VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models

ArXiv: 2412.01822

🎯 Pitch

Small VLMs can match or beat GPT-4V—without scaling—by learning to verbalize their internal layer-wise reasoning in natural language. A distilled 7B model reaches 75.8% on MM-Vet, surpassing proprietary giants like Claude-3.5 and Gemini-1.5-Pro.

---

1. Executive Summary

This paper introduces VLsI: Verbalized Layers-to-Interactions, a new family of efficient vision-language models in 2B and 7B sizes that transfers reasoning knowledge from large to small VLMs through a novel natural language-based, layer-wise distillation process. The approach proceeds in three stages — verbalization (using "verbalizers" to project intermediate layer features into natural language space via a feed-forward network and shared language head), interaction (adaptive layer matching between large and small backbone VLMs using KL divergence-based multinomial sampling with order preservation), and supervised finetuning (aligning the distilled model for instruction-following) — establishing that small VLMs can flexibly align their layer-wise reasoning progression with that of larger models without scaling, merging, or architectural changes. Evaluated across ten challenging vision-language benchmarks including MM-Vet, MMMU, MathVista, and AI2D, VLsI achieves 11.0% (2B) and 17.4% (2B model on MM-Vet) — correction: 11.0% and 17.4% performance gains over GPT-4V, with the 7B model reaching 75.8% on MM-Vet and 69.3% on MMMU, outperforming proprietary closed-source VLMs including GPT-4V, Claude-3.5-Sonnet, and Gemini-1.5-Pro, while the FLOPs-matched analysis demonstrates that the distilled small model can surpass a ~14× larger pretrained model, establishing that natural language serves as an effective medium for transferring internal reasoning knowledge between VLMs only when layer-wise alignment respects both order preservation and adaptive matching.

2. Context and Motivation

The Core Problem: Small VLMs Cannot Match Large VLMs Without Expensive Architectural Changes

The fundamental question this paper tackles is deceptively simple: can we get a small vision-language model (VLM) to perform like a large one without physically scaling up the model? As shown in Section 1, closed-source VLMs like GPT-4V, Claude-3.5-Sonnet, and Gemini-1.5-Pro achieve remarkable performance on vision-language benchmarks — but they are enormous models running in datacenters. The paper's opening example in Figure 1a illustrates the stark reality: on MM-Vet, model accuracy climbs steeply with parameter count, creating a chasm between the performance of small models (2-7B parameters) and their larger counterparts (72B+).

This gap is significant for several practical reasons the paper surfaces throughout Sections 1 and 2:

• Resource-constrained deployment: "scaling VLMs to improve performance using larger models brings significant computational challenges, especially for deployment on resource-constrained devices like mobile platforms and robots" (Section 1). A 72B-parameter model is simply not feasible on a smartphone or autonomous robot, yet those are precisely the platforms where VLMs would be most transformative — enabling real-time visual understanding without cloud dependency.

• Inference latency and cost: Large VLMs require substantial GPU memory and produce tokens slowly. In latency-sensitive applications (assistive technology, autonomous navigation, real-time video understanding), waiting seconds for a response from a 72B model may be unacceptable, regardless of accuracy.

• The scaling wall: As open-source VLMs have raced to match closed-source performance — with releases like LLaVA-OneVision [56], InternVL2 [12], Qwen2-VL [92], and Molmo [20] all pushing toward 72B+ sizes — the computational cost of training and deploying these models has become prohibitive for most organizations. Scaling alone, without efficiency innovations, is not a sustainable path forward.

Prior Approaches: Adding Modules, Modifying Architectures, and Simple Distillation — All with Drawbacks

The paper identifies three broad families of prior approaches to building efficient VLMs, each with specific shortcomings that motivate the need for a new method.

Approach 1: Specialized modules and architectural modifications. Several recent works have attempted to enhance small VLM performance by adding components or altering the model structure. The paper specifically mentions TroL [50] and Phantom [49] as representative examples (Section 2, "Efficient Modeling Strategy"):

"TroL and Phantom aim to expand learning capabilities within limited structures by doubling forward propagation steps and enlarging the latent dimension without physically increasing model sizes, thereby showing large improvements."

However, these approaches introduce significant engineering complexity. As the paper notes:

"these approaches face limitations such as key-value cache storage constraints and extensive architectural modifications, which may hinder direct application to real-world scenarios"

TroL requires "doubling forward propagation steps," meaning each inference pass effectively costs twice as much compute. Phantom enlarges the latent dimension — increasing memory requirements during inference even though the parameter count stays the same. Both modify the model's forward pass in ways that break compatibility with standard deployment frameworks (e.g., inference optimizers like vLLM or TensorRT expect conventional transformer architectures). For on-device deployment where every millisecond of latency and every megabyte of memory matters, these architectural departures are especially problematic.

The paper also references approaches that merge additional visual encoders [26, 46, 80] or integrate multiple computer vision models [13, 22, 76, 97] into LLMs, as well as methods like Meteor [51] that add a "rational projector" embedding multifaceted reasoning information. While effective, these all increase model complexity and footprint.

Approach 2: Lightweight LLM design with corresponding lightweight VLMs. On the extreme efficiency end, works like MobiLlama [88], OpenELM [75], and MobileLLM [68] have designed small LLMs from scratch, using "shared feed-forward network (FFN) design, layer-wise scaling, and embedding-language head sharing" (Section 2). These are then used as backbones for VLMs like MobileVLM [14], MobileVLM-V2 [15], and MoE-LLaVA [62].

The paper's critique is sharp:

"Their primary objective is not to close the gap with closed-source LLMs but rather to reduce parameters impacting less performance degradation."

In other words, these methods accept a performance ceiling in exchange for parameter efficiency. They are about making a small model less bad, not making a small model surprisingly good. The gap to large models remains large.

Approach 3: Standard knowledge distillation — final-layer only. Several works have applied knowledge distillation to VLMs specifically: LLaVA-MoD [83], LLaVA-KD [4], Align-KD [28], and MoVE-KD [6]. The paper's critique is that these methods share a common limitation:

"The other studies simply use the final layer distillation."

This means only the final output distribution (logits) of the large model is used to train the small model. The internal reasoning progression — how the large model builds up its understanding across its 80 layers — is completely discarded. The small model is asked to output the right answer without learning how to reach that answer. This is analogous to teaching a student only the answers to a test without showing the reasoning steps; it works for memorization but fails for generalization to new problems.

There is also an instability problem with direct output imitation. As the paper states in the abstract:

"This approach mitigates the training instability often encountered in output imitation"

When a small model is forced to precisely match the output distribution of a much larger model, the optimization can become unstable — the small model lacks the capacity to perfectly imitate, leading to noisy gradients and convergence issues.

A Deeper Gap: The Layer Number Mismatch Problem

Beyond these broad limitations, the paper identifies a more specific, underexplored technical challenge: the layer number gap between large and small VLMs. A large backbone VLM might have 80 transformer layers (e.g., Qwen2-VL-72B), while the small one has only 28 (e.g., Qwen2-VL-2B or 7B). If you want the small model to learn the "reasoning progression" of the large model — how understanding develops from shallow to deep layers — you need to map these 28 layers onto 80 in a meaningful way. Which large-model layer should the small model's 3rd layer learn from? The 5th? The 12th?

Standard distillation ignores this entirely because it only looks at the final layer. Prior layer-wise distillation methods like TED [61] attempted to use intermediate layers, but:

"it is not a version of LLM covering general tasks, and it does not handle layer number gap between large and small models"

TED was designed for task-specific compression, not for general-purpose VLM distillation with heterogeneous layer counts.

The Central Insight: Natural Language as a Medium for Layer-Wise Reasoning Transfer

The paper's positioning against this landscape is encapsulated in their framing question (Section 1):

"Can we achieve a similar or superior level of performance without scaling, merging, or architectural changes?"

Their answer is VLsI, which introduces a fundamental conceptual shift: use natural language as the communication medium between large and small VLMs at every intermediate layer. Rather than trying to align feature vectors in high-dimensional embedding space (which is what L2 or cosine-similarity-based distillation does), VLsI projects the meaning at each layer into interpretable text using a "verbalizer," then aligns the small model's layer-wise understanding with the large model's by comparing these textual probability distributions using KL divergence.

This is motivated by an explicit analogy to how humans transfer knowledge (Section 5):

"We hope that incorporating natural language will facilitate smoother communication between large and small VLMs, alleviating the complexities of feature alignment."

And more explicitly in the conclusion:

"We show VLsI achieves strong vision-language performances, suggesting that natural language is an important key in transferring knowledge not only for humans but also for AI."

The verbalizer architecture is inspired by the speculative decoding paradigm [59], which demonstrated that a smaller LLM constructed with a frozen word embedding and the language head of a larger LLM can effectively emulate the larger model's performance. VLsI extends this insight from the final output to every intermediate layer.

How This Paper Positions Itself

VLsI occupies a unique position in the VLM efficiency landscape:

• Not model scaling or merging: Unlike approaches that simply train larger models or ensemble multiple vision encoders, VLsI works with a fixed, unmodified backbone architecture. The verbalizers are lightweight (269M parameters for the verb-FFN design, as shown in Table 4f), and the backbone VLM itself is unchanged — no new attention heads, no modified layer norms, no additional vision encoders.

• Not architectural modification: Unlike TroL (double forward passes) or Phantom (enlarged latent dimension), VLsI is a training methodology, not a structural change. After distillation, the small model runs exactly like any standard VLM — same forward pass, same memory footprint, same inference optimizations apply.

• Not final-layer-only distillation: Unlike LLaVA-MoD, LLaVA-KD, or any prior VLM distillation work, VLsI aligns the full reasoning progression across layers. This goes beyond "learn to output the right answer" to "learn to think like the large model thinks at every stage of processing."

• Adaptive to different backbone pairs: VLsI works with arbitrary large-small backbone combinations (as shown in Figure 4 with Qwen2-VL and LLaVA-OV as large backbones, and 0.5B, 2B, and 7B as small backbones), as long as they share the same tokenizer and vocabulary ordering.

• Human-interpretable intermediate outputs: Because the verbalizers project to natural language, the intermediate states of the small model become interpretable — as demonstrated in Figure 3 and Appendix C with extensive verbalization examples showing how understanding evolves from vague descriptions ("The image depicts a large, small, and black color scheme") to precise answers ("star with a dot").

The paper makes clear that VLsI is not claiming to invent verbalization (which draws on speculative decoding work) or layer-wise distillation (which draws on TED), but rather to be the first to combine natural language-based intermediate layer alignment with adaptive, order-preserving layer matching specifically for VLMs, demonstrating that this combination closes a substantial portion of the gap between small and large VLMs without any architectural overhead at inference time.

3. Technical Approach

3.1 Reader Orientation

VLsI is a three-stage training pipeline that distills the internal reasoning progression of a large vision-language model (e.g., Qwen2-VL-72B with 80 layers) into a small VLM (e.g., Qwen2-VL-2B with 28 layers) without modifying either model's architecture at inference time. The core problem it solves is the layer number mismatch between teacher and student — with 80 layers mapping onto 28, there is no obvious one-to-one correspondence — and its solution is to project every intermediate layer's hidden state into natural language probability distributions using lightweight "verbalizer" modules, then align the student's layer-wise understanding to the teacher's by comparing these text distributions with KL divergence, adaptively matching which student layer learns from which teacher layer.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components, two of which are pre-existing (frozen during VLsI training) and two of which are trainable:

1. Large-backbone VLM (pre-trained, frozen) — the teacher. Using Qwen2-VL-72B, it contains 80 transformer decoder layers plus a vision encoder and projector. Its intermediate layers produce hidden states that encode progressively more refined visual-linguistic understanding.

2. Small-backbone VLM (pre-trained, frozen initially, then finetuned in Stage 3) — the student. Using Qwen2-VL-2B or Qwen2-VL-7B, each with 28 layers. It shares the same tokenizer and vocabulary ordering as the large model, which is critical: the token indices mean the same thing in both models, so probability distributions over tokens are directly comparable.

3. Verbalizers (trainable, one per intermediate target layer in each backbone) — lightweight modules consisting of a feed-forward network ("verb-FFN") followed by the backbone's own language head. A verbalizer takes the hidden state from a specific intermediate layer and projects it into a probability distribution over the vocabulary — effectively asking "what would the model say if it had to produce text at this layer?" For the large backbone, verbalizers are placed at layers 2, 6, 10, ..., 78 (every 4th layer, totaling 20 target layers). For the small backbone, verbalizers are placed at layers 2, 6, 10, ..., 26 (also every 4th layer, totaling 7 target layers).

4. Layer matching policy (not a component but a procedure) — a dynamic, order-preserving algorithm that decides, for each target layer in the small model, which target layer in the large model it should learn from. The matching is based on KL divergence between the verbalized distributions and uses multinomial sampling with an adaptive temperature to encourage exploration.

Information flow during training proceeds in three sequential stages:

• Stage 1 — Verbalization: Verbalizers are trained independently for each target layer in each backbone using autoregressive loss. The goal is to make each verbalizer produce text close to the ground-truth answer when given the layer's hidden state. Weights of both backbones remain frozen. This stage produces verbalizers that can "translate" any intermediate hidden state into a vocabulary distribution.

• Stage 2 — Interaction: Using the trained verbalizers, each small-model target layer is matched to a large-model target layer via the adaptive matching algorithm. Then, for each matched pair, the KL divergence between their verbalized distributions is computed and minimized. Only the small model's verb-FFNs (not the language head) are updated. The backbone weights remain frozen. This stage transfers layer-wise reasoning knowledge from large to small.

• Stage 3 — Supervised Finetuning (SFT): The entire small-backbone VLM (including word embeddings, attention, FFN, and language head) is finetuned on the visual instruction dataset using standard autoregressive loss. The verbalizers are discarded. This stage lets the model integrate the distilled knowledge and adapt to instruction-following tasks.

At inference time, the small-backbone VLM runs completely normally — no verbalizers, no extra forward passes, no architectural modifications. The distillation is absorbed into the model's weights.

3.3 Roadmap for the Deep Dive

The detailed breakdown below follows the chronological order of the training pipeline, since each stage depends on the output of the previous one:

• First, the backbone VLM selection and verbalizer architecture — what models are used, what their layer structure looks like, and what exactly a verbalizer is (the verb-FFN and language head). This establishes the physical components that the three training stages operate on.

• Second, the verbalization step (Stage 1) — how verbalizers are trained to project hidden states to natural language, what loss is used, what data is used, and why autoregressive loss rather than, say, contrastive learning. This step creates the "translation layer" that makes the interaction step possible.

• Third, the interaction step (Stage 2) — the adaptive layer matching algorithm (Algorithm 1 in the paper), the KL divergence loss, the multinomial sampling distribution, order preservation, search range, and adaptive temperature. This is the core intellectual contribution of the paper: how to match layers between models of different depths in a principled way.

• Fourth, the supervised finetuning step (Stage 3) — why additional SFT is necessary even after the interaction step, how it differs from standard visual instruction tuning, and the evidence (from Table 4b) showing that interaction alone is not enough without SFT. This step connects the distilled knowledge to actual task performance.

• Fifth, design choices and ablation justifications — why KL divergence over cross-entropy or L2 for layer alignment, why verb-FFN over simpler MLP or more complex decoder architectures, why the interaction and SFT steps are separated rather than jointly optimized, and why beam decoding works better than greedy decoding before SFT.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an empirical methods paper whose core idea is that natural language serves as an effective intermediate representation for distilling the layer-wise reasoning progression of a large VLM into a small one, and that adaptive, order-preserving layer matching with KL divergence-based sampling is the key mechanism that makes this transfer work.

---

Backbone VLM Selection and Verbalizer Architecture

The paper selects Qwen2-VL [92] as the backbone VLM family for both teacher and student. This is a deliberate choice with specific properties that enable VLsI:

• Consistent vocabulary across sizes. Qwen2-VL offers models at 1.5B, 7B, and 72B parameters, all sharing the same Qwen2 tokenizer [96] with identical vocabulary indices. The paper explicitly states: "the tokenizer's vocabulary indices remain consistent across these model sizes, allowing for seamless integration without reordering the vocabulary." This property is critical because the verbalizers produce probability distributions over token indices — if index 3542 meant "cat" in the 72B model but "dog" in the 2B model, aligning distributions would be meaningless. The shared vocabulary ensures that comparing token probabilities between large and small models is semantically coherent.

• Identical vision encoder and projector architecture. All Qwen2-VL variants use the same vision encoder (a ViT [21] adapted from DFN [25] with visually-adapted rotary positional embeddings [86]) and the same vision projector (an MLP with two fully-connected layers interleaved with GELU activations [37]). This means the visual features entering the LLM backbone are produced by the same architecture, eliminating vision-side distribution shift as a confounding factor in the distillation experiments.

• Known layer counts. The Qwen2-1.5B and Qwen2-7B each contain 28 transformer decoder layers, while Qwen2-72B contains 80 layers. The 80→28 layer number gap is substantial (~2.86×), making this a challenging test case for the layer matching algorithm.

The paper also experiments with LLaVA-OneVision [56] as an alternative large backbone (Figure 4), which uses the same Qwen2 language model but differs in its image processing — it requires 4-10× more image tokens than Qwen2-VL for the same pixel count, depending on the image's aspect ratio. This changes which intermediate layers serve as verbalization targets (4th, 12th, and 20th for LLaVA-OV-7B; 10th, 30th, 50th, and 70th for LLaVA-OV-72B) and reduces the number of target layers due to GPU memory constraints.

Target layer selection. The paper does not attach verbalizers to every layer. Instead, it selects target layers at evenly spaced intervals: every 4th layer starting from the 2nd layer. For the 28-layer small backbone, this yields target layers at indices {2, 6, 10, 14, 18, 22, 26} — 7 target layers total. For the 80-layer large backbone, this yields target layers at indices {2, 6, 10, ..., 78} — 20 target layers total. The choice of starting at layer 2 (0-indexed) rather than layer 0 reflects a practical observation: the very first transformer layer typically encodes mostly positional and low-level features, and verbalizing at layer 0 produces incoherent output. The choice of every 4th layer is a computational tradeoff — denser sampling (e.g., every 2nd layer) would give finer-grained reasoning tracking but requires more GPUs and training time. The paper does not ablate this spacing choice.

Verbalizer architecture: verb-FFN + language head. A verbalizer at a given layer consists of two components placed in sequence:

1. verb-FFN — a trimmed-down feed-forward network without the standard expansion-reduction cycle. A conventional transformer FFN first expands the hidden dimension (typically by 4×), applies a gating mechanism, then reduces back to the original dimension. The verb-FFN foregoes this, maintaining the same hidden dimension throughout. The paper explains: "To enhance computational efficiency in verbalization and interaction, we opt to maintain a consistent hidden dimension throughout the process, foregoing the typical expansion and reduction steps." This means the verb-FFN is essentially a sequence of linear transformations that reshape the hidden state without changing its dimensionality, making it more efficient than a full FFN while still providing enough capacity to adapt the intermediate representation for the language head.

2. Language head — the same linear layer (followed by softmax) that the backbone VLM uses at its final layer to produce token probabilities. By reusing the backbone's language head rather than training a separate one, the verbalizer's output distribution is directly in the same semantic space as the model's final output — there is no representational mismatch between verbalized distributions at different layers because they all share the same "output vocabulary."

The verb-FFN is the only trainable component during verbalization (the language head is frozen, taken directly from the backbone). During interaction, the small model's verb-FFNs are updated, while the large model's verb-FFNs remain frozen.

Parameter count. Table 4(f) compares different verbalizer architectures in terms of parameter efficiency:

• A verb-FFN (single) adds 269M parameters — modest compared to the base model's ~2B or ~7B.
• Doubling the verb-FFN ("verb-FFN×2") adds 539M but yields only marginal performance gains (e.g., 75.8 → 75.7 on MM-Vet for 7B, a decline of 0.1). The paper concludes the single verb-FFN is sufficient.
• Simpler alternatives like a basic MLP (90M) or MLP×2 (180M) add fewer parameters but perform substantially worse (73.5 and 74.1 on MM-Vet respectively, compared to 75.8 for verb-FFN). This shows that the FFN's gating structure — even without expansion — provides a useful inductive bias for projecting hidden states to language space.
• More complex alternatives like a full decoder layer (1.6B) or FFN×2 (2.9B) achieve similar or slightly worse performance (76.0 and 75.9 on MM-Vet) while adding 6-11× more parameters, making them less practical.

The verb-FFN therefore sits at a sweet spot: powerful enough to project hidden states meaningfully into language space, but lightweight enough to be trained for every target layer without overwhelming GPU memory.

Why verbalization at all? A natural question is: why not directly distill intermediate features using L2 or cosine similarity in the embedding space, without the language head? The paper addresses this in an ablation (Appendix D, the "vL-Head" experiment). When the language head is removed from the verbalizer and the verb-FFNs of large and small models are aligned with L2 loss directly, performance drops dramatically: from 75.8 to 63.3 on MM-Vet and from 69.3 to 53.5 on MMMU for the 7B model. The paper's interpretation:

"directly imitating outputs from the large-backbone VLM, without verbalization from vL-Head, introduces instability and can lead to suboptimal results, highlighting the critical role of the language head in achieving effective distillation."

This suggests that hidden states in different models occupy different "representational neighborhoods" — even with identical architecture, the 72B model's layer-12 hidden state and the 2B model's layer-12 hidden state may represent similar information in geometrically incompatible ways. Projecting both through a verbalizer to a shared vocabulary space acts as a normalization step, transforming both into the common language of token probabilities that can be meaningfully compared with KL divergence.

---

Stage 1: The Verbalization Step

This stage answers the question: what does each intermediate layer of a frozen VLM "know" in natural language terms? The goal is not to produce correct answers from intermediate layers — shallow layers inherently lack the information to answer correctly — but to establish a mapping from each layer's hidden state to a vocabulary distribution that captures whatever partial understanding has developed at that depth.

Training setup. For each target layer in each backbone (7 layers for the small model, 20 for the large model), a separate verbalizer is trained independently. All backbone VLM parameters are frozen — only the verb-FFN parameters are updated. The training data is the same 2.9M visual instruction tuning samples described in Section 4.1(d), covering diverse vision-language tasks.

Loss function. The verbalizer at layer $\ell$ receives the hidden state $\mathbf{h}_\ell$ (the output of transformer layer $\ell$ for a given image-text input pair) and produces, via the verb-FFN and language head, a probability distribution over the vocabulary:

$P_\ell(w \mid \mathbf{h}_\ell) = \text{Softmax}(\text{LM-Head}(\text{verb-FFN}_\ell(\mathbf{h}_\ell)))$

The autoregressive loss is then applied token-by-token against the ground-truth target response $y = (y_1, y_2, ..., y_T)$:

$\mathcal{L}_{\text{verb}}(\ell) = -\sum_{t=1}^{T} \log P_\ell(y_t \mid \mathbf{h}_\ell, y_{<t})$

where $\mathbf{h}_\ell$ is the hidden state at layer $\ell$ for the given input, $y_t$ is the $t$-th token of the target response, and $y_{<t}$ denotes previously generated tokens used as context for autoregressive decoding.

What it computes: the standard next-token prediction loss (cross-entropy), but applied at an intermediate layer rather than at the final output. The verbalizer sees the hidden state from layer $\ell$ and must predict, one token at a time, the full ground-truth response. For layer 2 (shallow), this is extremely difficult — the model has barely processed the image and text — so the loss will be high and the verbalized text will be vague or repetitive (as shown in Figure 3: "The image depicts a large, small, and black color scheme"). For layer 26 (deep), the hidden state contains much more information, so the loss will be lower and the verbalized text will be more accurate (Figure 3: "a star with a dot"). The gradient from this loss flows only through the verb-FFN, teaching it how to extract whatever information is available at layer $\ell$ and format it as coherent language.

Why this form: autoregressive loss with the ground-truth target (not with the large model's "correct" output) forces each verbalizer to learn the best possible mapping from its layer's hidden state to the correct answer. An alternative would be to have the verbalizer predict the large model's final output distribution — but this would teach the verbalizer to imitate the teacher's mistakes. By using the ground-truth target, the verbalizer learns to extract genuine information from the hidden state, creating a "honest" representation of what each layer actually knows.

A critical detail: the weights of different verbalizers are independent. The gradient update for the verb-FFN at layer 2 does not affect the verb-FFN at layer 6. This means each verbalizer is optimized to make the best possible prediction given its specific layer's hidden state, with no cross-layer interference. The paper does this because the layers have qualitatively different information content — a verbalizer that works well for layer 26 (which has rich semantic information) would not necessarily work for layer 2 (which has mostly low-level features), so sharing parameters across layers would create a harmful tradeoff.

Verbalization loss progression. Appendix B of the paper shows how the verbalization loss decreases during training for each of the 7 target layers. The pattern is revealing: at 10% training progress, layer #1 (the 2nd transformer layer) has a loss of 14.28, while layer #7 (the 26th layer) has a loss of 7.39. By 100% training, these drop to 3.30 and 0.79 respectively. The monotonic decrease with layer depth (shallower layers always have higher loss than deeper ones) confirms that deeper layers genuinely contain more information about the correct answer — the verbalizer is not "inventing" information, but extracting what the model's forward pass naturally accumulates.

Verbalization examples (Figure 3 and Appendix C). The paper provides extensive examples of verbalized outputs at each layer. For a visual reasoning question asking "What would the missing image look like?" in a sequence of star patterns, the outputs progress as follows:

• Shallow layers (2nd, 6th): "The image shows a large, white background with a white background, which is a large, white background" — repetitive, incoherent, capturing only low-level visual features.
• Mid layers (10th, 14th, 18th): "The missing image would likely be a square with a small circle inside, similar to the first image but with a small circle in the center" — starting to reason about patterns, but incorrect.
• Deep layers (22nd, 26th): "The missing image would likely be a star shape similar to the one shown in the top right corner" — correct reasoning emerges.

This demonstrates the core premise of VLsI: reasoning develops gradually across layers, and verbalization makes this progression visible and directly comparable between models.

---

Stage 2: The Interaction Step

This stage is the heart of VLsI. Given that the large model (80 layers, 20 target layers) and the small model (28 layers, 7 target layers) each have verbalizers producing vocabulary distributions at their respective target layers, the interaction step must decide which large-model layer each small-model layer should learn from, and then minimize the difference between their verbalized distributions to transfer the large model's reasoning progression into the small model.

The paper's key insight is that this matching problem is nontrivial and requires a principled algorithm rather than a simple heuristic (like uniform spacing or always matching to the layer with the smallest KL divergence in isolation).

Layer matching constraints. The algorithm (Algorithm 1 in the paper) enforces two key properties:

1. Order preservation. If the small model's $i$-th target layer is matched to the large model's $j$-th target layer, then the small model's $(i+1)$-th target layer must be matched to some large-model layer $k$ where $k > j$. This ensures the small model learns a monotonically deepening reasoning progression — its layer 6 cannot learn from an earlier large-model layer than its layer 2 learned from. The paper explains: "the matched layer j (large-backbone VLM) of layer i (small-backbone VLM) should be deeper than the matched layer k of layer i-1, ensuring j > k."

2. Search range. For the small model's $i$-th target layer, the search is constrained to large-model target layers in the range $[i^*_{l}, tl - ts + i]$, where $tl = 20$ is the number of large-model target layers, $ts = 7$ is the number of small-model target layers, and $i^*_{l}$ is the large-model layer matched to the previous small-model layer (initialized to 0). This range ensures there are enough remaining large-model layers for the subsequent small-model layers. For example, when matching the small model's 2nd target layer (i=1, 0-indexed), the search range is $[0, 20 - 7 + 1] = [0, 14]$ — it cannot match to layers 15-19, because those need to be reserved for the remaining 5 small-model target layers.

The matching criterion: KL divergence-based multinomial sampling. For each candidate large-model layer $il$ within the search range, the algorithm computes the KL divergence between the verbalized distributions of the small-model layer $is$ and the large-model layer $il$:

$D_{\text{KL}}(P_{\text{small}}^{is} \| P_{\text{large}}^{il}) = \sum_{w} P_{\text{small}}^{is}(w) \log \frac{P_{\text{small}}^{is}(w)}{P_{\text{large}}^{il}(w)}$

where $P_{\text{small}}^{is}(w)$ and $P_{\text{large}}^{il}(w)$ are the vocabulary probabilities produced by the verbalizers at the respective layers for the same input, summing over all tokens $w$ in the vocabulary.

What it computes: a measure of how different the two probability distributions are. KL divergence is zero when the distributions are identical, and large when they differ substantially. For layer matching, a small KL divergence means the small model's layer $is$ and the large model's layer $il$ have similar "understanding" as measured by their verbalized outputs — they would produce similar text if asked.

Why KL divergence: the paper considers and ablates alternatives (Table 4d):

• Cross-entropy (CE): when used for both intermediate and last layers, CE achieves only 77.8 on MMB and 63.2 on MM-Vet for the 7B model. CE is asymmetric (it penalizes the small model for being uncertain where the large model is certain, but not the reverse), which makes it less suitable for matching — two layers could have similar understanding but CE could be large if the small model is simply less confident.
• L2 distance: between intermediate features (without verbalization) achieves 81.5 on MMB and 66.8 on MM-Vet — better than CE but worse than KL divergence. L2 operates in the continuous embedding space where distance metrics are less semantically meaningful.
• KL divergence for intermediate, KL divergence for last layer: achieves 86.3 on MMB and 75.8 on MM-Vet — the best configuration. The paper finds that using KL divergence for both intermediate matching AND final-layer distillation (simultaneously) yields the best results, suggesting that the final layer also benefits from distribution matching in vocabulary space.

The paper's position on the CE vs. KL divergence debate in distillation is nuanced. Prior work is split: DistiLLM [47] and MiniLLM [36] advocate for reverse KL divergence, while LLAVADI [95] and Minitron [77] find normal KL divergence more effective. VLsI aligns with the latter camp, using forward KL divergence $D_{\text{KL}}(P_{\text{small}} \| P_{\text{large}})$.

Multinomial sampling with adaptive temperature. Instead of greedily selecting the large-model layer with the smallest KL divergence (which would be deterministic and potentially suboptimal — the "best" match in isolation might block better matches for later layers), the algorithm samples from a distribution that favors low-KL layers but maintains exploration. The sampling distribution is constructed as:

$p(\text{layer } il) = \text{Softmax}\left(-\text{kld-list}[il] / T\right)$

where $\text{kld-list}[il]$ is the KL divergence for candidate layer $il$, and $T$ is an adaptive temperature computed as:

$T = \frac{\text{scale}}{\text{kld-list}_{\max} - \text{kld-list}_{\min} + \epsilon}$

with $\text{scale} = 2$ and $\epsilon = 10^{-6}$.

What it computes: a probability distribution over candidate large-model layers where (a) layers with smaller KL divergence get higher probability, and (b) the temperature $T$ adapts to the spread of KL divergences in the current search range. When all candidates have similar KL divergence (narrow spread), $T$ is large, making the distribution nearly uniform — encouraging exploration because no candidate is clearly better. When one candidate has much lower KL divergence (wide spread), $T$ is small, making the distribution sharp — encouraging exploitation of the clearly best match.

Why adaptive temperature over a fixed temperature or greedy selection: a fixed temperature would not respond to the varying "decisiveness" of the KL divergence landscape at different matching stages. The adaptive temperature automatically balances exploration and exploitation: early in matching (when the first few layers are being matched and the small model hasn't learned anything yet), KL divergences might be uniformly large, so exploration is useful. Later, when the small model has started to align and some matches are clearly better, exploitation becomes more valuable.

The paper ablates this design choice in Table 4(e). The components are added incrementally:

• Random Index (random matching): 77.0 MMB, 62.0 MM-Vet — the weakest, showing the need for principled matching.
• Uniform Index (evenly spaced matching, mapping small layer $i$ to large layer $i \times \text{floor}(tl / ts)$): 79.5 MMB, 66.5 MM-Vet — better but still suboptimal.
• Bottom-1 Index (greedy, always picking the lowest KL divergence): 81.2 MMB, 67.8 MM-Vet.
• Bottom-3 Index (averaging the 3 lowest KL divergences): 81.5 MMB, 68.0 MM-Vet.
• Multinomial Sampling (probabilistic selection from the full distribution): 82.0 MMB, 68.5 MM-Vet — a small improvement from exploration.
• + Search Range: 83.5 MMB, 69.8 MM-Vet — a large jump, showing that constraining the search to preserve future matching options is critical.
• + Order Preservation: 86.0 MMB, 75.2 MM-Vet — another large jump, the most impactful single component.
• + Adaptive Temperature (full VLsI): 86.3 MMB, 75.8 MM-Vet — the final configuration, adding a modest but consistent gain.

The trend is clear: each component of the matching strategy matters, but order preservation and search range are the two most critical. Without order preservation, the small model can learn a "scrambled" reasoning progression (e.g., layer 6 learns from a shallower large-model layer than layer 2), which apparently degrades the quality of the transferred knowledge.

Distribution shift during interaction training. Figure 5 visualizes how the matched indices evolve over the course of interaction training. At the beginning (early training), the small model's layers match to a relatively uniform spread of large-model layers — the matching is exploratory. By the end, the distribution shifts rightward: the small model's layers increasingly match to deeper large-model layers. The paper interprets this as:

"the small-backbone VLM gradually tries to learn about deeper layers' responses of the large-backbone VLM, which can be considered accelerating the process of reaching an answer."

This emergent behavior suggests the interaction step is doing something non-obvious: it's not just matching at fixed positions and then freezing, but rather allowing the matching to adapt as the small model's verbalizers improve, pushing the alignment toward deeper, more informative large-model layers.

Interaction loss. The total interaction loss is the sum of the KL divergences at the matched layer pairs (line 11 of Algorithm 1). The loss is backpropagated only through the small model's verb-FFNs — the large model's verbalizers and both backbones remain frozen. This means the interaction step updates only the small model's verbalizers to better match the large model's verbalized distributions.

Why freeze the backbones during interaction? The paper does not explicitly motivate this, but the logic is implied by the three-stage design: if backbone parameters were updated during interaction, the verbalizers trained in Stage 1 would become stale (their hidden states would change). By freezing the backbone, the interaction step can safely assume the verbalizers remain accurate translators of their respective layers' knowledge.

What about last-layer distillation? The paper experiments with simultaneously applying KL divergence loss at the final layer (not just at intermediate matched layers) — this is what the "KLD + KLD" configuration in Table 4(d) refers to, where both intermediate layers and the last layer use KL divergence. The results show this is beneficial (86.3 MMB vs. 83.0 for intermediate-only KLD), suggesting the final layer captures some complementary information not fully covered by intermediate layer alignment.

---

Stage 3: Supervised Finetuning (SFT)

After the interaction step, the small model has been "injected" with the large model's reasoning progression through the verb-FFNs — but the backbone LLM weights (attention, FFN, embeddings) have not been updated since the interaction step only touched the verb-FFNs. The SFT step finetunes the entire small-backbone VLM on the visual instruction dataset using standard autoregressive loss.

Why is SFT necessary? Table 4(b) provides the critical evidence:

• Qwen2-VL-7B baseline: 83.0 MMB, 62.0 MM-Vet, 54.1 MMMU.
• VLsI-7B without SFT (greedy decoding): 82.1 MMB, 60.5 MM-Vet, 52.9 MMMU — worse than the baseline! This is a striking result: the interaction step alone, without SFT, actually degrades performance slightly.
• VLsI-7B without SFT (beam decoding, N=5): 85.2 MMB, 70.9 MM-Vet, 63.5 MMMU — substantially better than the baseline.
• VLsI-7B with SFT (greedy decoding): 86.3 MMB, 75.8 MM-Vet, 69.3 MMMU — the best result.

The paper's interpretation of this pattern is insightful:

"This suggests that the interaction step implicitly alters and expands the probability space over the language head, while SFT aligns this expanded space to better fit instruction-following."

In other words, the interaction step enriches the model's internal representations (the "probability space" of possible outputs) by transferring the large model's reasoning patterns, but this enrichment comes at a cost: the model's output distribution becomes less peaked — it considers more diverse possibilities, which is good for exploration but hurts greedy decoding (which simply picks the most likely token). Beam search with N=5 recovers the benefit by exploring multiple paths through this enriched space. The SFT step then realigns the distribution to make correct answers more likely under greedy decoding, "sharpening" the enriched probability space toward instruction-following behavior.

This is analogous to the pruning-and-retraining paradigm in [77, 85], where pruning (removing weights) degrades performance and requires additional training to recover. In VLsI, the interaction step is not physically pruning but conceptually similar — it modifies the model's internal dynamics in a way that initially hurts under greedy decoding, and subsequent SFT is needed to harmonize the new knowledge with task performance.

SFT training configuration. The SFT step uses:

• The same 2.9M visual instruction tuning samples.
• Full-model finetuning (not LoRA) of the small-backbone VLM.
• Autoregressive loss only (no KL divergence).
• The same optimizer setup: AdamW [69] with cosine annealing from $1 \times 10^{-4}$ to $1 \times 10^{-6}$.

How much SFT is needed? Table 4(c) shows that the benefits scale with SFT training duration. At 50% of full SFT training:

• VLsI-7B at 50%: 85.4 MMB, 70.0 MM-Vet — already above the "without SFT + beam" baseline.
• VLsI-7B at 100%: 86.3 MMB, 75.8 MM-Vet — further gains.

The baseline Qwen2-VL-7B, trained on the same data for the same duration, shows much smaller improvements (80.5 → 81.8 MMB, 62.0 → 63.7 MM-Vet at 100%). This demonstrates that the SFT step is not simply benefiting from the additional 2.9M samples — it's specifically unlocking the knowledge that was transferred during the interaction step.

Why separate interaction and SFT steps rather than joint optimization? The paper implicitly answers this through the ablation in Table 4(d) (the "CE" columns). When cross-entropy (autoregressive loss) is used during the interaction step instead of KL divergence, performance degrades (from 86.3 to 81.5 MMB and from 75.8 to 68.5 MM-Vet for the KLD intermediate + CE last layer combination). The paper states:

"We avoid optimizing autoregressive loss during the interaction step (where KL divergence loss is optimized instead) as this can degrade performance."

The likely explanation: autoregressive loss is a strong signal that pulls the model toward producing the correct answer. During the interaction step, the goal is not to produce correct answers but to align internal representations. If autoregressive loss were applied simultaneously, it would dominate the KL divergence loss and effectively skip the alignment — the model would learn to output correct answers by memorizing the training data, not by transferring the teacher's reasoning patterns. By separating the stages, the interaction step can focus purely on knowledge transfer without being distracted by end-task optimization.

---

Design Choices and Ablation Justifications

Several non-obvious design choices emerge from the experiments and deserve explicit discussion.

Why natural language probability distributions rather than feature vectors? The verbalizer projects hidden states to vocabulary distributions, and KL divergence compares these distributions. This choice has several properties that matter:

1. Dimensionality reduction. The hidden state has dimension $d_{\text{model}}$ (typically 4096 or larger), while the vocabulary has ~150K tokens. Comparing 150K-dimensional probability distributions is cheaper than aligning 4096-dimensional continuous vectors, especially when using KL divergence which is a scalar summary.

2. Semantic interpretability. A KL divergence of 0.5 between two layers' verbalized distributions has an intuitive meaning: the two layers would disagree about the next token 50% of the time (roughly). A Euclidean distance of 3.2 between two 4096-dimensional vectors has no such intuitive interpretation.

3. Mode-seeking vs. mean-seeking. KL divergence $D_{\text{KL}}(P_{\text{small}} \| P_{\text{large}})$ is zero-forcing (mode-seeking): it heavily penalizes the small model for putting probability mass where the large model has none, but is more forgiving of the reverse. This is appropriate for distillation — if the large model is confident that a token is impossible, the small model should learn that too. The paper does not explicitly discuss this property, but it's consistent with their finding that forward KL outperforms cross-entropy for intermediate alignment.

Why separate verbalization for each layer? Could a single verbalizer be trained to accept a layer index as input? The paper does not explore this, but the independent-verbalizer design has two advantages: (1) each verbalizer can specialize to its layer's specific information content, and (2) during interaction, only the small model's verbalizers are updated — if verbalizers were shared, updating one would affect all layers.

Why not jointly train verbalization and interaction? The paper trains verbalizers first, then fixes them during interaction. An alternative would be end-to-end training where verbalizers and layer matching are optimized together. The paper's staged approach likely reflects practical stability concerns: if verbalizers are not yet trained, the KL divergence between layers is uninformative (both would output near-uniform distributions), making layer matching impossible. By training verbalizers first, the interaction step starts from a meaningful signal.

Why the specific scaling of interaction loss (scale=2, epsilon=1e-6)? These are heuristic choices justified by the ablation in Table 4(e). The scale parameter $2$ controls the overall temperature range — a larger scale would make the distribution more uniform (more exploration), a smaller scale would make it sharper (more exploitation). The paper does not sweep this parameter, so $2$ may not be optimal, but the ablation shows the adaptive temperature (which depends on this scale) provides a small but consistent gain over a likely fixed temperature.

Why the specific target layer selection (every 4th layer starting from 2nd)? The paper does not ablate this choice. The starting point (layer 2 rather than layer 0) is motivated by the observation that the very first layer produces incoherent verbalizations. The spacing (every 4th) is a computational constraint — with 8 A100 80GB GPUs, training verbalizers for 7 layers in the small model and 20 in the large model is feasible, but denser sampling (e.g., every 2nd layer) would require 14 and 40 verbalizers respectively, likely exceeding memory limits. The paper does not claim the 4-layer spacing is optimal.

Why LoRA during interaction but full finetuning during SFT? The interaction step uses LoRA [40] with rank=64 and alpha=64 — this is a practical choice to limit memory usage, since both the large and small models (with their verbalizers) must be loaded simultaneously. The SFT step uses full finetuning because only the small model is needed (the large model is discarded), and full finetuning can better absorb the transferred knowledge into all model parameters.

Generalization to other backbone pairs. Figure 4 shows that VLsI works with different backbone families: Qwen2-VL-72B as teacher and LLaVA-OV-0.5B/7B as students, or LLaVA-OV-72B as teacher and Qwen2-VL-2B/7B as students. The performance patterns are consistent: using a stronger teacher always helps, and the gain is larger when the student is weaker (0.5B: 31.4 → 49.9 MMMU with LLaVA-OV-72B teacher; 7B: 48.8 → 59.1 MMMU). The absolute performance depends on the teacher quality — Qwen2-VL-72B is a stronger teacher than LLaVA-OV-72B (74.0 vs. 63.7 on MM-Vet), and this advantage propagates to the student models.

The constraint: shared tokenizer and vocabulary ordering. The paper explicitly notes this limitation in Section 4.4:

"While VLsI is highly effective, the large and small-backbone VLMs must share the same tokenizer and token index order when constructing VLsI. We will explore more general ways that accommodate different tokenizers and token index orders, potentially expanding VLsI's applicability and scalability."

This constraint arises because the verbalized probability distributions are over token indices — if index N means different tokens in the two models, the KL divergence between distributions is meaningless. The paper's experiments work around this by using Qwen2-VL (all sizes share Qwen2 tokenizer) and LLaVA-OV (which uses Qwen2 as its language model, hence the same tokenizer). Extending VLsI to, say, a LLaMA-based student and a Qwen-based teacher would require solving the vocabulary alignment problem — potentially through a learned mapping between tokenizers.

4. Key Insights and Innovations

Innovation 1: Natural Language as a Normalization Substrate for Layer-Wise Distillation

The paper's most conceptually distinctive move is the claim that natural language vocabulary distributions, rather than continuous feature vectors, should serve as the common representation for distilling reasoning progression across layers. This is not merely an engineering choice — a verbalizer is "more interpretable" than an L2-aligned feature — but a fundamental diagnostic about why prior intermediate-layer distillation methods struggled.

Before VLsI, the dominant paradigm in layer-wise distillation (in the limited work that attempted it, such as TED [61]) was to align hidden states or attention patterns in the continuous embedding space using L2 or cosine similarity. The implicit assumption was that two models with the same architecture produce geometrically comparable representations — that layer 12 of a 7B model and layer 12 of a 72B model occupy the same region of the embedding manifold. The paper's "vL-Head" ablation in Appendix D provides strong evidence that this assumption is false: when the language head is removed from the verbalizer and the verb-FFN outputs are aligned directly with L2 loss, performance collapses from 75.8 to 63.3 on MM-Vet and from 69.3 to 53.5 on MMMU for the 7B model. The authors' diagnosis is that "directly imitating outputs from the large-backbone VLM, without verbalization from vL-Head, introduces instability" — but the deeper interpretation is that continuous feature alignment fails because there is no shared coordinate system across models of different scales. The 72B model's hidden state at some layer and the 7B model's hidden state at the corresponding depth encode similar information in geometrically incompatible subspaces.

The verbalizer solves this by projecting both models' hidden states through the same language head — a linear classifier over a shared vocabulary — producing probability distributions in a space where distances are semantically meaningful. KL divergence between two vocabulary distributions has an intrinsic interpretation (information-theoretic disagreement about what token should come next) that Euclidean distance between two 4096-dimensional vectors lacks. This is a conceptual shift from "make the features similar" to "make the models say similar things at each layer" — which is both more principled (it uses the model's own output interface as the comparison metric) and empirically validated by the ablation.

This framing also explains why the verbalizer must use the backbone's own language head rather than a separately trained one: using a custom vocabulary projector would reintroduce the representational mismatch problem that the shared language head solves. The language head is the only component that both the large and small models agree on semantically — it was trained on the same tokenizer with the same vocabulary indices — making it the natural choice for a shared projection space.

The significance of this innovation extends beyond VLsI: it suggests that for any heterogeneous model distillation problem (different sizes, different architectures, different modalities), vocabulary-space alignment via a shared tokenizer and language head may be fundamentally more robust than feature-space alignment, because it operates in a representation shared by design rather than one that must be learned to be shared.

---

Innovation 2: Order-Preserving, Sampling-Based Layer Matching as a Principled Solution to the Depth Mismatch Problem

The paper identifies and solves a problem that prior work had largely ignored: how to map layers between teacher and student models of different depths in a way that respects the sequential nature of reasoning. This problem — which the paper calls the "layer number gap" — is structurally analogous to sequence alignment in bioinformatics or dynamic time warping in signal processing, but had not been framed in those terms in the distillation literature.

Prior approaches to layer-wise distillation either assumed matched depths (trivial one-to-one mapping) or used naive heuristics: uniform spacing (map student layer i to teacher layer i × floor(L_teacher / L_student)) or greedy selection (each student layer picks the teacher layer with smallest loss independently). Table 4(e) quantifies the cost of these simplifications: uniform indexing achieves 79.5 on MMB and 66.5 on MM-Vet, while the full VLsI matching strategy reaches 86.3 and 75.8 — a gap of ~7 and ~9 percentage points respectively. This 7–9 point gap represents the penalty for ignoring the sequential structure of layer-wise reasoning.

The conceptual contribution is not the KL divergence metric itself — which is a standard distributional distance — but the formulation of layer matching as a constrained sequential sampling problem with two properties that are genuinely novel in combination:

1. Order preservation constraint enforces that the reasoning progression learned by the student is monotonically deepening — the student's layer 6 cannot learn from an earlier teacher layer than its layer 2 learned from. This is not an arbitrary rule; it reflects the inductive bias that visual-linguistic reasoning builds up cumulatively across layers, and scrambling this progression would teach the student a nonsensical internal dynamic. The ablation shows this is the single most impactful component of the matching strategy (MM-Vet jumps from 69.8 to 75.2 when order preservation is added).

2. KL-divergence-based multinomial sampling with adaptive temperature treats layer matching as a stochastic optimization rather than a deterministic assignment. The intuition is that the "best" match for layer i in isolation may create a dead end for layers i+1, i+2, ... — greedy matching optimizes locally but can produce globally suboptimal alignments. The multinomial sampling introduces controlled randomness, and the adaptive temperature (which scales with the spread of KL divergences in the current search range) automatically balances exploration and exploitation: when all candidate teacher layers are similarly good (low variance in KL), the temperature is high and exploration dominates; when one candidate is clearly best, the temperature is low and exploitation dominates.

This formulation is fundamentally different from attention-based or dynamic programming approaches to sequence alignment. It is online and adaptive: the matching distribution evolves during training as the student's verbalizers improve (Figure 5 shows the matched indices shifting toward deeper teacher layers over time). This means the matching is not a one-time preprocessing step but an integral part of the learning dynamics — the student "chooses" which teacher layers to learn from based on where it currently is in its own development.

The significance of this contribution extends to any teacher-student setup with heterogeneous depths, not just VLMs: language model distillation, encoder-decoder transfer, or even cross-modal distillation (e.g., video→image) where layer counts differ. The paper provides a template — order-preserving sequential sampling with distributional distance — that could generalize to other modalities and distance metrics.

---

Innovation 3: The Interaction-SFT Staging as a Deliberate Decoupling of Knowledge Transfer from Task Optimization

A subtle but important finding is that the interaction step and the SFT step must be separated — applying end-task optimization (autoregressive loss) during interaction actively degrades performance. This is counterintuitive from a standard multi-task learning perspective, where jointly optimizing auxiliary and primary objectives typically helps. Table 4(d) quantifies the penalty: using cross-entropy (autoregressive loss) for intermediate layers during interaction drops MMB from 86.3 to 81.0 and MM-Vet from 75.8 to 67.2 compared to using KL divergence.

The paper's diagnostic — that the interaction step "implicitly alters and expands the probability space over the language head, while SFT aligns this expanded space to better fit instruction-following" — frames this as a representation enrichment followed by sharpening process. This is conceptually similar to the two-phase dynamic in pruning-based distillation [77, 85], where structural changes initially degrade performance and require retraining, but the mechanism is different: in VLsI, there is no structural change, only a change in the distribution of internal representations.

The deeper insight is about the nature of knowledge transfer in neural networks. The interaction step, by minimizing KL divergence to the teacher's layer-wise vocabulary distributions, teaches the student what the teacher knows at each stage of processing — effectively expanding the student's representational capacity by injecting the teacher's reasoning patterns. But this expansion comes at a cost: the student's output distribution becomes more "diffuse" (less confident) because it now considers a richer set of possible interpretations, many of which are locally plausible but globally incorrect. Greedy decoding — which simply picks the most likely next token — performs worse under this expanded distribution because the probability mass is spread across more options. Beam search with N=5 recovers some of the benefit (Table 4b: 85.2 MMB, 70.9 MM-Vet without SFT but with beam decoding) because it can explore multiple paths through the expanded space.

The SFT step then "sharpens" the distribution: standard autoregressive training on the visual instruction dataset teaches the model which of its now-enriched set of internal representations lead to correct answers, effectively collapsing the expanded probability space back toward task-relevant outputs while preserving the representational richness gained from the teacher.

This two-phase dynamic — first learn to think broadly, then learn to answer correctly — is a powerful diagnostic principle that extends beyond VLsI. It suggests that for any distillation task where the teacher's internal dynamics are substantially richer than the student's, directly optimizing for end-task performance will short-circuit the knowledge transfer because the task loss will dominate and the model will revert to memorization rather than absorbing reasoning patterns. The separation of "representation transfer" (KL divergence, no task loss) from "task alignment" (autoregressive loss, no KL divergence) is therefore not a convenience but a necessity for effective distillation in capacity-mismatched settings.

The evidence that beam decoding substantially recovers interaction-only performance — and that SFT pushes performance further — makes this interpretation falsifiable and testable: one could measure the entropy of the student's output distribution after interaction vs. after SFT and predict that entropy decreases, consistent with the "expansion then sharpening" narrative.

---

Innovation 4: Verifying That Test-Time Compute With Distillation Can Substitute for Pretraining Scale — Extending the Inference-vs-Training Tradeoff to VLMs

While the paper does not frame this as its primary contribution, Figure 4 and Table 4(a) provide the first systematic evidence in the VLM domain that a distilled small model can approach or surpass the performance of models with substantially more parameters, without architectural modifications at inference time. This extends the inference-compute-vs-pretraining tradeoff literature (e.g., the compute-optimal inference scaling findings for LLMs discussed in Section 7 of the reference example paper) to vision-language models specifically.

The numbers are striking: VLsI-0.5B (using LLaVA-OV-72B as teacher) achieves 72.5 on MMB and 49.9 on MMMU — surpassing LLaVA-OV-7B's baseline (80.8 MMB but 48.8 MMMU) on MMMU despite being 14× smaller. VLsI-2B (using Qwen2-VL-72B as teacher) achieves 51.4 MMMU, exceeding Qwen2-VL-7B's pretraining baseline of 54.1 on some metrics and substantially outperforming the 7B model on MM-Vet (64.8 vs. 62.0). The most dramatic case is VLsI-7B (Qwen2-VL-72B teacher) reaching 69.3 MMMU and 75.8 MM-Vet — both numbers exceeding the teacher Qwen2-VL-72B's own performance (64.3 MMMU, 73.9 MM-Vet) by 5.0 and 1.9 points respectively, meaning the student outperforms the teacher.

This "student surpasses teacher" finding is notable because standard distillation typically produces students that asymptotically approach but do not exceed teacher performance. The paper attributes this to the combination of (a) the teacher's reasoning patterns being more efficiently encoded in the student after SFT and (b) the SFT step adding task-specific fine-tuning that the frozen teacher did not receive. But the deeper implication is about knowledge compression: the 7B model, by learning the 72B model's layer-wise reasoning dynamics rather than just its outputs, achieves a more efficient encoding of the relevant knowledge — less capacity wasted on patterns that don't matter for the task, more capacity focused on what the teacher actually does internally.

This finding connects to a broader debate about whether distillation transfers "skill" or just "answers." The fact that the student can outperform the teacher — and that this requires layer-wise transfer (final-layer-only distillation, by contrast, does not achieve this) — suggests that VLsI transfers something closer to skill, a reasoning process that generalizes better than the teacher's specific output distribution.

The practical implication is a concrete alternative to the "train the biggest model you can afford" paradigm: train a very large teacher once, use VLsI to produce a small student that retains (or exceeds) the teacher's performance, and deploy the small model with no architectural overhead. The cost is a one-time distillation cost; the benefit is deployment at a fraction of the compute and memory.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper compiles a diverse visual instruction tuning dataset totaling 2.9 million samples from various sources spanning multiple vision-language capabilities. For evaluation, ten challenging benchmarks are selected: QBench [93], AI2D [44], ChartQA [72], POPE [58], HallusionBench (HallB) [64], MME [30], MathVista [71], MMBench (MMB) [67], MMBench-Chinese (MMBCN) [67], MM-Vet [100], MMMU [102], MMStar [10], SEED-Bench-2-Plus [55], BLINK [31], CV-Bench [89], and LLaVA-Wilder [106]. Detailed descriptions of each benchmark appear in Appendix A. The training dataset composition is specified in Section 4.1(d), including ShareGPT4o-Images (57K), ShareGPT4V (755K), ALLaVA-VFLAN/Text (548K), MiniGemini (27K), LLaVA-HD (116K), mPLUG-DocOwl1.5-Downstream/Reasoning (599K), GLLaVA (177K), MathVision (3K), MathInstruct (262K), and MathPlus (304K) — totaling 2.9M samples.

• Base model(s). The primary backbone is Qwen2-VL [92], selected for "its high performance on the OpenVLM-Leaderboard" and because "the tokenizer's vocabulary indices remain consistent across these model sizes, allowing for seamless integration without reordering the vocabulary." Three scales are used: Qwen2-VL-2B and Qwen2-VL-7B as small backbones (each with 28 transformer layers), and Qwen2-VL-72B as the large backbone (80 layers). The vision encoder is a ViT adapted from DFN [25] with visually-adapted rotary positional embeddings [86], and the vision projector is an MLP with two fully-connected layers interleaved with GELU activations. As an alternative backbone for generalization experiments (Figure 4, Table 4a), LLaVA-OneVision [56] is also used, which shares Qwen2 as its language model but differs in image token processing (requiring 4–10× more image tokens than Qwen2-VL for equivalent pixel counts).

• Metrics. Performance is measured as accuracy (%) on each benchmark's standard evaluation protocol. For multiple-choice benchmarks (MMB, MMBCN, MMMU, MMStar, SEED-2-Plus, BLINK, CV-Bench, AI2D, MathVista), accuracy is the fraction of questions where the model selects the correct option. For open-ended generation benchmarks (MM-Vet, MM-Vet-v2, LLaVA-Wilder), an LLM-based evaluator is used following the benchmark's official protocol. No custom metrics or re-weighting schemes are introduced. Tables 1, 2, and 3 report these accuracy numbers directly.

• Baselines. The paper compares against an extensive set of open-source VLMs across different size categories. In Table 1 (7–13B models): LLaVA-NeXT-7B/8B/13B [66], MM1-7B and MM1-MoE-7B×32 [74], MiniGemini-HD-7B/13B [60], Cambrian-1-8B/13B [89], Eagle-8B/13B [82], VILA1.5-8B/13B [63], VILA2-8B [27], CogVLM2-8B [38], LLaVA-OneVision-7B [56], InternVL2-8B [11], MiniCPM-V2.5-8B and MiniCPM-V2.6-8B [98], TroL-7B [50], Phantom-7B [49], and Qwen2-VL-7B [92]. In Table 2 (≤4B models): MiniCPM-2.4B and MiniCPM-V2-2.8B [41], MM1-3B and MM1-MoE-3B×64 [74], ALLaVA-3B [7], VILA1.5-3B [63], InternVL2-2B/4B [11], TroL-1.8B/3.8B [50], Phantom-1.8B/3.8B [49], DeepSeek-VL-1.3B [70], MobileVLM-1.7B [14], MobileVLM-V2-1.7B [15], MoE-LLaVA-1.8B×4 [62], Mini-Gemini-2B [60], and Qwen2-VL-2B [92]. Table 3(b) extends comparison to larger open-source (LLaVA-NeXT-34B [66], VILA1.5-40B [63], Cambrian-34B [89], Molmo-72B [20], LLaVA-OV-72B [56], LLaMA-3.2-Vision, InternVL2-76B [11], Qwen2-VL-72B [92]) and closed-source VLMs (GPT-4V [9], GPT-4o, Claude-3.5-Sonnet [1], Gemini-1.5-Pro [87], NVLM-D-72B [19]). The direct teacher model (Qwen2-VL-72B) serves as a critical upper-bound baseline on all benchmarks.

• Generation budget / compute accounting. Inference is performed using greedy search for text generation (consistent with Qwen2-VL's standard configuration), and each model is evaluated at its native parameter count — no test-time compute scaling is applied. The paper does not measure FLOPs or wall-clock time for inference, but the parameter counts of the verbalizers are carefully reported in Table 4(f): the final verb-FFN architecture adds 269M parameters during training, but these are discarded at inference time; the deployed model has exactly the same parameter count and architecture as the base backbone VLM (2B or 7B). For the FLOPs-matched comparison implicit in the distillation setup, the one-time training cost of verbalization, interaction, and SFT is not amortized over inference queries — the paper treats this as a fixed training investment whose return is zero-overhead inference.

• Cross-validation / statistical protocol. No cross-validation or statistical significance testing is reported. All results in Tables 1–3 are single-run evaluations on the standard test splits of each benchmark. For ablation studies (Table 4), the same benchmark suite is used to compare configurations, but no confidence intervals or error bars are provided. The paper does not discuss potential variance from different random seeds during verbalizer training, interaction matching, or SFT — all results represent a single trained model per configuration.

Main Quantitative Results

The paper's experimental results are organized across four comparison axes: (1) headline performance of VLsI-7B and VLsI-2B against open-source VLMs of similar scale, (2) performance against smaller-scale open-source VLMs, (3) performance against VLMs with architectural modifications and closed-source models on challenging benchmarks, and (4) generalization across backbone pairs and the scaling properties of the distillation process. Each axis tells a progressively more detailed story about VLsI's effectiveness.

Headline Results: VLsI-7B Against Contemporaneous 7–13B Open-Source VLMs

Table 1 presents the primary comparison for VLsI-7B against seventeen existing open-source VLMs in the 7–13B parameter range across eleven benchmarks. The headline numbers are immediately striking: VLsI-7B achieves the top score on 8 out of 11 benchmarks and the second-best on the remaining 3.

On the most challenging integrated reasoning benchmarks, VLsI-7B's margins are substantial:

• MM-Vet: 75.2%, surpassing the next-best open-source model (Phantom-7B at 70.8%) by 4.4 points and Qwen2-VL-7B (its own pretraining baseline at 62.0%) by 13.2 points.
• MMMU: 69.3%, exceeding Qwen2-VL-7B (54.1%) by 15.2 points and the next-best open-source model in this size range (Phantom-7B at 51.2%) by 18.1 points.
• MathVista: 74.7%, surpassing LLaVA-OneVision-7B (63.2%) by 11.5 points and the next-best open-source model (Phantom-7B at 70.9%) by 3.8 points.
• BLINK: 74.2%, the highest among all models listed in Table 1 where this metric is reported (the next highest is Qwen2-VL-7B at 65.7%).
• MMB: 86.3%, exceeding InternVL2-8B (81.7%) by 4.6 points and TroL-7B (83.5%) by 2.8 points.
• MMBCN: 85.5%, exceeding Phantom-7B (84.7%) by 0.8 points and InternVL2-8B (81.2%) by 4.3 points.

On benchmarks where VLsI-7B does not rank first, it still places second:

• AI2D: 87.3%, behind InternVL2-8B (83.8% — wait, this would put VLsI-7B at first by 3.5 points. Re-reading: InternVL2-8B achieves 83.8, VLsI-7B achieves 87.3 — so VLsI-7B is actually first here as well. Checking all 11 benchmarks in Table 1 systematically: QBench (77.5, tied first with Qwen2-VL-7B), AI2D (87.3, first), ChartQA (86.1, first), POPE (88.6, second behind Qwen2-VL-7B's 88.9), HallB (74.2, first), MME (2338, second behind MiniCPM-V2.6-8B's 2348), MathVista (74.7, first), MMB (86.3, first), MMBCN (85.5, first), MM-Vet (75.2, first), MMMU (69.3, first). So VLsI-7B achieves first place on 9 of 11 benchmarks and second place on 2 (POPE by 0.3 points, MME by 10 points out of ~2350).

The comparison against Qwen2-VL-7B — the exact same architecture without VLsI distillation — isolates the contribution of the proposed method: average improvement across the benchmarks where both are reported (QBench, AI2D, ChartQA, POPE, HallB, MME, MathVista, MMB, MMBCN, MM-Vet, MMMU) shows gains ranging from 0.0 (QBench, tied) to 15.2 points (MMMU), with particularly large jumps on the hardest reasoning tasks. This directly supports the claim that VLsI adds capability without architectural change — the model is identical at inference time, yet performs dramatically better.

Results for Smaller-Scale VLMs: VLsI-2B Against ≤4B Models

Table 2 shifts focus to the more resource-constrained regime, comparing VLsI-2B against twenty-one open-source VLMs in the ≤4B parameter range. The pattern of dominance is similar, though with slightly less uniform first-place finishes:

• MM-Vet: 64.8%, exceeding the next-best small model (Phantom-1.8B at 54.1%) by 10.7 points and Qwen2-VL-2B (49.5%) by 15.3 points.
• MMMU: 51.4%, surpassing Qwen2-VL-2B (41.1%) by 10.3 points and Phantom-1.8B (40.6%) by 10.8 points.
• MathVista: 68.4%, exceeding Phantom-1.8B (60.9%) by 7.5 points and Qwen2-VL-2B (43.0%) by 25.4 points — an enormous jump that highlights the knowledge transfer's impact on mathematical reasoning at small scales.
• AI2D: 89.0%, far above the next-best small model (InternVL2-4B at 78.9%) by 10.1 points, and Qwen2-VL-2B itself at 60.2% by 28.8 points.
• ChartQA: 85.8%, above Phantom-1.8B (87.0%) by... actually this would put Phantom-1.8B first by 1.2 points — re-checking: Phantom-1.8B gets 87.0, VLsI-2B gets 85.8. So VLsI-2B is second on ChartQA. Meanwhile, Qwen2-VL-2B gets 73.5%, meaning VLsI-2B improves by 12.3 points over its pretraining baseline.
• HallB: 70.0%, surpassing Phantom-1.8B (62.2%) by 7.8 points and Qwen2-VL-2B (61.2%) by 8.8 points.
• BLINK: (not reported in Table 2 for most models, but appears in Table 3b for VLsI-2B at 52.4%).

The most dramatic baseline-to-VLsI improvements for the 2B model are on AI2D (28.8 points), MathVista (25.4 points), MM-Vet (15.3 points), and MMMU (10.3 points) — all benchmarks requiring sophisticated reasoning and knowledge integration, exactly the capabilities that layer-wise distillation from a 72B teacher would be expected to enhance.

A notable pattern emerges when comparing VLsI-2B against 7B-scale models from Table 1: VLsI-2B (64.8 on MM-Vet) outperforms VILA1.5-8B (43.2), Eagle-8B (not reported for MM-Vet), and LLaVA-NeXT-7B (43.9) — models with 3.5× more parameters. On MMMU, VLsI-2B (51.4) exceeds all 7–8B models in Table 1 except Qwen2-VL-7B (54.1) and VLsI-7B (69.3). This directly supports the claim that effective distillation can substitute for 3–4× parameter scaling, and in some cases more — VLsI-2B even surpasses the 7B InternVL2-8B on AI2D (89.0 vs. 83.8).

Comparison Against Architecturally Modified VLMs and Closed-Source Models

Table 3(a) specifically targets VLMs that employ "additional modules and projectors" — models like OmniFusion-7B [34], DeepSeek-VL-7B [70], MoVA-7B [43], Eagle-8B [82], CoLLaVO-7B [52], MoAI-7B [53], and Meteor-7B [51] that extend the standard VLM architecture with extra components. On four core benchmarks:

• MMB: VLsI-2B (81.7) and VLsI-7B (86.3) outperform all listed models with architectural modifications except Meteor-7B (82.9) for the 2B case; the 7B version leads by 3.4 points over the nearest competitor (Meteor-7B at 82.9).
• MathVista: VLsI-2B (68.4) substantially exceeds all modification-based models (highest: CoLLaVO-7B at 57.6, gap of 10.8 points). VLsI-7B (74.7) is even further ahead.
• MM-Vet: VLsI-2B (64.8) and VLsI-7B (75.8) dominate — the nearest modification-based model that reports MM-Vet is MoAI-7B at 43.7, a gap of 21.1 and 32.1 points respectively, though direct comparison is complicated by missing entries (Eagle, CoLLaVO, MoVA do not report MM-Vet).
• MMMU: VLsI-2B (51.4) exceeds all modification-based models (highest: Meteor-7B at 48.3), and VLsI-7B (69.3) opens a 21.0 point gap.

This comparison is significant because it directly addresses the paper's motivating question: "Can we achieve a similar or superior level of performance without scaling, merging, or architectural changes?" The answer from Table 3(a) is an unqualified yes — the architecture-free VLsI approach matches or exceeds every method that adds specialized modules, extra vision encoders, or rationale projectors.

Table 3(b) provides the most ambitious comparison: VLsI against much larger open-source models and proprietary closed-source VLMs. On an expanded set of ten difficult benchmarks:

• VLsI-7B vs. GPT-4V (0409): On MM-Vet, 75.8 vs. 67.5 (+8.3 points); on MMMU, 69.3 vs. 61.7 (+7.6 points); on MathVista, 74.7 vs. 54.7 (+20.0 points); on AI2D, 87.3 vs. 78.6 (+8.7 points). On BLINK, VLsI-7B (59.7) slightly exceeds GPT-4V (58.3), though GPT-4V leads on LLaVA-Wilder (71.5 vs. 92.0 — actually VLsI-7B gets 92.0 and GPT-4V gets 71.5, so VLsI-7B leads by 20.5 points there).

• VLsI-7B vs. GPT-4o (0806): On MM-Vet, 75.8 vs. 75.1 (+0.7, essentially tied); on MMMU, 69.3 vs. 69.9 (-0.6, essentially tied); on MMStar, 73.6 vs. 64.7 (+8.9 points); on MathVista, 74.7 vs. 62.7 (+12.0 points); on BLINK, 59.7 vs. 64.7 (-5.0 points). The results show VLsI-7B is competitive with (and on several benchmarks, notably better than) GPT-4o, a model widely regarded as state-of-the-art.

• VLsI-7B vs. Claude-3.5-Sonnet: On MM-Vet, 75.8 vs. 66.0 (+9.8 points); on MMMU, 69.3 vs. 65.9 (+3.4 points); on MathVista, 74.7 vs. 61.6 (+13.1 points); on BLINK, 59.7 vs. 28.2 (+31.5 points). The BLINK result is particularly notable — BLINK tests core visual perception (depth, correspondence, spatial reasoning), and VLsI-7B more than doubles Claude-3.5's score.

• VLsI-7B vs. Gemini-1.5-Pro: On MM-Vet, 75.8 vs. 64.0 (+11.8 points); on MMMU, 69.3 vs. 60.6 (+8.7 points); on MathVista, 74.7 vs. 57.7 (+17.0 points); on BLINK, 59.7 vs. 59.1 (+0.6, tied).

• VLsI-7B vs. its own teacher (Qwen2-VL-72B): On MM-Vet, 75.8 vs. 73.9 (+1.9 points); on MMMU, 69.3 vs. 64.3 (+5.0 points); on MathVista, 74.7 vs. 69.7 (+5.0 points); on MMStar, 73.6 vs. 68.6 (+5.0 points). The student outperforms the teacher on 8 of 10 benchmarks (the exceptions being AI2D, where the teacher leads 88.3 vs. 87.3, and LLaVA-Wilder, where the teacher leads 84.1 to... actually VLsI-7B gets 92.0 and Qwen2-VL-72B gets 84.1, so VLsI-7B leads by 7.9 points. Checking: SEED-2-Plus: VLsI-7B 74.9, Qwen2-VL-72B 72.3 — student ahead; CV-Bench: VLsI-7B 89.1, Qwen2-VL-72B 74.3 — student ahead by 14.8 points; BLINK: VLsI-7B 59.7, Qwen2-VL-72B 60.5 — teacher ahead by 0.8 points). So VLsI-7B outperforms its 72B teacher on 8 of 10 benchmarks, sometimes substantially.

• VLsI-2B against large models: Even the 2B version achieves remarkable comparisons — VLsI-2B (64.8 MM-Vet) substantially exceeds GPT-4V (67.5 — actually GPT-4V leads here by 2.7 points, wait: VLsI-2B gets 64.8, GPT-4V gets 67.5, so GPT-4V leads by 2.7 on MM-Vet); on AI2D, VLsI-2B (89.0) exceeds all closed-source models listed; on CV-Bench, VLsI-2B (90.1) equals VLsI-7B (89.1 — actually VLsI-2B gets 90.1 and VLsI-7B gets 89.1, so the 2B model is ahead) and substantially exceeds Qwen2-VL-72B (74.3).

The key result from Table 3(b) is unambiguous: a 7B model trained with VLsI matches or exceeds frontier proprietary models (GPT-4o, Claude-3.5, Gemini-1.5-Pro) and its own 72B teacher on the majority of challenging vision-language benchmarks, using zero architectural modifications at inference time.

Generalization Across Backbone Pairs and Scaling Behavior

Figure 4 provides the only multi-backbone comparison in the paper, answering: does VLsI work with different teacher-student pairs, and how does performance scale with teacher quality and student size?

The results are presented as a grid: small backbones of 0.5B, 2B, and 7B parameters, each trained with either Qwen2-VL-72B or LLaVA-OV-72B as the large teacher, evaluated on MM-Vet and MMMU.

Scaling with student size (within a fixed teacher): For Qwen2-VL-72B as teacher:

• 0.5B student: (not reported for Qwen2-VL — the 0.5B experiments use LLaVA-OV as backbone, as shown in Table 4a)
• 2B student: MM-Vet 64.8, MMMU 51.4
• 7B student: MM-Vet 75.8, MMMU 69.3

The jump from 2B to 7B (3.5× more parameters) yields +11.0 MM-Vet points and +17.9 MMMU points — substantial improvements consistent with the larger student having greater capacity to absorb the teacher's reasoning patterns.

For LLaVA-OV-72B as teacher:

• 0.5B student: MM-Vet 50.7, MMMU 49.9 — remarkably, the 0.5B model trained with VLsI achieves 49.9 MMMU, which exceeds the untrained LLaVA-OV-7B baseline (48.8 MMMU, as implied by the baseline table in Appendix D) by 1.1 points, despite being 14× smaller. The MM-Vet performance (50.7) exceeds LLaVA-OV-7B's baseline (57.5)... actually wait, checking: the baseline LLaVA-OV-7B gets 57.5 on MM-Vet and 48.8 on MMMU. So VLsI-0.5B (LLaVA-OV-72B teacher) gets 50.7 MM-Vet (7.2 points below LLaVA-OV-7B baseline) but 49.9 MMMU (1.1 points above). For a 14× parameter reduction, this is a strong result on MMMU specifically.
• 7B student: MM-Vet 61.6, MMMU 59.1 — improving over LLaVA-OV-7B baseline of 57.5 and 48.8 by 4.1 and 10.3 points respectively.

Effect of teacher quality: Comparing Qwen2-VL-72B vs. LLaVA-OV-72B as teacher for the same 7B student:

• Qwen2-VL-72B teacher: MM-Vet 75.8, MMMU 69.3
• LLaVA-OV-72B teacher: MM-Vet 61.6, MMMU 59.1

The Qwen2-VL-72B teacher yields 14.2 and 10.2 point improvements respectively over the LLaVA-OV-72B teacher. This is consistent with the baseline teacher performances: Qwen2-VL-72B achieves 74.0 MM-Vet and 64.5 MMMU vs. LLaVA-OV-72B's 63.7 and 56.8. The gain from a stronger teacher propagates meaningfully to the student — a better teacher makes for a better student, which is expected but important to confirm quantitatively.

Cross-family generalization: The fact that VLsI works with both Qwen2-VL and LLaVA-OV as teachers (and LLaVA-OV as both teacher and student) demonstrates that the method is not specific to one backbone architecture. However, both families share the Qwen2 tokenizer (LLaVA-OV uses Qwen2 as its LLM), consistent with the paper's stated constraint that "the large and small-backbone VLMs must share the same tokenizer and token index order."

Ablation Studies and Robustness Checks

Table 4 presents six groups of ablations examining different aspects of the VLsI pipeline. All ablations are reported on four challenging benchmarks: MMB, BLINK, MM-Vet, and MMMU, using VLsI-7B (Qwen2-VL-72B teacher) unless otherwise specified. Each ablation group is internally controlled — only the stated component is varied while others remain at their VLsI defaults.

Backbone capability and scale (Table 4a): This ablation examines the effect of student scale and backbone family by training VLsI with LLaVA-OV-0.5B and LLaVA-OV-7B as small backbones, using LLaVA-OV-72B as teacher. For LLaVA-OV-0.5B, VLsI achieves 72.5 MMB and 49.9 MMMU, compared to the LLaVA-OV-0.5B baseline of 52.1 MMB and 31.4 MMMU — gains of 20.4 and 18.5 points respectively. This demonstrates that even at extremely small scales (0.5B parameters), the method transfers substantial knowledge. For LLaVA-OV-7B, VLsI achieves 86.1 MMB and 59.1 MMMU, compared to the baseline of 80.8 MMB and 48.8 MMMU — gains of 5.3 and 10.3 points. The smaller absolute gain for the larger student is consistent with the baseline being stronger to begin with (higher ceiling effect). The comparison across 0.5B, 2B, and 7B students (with their respective teachers) confirms monotonic improvement with student size — no saturating trend is observed over this range.

Effect of the SFT step (Table 4b): This critical ablation compares VLsI with and without the final SFT stage, for both 2B and 7B models. Without SFT and using greedy decoding, VLsI-7B achieves 82.1 MMB (vs. Qwen2-VL-7B baseline of 83.0) and 60.5 MM-Vet (vs. 62.0 baseline) — worse than the pretrained baseline. For VLsI-2B, the pattern is similar: without SFT, MMB is 73.2 (vs. 74.9 baseline) and MM-Vet is 47.9 (vs. 49.5 baseline). This negative result is crucial: it shows that the interaction step alone, despite successfully transferring knowledge, degrades standard evaluation performance under greedy decoding. The paper's interpretation — that interaction "expands" the probability space, hurting greedy search but enriching beam search — is validated by the second row: with beam decoding (N=5) and no SFT, VLsI-7B jumps to 85.2 MMB and 70.9 MM-Vet, substantially exceeding the baseline. Adding SFT with greedy decoding yields the best results across all metrics (86.3 MMB, 75.8 MM-Vet, 69.3 MMMU for 7B), confirming that SFT is not optional but essential for recovering and surpassing baseline performance under standard decoding.

Training data contribution vs. VLsI method (Table 4c): This ablation addresses a potential confounding factor: perhaps VLsI's gains come not from the distillation process but simply from additional training on the 2.9M visual instruction dataset (the "Recipe-dataset Sampling" or RS). The experiment trains both the baseline Qwen2-VL-7B and VLsI-7B on 50% and 100% of the RS data, measuring performance at each point. For baseline Qwen2-VL-7B, additional data yields modest improvements: from 80.5 to 81.8 MMB (+1.3), 50.8 to 52.3 BLINK (+1.5), 62.0 to 63.7 MM-Vet (+1.7), 54.1 to 55.2 MMMU (+1.1) when going from 50% to 100% of the RS data. For VLsI-7B, the gains are dramatically larger: from 82.1 to 86.3 MMB (+4.2), 49.6 to 59.7 BLINK (+10.1), 60.5 to 75.8 MM-Vet (+15.3), 52.9 to 69.3 MMMU (+16.4). The interaction step at 50% RS data already lifts VLsI-7B above the baseline (82.1 vs. 80.5 MMB, 60.5 vs. 62.0 MM-Vet — actually baseline leads on MM-Vet at this point), but the critical finding is that the rate of improvement from additional SFT is substantially higher for VLsI than for the baseline, confirming that the SFT step is unlocking transferred knowledge rather than just benefiting from data quantity.

Operations for intermediate and last layers (Table 4d): This ablation systematically varies the loss function used for intermediate-layer alignment (IL-Ops) and last-layer distillation (LL-Ops). Using cross-entropy (CE) for intermediate layers — with or without CE/KLD at the last layer — consistently underperforms using KL divergence (KLD). The best configuration is KLD for intermediate layers and KLD for the last layer: 86.3 MMB, 59.7 BLINK, 75.8 MM-Vet, 69.3 MMMU. Removing the last-layer distillation entirely (KLD intermediate, no last-layer ops) drops performance to 83.0 MMB and 69.5 MM-Vet — a loss of 3.3 MMB and 6.3 MM-Vet points, showing that last-layer distillation provides complementary information not captured by intermediate alignment alone. Using L2 loss for intermediate alignment (with KLD last) yields 81.7 MMB and 67.0 MM-Vet — worse than KLD intermediate but still above the CE intermediate configurations, suggesting distribution matching in vocabulary space is preferable but continuous alignment is not entirely ineffective. The ablation with L2 intermediate and no verbalizer language head (Appendix D, "vL-Head" experiment) shows the most dramatic degradation: 63.3 MM-Vet and 53.5 MMMU, confirming that verbalization through the language head is essential — direct feature alignment fails catastrophically.

Components in the layer matching strategy (Table 4e): This ablation incrementally adds components to the matching algorithm, starting from simple baselines and building up to the full VLsI strategy. Random index matching (no structure) achieves 77.0 MMB and 62.0 MM-Vet — the baseline for evaluating each component's contribution. Uniform index (evenly spaced) improves to 79.5 MMB and 66.5 MM-Vet, a gain of 2.5 and 4.5 points. Switching to Bottom-1 (greedy selection of lowest-KL layer) yields 81.2 MMB and 67.8 MM-Vet. Bottom-3 (averaging the 3 lowest-KL layers) gives 81.5 MMB and 68.0 MM-Vet — only marginal improvement over Bottom-1. Multinomial sampling (without search range or order preservation) reaches 82.0 MMB and 68.5 MM-Vet. Adding the search range constraint improves to 83.5 MMB and 69.8 MM-Vet — a gain of 1.5 and 1.3 points, confirming that ensuring enough teacher layers remain for later student layers matters. Adding order preservation produces the largest single jump: 86.0 MMB (+2.5) and 75.2 MM-Vet (+5.4). The full configuration with adaptive temperature reaches 86.3 MMB and 75.8 MM-Vet. The ordering of gains reveals a clear hierarchy: order preservation > search range ≈ multinomial sampling > adaptive temperature, but all components contribute positively. The adaptive temperature adds a small but consistent benefit (0.3 MMB, 0.6 MM-Vet), confirming that dynamically adjusting the exploration-exploitation tradeoff based on the local KL divergence landscape is helpful.

Verbalizer architecture (Table 4f): This ablation compares different architectural choices for the verbalizer, reporting both parameter count and performance. The verb-FFN (269M parameters) achieves 86.3 MMB and 75.8 MM-Vet. Doubling the verb-FFN (539M) yields essentially identical performance (85.8 MMB, 75.8 MM-Vet), showing the single verb-FFN is sufficient and more parameters provide no benefit. A full decoder layer (Decoder×2, 3.3B parameters) achieves slightly lower MM-Vet (76.2 vs. 75.8 — actually Decoder×2 gets 76.2 which is 0.4 points higher, but MMB is 85.5 vs. 86.3, lower by 0.8) — and the Decoder configuration without doubling (1.6B) achieves 85.7 MMB and 76.0 MM-Vet. The FFN×2 architecture (2.9B) achieves 86.3 MMB and 75.9 MM-Vet. Simpler MLP architectures (90M and 180M) perform substantially worse: 83.8 MMB and 73.5 MM-Vet for the 90M MLP. The key finding is the efficiency sweet spot: the 269M verb-FFN achieves essentially the same performance as architectures with 3.3B parameters (12× larger), confirming the design choice to maintain constant hidden dimension without expansion-reduction provides sufficient capacity without waste.

Verbalization loss progression (Appendix B): The paper provides detailed loss curves during verbalization training, showing per-layer autoregressive loss at 10% increments through training. At 10% progress, layer #1 (2nd transformer layer) has loss 14.28 while layer #7 (26th layer) has loss 7.39 — a factor of ~2× difference confirming deeper layers contain substantially more information about the target answer. By 100% progress, these drop to 3.30 and 0.79 respectively — both improve, but the relative gap widens (factor of ~4.2×). All layers show monotonically decreasing loss, and the deeper layers always have lower loss than shallower layers at every training percentage — confirming that the layer-wise reasoning progression captured by the verbalizers is genuine and not an artifact of training.

Interaction step distribution shift (Figure 5): Rather than an ablation, this is a diagnostic analysis showing how matched layer indices shift from early to late interaction training. At the beginning, the small model's target layers match to a relatively uniform spread across the large model's target layers. By the end, the distribution concentrates at deeper large-model layers — the small model "moves" its matching toward later stages of the teacher's reasoning. This emergent behavior is important because it suggests the matching is not merely finding static correspondences but adapting as the student's verbalizers improve, progressively targeting more informative teacher layers.

Critical Assessment

The experimental results in this paper are, by most conventional standards, impressively strong — VLsI-7B consistently ranks first or second across eleven benchmarks against similarly-sized models, surpasses its own 72B teacher on most metrics, and competes with or exceeds proprietary frontier models. However, several methodological features of the evaluation warrant scrutiny before accepting the paper's strongest claims at face value.

The central empirical claim is that VLsI achieves "significant performance gains of 11.0% (2B model) and 17.4% (7B model) over GPT-4V" (Section 1). These numbers come from aggregating across the ten benchmarks in Table 3(b) — but the paper does not specify the aggregation method (arithmetic mean? geometric mean? benchmark-weighted?). More importantly, the baseline GPT-4V numbers are themselves from different evaluation dates and configurations. GPT-4V (0409) on MathVista scores 54.7, while VLsI-7B scores 74.7 — a genuine 20-point improvement on that benchmark. But on MM-Vet, GPT-4V scores 67.5 and VLsI-7B scores 75.8 — an 8.3-point improvement. The average gain clearly depends on which benchmarks are included and how they are weighted, and the paper provides no details on this calculation. The 11.0% and 17.4% figures should therefore be interpreted as approximate summaries rather than precisely defined metrics.

A more fundamental concern is the single-backbone evaluation setup. While Figure 4 demonstrates generalization across Qwen2-VL and LLaVA-OV as backbone families, both share the Qwen2 tokenizer and vocabulary. The paper explicitly acknowledges this dependency (Section 4.4: "the large and small-backbone VLMs must share the same tokenizer and token index order"), and no experiments test VLsI across tokenizer boundaries (e.g., Qwen teacher to LLaMA student). This means the reported gains are, strictly speaking, demonstrated only for models within the Qwen2 tokenizer ecosystem. Whether VLsI's verbalization-based alignment works when the vocabulary spaces differ — requiring a learned mapping between tokenizers — is untested and non-trivial. The paper's framing of VLsI as a general method "adaptable across various model architectures" (contribution bullet 3) is therefore aspirational rather than demonstrated — "architectures" here means VLM architectures sharing a tokenizer, which is a narrower claim.

The comparison against closed-source models in Table 3(b) is eye-catching but has methodological weaknesses. The closed-source model scores are taken from different evaluation sources — not all benchmarks report results for all models, and the evaluation protocols may differ. For instance, the paper's evaluation of VLsI uses greedy decoding on 8 A100 GPUs; the exact evaluation configuration for GPT-4o, Claude-3.5, and Gemini-1.5-Pro is unspecified and likely differs (different prompting, potentially different image preprocessing). The paper does not re-evaluate these models under a unified protocol, meaning some portion of the performance gap could stem from evaluation differences rather than genuine capability differences. That said, the gaps on benchmarks like MathVista (+20.0 over GPT-4V, +12.0 over GPT-4o) are large enough that evaluation variance alone is unlikely to explain them entirely.

A critical missing comparison is VLsI against simple finetuning baselines. The SFT step trains the small model on 2.9M visual instruction samples — but what if one simply fine-tuned Qwen2-VL-7B on this dataset without any distillation at all? Table 4(c) partially addresses this, showing that the baseline Qwen2-VL-7B improves from 80.5 to 81.8 MMB and 62.0 to 63.7 MM-Vet with additional SFT data — but this is only reported for 50% and 100% of the RS data, and the baseline was not trained to convergence on the full dataset as a separate experiment. A proper baseline would be: Qwen2-VL-7B trained on the full 2.9M dataset for the same number of steps as VLsI's total training (verbalization + interaction + SFT), to isolate how much of the gain comes from the distillation process vs. simply seeing more data. The paper's numbers suggest the distillation is the dominant factor (baseline gains are 1–2 points while VLsI gains are 10–15 points), but a controlled experiment with matched training compute would make this case airtight.

The ablation of the SFT step reveals a subtle concern about the method's robustness. VLsI without SFT actually underperforms the pretrained baseline under greedy decoding (Table 4b). The paper's diagnosis — that interaction "expands" the probability space — is plausible but not directly verified. No experiment measures output entropy or distribution sharpness before and after SFT. The recovery through beam search (N=5) is demonstrated, but beam search increases inference cost by 5×, partially defeating the purpose of an efficient model. The fact that SFT is required to make the distilled model work under standard greedy decoding means VLsI is a three-stage pipeline where the first two stages alone are counterproductive — a practitioner implementing VLsI must commit to all three stages, and debugging intermediate failures would be difficult since Stage 2 appears to regress performance.

The layer matching ablation (Table 4e) reveals that order preservation and search range together account for the majority of the gain over naive matching strategies. However, the matching strategy is evaluated only as a complete procedure — no experiment varies the number of target layers or their spacing. What if target layers were selected adaptively based on where the KL divergence changes most rapidly (e.g., at points where the verbalized output qualitatively shifts)? The fixed every-4th-layer spacing is computationally convenient but potentially suboptimal — layers where reasoning transitions happen might not align with uniform spacing.

A significant missing experiment is the scaling behavior with teacher size. The paper uses Qwen2-VL-72B as the teacher throughout, but what if a 14B or 32B teacher were used? Does VLsI's benefit scale smoothly with teacher capability, or does it saturate? Figure 4 compares Qwen2-VL-72B vs. LLaVA-OV-72B as teachers, showing the stronger teacher yields better students, but no intermediate teacher sizes are tested. This makes it impossible to determine the "distillation efficiency" — how many teacher parameters are needed per point of student improvement, and whether diminishing returns set in.

The computational cost of VLsI training is not quantified in the paper. Verbalization requires training ∼27 verbalizers (7 for the small model + 20 for the large), each on the full 2.9M dataset. Interaction requires computing KL divergences for all candidate layer pairs and backpropagating through the small model's verbalizers. SFT requires full-model finetuning. The total training FLOPs or GPU-hours are not reported, making it difficult to assess whether VLsI's gains justify the training cost compared to simply training a larger model from scratch with the same total compute. The one-time training cost argument is valid (training is expensive but inference is free), but the absolute cost matters for practical adoption.

Finally, the paper's evaluation is entirely on English-language vision-language benchmarks with standard answer formats. No experiments test VLsI on multilingual tasks, video understanding, embodied AI, or other modalities. The method's generalization to these settings is unknown, and the shared tokenizer constraint may be more limiting in multilingual contexts where tokenizers differ across languages. The paper acknowledges limitations only in passing (Section 4.4 focuses on the tokenizer constraint), which somewhat overstates the demonstrated generality of the approach.

These concerns do not invalidate the paper's core contributions — VLsI clearly achieves substantial, benchmark-verified improvements on a difficult set of vision-language tasks — but they do suggest the strongest framing ("a new VLM family that... achieves notable performance gains over GPT-4V") should be tempered with awareness that the results are demonstrated on a specific backbone ecosystem (Qwen2-VL and LLaVA-OV, both using Qwen2 tokenizer), against baselines that are evaluated under different protocols, and with a training pipeline whose computational cost is unquantified.

6. Limitations and Trade-offs

The Shared Tokenizer Constraint Restricts VLsI to a Single Vocabulary Ecosystem

VLsI's entire verbalization and interaction mechanism depends on the large and small backbone VLMs sharing the same tokenizer with identical vocabulary index ordering. The paper acknowledges this explicitly in Section 4.4:

"While VLsI is highly effective, the large and small-backbone VLMs must share the same tokenizer and token index order when constructing VLsI. We will explore more general ways that accommodate different tokenizers and token index orders, potentially expanding VLsI's applicability and scalability."

The consequence is that VLsI is currently demonstrated only within the Qwen2 tokenizer ecosystem — Qwen2-VL and LLaVA-OV both use Qwen2 as their language model. A practitioner wanting to distill from a GPT-4V-level teacher (which uses a different tokenizer) to a LLaMA-based student (different tokenizer again) cannot apply VLsI directly. The verbalizer produces a probability distribution over the teacher's vocabulary indices, and the student's language head interprets those same indices — if index 4253 means "photograph" in the teacher and "carburetor" in the student, the KL divergence between distributions is semantically meaningless. This is not a minor implementation detail — it constrains the entire method to backbone families where vocabulary alignment is guaranteed a priori. The paper provides no evidence on whether VLsI would work across tokenizer boundaries with a learned mapping, and the ablation removing the language head entirely (Appendix D) shows performance collapses (MM-Vet drops from 75.8 to 63.3), suggesting the shared vocabulary is load-bearing. The mitigation status is explicitly future work — the paper flags this as an open problem and does not attempt to solve it.

---

VLsI Training Cost Is Unquantified, Making Cost-Benefit Analysis Impossible

The paper reports zero numbers on the computational cost of VLsI's three-stage training pipeline — no GPU-hours, no FLOP counts, no wall-clock time, not even a qualitative comparison to the cost of simply training a larger model. This matters because a practitioner deciding whether to use VLsI needs to know: is the one-time distillation cost less than, equal to, or greater than the cost of pretraining a larger student from scratch? The training pipeline is substantial: Stage 1 trains 27 independent verbalizers (7 for the small model, 20 for the large) on 2.9M samples; Stage 2 computes KL divergences across all candidate layer pairs and backpropagates through the small model's verbalizers while both models (72B + 2/7B) must reside in GPU memory simultaneously; Stage 3 full-finetunes the small model on the same 2.9M samples. The paper notes that 8 NVIDIA A100 80GB GPUs are used (Section 4.1), gradient accumulation with 16 steps is employed, and the 72B model configuration uses only 2 batches per GPU to fit in memory — all of which hints at substantial resource consumption. But without concrete numbers, the headline claim that VLsI achieves gains "without scaling, merging, or architectural changes" omits the scaling of the training budget required to achieve those gains. The mitigation status: not addressed. The paper provides no training cost analysis and no comparison to alternative approaches matched on total training FLOPs.

---

The Interaction Step Alone Degrades Greedy Decoding Performance — VLsI Requires All Three Stages

Table 4(b) reveals a non-obvious failure mode that has practical consequences for anyone attempting to implement or debug VLsI: after the interaction step (Stage 2), the model's performance under standard greedy decoding is worse than the pretrained baseline. For VLsI-7B without SFT, MMB drops from 83.0 (baseline) to 82.1 and MM-Vet drops from 62.0 to 60.5. For VLsI-2B, MMB drops from 74.9 to 73.2 and MM-Vet drops from 49.5 to 47.9. The paper's interpretation — that the interaction step "expands the probability space" and SFT "sharpens" it — is plausible but unverified, since no experiment directly measures output distribution entropy or calibration before and after each stage. The practical consequence is that VLsI is an all-or-nothing pipeline: a practitioner cannot stop after Stage 2 and expect any benefit, and tuning hyperparameters for the interaction step is difficult because the training signal (improving Stage 2 metrics) is anti-correlated with the deployment metric (greedy decoding accuracy). If the SFT step is insufficiently long or improperly configured, the final model may underperform. The beam search results (85.2 MMB, 70.9 MM-Vet without SFT but with N=5 beam decoding) confirm that the knowledge is present in the model after interaction — it is just inaccessible under greedy decoding. But beam search increases inference cost 5×, partially defeating the efficiency motivation. The mitigation status: the paper identifies this phenomenon and provides beam search as a partial workaround, but offers no solution that makes Stage 2 outputs directly usable without either beam search or SFT. A diagnostic measuring output entropy at each stage would help users know when SFT has sufficiently "sharpened" the distribution, but no such metric is proposed.

---

Hard Problems Show Diminishing Returns — the Method Amplifies Existing Capability but Cannot Create It

The difficulty-stratified results implicit in the benchmark evaluations reveal a capability bound that constrains VLsI's applicability. On benchmarks testing sophisticated multimodal reasoning — MMMU (college-level expert knowledge across six disciplines) and MathVista (mathematical reasoning in visual contexts) — the absolute performance, while strongly improved over the baseline, still leaves substantial room below practical reliability. VLsI-2B reaches 51.4 on MMMU and 68.4 on MathVista; VLsI-7B reaches 69.3 on MMMU and 74.7 on MathVista. These are impressive relative gains over their respective baselines (MMMU: +10.3 and +15.2 points; MathVista: +25.4 and +16.5 points), but on MMMU the 2B model still fails nearly half the questions and even the 7B model fails nearly one-third. This is analogous to the finding in the test-time compute scaling literature that "hardest problems show near-zero improvement regardless of budget" — VLsI can transfer what the teacher knows, but if the teacher's capability on a problem class is limited (Qwen2-VL-72B itself scores 64.3 on MMMU), the student inherits those limits. The paper does not analyze performance stratified by question difficulty within benchmarks — all results are aggregate benchmark scores — so it is unknown whether VLsI's gains are concentrated on easier subsets (where the teacher is already strong) or evenly distributed. The qualitative verbalization examples in Appendix C show VLsI successfully answering questions about pattern recognition (stars and dots), image atmosphere description, and chart interpretation — but these are all within the teacher's demonstrated competence range. What happens on genuinely novel visual reasoning problems that neither the teacher nor the training distribution covers is untested. The mitigation status: this limitation is inherent to distillation-based approaches and the paper does not claim otherwise — VLsI is explicitly about transferring existing large-model capability to small models, not about creating new capability. But the framing in the abstract ("achieving notable performance gains over GPT-4V") could be read as claiming general-purpose parity, when the results are more precisely described as strong improvements within the teacher's capability envelope.

---

Evaluation Is Confined to English-Language, Static-Image Benchmarks — No Multimodal, Temporal, or Cross-Lingual Testing

Every benchmark in Tables 1–3 evaluates VLsI on English-language visual question answering with static images: MM-Vet (single-image QA), MMMU (college-level image-text problems), MathVista (visual math), AI2D (diagram understanding), BLINK (visual perception), etc. The paper provides no results on: video understanding (temporal reasoning across frames), multilingual vision-language tasks (where the shared tokenizer assumption may break if different languages use different subword vocabularies), embodied AI (where visual inputs are egocentric and sequential), or interactive settings (multi-turn visual dialogue). The shared tokenizer constraint — both models must use identical vocabulary indices — is especially problematic for multilingual deployment: if the Qwen2 tokenizer is optimized for English and Chinese, a deployment targeting Arabic or Japanese may require a different tokenizer, breaking VLsI's core mechanism. The 2.9M training dataset composition (Section 4.1(d)) is entirely English-language and static-image focused, drawn from datasets like ShareGPT4V, AI2D, ChartQA, and MathVista. This means VLsI's demonstrated effectiveness is bounded by the training distribution — whether the layer-wise reasoning patterns transferred from the teacher generalize to video temporal dynamics, cross-lingual semantic alignment, or interactive visual reasoning is unknown. The paper does not claim generalizability to these settings, but it also does not flag English/static-image as a scope limitation. The mitigation status: not addressed. The paper's conclusion that "natural language is an important key in transferring knowledge not only for humans but also for AI" implicitly claims universality, but the evaluation provides evidence only for English-language visual QA on static images.

---

No Statistical Significance Reporting or Multi-Seed Validation — Effect Sizes Are Unbounded

All results in Tables 1–4 are single-run evaluations with no confidence intervals, error bars, standard deviations, or multi-seed validation. The test sets for these benchmarks range from ~500 questions (MM-Vet) to ~11,500 (MMMU), but the paper does not report whether performance differences between configurations are statistically significant. This matters most for the headline numeric claims: VLsI-7B achieves 75.8 on MM-Vet vs. GPT-4V's 67.5 — a +8.3 point difference. But if the standard error on MM-Vet (500 questions) is, say, ±2 points, the actual gap could be anywhere from +4.3 to +12.3 at a 95% confidence interval. Similarly, the comparison against GPT-4o on MM-Vet is essentially tied (75.8 vs. 75.1, difference of 0.7 points) — without error estimates, this could just as easily be a genuine 3-point gap in either direction. The layer matching ablation (Table 4e) reports gains as small as 0.3 MMB and 0.6 MM-Vet from adaptive temperature — these could be noise given no error bars. The paper also provides no evidence that the key results replicate across random seeds. The verbalizer training and the multinomial sampling in the interaction step both involve randomness; different initializations or sampling trajectories could produce different layer matchings and different final performance. The mitigation status: not addressed. The absence of statistical rigor is common in the VLM benchmarking literature — many of the papers VLsI compares against also report single-point estimates — but it means the paper's precise numeric rankings (first place on 9 of 11 benchmarks for VLsI-7B in Table 1) should be interpreted as best estimates rather than statistically robust orderings.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a genuinely new axis in the efficiency-vs-performance tradeoff for vision-language models: distillation through intermediate natural language verbalization rather than through final-output imitation or continuous feature alignment. Before VLsI, the dominant paradigm for making small VLMs perform well was either (a) accept the performance ceiling and optimize for parameter count (the MobileVLM/MoE-LLaVA approach), or (b) modify the architecture to squeeze more computation into the same parameter budget (the TroL/Phantom approach of double forward passes and enlarged latent dimensions). Both essentially concede that a small model, as conventionally trained, cannot approach large-model capability.

VLsI changes this equation by demonstrating that the reasoning progression encoded across a large model's layers is transferable to a small model without architectural modification, provided the transfer medium is natural language vocabulary distributions rather than continuous features. The magnitudes are significant enough to shift the burden of proof: a 7B model matching GPT-4o on MM-Vet and MMMU, and a 2B model exceeding many 7–8B baselines on AI2D and MathVista, is not an incremental improvement over prior distillation work — it represents a step change in what small models can achieve.

The conceptual reframing is this: the problem of heterogeneous-depth distillation is best understood as a sequential alignment problem in a shared semantic space, not as a feature-matching problem in a learned embedding space. This reframing matters because it suggests a research program around "verbalized intermediate representations" that is distinct from both standard knowledge distillation (which operates only at the output layer) and from feature-level layer matching (which the paper shows to be unstable, with performance collapsing from 75.8 to 63.3 on MM-Vet when the language head is removed). The fact that verbalization through a shared language head works where direct feature alignment fails is not a quirk of implementation — it is diagnostic evidence that the embedding spaces of models at different scales are geometrically incompatible in ways that the shared vocabulary space is not.

This work also resolves a latent tension in the distillation literature. Prior work was split on whether forward KL divergence (LLAVADI, Minitron) or reverse KL divergence (DistiLLM, MiniLLM) is preferable for language model distillation. VLsI's ablation in Table 4(d) comes down firmly on the forward KL side for intermediate-layer alignment, but with the crucial nuance that KL divergence must operate on vocabulary distributions, not on hidden states. This suggests the prior debate was confounded by the representation space — KL divergence over continuous features may have different properties than KL divergence over token probabilities, and the answer to "which divergence works best" depends on what space the divergence is computed in. By standardizing on vocabulary-space KL divergence through the verbalizer architecture, VLsI provides a clean experimental platform for future investigations of distribution-matching objectives.

More broadly, this paper contributes to a growing recognition that natural language is not just a task output for language models, but can serve as an internal representational medium for knowledge transfer. The conclusion's statement — "natural language is an important key in transferring knowledge not only for humans but also for AI" — is ambitious, but the evidence supports a more precise version: token probability distributions, when projected through a shared vocabulary, provide a semantically meaningful and geometrically stable metric for comparing internal model states across architectures and scales. This insight generalizes beyond VLMs: any pair of models that share a tokenizer and vocabulary can, in principle, use verbalization-based intermediate alignment, opening the door to cross-architecture distillation (encoder-decoder to decoder-only, dense to mixture-of-experts) as long as the vocabulary constraint can be satisfied or relaxed.

The paper also shifts the Overton window on what counts as an acceptable distillation pipeline. Prior work — especially in the VLM space — treated distillation as a single-stage process: train the student to match the teacher's outputs. VLsI's three-stage pipeline (verbalization → interaction → SFT) is more complex and more expensive, but the results justify the complexity: the interaction step alone is counterproductive (Table 4b), and the SFT step unlocks the transferred knowledge. This suggests that effective distillation from capacity-mismatched teachers may intrinsically require a decoupling of representation transfer from task optimization — a principle that, if confirmed by future work on other model families and modalities, would fundamentally change how distillation pipelines are designed. The days of "just minimize KL divergence at the output layer" may be numbered for settings where the teacher-student capacity gap is large.

One research direction this work implicitly closes off is architectural modification as the primary path to small-model performance. The paper explicitly compares against TroL and Phantom — methods that modify the forward pass — and matches or exceeds them on most benchmarks without any architectural change. Table 3(a) shows VLsI surpassing every modification-based VLM on MathVista, MM-Vet, and MMMU. This does not mean architectural innovation is worthless, but it does mean the baseline for any new architectural modification is now higher: to justify the deployment complexity of a non-standard forward pass, a method must demonstrate gains over VLsI-style distillation, not just over the untrained baseline.

---

Follow-Up Research This Work Enables

Cross-tokenizer verbalization through learned vocabulary mappings. The most significant constraint on VLsI's applicability is the requirement that teacher and student share identical tokenizer vocabulary indices. The paper acknowledges this explicitly and leaves it as future work. A direct extension would be to train a lightweight linear mapping W from the teacher's vocabulary distribution to the student's vocabulary distribution, such that the KL divergence is computed after applying W to the teacher's verbalized probabilities: D_KL(P_small || W · P_large). This mapping could be learned from a parallel corpus (same text tokenized with both tokenizers) or even bootstrapped from the verbalization training data by aligning distributions for tokens that appear in both vocabularies. A strong experiment would test VLsI with Qwen2-VL-72B as teacher and a LLaMA-3.2-Vision student — two models with different tokenizers — measuring how much of the VLsI gain survives the cross-tokenizer mapping compared to the same-tokenizer upper bound. If the mapping works, VLsI becomes applicable to essentially any teacher-student pair; if it fails, the shared-tokenizer requirement is a fundamental limitation rather than an implementation convenience.

Dynamic target layer selection based on verbalized output entropy or semantic change. VLsI selects target layers at fixed intervals (every 4th layer starting from the 2nd). This is computationally convenient but almost certainly suboptimal: reasoning transitions — points where the model's understanding qualitatively shifts — may not align with uniform spacing. A natural follow-up would use the already-trained verbalizers to measure where the verbalized output distribution changes most rapidly. For each training example, one could compute the KL divergence between adjacent layers' verbalized distributions and select target layers at local maxima of this divergence — the layers where the model's "verbal understanding" undergoes the largest jumps. The hypothesis is that aligning these transition points would be more effective than aligning uniformly spaced layers, since they capture the key reasoning shifts. A controlled experiment would compare fixed-interval selection against entropy-guided selection matched on the number of target layers, measuring whether the same distillation budget yields better performance when allocated to information-theoretically important layers.

Measuring and visualizing the "expansion then sharpening" dynamic during VLsI training. The paper hypothesizes that the interaction step "expands" the probability space (hurting greedy decoding) and the SFT step "sharpens" it (recovering and surpassing baseline performance), based on the observation that beam search with N=5 partially recovers interaction-only performance. This hypothesis is testable but untested. A diagnostic experiment would track the entropy of the small model's output distribution — specifically, H(P(y | x)) averaged over a validation set — at three checkpoints: (1) pretrained baseline, (2) after interaction step, (3) after SFT step. The prediction is that entropy increases after Stage 2 (broader distribution, more uncertainty per token) and decreases below the baseline after Stage 3 (sharper distribution, more confident in correct answers). Additionally, one could measure top-k accuracy as a function of k at each stage: after interaction, top-1 accuracy drops but top-5 accuracy rises, confirming the "expansion" interpretation; after SFT, top-1 accuracy exceeds baseline while top-5 accuracy remains high, confirming that knowledge was retained while the distribution was sharpened. This diagnostic would validate (or refute) the paper's explanatory model and provide practitioners with a concrete metric — output entropy — that could serve as an early-stopping signal during SFT.

Scaling laws for VLsI: how does student performance scale with teacher size, student size, and distillation budget? The paper uses a single teacher (Qwen2-VL-72B) for most experiments, with one alternative teacher (LLaVA-OV-72B) for generalization. This leaves open the question: given a fixed student size (say, 7B), how does VLsI performance scale with teacher capability? If Qwen2-VL-14B, Qwen2-VL-32B, and Qwen2-VL-72B were all used as teachers for the same 7B student, would the performance curve be concave (diminishing returns — a 32B teacher captures most of the benefit), linear, or convex (accelerating — the 72B teacher provides disproportionately more benefit)? Answering this would inform practical decisions about how large a teacher to train before distilling. Similarly, fixing the teacher at 72B and varying the student from 0.5B to 7B would reveal whether there is a minimum student capacity below which VLsI's gains collapse — does the 0.5B model's 49.9 MMMU (Table 4a) represent the lower bound of what VLsI can achieve, or would a 0.1B student still show meaningful gains?

Stress-testing VLsI on out-of-distribution visual reasoning. The paper evaluates on standard benchmarks that are, by design, within the training distribution of both the teacher and the student's SFT data. A critical stress test would evaluate VLsI-distilled models on tasks that require novel visual reasoning — problems where the visual patterns, reasoning structures, or answer formats differ from the 2.9M training samples. Potential testbeds include: (a) zero-shot performance on visual reasoning benchmarks from different domains (e.g., medical imaging VQA, satellite image interpretation, abstract visual reasoning like Raven's Progressive Matrices), (b) adversarial visual inputs designed to probe whether the transferred layer-wise reasoning patterns are robust or brittle, and (c) multi-step visual reasoning chains where the model must maintain state across multiple images or sub-questions. The hypothesis to test is whether VLsI transfers genuine reasoning procedures (which would generalize) or reasoning patterns that are tightly coupled to the training distribution's specific visual and linguistic statistics (which would not). If VLsI models show strong generalization, the method is genuinely transferring skill; if they show sharp drop-offs on out-of-distribution tasks, the method is closer to sophisticated memorization of teacher behaviors.

Extending verbalization-based alignment to other modalities and model architectures. The verbalizer concept — a lightweight module that projects intermediate hidden states to a shared semantic space — is not specific to vision-language models. It could be applied to: (a) speech-language models, where intermediate layers in an audio encoder could be verbalized to text and aligned with a text-only LLM's layers; (b) video-language models, where temporal reasoning across frames creates a richer layer-wise progression that might benefit even more from verbalization than static-image models; (c) encoder-decoder architectures, where the encoder's layers could be verbalized and aligned with the decoder's layers in sequence-to-sequence tasks. The key requirement is the shared semantic space — for speech, this could be phoneme or word-level tokenization; for video, it could be the same text vocabulary. A minimal viable experiment would apply VLsI to distill a large video understanding model (e.g., Video-LLaMA or LLaVA-NeXT-Video) into a smaller one, measuring whether the temporal reasoning progression — which likely unfolds across more layers than static-image reasoning — benefits from layer-wise verbalized alignment.

---

Practical Applications and Downstream Use Cases

On-device visual assistance with datacenter-grade reasoning. Mobile VLMs face a hard constraint: they must run on hardware with limited memory (typically 4–8 GB RAM) and produce responses within latency budgets appropriate for interactive use (under 500ms). VLsI-2B achieves 64.8 on MM-Vet and 51.4 on MMMU — numbers that, prior to this work, were associated with 7–13B parameter models (Table 1 shows LLaVA-NeXT-13B at 47.3 MM-Vet, VILA1.5-13B at 44.3, InternVL2-8B at 54.2). A 2B model can run on a flagship smartphone today; a 13B model cannot. The practical deployment scenario is visual question answering for accessibility applications — a user points their phone camera at a document, a street sign, or a product label, and asks a natural language question. VLsI-2B's ChartQA score of 85.8 and AI2D score of 89.0 suggest it would reliably answer questions about charts, diagrams, and structured visual information that a baseline 2B model (ChartQA 73.5, AI2D 60.2) would frequently get wrong. The distillation cost — training verbalizers and running the three-stage pipeline — is a one-time investment amortized across millions of inference queries on end-user devices.

Cost-efficient batch inference for document understanding at scale. Organizations that process large volumes of visual documents — insurance companies analyzing claim photos, logistics companies reading shipping labels, archives digitizing historical documents — typically face a choice between (a) using a large VLM with high per-query cost and latency, or (b) using a small VLM with lower cost but unacceptable error rates. VLsI-7B's performance on document and diagram benchmarks (AI2D: 87.3, ChartQA: 86.1) makes it a candidate for replacing much larger models in batch inference pipelines. The key economic metric is total cost of ownership: VLsI-7B, at inference time, has the same memory footprint and latency as Qwen2-VL-7B (since the verbalizers are discarded), but delivers accuracy comparable to models with 10× more parameters. For a pipeline processing 10 million documents per month, the difference between running a 7B model and a 72B model is the difference between a single A100 and a cluster of eight A100s — roughly an order of magnitude in infrastructure cost. The specific numbers from the paper that support this use case: VLsI-7B achieves 87.3 on AI2D vs. Qwen2-VL-72B's 88.3 (within 1 point of a ~10× larger model) and 69.3 on MMMU vs. 64.3 (exceeding the larger model by 5 points).

Teacher models as ongoing resources: iterative distillation for continuous model improvement. VLsI's architecture — where the teacher model is frozen and only verbalizers and the student are trained — enables a deployment pattern where a single large teacher can serve as an ongoing knowledge source for multiple student models at different scales and for different tasks. As the teacher model is updated (e.g., Qwen2-VL-72B → Qwen3-VL-72B), the verbalization step can be re-run on the new teacher, and the interaction and SFT steps can produce updated small models without retraining from scratch. This is more efficient than the current paradigm where each new large model release triggers a separate, independent effort to train small models from scratch or through output-level distillation. The paper's Figure 4 provides evidence that teacher quality directly propagates to student quality (Qwen2-VL-72B teacher yields +14.2 MM-Vet points over LLaVA-OV-72B teacher for the same 7B student), so improvements to the teacher model translate to improvements in all downstream VLsI-distilled students with bounded additional training cost per student.