Training Methods2025~87 minarXiv:2510.19307

Unified Reinforcement and Imitation Learning for Vision-Language Models

score 9Byung-Kwan Lee, Ryo Hachiuma, Yong Man Ro +2NVIDIA↑ 32Oct 22, 2025cs.CV

🎯 Pitch

A student 7B VLM trained via GAIL-style adversarial imitation from multiple 72B teachers can match its teachers on MathVista and MMMU—without any thinking tokens. The key is an LLM-based discriminator that provides a binary similarity reward on top of standard answer rewards, which together shrink the gap to GPT-4o by over 10 points on average across 14 benchmarks.

1. Executive Summary

This paper introduces Unified Reinforcement and Imitation Learning (RIL), a training framework that enables lightweight VLMs (e.g., 7B models) to emulate the text-generation capabilities of substantially larger VLMs (e.g., 72B models) by combining GRPO-based reinforcement learning with a GAIL-style adversarial imitation learning setup. RIL employs an LLM-based discriminator—initialized from the student VLM's own architecture—that provides a binary similarity reward signaling how closely student-generated responses resemble teacher outputs, alongside an LLM-as-a-Judge answer reward that separately verifies factual correctness against ground truth. Across 14 vision-language benchmarks, RIL-trained Qwen2.5-VL-7B and InternVL3-8B models substantially narrow the gap to state-of-the-art open- and closed-source VLMs and in several instances surpass them—for example, Qwen2.5-VL-RIL-7B achieves a 79.7% score on MathVista compared to the base model's 67.8% and matches or exceeds the 72B teacher on several benchmarks—while maintaining the original inference speed by avoiding explicit think-answer chains. The work establishes that imitation from multiple diverse teacher VLMs is consistently more effective than from any single teacher, and that the approach yields particularly strong gains when applied to student models that have already undergone knowledge distillation, demonstrating that an adversarial imitation framework operating purely on natural language responses can transfer sophisticated generation capabilities across model scales without requiring aligned image embeddings or shared tokenizers.

2. Context and Motivation

The Core Problem: Making Powerful VLMs Practically Deployable

The fundamental tension this paper tackles is one that has become increasingly acute as VLMs have matured: the models with the strongest vision-language capabilities are too large to deploy in the settings where they would be most valuable. State-of-the-art VLMs like Qwen2.5-VL-72B, InternVL3-78B, GPT-4o, and Gemini achieve impressive performance across diverse benchmarks, but their computational and memory requirements make them impractical for smartphones, augmented reality devices, edge computing, and other resource-constrained environments where low latency and small model footprints are non-negotiable.

This is not merely an inconvenience—it is a deployment bottleneck that determines whether advanced multimodal AI reaches end-users or remains locked in the cloud. The paper's introduction frames this explicitly: "The pursuit of artificial general intelligence (AGI) has gained momentum," yet "the sheer size and computational demands of these models present significant barriers to deployment in resource-constrained environments such as mobile and embedded devices." The practical stakes are high: if we cannot make strong vision-language capabilities available on-device, we sacrifice privacy (data must leave the device), availability (connectivity-dependent), and latency (network round-trips), while also concentrating access to advanced AI in organizations that can afford datacenter-scale inference.

Where Prior Approaches Fall Short

The paper identifies four broad strategies that have historically driven VLM performance improvements, and explains why each is insufficient as a complete solution to the deployment problem:

1. Scaling model size and training data. This is the dominant paradigm—train larger models on more instruction-tuning data. The results are undeniable (Qwen2.5-VL-72B, InternVL3-78B, LLaVA-OneVision-72B, Claude-3.5 Sonnet, GPT-4o), but they push deployable models away from resource-constrained settings, not toward them. A 72B-parameter model requires multiple high-end GPUs for inference; it simply cannot run on a phone.

2. Architectural modifications. Recent work has explored adding auxiliary vision modules (CoLLaVO, MoAI), multiple vision encoders (Mini-Gemini, MoVA, Eagle), specialized reasoning components (Meteor), or modified layer propagation schemes (TroL, Phantom). The paper argues that while these approaches can improve performance at a given model size, they introduce two problems: increased architectural complexity that makes integration and maintenance harder, and in the case of reasoning modules, substantially increased inference latency from the verbose intermediate "think" steps.

3. The think-answer paradigm with reinforcement learning. Following DeepSeek-R1's success with GRPO, numerous VLM works have adopted a "think before you answer" approach where models generate explicit reasoning chains (LMM-R1, Vision-R1, R1-V, OpenVLThinker, R1-OneVision, R1-Zero, MM-Eureka). The paper directly critiques this trend: "the verbose 'think' responses preceding answers can significantly increase inference latency and computational memory requirements." The core tension is that while these methods improve accuracy—particularly on math and structured reasoning tasks—they do so by generating more tokens at inference time, which is precisely the opposite of what resource-constrained deployment needs.

4. Knowledge distillation. Traditional knowledge distillation transfers knowledge from a large teacher to a smaller student by having the student match the teacher's high-dimensional features or logit distributions. While partially effective (the paper cites MiniLLM, DistilLLM, LLaVA-KD, and the authors' own VLsI), these methods suffer from a critical architectural constraint: they require the student and teacher to share the same image embedding strategies and language tokenizers, including vocabulary, index order, and sequence length. This means you cannot distill from a Qwen2.5-VL teacher with one tokenizer to an InternVL3 student with a different one—a severe limitation given the diversity of VLM architectures. Furthermore, the paper notes that common distillation methods "often rely on KL divergence—a static, non-trainable metric devoid of contextual understanding—to measure feature similarity." The KL divergence between two high-dimensional distributions is a crude signal: it captures global distributional similarity but cannot assess whether the student's response is stylistically appropriate or whether it gets the meaning right.

A deeper issue unifies these four approaches: none of them directly optimizes for what actually matters at deployment time—the quality of the natural language responses the model generates. Scaling increases capacity, architectural modifications add inductive biases, think-answer paradigms add computation, and distillation matches internal representations. But the end-user experience depends entirely on the text the model produces. The paper cites prior work (VLsI and Eagle) arguing that "natural language response-based distillation is more effective than high-dimensional feature distillation," emphasizing what they call the verbalization effect—the idea that how a proficient model articulates its answers in natural language carries information that pure feature matching misses. This insight motivates RIL's core design choice: operate exclusively on generated text responses rather than internal model states.

Conflicting Demands: Accuracy vs. Efficiency vs. Generality

The paper's motivation crystallizes around three simultaneous requirements that existing methods fail to satisfy:

Strong accuracy across diverse vision-language tasks (not just math or structured reasoning).
Low inference latency (no lengthy think chains, no increased model footprint).
Broad compatibility across different VLM architectures, image encoders, and tokenizers.

Methods that excel on one dimension typically sacrifice the others. Scaling model size and think-answer RL improve accuracy at the cost of efficiency. Distillation improves efficiency but is architecturally fragile and provides a weaker learning signal. Architectural modifications are typically model-specific and do not transfer across VLM families. This trilemma—simultaneously achieving accuracy, efficiency, and generality—defines the gap that RIL aims to fill.

How the Paper Positions Itself Relative to Existing Work

The paper positions RIL at the intersection of two established but previously separate frameworks:

From reinforcement learning for VLMs, RIL inherits the GRPO mechanism (Group Relative Policy Optimization) and its improved variant Dr.GRPO, which provide unbiased advantage estimates for policy updates. Prior works applying RL to VLMs (DeepSeek-R1, Perception-R1, VLM-R1) typically use only an answer correctness reward—the model gets a binary signal based on whether its final answer matches the ground truth. RIL adopts this answer reward component (routed through LLM-as-a-Judge rather than simple answer parsing) but argues it is insufficient alone because it provides no signal about the quality of expression—a factually correct but poorly articulated response receives the same reward as a correct and well-written one.

From imitation learning, RIL inherits the GAIL (Generative Adversarial Imitation Learning) framework, which casts learning from an expert as a minimax game between a generator (the student) and a discriminator trained to distinguish student outputs from expert outputs. The key innovation relative to standard GAIL is four specific modifications that adapt the framework from its original robotics context to VLM training:

Combined GRPO + GAIL with explicit reward design: rather than the discriminator's score serving as the sole reward, RIL provides a composite reward with separate similarity and correctness components, ensuring the student pursues both style and substance simultaneously.
Binary discriminator output: the paper converts the discriminator's continuous score (0 to 1) into a binary signal by thresholding at 0.5, providing a cleaner, more decisive learning gradient. The authors argue this stabilizes training because "models often struggle to interpret subtle differences in continuous rewards (e.g., why a score of 0.21 should be preferred over 0.20)."
LLM-as-a-Judge for answer verification: rather than relying on exact-match answer parsing (which works only for domains like math with well-defined answer formats and breaks on open-ended visual questions), RIL uses Qwen2.5-32B as an LLM-based judge to evaluate whether the generated response matches the ground truth semantically. This expands the approach's scope to general visual question answering, chart understanding, and other tasks where answers may be phrased differently but mean the same thing.
Teacher responses included in GRPO optimization: during each training step, the student sees not only its own generated responses but also cached responses from the teacher VLMs. This means the GRPO optimization has access to exemplars of what correct, well-articulated responses look like—a form of direct guidance that helps the student navigate toward good outputs even when its own initial attempts are poor.

The paper explicitly distinguishes RIL from standard knowledge distillation along multiple axes. Unlike distillation, RIL operates on natural language strings, not on internal representations. This means the student and teacher can have completely different vision encoders (CLIP, SigLIP, ConvNeXt), different language backbones (Qwen2, InternLM), different tokenizers (vocabulary sizes, tokenization algorithms), and different image resolution strategies—none of which is possible with standard distillation. The discriminator in RIL is a trainable, language-aware component that learns to assess similarity in a contextually nuanced way, rather than a static distance metric like KL divergence. This is what the paper means when it claims RIL is "agnostic to the specific image embedding strategies or language tokenizers used by the student or teacher VLMs."

The Specific Operational Gap: How to Transfer Sophisticated Output Patterns Without Architectural Alignment

Stepping back, the paper's central motivation can be restated as a specific technical question: Given a small VLM and access to high-quality outputs from a large VLM (or multiple large VLMs), how can we transfer the large model's sophisticated answer-generation capability without requiring any architectural compatibility between them? The answer must work for diverse VLM families (Qwen, InternVL), must maintain inference efficiency (no think chains), and must handle the full breadth of visual question answering tasks (not just math and structured output domains). RIL is the proposed answer: combine GRPO's policy optimization machinery with an adversarially-trained discriminator that learns to recognize good responses by example, and provide both imitation and correctness guidance through a composite reward.

3. Technical Approach

3.1 Reader Orientation

The paper presents RIL as a training algorithm — not a new model architecture or a dataset — that takes an already-pretrained, instruction-tuned small VLM (the student) and teaches it to produce text responses that resemble those from much larger, more capable VLMs (the teachers) while simultaneously ensuring those responses are factually correct. The system solves the problem of transferring sophisticated generative capabilities across model scales without architectural alignment: rather than matching internal representations or logit distributions (which requires compatible tokenizers and vision encoders), RIL operates purely on the generated natural language strings, using an adversarially trained discriminator to define what "good" output looks like and a separate LLM judge to verify correctness against ground truth.

3.2 Big-Picture Architecture (Diagram in Words)

The RIL framework has five interacting components, with data flowing through them in a loop:

Student VLM ( $\pi_\theta$ ) — the small VLM being trained (e.g., Qwen2.5-VL-7B). It generates text responses to image-question pairs, receives composite rewards, and updates its parameters via a GRPO-style policy gradient.
Teacher VLM(s) — one or more large, frozen VLMs (e.g., Qwen2.5-VL-72B, InternVL3-78B) that provide reference text responses. These are pre-generated and cached to avoid repeated inference during training.
Discriminator ( $D_\phi$ ) — a language model initialized from the student VLM's own architecture but with a scalar-output head replacing the vocabulary head. It is trained to output low scores (close to 0) for teacher responses and high scores (close to 1) for student responses, and its thresholded output provides the similarity reward signal.
LLM-as-a-Judge — a frozen large language model (Qwen2.5-32B) that evaluates whether a generated response matches the semantic meaning of the ground truth answer, producing a binary answer reward.
RIL Training Loop — an iterative process where the discriminator is updated to distinguish student from teacher outputs, then the student VLM is updated via Dr.GRPO using a composite reward that combines similarity (from the discriminator) and correctness (from the judge), with teacher responses included in the optimization batch to provide direct exemplars.

Information flows as follows: image-question pairs enter → student VLM generates responses → discriminator scores each response for similarity to teacher style → LLM-as-a-Judge evaluates each response for factual correctness → composite reward is computed → student VLM parameters are updated via policy gradient → discriminator is retrained on fresh student outputs to maintain its discriminative ability → cycle repeats.

3.3 Roadmap for the Deep Dive

First, the SFT warm-up phase (Section 3.3 preamble), which is prerequisite context for understanding how the student VLM is initialized before the RIL loop begins.
Second, the discriminator architecture and pre-training (Section 3.2), because the discriminator is the novel component that enables imitation learning in text space and must be competent before the RIL loop starts.
Third, the RIL training loop itself (Section 3.3), covering how student and teacher responses are gathered, how the discriminator is updated, how the composite reward is constructed from similarity and answer signals, and how Dr.GRPO is applied to update the student — this is the core algorithm.
Fourth, the specific design choices (binary thresholding, multiple teachers, teacher responses in the optimization batch) and why they matter for training stability and performance.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a training methodology paper whose core idea is that a small VLM can learn to produce high-quality text responses by simultaneously (a) imitating the output patterns of large VLMs through an adversarially trained text discriminator, and (b) optimizing for factual correctness through an LLM-based judge, with both signals combined into a single composite reward used to update the student via Dr.GRPO.

Supervised Fine-Tuning (SFT) Warm-Up Phase

Before the RIL loop begins, the student VLM undergoes a preliminary supervised fine-tuning stage on a comprehensive 4-million-sample visual instruction tuning dataset. This dataset spans general visual question answering, dense image captioning, chart/diagram/document understanding, common-sense knowledge, scientific and mathematical problem-solving, and multi-dimensional reasoning — drawn from dozens of sources including COCO-ReM, VQA-v2, MAVIS, Geometry3K, ScienceQA, AI2D, ChartQA, DocVQA, and many others (full list in Appendix C).

The SFT phase serves as a crucial warm-up: it acclimates the student VLM to the target data distribution and ensures it has basic instruction-following capability before the more complex RIL optimization begins. The paper states that this SFT stage uses the AdamW optimizer with a linearly decayed learning rate from $1 \times 10^{-5}$ to $1 \times 10^{-6}$ . After SFT, a 40,000-sample subset is curated from the full 4M dataset for the RIL loop using log-probability sampling and overlong filtering. The SFT checkpoint serves as the initial policy $\pi_{\theta_{\text{init}}}$ for the RIL process and also as the reference model $\pi_{\text{ref}}$ for the KL-divergence penalty in the Dr.GRPO objective.

Discriminator Architecture and Pre-training

The discriminator is the central mechanism that enables imitation learning in text space. Unlike standard knowledge distillation, which matches internal model representations, the RIL discriminator operates on natural language text strings and learns to assess whether a response "reads like" it came from a teacher VLM or a student VLM.

Architecture. The discriminator $D_\phi$ is a language model whose backbone architecture and initial parameters are identical to the student VLM's. The paper explicitly states this is done "to promote training stability and mitigate the balance problem common in adversarial setups — where one component might overpower the other." The balance problem refers to the well-known GAN training instability where the discriminator becomes too strong too quickly (providing no useful gradient to the generator) or too weak (providing a meaningless signal). By starting the discriminator from the same initialization as the student, the two begin on equal footing.

The only architectural modification is to the output head: the standard language modeling head, which maps a $d$ -dimensional hidden state to a $v$ -dimensional vocabulary logit vector ( $\mathbb{R}^{d \times v}$ ), is replaced with a linear discriminator head that maps to a single scalar ( $\mathbb{R}^{d \times 1}$ ). The input to this head is the representation of the final sequence token from the last layer of the backbone model. A sigmoid function is applied to this scalar to produce a score in $[0, 1]$ .

Input format. The discriminator receives a question $q$ and a generated text response $o$ as input, formatted using a specific prompt template shown in Figure 3 of the paper. This prompt structures the input so the discriminator can process the question-response pair as a unified text sequence, enabling it to evaluate the response in context rather than in isolation.

Pre-training objective. Before the RIL loop begins, the discriminator is pre-trained on a dataset of responses collected from both student and teacher VLMs. For each question $q$ , $N$ text responses are generated from the student VLM (denoted $\{\mathbf{o}_i^{(s)}\}_{i=1}^N$ ) and $N$ from the teacher VLM (denoted $\{\mathbf{o}_i^{(t)}\}_{i=1}^N$ ). The discriminator is trained to maximize:

$\max_\phi \mathcal{L}(\phi) = \frac{1}{N} \sum_{i=1}^N \left\{ \log D_\phi(q, o_i^{(s)}) + \log(1 - D_\phi(q, o_i^{(t)})) \right\}$

where $D_\phi(q, o_i^{(s)})$ is the discriminator's score for a student response, and $D_\phi(q, o_i^{(t)})$ is its score for a teacher response.

What it computes: the standard binary cross-entropy objective for a classifier distinguishing between two classes. The first term $\log D_\phi(q, o_i^{(s)})$ encourages the discriminator to output values close to 1 for student-generated responses — it is maximized when $D_\phi$ is confident that the response came from the student. The second term $\log(1 - D_\phi(q, o_i^{(t)}))$ encourages the discriminator to output values close to 0 for teacher-generated responses — it is maximized when $D_\phi$ is confident that the response came from the teacher. The sum is a single scalar per training example.

Why this form: this is the standard maximum-likelihood objective for binary classification, and it directly establishes the desired behavior: the discriminator learns to map teacher responses to 0 and student responses to 1. This directional choice — teacher = 0, student = 1 — is important because it determines the interpretation of the similarity reward later: a discriminator score below 0.5 indicates the response resembles a teacher output (since teachers were trained to receive 0), and this threshold is used to produce the binary similarity reward.

Pre-training data scale. The paper reports generating $N = 16$ responses per question for both student and teacher, and using 40,000 questions to collect a total of 1.2 million samples ( $40\text{K} \times 16 \times 2$ ). This pre-training phase takes approximately 1 to 3 days on 256 NVIDIA A100 GPUs, using vLLM for fast text generation and DeepSpeed ZeRO-3 for distributed training.

Practical implementation details. The paper uses $N = 16$ generations per question for pre-training, AdamW optimizer with learning rate linearly decayed from $1 \times 10^{-5}$ to $1 \times 10^{-6}$ , and 8 GPUs for vLLM-based generation followed by 8 GPUs with DeepSpeed ZeRO-3 for training.

Why pre-training matters. The paper emphasizes that "without adequate pre-training, $D_\phi$ would yield unreliable scores, undermining its utility and potentially degrading overall model performance." If the discriminator cannot reliably distinguish student from teacher outputs, its similarity reward signal is noise, and the student receives no meaningful imitation guidance. The pre-training ensures the discriminator enters the RIL loop with basic discriminative competence.

The RIL Training Loop: Core Algorithm

The RIL loop is the iterative process that alternates between updating the discriminator and updating the student VLM. The full procedure is specified in Algorithm 2 in Appendix B. Here is the step-by-step flow:

Step 1: Initialize models. The reference model $\pi_{\text{ref}}$ is set to the SFT checkpoint $\pi_{\theta_{\text{init}}}$ , and the training model $\pi_\theta$ is initialized to the same checkpoint. The discriminator $D_\phi$ is loaded from pre-training. Teacher responses have been pre-generated and cached.

Step 2: Sample a batch and generate student responses. For each question $q$ in a training batch $\mathcal{B}$ , the current student policy $\pi_{\theta_{\text{old}}}$ (frozen copy of the current parameters) generates $G$ text responses, denoted $\{o_i^{(s)}\}_{i=1}^G$ . The paper uses $G = 4$ responses per question during the RIL loop. Generation parameters are: temperature = 1.0, top-p = 0.95, top-k = 50, repetition penalty = 1.05 — these settings encourage diverse outputs.

Step 3: Retrieve teacher responses. For the same question $q$ , $G$ pre-generated teacher responses are extracted from the cache, denoted $\{o_i^{(t)}\}_{i=1}^G$ . These are fixed and do not change during training (teacher VLMs are frozen). This yields a combined set of $2G$ responses $\{o_i\}_{i=1}^{2G}$ for each question — half from the student, half from the teacher.

Step 4: Update the discriminator. Using the $2G$ responses, the discriminator $D_\phi$ is updated via Equation 1 (the same binary cross-entropy objective used in pre-training). The paper uses $\mu = 1$ discriminator update iteration per RIL cycle, finding that more iterations provide diminishing returns (Table 5a). This step ensures the discriminator adapts to the evolving student output distribution and maintains its ability to distinguish student from teacher responses.

Why update the discriminator during training? As the student policy improves, its generated responses shift in distribution. A discriminator trained only once on initial student outputs would become increasingly unreliable as the student's output quality improves (because student responses that look more like teacher responses might receive scores near 0.5 — ambiguous and uninformative). Continuous re-training keeps the discriminator calibrated to the frontier of student capability. Table 5d confirms this empirically: using a fixed, non-updated discriminator outperforms RL-only baselines but significantly underperforms the continuously trained discriminator.

Step 5: Compute rewards and advantages. For each of the $2G$ responses, a composite reward $R(q, o_i)$ is computed as the sum of two binary components:

Similarity Reward: $\mathbb{1}(D_\phi(q, o_i) < 0.5)$ — this indicator returns 1 if the discriminator's score for response $o_i$ is below 0.5, meaning the discriminator classifies it as teacher-like (recall: teacher = 0 target during training). It returns 0 otherwise.
Answer Reward: the output of LLM-as-a-Judge $(q, a, o_i)$ , where $a$ is the ground truth answer. The judge evaluates whether the generated response $o_i$ is semantically consistent with $a$ , returning a binary 1 (correct) or 0 (incorrect). The judge uses the prompt template shown in Figure 4 of the paper and is implemented using Qwen2.5-32B.

The total reward is $R(q, o_i) = \mathbb{1}(D_\phi(q, o_i) < 0.5) + \text{LLM-as-a-Judge}(q, a, o_i)$ , producing a value in $\{0, 1, 2\}$ for each response.

Why binary rewards? The paper argues that continuous discriminator scores "can introduce ambiguity into the learning signal" because models struggle to interpret subtle differences (e.g., why a score of 0.21 should receive a different update than 0.20). Binarizing at 0.5 provides a clean, decisive signal. Figure 6 empirically validates this choice: binary rewards consistently outperform continuous and multi-level discretized rewards across four benchmarks. The answer reward is inherently binary since the LLM-as-a-Judge makes a yes/no determination.

Why include the answer reward at all? The discriminator alone captures stylistic similarity but "does not inherently verify factual correctness against ground truth answers, which could otherwise lead to performance degradation." A student could learn to produce responses that read like teacher outputs but are factually wrong — high similarity, low accuracy. The answer reward prevents this by providing an independent correctness signal that the student must also optimize.

Advantage computation. Once all $2G$ responses have been scored, advantages $\hat{A}_i$ are computed using the Dr.GRPO formulation:

$\hat{A}_i = R(q, o_i) - \frac{1}{2G} \sum_{j=1}^{2G} R(q, o_j)$

where $R(q, o_i)$ is the composite reward for response $o_i$ and the second term is the mean reward across all $2G$ responses for question $q$ .

What it computes: the advantage for each response is its reward minus the average reward of all responses in the group for that question. A positive advantage means the response scored above average; a negative advantage means it scored below average.

Why this form: Dr.GRPO uses group-relative advantages rather than absolute rewards because it removes the need for a learned value function (critic). By normalizing rewards within each group, the advantage estimates are unbiased and the scale of rewards does not affect the policy gradient — only the relative ordering within the group matters. This is computationally simpler than training a separate value network and avoids the instability issues that can arise from poorly estimated value baselines. The $2G$ responses for Dr.GRPO include both student and teacher responses, meaning teacher responses that receive high rewards (similarity + correctness) will have positive advantages and will be imitated.

Step 6: Update the student VLM. The student policy $\pi_\theta$ is updated by maximizing the Dr.GRPO objective over all $2G$ responses:

$\max_\theta \mathcal{L}(\theta) = \frac{1}{2G} \sum_{i=1}^{2G} \left\{ \min\left[r_i(\theta)\hat{A}_i, \text{clip}\left(r_i(\theta), 1-\epsilon, 1+\epsilon\right)\hat{A}_i\right] - \beta \mathcal{D}_{\text{KL}}\left(\pi_\theta \mid \pi_{\text{ref}}\right) \right\}$

where:

$r_i(\theta) = \frac{\pi_\theta(o_i \mid q)}{\pi_{\theta_{\text{old}}}(o_i \mid q)}$ is the probability ratio — how much more (or less) likely the current policy makes response $o_i$ compared to the policy that generated it. A value > 1 means the current policy assigns higher probability to this response than before; < 1 means lower probability.
$\hat{A}_i$ is the group-relative advantage from Step 5.
$\epsilon$ is the clipping hyperparameter, set to 0.2 throughout all experiments.
$\beta$ controls the strength of the KL-divergence penalty $\mathcal{D}_{\text{KL}}(\pi_\theta \mid \pi_{\text{ref}})$ , which measures how far the current policy has diverged from the reference (initial SFT) policy. KL divergence is a measure of distributional distance — it increases when $\pi_\theta$ assigns high probability to outputs that $\pi_{\text{ref}}$ assigns low probability to.
$\pi_{\text{ref}}$ is the initial SFT checkpoint, frozen throughout RIL training.

What it computes: a clipped surrogate objective with a KL penalty — the standard PPO/GRPO formulation. The $\min$ operation inside the sum compares the unclipped probability ratio times advantage with the clipped version, taking whichever is smaller. When the advantage is positive (good response), the clip prevents the policy from increasing its probability too aggressively (the ratio cannot exceed $1 + \epsilon$ ). When the advantage is negative (bad response), the clip prevents the policy from decreasing its probability too aggressively (the ratio cannot fall below $1 - \epsilon$ ). The KL penalty $\beta \mathcal{D}_{\text{KL}}$ is subtracted from this term, so the overall objective balances reward maximization against staying close to the reference distribution.

Restated operationally: for each of the $2G$ responses, the objective computes whether the student should increase or decrease the probability of generating that response (based on whether its advantage is positive or negative), clips the magnitude of change to prevent destructive updates, and subtracts a penalty for diverging too far from the initial SFT behavior. The result is a scalar objective that is maximized by gradient ascent, updating $\pi_\theta$ 's parameters.

Why this form: the clipping mechanism is the key innovation of PPO/GRPO-style objectives — it creates a trust region that prevents the policy from changing so much in a single update that it collapses or enters a degenerate regime. Without clipping, a single high-advantage response could cause the policy to massively increase its probability for that specific output, overfitting to individual examples and destabilizing training. The KL penalty provides an additional regularizer that prevents catastrophic forgetting of the SFT capabilities. Together, they ensure stable, gradual improvement.

Why include teacher responses in the optimization batch? This is a critical design choice. The $2G$ responses include both student-generated and teacher-generated text. When the student VLM is weak and fails to produce any correct responses for a given question, the GRPO update would have no positive examples to learn from — all $G$ student responses would have low advantages. By including teacher responses (which typically receive both the similarity reward and often the answer reward), the optimization batch always contains at least some high-advantage examples, providing "clearer guidance, especially when the student VLM lacks sufficient domain-specific knowledge compared to larger models." This also "creates opportunities for the student VLM to potentially surpass the performance of larger teachers" because the student is not merely imitating — it is optimizing a composite objective that can in principle exceed teacher performance if the answer reward provides additional signal beyond what the teacher's responses demonstrate.

Training hyperparameters for the RIL loop. The paper uses:

$\mu = 1$ update iteration for both discriminator and student per RIL cycle (Table 5a shows more iterations risk overfitting, especially for the student)
$\epsilon = 0.2$ clipping parameter
$\beta$ KL penalty coefficient (swept in Table 5b — too large $\beta$ hinders performance by being overly restrictive)
Static learning rate of $1 \times 10^{-6}$ for both student and discriminator during the RIL loop
Gradient accumulation with 6 steps, 4 batches per GPU, 8 GPUs total (effective batch size = $4 \times 6 \times 8 = 192$ batches)
Temperature = 1.0, top-p = 0.95, top-k = 50, repetition penalty = 1.05 for text generation
$G = 4$ responses generated per question per RIL iteration

Computational cost. The RIL loop (with teacher responses pre-cached) takes approximately 3 to 5 days on 8 NVIDIA A100 GPUs for 40,000 training questions. The paper notes that costs can be mitigated by offloading discriminator and student weights between CPU and GPU using DeepSpeed API, loading only the model being updated at each step — feasible because both share the same architecture.

Multiple Teacher VLMs

A significant performance lever identified in the paper is using responses from multiple large teacher VLMs rather than a single teacher. Table 2 demonstrates this empirically: using both Qwen2.5-VL-72B and InternVL3-78B as teachers consistently outperforms using either alone across all benchmarks.

Mechanism. When multiple teachers are used, the discriminator is trained to output zero for responses from any teacher VLM (i.e., both Qwen2.5-VL-72B and InternVL3-78B responses are treated as the "expert" class) and one for student responses. The paper describes this simply: "given a response 'T1' from one teacher VLM and 'T2' from another teacher VLM, and a response 'S' from the student VLM, then the discriminator is trained to output zero when 'S' is provided and one when either 'T1' or 'T2' is provided."

Why multiple teachers help. The paper attributes this to two factors. First, "the richer diversity of textual responses" from multiple teachers strengthens the discriminator's training because it sees a wider variety of high-quality outputs as the "teacher" class, making it more robust to stylistic variation and less likely to overfit to idiosyncrasies of any single teacher. Second, incorporating this diverse set into the GRPO optimization provides the student with "clearer and more varied exemplars of correct answer generation" — when the student sees multiple valid ways to answer the same question, it can learn to generalize rather than copying a single teacher's specific phrasing patterns. This is particularly important for open-ended visual questions where many different phrasings can all be correct; a single teacher's output represents only one valid formulation.

Parameter Update Strategy

Table 5c in the paper reports an ablation study on which parameter groups of the student VLM should be updated during RIL training. The finding is that training the self-attention layers, word embeddings, and language head yields the most significant gains, while updating feed-forward networks (FFN) or layer normalization parameters provides diminishing returns.

Why this pattern? Self-attention layers control how the model relates different parts of the input (image tokens, question tokens, generated tokens), so updating them directly affects the model's ability to attend to relevant visual and textual context when formulating responses. Word embeddings and the language head are the interface between the model's internal representations and the vocabulary — updating them allows the model to adjust which words it selects and how it represents them. FFN layers, by contrast, process information within individual positions after attention has mixed information across positions; they may be less critical for the kind of stylistic and factual alignment RIL targets. Layer normalization parameters control activation scaling and shifting, which the paper finds is relatively stable and does not need adjustment for RIL to work effectively.

Interaction with Prior Knowledge Distillation

Table 5f reports a particularly important finding: RIL is more effective when applied to student models that have already undergone knowledge distillation (MiniLLM, DistilLLM, LLaVA-KD, VLsI) compared to non-distilled counterparts. The paper hypothesizes that "prior feature alignment through distillation primes them for RIL's alignment mechanisms."

Why this synergy exists. Knowledge distillation aligns internal model representations — feature maps, attention patterns, hidden states — between teacher and student. This creates internal representations in the student that are structurally similar to those in the teacher. RIL then operates on top of this aligned internal foundation, refining the surface-level text generation to more closely match teacher outputs. A student that already has teacher-like internal representations will produce text that is closer to teacher text to begin with, giving RIL a stronger starting point and making the discriminator's job easier (since initial student outputs are already somewhat teacher-like, the discriminator must learn finer distinctions). Additionally, distillation may transfer some of the teacher's knowledge about what constitutes a good response, making the student more receptive to the imitation signal.

Why Operate on Text Rather Than Features?

The paper makes a deliberate architectural choice to operate exclusively on generated text responses rather than internal model states. Section 4.3 provides the rationale:

Architectural independence. Operating on text means the student and teacher VLMs can use "completely different vision encoders (CLIP, SigLIP, ConvNeXt), different language backbones (Qwen2, InternLM), different tokenizers (vocabulary sizes, tokenization algorithms), and different image resolution strategies." None of this is possible with feature-level distillation, which requires the student to match the teacher's internal representations — and therefore requires compatible architectures.

Contextual similarity assessment. The discriminator, being a language model itself, "learns to discern nuanced similarities and stylistic differences between student and teacher outputs." This is fundamentally different from a static distance metric like KL divergence, which measures global distributional similarity without any language understanding. The discriminator can learn that "approximately 20 percent" and "around one fifth" are similar expressions (even though they share no tokens), or that a teacher-like response explains reasoning before giving the answer while a student-like response jumps straight to the conclusion — distinctions that token-level KL divergence on logits would miss entirely.

The verbalization effect. The paper explicitly invokes the concept from prior work (VLsI) that "how proficient VLMs articulate correct and contextually appropriate answers" carries information beyond what features capture. The specific phrasing, level of detail, logical flow, and tone of a response are all aspects of generation quality that manifest in the text itself — not just in the semantic content. RIL's text-level discriminator can capture these surface patterns directly.

4. Key Insights and Innovations

Innovation 1: Text-Level Adversarial Imitation as an Alternative to Feature-Level Distillation

The dominant paradigm for transferring knowledge from large to small VLMs has been feature-level distillation — having the student match the teacher's internal representations, whether logit distributions (Hinton et al., 2015), hidden states, or attention patterns. This approach has an architectural dependency that is so fundamental it is rarely questioned: because the student must reproduce the teacher's internal activations, both models must share compatible vision encoders, language backbones, tokenizers, and embedding dimensionalities. RIL's central conceptual move is to relocate the knowledge transfer interface from internal model states to natural language text strings, using an adversarially trained language-model discriminator rather than a static distance metric (KL divergence) as the similarity function. This is not an incremental improvement to distillation — it is a category shift in what is being transferred (surface generation patterns rather than internal representations) and how similarity is assessed (by a trainable, context-aware language model rather than a fixed mathematical divergence).

The contrast with prior work makes the shift clear. Standard distillation methods (MiniLLM, DistilLLM, LLaVA-KD) and the authors' own prior method VLsI all operate on logits or intermediate features, requiring the student to minimize KL divergence between its output distribution and the teacher's. KL divergence is a global, symmetric measure that treats all vocabulary items as independent — it cannot capture whether "approximately 20 percent" and "around one fifth" are semantically interchangeable, nor can it assess whether a response's logical structure mirrors teacher-like reasoning patterns. RIL's discriminator, by contrast, is a full language model that processes the question-response pair as a unified text sequence and learns to make a holistic judgment: does this response read like it came from a large VLM? This is a fundamentally richer signal because it captures stylistic patterns, reasoning structure, lexical choices, and answer formatting simultaneously, without needing to decompose them into separate objectives.

The significance extends beyond performance gains. By decoupling knowledge transfer from architectural compatibility, RIL makes the choice of teacher and student VLMs independent design decisions. A practitioner can pair a Qwen2.5-VL teacher (using CLIP vision encoder, Qwen2 tokenizer) with an InternVL3 student (using InternViT vision encoder, InternLM tokenizer) — a combination that is impossible with feature-level distillation. This architectural agnosticism is a conceptual advance, not just a practical convenience: it reframes VLM compression from an architecture-constrained optimization problem to a text-supervised learning problem where any model capable of generating text can be the student and any model that produces high-quality text can be the teacher. The evidence in Table 4 bears this out: RIL-trained Qwen2.5-VL-7B and InternVL3-8B both show substantial gains, and the approach works across model families with completely different design philosophies.

Innovation 2: The Discriminator as a Dynamic, Co-Evolving Quality Metric

Standard imitation learning frameworks (including the original GAIL on which RIL is based) typically use a fixed expert policy whose outputs are treated as ground-truth demonstrations. The discriminator is trained once to distinguish expert from novice outputs, and the generator is then trained against this fixed discriminator. RIL introduces a critical modification that changes the learning dynamics: the discriminator is continuously re-trained throughout the optimization loop on the student's evolving output distribution. This transforms the discriminator from a static quality evaluator into a co-evolving critic that adapts to the student's current capability frontier.

To appreciate why this matters, consider what would happen with a fixed discriminator. At the start of training, the student generates poor responses that a pre-trained discriminator can easily distinguish from teacher outputs — scores are reliably near 1 for student and 0 for teacher, providing a clear gradient. As the student improves and its responses become more teacher-like, the fixed discriminator's scores shift toward 0.5 (uncertainty), because the student is now producing outputs in the boundary region between what the discriminator was trained to recognize as student vs. teacher. The gradient signal degrades precisely when the student needs it most — when it is close to matching teacher quality but needs fine-grained feedback to close the remaining gap. Continuous discriminator re-training prevents this: by re-training on the student's latest outputs, the discriminator learns to make progressively finer distinctions, maintaining a sharp gradient signal throughout training.

The empirical evidence in Table 5d quantifies the importance of this design choice: using a fixed (pre-trained only) discriminator provides some benefit over RL-only baselines, but the fully continuous discriminator delivers substantially larger gains. The paper characterizes this as addressing "the balance problem common in adversarial setups" where one component overpowers the other, but the deeper insight is about maintaining information content in the reward signal as the student's capabilities evolve. This is a conceptual contribution to the design of adversarial imitation learning systems more broadly — it suggests that for tasks where the expert-generator gap narrows substantially over training, discriminator co-evolution is not merely a stability mechanism but a necessary condition for sustained learning.

Innovation 3: Composite Reward Design That Separates Style from Substance

Prior work applying RL to VLM training (DeepSeek-R1, Vision-R1, VLM-R1, Perception-R1) uses a single reward signal — typically answer correctness, determined by parsing the model's output and comparing against a ground truth answer. This conflates two distinct dimensions of generation quality: whether the response is factually correct (substance) and whether it is well-articulated in the manner of a capable model (style). RIL's key insight is that these two dimensions should be rewarded through separate, independently computed signals that are then combined additively, because they capture different aspects of what makes a response good and can be optimized simultaneously without interference.

The separation is not a superficial engineering choice. The similarity reward from the discriminator captures aspects of generation quality that correctness signals cannot: appropriate level of detail, logical structure, explanatory clarity, and stylistic alignment with large-model outputs. These are precisely the dimensions along which smaller VLMs typically underperform even when their answers happen to be correct — a 7B model might produce the right answer but in a terse, uninformative, or awkwardly phrased way. The answer reward from LLM-as-a-Judge captures factual accuracy independent of style — a response can be teacher-like in its phrasing but factually wrong, or correct but phrased like a small model. By summing these orthogonal signals, RIL creates a composite reward landscape where the global optimum is responses that are both correct and well-articulated.

The contrast with prior RL-for-VLMs work is instructive. Methods like DeepSeek-R1 and its VLM derivatives use only answer rewards, which means the optimization is blind to response quality beyond correctness. A student that learns to output correct answers but in a degenerate, terse, or uninformative format would receive the same reward as one that produces correct, well-explained, teacher-like responses. The prevalence of "think-answer" formats in these prior approaches can be partly understood as a workaround: by forcing the model to generate explicit reasoning tokens before answering, these methods create an implicit style constraint (the reasoning must be logically coherent to reach the right answer), but at the cost of substantially increased inference latency and compute. RIL achieves style guidance without requiring a format change, by making imitation of teacher style an explicit optimization target through the discriminator reward.

The empirical validation of this design choice is distributed across multiple results. Table 1 shows that RIL (with both similarity and answer rewards) significantly outperforms RL-only baselines (GRPO and Dr.GRPO with only answer rewards) across all 14 benchmarks. Figure 5 shows that both similarity and answer rewards increase over the course of RIL training, confirming that the student is simultaneously improving on both dimensions. Figure 6 demonstrates that binarizing the similarity reward — making the style signal decisive rather than continuous — improves performance, suggesting that clean separation and clear signals matter for optimization.

Innovation 4: Multiple Diverse Teachers as a Training Signal Amplifier

The default assumption in knowledge distillation and imitation learning is that a single expert provides the target behavior. RIL demonstrates empirically that using multiple large teacher VLMs simultaneously — with responses from all of them treated as the "expert" class — consistently and substantially outperforms using any single teacher. This finding challenges the implicit assumption that the best teacher is simply the single highest-performing model and reveals that teacher diversity is an independent axis of training signal quality.

The mechanism by which multiple teachers improve learning is not obvious from the architecture alone, and the paper's explanation suggests two distinct effects. The first is on the discriminator: when trained to recognize responses from multiple teacher VLMs all as the positive class, the discriminator must learn a more abstract, general notion of "high-quality VLM output" rather than overfitting to the specific phrasing patterns, common phrases, or formatting conventions of any single teacher model. This makes the similarity reward more robust — it captures what makes a response teacher-like in a general sense rather than what makes it specifically Qwen2.5-VL-72B-like. The second is on the student optimization: when teacher responses from multiple models are included in the GRPO batch, the student sees multiple valid ways to answer the same question, providing richer exemplars for policy improvement. Instead of learning to copy one teacher's specific answer style, the student learns that several different phrasings, levels of detail, and reasoning structures are all valid — which should promote generalization.

The significance of this finding is not just the performance improvement (visible in Table 2) but the implication it carries for how we think about teacher selection in model compression. The conventional wisdom — pick the single best teacher available — may be suboptimal. A diverse ensemble of strong teachers can provide a more informative training signal than any individual member, even if some ensemble members are individually weaker than the single best model. This is reminiscent of ensemble distillation effects observed in other contexts but applied here to the qualitatively different setting of adversarial imitation in text space. It also has practical implications: organizations with access to multiple large VLMs (through APIs or open-source releases) can combine their outputs to train a single small model that benefits from the collective strengths of all of them, potentially exceeding what could be achieved by distilling from any one alone.

Innovation 5: Synergy with Prior Distillation — A Two-Stage Knowledge Transfer Architecture

One of the paper's most intriguing findings is that RIL is substantially more effective when applied to student VLMs that have already undergone knowledge distillation compared to those trained only with standard SFT (Table 5f). This is not merely an empirical curiosity — it reveals a two-stage knowledge transfer dynamic where feature-level distillation and text-level imitation are complementary rather than competing approaches, and where their combination produces gains that exceed either alone.

The paper's hypothesis — that "prior feature alignment through distillation primes [the student] for RIL's alignment mechanisms" — points to a deeper architectural insight. Knowledge distillation aligns the student's internal representations with the teacher's, meaning the student's hidden states, attention patterns, and intermediate features already encode information in a teacher-like format before RIL begins. When RIL then trains the student to produce teacher-like text, it is operating on top of an internal representation space that is already structured similarly to the teacher's — the mapping from internal states to output text is being refined, but the internal states themselves are already in the right neighborhood. For a non-distilled student, RIL must simultaneously learn to restructure internal representations and refine output text, a harder joint optimization problem.

This finding suggests a general principle for model compression: feature-level and text-level knowledge transfer address different aspects of the student-teacher gap and can be composed sequentially. Feature-level distillation (whether logit-based, hidden-state-based, or attention-based) transfers the teacher's internal computation patterns — how the model processes information. Text-level imitation through RIL transfers the teacher's surface generation patterns — what the model ultimately produces. Neither is a complete solution alone, but together they cover complementary aspects of the knowledge transfer problem. This is a conceptual contribution to the design of VLM compression pipelines: rather than choosing between distillation approaches, practitioners can stack them, with feature-level alignment creating a foundation that makes subsequent text-level imitation more effective.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use 14 vision-language evaluation benchmarks: AI2D, ChartQA, MathVista, MMB, MMB CN, MM-Vet, MM-Vet-v2, MMMU, MMMU-Pro, MMStar, BLINK, SEED, SEED2+, and RealWorldQA. Each benchmark targets a specific capability — AI2D tests diagram understanding, ChartQA and DocVQA test chart/document comprehension, MathVista tests mathematical reasoning in visual contexts, MMB tests general multimodal perception, MM-Vet and MM-Vet-v2 test integrated vision-language capabilities, MMMU and MMMU-Pro test multi-discipline college-level multimodal understanding, BLINK tests visual perception versus recognition, SEED tests multimodal generative comprehension, and RealWorldQA tests real-world visual question answering. The paper also reports the average score across all 14 benchmarks as a summary metric. Exact test splits follow the standard evaluation protocols associated with each benchmark — no custom splits are introduced.
Base model(s). RIL is validated on five student VLM families spanning a wide parameter range: Qwen2.5-VL (3B and 7B variants), InternVL3 (1B, 2B, and 8B variants), Qwen2-VL (2B), InternVL2 (1B, 2B, 4B), and InternVL2.5 (1B, 2B, 4B, 8B). The primary focus is on the 7B–8B scale (Qwen2.5-VL-7B and InternVL3-8B), which the paper positions as the sweet spot for demonstrating that lightweight models can rival much larger counterparts. For teacher VLMs, the paper uses Qwen2.5-VL-72B and InternVL3-78B — the largest, highest-performing variants in their respective families — selected because of their strong benchmark performance on the VLM Leaderboard. A frozen Qwen2.5-32B serves as the LLM-as-a-Judge for answer verification throughout all experiments.
Metrics. The primary metric for every benchmark is accuracy (%) — the fraction of test questions for which the model's generated response is judged correct by the benchmark's standard evaluation protocol. For most benchmarks (math, multiple-choice, structured answer tasks), this uses exact-match or rubric-based grading provided by the benchmark authors. For the answer reward used internally during RIL training, correctness is determined by LLM-as-a-Judge (Qwen2.5-32B) using the prompt template shown in Figure 4, which asks the judge to assess whether the predicted answer is semantically equivalent to the ground truth. The paper also reports average scores across all 14 benchmarks as an aggregate performance measure. No confidence intervals or statistical significance tests are reported for any result.
Baselines. The experiments compare RIL against multiple categories of baselines:
- RL-only methods: GRPO (Shao et al., 2024) and Dr.GRPO (Liu et al., 2025) trained with only answer rewards from LLM-as-a-Judge, without the discriminator or similarity rewards (Table 1). This isolates the contribution of the imitation learning component.
- Single-teacher RIL: RIL trained with only one teacher VLM (either Qwen2.5-VL-72B or InternVL3-78B alone) rather than both simultaneously (Table 2), isolating the contribution of multi-teacher diversity.
- Open-source VLMs across scales: a comprehensive set of recent VLMs at comparable parameter counts, including Qwen2-VL-7B, InternVL2-8B, InternVL2.5-8B, LLaVA-OneVision-7B, MiniCPM-V2.5/2.6/o2.6-8B, Ovis2-8B/4B/2B/1B, Phi-3.5-Vision-4B, Phi-4-Multimodal-5.6B, and smaller variants (Tables 3 and 4).
- Large open-source VLMs: Qwen2.5-VL-72B, InternVL3-78B, NVLM-72B, Molmo-72B, LLaVA-OneVision-72B (Table 4).
- Closed-source VLMs: GPT-4o, GPT-4V, Gemini-1.5-Pro, Claude-3.5 Sonnet (Table 4).
- Distillation baselines: MiniLLM, DistilLLM, LLaVA-KD, and VLsI applied to the same Qwen2-VL backbone (Table 5f), used for the synergy analysis rather than as performance baselines per se.
- Answer parsing baseline: replacing LLM-as-a-Judge with conventional answer parsing for reward computation (Table 5e), isolating the contribution of flexible LLM-based evaluation.
Generation budget / compute accounting. During RIL training, the generation budget per question is $G = 4$ responses from the student and $G = 4$ cached responses from the teacher(s) per RIL iteration (total $2G = 8$ responses in the optimization batch). The pre-training phase generates $N = 16$ responses per question from both student and teacher for discriminator training (1.2M total samples from 40K questions). Teacher responses are pre-generated and cached, so teacher inference cost is not incurred during the RIL loop. For evaluation, models use their default generation hyperparameters with a single response per question (greedy or default temperature decoding, not specified explicitly). The paper reports total training compute in GPU-days: discriminator pre-training takes 1–3 days on 256 A100 GPUs, SFT takes 3–5 days on 8 A100 GPUs (the 4M-sample SFT dataset), the RIL loop takes 3–5 days on 8 A100 GPUs. No FLOPs-matched comparisons are performed — the paper evaluates only the resulting model quality, not the training efficiency relative to alternatives.
Cross-validation / statistical protocol. There is no cross-validation, no repeated runs with different random seeds, and no statistical significance testing reported anywhere in the paper. All results are single-run evaluations on the standard test splits of each benchmark. The RIL training uses a fixed 40K-sample subset of the 4M SFT dataset, curated via log-probability sampling and overlong filtering, but the specific 40K subset is not described as being validated across multiple splits. This means the reported performance figures should be interpreted as point estimates without quantified uncertainty.

Main Quantitative Results

RIL Outperforms RL-Only Training with Answer Rewards (Table 1, Figure 1, Figure 2)

Table 1 is the primary head-to-head comparison between RIL and RL-only baselines. Using Qwen2.5-VL-7B as the student and Qwen2.5-VL-72B as the single teacher:

Base model (Qwen2.5-VL-7B, before any RL): accuracy ranges from 38.3% (MMMU-Pro) to 87.3% (ChartQA), with an average of 68.5% across the 14 benchmarks reported in Table 1.
GRPO with answer reward only: 66.7% average — actually below the base model, indicating that naive RL with only answer rewards can degrade performance, likely because the optimization drifts the model away from its SFT capabilities without sufficient guidance.
Dr.GRPO with answer reward only: 68.9% average — a marginal improvement over the base model (+0.4 percentage points), showing that the unbiased advantage estimates from Dr.GRPO help stabilize training but provide limited gains without the imitation component.
RIL with single teacher (Qwen2.5-VL-72B): 70.7% average — a clear improvement over both base (68.5%) and Dr.GRPO (68.9%), driven by large gains on specific benchmarks like MathVista (67.8% → 69.4%), MMMU (55.0% → 56.5%), and ChartQA (87.3% → 93.6%).
RIL with both teachers (Qwen2.5-VL-72B + InternVL3-78B): the version labeled "RIL (Both)" achieves 71.1% average on Qwen2.5-VL-7B and 74.8% on InternVL3-8B. The key benchmark-level gains for Qwen2.5-VL-7B are MathVista (67.8% → 79.7%, +11.9 points), ChartQA (87.3% → 95.6%, +8.3 points), BLINK (56.4% → 70.0%, +13.6 points), and MMB (83.5% → 86.3%, +2.8 points).

These numbers appear to correspond to the "RIL (Both)" rows in Table 3, though Table 1 and Table 3 report overlapping but not identical benchmark sets — Table 1 includes MM-Vet-v2 and MMB CN while Table 3 includes RWQA.

For InternVL3-8B, the gains are similarly substantial: the base model averages approximately 70.8% before RIL and reaches 74.8% after RIL with both teachers (Table 3, "InternVL3-RIL-8B (Both)" row), with particularly large jumps on ChartQA (86.6% → 95.3%), MathVista (71.6% → 77.8%), and MMB (83.4% → 88.1%).

The visual presentation in Figure 1 reinforces these results: Qwen2.5-VL-7B shows a 2.6 percentage point improvement over the base model for RIL (Both) versus a 0.4 point improvement for Dr.GRPO alone, and Figure 2 positions both RIL-trained 7B/8B models in the competitive landscape, showing them approaching or matching much larger open-source VLMs and closing the gap to closed-source models.

Multiple Teachers Consistently Outperform Single Teachers (Table 2)

Table 2 directly compares RIL training with single teachers versus both teachers combined, using Qwen2.5-VL-7B as the student. The results are consistent across all 14 benchmarks:

Single teacher (Qwen2.5-VL-72B only): average accuracy is approximately 70.7%.
Single teacher (InternVL3-78B only): average accuracy is approximately 70.1%.
Both teachers together: average accuracy is approximately 71.1%, outperforming either single teacher. The gains are not uniform — some benchmarks benefit more from the diversity (MathVista sees larger gains with both teachers than either alone), while others are relatively insensitive — but the trend is monotonically in favor of multiple teachers across every individual benchmark.

This demonstrates that teacher diversity is an independent axis of training signal quality, not merely a matter of selecting the single strongest teacher. The mechanism is attributed to richer response diversity strengthening the discriminator and providing more varied exemplars in the GRPO optimization.

RIL Narrows the Gap to Large Open-Source and Closed-Source VLMs (Tables 3 and 4, Figures 1 and 2)

Tables 3 and 4 present the comprehensive comparison of RIL-trained models against a wide range of existing VLMs at various scales.

At the 7B–8B scale (Table 3, top section):

Qwen2.5-VL-RIL-7B (Both) achieves an average of 72.8% across the 14 benchmarks versus the base Qwen2.5-VL-7B's 68.5%. This places it ahead of all other 7–8B open-source models listed, including InternVL2-8B (64.2%), InternVL2.5-8B (70.1%), LLaVA-OneVision-7B (69.9%), MiniCPM-V2.5-8B (63.5%), MiniCPM-V2.6-8B (65.0%), and Ovis2-8B (72.5%).
InternVL3-RIL-8B (Both) achieves an average of 73.7%, surpassing its base model (70.8%) and all other 7–8B competitors, and nearly matching Ovis2-16B (74.1%), a 16B model.
Specific standout benchmark results: Qwen2.5-VL-RIL-7B achieves 95.6% on ChartQA (vs. 87.3% base, an 8.3-point gain), 79.7% on MathVista (vs. 67.8%, +11.9 points), and 70.0% on BLINK (vs. 56.4%, +13.6 points). InternVL3-RIL-8B achieves 95.3% on ChartQA (vs. 86.6% base), 77.8% on MathVista (vs. 71.6%), and 80.1% on MM-Vet (vs. 78.5%).

At smaller scales (Table 3, bottom sections):

Qwen2.5-VL-RIL-3B achieves 73.2% average, a substantial jump from the base model's 65.4% (+7.8 points), and notably outperforms several 7–8B models including the base Qwen2.5-VL-7B (68.5%) and InternVL3-8B (70.8%) — this is a striking result because it means a 3B model trained with RIL can match or exceed the performance of much larger base models.
InternVL3-RIL-2B achieves 69.3% average (vs. 64.3% base).
InternVL3-RIL-1B achieves 63.7% average (vs. 58.2% base) — even a 1B model shows non-trivial gains from RIL.

Comparison with large VLMs (Table 4): The RIL-trained 7B/8B models are compared against 72B–78B open-source VLMs and closed-source models. Qwen2.5-VL-RIL-7B's 72.8% average approaches Qwen2.5-VL-72B's 75.1% and InternVL3-78B's 76.8%, substantially narrowing the ~7-8 point gap between the base 7B model and its 72B counterpart. On specific benchmarks, the RIL-trained models occasionally exceed their teachers: for example, Qwen2.5-VL-RIL-7B achieves 79.7% on MathVista versus the teacher's 74.8%, and 70.0% on BLINK versus the teacher's 61.7% (the paper claims "several instances" of surpassing teacher performance). Against closed-source models, the gap narrows but does not close: GPT-4o achieves approximately 82.8% average, Claude-3.5 Sonnet approximately 79.3%, and Gemini-1.5-Pro approximately 80.3% — the RIL models remain 5–10 points behind these frontier systems.

Figure 2 visualizes this competitive positioning, showing the RIL-trained models' average performance across the 14 benchmarks relative to the broader landscape.

Training Dynamics Show Simultaneous Improvement in Style and Correctness (Figure 5)

Figure 5 plots three metrics over the course of RIL training for both Qwen2.5-VL-7B and InternVL3-8B:

Similarity reward (left panel): steadily increases from roughly 0.5–0.6 at the start of RIL training to roughly 0.9 by the end, indicating that an increasing fraction of student responses are classified as teacher-like by the discriminator (scores below the 0.5 threshold).
Answer reward / accuracy reward (middle panel): also steadily increases over training, from roughly 0.65 to roughly 0.75, indicating that the student is simultaneously generating more factually correct responses.
Average benchmark performance (right panel): increases from roughly 68–69% to 71–72% for Qwen2.5-VL-7B and from roughly 72% to 74–75% for InternVL3-8B, mirroring the reward trends.

The key takeaway from these dynamics is that the two reward signals are not in tension — the student improves on both dimensions simultaneously, and these improvements translate into benchmark performance gains. There is no evidence of a tradeoff where becoming more teacher-like in style comes at the cost of correctness, or vice versa.

Ablation Studies and Robustness Checks

Discriminator and student update iterations ( $\mu$ ) (Table 5a): Setting $\mu = 1$ (one update for both discriminator and student per RIL cycle) is sufficient for strong performance. Increasing $\mu$ for the discriminator provides marginal additional benefit, but increasing it for the student ( $\mu = 5$ ) actually degrades performance, suggesting the student is prone to overfitting when updated too aggressively within each cycle. All subsequent experiments use $\mu = 1$ for both components.

KL-divergence penalty coefficient ( $\beta$ ) (Table 5b): Sweeping $\beta \in \{0.0, 0.01, 0.04, 0.1\}$ on Qwen2.5-VL-7B shows that $\beta = 0.04$ yields the best average accuracy (71.1%), with both lower and higher values degrading performance. $\beta = 0.0$ (no KL penalty) drops to roughly 69%, confirming that constraining policy drift from the SFT checkpoint is critical. $\beta = 0.1$ (overly restrictive) also degrades performance (roughly 70%), confirming that too much regularization prevents the student from learning. The paper interprets this as evidence that "while constraining policy updates is crucial for stability, an overly restrictive penalty can hinder performance."

Parameter groups updated in student VLM (Table 5c): Training self-attention layers, word embeddings, and the language head yields the best performance. Specifically, updating only attention layers produces an average of approximately 70.0%; adding word embeddings and the language head brings it to 71.1%. Including feed-forward network (FFN) and layer normalization parameters in the update reduces performance slightly (approximately 70.5%), suggesting these components are better kept frozen during RIL. The interpretation is that attention and output-projection layers are most directly involved in shaping response patterns, while FFN and normalization parameters are more stable and do not need adjustment.

Continuous vs. fixed discriminator (Table 5d): Three configurations are compared:

No discriminator at all (RL-only, marked ✗): baseline performance, approximately 68.9% (Dr.GRPO only).
Fixed discriminator (pre-trained, not updated during RIL loop, marked $\triangle$ ): improves over RL-only to approximately 70.0%, confirming that even a static similarity reward provides useful guidance.
Continuously updated discriminator (marked ✓): achieves 71.1%, substantially outperforming the fixed discriminator. This validates that co-evolving the discriminator with the student is critical for full RIL effectiveness.

LLM-as-a-Judge vs. answer parsing (Table 5e): Replacing LLM-as-a-Judge with conventional answer parsing ( $\triangle$ ) significantly degrades performance, dropping average accuracy from 71.1% to approximately 69.5%. The paper attributes this to answer parsing's inability to handle open-ended visual questions where answers can be phrased in multiple valid ways — e.g., "The answer is twenty percent" vs. a ground truth of "20%" — and its complete failure on tasks without well-defined answer formats. This is presented as validation that LLM-as-a-Judge's flexible semantic evaluation is essential for RIL to work across diverse vision-language tasks.

Importance of SFT warm-up (Table 5e): The SFT stage before RIL is shown to be important: applying RIL directly to the pre-trained student without SFT (denoted ✗ for SFT) yields lower performance than SFT+RIL. However, the gain from RIL on top of SFT is substantially larger than the gain from SFT alone, confirming that RIL provides benefits beyond what the SFT warm-up captures.

Binary vs. continuous vs. multi-level similarity reward (Figure 6): This ablation tests different discretization schemes for the similarity reward on four benchmarks (MathVista, MMB, MM-Vet, MMMU) with both Qwen2.5-VL-7B and InternVL3-8B. Binary rewards (threshold at 0.5) consistently outperform continuous rewards (raw discriminator scores) and multi-level discretizations (3, 5, or 10 levels). For example, on MathVista with Qwen2.5-VL-7B, binary achieves approximately 79.7% versus continuous at approximately 77.5%, with similar gaps on other benchmarks. The paper's interpretation is that "binary feedback provides clearer and more reliable learning signals" because models struggle to interpret why a score of 0.21 should receive a different update than 0.20.

Number of teacher responses per question (Figure 7): Varying the number of teacher-generated responses used during RIL training shows monotonically improving student performance from 1 to 16 responses per question on Qwen2.5-VL-7B. The average accuracy across four benchmarks (MathVista, MMB, MM-Vet, MMMU) increases from roughly 69% with 1 teacher response to roughly 71.5% with 16 responses, with the curve showing diminishing returns after approximately 8 responses. This validates that "richer supervision from diverse teacher outputs enhances the student VLM's overall generalization ability."

RIL applied to distilled student models (Table 5f): This is one of the most striking ablation results. When RIL is applied to Qwen2-VL-7B students that have already undergone knowledge distillation (MiniLLM, DistilLLM, LLaVA-KD, VLsI), the performance gains are consistently larger than when applied to non-distilled students. For example, VLsI + RIL achieves approximately 72.5% average versus VLsI alone at approximately 70.5% and RIL alone (no prior distillation) at approximately 71.1%. This synergy is attributed to feature-level alignment from distillation creating internal representations that are already teacher-like, making RIL's surface-level text imitation more effective.

Critical Assessment

Claim: RIL Significantly Outperforms RL-Only Baselines

The evidence in Table 1 supports this claim, but the magnitude is more modest than the abstract's framing ("significant performance gains") suggests. RIL with both teachers (71.1%) outperforms Dr.GRPO with answer rewards only (68.9%) by 2.2 percentage points on average across the benchmarks reported in Table 1. This is a real improvement, and the gains on specific benchmarks are larger (MathVista: +11.9 points, ChartQA: +8.3 points), but the average gain is concentrated in a subset of benchmarks while others show minimal movement (MMB: +2.8 points, SEED: +3.5 points). The "significant" qualifier is justified for the domains where RIL excels but overstated as a blanket characterization. Moreover, a notable finding the paper does not emphasize is that Gr.GRPO alone (68.9%) barely improves over the base model (68.5%), and Gr.GRPO actually underperforms the base model (66.7% vs. 68.5%). This suggests that the answer reward alone, even with unbiased advantage estimates, provides a very weak training signal for VLM improvement — which in turn makes the imitation component's contribution (lifting from 68.9% to 71.1%) more meaningful in context.

Claim: Multiple Teachers Consistently Outperform Single Teachers

Table 2 strongly supports this claim — the trend holds across every benchmark without exception. The average gain from using both teachers versus the best single teacher is approximately 0.4–1.0 percentage points, which is modest but consistent. The mechanism (diversity strengthens the discriminator and provides richer exemplars) is plausible and aligns with the training-dynamics data. However, only one combination of teachers is tested (Qwen2.5-VL-72B + InternVL3-78B). It is unknown whether adding a third teacher (e.g., LLaVA-OneVision-72B) would provide further gains, or whether the benefit saturates at two. It is also unknown whether the specific combination of Qwen and InternVL teachers matters — would combining two teachers from the same model family (e.g., Qwen2.5-VL-72B and Qwen2-VL-72B) produce similar gains, or is cross-family diversity the active ingredient? These ablation gaps mean the claim is supported for the tested configuration but the generality of "multiple teachers help" is not established.

Claim: RIL-Trained Models Narrow the Gap to Large Open-Source and Closed-Source VLMs

This claim is supported by Tables 3 and 4, but with important nuance. RIL-trained 7B/8B models do substantially outperform their base versions and surpass most same-scale competitors. The claim that they "narrow the gap" to large models is true — Qwen2.5-VL-RIL-7B (72.8%) is closer to Qwen2.5-VL-72B (75.1%) than the base 7B model (68.5%) is, closing roughly 40% of the gap. The claim that they "in several instances, surpass them" is also true but selective: on MathVista, the RIL 7B model (79.7%) beats the 72B teacher (74.8%), but on most benchmarks the teacher maintains a clear lead. Against closed-source models, the gap remains substantial — GPT-4o (82.8%) leads RIL-7B by ~10 points on average — so "competitive with" is a reasonable characterization only if understood as "competitive for a 7B model," not "competitive in absolute terms."

The broader concern is that all comparisons are on evaluation benchmarks only, with no assessment of qualitative response differences. The discriminator's similarity reward ensures the student produces teacher-style responses, but Tables 3 and 4 only report accuracy — they do not evaluate whether RIL-trained models actually produce more detailed, better-structured, or more helpful responses in the way the imitation objective intends. The fact that benchmark accuracy improves does not directly validate that the imitation signal is transferring the right aspects of teacher behavior, as opposed to surface-level patterns that happen to correlate with benchmark correctness.

What's Missing From the Experimental Design

Single-run evaluations without confidence intervals. Every number in every table is a single evaluation with no error bars, no multiple seeds, and no statistical testing. Given that benchmark performance differences of 1–2 percentage points are treated as meaningful (e.g., in the hyperparameter ablations in Table 5), the absence of any variance estimate makes it impossible to distinguish real improvements from noise. This is particularly problematic for the ablation studies where the differences between configurations are small (e.g., Table 5c: 70.0% vs. 70.5% vs. 71.1%).

No comparison to alternative efficiency methods. RIL is positioned as an alternative to think-answer RL methods (Vision-R1, LMM-R1, etc.), but no direct comparison is provided. The paper argues RIL is superior because it avoids inference-time latency from think chains, but this advantage is never quantified — there is no latency measurement, no comparison of inference FLOPs between RIL-trained models and think-answer models at comparable accuracy, and no experiment showing that a think-answer approach applied to the same base model would or would not achieve similar accuracy gains.

Limited teacher diversity ablation. As noted above, only one multi-teacher configuration is tested. The paper's headline finding about multiple teachers would be strengthened by ablating across different numbers of teachers (2, 3, 4), different teacher combinations (same-family vs. cross-family), and teachers of varying quality (to test whether diversity or absolute quality matters more).

No training efficiency comparison. The paper reports training time (1–3 days for discriminator pre-training + 3–5 days for SFT + 3–5 days for RIL = 7–13 days total on 256 + 8 A100 GPUs), but provides no comparison to the training cost of alternative approaches. How does the total compute compare to training an think-answer RL model like Vision-R1 from the same base checkpoint? How does it compare to simply doing more SFT on a larger dataset? Without this, the claim that RIL is an "efficient training algorithm" is supported only relative to the goal (small model training) rather than relative to competing methods.

No evaluation of generation quality beyond accuracy. The entire motivation for the discriminator-based imitation component is to transfer response quality, not just correctness. But the evaluation uses only benchmark accuracy, which captures correctness but not whether responses are better-explained, more detailed, or more natural. A human evaluation or LLM-based quality assessment of generated responses (analogous to LMSys-style side-by-side comparisons) would directly test whether the imitation component achieves its stated goal.

Test set overlap with training data. The SFT dataset (Appendix C) draws from dozens of sources, including several that are also used as evaluation benchmarks (AI2D, ChartQA, MathVista, ScienceQA). The paper does not discuss whether the SFT or RIL training data includes examples that overlap with the evaluation test sets, which is a potential contamination concern. Standard practice would be to filter out any training examples that appear in evaluation splits, but this is not mentioned.

Conditions and Boundaries

The effectiveness of RIL appears to depend on several conditions that the experiments do not fully map:

Teacher-student quality gap: RIL uses 72B teachers for 7B students (a ~10× scale difference). It is unclear whether the approach works with a smaller gap (e.g., 13B teacher for 7B student) or a larger gap (e.g., 405B teacher for 7B student). The discriminator's ability to distinguish outputs presumably depends on the gap size — too small a gap and there is no signal, too large and the discriminator task is too easy (trivial separation, providing no nuanced gradient).
Student model family: RIL is tested on Qwen and InternVL families. It would likely work on other VLM architectures (the paper claims architectural agnosticism), but this claim is not tested. The discriminator being initialized from the student's architecture means the approach is defined for any student, but whether the training dynamics transfer across fundamentally different architectures (e.g., a LLaVA-style student with a different vision-language connector design) is unknown.
Benchmark domain: The largest gains are on ChartQA, MathVista, and BLINK — tasks with relatively structured answer formats and clear correctness criteria. Gains on open-ended benchmarks (MM-Vet, SEED) are more modest. This may reflect the answer reward being most informative for questions with well-defined correctness, and the similarity reward providing less benefit where multiple very different phrasings are all equally valid.
Performance ceiling: The training dynamics (Figure 5) show similarity and answer rewards increasing but the benchmark accuracy curve appears to be flattening toward the end of training. It is unclear whether longer RIL training would yield further gains or whether the approach saturates. The paper does not report the number of training iterations or epochs, making it difficult to assess whether the reported results represent convergence or an arbitrary stopping point.

6. Limitations and Trade-offs

6.1 Difficulty Estimation Cost Is Unmeasured and Potentially Dominates Training Budget

The assumption or constraint. The discriminator that enables RIL's imitation learning must be pre-trained to distinguish student from teacher responses before the RIL loop can begin (Section 3.2). This requires generating $N = 16$ responses per question from both the student and teacher VLMs across 40,000 questions — a total of 1.2 million samples. The paper reports this pre-training takes 1–3 days on 256 NVIDIA A100 GPUs (Section 4.1 final paragraph), and the full pipeline (SFT + discriminator pre-training + RIL loop) totals 7–13 days on a mix of 256 and 8 A100 GPUs. The paper acknowledges that "RIL requires increased computational costs during training due to teacher, student, discriminator, and LLM-as-a-Judge" compared to GRPO (Appendix C), but frames this as mitigated by weight offloading between CPU and GPU "using the same model architecture for both the discriminator and student."

The consequence. The practical question for a practitioner is: how much more expensive is RIL than the baseline it claims to improve upon (Dr.GRPO with answer rewards only), and does the performance gain justify the cost? The paper provides no answer. No training FLOPs comparison, no wall-clock time comparison, and no cost-efficiency metric (e.g., benchmark accuracy per GPU-hour) is reported for RIL versus any baseline. The 256×GPU configuration for discriminator pre-training is substantial — it represents a training-time cost that a practitioner would need to weigh against simply training a different model or using a different approach entirely. The paper's headline efficiency claim — that RIL-trained models maintain original inference speed — addresses deployment efficiency but says nothing about training efficiency. For an organization deciding whether to adopt RIL, the absence of any cost-efficiency analysis means the core resource tradeoff (how much training compute for how much accuracy gain?) is unquantified.

What evidence exists in the paper. The paper reports training times for each phase (Section 4.1, Appendix C) but provides no baseline comparison — no GRPO-only training time, no Dr.GRPO-only training time, and no cost-per-accuracy-gain analysis. Table 1 shows that Dr.GRPO with answer rewards only achieves 68.9% average accuracy versus RIL at 71.1% (+2.2 points). Whether this gain justifies the discriminator pre-training cost (1–3 days on 256 GPUs) and the additional discriminator updates during the RIL loop is not addressed.

Mitigation status. The paper partially acknowledges the cost concern in Appendix C, noting that "discriminator and student costs can be at least mitigated" through CPU-GPU weight offloading since both share the same architecture. This reduces memory pressure but does not reduce the total FLOPs required — the discriminator must still be trained. No comparison to baseline training costs is proposed as future work.

6.2 Hard Problems Show Diminishing Returns and No Systematic Difficulty Analysis

The assumption or constraint. The paper's experimental results cover 14 benchmarks spanning diverse vision-language tasks, but the results are reported as aggregate accuracy per benchmark. There is no analysis of which types of questions within each benchmark benefit from RIL and which do not. The gains are not uniform: Qwen2.5-VL-RIL-7B improves by 11.9 points on MathVista (67.8% → 79.7%) and 13.6 points on BLINK (56.4% → 70.0%), but only 1.5 points on MM-Vet (71.8% → ~73.3%, reading from Table 1 and converting the 80.4 in Table 3 which appears to be MM-Vet-v2) and shows regression on some tasks — the base model's 87.3% on ChartQA goes to 95.6%, but MMMU-Pro moves from 38.3% to 48.5% (+10.2 points) while SEED2+ moves from 70.4% to 72.8% (+2.4 points). The paper does not stratify results by question difficulty, domain, or required reasoning depth.

The consequence. Without difficulty-level analysis, a practitioner cannot predict where RIL's gains will materialize. Is RIL improving the student on questions it was already getting right (confidence calibration, better formatting), on borderline questions (pushing near-correct answers to correct), or on genuinely hard questions outside the student's initial capabilities? The large gains on MathVista and ChartQA — benchmarks where answer formats are relatively structured — combined with more modest gains on open-ended benchmarks (SEED, MM-Vet) suggest RIL may primarily improve the student's ability to articulate answers it already knows how to solve conceptually, rather than teaching it to solve fundamentally new types of problems. If this is the case, RIL's value is bounded by the base model's core reasoning capability — the same constraint that limits test-time compute scaling in reasoning models. But the paper provides no analysis to confirm or refute this interpretation.

What evidence exists in the paper. The only indirect evidence comes from comparing benchmark-level gains. The largest absolute improvements are on tasks with structured outputs (ChartQA: +8.3 points, MathVista: +11.9 points, BLINK: +13.6 points) where the discriminator's similarity reward can guide the student toward specific formatting and reasoning patterns. Open-ended tasks (MM-Vet, SEED) show smaller gains. But this is post-hoc pattern-spotting; the paper does not analyze question-level difficulty, per-question pass@1 of the base model, or any difficulty stratification.

Mitigation status. Not addressed. The paper does not mention difficulty analysis or question-level stratification as future work.

6.3 No Comparison to Alternative Efficiency Methods and the Think-Answer Baseline It Critiques

The assumption or constraint. The paper's introduction and related work sections (Sections 1 and 2) explicitly critique think-answer RL methods (DeepSeek-R1, Vision-R1, LMM-R1, VLM-R1) for increasing inference latency and memory requirements through verbose reasoning chains. RIL is positioned as superior because it achieves performance gains "while preserving the original inference speed by avoiding lengthy intermediate reasoning steps" and "trained models do not require an explicit 'think' phase before generating answers." This is a central claim — that RIL offers comparable or better accuracy gains than think-answer methods but without the latency cost. Yet the paper provides no direct comparison to any think-answer VLM, whether in terms of accuracy, inference latency, or training cost.

The consequence. A practitioner choosing between RIL and a think-answer approach (e.g., applying Vision-R1 to the same Qwen2.5-VL-7B base model) has no data to inform the decision. The paper's critique of think-answer methods is qualitative ("verbose 'think' responses... can significantly increase inference latency"), but the magnitude of the latency difference is never quantified. How much slower is a think-answer VLM at comparable accuracy? Conversely, does a think-answer approach achieve higher accuracy than RIL at the cost of that latency? Without a head-to-head comparison on the same base models using the same training data, the paper's claim of superiority on the efficiency dimension is asserted rather than demonstrated.

What evidence exists in the paper. None. No think-answer VLM is evaluated, no latency measurements are reported for any model (RIL-trained or otherwise), and no FLOPs-per-inference analysis is provided. The evaluation in Tables 3 and 4 compares RIL models against standard open-source VLMs at inference time, but these are all single-pass generation models — none use think-answer chains. The think-answer methods critiqued in the introduction (LMM-R1, Vision-R1, R1-V, etc.) do not appear in any comparison table.

Mitigation status. Not addressed. The paper acknowledges in the Discussion and Limitation section (Section 4.3) that "RIL's current implementation has been focused primarily on the post-instruction tuning alignment phase" and suggests extending the discriminator to earlier training stages as future work, but does not mention the missing think-answer comparison.

6.4 Single-Run Evaluations With No Confidence Intervals and Modest Aggregate Gains

The assumption or constraint. Every quantitative result in the paper — all benchmark accuracies, all ablation study numbers, all training dynamics curves — comes from a single training run evaluated once on the standard test split of each benchmark. There are no error bars, no multiple random seeds, no statistical significance tests, and no cross-validation (beyond the standard benchmark splits, which are fixed). The paper's reproducibility checklist explicitly answers "N/A" to the question of whether error bars or statistical significance information is reported, stating simply "N/A" with no justification.

The consequence. Many of the comparisons that support the paper's claims involve differences of 1–3 percentage points in aggregate accuracy. For example: RIL with both teachers (71.1%) versus Dr.GRPO alone (68.9%) is a 2.2-point difference (Table 1); the $\beta = 0.04$ versus $\beta = 0.01$ ablation is roughly a 1-point difference (Table 5b); the continuously updated discriminator versus fixed discriminator is roughly a 1-point difference (Table 5d). Without any variance estimate, a practitioner cannot determine whether these differences are statistically reliable or within the range of run-to-run variability. This is particularly acute for the ablation studies, where the conclusions about optimal hyperparameters ( $\mu = 1$ , $\beta = 0.04$ , which parameter groups to update) are based on single-run comparisons with differences that could plausibly be noise. The test sets of these benchmarks typically contain hundreds to a few thousand questions — a 1-point swing on a 500-question test set represents only 5 questions, which could easily vary between training runs due to random seed effects in data ordering or generation sampling.

What evidence exists in the paper. The checklist explicitly acknowledges no error bars are reported. The training dynamics curves in Figure 5 show steady improvement but are single-run curves — there is no shaded region indicating variance across multiple runs or across batches within a single run. The performance tables report numbers to one decimal place, implying precision that the experimental design does not support.

Mitigation status. Not addressed. The paper's reproducibility checklist marks "N/A" for statistical significance with the justification that there are "no theoretical formulations that need to be proved," which does not address the empirical need for variance estimates. No mention is made of this as a limitation in Section 4.3.

6.5 Architectural Agnosticism Is Claimed but Not Tested Across Diverse VLM Families

The assumption or constraint. One of RIL's three core contributions is described as "Broad Applicability and Flexibility: RIL exhibits wide applicability, functioning effectively with diverse VLMs irrespective of their underlying image embedding strategies or language tokenizers" (Section 1). The paper argues this is a key advantage over feature-level distillation, which requires compatible architectures. The discriminator operates on generated text, so "it is agnostic to the specific image embedding strategies or language tokenizers used by the student or teacher VLMs, ensuring broad compatibility" (Section 1).

The consequence. This claimed architectural agnosticism is only tested on two VLM families — Qwen2.5-VL and InternVL3 — and only in a restricted cross-family configuration: the teachers are Qwen2.5-VL-72B and InternVL3-78B. The student is always from one of these two families. Missing are tests with fundamentally different architectures: a LLaVA-style student (with a simple MLP connector rather than Qwen's or InternVL's more complex vision-language fusion), a student using a completely different vision encoder (e.g., SigLIP rather than the CLIP-style or InternViT encoders used by Qwen and InternVL), or a student from a family not represented in the teacher set. The paper also does not test whether a teacher from one family can train a student from a third family not in the teacher set — e.g., Qwen2.5-VL-72B as the sole teacher for a LLaVA-OneVision-7B student. The claim of broad applicability is extrapolated from the architectural principle (text-based operation) rather than empirically validated across a diverse range of architectures.

What evidence exists in the paper. RIL is evaluated on five student scales within two model families: Qwen2.5-VL (3B, 7B) and InternVL3 (1B, 2B, 8B). Both families use relatively similar high-level architectures (vision encoder → adapter → LLM backbone), and the cross-family teacher configuration (Table 2) uses both Qwen and InternVL teachers for both Qwen and InternVL students. The RIL results on smaller InternVL3 variants (1B, 2B) suggest the method works across scales within a family, but this does not test the claim of cross-architecture generality.

Mitigation status. The paper does not acknowledge this as a limitation. Section 4.3 argues for architectural agnosticism based on the principle that text-based operation avoids feature-level constraints, but does not propose testing this claim across additional VLM families as future work.

6.6 The Discriminator Operates on a Narrow Definition of "Quality" — Teacher-Likeness — Without Verifying Response Substance

The assumption or constraint. The discriminator's training objective (Equation 1) is to distinguish student-generated text from teacher-generated text. The similarity reward derived from it (binary threshold at 0.5) rewards the student for producing responses that are indistinguishable from teacher outputs from the discriminator's perspective. The paper explicitly notes that "the discriminator alone primarily captures stylistic similarity and does not inherently verify factual correctness" and that the separate answer reward (LLM-as-a-Judge) is included to compensate for this (Section 3.3). However, the composite reward $R = \text{similarity} + \text{correctness}$ places equal weight on both components — a factually correct but non-teacher-like response receives the same reward (1 + 0 = 1) as a teacher-like but incorrect response (0 + 1 = 1).

The consequence. The equal-weight composite reward creates a potential blind spot: the student can achieve high total reward by optimizing the similarity signal even when correctness is low, or vice versa. The training dynamics (Figure 5) show both similarity and answer rewards increasing together, suggesting this is not an issue in practice for the tested configuration. But this co-improvement may be specific to the tested teacher-student pairs — if teacher responses themselves have high factual correctness rates, then becoming more teacher-like and becoming more correct are correlated objectives, and the equal-weight sum does not create tension. If a teacher model were highly articulate but produced factually incorrect answers at a non-trivial rate, the composite reward would reward the student for imitating those incorrect but teacher-like responses. More fundamentally, the discriminator reduces "response quality" to "teacher-likeness" — it cannot distinguish between teacher responses that are genuinely high-quality (detailed, well-reasoned, clear) and teacher responses that happen to be verbose or stylistically distinctive but not substantively better. A student that learns to imitate surface-level patterns (sentence length, common phrases, formatting conventions) without improving substantive reasoning could receive high similarity rewards.

What evidence exists in the paper. The paper's benchmark evaluations measure accuracy, not response quality. There is no analysis of whether RIL-trained models produce responses that are actually better-structured or more informative (beyond correctness), nor any human evaluation or LLM-based quality assessment of the generated text. The steady increase in both similarity and answer rewards (Figure 5) is consistent with correlated improvement but does not rule out surface-level imitation of style without substance — if the answer reward captures correctness independently, the similarity reward's contribution could be purely stylistic without affecting benchmark scores.

Mitigation status. The paper partially addresses this by noting the discriminator's limitation and including the answer reward as a corrective, but this only protects against factual errors, not against hollow stylistic imitation. There is no discussion of evaluating response quality beyond correctness, and no proposal for weighting the two reward components differently or making the similarity reward conditional on correctness. The claim that RIL captures a "deeper 'verbalization effect'" (Section 3.3) — the idea that how proficient VLMs articulate answers carries useful information — is asserted but not directly validated through any quality-of-expression evaluation.

7. Implications and Future Directions

How This Work Changes the Landscape

RIL introduces a methodological shift in how the field approaches VLM compression: it relocates the knowledge transfer interface from internal model states to natural language text strings, using an adversarially trained language-model discriminator rather than a static distance metric as the similarity function. This is not an incremental improvement to distillation — it is a category change in what is transferred (surface generation patterns rather than internal representations) and how similarity is assessed (by a trainable, context-aware language model rather than a fixed mathematical divergence like KL divergence).

The magnitude of this shift is best understood as opening a new axis of the VLM design space rather than solving an existing problem definitively. Prior to RIL, the dominant compression paradigm was feature-level distillation — matching logits, hidden states, or attention patterns between teacher and student. This paradigm works but imposes a hard constraint: the teacher and student must share compatible vision encoders, language backbones, tokenizers, and embedding dimensionalities. RIL demonstrates that this constraint is not inherent to the knowledge transfer problem — it is an artifact of choosing to operate on internal representations. By operating on generated text, RIL makes the teacher and student choices independent design decisions, enabling combinations that are architecturally impossible with feature-level distillation (e.g., a Qwen2.5-VL teacher training an InternVL3 student, as demonstrated in Table 2).

The conceptual contribution is not that text-level supervision works (instruction tuning already shows that), but that adversarially trained text-level supervision can serve as a drop-in replacement for feature-level alignment — learning what makes a response "teacher-like" through a discriminator rather than through KL divergence on logits. This reframes VLM compression from an architecture-constrained optimization problem to a text-supervised learning problem where any model capable of generating text can be the student, and any model that produces high-quality text can be the teacher. The practical significance is immediate: practitioners can now combine the best available teacher VLMs with the most deployment-suitable student architectures regardless of their internal design, something that was impossible under the feature-distillation paradigm.

Reconciling prior contradictions. The paper resolves an implicit tension in the VLM compression literature between two observations: (1) feature-level distillation can improve student performance but requires architectural compatibility, and (2) natural language responses carry useful training signal (the "verbalization effect" from VLsI and Eagle) but prior text-based methods lacked a mechanism for structured imitation. RIL synthesizes these observations: a discriminator provides structured imitation in text space, while a separate answer reward ensures factual correctness, combining the architectural flexibility of text-based transfer with the learning stability of adversarial training. The finding that RIL works especially well on students that have already undergone feature-level distillation (Table 5f) further suggests that these two paradigms are complementary rather than competing — feature alignment creates an internal foundation that text-level imitation then refines at the output surface.

Research directions that become more attractive. RIL makes text-level adversarial imitation a viable approach for VLM training, which opens several natural extensions: applying the same framework to other modalities (audio, video), using the discriminator during pre-training rather than only post-hoc alignment, and exploring different discriminator architectures (larger models, multi-task discriminators that assess specific quality dimensions). It also makes research on discriminator robustness more urgent — the discriminator is the linchpin of the entire approach, and its failure modes (rewarding surface-level patterns without substance, potential for adversarial examples that fool the discriminator) directly determine RIL's ceiling.

Research directions that become less attractive. The paper's strong results on RIL versus RL-only baselines (Table 1) suggest that pure RL with answer rewards is a weak training signal for general VLM improvement. The GRPO baseline actually underperforms the base model (66.7% vs. 68.5%), and even the improved Dr.GRPO barely exceeds it (68.9% vs. 68.5%). This implies that the field's recent enthusiasm for applying DeepSeek-R1-style RL to VLMs may be misallocated unless accompanied by a richer reward structure — answer correctness alone provides too sparse a signal for meaningful policy improvement on diverse vision-language tasks. The think-answer paradigm (Vision-R1, LMM-R1, VLM-R1) can be understood partly as a workaround for this sparsity — forcing the model to externalize reasoning creates intermediate states against which rewards can be defined — but at the cost of inference latency that RIL avoids. RIL's success with a composite reward that separates style from substance suggests that designing richer reward functions may be more fruitful than designing more elaborate generation formats.

Follow-Up Research This Work Enables

Cross-architecture generalization: testing RIL with a LLaVA student and a Qwen teacher. The paper claims architectural agnosticism based on the principle that text-based operation avoids feature-level constraints, but this is only tested within two relatively similar VLM families (Qwen2.5-VL and InternVL3). A direct test of the claim would train a LLaVA-OneVision-7B student (which uses a simple MLP vision-language connector and a different training recipe) using Qwen2.5-VL-72B as the sole teacher — two models that share neither vision encoder architecture, language backbone, nor tokenizer. If RIL succeeds on this pair, the architectural agnosticism claim is validated. If it fails or underperforms, the claim needs to be qualified: text-level transfer may work across families with similar high-level design philosophies but not across fundamentally different VLM paradigms. This experiment also tests whether the discriminator's initialization from the student architecture (necessary for the balance problem mitigation) creates an implicit dependency on architecture-specific inductive biases.

Scaling the teacher-student gap: how large can the quality difference be before the discriminator signal degrades? RIL uses 72B teachers for 7B students (~10× scale difference), and the discriminator learns to distinguish their outputs. The dynamics of this distinction matter: if the gap is too small (e.g., 13B teacher for 7B student), the discriminator may struggle to find reliable differences, providing no useful gradient. If the gap is too large (e.g., a hypothetical 405B teacher for a 1B student), the discriminator task becomes trivially easy — perfect separation with a score of 1.0 for all student outputs and 0.0 for all teacher outputs — which provides no nuanced gradient either (the discriminator is too confident to give meaningful feedback on how student outputs differ from teacher outputs). This suggests an optimal gap size where the discriminator is challenged but not overwhelmed. A systematic experiment varying the teacher size (7B, 13B, 34B, 72B, 405B if available) while keeping the student fixed (7B) and measuring both discriminator accuracy and downstream student performance would map this relationship. The hypothesis is an inverted-U curve: student performance peaks at an intermediate teacher-student gap where the discriminator provides the most informative gradient.

What does the discriminator actually learn? Probing for surface vs. substantive features. The paper's discriminator is trained to output low scores for teacher responses and high scores for student responses, but it is never analyzed to determine what features it uses to make this distinction. Does the discriminator learn to recognize substantive quality differences (reasoning depth, factual precision, logical coherence), or does it primarily latch onto surface-level patterns (sentence length, common phrases, formatting conventions)? A probing experiment would generate responses that are stylistically teacher-like but factually incorrect (e.g., by taking teacher responses and introducing subtle factual errors while preserving style) and measure the discriminator's scores on these. If the discriminator gives low scores (teacher-like) to incorrect-but-stylish responses, it is primarily a surface-level style detector. If it gives intermediate or high scores, it is picking up on some signal of factual reliability. This matters because the similarity reward's value depends on what "similarity" actually means — if it is purely surface-level, the imitation component is a stylistic regularizer that may not generalize to new domains, whereas if it captures deeper quality features, the discriminator is learning something more transferable. The steady co-increase of similarity and answer rewards in Figure 5 is consistent with either interpretation, since teacher responses are both stylistically distinctive and factually correct on average.

Training-time cost-efficiency: FLOPs-matched comparison against think-answer RL methods. The paper's central critique of think-answer RL methods is that they increase inference latency, but no comparison is provided. A direct experiment would take the same Qwen2.5-VL-7B base model, train one variant with RIL (as in the paper) and another with a think-answer RL approach (e.g., applying Vision-R1 or VLM-R1 methodology using the same 40K training questions), and compare both accuracy and inference cost. The inference cost dimension should include both latency (milliseconds per query) and total FLOPs per query (since think-answer methods generate more tokens). The hypothesis from the paper's framing is that think-answer methods might achieve comparable or higher accuracy but at substantially higher inference cost, while RIL achieves its gains at the base model's original inference cost. Even if think-answer methods achieve higher accuracy, quantifying the cost differential would let practitioners make informed tradeoffs. The paper's current silence on this comparison is its most significant evidential gap.

Discriminator-guided data selection: can the discriminator identify high-quality training examples from unlabeled data? The discriminator, once trained, produces a scalar score that estimates how teacher-like any given response is. This score could be used to filter or weight training data — for example, selecting the most teacher-like responses from a pool of student-generated candidates for further training, or identifying which questions the student struggles with most (where student responses are consistently scored as non-teacher-like) for targeted data collection. A concrete experiment: use the trained discriminator to rank a large pool of student-generated responses to novel questions, select the top-K most teacher-like and bottom-K least teacher-like, and measure the actual quality (by LLM-as-a-Judge or human evaluation) of these responses. If the discriminator's rankings correlate well with actual response quality, it can serve as a quality filter for self-training or data augmentation pipelines without requiring ground-truth answers. If the correlation is weak (because the discriminator is picking up on stylistic confounds), this would reveal a fundamental limitation of the adversarial imitation approach.

Multi-dimensional discriminator: decomposing "teacher-likeness" into interpretable quality dimensions. The current discriminator produces a single scalar score that conflates all aspects of response quality — factual accuracy, reasoning structure, level of detail, clarity, formatting, and domain-appropriate tone. A natural extension is to train multiple discriminators, each specialized to a specific quality dimension, and combine their scores into a richer reward signal. For example, one discriminator could be trained specifically on pairs of responses that differ in reasoning depth (but are both factually correct), another on pairs that differ in clarity (but cover the same content), and another on domain-specific conventions (chart-reading answers vs. general VQA answers). The student would then receive separate similarity rewards for reasoning quality, clarity, and domain appropriateness, enabling more targeted improvement. This would also make the training process more interpretable — a practitioner could see which quality dimensions are improving and which are not, rather than observing only a single aggregate similarity curve as in Figure 5.

Practical Applications and Downstream Use Cases

On-device VLM deployment with cloud-teacher supervision. The most direct application of RIL is training small VLMs for smartphones, AR devices, and edge computing where 72B models cannot run. A 3B model trained with RIL achieves 73.2% average accuracy across the 14 benchmarks (Table 3), which places it ahead of the base 7B Qwen2.5-VL model (68.5%) and nearly all other 7–8B open-source models. This means a practitioner deploying to a memory-constrained device (where even a 7B model with 4-bit quantization pushes storage limits) can use a 3B RIL-trained model at approximately half the parameter footprint of a 7B model while achieving better accuracy. The inference cost is unchanged from the base model (no think chains, no additional forward passes), so the deployment benefit is pure: better accuracy at the same latency and memory. The teachers (72B VLMs) run in the cloud once to generate training data, and the resulting small model runs entirely on-device.

Cross-family model compression for organizations with heterogeneous model access. Because RIL operates on text, an organization that has API access to a closed-source teacher VLM (e.g., GPT-4o via API, but not its weights or internal representations) can use that teacher's text outputs to train an open-source student VLM of their choice. This is impossible with feature-level distillation, which requires access to the teacher's internal representations (logits, hidden states). Concretely: a team could query GPT-4o on their domain-specific question set, cache the text responses, and use RIL to train an InternVL3-8B student that imitates GPT-4o's answer style, even though GPT-4o's architecture is unknown and its tokenizer is incompatible with InternVL3. This makes closed-source VLMs viable teachers for the first time, since only their text outputs are needed. The paper's reported gains from multiple teachers (Table 2) suggest that combining outputs from multiple API-accessible models (GPT-4o, Claude, Gemini) could provide even stronger training signal than any single one.

Cost-efficient VLM fine-tuning for domain-specific applications. The RIL training loop uses a relatively small curated dataset (40K questions, downsampled from a 4M SFT dataset via log-probability sampling) and achieves substantial gains over the SFT baseline. For an organization fine-tuning a VLM for a specific domain (medical imaging QA, legal document understanding, industrial inspection), the RIL recipe provides a template: (1) SFT the student on domain data, (2) collect teacher responses on the same domain data (possibly from a larger, more expensive model queried once), (3) pre-train the discriminator and run the RIL loop. The paper reports that the RIL loop takes 3–5 days on 8 A100 GPUs for 40K questions — a cost that is feasible for many applied ML teams. The specific gains on structured-output tasks (ChartQA: +8.3 points, MathVista: +11.9 points) suggest RIL is particularly well-suited for domains where answers follow predictable formats, which describes many industrial applications (form understanding, report generation, standardized QA).

Self-improvement pipelines with iterative teacher-student role swapping. The paper shows that RIL-trained students can occasionally surpass their teachers on specific benchmarks (Qwen2.5-VL-RIL-7B achieves 79.7% on MathVista versus the 72B teacher's 74.8%). This opens the possibility of an iterative bootstrapping process: train a student with RIL using current teachers → deploy the improved student as a new "teacher" for the next generation of even smaller students → repeat. Because the student can exceed the teacher on some dimensions (thanks to the answer reward providing correctness signal independent of teacher behavior), each generation could in principle produce outputs better than the previous generation's teacher, creating a self-improvement loop. The key risk is that discriminators trained on successive generations might overfit to stylistic artifacts of the specific teacher set, requiring careful monitoring of response diversity. The paper's finding that multiple diverse teachers work better than single teachers (Table 2) suggests that maintaining teacher diversity in such a loop would be essential.

When to Prefer This Method

The paper explicitly positions RIL against two categories of alternatives: feature-level knowledge distillation methods (MiniLLM, DistilLLM, LLaVA-KD, VLsI) and think-answer RL methods (Vision-R1, LMM-R1, VLM-R1). The tradeoffs can be characterized from the paper's evidence:

Prefer RIL over feature-level distillation when:

The student and teacher VLMs use different vision encoders, tokenizers, or language backbones — RIL operates on text and imposes no architectural compatibility constraints (Section 4.3).
The teacher is a closed-source model accessible only through API text outputs, with no access to internal logits or hidden states (Section 1, "agnostic to the specific image embedding strategies or language tokenizers").
Response quality (style, clarity, reasoning structure) matters alongside factual correctness — RIL's discriminator provides an explicit imitation signal for these dimensions, while KL divergence on logits captures only distributional similarity (Section 3.3, the "verbalization effect").
The student has already undergone feature-level distillation — RIL shows particularly strong synergy with prior distillation (Table 5f), suggesting a two-stage pipeline where feature alignment creates a foundation and RIL refines output quality.

Prefer RIL over think-answer RL when:

Inference latency is a hard constraint — RIL maintains the base model's original inference speed (no think chains), while think-answer methods generate additional reasoning tokens that increase both latency and memory (Section 1).
The deployment setting involves general visual question answering, not just domains with well-defined answer formats (math, structured reasoning) — RIL's LLM-as-a-Judge enables reward computation on open-ended questions where answer parsing fails (Table 5e shows significant degradation when parsing replaces the judge).
Training compute must be amortized across many queries — RIL's training cost (7–13 days on 256 + 8 A100 GPUs) is incurred once, producing a model that runs at the original inference speed for all subsequent queries, whereas think-answer methods incur the latency penalty on every query.

Prefer think-answer RL or scaling pretraining over RIL when:

Accuracy on hard reasoning problems is the sole objective and inference latency is not constrained — think-answer methods may achieve higher accuracy on math and structured reasoning tasks by externalizing the reasoning process (though the paper provides no direct comparison to substantiate this).
The base model's pass@1 on the target task is near zero — RIL, like test-time compute scaling, amplifies existing capability but does not create it, and the paper provides no difficulty-stratified analysis to determine where gains concentrate. If the student cannot produce any correct answers even with the answer reward signal, the imitation component alone is insufficient (the composite reward relies on the answer reward to prevent hollow stylistic imitation).

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning2025· multimodal← LLM-judged similar
- Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodology2025· multimodal← LLM-judged similar
- G1: Bootstrapping Perception and Reasoning Abilities of Vision-Language Model via Reinforcement Learning2025· training-methods← LLM-judged similar
- MMaDA: Multimodal Large Diffusion Language Models2025· multimodal← in-domain predecessor
- From Narrow to Panoramic Vision: Attention-Guided Cold-Start Reshapes Multimodal Reasoning2026· multimodal← similar topic
- Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory2025· multimodal← similar topic
- InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency2025· multimodal← similar topic
- CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning2025· multimodal← similar topic
- Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey2024· multimodal← similar topic
Efficient Long-context Language Model Training by Core Attention Disaggregation2025· training-methods← in-domain predecessor
- When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training2024· architecture← LLM-judged similar
- BitNet Distillation2025· training-methods← in-domain predecessor
- Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention2025· training-methods← similar topic
- DCP: Addressing Input Dynamism In Long-Context Training via Dynamic Context Parallelism2025· training-methods← similar topic

What it influenced

Later papers in the same chain. Direction reversed.

DaMo: Data Mixing Optimizer in Fine-tuning Multimodal LLMs for Mobile Phone Agents2025· training-methods→ in-domain predecessor
- Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training2026· training-methods→ LLM-judged similar
- E2Rank: Your Text Embedding can Also be an Effective and Efficient Listwise Reranker2025· training-methods→ in-domain predecessor
Rainbow: Combining Improvements in Deep Reinforcement Learning2000· training-methods→ similar topic
- Sequence to Sequence Learning with Neural Networks2000· training-methods→ in-domain predecessor
- Off-Policy Deep Reinforcement Learning without Exploration2018· training-methods→ similar topic
- Robust Preference Learning for Storytelling via Contrastive Reinforcement Learning2022· training-methods→ similar topic
- Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewards2026· training-methods→ similar topic
G1: Bootstrapping Perception and Reasoning Abilities of Vision-Language Model via Reinforcement Learning2025· training-methods→ similar topic
- GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning2025· multimodal→ LLM-judged similar
- Quartet: Native FP4 Training Can Be Optimal for Large Language Models2025· training-methods→ in-domain predecessor
- SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution2025· training-methods→ similar topic
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example2025· training-methods→ similar topic