RL makes MLLMs see better than SFT

🎯 Pitch

Fine-tuning with RL doesn't just improve MLLM outputs—it fundamentally rewires the vision encoder to produce sharper, precisely localized representations that SFT fails to achieve. A mere preference alignment on this 'RL-enhanced' vision backbone creates encoders that beat significantly larger models, all for less than 1% of standard pretraining cost.

1. Executive Summary

This paper analyzes how post-training strategies—specifically Supervised Fine-Tuning (SFT) versus Direct Preference Optimization (DPO), a form of reinforcement learning—differentially reshape both Multimodal Language Models (MLLMs) and their underlying vision encoders across the Cambrian benchmark suite using Qwen2.5 LLMs paired with SigLIP2 vision models. The central finding is that DPO produces stronger and precisely localized visual representations compared to SFT, yielding gains of +2.4 to +4.2 percentage points on strongly vision-centric tasks (OCR & Chart VQA, Vision-Centric VQA) while showing negligible advantage on knowledge-intensive benchmarks. The paper then formalizes this process into a simple recipe called Preference-Instructed Vision OpTimization (PIVOT), demonstrating that a PIVOT-enhanced SigLIP2-So/16 encoder outperforms both its unenhanced counterpart and the substantially larger SigLIP2-g/16 encoder across all VQA categories in downstream MLLM evaluation, establishing that existing vision models possess substantial untapped potential for MLLM-ready representations that can be unlocked through preference alignment rather than scaling model size alone.

2. Context and Motivation

The Core Problem: We Don't Know How Post-Training Reshapes Visual Representations

The fundamental question this paper tackles is one that has been systematically overlooked in multimodal language model research: how do different post-training strategies—SFT versus RL—reshape the vision encoder's internal representations, and do these differences matter for downstream performance? This matters because the field has operated under an implicit, LLM-centric assumption that MLLM capabilities are primarily inherited from the pretrained language model backbone, given its immense parameter scale and remarkable reasoning abilities. This assumption has created what the authors characterize as a "significant void in our understanding of how SFT and RL differ in reshaping visual representations" (Section 1).

This gap is significant for several interconnected reasons that the paper surfaces across Sections 1–3:

The training paradigm is shifting without corresponding analysis. The MLLM community is rapidly adopting RL-based post-training (DPO, PPO, GRPO) following the success of RLHF in pure language models, with at least 13 recent works listed in Table A employing RL for MLLM training. However, this adoption has occurred without any systematic comparison of how SFT and RL differentially affect the model's visual processing pipeline. The field knows RL helps—but not why, how, or where in the architecture the improvements manifest.
The vision encoder is treated as a black box. Prior work on MLLM architecture has explored whether to update the vision encoder during training (Bai et al., 2025; Li et al., 2024) and how many encoders to use (Tong et al., 2024b,a), but has not investigated what actually happens inside the vision encoder under different training objectives. As the authors note in Section 4, "research has progressed little beyond the preliminary finding that fine-tuning the vision encoder yields better outcomes than keeping it frozen."
If RL reshapes vision differently than SFT, this has direct implications for architecture design. Understanding how training strategies affect visual representations would inform decisions about whether to freeze or update vision encoders, which training objectives to use, and whether existing vision pretraining recipes are genuinely optimal for MLLM deployment.

Conflicting Signals in the Literature

The paper is motivated by a tension in existing findings about RL's effects on model representations. On one side, several works demonstrate that applying RL to MLLMs—particularly DPO—improves performance on benchmarks (Yu et al., 2024, 2025; Wang et al., 2024b; Sun et al., 2024a) and reduces hallucination (Yang et al., 2025c; Yu et al., 2024; Fu et al., 2025b). On the other side, these studies attribute improvements almost exclusively to better language generation or alignment of the LLM component, never investigating whether the vision encoder itself undergoes meaningful change.

This oversight is particularly striking because some prior work has provided indirect hints that visual processing changes under different training regimes. Chu et al. (2025) demonstrated that RL-trained models generalize better to new environments than SFT-trained ones when adapting to card games and map navigation, while Shenfeld et al. (2025) showed that RL forgets less than SFT during adaptation. These findings suggest that RL induces fundamentally different kinds of representational change than SFT, but neither study examined the vision encoder specifically. The paper explicitly positions itself as filling this gap: "Unlike RL-vs.-SFT studies, [we] evaluate on common benchmarks rather than specialized settings" (Appendix A.3), and crucially, we analyze "how these distinct trainings impact MLLMs" at the level of visual representations.

Where Existing Approaches Fall Short

The paper identifies specific limitations in prior work along three axes:

Lack of controlled comparison between SFT and DPO. Most prior RL-for-MLLM work compares a pretrained (Stage 1) model against the same model further trained with DPO, which confounds the effect of additional training with the effect of the training objective. As the authors note in Section 3.1: "Prior works like MPO (Wang et al., 2024b) typically focus on comparing a pre-trained model (Stage 1) against the same model further trained with DPO, which does not provide a fair evaluation of DPO versus SFT." Without a controlled comparison using identical data and training budgets, it is impossible to determine whether DPO's reported benefits stem from the preference optimization itself or simply from seeing more training examples.

No analysis of difficulty-dependent effects. Even among studies that claim RL improves MLLMs, there has been no systematic investigation of which types of tasks benefit most. The paper's benchmark suite, adapted from Cambrian (Tong et al., 2024a), deliberately spans 16 tasks across four categories (General VQA, Knowledge VQA, OCR & Chart VQA, Vision-Centric VQA) precisely to disentangle where RL's benefits concentrate. The finding that DPO provides substantial gains on vision-centric tasks but negligible improvement on knowledge tasks (+0.3%p on ScienceQA and MathVista, as reported in Section 3.2) explains why prior work focusing on narrow benchmarks may have reached incomplete or contradictory conclusions.

Vision encoder evaluation restricted to MLLM benchmarks. Perhaps most critically, prior work exclusively evaluates vision encoder quality through MLLM downstream task performance—a metric that conflates visual representation quality with the LLM's reasoning capabilities. The paper introduces a novel evaluation protocol in Section 4 that isolates the vision encoder from the LLM and evaluates it on classic vision tasks (ImageNet classification, semantic segmentation) and representation analysis (gradient visualization, representational alignment). This protocol reveals that DPO-trained encoders outperform SFT-trained encoders by +1.83%p to +1.96%p on ImageNet Top-1 accuracy and by up to +1.08%p on patch-level segmentation recall—effects that would be invisible in standard MLLM evaluations.

The Missing Piece: Gradient-Level Evidence

A particularly important gap the paper identifies is the absence of any mechanistic explanation for how different training objectives might reshape vision. There is no prior work visualizing or analyzing the gradient signals that flow backward from the LLM into the vision encoder during post-training, despite the fact that these gradients are the only mechanism by which the vision encoder can change. The authors hypothesize in Section 4.2 that DPO's advantage stems from its contrastive objective—differentiating between chosen and rejected responses—providing more precise, semantically localized gradient signals compared to SFT's maximum likelihood objective, which simply maximizes the probability of the chosen response. The Grad-CAM visualizations in Figure 7 provide the first empirical evidence for this hypothesis, showing that DPO gradients concentrate on question-relevant image regions while SFT gradients are "scattered" across the image.

How This Paper Positions Itself

The paper frames its contribution not as proposing a new method but as conducting a "timely exploration" (Section 1) and "critical yet under-explored analysis" (Abstract) of an overlooked dimension of MLLM training. This is best understood through three positioning choices:

First, it deliberately focuses on DPO as the representative RL method. Section 2 documents that DPO has become the "predominant choice" for MLLM research (Table A), and the paper follows this trend "for simplicity." This is not a claim that DPO is uniquely effective—Section 6 explicitly suggests future work comparing PPO and GRPO—but rather a recognition that understanding the most commonly used method provides maximum practical value.

Second, it introduces PIVOT as a "simple recipe" rather than a new algorithm. The paper is careful to distinguish PIVOT from CLIP-style contrastive pretraining (Figure 11) and to position it as an under-explored training regime that reveals latent potential in existing vision models. The emphasis is on the finding that "even state-of-the-art encoders have substantial room for MLLM evolution" (Section 5.2), with PIVOT requiring "less than 1% of the computational cost of standard vision pretraining" (Abstract). This is a striking claim given that SigLIP2 was trained on up to 2K TPUv5e chips, while PIVOT uses only 8 H100 GPUs for 18 hours.

Third, it connects to broader questions about pretraining versus post-training compute allocation. By demonstrating that a small amount of carefully targeted post-training can surpass the benefits of scaling model size (SigLIP1-So/14 + PIVOT > SigLIP2-So/16, and SigLIP2-So/16 + PIVOT > SigLIP2-g/16 in Table 1), the paper implicitly argues that the field's focus on ever-larger vision pretraining may be misallocated. The finding that even vision-only self-supervised models like MAE and DINOv2, and a purely supervised ImageNet-trained ViT, all benefit from PIVOT (Table 1, bottom) suggests the improvement is not specific to image-language pretrained backbones—it is a general phenomenon of how preference alignment reshapes visual representations for language-guided tasks.

The paper also explicitly connects to the parallel shift from SFT to RL in pure LLM development (Christiano et al., 2017; Ouyang et al., 2022), but with a crucial difference: while the LLM community widely accepts that RLHF improves language generation quality and alignment, there has been no analogous investigation of whether RLHF improves visual perception quality. This paper's key contribution is establishing that it does—and, critically, that it does so through a different mechanism (localized gradient signals producing fine-grained visual features) than the language-side improvements typically attributed to RLHF.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily an analysis and empirical study rather than a new method, and its core idea is that the choice of MLLM post-training objective—supervised fine-tuning (SFT) versus reinforcement learning via Direct Preference Optimization (DPO)—fundamentally reshapes the vision encoder's internal representations, with DPO producing features that are stronger, more precisely localized, and more useful for downstream MLLM tasks. The system being built is a controlled experimental framework for training MLLMs under identical data conditions but different objectives, then isolating the vision encoder to measure how its representations have changed, allowing the authors to study cause-and-effect relationships that were previously invisible in standard end-to-end MLLM evaluations.

3.2 Big-Picture Architecture (Diagram in Words)

The experimental system has five major components arranged in a linear pipeline:

Base MLLM Architecture — a standard vision-language model composed of a pretrained vision encoder (SigLIP2 in four sizes: B/16, L/16, So/16, g/16), a 2-layer MLP projector, and a Qwen2.5 LLM (four scales: 0.5B, 1.5B, 3B, 7B). This is the "canvas" on which different post-training strategies are applied.
Stage 1 Pre-Training — the standard MLLM initialization procedure: first projector-only alignment on LAION/CC/SBU-558K, then end-to-end training of all parameters on the LLaVA-OneVision 3.2M dataset. This produces a "base MLLM" that can perform visual question answering but has not undergone preference alignment.
Stage 2 Post-Training — the controlled experimental manipulation. Using the identical dataset of 20K preference pairs from the MPO dataset, the base MLLM undergoes either SFT (training only on the chosen response via maximum likelihood) or DPO (training on the contrast between chosen and rejected responses relative to a reference model). Both train all parameters, including the vision encoder.
Vision Encoder Extraction and Probing — after Stage 2, the vision encoder and projector are surgically detached from the LLM. These frozen components are then evaluated on classic vision tasks (ImageNet linear probe classification, ADE20K segmentation probing) to measure representational quality independently of the LLM's reasoning capabilities.
PIVOT Evaluation Pipeline — the extracted PIVOT-enhanced vision encoder is paired with a fresh Qwen2.5-1.5B LLM, undergoes projector-only pretraining on LAION/CC/SBU-558K, then instruction finetuning of projector and LLM on Cambrian's 737K dataset with the vision encoder frozen. This measures how useful the PIVOT representations are in a realistic MLLM deployment.

Information flows as follows: pretrained vision model + pretrained LLM → Stage 1 pre-training on 3.2M samples → base MLLM → Stage 2 post-training with either SFT or DPO on 20K preference pairs → trained MLLM → either (a) evaluate MLLM directly on 16 VQA benchmarks, or (b) extract vision encoder → probe on ImageNet/ADE20K/gradient visualization/representational alignment, or (c) integrate frozen PIVOT encoder into new MLLM → train projector and LLM on Cambrian-737K → evaluate on 16 VQA benchmarks.

3.3 Roadmap for the Deep Dive

First, the controlled training setup (Section 3.1): the two-stage MLLM training pipeline, the model scaling configurations, and the precise data budget at each stage. This establishes the "laboratory conditions" under which all comparisons are made.
Second, the SFT and DPO objectives as implemented for MLLM post-training: the mathematical forms, the key differences between them, and the critical experimental choice to use the same number of training samples for both. This is where we explain what is being compared and why the comparison is fair.
Third, the evaluation protocol for MLLMs: the Cambrian benchmark suite with its 16 tasks across 4 categories, and how this decomposition enables the paper's key finding about difficulty-dependent effects.
Fourth, the vision encoder isolation and probing methodology: the procedures for ImageNet linear probe classification, semantic segmentation probing using ADE20K, gradient visualization via Grad-CAM, and representational alignment measurement. These are the tools that let us "look inside" the vision encoder.
Fifth, the PIVOT recipe: how the Stage 1 + Stage 2 training is repurposed as a standalone vision encoder enhancement procedure, and the three-stage evaluation pipeline that verifies the enhanced encoders work well with a completely new LLM head.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an empirical analysis paper with a practical recipe contribution. The core idea is that DPO and SFT produce qualitatively different visual representations when used for MLLM post-training, and that DPO's representations are strictly better for vision-related tasks because the contrastive preference objective sends more focused, semantically meaningful gradient signals back into the vision encoder compared to the maximum-likelihood SFT objective. The technical machinery serves to (a) establish fair experimental conditions for comparing the two objectives, (b) measure the resulting MLLM performance on a diverse set of tasks, and (c) isolate and characterize the vision encoder's representations independent of the LLM.

Controlled Two-Stage MLLM Training Pipeline

Stage 1: Pre-training. The MLLM is constructed from three components: a SigLIP2 vision encoder (available in four sizes—B/16 at 86M parameters, L/16, So/16 at 400M parameters, and g/16 at 1B parameters—all operating at 384×384 pixel input resolution), a 2-layer MLP projector, and a Qwen2.5-Instruct LLM (available in four scales: 0.5B, 1.5B, 3B, and 7B parameters). The training proceeds in two sub-stages:

First, projector-only pre-training: only the 2-layer MLP projector is trained, while both the vision encoder and LLM are frozen. The dataset is LAION/CC/SBU-558K, a standard multimodal alignment dataset containing 558,000 image-text pairs. This stage establishes initial alignment between the vision encoder's output embedding space and the LLM's input embedding space, preventing the LLM from receiving out-of-distribution visual tokens during subsequent full training.

Second, end-to-end pre-training: all model parameters—vision encoder, projector, and LLM—are trained jointly on the LLaVA-OneVision-3.2M dataset, which contains 3.2 million samples spanning Visual Question Answering, vision-grounded dialogue, and image captioning tasks. This stage uses the standard LLaVA-OneVision finetuning script with a learning rate of $1 \times 10^{-5}$ and a batch size of 256. The output of Stage 1 is a base MLLM that can perform visual question answering but has not been exposed to preference data.

Design choice—why two sub-stages: The projector-only phase prevents the LLM from receiving arbitrarily misaligned visual features during the initial steps of joint training, which could cause catastrophic forgetting in the pretrained language model. This two-phase approach is standard in the MLLM literature (Liu et al., 2023a; Li et al., 2023) and is adopted here to ensure the base model is of reasonable quality before the controlled Stage 2 experiment begins.

Stage 2: Post-training (the controlled comparison). The base MLLM from Stage 1 undergoes a second round of full-parameter training, but now using one of two objectives applied to the same dataset. This is where the paper's central experimental manipulation occurs.

The post-training dataset, denoted $X_{\text{PT}} = \{x_0, x_1, \dots, x_T\}$ , consists of 20,000 instances randomly sampled from the MPO (Mixed Preference Optimization) dataset (Wang et al., 2024b). Each instance $x_i$ is a tuple $\{I_i, q_i, y_i^c, y_i^r\}$ containing an image $I_i$ , a natural language query $q_i$ , a chosen (preferred) response $y_i^c$ , and a rejected (non-preferred) response $y_i^r$ . The chosen and rejected responses were produced by MLLMs and annotated with human preference judgments in the MPO dataset.

Critical experimental control: Both SFT and DPO receive exactly the same 20K samples. The SFT variant uses only the $(I_i, q_i, y_i^c)$ triple (ignoring $y_i^r$ ), while the DPO variant uses the full preference pair $(I_i, q_i, y_i^c, y_i^r)$ . This ensures that any performance differences are attributable to the training objective, not to differing amounts of training data. This is in contrast to prior work (Wang et al., 2024b; Sun et al., 2024a) which compared a Stage 1 model against a Stage 1 + DPO model—a comparison that conflates "more training" with "better training objective."

Data scale rationale: The 20K sample count was chosen because it "is a scale comparable to recent DPO studies for MLLMs" (Section 3.1), specifically citing Yu et al. (2024), Yu et al. (2025), and Yang et al. (2025c). The data scaling experiment in Figure 4 and Table H later sweeps from 3K to 40K samples to verify that the 20K point is representative.

Model scaling protocol: The paper conducts experiments across all combinations of four vision encoder sizes and four LLM sizes, but for the main scaling analyses (Figures 2 and 3), one component is varied while the other is fixed. Specifically:

Figure 2: vision encoder varies (B/16 → L/16 → So/16 → g/16) with Qwen2.5-3B fixed as the LLM.
Figure 3: LLM varies (0.5B → 1.5B → 3B → 7B) with SigLIP2-So/16 fixed as the vision encoder.
Figure 4 and the data scaling experiments: Qwen2.5-1.5B + SigLIP2-So/16 is fixed while data varies from 3K to 40K.

This factorial design allows the paper to claim that DPO's superiority holds across model scales rather than being an artifact of a particular configuration.

SFT and DPO Objectives: Mathematical Form and Intuition

The paper focuses on two post-training objectives, both applied to the same $X_{\text{PT}}$ preference dataset, but using fundamentally different optimization signals.

Supervised Fine-Tuning (SFT) objective. The SFT loss is the standard maximum-likelihood objective used throughout the LLM and MLLM literature for instruction tuning:

$L_{\text{SFT}} = -\mathbb{E}_{i \sim X_{\text{PT}}} \log \pi_{\theta}(y_i^c | I_i, q_i)$

where $\pi_{\theta}$ is the MLLM being trained (parameterised by $\theta$ ), $I_i$ is the input image, $q_i$ is the query text, $y_i^c$ is the chosen response, and the expectation is taken over the post-training dataset $X_{\text{PT}}$ .

What it computes: For each training example, the model generates a probability distribution over the chosen response tokens conditioned on the image and query, and the loss is the negative log-likelihood of the correct sequence. Minimizing this loss means increasing the probability the model assigns to the chosen response. This is computed token-by-token using teacher forcing: the model predicts each token given the ground-truth prefix, and the per-token losses are averaged.

Why this form: Maximum-likelihood estimation is the standard objective for instruction tuning because it directly encourages the model to reproduce the desired output distribution. It is simple, stable, and has well-understood convergence properties. However, it has a critical limitation for vision encoder training: the gradient signal only pushes toward the chosen response. There is no penalty for assigning high probability to incorrect but plausible alternatives, no mechanism for the model to learn that a rejected response $y_i^r$ is wrong, and no requirement that the model distinguish fine-grained visual details that differentiate the chosen from the rejected answer. The objective effectively says "make this answer likely" without saying "and make that alternative unlikely."

Direct Preference Optimization (DPO) objective. The DPO loss (Rafailov et al., 2023) is a contrastive objective that operates on preference pairs without requiring an explicit reward model:

$L_{\text{DPO}} = -\mathbb{E}_{i \sim X_{\text{PT}}} \log \sigma \left( \beta \left( \log \frac{\pi_{\theta}(y_i^c | I_i, q_i)}{\pi_{\text{ref}}(y_i^c | I_i, q_i)} - \log \frac{\pi_{\theta}(y_i^r | I_i, q_i)}{\pi_{\text{ref}}(y_i^r | I_i, q_i)} \right) \right)$

where $\sigma(\cdot)$ is the logistic (sigmoid) function, $\beta$ is a temperature hyperparameter controlling the strength of preference alignment, $\pi_{\theta}$ is the model being trained, $\pi_{\text{ref}}$ is a frozen reference model (typically the base model before DPO training), $y_i^c$ is the chosen response, $y_i^r$ is the rejected response, and the expectation is taken over $X_{\text{PT}}$ .

What it computes: The DPO loss encourages the model to increase the relative probability of the chosen response compared to the rejected response, where "relative" means measured against the reference model's probabilities. The inner term $\log \frac{\pi_{\theta}(y^c)}{\pi_{\text{ref}}(y^c)} - \log \frac{\pi_{\theta}(y^r)}{\pi_{\text{ref}}(y^r)}$ is a "reward difference"—the model's improvement in the chosen response's log-probability minus its improvement in the rejected response's log-probability, both measured relative to the reference model. Multiplying by $\beta$ scales this reward difference into logit space for the sigmoid. The sigmoid $\sigma(\cdot)$ maps the scaled reward difference to a probability in $[0, 1]$ , representing the model's predicted probability that the chosen response is preferred over the rejected response. The outer negative log-loss then penalizes the model when this predicted preference probability is low.

In operational terms: for each preference pair, the model computes four log-probabilities (chosen and rejected under both current and reference models), forms the reward difference, feeds it through a sigmoid, and takes the negative log. The gradient pushes the model to (a) increase $\pi_{\theta}(y^c)$ relative to $\pi_{\text{ref}}(y^c)$ and (b) decrease $\pi_{\theta}(y^r)$ relative to $\pi_{\text{ref}}(y^r)$ .

Why this form: The DPO objective has a fundamental property that the SFT objective lacks: it is contrastive. The model must learn to distinguish the chosen from the rejected response, which requires it to identify what visual information in the image supports the chosen answer over the rejected one. Consider the example shown in Figure 7: the query is "Are there curtains on the window?" with a chosen response mentioning "curtains appear to be patterned with a floral design" and a rejected response claiming "curtains appear to be white." To assign higher probability to the chosen response, the model's vision encoder must produce features that encode the curtain's pattern (presence of floral design) rather than just its existence or color. The SFT objective, by contrast, would be satisfied as long as the model says something about curtains—it never encounters the "white curtains" hypothesis and therefore receives no gradient signal discouraging that interpretation. This is the mechanism by which DPO drives the vision encoder toward fine-grained, precisely localized features.

The reference model $\pi_{\text{ref}}$ serves as a regularizer: without it, the DPO objective could be satisfied by arbitrarily increasing $\pi_{\theta}(y^c)$ and decreasing $\pi_{\theta}(y^r)$ , potentially causing the model to diverge from reasonable language generation. The reference model anchors the optimization, ensuring that the model only needs to improve relative to its own starting point rather than achieving some absolute probability threshold.

Training hyperparameters for Stage 2. The DPO training uses the LLaVA-OneVision DPO script with a learning rate of $1 \times 10^{-6}$ and a batch size of 256. The SFT training for Stage 2 uses the standard finetuning script with a learning rate of $1 \times 10^{-5}$ and a batch size of 256, but crucially "removes the vision-encoder-specific LR" (Appendix F.1), mirroring the DPO setup where no component-specific learning rates are used. The temperature $\beta$ for DPO is not explicitly stated in the paper but follows the default from the LLaVA-OneVision DPO script.

Why different learning rates: The paper notes (Appendix F.1) that "DPO requires substantially smaller LRs than SFT, partly because DPO accounts for both chosen and rejected responses, effectively doubling the batch size per iteration compared to SFT." This observation is consistent with prior work: InternVL2.5 (Wang et al., 2024b) uses $2 \times 10^{-7}$ for DPO and $4 \times 10^{-5}$ for SFT. The paper treats learning rate as a hyperparameter to be tuned per-objective rather than forced to be identical, since the objectives have fundamentally different gradient characteristics.

MLLM Evaluation Protocol: The Cambrian Benchmark Suite

The paper evaluates MLLMs using the benchmark suite from Cambrian (Tong et al., 2024a), which covers 16 individual tasks organized into four categories. This decomposition is essential to the paper's central finding that DPO's benefits are concentrated on vision-intensive tasks.

The four benchmark categories:

General VQA (4 tasks): GQA (Hudson & Manning, 2019), SEED-image (Ge et al., 2023), MME-perception (Fu et al., 2023), and MMBench (Liu et al., 2024b). These tasks test general-purpose visual understanding without requiring specialized knowledge or fine-grained perception.
Knowledge VQA (4 tasks): AI2D (Hiippala et al., 2021), ScienceQA-image (Lu et al., 2022), MathVista-math (Lu et al., 2023), and MMMU-vision (Yue et al., 2024). These tasks require scientific reasoning, mathematical problem-solving, or domain-specific knowledge applied to visual inputs.
OCR & Chart VQA (4 tasks): TextVQA (Singh et al., 2019), DocVQA (Mathew et al., 2021), ChartQA (Masry et al., 2022), and OCRBench (Liu et al., 2023b). These tasks require reading text in images, interpreting document layouts, or understanding charts and graphs.
Vision-Centric VQA (4 tasks): MMVP (Tong et al., 2024b), RealWorldQA (xAI, 2024), CVBench-2D, and CVBench-3D (Tong et al., 2024a). These tasks specifically probe fine-grained visual perception, including distinguishing visually similar objects, understanding spatial relationships, and reasoning about 3D structure from 2D images.

Category designation rationale: The paper labels OCR & Chart VQA and Vision-Centric VQA as "strongly vision-related" tasks, General VQA as moderately vision-related, and Knowledge VQA as "weakly vision-related." This designation is based on the degree to which task success depends on extracting detailed visual information versus applying learned knowledge. OCR tasks, for example, require reading small text that may span only a few pixels—a failure of vision directly causes a failure of the MLLM regardless of the LLM's reasoning capability. Knowledge tasks, by contrast, can often be answered correctly even with coarse visual understanding if the LLM possesses the relevant factual knowledge.

Score computation and aggregation. For each benchmark, the standard evaluation metric is accuracy (percentage of correctly answered questions). The exception is MME, which produces scores on a 0–2000 scale; these scores are rescaled by a factor of 20 (dividing by 20) to map to a 0–100 scale, making them comparable with other benchmarks in the average computation. The "Average (All)" score reported in figures and tables is the arithmetic mean across all 16 benchmarks after this rescaling. The per-category scores are the arithmetic means within each 4-task group.

Evaluation implementation details. For most benchmarks, the paper uses the Cambrian source code directly. The one exception is DocVQA: the Cambrian implementation requires manual submission of result CSV files to an evaluation server, which is impractical for the scale of experiments. Instead, DocVQA scores are obtained using the lmms-eval library (Zhang et al., 2024).

Why this decomposition matters: The paper's key empirical finding—that DPO outperforms SFT by $+4.2$ percentage points on OCR & Chart VQA and $+2.4$ percentage points on Vision-Centric VQA with SigLIP2-L/16, but only $+0.3$ percentage points on Knowledge VQA (Section 3.2)—would be invisible if only aggregate performance were reported. The decomposition enables the inference that DPO specifically improves visual processing, since the largest gains appear on tasks where visual detail is the primary bottleneck, while tasks bottlenecked by LLM knowledge show minimal improvement.

Vision Encoder Isolation and Probing Methodology

A core methodological contribution of the paper is the protocol for evaluating the vision encoder independently of the LLM. This enables the paper's central claim that DPO reshapes visual representations, not just language generation.

Encoder extraction procedure. After Stage 2 post-training is complete, the vision encoder and multimodal projector are physically separated from the LLM. The resulting frozen components can then be evaluated on classic computer vision tasks. Two configurations are used:

Encoder-only features: features extracted from the vision encoder's final layer output, before the projector. These are the patch-level features that the vision encoder produces from the raw image.
Encoder + projector features: features that have passed through both the vision encoder and the 2-layer MLP projector. These are the "visual tokens" that would normally be fed as input to the LLM.

Linear probe evaluation for ImageNet classification. The paper evaluates the extracted vision features using linear probing on ImageNet-1K classification, following the protocol from the OpenAI CLIP repository. Specifically:

The feature extractor (vision encoder, or encoder + projector) is frozen.
A scikit-learn Logistic Regression classifier with L2 regularization is trained on the extracted features, sweeping over lambda (regularization strength) values with a maximum of 1000 iterations.
Because evaluating on the full 1.28M ImageNet training set is computationally expensive, validation is performed on a 50K random subset of the ImageNet validation data.

Additionally, for rapid screening experiments, the paper implements a prototype-based linear classifier: the features of all training examples belonging to each class are averaged to form a class prototype vector, and classification is performed by nearest-prototype matching. The paper verifies that this faster method yields "similar performance trends to the standard Logistic Regression approach" (Appendix F.2).

What ImageNet accuracy measures for a vision encoder extracted from an MLLM: The linear probe measures how linearly separable the visual features are with respect to object categories, which serves as a proxy for the quality and discriminability of the representations. A higher Top-1 accuracy indicates that the vision encoder produces features where different object classes occupy distinct, well-separated regions of the embedding space. Critically, because the probe is linear, it cannot add representational capacity—it can only measure what is already present in the frozen features.

Segmentation probing with ADE20K. To measure the vision encoder's localization ability—its capacity to produce features that distinguish different semantic regions within an image—the paper uses a patch-level segmentation probing protocol based on Covert et al. (2025). The procedure is:

The vision encoder is frozen.
A two-layer MLP head is attached and trained as a patch-level classifier on the ADE20K dataset (Zhou et al., 2017), which contains pixel-level semantic segmentation annotations across 150 object and stuff categories.
Training uses 5 epochs and a learning rate of $1 \times 10^{-3}$ , following the default configuration from the Covert et al. codebase.
Evaluation uses patch-level recall as the primary metric, averaged over 6 random seeds. Patch-level recall measures what fraction of ground-truth object pixels are correctly classified by the probing head.

Why segmentation probing is informative: Segmentation requires the vision encoder to produce features that vary spatially according to object boundaries. A vision encoder that pools all image regions into a single global representation (as some contrastively trained models tend to do) would perform poorly on this task regardless of how good its global ImageNet features are. The finding that DPO-trained encoders outperform SFT-trained encoders on this metric (by $+1.08$ percentage points for CLIP-L/14 336px, as reported in Figure 8) indicates that DPO training preserves or enhances the spatial specificity of visual features—consistent with the paper's claim that DPO produces "precisely localized visual representations."

Gradient visualization via Grad-CAM. To provide mechanistic evidence for how DPO reshapes visual representations differently from SFT, the paper uses gradient-weighted class activation mapping (Grad-CAM; Selvaraju et al., 2017) to visualize which image regions receive the strongest gradient signals during post-training.

The procedure, detailed in Appendix F.3, is:

A training pipeline is constructed using a single sample from the post-training dataset.
The loss for that sample is computed according to either $L_{\text{SFT}}$ or $L_{\text{DPO}}$ as defined in Equation (1).
A backward pass is performed, and the gradients with respect to the vision encoder's feature activations $A := \Phi_{\text{ViT}}(I)$ are extracted.
The gradient magnitude at each spatial position (each ViT patch token) is measured and visualized as a heatmap overlaid on the input image.

The visualization is performed "around the 20th step" of training rather than at initialization, because the standard LLaVA-OneVision cosine learning rate scheduler sets the learning rate near zero at the start, producing uninformative gradients at very early steps.

What Grad-CAM reveals: The resulting heatmaps (Figure 7 and Appendix Figure E) show that DPO gradients concentrate strongly on question-relevant image regions (e.g., the window area when asking about curtains, the text region when asking about a document date), while SFT gradients are more diffusely distributed across the image. This provides direct evidence for the paper's central mechanistic hypothesis: DPO's contrastive objective forces the model to attend to the specific visual details that distinguish the chosen from the rejected response, producing more informative gradient signals for the vision encoder.

Representational alignment measurement. To measure how well the vision encoder's representations align with language model representations—a desirable property for MLLM integration—the paper uses the representational alignment metric from Huh et al. (2024), implemented via the Platonic Representation repository. The procedure is:

Vision encoders trained within MLLMs alongside LLMs of different sizes (0.5B, 1.5B, 3B Qwen2.5) are evaluated.
Alignment scores are computed between each vision encoder and five reference LLMs: Gemma-2B, Gemma-7B (Team et al., 2024b), LLaMA-3-8B (Dubey et al., 2024), Mixtral-8x7B (Jung et al., 2010), and Bloomz-7B (Muennighoff et al., 2022).
Scores are computed on the 'minhuh/prh' dataset (1024 examples).
Due to the small dataset size, results are averaged over three random seeds.

What alignment measures: A higher alignment score indicates that the vision encoder's representations have a similar geometric structure to the LLM's representations—nearby images in vision-encoder space correspond to nearby sentences in LLM space, and the overall topology of the two spaces is similar. This matters for MLLM performance because the projector (which is typically a relatively shallow MLP) can more easily map between aligned spaces than between spaces with fundamentally different geometries.

The PIVOT Recipe: Repurposing MLLM Training as Vision Encoder Enhancement

Section 5 reframes the two-stage MLLM training process as a standalone vision encoder enhancement procedure called Preference-Instructed Vision OpTimization (PIVOT). The key insight is that training a vision model with an LLM head using DPO produces visual representations that remain beneficial even when the original LLM is discarded and replaced with a completely new one.

PIVOT training procedure. The PIVOT process mirrors exactly the training pipeline described in Sections 3.1 and F.1, but is applied to any vision encoder (not just SigLIP2) and always uses a Qwen2.5-1.5B LLM as the training head:

The vision encoder is attached to a Qwen2.5-1.5B LLM via a 2-layer MLP projector.
Stage 1 pre-training: Projector-only training on LAION/CC/SBU-558K, followed by end-to-end training of all parameters on LLaVA-OneVision-3.2M.
Stage 2 post-training: DPO training on 20K preference pairs from MPO. This is what makes it "PIVOT" rather than generic MLLM training—the DPO objective is the defining characteristic.
After training, the vision encoder (and optionally the projector) is extracted, frozen, and used as a standalone visual feature extractor.

Computational cost. The paper reports that PIVOT training requires "just 18 hours of training on 8 H100 GPUs" (Section 5, Abstract). This is compared to SigLIP2's training, which used "up to 2K TPUv5e chips" (Section 5). The paper claims PIVOT uses "less than 1% of the computational cost of standard vision pretraining."

PIVOT evaluation protocol (Stage 3). To evaluate whether the PIVOT-enhanced encoder actually produces better features for MLLM use, a three-stage pipeline is used (Figure D, Appendix F.6):

The frozen PIVOT-enhanced vision encoder is paired with a fresh Qwen2.5-1.5B LLM (not the one used during PIVOT training) via a new randomly initialized projector.
Projector-only pretraining: Only the new projector is trained on LAION/CC/SBU-558K, aligning the PIVOT encoder's features with the new LLM's input space.
Instruction finetuning: The projector and LLM are trained jointly on Cambrian's 737K instruction-tuning dataset, with the PIVOT-enhanced vision encoder remaining frozen. The training uses a batch size of 256 and a learning rate of $1 \times 10^{-5}$ .

The resulting MLLM is evaluated on the same 16-benchmark Cambrian suite. This protocol is adopted from prior work (Cambrian, DINO-MLLM, MLLM-Data) specifically because it "allows us to study visual representations efficiently" by isolating the contribution of the vision encoder from confounding factors like the LLM's familiarity with the training data.

Projector reuse ablation (Appendix B.2). The paper investigates whether reusing the PIVOT-trained projector (rather than training a fresh one) during Stage 3 improves performance. Three configurations are tested:

PIVOT-projector 0: the PIVOT-trained projector is entirely discarded; a fresh 2-layer MLP is trained.
PIVOT-projector 1: the first linear layer of the PIVOT-trained projector is frozen and reused; one additional randomly initialized layer is appended and trained (total 2 layers).
PIVOT-projector 2: both layers of the PIVOT-trained projector are frozen; two additional layers are appended (total 4 layers).

The results in Table B show that the 1+1 configuration (frozen first layer + one new layer) achieves the best downstream performance—for example, 54.6% average with Qwen2.5-1.5B compared to 52.4% with no projector reuse and 54.3% with two frozen layers + two new layers. The paper adopts the 1+1 configuration for all PIVOT experiments in Section 5.

Why PIVOT works with diverse vision encoders. The paper applies PIVOT to a range of vision encoders with fundamentally different pretraining objectives: contrastive image-language models (CLIP-L/14 336px, SigLIP1-So/14), vision-only self-supervised models (DINOv2-g/14 378px, MAE-H/14 224px, MOCO-B/16 224px), and a purely supervised ImageNet-trained model (SupViT-H/14 224px). The consistent improvement across all five (Table 1, bottom section) suggests that PIVOT is not merely fine-tuning image-language aligned encoders further in the same direction—it is adding a capability (fine-grained, language-guided feature localization) that was absent from the original pretraining, regardless of what that pretraining was.

Ensemble with PIVOT. The paper also demonstrates that PIVOT composes with multi-encoder ensembling (Tong et al., 2024b). An ensemble of SigLIP1-So/14 + ConvNeXt-XXL achieves 51.4% average VQA, while SigLIP1-So/14+PIVOT alone achieves 53.2%. Combining SigLIP1-So/14+PIVOT with ConvNeXt-XXL further raises performance to 53.6%, showing that the representational improvements from PIVOT are complementary to the benefits of combining different vision architectures.

Training data scale for Stage 3. The paper verifies that PIVOT's benefits persist when Stage 3 uses more training data (Table C). When trained on LLaVA-OV-3M instead of Cambrian-737K, the PIVOT-enhanced SigLIP2-So/16 maintains a $+2.3$ percentage point advantage over the baseline (59.2% vs. 56.9%), compared to $+3.2$ percentage points with Cambrian-737K (55.6% vs. 52.4%). The gap is somewhat smaller but remains substantial, indicating that PIVOT representations are genuinely better starting points for MLLM training, not just artifacts of a particular data budget.

Full-parameter training in Stage 3. The paper's main PIVOT evaluation freezes the vision encoder during Stage 3 to isolate its representational contribution. However, the paper also tests a variant where all parameters—including the vision encoder—are updated during Stage 3 (Table D). The PIVOT-enhanced encoder still outperforms the baseline ( $+0.7$ percentage points with Qwen2.5-1.5B, $+0.9$ percentage points with Qwen2.5-0.5B), demonstrating that the initial representations matter even when subsequent training can modify them. The persistent advantage suggests that PIVOT produces representations that serve as better initialization points, not just better frozen features.

4. Key Insights and Innovations

Innovation 1: The Vision Encoder Is Not a Passive Sensor — Post-Training Objectives Fundamentally Reshape How MLLMs "See"

The paper's most fundamental conceptual contribution is overturning the dominant LLM-centric assumption about where MLLM capabilities originate. Before this work, the field implicitly treated the vision encoder as a relatively static feature extractor whose quality was determined entirely by its pretraining (CLIP, SigLIP, DINOv2, etc.) — any improvements from MLLM training were attributed to the LLM learning to better reason about visual features, not to the vision encoder producing better features. The evidence for this assumption was entirely circumstantial: the LLM contains orders of magnitude more parameters and undergoes more dramatic behavioral changes during instruction tuning, so it seemed natural that learning was concentrated there.

This paper provides the first systematic evidence that this assumption is wrong. By surgically extracting vision encoders from MLLMs post-training and evaluating them on classic vision tasks (ImageNet classification, Figure 6; ADE20K segmentation, Figure 8), the paper demonstrates that MLLM training rewrites visual representations in measurable, functionally significant ways. The vision encoder's ImageNet Top-1 accuracy increases by up to +4.4 percentage points when trained alongside a larger LLM (SigLIP2-So/16 with 7B vs. 0.5B Qwen2.5, Figure 6 left), and its patch-level segmentation recall improves by up to +1.08 percentage points for CLIP-L/14 when trained with DPO rather than SFT (Figure 8). These are not subtle effects — they represent meaningful improvements in the standalone quality of visual features, detectable without any language model in the loop.

Why this is a fundamental shift rather than incremental: Prior work on MLLM architecture explored whether to update the vision encoder (frozen vs. unfrozen; Liu et al., 2023a vs. Li et al., 2024) and how many encoders to use (Tong et al., 2024b,a), but never asked what actually changes inside the encoder under different training objectives. This paper establishes that the training objective choice is not merely an optimization detail — it is a first-class design decision that determines the kind of visual features the MLLM has access to. The implication is that vision encoder training should not be treated as a one-time pretraining step separate from MLLM development, but rather as something that can be substantially improved through the right post-training objective. This reframes the vision encoder from a fixed input device to a trainable component whose representations are shaped by the language-guided optimization it participates in.

Evidence anchor: Figure 6 (ImageNet accuracy scaling with both vision encoder and LLM size), Figure 8 (segmentation probing across six encoders), and Finding 3 in Section 4.2 ("MLLM training not only adapts the language model but also reshapes the visual representations that determine how the model sees an image").

Innovation 2: DPO's Contrastive Objective Produces Precisely Localized Visual Features by Design, Not by Accident

The second conceptual contribution is the mechanistic explanation for why DPO improves visual representations more than SFT, supported by the first gradient-level evidence of how different training objectives propagate information back into the vision encoder. The paper's Grad-CAM visualizations (Figure 7, Appendix Figure E) reveal a previously unobserved phenomenon: the gradient signals that flow backward from the LLM into the vision encoder are qualitatively different under DPO versus SFT. SFT gradients are diffusely distributed across the image, while DPO gradients concentrate sharply on question-relevant regions.

This is not a minor implementation detail — it reflects a fundamental difference in what the two objectives demand from the vision encoder. The SFT objective asks: "given this image and query, maximize the probability of this answer." This can be satisfied by improving any aspect of the visual representation that helps predict the chosen tokens, including global scene-level features, object category information, or even statistical regularities in the language model's output distribution that have little to do with visual detail. The DPO objective asks: "given this image and query, make this answer more probable than that alternative." This forces the model to identify what visual information distinguishes the chosen from the rejected response. If the chosen response says "floral pattern" and the rejected says "white curtains," the only way to satisfy the DPO objective is to produce visual features that encode the curtain's pattern — and the gradient signal will be strongest at the image locations where that pattern information resides.

Comparison to prior understanding: Prior work on RL for MLLMs (Yu et al., 2024; Wang et al., 2024b; Sun et al., 2024a) attributed DPO's benefits entirely to better language generation or alignment — the LLM learning to produce more helpful, less harmful, or less hallucinated text. The possibility that DPO was also improving visual perception was never investigated, because no one looked at the vision encoder separately. The gradient visualization evidence in this paper provides the first mechanistic bridge between the DPO objective and visual representation quality: DPO doesn't just produce better answers, it produces better seeing.

Why this insight has implications beyond performance: Knowing that DPO improves visual localization by design — not by accident — changes how practitioners should think about preference data construction. If the gradient concentration effect depends on the contrast between chosen and rejected responses highlighting specific visual differences, then preference pairs where the chosen and rejected responses differ in visually meaningful ways (object attributes, spatial relationships, fine-grained categories) should produce stronger visual improvements than pairs where they differ in factual knowledge or reasoning style. This hypothesis is explicitly suggested by the paper's observation that for global queries like "Describe this photo in detail," DPO and SFT produce similarly diffuse gradients (Appendix Figure E, bottom rows) — the contrastive advantage requires localized visual distinctions in the preference data. This opens a new dimension of preference dataset design that was previously invisible.

Evidence anchor: Figure 7 (Grad-CAM visualizations showing DPO gradients concentrated on question-relevant regions), Appendix Figure E (additional examples including global queries), Finding 4 in Section 4.2 ("DPO steers the vision encoder toward a more fine-grained analysis of visual information, improving its object localization capabilities").

Innovation 3: Vision Model Quality for MLLMs Is Severely Underestimated by Standard Pretraining Metrics — PIVOT Reveals Massive Untapped Potential

The third conceptual contribution is the demonstration that state-of-the-art vision models — including SigLIP2, which was specifically designed as an MLLM backbone and trained on 40B image-text pairs — have substantial room for improvement when measured by their actual utility in MLLMs. The PIVOT results in Table 1 reveal a pattern that should be surprising given the scale of modern vision pretraining: a 400M-parameter SigLIP2-So/16 encoder, after PIVOT enhancement (requiring <1% of the original pretraining compute), outperforms the 1B-parameter SigLIP2-g/16 across all VQA categories (55.6% vs. 53.9% average). Even more striking, a previous-generation SigLIP1-So/14 encoder (30B pretraining samples) with PIVOT outperforms the next-generation SigLIP2-So/16 (40B pretraining samples, improved training recipe) by 53.2% to 52.4%.

These results challenge the default assumption that better vision pretraining metrics — larger models, more data, improved contrastive objectives — directly translate to better MLLM performance. They suggest instead that there is a gap between the representations that vision pretraining optimizes for and the representations that MLLMs actually need. Standard vision pretraining (CLIP, SigLIP) optimizes for global image-text alignment: the pooled image embedding should be close to the paired text embedding in a shared space. But MLLMs consume vision encoder outputs as sequences of patch-level tokens, not as global embeddings, and they need features that support fine-grained spatial reasoning, precise attribute discrimination, and localization of small objects — capabilities that global alignment objectives do not explicitly encourage.

What makes this a conceptual advance rather than just a strong result: The paper is not merely demonstrating that "training more helps." It is identifying a specific type of training — preference alignment with contrastive language feedback — that produces qualitatively different representations than scaling up pretraining. The finding that even DINOv2 (a 1B-parameter model trained on 2B images with no language supervision), MAE (trained purely with pixel reconstruction), and a supervised ImageNet classifier all improve under PIVOT (Table 1, bottom section) indicates that this gap is universal — it exists regardless of pretraining objective and can be partially closed with a small amount of targeted post-training. This reframes vision model development for MLLMs: rather than competing to build ever-larger pretrained encoders, there may be more efficient paths through purpose-built post-training procedures that specifically target MLLM-relevant visual capabilities.

Why this is not just "fine-tuning helps": Standard fine-tuning of vision encoders on downstream tasks is well-known to improve performance on those tasks. What distinguishes PIVOT is (a) the training uses language preference signals, not vision labels, (b) the resulting improvements transfer to a completely new LLM head the encoder was never trained with, and (c) the scale of improvement relative to compute investment is disproportionate — surpassing next-generation models trained on 10,000× more data. This suggests PIVOT is not "fine-tuning" in the conventional sense of adapting representations to a specific distribution, but rather a process of aligning visual representations with the kind of fine-grained, spatially localized, language-relevant features that MLLMs require.

Evidence anchor: Table 1 (PIVOT-enhanced encoders outperforming larger and newer baselines), Section 5 claim of <1% pretraining compute, Finding 6 ("Existing vision models possess substantial potential for improvement within MLLMs").

Innovation 4: Larger LLMs Provide Better Visual Supervision — The LLM Is Not Just a Consumer of Vision, It Is a Teacher

The fourth conceptual insight concerns the direction of information flow in MLLM training. The standard view treats the LLM as a consumer of visual features — the vision encoder produces representations, and the LLM learns to use them for language generation. The paper's scaling results (Figure 6 left, Figure 10) suggest a complementary role: the LLM is also a teacher that provides supervisory signals back to the vision encoder, and larger LLMs are better teachers.

The evidence for this is twofold. First, vision encoders trained alongside larger LLMs produce better standalone visual features: SigLIP2-So/16 achieves +4.4 percentage points higher ImageNet Top-1 accuracy when trained with a 7B LLM vs. a 0.5B LLM (Figure 6 left). This improvement is measured after the LLM is removed, meaning it reflects genuinely better visual representations, not better language generation. Second, vision encoders trained with larger LLMs show stronger representational alignment with reference language models (Figure 10) — the visual feature space becomes more structurally similar to language model feature spaces as the training LLM grows.

The mechanism, as the paper hypothesizes, is that larger LLMs produce more informative gradient signals during backpropagation. A larger LLM has more capacity to model the intricate relationships between visual features and language outputs, and its loss surface with respect to the vision encoder's parameters is therefore richer — it provides more precise feedback about which aspects of the visual representation are useful for language generation and which are not. The vision encoder, receiving these richer gradients, learns to produce features that are better structured for language-guided tasks.

Comparison to prior assumptions: The dominant framing in MLLM research has been that the LLM's role is to interpret visual features, with the implicit corollary that a sufficiently powerful LLM could compensate for weaker vision. Scaling studies typically focus on increasing LLM size to improve reasoning, not on the LLM's backward influence on vision. This paper shows that LLM scale matters for visual representation quality even when controlling for the vision encoder — a finding that has direct implications for MLLM architecture design. It suggests that training a vision encoder with a temporarily larger LLM head (as PIVOT does) could be a cost-effective strategy: use the large LLM's superior teaching signal during training, then deploy with a smaller, more efficient LLM while retaining the improved vision encoder.

Why this is a discovery, not an assumption: The paper explicitly acknowledges that the field had an "implicit, LLM-centric assumption about the source of MLLM capabilities" (Section 1) — but that assumption concerned the LLM's role in generating answers, not its role in training vision. The teacher-student dynamic revealed here was not predicted by existing theory and was discovered through the vision encoder isolation experiments that are this paper's primary methodological innovation.

Evidence anchor: Figure 6 left (ImageNet accuracy scaling with LLM size), Figure 10 (representational alignment increasing with LLM scale), Finding 5 ("The vision encoder benefits from a larger LLM, which provides more informative backward signals for visual representation within an MLLM").

Innovation 5: SFT Memorizes, RL Localizes — A New Taxonomy of Training Objective Effects on Multimodal Representations

The final conceptual contribution is a reframing of the SFT vs. RL debate from a question of which performs better to a question of what kind of representations each produces. The paper's results, when synthesized, suggest a crisp functional distinction: SFT encourages the model to reproduce correct answers, which primarily improves global, category-level visual features; DPO encourages the model to discriminate between correct and incorrect answers, which specifically improves fine-grained, spatially localized visual features.

This taxonomy explains the otherwise puzzling pattern in the MLLM benchmark results (Figures 2, 3, 4): DPO dramatically outperforms SFT on OCR & Chart VQA and Vision-Centric VQA (tasks requiring reading small text, counting objects, discriminating visually similar items), but provides minimal advantage on Knowledge VQA (tasks solvable with coarse visual understanding plus domain knowledge) or General VQA (tasks solvable with scene-level features). It also explains the gradient visualization results (Figure 7): SFT gradients are diffuse because improving global scene features helps marginally with most answers; DPO gradients are localized because discriminating between specific answer alternatives requires attending to specific image regions. And it explains the segmentation probing results (Figure 8): DPO-trained encoders produce better localization maps because their training objective explicitly rewarded fine-grained spatial discrimination.

What makes this a taxonomic advance: Prior work comparing SFT and RL (Chu et al., 2025; Shenfeld et al., 2025) characterized the difference as "SFT memorizes, RL generalizes" — a claim about out-of-distribution behavior. This paper's taxonomy is orthogonal and complementary: it's about representational granularity, not generalization. Even on in-distribution tasks, DPO outperforms SFT when the task requires fine-grained visual discrimination, regardless of whether the specific examples are novel. This suggests that DPO and SFT are not simply better or worse versions of the same thing — they are optimizing for fundamentally different kinds of visual competence, and the choice between them should depend on the visual demands of the target application.

Evidence anchor: Figure 2 (DPO gains concentrated on vision-intensive tasks), Figure 7 (Grad-CAM localization patterns), Figure 8 (segmentation probing results), Section 3.2 Finding 2 (quantified performance gaps by category), Appendix Figure E (global vs. local query gradient patterns).

5. Experimental Analysis

Evaluation Methodology

Dataset. All MLLM experiments use the Cambrian benchmark suite (Tong et al., 2024a), covering 16 individual tasks across four categories: General VQA (GQA, SEED-image, MME-perception, MMBench), Knowledge VQA (AI2D, ScienceQA-image, MathVista-math, MMMU-vision), OCR & Chart VQA (TextVQA, DocVQA, ChartQA, OCRBench), and Vision-Centric VQA (MMVP, RealWorldQA, CVBench-2D, CVBench-3D). The MLLM training data consists of publicly available datasets: LAION/CC/SBU-558K for projector-only pretraining, LLaVA-OneVision-3.2M for Stage 1 end-to-end pretraining (Li et al., 2025a), and a 20K random subset of the MPO-1.2 dataset (Wang et al., 2024b) for Stage 2 post-training. Vision encoder probing experiments use ImageNet-1K for classification (evaluated on a 50K random subset of the validation set for efficiency) and ADE20K (Zhou et al., 2017) for semantic segmentation. The representational alignment measurement uses the 'minhuh/prh' dataset (1,024 examples) from Huh et al. (2024). The full list of benchmarks with citations appears in Table E.
Base model(s). The MLLM architecture follows the LLaVA-OneVision framework (Li et al., 2025a), combining a SigLIP2 vision encoder (Tschannen et al., 2025) at four scales—B/16 (86M parameters), L/16, So/16 (400M), and g/16 (1B)—all operating at 384×384 pixel input resolution, with a Qwen2.5-Instruct LLM (Yang et al., 2025a) at four scales (0.5B, 1.5B, 3B, 7B parameters), connected by a 2-layer MLP projector. For the PIVOT evaluation in Section 5, additional vision encoders are tested: CLIP-L/14 336px (Radford et al., 2021), SigLIP1-So/14 (Zhai et al., 2023), DINOv2-g/14 378px (Oquab et al., 2024), MAE-H/14 224px (He et al., 2022), MOCO-B/16 224px (He et al., 2020), and a supervised ViT-H/14 224px (Dosovitskiy et al., 2021). The FLOPs-matched comparison in Section 5 uses Qwen2.5-1.5B as the fixed LLM head for all PIVOT-enhanced encoders. The paper argues (Section 4) that PaLM 2-S*—while not used here—would be "representative of the capabilities of many contemporary LLMs," with the Qwen2.5 family serving an analogous role in this study by providing a range of model scales with publicly available weights and established MLLM integration patterns.
Metrics. For MLLM evaluation, the primary metric is accuracy (% of correctly answered questions) on each benchmark, with per-category averages computed as the arithmetic mean across the four tasks in that category and "Average (All)" computed across all 16 benchmarks. MME scores (originally 0–2000 scale) are rescaled by a factor of 20 to map to 0–100 for comparability. For ImageNet classification probing, the metric is Top-1 accuracy using scikit-learn Logistic Regression with L2 regularization on frozen features. For segmentation probing, the metric is patch-level recall on ADE20K, averaged over 6 random seeds. For representational alignment, the metric is the alignment score from Huh et al. (2024), computed via the Platonic Representation repository and averaged over three random seeds. DocVQA scores are obtained using the lmms-eval library (Zhang et al., 2024) rather than the Cambrian implementation, which requires manual submission.
Baselines. The primary experimental comparison is between two Stage 2 post-training strategies applied to the same base MLLM: SFT (supervised fine-tuning on chosen responses only, following standard instruction tuning practice as in Ouyang et al., 2022 and Touvron et al., 2023b) and DPO (Direct Preference Optimization on preference pairs, following Rafailov et al., 2023). Both use identical training data (20K samples from MPO) but differ in the objective function. For the PIVOT vision encoder evaluation in Section 5, the baselines are the original (unenhanced) vision encoders paired with Qwen2.5-1.5B and trained on Cambrian-737K or LLaVA-OV-3M, following the evaluation protocol of Cambrian (Tong et al., 2024a), DINO-MLLM (Fan et al., 2025), and MLLM-Data (Han et al., 2025). Two model ensemble baselines from Tong et al. (2024b) are also included: SigLIP1-So/14 + DINOv2-g/14 (700M total parameters) and SigLIP1-So/14 + ConvNeXt-XXL (1.25B total parameters).
Generation budget / compute accounting. The paper does not use generation budgets in the sense of sampling multiple completions (as in best-of-N). Instead, the primary resource unit is training data: the number of samples seen during Stage 2 post-training, controlled to be identical (20K) for both SFT and DPO. The data scaling experiment in Figure 4 sweeps from 3K to 40K samples. For the PIVOT computational cost comparison, the paper reports wall-clock time and hardware: "just 18 hours of training on 8 H100 GPUs" versus SigLIP2's reported training on "up to 2K TPUv5e chips." The "# samples seen" column in Table 1 reports the total pretraining samples for each original encoder (e.g., 30B for SigLIP1-So/14, 40B for SigLIP2 variants) plus the 3M + 20K PIVOT samples, following the accounting convention of Cherti et al. (2023) and Zhai et al. (2023). For MLLM training hyperparameters, Stage 1 uses a learning rate of 1×10⁻⁵ and batch size 256; Stage 2 DPO uses 1×10⁻⁶ and batch size 256; Stage 2 SFT uses 1×10⁻⁵ and batch size 256 (with vision-encoder-specific LR removed, mirroring DPO).
Cross-validation / statistical protocol. The paper does not employ cross-validation for MLLM strategy selection, as the controlled comparison between SFT and DPO uses identical data and hyperparameter search spaces. For segmentation probing, results are averaged over 6 random seeds to account for variability in the MLP head initialization and training. For representational alignment, results are averaged over 3 random seeds due to the small size of the evaluation dataset (1,024 examples). The MLLM benchmarks are standard test sets with no training on test data, and the ImageNet probing uses the standard validation split. The data scaling experiments (Figures 4, 5; Table H) report single-run results without confidence intervals, which the paper does not explicitly justify but which follows the convention of the Cambrian benchmark suite.

Main Quantitative Results

MLLM Performance: DPO vs. SFT Across Model Scales

The central experimental finding is that DPO consistently outperforms SFT as a Stage 2 post-training strategy for MLLMs, with gains concentrated on vision-intensive tasks and holding across both vision encoder and language model scaling.

Vision encoder scaling with fixed LLM (Figure 2, Table F). When the Qwen2.5-3B LLM is held constant and the SigLIP2 vision encoder scales from B/16 (86M) to g/16 (1B), DPO achieves higher average accuracy than SFT at every scale. With SigLIP2-L/16, the DPO-trained model reaches 62.6% average vs. 60.8% for SFT—a +1.8 percentage point gap. The advantage is most pronounced on OCR & Chart VQA (+4.2pp with L/16) and Vision-Centric VQA (+2.4pp with L/16), while Knowledge VQA shows nearly identical performance (e.g., 52.0% vs. 51.5% for the four Knowledge tasks with L/16, a +0.5pp difference). As the vision encoder grows from B/16 to g/16, the DPO-trained MLLM's Vision-Centric VQA score improves by +4.5pp (from 45.3% to 52.0% average across the four Vision-Centric tasks) and OCR & Chart VQA improves by +10.6pp (from 57.2% to 66.0% average), while Knowledge VQA improves by only +1.9pp (from 54.2% to 56.1%). This confirms Finding 1: increasing vision encoder capacity is particularly important for tasks requiring fine-grained visual understanding.

Language model scaling with fixed vision encoder (Figure 3, Table G). When SigLIP2-So/16 is held constant and the Qwen2.5 LLM scales from 0.5B to 7B, DPO again outperforms SFT at every scale. With the 3B LLM, DPO achieves 63.9% average vs. 62.3% for SFT (+1.6pp); with the 7B LLM, DPO reaches 68.9% vs. 66.2% for SFT (+2.7pp). The per-category gaps with the 7B LLM are: General VQA +2.7pp, OCR & Chart VQA +3.1pp, Vision-Centric VQA +4.2pp, and Knowledge VQA +1.2pp. The DPO advantage persists and even widens slightly at larger LLM scales, confirming Finding 2: preference alignment produces MLLMs with superior performance to SFT, especially on strongly vision-related tasks, and this holds regardless of the language model's size.

Data scaling (Figure 4, Table H). When the MLLM configuration is fixed (Qwen2.5-1.5B + SigLIP2-So/16) and Stage 2 post-training data varies from 3K to 40K samples, DPO achieves high performance even with minimal data. At 3K samples, DPO reaches 60.4% average accuracy—already surpassing the SFT model trained on 40K samples (59.5%). SFT improves gradually and monotonically from 56.4% (3K) to 59.5% (40K), while DPO improves from 60.4% to 61.3% over the same range, suggesting DPO saturates earlier but at a substantially higher performance level. The most dramatic gap occurs at 3K samples: +4.0pp for DPO over SFT (60.4% vs. 56.4%). The per-domain breakdown in Appendix Figure B reveals an interesting pattern in Knowledge VQA: the DPO-SFT gap shrinks from +4.2pp at 3K to +0.7pp at 10K, suggesting that additional SFT data eventually compensates for the lack of preference signal on knowledge-intensive tasks, while DPO's advantage on vision-centric and OCR tasks remains robust.

Performance under distribution shift (Appendix Figure A). When the Stage 2 post-training data includes samples from a shifted distribution (longer responses or special tokens like <thinking> and <review>), DPO maintains robust performance while SFT degrades sharply. With 80% shifted-distribution samples, the DPO-trained MLLM achieves 62.2% average vs. 53.2% for SFT—a gap of +9.0pp. The degradation is most severe in Vision-Centric VQA, where the gap reaches +17.9pp at 80% shift. This aligns with prior findings by Chu et al. (2025) and Shenfeld et al. (2025) that RL-trained models are more robust to distribution change, but extends the evidence from specialized environments (card games, robot planning) to the broad set of 16 general VQA benchmarks.

Vision Encoder Probing: Quantifying Representational Change

The paper's core methodological contribution is isolating the vision encoder from the LLM and measuring how its representations change under different post-training objectives.

ImageNet classification (Figure 6, Appendix Figure C). When vision encoders are extracted from SFT-trained vs. DPO-trained MLLMs and evaluated via linear probe on ImageNet-1K, DPO-trained encoders consistently achieve higher accuracy. With the encoder+projector features (Figure 6), DPO outperforms SFT by +1.83pp for SigLIP2-So/16 paired with Qwen2.5-3B, and by +1.96pp for SigLIP2-L/16 with Qwen2.5-1.5B. With encoder-only features (Appendix Figure C, bypassing the projector), the same pattern holds: DPO-trained encoders produce more discriminative features. Additionally, vision encoders trained alongside larger LLMs achieve higher ImageNet accuracy: SigLIP2-So/16 trained with a 7B LLM shows +4.4pp higher Top-1 accuracy than the same encoder trained with a 0.5B LLM (Figure 6 left), supporting the finding that larger LLMs provide richer supervisory signals to the vision encoder during joint training.

Segmentation probing (Figure 8, Appendix Figure F). When the extracted vision encoders are evaluated on patch-level semantic segmentation via a two-layer MLP probe trained on ADE20K, DPO-trained encoders consistently outperform SFT-trained ones across six different backbone architectures. The quantitative gains over SFT (shown above the DPO bars in Figure 8) range from +0.55pp for SigLIP2-So/16 to +1.08pp for CLIP-L/14 336px. The qualitative results in Figure 9 and Appendix Figure F show that DPO-trained encoders produce segmentation maps with crisper object boundaries and better alignment with ground-truth annotations, while SFT-trained encoders generate more diffuse, less accurate maps. This directly supports Finding 4: DPO steers the vision encoder toward more fine-grained analysis of visual information, improving its object localization capabilities.

Data scaling for vision representations (Figure 5). Unlike MLLM benchmark performance, which improves with more post-training data for both SFT and DPO (Figure 4), the quality of visual representations—measured by ImageNet linear probe accuracy—benefits from data scaling only under DPO. As Stage 2 data increases from 3K to 40K, the DPO-trained encoder's ImageNet accuracy continues to improve, while the SFT-trained encoder's accuracy plateaus or even slightly degrades. This divergence suggests that SFT's MLLM benchmark improvements with more data (Figure 4) come primarily from better language generation or LLM adaptation, not from improved visual features, whereas DPO genuinely improves the vision encoder's representations with additional preference data.

Representational alignment (Figure 10). Vision encoders trained with DPO show stronger alignment scores with reference LLMs compared to SFT-trained encoders, and encoders trained with larger LLMs show consistently higher alignment across all five reference models (Gemma-2B, Gemma-7B, LLaMA-3-8B, Mixtral-8x7B, Bloomz-7B). For example, the gap between DPO and SFT alignment scores grows as the training LLM increases from 0.5B to 3B, and the absolute alignment increases monotonically with LLM size under both objectives. This provides convergent evidence for Finding 5: larger LLMs transmit more informative backward signals that produce vision representations better structured for language-guided tasks.

PIVOT: Preference Alignment as Vision Encoder Enhancement

Section 5 reframes the Stage 1 + Stage 2 DPO training as a standalone vision encoder enhancement procedure and evaluates its efficacy across diverse vision backbones.

SigLIP1 to SigLIP2 comparison (Table 1, top section). The original SigLIP1-So/14 (30B pretraining samples) achieves 50.9% average VQA when paired with Qwen2.5-1.5B. After PIVOT enhancement (+3M + 20K samples), the same encoder reaches 53.2%—surpassing the next-generation SigLIP2-So/16 (40B pretraining samples) at 52.4%. The PIVOT-enhanced SigLIP1 outperforms its unenhanced self across all four categories, with the largest gains in OCR & Chart VQA (+4.5pp, from 42.3% to 46.8%) and Vision-Centric VQA (+1.9pp, from 49.8% to 51.7%). The SFT-only variant (+SFT in Table 1) achieves 52.2%, confirming that the DPO component specifically contributes +1.0pp over SFT-based enhancement.

SigLIP2-So/16 to SigLIP2-g/16 comparison (Table 1, top section). The 400M-parameter SigLIP2-So/16 with PIVOT achieves 55.6% average—outperforming the 1B-parameter SigLIP2-g/16 (53.9%) despite having 2.5× fewer parameters. The PIVOT-enhanced So/16 surpasses the g/16 baseline in all four categories: General VQA (68.1% vs. 66.5%), OCR & Chart VQA (53.9% vs. 50.8%), Vision-Centric VQA (52.4% vs. 51.9%), and Knowledge VQA (48.1% vs. 46.4%). The SFT-enhanced So/16 (54.6%) already surpasses the g/16 baseline, but PIVOT provides an additional +1.0pp.

Scaling to larger PIVOT encoders (Table 1, top section). When PIVOT is applied to the already-large SigLIP2-g/16, performance improves from 53.9% to 56.7% average (+2.8pp). The SFT-enhanced g/16 reaches 55.4%, so the DPO component contributes +1.3pp specifically. Across all three SigLIP variants (So/14, So/16, g/16), the pattern is consistent: PIVOT > SFT enhancement > original encoder, with the DPO-specific advantage ranging from +1.0pp to +1.3pp.

Classic vision encoders with PIVOT (Table 1, middle section). All five non-SigLIP encoders improve under PIVOT:

CLIP-L/14 336px: 46.3% → 49.5% (+3.2pp)
DINOv2-g/14 378px: 40.9% → 43.6% (+2.7pp)
MAE-H/14 224px: 36.8% → 39.7% (+2.9pp)
MOCO-B/16 224px: 35.3% → 37.5% (+2.2pp)
SupViT-H/14 224px: 35.5% → 37.7% (+2.2pp)

The improvements are largest in General VQA and Vision-Centric VQA, with OCR & Chart VQA showing modest gains for encoders not originally trained with text supervision (e.g., DINOv2 improves from 17.6% to only 18.7% on OCR & Chart—the OCR capability is largely determined by pretraining). This demonstrates that PIVOT is effective regardless of the original pretraining objective, including purely vision-only self-supervised methods (MAE, MOCO, DINOv2) and supervised classification (SupViT).

Model ensemble with PIVOT (Table 1, bottom section). Combining SigLIP1-So/14+PIVOT with ConvNeXt-XXL achieves 53.6% average, compared to the standard SigLIP1-So/14 + ConvNeXt-XXL ensemble at 51.4% (+2.2pp) and SigLIP1-So/14 + DINOv2-g/14 at 49.4%. Notably, SigLIP1-So/14+PIVOT alone (53.2%) already outperforms both baseline ensembles, despite using only 400M parameters vs. 700M–1.25B.

Effect of larger Stage 3 data (Table C). When PIVOT-enhanced SigLIP2-So/16 is evaluated with LLaVA-OV-3M instead of Cambrian-737K during Stage 3, the advantage over the baseline persists: 59.2% vs. 56.9% (+2.3pp) with Qwen2.5-1.5B, and 50.6% vs. 49.0% (+1.6pp) with Qwen2.5-0.5B. The gap is somewhat smaller than with Cambrian-737K (+3.2pp for 1.5B), suggesting that more Stage 3 data partially compensates for weaker initial visual representations, but PIVOT's benefit remains substantial.

Effect of full-parameter Stage 3 training (Table D). When all parameters—including the vision encoder—are updated during Stage 3 (departing from the frozen-encoder evaluation protocol), the PIVOT-enhanced encoder still provides an advantage: +0.7pp with Qwen2.5-1.5B (55.2% vs. 54.5%) and +0.9pp with Qwen2.5-0.5B (46.0% vs. 45.1%). The smaller gap compared to frozen-encoder evaluation confirms that some of PIVOT's benefit can be recovered through additional training, but the persistent advantage demonstrates that PIVOT representations serve as better initialization points even when subsequent training is allowed to modify them.

Ablation Studies and Robustness Checks

Training data scale for MLLM benchmarks (Figure 4, Table H, Appendix Figure B): Sweeping Stage 2 data from 3K to 40K samples shows DPO achieves near-peak performance with only 3K–5K samples (60.4% at 3K, 60.6% at 5K, 61.0% at 20K), while SFT improves gradually throughout the range (56.4% → 57.6% → 59.1% → 59.5%). The DPO-SFT gap is largest at small data scales (+4.0pp at 3K) and narrows but never closes at 40K (+1.8pp). Per-category breakdown (Appendix Figure B) reveals that on Knowledge VQA, the gap shrinks from +4.2pp at 3K to +0.7pp at 10K, suggesting SFT can largely match DPO on knowledge-intensive tasks given sufficient data, but DPO's advantage on OCR & Chart and Vision-Centric VQA remains robust (>+2pp even at 40K).

Effect of distribution shift in post-training data (Appendix Figure A): When 0%, 50%, or 80% of Stage 2 data consists of samples with out-of-distribution characteristics (longer responses, special tokens), DPO maintains robust performance (62.2% → 62.5% → 62.2%) while SFT degrades substantially (62.2% → 57.5% → 53.2%). The degradation is most pronounced in Vision-Centric VQA (17.9pp gap at 80% shift). This robustness check demonstrates that DPO's advantage is not an artifact of the specific MPO data distribution and suggests DPO-trained models are less prone to catastrophic forgetting or overfitting to superficial dataset characteristics.

PIVOT projector reuse (Table B): When integrating PIVOT-enhanced encoders into new MLLMs during Stage 3, three projector configurations are tested: discarding the PIVOT-trained projector entirely (0 reused layers), freezing and reusing the first linear layer with one new trainable layer appended (1+1, total 2 layers), and freezing both PIVOT-trained layers with two new layers appended (2+2, total 4 layers). The 1+1 configuration achieves the best performance: 54.6% average with Qwen2.5-1.5B vs. 52.4% for 0 reused layers and 54.3% for 2+2. With Qwen2.5-0.5B, the 1+1 configuration reaches 45.2% vs. 42.9% for 0 reused layers. The 2+2 underperformance relative to 1+1 suggests that deeper projector architectures are not inherently better and that the intermediate representations from the PIVOT-trained projector's first layer provide a beneficial initialization that a single additional layer can effectively adapt.

Stage 3 training data size for PIVOT evaluation (Table C): When PIVOT-enhanced encoders are evaluated with larger Stage 3 data (LLaVA-OV-3M vs. Cambrian-737K), the advantage over the baseline encoder persists but narrows. SigLIP2-So/16+PIVOT with Cambrian-737K achieves 55.6% vs. 52.4% baseline (+3.2pp); with LLaVA-OV-3M, 59.2% vs. 56.9% (+2.3pp). The narrowing from +3.2pp to +2.3pp is consistent with the intuition that more downstream training data can partially compensate for suboptimal initial representations, but the persistent +2.3pp gap indicates that even 3M instruction-tuning samples are insufficient to fully close the representational quality gap.

Full-parameter vs. frozen-encoder Stage 3 training (Table D): This ablation tests whether PIVOT's benefits survive when the vision encoder is allowed to be updated during downstream MLLM training (departing from the controlled frozen-encoder protocol). The PIVOT-enhanced encoder's advantage shrinks from +3.2pp (frozen, Table 1) to +0.7pp (fully trained, Table D) with Qwen2.5-1.5B, confirming that some of PIVOT's benefit can be recovered through additional training. However, the persistent +0.7pp gap indicates PIVOT provides genuinely better initialization. With Qwen2.5-0.5B, the gap is +0.9pp (46.0% vs. 45.1%). Full-parameter training also improves the baseline substantially (52.4% → 54.5% with 1.5B), demonstrating that the standard Cambrian-737K frozen-encoder evaluation protocol, while useful for isolating encoder quality, underestimates achievable performance when all parameters can adapt.

DPO vs. SFT for PIVOT (Table 1, top section): For both SigLIP2-So/16 and SigLIP2-g/16, the PIVOT (DPO) variant outperforms the SFT-enhanced variant by +1.0pp and +1.3pp respectively when used as frozen encoders with Qwen2.5-1.5B. This demonstrates that the representational benefits of DPO observed in Section 4 (ImageNet, segmentation, gradient localization) translate to improved downstream MLLM performance, and that SFT-based enhancement—which resembles the language alignment stage in Perception Encoder (Bolya et al., 2025)—is strictly inferior to PIVOT for building MLLM-ready vision encoders.

Gradient visualization with global vs. local queries (Appendix Figure E, bottom rows): For queries that ask for broad image descriptions ("Please describe this photo in detail," "What are the main objects or subjects in the image?"), both DPO and SFT produce similarly diffuse gradient patterns across the entire image—in contrast to the localized DPO gradients observed for specific visual queries. This ablation supports the mechanistic hypothesis that DPO's localization benefit depends on the preference data containing visually discriminative contrasts: when the chosen and rejected responses differ only in global descriptive details (which objects are mentioned, general scene characterization), the contrastive signal does not provide spatial localization pressure, and DPO behaves similarly to SFT.

Representational alignment across LLM scales and multiple reference models (Figure 10): The alignment measurement is robust to the choice of reference LLM: vision encoders trained with larger LLMs show higher alignment scores consistently across all five reference models. The gap between DPO and SFT is visible across reference models, though the absolute alignment values vary. The use of three random seeds for averaging addresses the small dataset size (1,024 examples), though the paper does not report variance across seeds.

Critical Assessment

Does the Evidence Support the Paper's Central Claims?

Claim 1: RL (DPO) produces stronger and precisely localized visual representations compared to SFT. This claim is supported by converging evidence from multiple independent measurements: DPO-trained encoders achieve higher ImageNet Top-1 accuracy (+1.83pp to +1.96pp, Figure 6), better segmentation patch-level recall (+0.55pp to +1.08pp across six encoders, Figure 8), more question-localized Grad-CAM activations (Figure 7, Appendix Figure E), and stronger representational alignment with language models (Figure 10). The gradient visualization provides mechanistic evidence that the localization improvement arises from DPO's contrastive objective, and the segmentation probing provides quantitative evidence that this localization translates to measurable task performance. However, the evidence is correlational rather than causal—there is no experiment that directly manipulates gradient localization and measures its effect on downstream performance. A causal demonstration would require, for example, training with synthetically constructed preference pairs that vary only in the spatial specificity of the visual distinction between chosen and rejected responses, then measuring whether this produces corresponding differences in segmentation performance. Without such an experiment, the claimed mechanism (contrastive objective → localized gradients → fine-grained features) remains a well-supported hypothesis rather than an established causal chain.

Claim 2: DPO's MLLM advantage is concentrated on strongly vision-related tasks while SFT is competitive on knowledge-intensive benchmarks. This claim is strongly supported by the consistent pattern across all scaling experiments. With SigLIP2-L/16 and Qwen2.5-3B (Table F), DPO leads SFT by +4.2pp on OCR & Chart VQA and +2.4pp on Vision-Centric VQA, but only +0.5pp on Knowledge VQA (where the four Knowledge tasks average 52.0% vs. 51.5%). With the 7B LLM (Table G), the gaps are +3.1pp (OCR & Chart), +4.2pp (Vision-Centric), and +1.2pp (Knowledge). However, "Knowledge VQA" is an aggregate of four tasks, and the paper does not analyze whether the knowledge gap is uniformly small or whether some knowledge tasks show larger DPO benefits than others. The per-task breakdowns in Tables F–H show some variability: for example, MMMU-vision shows minimal differences (42.6% vs. 42.3% for So/16+3B), while AI2D sometimes favors DPO by larger margins (75.9% vs. 75.9%—tied in this case, but other configurations show gaps). The claim's reliability is anchored in the aggregate pattern rather than individual task-level statistical significance, which is not reported.

Claim 3: PIVOT-enhanced encoders outperform larger and next-generation vision models despite requiring <1% of pretraining compute. This claim is supported by Table 1's headline comparisons: SigLIP1-So/14+PIVOT (53.2%) > SigLIP2-So/16 (52.4%), and SigLIP2-So/16+PIVOT (55.6%) > SigLIP2-g/16 (53.9%). The compute comparison is based on hardware reports: PIVOT uses 8 H100 GPUs for 18 hours versus SigLIP2's reported use of "up to 2K TPUv5e chips" for full pretraining. However, the paper does not provide a direct FLOPs comparison. The "<1%" figure appears to be an order-of-magnitude estimate based on hardware count and training duration, but the actual ratio depends on utilization rates, TPUv5e vs. H100 FLOPs, and the number of training steps for SigLIP2 (which is not reported). The "# samples seen" column in Table 1 shows 30B + 0.003B for PIVOT-enhanced SigLIP1 vs. 40B for SigLIP2—the PIVOT samples are 0.01% of the pretraining samples, but sample count is not a direct compute metric since PIVOT samples involve both forward and backward passes through an LLM (Qwen2.5-1.5B) that adds compute not present in standard vision pretraining. A rigorous FLOPs-matched comparison would require accounting for the LLM's contribution to PIVOT's training cost. The claim should be understood as "PIVOT requires dramatically less compute than full vision pretraining" rather than a precisely quantified ratio.

Claim 4: Larger LLMs provide better visual supervision, improving the vision encoder's standalone quality. This is supported by Figure 6 (left), where SigLIP2-So/16 trained with a 7B LLM achieves +4.4pp higher ImageNet accuracy than when trained with a 0.5B LLM, and by Figure 10, where alignment scores increase monotonically with LLM size. However, the claim conflates two possible mechanisms: (1) larger LLMs genuinely provide more informative gradient signals per training step, and (2) larger LLMs may converge to better language generation during joint training, which indirectly improves the vision encoder through better task performance, not through intrinsically better gradients. The paper does not disentangle these. A key missing experiment is controlling for final MLLM performance: if two LLMs of different sizes are trained to equivalent MLLM accuracy (by varying training duration), do their vision encoders still show quality differences? If yes, the gradient quality mechanism is supported; if no, the effect is mediated by the LLM's own learning trajectory rather than its capacity per se.

Genuine Weaknesses and Missing Evidence

Single benchmark suite for MLLM evaluation. All MLLM results use the Cambrian benchmark suite with 16 tasks. While this is more diverse than most prior work (which often reports on 3–5 benchmarks), it represents one particular task distribution and difficulty profile. The paper does not evaluate on hallucination benchmarks (AMBAR, MME-hallucination), multi-image or video understanding tasks, or open-ended generation quality metrics. Since the central finding is that DPO improves fine-grained visual perception, hallucination benchmarks—where models must avoid generating plausible-sounding but visually unsupported details—would be a particularly strong test of the claim.

Single model family. All experiments use SigLIP2 vision encoders and Qwen2.5 LLMs (except the PIVOT encoder diversity experiments, which test other vision backbones but always with Qwen2.5-1.5B as the LLM head). The paper acknowledges this limitation in Appendix E.2: "a natural extension is to investigate if our findings generalize to other MLLM architectures, such as InternVL and Qwen-VL, or when using different LLM backbones like LLaMA and Gemma." Without such evidence, it is unknown whether the DPO advantage for vision is specific to the Qwen2.5 architecture or a general property of preference alignment.

The DPO hyperparameter $\beta$ is not reported. The temperature parameter controlling preference alignment strength is a critical hyperparameter for DPO—too small and the objective collapses to SFT, too large and it causes training instability. The paper states it "follows the default from the LLaVA-OneVision DPO script" (deep dive in Section 3.4) but does not report the actual value or any sensitivity analysis. If the DPO advantage depends on careful $\beta$ tuning, this would affect the generalizability of the findings.

No learning rate sweep for the controlled comparison. SFT and DPO use different optimal learning rates (1×10⁻⁵ vs. 1×10⁻⁶), which the paper justifies based on prior work and the doubled effective batch size of DPO. However, this means the controlled comparison is not perfectly controlled: if SFT were run with a lower learning rate more similar to DPO's, would the performance gap narrow? The paper could strengthen its causal claim by showing that the DPO advantage is robust to learning rate choices for both methods.

ImageNet probing uses a 50K subset, not full validation. This is a practical compromise but introduces sampling variance that is not quantified. The prototype-based linear classifier used for rapid screening is not the primary metric reported in figures, but its relationship to the Logistic Regression results is described as having "similar performance trends" without quantitative correlation.

Statistical significance is not reported for any result. No confidence intervals, standard errors, or hypothesis tests appear anywhere in the paper. The MLLM benchmark results are single-run evaluations. The segmentation results are averaged over 6 seeds, and the alignment results over 3 seeds, but variance is not reported. For the headline DPO-SFT comparisons (e.g., +4.2pp on OCR & Chart with L/16), the reader cannot assess whether this difference is statistically reliable or within the noise floor of the evaluation protocol.

The PIVOT evaluation design conflates training data and training objective. PIVOT always uses both Stage 1 (3M SFT samples) and Stage 2 (20K DPO samples). The paper does not include a "Stage 1 only" baseline (SFT on 3M + 20K, no DPO phase) that would isolate whether PIVOT's benefit comes from the additional 20K training samples, the DPO objective, or their combination. The SFT-enhanced encoders (Table 1, "+SFT" rows) partially address this by replacing DPO with SFT for the 20K Stage 2 samples, but these use the same total number of training steps. The key missing ablation is a "Stage 1 + no Stage 2" encoder—this would show whether the 3M SFT samples are sufficient and whether any Stage 2 training (regardless of objective) provides the observed benefit.

Missing Experiments That Would Strengthen the Paper

Causal intervention on gradient localization. Train a DPO variant where the chosen and rejected responses differ only in global attributes (e.g., scene category) vs. local attributes (e.g., object color), and measure whether the resulting vision encoder's localization ability differs as predicted by the gradient concentration hypothesis.

FLOPs-matched pretraining vs. PIVOT comparison. Estimate the actual FLOPs for SigLIP2 pretraining vs. PIVOT training (including the LLM's contribution), and determine the exact ratio rather than the order-of-magnitude estimate. This would make the "<1%" claim precise and falsifiable.

Cross-architecture replication. Train the same controlled SFT vs. DPO comparison using LLaMA-3 as the LLM backbone and a non-SigLIP vision encoder (e.g., DFN-CLIP, EVA-CLIP), and verify that DPO's vision-centric advantage replicates across architectures.

Hallucination benchmark evaluation. Since the paper's mechanism suggests DPO produces features that better distinguish visually present from visually absent details, evaluation on hallucination benchmarks (POPE, AMBER, MME-hallucination) would provide a direct test.

Oracle difficulty binning for vision tasks. The paper finds that DPO helps most on "strongly vision-related tasks"—but this is a post-hoc categorization based on task type, not a measured property of individual samples. The analysis would be stronger if individual questions were scored for "visual difficulty" (e.g., how many other models fail, or human judgment of required visual acuity) and the DPO-SFT gap were plotted as a function of this difficulty.

Conditional Nature of the Claims

The claims hold under the following conditions, which should be understood as boundary conditions rather than limitations:

The vision encoder is updated during post-training. The paper's core findings apply only when the vision encoder receives gradients from the LLM. If the encoder is frozen (as in some MLLM configurations), the representational differences cannot occur.
The preference data contains visually discriminative contrasts. The Grad-CAM results with global queries (Appendix Figure E) suggest that DPO's localization benefit requires the chosen and rejected responses to differ in ways that implicate specific image regions. Preference data where responses differ primarily in reasoning style, factual detail, or language quality may not produce the same visual representation improvements.
The base MLLM has non-trivial visual capability. The paper evaluates on a model series (SigLIP2 + Qwen2.5) that achieves strong baseline performance. Whether DPO's advantage extends to very weak vision encoders or very small LLMs (<0.5B) is untested.
The evaluation is limited to single-image, short-answer VQA. The 16 Cambrian benchmarks are all single-image tasks with relatively short expected answers. Open-ended generation, multi-image reasoning, video understanding, and embodied tasks may show different DPO-SFT tradeoffs not captured here.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted for in Efficiency Claims

The assumption or constraint. The compute-optimal framework the paper critiques and the adaptive allocation it advocates both depend on knowing something about the nature of each input before deciding how to process it. In this paper's context, the relevant "difficulty" is not a scalar problem-difficulty estimate (as in the reference example) but rather the degree to which a given task demands fine-grained visual discrimination—the very dimension along which DPO's advantage is concentrated. The paper's analysis establishes that DPO is superior specifically for "strongly vision-related tasks" (OCR & Chart VQA, Vision-Centric VQA) while providing minimal benefit on knowledge-intensive benchmarks. However, the paper provides no mechanism for identifying in advance which category a test-time query falls into. An MLLM deployed with PIVOT does not know whether the next user question requires reading small text in an image (where DPO-trained features would help) or answering a science question (where they might not matter). The paper does not address the cost or feasibility of making this determination, and the headline efficiency gains of PIVOT—surpassing larger models with less than 1% of pretraining compute—are computed without any mechanism for adaptive deployment.

The consequence. Without a practical way to route queries to PIVOT-enhanced encoders only when they provide benefit, a practitioner faces an uncomfortable choice: use PIVOT uniformly (paying the training cost even for knowledge tasks where it provides negligible improvement) or maintain multiple encoders and a routing mechanism (adding inference-time complexity and latency). Neither option is cost-free, and the paper provides no guidance on which regime is preferable. In the worst case, a practitioner might invest 18 GPU-hours on 8 H100s to produce a PIVOT-enhanced encoder, only to find that for their particular query distribution (which might skew toward knowledge-intensive tasks), the improvement is $+0.3$ percentage points rather than the $+4.2$ percentage points advertised for vision-centric tasks. The paper's benchmark suite, which averages across 16 diverse tasks, produces headline numbers that are a mixture of large vision-task gains and near-zero knowledge-task gains—a practitioner whose traffic is 90% knowledge VQA would experience substantially less benefit than the average suggests.

What evidence exists in the paper. The task-category decomposition in Figures 2 and 3 provides direct evidence of the heterogeneity. With SigLIP2-L/16 and Qwen2.5-3B (Table F), DPO outperforms SFT by $+4.2$ percentage points on OCR & Chart VQA and $+0.5$ percentage points on Knowledge VQA. The data scaling results in Appendix Figure B show that on Knowledge VQA, the DPO-SFT gap shrinks from $+4.2$ percentage points at 3K samples to $+0.7$ percentage points at 10K—meaning that for knowledge tasks specifically, simply collecting more SFT data nearly matches PIVOT's benefit at a potentially lower training complexity. The paper's own results thus demonstrate that the DPO advantage is highly task-dependent, but no mechanism for task-dependent deployment is proposed or evaluated.

Mitigation status. The paper does not address this limitation. Section 6 suggests future work on "whether novel dataset formats could be designed to better leverage DPO for learning stronger visual representations," but this concerns training data design, not deployment-time task identification. The absence of any discussion about how practitioners should decide when to use PIVOT versus standard encoders is a significant gap between the paper's experimental validation and its practical deployment.

The "<1% of Pretraining Compute" Claim Is an Order-of-Magnitude Estimate Without Rigorous Accounting

The assumption or constraint. The paper's most attention-grabbing quantitative claim is that PIVOT requires "less than 1% of the computational cost of standard vision pretraining" (Abstract, Section 5). This comparison is based on hardware reports: PIVOT uses 8 H100 GPUs for 18 hours, while SigLIP2 training used "up to 2K TPUv5e chips" (Section 5). The "# samples seen" column in Table 1 reports 30B + 0.003B for PIVOT-enhanced SigLIP1 versus 40B for SigLIP2—the PIVOT samples are approximately 0.01% of the pretraining samples. However, neither hardware count nor sample count is a direct measure of total FLOPs, and the paper does not perform an actual FLOPs-matched comparison.

The consequence. The "<1%" figure is likely correct in order-of-magnitude terms—PIVOT clearly uses dramatically less compute than training SigLIP2 from scratch—but the lack of rigor has two practical consequences. First, the number cannot be replicated or verified without access to SigLIP2's training configuration (batch size, number of training steps, TPUv5e utilization rates), which is not publicly reported in sufficient detail. Second, the comparison ignores an important cost asymmetry: PIVOT training involves forward and backward passes through a Qwen2.5-1.5B language model (approximately 1.5 billion parameters) in addition to the vision encoder. The LLM's contribution to the per-sample FLOPs during PIVOT training is substantial—the 1.5B LLM has roughly 3.75× more parameters than the 400M SigLIP2-So/16 encoder, and the self-attention operations in the LLM scale quadratically with sequence length. A rigorous FLOPs comparison would need to account for the LLM's contribution to PIVOT's training cost, which the "<1%" figure implicitly absorbs into the hardware-hours estimate but does not decompose. A practitioner trying to budget for PIVOT training needs to know not just the hardware requirement but also the sensitivity to LLM size—using a larger LLM head might improve the resulting encoder (per Finding 5) but at substantially higher training cost, a tradeoff the paper does not quantify.

What evidence exists in the paper. Table 1 reports "# samples seen" following the convention of Cherti et al. (2023) and Zhai et al. (2023), which counts training examples rather than FLOPs. Section 5 states "just 18 hours of training on 8 H100 GPUs" and contrasts this with SigLIP2's "up to 2K TPUv5e chips." Appendix F.6 describes the PIVOT training configuration but does not decompose the LLM's computational contribution from the encoder's. The paper does not report FLOPs for any training stage.

Mitigation status. The paper is transparent that the comparison is based on hardware and sample counts rather than a FLOPs-matched analysis. The "<1%" figure is labeled as an estimate, and the paper never claims it is exact. However, the paper does not acknowledge the LLM cost asymmetry as a caveat to this estimate, and no ablation studies vary the LLM head size to measure how PIVOT's cost scales with LLM capacity. A practitioner reading the paper might reasonably infer that PIVOT's cost is independent of the LLM choice, which is not established.

All Results Are on a Single Benchmark Suite with a Single Model Family, and No Statistical Significance Is Reported

The assumption or constraint. The paper's empirical findings rest entirely on the Cambrian benchmark suite (16 tasks spanning four categories) evaluated on MLLMs built from the SigLIP2 vision encoder family paired with the Qwen2.5 LLM family. For the PIVOT encoder diversity experiments in Table 1, additional vision backbones are tested (CLIP, DINOv2, MAE, MOCO, SupViT), but always with Qwen2.5-1.5B as the LLM head during PIVOT training and as the downstream MLLM's language component. The paper acknowledges this limitation explicitly in Appendix E.2: "a natural extension is to investigate if our findings generalize to other MLLM architectures, such as InternVL and Qwen-VL, or when using different LLM backbones like LLaMA and Gemma." Critically, no confidence intervals, standard errors, or hypothesis tests are reported for any result in the paper.

The consequence. Two distinct risks arise. First, the generalizability risk: the paper's central finding—that DPO reshapes visual representations to be more fine-grained and localized—may depend on specific properties of the Qwen2.5 architecture (e.g., how its attention mechanism propagates gradients back to the vision encoder) or the SigLIP2 pretraining (e.g., the fact that SigLIP2 already produces relatively good patch-level features). If a different LLM backbone produces less informative gradient signals during joint training, the DPO-SFT gap might shrink or disappear. The PIVOT encoder diversity experiments partially address this by showing that DPO benefits encoders with radically different pretraining objectives, but always with Qwen2.5-1.5B as the gradient source. Second, the statistical reliability risk: with a test set of 500 questions (the MATH benchmark in the reference example), a $+4.2$ percentage point gap might or might not be statistically significant depending on variance. The Cambrian benchmark suite uses larger test sets (MME has several thousand examples; individual VQA benchmarks typically have 1,000–10,000 test instances), making point estimates more reliable, but without reported variance, a practitioner cannot assess whether the $+1.0$ to $+1.3$ percentage point DPO-SFT gaps in the PIVOT experiments (Table 1) are robust or within the noise floor of MLLM evaluation—which is known to be non-trivial due to prompt sensitivity, decoding temperature, and evaluation metric implementation details (the paper itself switches DocVQA evaluation from Cambrian's submission-based protocol to lmms-eval due to practical concerns, a change that could introduce score discrepancies).

What evidence exists in the paper. The benchmark list in Table E provides the task names and citations but not test set sizes or variance estimates. The per-task scores in Tables F–H are single-run point estimates without error bars. The paper uses cross-validation only for segmentation probing (6 random seeds) and representational alignment (3 random seeds), but not for MLLM evaluation. The PIVOT encoder diversity results (Table 1, middle section) demonstrate generality across vision architectures but, as noted, always use Qwen2.5-1.5B. The distribution-shift experiment (Appendix Figure A) uses only the Qwen2.5-3B + SigLIP2-So/16 configuration.

Mitigation status. The paper explicitly acknowledges the model family limitation as future work, but does not discuss the absence of statistical reporting or argue that the test set sizes make it unnecessary. The practical consequence is that a practitioner considering adopting PIVOT for a LLaMA-based or Gemma-based MLLM would need to conduct their own replication study to verify that the DPO vision benefits transfer—the paper provides suggestive but not definitive evidence.

Hard Visual Problems with Near-Zero Baseline Performance Are Unaddressed

The assumption or constraint. The paper's analysis reveals that DPO's benefit is concentrated on tasks where the base model already possesses non-trivial visual capability. This is visible most clearly in the OCR & Chart VQA results for encoders that were not pretrained with text supervision. In Table 1, DINOv2-g/14 (a vision-only self-supervised model with no text reading capability) achieves only 17.6% on OCR & Chart VQA in its original form. After PIVOT, it improves to just 18.7%—a $+1.1$ percentage point gain, compared to $+4.5$ percentage points for SigLIP1-So/14 on the same category. Similarly, MAE-H/14 improves from 17.3% to 18.2% on OCR & Chart ( $+0.9$ percentage points), and MOCO-B/16 from 17.1% to 17.6% ( $+0.5$ percentage points). The pattern is stark: PIVOT provides large gains on OCR & Chart only for encoders that were originally trained with image-language supervision (CLIP, SigLIP variants), where reading capability already exists and can be refined. For vision-only pretrained models, the OCR capability simply is not present and PIVOT cannot create it.

The consequence. PIVOT amplifies existing visual capabilities but does not create fundamentally new ones. This places a hard boundary on what PIVOT can achieve: if a vision encoder lacks the capacity to represent certain visual information (fine-grained text, small object attributes, spatial relationships requiring high-resolution features), PIVOT cannot add that capacity—it can only refine what is already there. A practitioner evaluating whether PIVOT is worth the training cost for their specific vision encoder should first assess the encoder's baseline performance on the target task category. For encoders with very low baseline scores (e.g., DINOv2 on OCR at 17.6%), the expected PIVOT gain is small relative to the absolute performance gap that would need to be closed. The paper's headline claims about PIVOT's effectiveness are implicitly conditioned on the encoder already possessing the relevant visual capabilities at a non-trivial level—they apply to strong encoders being made stronger, not to weak encoders being made competent.

What evidence exists in the paper. Table 1, middle section, directly demonstrates this limitation: the OCR & Chart VQA improvements for non-text-pretrained encoders are $+0.5$ to $+1.1$ percentage points, compared to $+2.7$ to $+4.5$ percentage points for language-supervised encoders. The paper does not explicitly analyze or discuss this pattern, but the numbers are present in the table. The gradient visualization results (Figure 7) provide a mechanistic explanation: DPO concentrates gradients on question-relevant image regions, but if the vision encoder's architecture or pretraining prevents it from producing discriminative features at those regions (e.g., a DINOv2 encoder has never learned that small high-contrast patterns might be text characters), concentrated gradients cannot compensate for absent representational capacity.

Mitigation status. The paper does not acknowledge this capability boundary as a limitation. Section 6 states that "various future directions can be explored" but frames these as extensions (e.g., testing other RL algorithms) rather than acknowledging that PIVOT's benefits have sharp limits for certain encoder-task combinations. A practitioner considering PIVOT for a non-standard vision encoder or a specialized visual domain would need to discover this boundary through their own experimentation, as the paper provides no guidance on when the gains are likely to be small.

The Revision Model Training Fragility Is a Cautionary Tale Not Analyzed in the Paper's Context

The assumption or constraint. This limitation concerns a pattern that the reference example paper identifies and that is relevant here as a structural concern, though the current paper does not train revision models. The broader issue is that DPO-based training introduces a dependency on preference data quality that SFT does not share. While the paper's main experiments use the MPO dataset—a carefully curated human-annotated preference dataset—the distribution-shift experiment in Appendix Figure A demonstrates that DPO's advantage is sensitive to data characteristics. When the post-training data includes samples from a shifted distribution (longer responses, special tokens like <thinking> and <review>), SFT performance degrades substantially while DPO remains robust—but this robustness is demonstrated only for the specific types of distribution shift tested. The paper does not explore what happens when preference data contains systematic errors (e.g., rejected responses that are actually correct, chosen responses that contain subtle errors, or preference pairs where the visual distinction between chosen and rejected is ambiguous or non-existent).

The consequence. DPO's mechanism relies on the chosen response being genuinely better than the rejected response in ways that implicate visual information. If a practitioner constructs their own preference dataset and inadvertently includes pairs where the "chosen" response is preferred for non-visual reasons (e.g., better phrasing, more polite tone, or simply annotator bias), DPO's gradient signal to the vision encoder is effectively noise—it encourages the encoder to produce features that distinguish the two responses, but the distinction is not grounded in visual differences, so the resulting features may not improve vision. In the worst case, this could actively degrade visual representations by optimizing for spurious correlations between visual features and arbitrary language preferences. This is a risk that does not exist for SFT, which simply maximizes the probability of the chosen response without contrastive pressure. The paper's demonstration that DPO is robust to distribution shift in the form of the data (longer responses, special tokens) does not address robustness to distribution shift in the quality of the preference signal itself.

What evidence exists in the paper. The distribution-shift experiment (Appendix Figure A) is the closest the paper comes to testing DPO's robustness to data quality issues, but it manipulates only surface-level characteristics (response length, presence of special tokens), not the correctness of the preference labels. The Grad-CAM results with global queries (Appendix Figure E, bottom rows) show that when chosen and rejected responses do not differ in visually localized ways, DPO gradients become diffuse like SFT gradients—this provides mechanistic evidence that the DPO-SFT gap depends on the preference data containing visually discriminative contrasts, but the paper does not systematically vary the nature of the preference signal to measure this dependence.

Mitigation status. Not addressed. The paper uses a single preference dataset (MPO) for all Stage 2 experiments, and no ablation studies vary the quality, source, or visual specificity of the preference data. Section E.2 mentions future work on "whether novel dataset formats could be designed to better leverage DPO for learning stronger visual representations," which implicitly acknowledges that dataset design matters, but the paper does not provide practitioners with guidance on what properties a preference dataset should have to maximize vision encoder improvement. This is a significant gap for adoption, since practitioners deploying PIVOT would need to create or select preference data and currently have no empirical basis for doing so beyond replicating the MPO setup exactly.

Sequential Training Cost Is Reported, but Inference-Time Implications Are Not Analyzed

The assumption or constraint. The paper's practical contribution—PIVOT—is framed entirely around training-time improvements: a PIVOT-enhanced encoder costs less than 1% of pretraining compute and produces better downstream MLLM performance than larger, more expensively trained encoders. However, the paper does not discuss whether PIVOT training changes the inference-time properties of MLLMs built with the enhanced encoder. Specifically, the PIVOT training procedure involves full-parameter updates to the vision encoder during Stage 1 (3M samples of SFT) and Stage 2 (20K samples of DPO). The resulting encoder produces features that are qualitatively different—more localized, more fine-grained—but the paper does not measure whether these features affect inference latency, memory consumption, or the number of visual tokens needed for effective MLLM operation.

The consequence. A practitioner deploying a PIVOT-enhanced MLLM needs to know whether the improved representations enable any inference-time optimizations (e.g., fewer visual tokens, smaller projector, lower-resolution inputs) or whether they impose any new costs (e.g., if the improved localization means the model is more sensitive to input resolution, or if the features require different normalization for the new LLM head). The paper's Stage 3 evaluation protocol (frozen encoder + trained projector and LLM) is designed to isolate the encoder's representational quality, but this protocol does not measure real-world deployment characteristics like tokens-per-second throughput or peak GPU memory during inference. Since the paper's core claim is that PIVOT enables smaller encoders to replace larger ones at lower cost, a complete cost analysis would need to include inference costs as well as training costs—especially since the "larger encoder" being replaced (e.g., SigLIP2-g/16 at 1B parameters vs. SigLIP2-So/16+PIVOT at 400M) would have different inference characteristics independent of visual token quality.

What evidence exists in the paper. The paper provides no inference-time measurements. Table 1 reports "# Params" for each encoder and the total ensemble parameter counts, from which a reader could infer that replacing SigLIP2-g/16 (1B params) with PIVOT-enhanced SigLIP2-So/16 (400M params) reduces the vision encoder's parameter count by 60%. However, encoder parameter count is only one component of inference cost—the projector dimensions, the sequence length of visual tokens, and the LLM's self-attention cost all contribute, and the paper does not report any of these for the PIVOT evaluation MLLMs. The number of visual tokens per image is determined by the vision encoder's patch size and image resolution, which are not changed by PIVOT (both So/16 and g/16 use 16×16 patches at 384px, producing the same number of tokens), so the inference cost reduction comes primarily from reduced encoder parameters and potentially reduced projector dimensions, neither of which is analyzed in cost terms.

Mitigation status. Not addressed. The paper's framing is entirely training-cost-focused, and the practical deployment implications of switching vision encoders are left to the reader to infer from the parameter counts in Table 1. For a paper whose central practical claim is about efficiency ("less than 1% of the computational cost"), the absence of inference cost analysis is a notable omission, especially since MLLM inference is typically the dominant cost in production deployments where models serve many queries.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes the role of the vision encoder in multimodal language models—from a passive sensor whose quality is determined at pretraining time, to an active, trainable component whose representations are shaped by the language-guided optimization it participates in during MLLM post-training. This is not an incremental contribution to vision encoder architecture design; it is a conceptual shift in where researchers should look when trying to improve MLLM visual capabilities. Before this work, the dominant assumption was that better MLLM vision came from scaling vision pretraining (larger models, more data, improved contrastive objectives like SigLIP2 over SigLIP1) or from architectural innovations (multiple encoders, higher resolution, different patch sizes). This paper demonstrates that the training objective used during MLLM post-training—SFT versus DPO—is itself a first-class design decision that produces qualitatively different visual representations, with effects measurable even when the LLM is completely removed and the encoder is evaluated on classic vision tasks like ImageNet classification and semantic segmentation.

The magnitude of this shift should not be overstated—this is not a new training algorithm or architecture, but rather a diagnostic discovery that reveals a previously invisible dimension of MLLM design. The paper shows that a 400M-parameter SigLIP2-So/16 encoder, after PIVOT enhancement with DPO, outperforms a 1B-parameter SigLIP2-g/16 (55.6% vs. 53.9% average VQA in Table 1), and that a previous-generation SigLIP1-So/14 with PIVOT surpasses the next-generation SigLIP2-So/16 (53.2% vs. 52.4%). These are not marginal gains—they represent a path to MLLM improvement that is orthogonal to the scaling paradigm the field has primarily pursued. The practical consequence is a redirecting of research attention: improving the post-training objective is easier, cheaper, and potentially more impactful than scaling pretraining further, at least for vision encoders that already possess non-trivial base capabilities.

The paper also resolves a latent contradiction in the MLLM literature about where RL's benefits manifest. Prior work (Yu et al., 2024; Wang et al., 2024b; Sun et al., 2024a) uniformly attributed DPO's MLLM improvements to better language generation or alignment in the LLM component. The possibility that DPO was also improving visual perception was never investigated—not because evidence contradicted it, but because no one had designed an experiment that could detect it. By surgically extracting vision encoders from MLLMs and evaluating them on classic vision benchmarks (ImageNet linear probe, ADE20K segmentation), this paper provides the first direct evidence that DPO does not just produce better answers—it produces better seeing. The Grad-CAM visualizations in Figure 7 provide mechanistic evidence for why: DPO's contrastive objective concentrates gradient signals on question-relevant image regions, producing precisely localized features, while SFT's maximum-likelihood objective produces diffuse gradients that improve global features but not fine-grained spatial discrimination.

This creates a new taxonomy of training objective effects: SFT improves global, category-level visual features; DPO improves fine-grained, spatially localized visual features. This taxonomy explains the otherwise puzzling pattern in the benchmark results (DPO dramatically outperforms SFT on OCR & Chart VQA and Vision-Centric VQA, but provides minimal advantage on Knowledge VQA) and provides a principled basis for choosing between training objectives based on the visual demands of the target application. It also predicts that preference dataset design is now a visual representation design problem, not just a language alignment problem—the specific visual distinctions present in the chosen vs. rejected responses will determine which visual capabilities the encoder develops. This reframing opens an entirely new dimension of dataset engineering that was invisible under the LLM-centric assumption.

Finally, the paper identifies a teacher-student dynamic between the LLM and the vision encoder that reverses the standard information-flow model. The conventional view treats the LLM as a consumer of visual features; this paper shows that the LLM is also a teacher, with larger LLMs providing richer gradient signals that produce better standalone visual representations (Figure 6: +4.4pp ImageNet accuracy when training with a 7B vs. 0.5B LLM). This suggests that training a vision encoder with a temporarily larger LLM head, then deploying with a smaller one, could be a cost-effective strategy—an approach that has no analog in current MLLM training practice but is directly enabled by this paper's findings.

Follow-Up Research This Work Enables

Systematic preference dataset ablation for visual representation quality. The paper's gradient visualization results (Figure 7, Appendix Figure E) demonstrate that DPO's localization benefit depends on the chosen and rejected responses differing in visually discriminative ways—when queries are global ("Describe this photo in detail"), DPO and SFT produce similarly diffuse gradients. This suggests a directly testable hypothesis: the visual quality of DPO-trained encoders is a function of the visual specificity of the preference pairs. A follow-up study could construct multiple preference datasets from the same image-query pairs but with systematically varied relationships between chosen and rejected responses: (a) responses that differ only in object attributes (color, material, count), (b) responses that differ only in spatial relationships, (c) responses that differ only in factual knowledge, (d) responses that differ in language style but not content. Training DPO on each dataset and measuring both the resulting encoder's segmentation probing performance and the gradient localization patterns would establish a causal link between preference data design and visual representation quality. If dataset (a) produces the strongest localization and dataset (c) produces no improvement over SFT, this would provide a precise recipe for practitioners constructing preference data for vision improvement. The paper's existing results already suggest this will be the case—the OCR & Chart improvements for text-pretrained vs. non-text-pretrained encoders (Table 1: +4.5pp for SigLIP1 vs. +0.5pp for MOCO) indicate that the preference signal can only refine capabilities the encoder already possesses in rudimentary form.

Cross-architecture replication with LLaMA and InternVL backbones. The paper's findings are demonstrated exclusively on the Qwen2.5 LLM family (with SigLIP2 or diverse vision backbones). A critical replication study would reproduce the controlled SFT vs. DPO comparison from Section 3 using LLaMA-3 (8B) as the LLM backbone and a non-SigLIP vision encoder such as DFN-CLIP or EVA-CLIP, following the same two-stage training protocol (3.2M Stage 1 samples, 20K MPO Stage 2 samples) and the same vision encoder extraction and probing pipeline (ImageNet linear probe, ADE20K segmentation, Grad-CAM). The key question is whether the DPO vision benefit is specific to the Qwen2.5 architecture's gradient propagation characteristics—for instance, Qwen2.5 might have architectural properties (attention patterns, layer normalization placement, residual stream width) that make its gradients particularly informative for vision encoder training, while LLaMA-3's different architecture might produce weaker or qualitatively different gradient signals. If the DPO-SFT gap replicates in magnitude and task-dependence, the finding is established as a general property of preference alignment rather than a Qwen-specific phenomenon. If it does not replicate, the finding becomes architecture-conditional, and the practical recommendation becomes "use Qwen2.5 as the training head even if deploying with a different LLM"—which is a more nuanced but still actionable prescription.

Hallucination benchmark evaluation of PIVOT-enhanced MLLMs. The paper's mechanistic claim—that DPO produces features which better distinguish visually present from visually absent details—makes a direct prediction about hallucination behavior: MLLMs built with PIVOT-enhanced encoders should hallucinate fewer visual details than those built with standard or SFT-enhanced encoders, because the vision encoder produces features that more precisely encode what is actually in the image versus what is semantically plausible. This prediction is testable on standard hallucination benchmarks (POPE, AMBER, MME-hallucination, CHAIR for image captioning) using the same Stage 3 evaluation protocol from Section 5 (frozen PIVOT encoder + trained Qwen2.5-1.5B). A strong positive result—e.g., PIVOT reducing hallucination rates by 20–30% relative to the baseline encoder—would substantially strengthen the paper's practical value proposition, since hallucination reduction is a major deployment concern for MLLMs. A null result would suggest that DPO's localization improvements are limited to discriminative tasks (VQA) and do not transfer to generative tasks (captioning), which would refine our understanding of what "better visual representations" actually means for different downstream use cases.

PIVOT with PPO and GRPO: do other RL algorithms also improve vision? The paper focuses on DPO as the representative RL method, following its dominance in recent MLLM work (Table A). However, PPO and GRPO are increasingly used in the LLM community and have different optimization characteristics—PPO uses an explicit reward model, while GRPO uses group-wise relative rankings without a reference model. A systematic comparison of DPO, PPO, and GRPO applied to the same preference data (MPO 20K) using the same vision encoder probing protocol would establish whether the vision improvement is a general property of preference-based optimization or specific to DPO's implicit reward formulation. The prediction from the paper's gradient concentration mechanism is that any objective which forces the model to discriminate chosen from rejected responses based on visual content should produce localization benefits—PPO with a well-trained reward model should show similar Grad-CAM patterns to DPO, while GRPO (which compares multiple candidate responses within a group) might produce even stronger localization if the candidate responses differ in visually diverse ways. If PPO and GRPO show similar or stronger vision improvements, the practical recommendation broadens: any preference-based post-training improves visual representations, and practitioners can choose the RL algorithm based on non-vision considerations (training stability, compute efficiency) without sacrificing this benefit.

PIVOT for domain-specific visual capabilities. The paper demonstrates PIVOT on general-purpose vision encoders evaluated on broad VQA benchmarks. A natural extension is to test whether PIVOT can instill domain-specific visual capabilities that were absent from the original pretraining. For example: medical image understanding (train PIVOT on radiology preference pairs where chosen and rejected responses differ in pathology detection), satellite image analysis (preference pairs requiring discrimination of land-use categories or infrastructure types), or fine-grained species identification (preference pairs distinguishing visually similar bird or plant species). The key question is whether PIVOT can create new visual competencies (not just refine existing ones) when the preference data is sufficiently targeted. The paper's negative result with OCR for non-text-pretrained encoders (DINOv2 improving only +1.1pp on OCR & Chart VQA after PIVOT, vs. +4.5pp for SigLIP1) suggests a boundary condition: PIVOT cannot create capabilities that require fundamentally different feature types than the encoder learned during pretraining. Domain-specific experiments would map this boundary more precisely—does a general-purpose CLIP encoder improve on medical image tasks after PIVOT with medical preference data, or is the gap between natural images and radiology too large for preference alignment to bridge?

FLOPs-matched PIVOT vs. vision pretraining scaling with rigorous compute accounting. The paper's "<1% of pretraining compute" claim is an order-of-magnitude estimate based on hardware count and sample count, not a precise FLOPs comparison. A rigorous follow-up would: (a) estimate the total FLOPs for SigLIP2 pretraining using reported or inferred batch sizes, training steps, and model sizes; (b) estimate the total FLOPs for PIVOT training including both the vision encoder and the Qwen2.5-1.5B LLM's forward and backward passes; (c) produce a FLOPs-matched scaling curve that compares PIVOT-enhanced encoders against vision encoders pretrained with increasing compute budgets. The prediction from the paper's results is that PIVOT will lie far above the pretraining scaling curve—achieving accuracy that would require 100× or more pretraining FLOPs to match—but the exact multiplier is unknown. This analysis would also quantify the LLM's contribution to PIVOT's training cost, enabling a cost-benefit analysis of using larger vs. smaller LLM heads during PIVOT training. If the LLM contributes, say, 60% of PIVOT's total FLOPs, then the cost of upgrading from a 1.5B to a 3B LLM head (which should improve the resulting encoder per Finding 5) becomes a concrete engineering tradeoff rather than an open question.

Practical Applications and Downstream Use Cases

Cost-efficient MLLM deployment with smaller vision encoders. The most directly actionable finding is that a PIVOT-enhanced 400M-parameter vision encoder (SigLIP2-So/16) can replace a 1B-parameter encoder (SigLIP2-g/16) without sacrificing MLLM performance—in fact, improving it from 53.9% to 55.6% average VQA (Table 1). For production deployments where vision encoder inference cost is a meaningful fraction of total latency or memory consumption, this 60% reduction in vision encoder parameters translates directly to faster per-query inference and lower GPU memory requirements, without any change to the MLLM architecture or the LLM component. Since PIVOT training costs only 18 hours on 8 H100 GPUs (compared to the thousands of GPU-days required to pretrain a larger encoder), the training investment pays for itself rapidly in inference savings for high-throughput deployments. A team currently using SigLIP2-g/16 in production could switch to PIVOT-enhanced SigLIP2-So/16 and achieve both better accuracy and lower inference cost—a rare win-win in deployment engineering.

Upgrading legacy MLLMs without retraining the full model. The PIVOT evaluation protocol in Section 5 demonstrates that a PIVOT-enhanced vision encoder can be paired with a completely new LLM and projector, undergoing only lightweight alignment training (projector-only pretraining on 558K samples + instruction finetuning of projector and LLM on Cambrian-737K), while keeping the vision encoder frozen. This means that organizations with existing MLLM deployments built around a particular LLM (e.g., LLaMA-3-8B fine-tuned on proprietary instruction data) can upgrade their vision encoder to a PIVOT-enhanced version without retraining the LLM or discarding their instruction-tuning investment. The procedure would be: (1) train a PIVOT-enhanced version of their current vision encoder using a Qwen2.5-1.5B head (since the LLM during PIVOT training is discarded anyway); (2) attach the frozen PIVOT encoder to their existing LLM via a new projector; (3) train only the projector on a small alignment dataset. The paper's results with full-parameter Stage 3 training (Table D) suggest that even if the LLM is subsequently fine-tuned end-to-end with the new encoder, the PIVOT advantage persists (+0.7pp with Qwen2.5-1.5B), so the upgrade provides benefit regardless of the downstream training strategy.

Preference data curation as a visual capability development tool. The paper's finding that DPO's vision improvement depends on the contrastive signal from chosen vs. rejected responses (Figure 7, Appendix Figure E) implies that preference dataset design is now a lever for developing specific visual competencies. A team building an MLLM for document understanding, for instance, could curate preference pairs where the chosen and rejected responses differ specifically in OCR accuracy (correct vs. incorrect text reading), spatial layout understanding (correct vs. incorrect description of element positions), or table structure interpretation. Training with DPO on this curated data should specifically improve the vision encoder's features for document-related visual tasks, as measured by the same probing protocol used in the paper (e.g., segmentation-like probing on document layout datasets). This turns preference data from a generic "alignment" tool into a targeted "visual skill" training tool—a shift in how practitioners think about post-training data that is directly enabled by this paper's diagnostic framework.

Efficient multi-encoder MLLMs via PIVOT rather than ensemble scaling. The paper's ensemble results (Table 1, bottom section) show that SigLIP1-So/14+PIVOT alone (53.2% average) outperforms both the SigLIP1-So/14 + DINOv2-g/14 ensemble (49.4%, 700M parameters) and the SigLIP1-So/14 + ConvNeXt-XXL ensemble (51.4%, 1.25B parameters). This suggests that for practitioners currently using multi-encoder ensembles to improve visual representation quality (a common approach in Cambrian and related work), replacing multiple encoders with a single PIVOT-enhanced encoder can simultaneously improve performance and reduce parameter count. The practical workflow would be: take the primary encoder from the ensemble, apply PIVOT, and evaluate whether the enhanced single encoder matches or exceeds the ensemble's performance. If it does (as it did for SigLIP1-So/14+PIVOT vs. both ensembles tested), the ensemble's additional encoders can be removed entirely, simplifying the architecture and reducing inference cost. Even if the PIVOT-enhanced single encoder doesn't fully match the ensemble, a reduced ensemble (PIVOT-enhanced primary + one auxiliary encoder) might outperform the original multi-encoder setup at lower total parameter count—a hypothesis directly testable using the paper's evaluation protocol.