Let ViT Speak: Generative Language-Image Pre-training

🎯 Pitch

GenLIP shows that a Vision Transformer trained directly to generate captions—with no contrastive loss or text decoder—can match or beat CLIP-style encoders while using 5× less data. Its key trick is a gated attention mechanism that prevents the model from collapsing into an “attention sink” mode that would otherwise destroy discriminative visual representations.

1. Executive Summary

This paper introduces Generative Language-Image Pre-training (GenLIP), a minimalist generative pretraining framework that trains a Vision Transformer (ViT) to predict language tokens directly from visual tokens using a single autoregressive language modeling objective, eliminating contrastive batch construction and additional text decoders. Across frozen-feature MLLM evaluations on 14 multimodal benchmarks—including document understanding, general VQA, and captioning—using Qwen2.5-1.5B and Qwen2.5-7B backbones under the LLaVA-NeXT framework, GenLIP matches or surpasses strong baselines like SigLIP2 (pretrained on 40B pairs) while using only 8B pretraining samples from Recap-DataComp-1B, achieving gains of 2.0–3.7 points in overall average score across three model scales (L/16, So/16, g/16). The paper introduces a gated attention mechanism (a per-token sigmoid gate modulating attention outputs) to suppress attention sink behavior—where the first visual token absorbs disproportionate attention mass—which would otherwise degrade discriminative visual representations, and a two-stage native-aspect-ratio adaptation procedure that improves detail-sensitive OCR and chart understanding. GenLIP's advantages are most pronounced on Doc&OCR benchmarks, where it outperforms SigLIP2 by up to 5.9 points averaged over seven tasks, establishing that a simpler generative pretraining paradigm can serve as a data-efficient foundation for MLLM vision encoders, though its benefits remain concentrated on tasks within the base model's visual-textual alignment capability and have not yet been validated at pretraining scales beyond 8B samples.

2. Context and Motivation

The Core Problem: Vision Encoders Are Optimized for the Wrong Objective

The fundamental problem GenLIP addresses is a mismatch between how vision encoders are pretrained and how they are ultimately used in multimodal large language models (MLLMs). In a typical MLLM architecture, there are three components: a vision encoder that processes images, a connector that translates visual features into the language model's input space, and a large language model (LLM) that performs reasoning. The LLM is fundamentally an autoregressive next-token predictor — it operates by generating text sequentially, conditioning on all preceding context. Yet the vision encoders that feed into these LLMs are predominantly trained with contrastive objectives that have nothing to do with autoregressive generation.

This disconnect matters for two reasons. First, objective mismatch: contrastive learning teaches the vision encoder to produce representations where matched image-text pairs are close in embedding space and mismatched pairs are far apart. This is a discriminative task — the model learns to distinguish whether a caption belongs to an image. But when this encoder is plugged into an MLLM, the LLM is trying to generate text conditioned on visual features. The kinds of visual representations that excel at a binary match/no-match decision may not be optimal for autoregressive generation, where the model needs to extract structured, sequential, fine-grained visual information that supports multi-step reasoning.

Second, architectural mismatch: the contrastive paradigm typically uses a dual-encoder (two-tower) architecture — one tower for images, one for text — that encodes each modality independently and only interacts through a dot product in a shared embedding space. There is no cross-modal attention, no opportunity for visual features to be informed by textual context during encoding. In an MLLM, the vision encoder processes the image in complete isolation from the text query, producing a fixed set of visual tokens that the LLM must then interpret. A pretraining paradigm that already mixes visual and textual information during encoding — through early fusion and autoregressive prediction — might produce visual features that are more naturally consumable by the LLM.

The authors frame this tension explicitly in Section 1:

"contrastive pretraining introduces an objective mismatch with the generative nature of MLLMs: contrastive learning favors discriminative alignment, whereas MLLMs are ultimately optimized for next token prediction."

This is not merely an aesthetic concern. The vision encoder serves as the perceptual bottleneck for the entire MLLM system — every piece of visual information the LLM can reason about must first pass through this encoder. If the encoder discards fine-grained spatial information, text-like visual patterns, or compositional structure because the contrastive objective doesn't reward preserving them, no amount of LLM reasoning can recover what was lost.

Why This Problem Is Important: The Vision Encoder as Bottleneck

The paper argues that the vision encoder "fundamentally determines the upper bound of an MLLM's visual understanding capability" (Section 1). This is a strong claim that deserves unpacking.

In a modular MLLM (vision encoder + projector + LLM), the LLM never sees raw pixels. It sees only the visual tokens output by the encoder. If those tokens fail to capture a detail — say, the text "40378170" in an invoice, or the exact spatial arrangement of geometric shapes — the LLM has no mechanism to recover that information. The projector (typically a 2-layer MLP) performs a linear transformation that can re-weight or re-combine visual features, but cannot synthesize new visual information that the encoder didn't extract. The LLM can reason over what it receives, but it cannot see what isn't there.

This makes the vision encoder the critical gating component for downstream performance on detail-sensitive tasks. The paper's emphasis on document understanding and OCR benchmarks (ChartQA, OCRBench, DocVQA, TextVQA, InfoVQA) reflects exactly this concern — these tasks require the vision encoder to faithfully preserve character shapes, spatial layouts, and text structure, which are precisely the kinds of fine-grained visual details that might be lost in representations optimized only for global image-text alignment.

The practical importance is underscored by the scale at which these systems are deployed. MLLMs are used for document analysis, chart interpretation, UI understanding, and scientific figure comprehension — all tasks where missing a single digit or misreading a spatial relationship produces an incorrect answer. If a simpler pretraining paradigm can produce vision encoders that are better at these tasks while requiring less training data, that represents a significant practical advance in both cost and capability.

Prior Approaches and Where They Fall Short

The paper situates existing vision-language pretraining methods into three categories (Section 2), each with identifiable limitations.

Dual-Encoder Contrastive Pretraining (CLIP, SigLIP, SigLIP2)

This is the dominant paradigm, exemplified by CLIP and its successors. The architecture consists of two separate encoders — one for images, one for text — trained with a contrastive loss that pulls matched image-text pairs together in a shared embedding space and pushes unmatched pairs apart.

The strengths are well-established: CLIP-style models produce strong global representations that transfer well to classification and retrieval tasks, and the contrastive objective scales effectively with data. However, the paper identifies three specific shortcomings in the context of MLLM vision encoders:

Objective mismatch: As discussed above, the contrastive loss optimizes for discriminative alignment, not generative conditioning. The visual features are learned under the pressure to "know whether this text matches this image," not to "provide the information needed to generate this text from this image."
No deep cross-modal interaction: The dual-encoder design means that visual and textual features are computed independently and only interact through a cosine similarity at the end. There is no cross-attention between modalities during encoding. This limits the model's ability to learn fine-grained, text-conditioned visual representations — for instance, learning to attend to text regions in an image because the caption mentions them.
Global alignment bias: Contrastive objectives with image-level captions naturally encourage global image-text alignment. The paper notes that this "does not facilitate deep cross-modal interaction" (Section 2), meaning that local, patch-level alignment between visual regions and textual concepts is not explicitly learned. This is especially problematic for OCR and document understanding, where the correspondence between specific pixel regions and specific characters matters.

Recent improvements to contrastive pretraining — higher-quality captions, dense region-level supervision, language rewrites — address some of these issues but do not change the fundamental objective or architecture. They make the contrastive signal cleaner but don't resolve the generative mismatch.

Encoder-Decoder Generative Pretraining (CapPa, AIMv2, OpenVision2)

This family of methods directly addresses the objective mismatch by training with a generative language modeling loss. The typical architecture couples a vision encoder with a separate text decoder: the vision encoder processes the image into a set of features, and the text decoder autoregressively generates a caption conditioned on those features. The vision encoder is updated through gradients that flow backward through the text decoder.

Examples include:

CapPa: A ViT encoder feeds into a text decoder trained with a captioning loss.
AIMv2: Couples a vision encoder with a multimodal decoder that autoregressively generates raw image patches and text tokens.
OpenVision2: Stacks a text decoder on top of a ViT and pretrains with a captioning objective, using stronger captions from Recap-DataComp-1B v2.

While these methods close the objective gap, the paper argues they introduce redundant architectural complexity that limits efficiency:

Indirect optimization: The vision encoder is never directly trained to produce the correct text — it only receives gradients through the text decoder's cross-attention layers. This creates an information bottleneck: the vision encoder must learn to produce features that the text decoder can use, rather than features that directly encode the language-relevant information. The paper refers to this as "indirect optimization" that "can limit efficiency when the goal is to learn a scalable vision encoder for MLLMs."
Separate text decoder becomes dead weight: When the pretrained vision encoder is extracted and plugged into an MLLM, the text decoder is discarded. All the parameters and computation invested in training that decoder contribute nothing to the final vision encoder. This is fundamentally inefficient — you're training a whole language model just to provide a training signal for the vision encoder, and then throwing it away.
Ineffective cross-modal interaction: In the standard encoder-decoder setup, cross-modal interaction only happens in the decoder's cross-attention layers. The vision encoder itself processes images in isolation. The paper's GenLIP instead performs early fusion — visual and textual tokens are concatenated and processed jointly from the first transformer layer — allowing cross-modal interaction to happen throughout the entire depth of the network.

The paper also mentions hybrid approaches like CoCa and SigLIP2 that combine contrastive and generative objectives, but notes that these further increase complexity by requiring multiple towers and multiple loss functions.

Single-Transformer Pretraining (SuperClass, VL-BEiT, SAIL, NEO)

A smaller body of work has explored pretraining with a single transformer that processes both modalities. The paper acknowledges these as architecturally similar but distinguishes GenLIP on several grounds:

SuperClass uses a single transformer with token-level classification targets — still discriminative, not generative.
VL-BEiT and OneR unify modalities in a single tower but rely on multiple training objectives (masked language modeling, masked image modeling, image-text matching), increasing complexity.
SAIL and NEO use a single transformer with a language modeling objective, but focus on building native MLLMs — models where the vision encoder and LLM are the same transformer, not modular architectures where a pretrained vision encoder is plugged into a separate LLM.
SAIL specifically (which GenLIP is "architecturally close to") leverages pretrained LLMs as the backbone and trains on large-scale instruction-tuning data. GenLIP, by contrast, is pretrained from scratch on caption data, without relying on a pretrained LLM.

The key distinction the paper emphasizes (Section 2, Discussion) is:

"GenLIP is designed to pretrain a scalable vision encoder from scratch to better serve modular MLLMs, whereas SAIL focuses on building a native MLLM with a simplified architecture based on pretrained LLMs. This distinct goal also leads to different design choices."

The Attention Sink Problem: An Architectural Challenge Specific to Single-Tower Generative Pretraining

A major gap that becomes apparent only when you try the single-tower generative approach is the attention sink phenomenon. The paper observes (Section 3.2) that when a single transformer processes concatenated visual-then-textual sequences with a Prefix-LM attention mask (bidirectional over visual tokens, causal over text tokens), attention becomes "overly concentrated on the first token of the input sequence."

This is not a minor implementation detail — it fundamentally compromises the vision encoder's quality. The mechanism is:

Because the text tokens attend causally, they can see all visual tokens that precede them.
The model discovers a shortcut: compress useful visual information into a small number of early visual tokens (especially the first one, $v_0$ ) that all subsequent text tokens can easily access.
This produces efficient language prediction — the training loss improves — but at the cost of degrading spatial diversity in visual representations. The remaining visual tokens become information-poor, carrying little discriminative content.

The consequences are severe and empirically documented: the GenLIP variant without gated attention exhibits "loss spikes during pretraining," "attention distributions where the first token absorbs most of the attention mass," and critically, fails at discriminative tasks (Table 8: 76.2% ImageNet-1K top-1 accuracy for GenLIP-So/16 without gated attention vs. 84.3% with it). The paper notes that this makes the pretrained ViT "undesirable for our target usage as a vision encoder for MLLMs," since MLLMs need spatially distributed, information-rich visual features.

Prior works had documented attention sink in pure language models (StreamingLLM) and in pure vision transformers (DINOv2's register tokens), but its manifestation in the mixed-modality Prefix-LM setting — and its specific impact on visual representation quality — was not previously characterized. The paper's introduction of gated attention (a per-token sigmoid gate element-wise multiplied with attention outputs) as a solution is motivated by prior work on LLMs but adapted to this cross-modal scenario.

How GenLIP Positions Itself

The paper frames GenLIP not as another incremental improvement to existing methods, but as a philosophical departure toward minimalism. The core insight is articulated in the title: "Let ViT Speak" — instead of building complex scaffolding (separate text encoders, text decoders, multiple loss functions) to teach a vision encoder about language, just make the vision encoder directly generate language tokens.

This philosophy translates into three concrete design principles that distinguish GenLIP from all prior categories:

Single transformer, single objective: One transformer processes concatenated visual and textual tokens with a single autoregressive language modeling loss. No separate text encoder (like CLIP), no separate text decoder (like CapPa/AIMv2), no contrastive loss, no masked modeling objectives. This is simpler than all three prior categories.
Direct optimization: The language modeling loss is applied directly to the vision encoder's own output (through a language modeling head), not mediated through a separate decoder. The vision encoder learns to predict text tokens itself, rather than learning to produce features that help another module predict text tokens.
Early fusion with Prefix-LM attention: Visual and textual tokens are concatenated and processed jointly from the first transformer layer, with bidirectional attention over visual tokens and causal attention over text tokens. This allows cross-modal interaction throughout the entire network depth, unlike encoder-decoder architectures where cross-modal interaction is confined to the decoder's cross-attention layers.

The paper explicitly positions this as filling a gap that prior work had not addressed: a method that is generative (solving the objective mismatch), single-tower (solving the architectural overhead), and encoder-focused (optimizing for vision encoder quality in modular MLLMs, not for building a unified native MLLM). As stated in the introduction:

"To unleash the full potential of generative vision-language pretraining, we advocate for a minimalist design philosophy: remove unnecessary modules and train the vision backbone as directly as possible."

The claimed advantages — simplicity, scalability, and performance — flow from this design philosophy. Simplicity comes from the single-transformer, single-objective architecture. Scalability is demonstrated empirically through data and model scaling experiments. Performance is validated by matching or surpassing contrastive and encoder-decoder baselines with substantially less pretraining data, with particularly strong results on detail-sensitive OCR tasks that benefit from the early-fusion cross-modal interaction.

The paper does not claim that GenLIP is universally superior — it acknowledges limitations including the academic-scale validation setting, the unexplored scaling behavior beyond 8B samples, and the reliance on high-quality captions. But it argues that the results suggest a simpler, better-aligned pretraining paradigm can serve as a strong foundation for MLLM vision encoders, and that the findings "chart a more direct, efficient, and scalable course for developing powerful vision-language models."

3. Technical Approach

This is an empirical methods paper whose core idea is to pretrain a Vision Transformer (ViT) for use as a vision encoder in modular MLLMs by making it directly generate language tokens from visual tokens using a single autoregressive language modeling objective, eliminating all auxiliary modules (text encoders, text decoders) and auxiliary losses (contrastive, masked modeling) in favor of a minimalist single-transformer, single-objective design.

3.1 Reader Orientation

GenLIP is a vision encoder pretraining system that treats a ViT as if it were a language model: you feed it an image, and it directly outputs text describing that image. The key insight is that if you train a ViT this way — by making it speak — the visual representations it learns internally are naturally aligned with how downstream LLMs operate (autoregressive next-token prediction), avoiding the objective mismatch introduced by contrastive pretraining. The "shape" of the solution is a single transformer that processes a concatenated sequence of visual patches followed by text tokens, trained end-to-end with a standard causal language modeling loss applied only to the text portion, with careful architectural modifications (Prefix-LM attention, gated attention, multimodal positional encoding) to make this cross-modal autoregressive training both stable and effective for learning spatially rich visual features.

3.2 Big-Picture Architecture (Diagram in Words)

The GenLIP system has four major components:

Image Patch Embedding Layer — a standard convolutional patch embedding (as in ViTs) that partitions an input image into a sequence of non-overlapping patches and embeds each into a vector. This converts [H, W, 3] pixels into [M, d] patch tokens, where M is the number of patches.
Text Token Embedding Layer — an off-the-shelf tokenizer (Qwen3, vocabulary size 151,936) converts the caption text into subword token IDs, which are then embedded into vectors of the same dimension d as the visual patches. This produces [L, d] text token embeddings.
Unified Transformer (Prefix-LM with Gated Attention and MRoPE) — a standard transformer backbone (configurations: L/16 with 24 layers & 1024 dims; So/16 with 27 layers & 1152 dims; g/16 with 40 layers & 1536 dims) modified in three ways: (i) Prefix-LM attention masks allow bidirectional attention over visual tokens and causal attention over text tokens; (ii) multimodal rotary position encoding (MRoPE) injects position information into queries and keys, handling the multimodal sequence without learned absolute position embeddings; (iii) a gated attention mechanism applies a per-token sigmoid gate to attention outputs to suppress attention sink behavior. This component processes the concatenated [M visual tokens, L text tokens] sequence.
Language Modeling Head — a linear projection (plus LayerNorm) on top of the transformer output that predicts the next text token in the vocabulary. The loss is computed only on text positions.

Information flow: An image and its caption enter the system → the image is patched into M visual tokens via convolutional embedding → the caption is tokenized into L text tokens via Qwen3 tokenizer → both token sequences are concatenated into S = [v_0, ..., v_M, t_0, ..., t_L] → the unified transformer processes this entire sequence with Prefix-LM attention (visual tokens see each other bidirectionally; text tokens see all visual tokens and causally all preceding text tokens) → the LM head predicts each text token t_k conditioned on all visual tokens and preceding text tokens → the negative log-likelihood of the text sequence is backpropagated through the entire transformer, including the visual token positions, training the ViT to produce representations that support text generation.

When GenLIP is deployed as a vision encoder in an MLLM, the tokenizer, text embeddings, and LM head are discarded. Only the patch embedding layer and the transformer blocks are retained. The Prefix-LM attention degrades to standard full bidirectional attention since only visual tokens are present. The output features from the final LayerNorm layer are fed through a 2-layer MLP projector into the LLM.

3.3 Roadmap for the Deep Dive

First, the data format and model architecture (including Prefix-LM attention and MRoPE), since the entire framework is built around processing concatenated multimodal sequences through a single transformer — understanding the sequence structure and attention pattern is foundational.
Second, the training objective (the autoregressive language modeling loss applied only to text tokens), because this is the only learning signal and understanding exactly what it computes and why it suffices is the core intellectual move of the paper.
Third, the extraction procedure for using GenLIP as a frozen vision encoder, since the paper's primary evaluation is frozen-feature transfer and the interface between the pretrained model and the downstream MLLM matters.
Fourth, the gated attention mechanism, because it addresses a critical failure mode (attention sink) that would otherwise prevent GenLIP from learning useful visual representations, and its design involves specific interactions with the Prefix-LM attention pattern.
Fifth, the two-stage pretraining recipe (fixed-resolution stage 1, native-aspect-ratio stage 2), including exact hyperparameters, datasets, and regularization choices, since the scaling and performance claims depend on these details.
Sixth, a discussion distinguishing GenLIP from prior single-transformer and encoder-decoder generative works, clarifying what precisely is novel versus shared.

3.4 Detailed, Sentence-Based Technical Breakdown

Data Format: Concatenated Multimodal Sequences

All pretraining data consists of image-text pairs {(I_i, T_i)}_{i=1}^N, where each image I_i is paired with a corresponding caption T_i (Section 3.1, "Data Format"). Each image is partitioned into a sequence of non-overlapping patches using a convolutional patch embedding layer, exactly as in standard ViT architectures. Given an input image of resolution 224 × 224 with a patch size of 16 (yielding 14 × 14 = 196 patches for the fixed-resolution stage), the patch embedding layer produces a sequence of M visual token embeddings {v_0, v_1, ..., v_M}, each of dimension d (the model's hidden dimension).

The corresponding text T_i is tokenized into a sequence of L subword tokens {t_0, t_1, ..., t_L} using the Qwen3 tokenizer with a vocabulary size of 151,936. These token IDs are embedded into vectors of the same dimension d through a learned text embedding lookup table.

The crucial design choice is that these two sequences are concatenated into a single sequence, with visual tokens appearing first:

$S = [v_0, \ldots, v_M, t_0, \ldots, t_L]$

where S is the input sequence of length M + L + 1 (accounting for the fact that visual tokens are indexed from 0 to M, and text tokens from 0 to L). The visual prefix is deliberately placed before the text suffix so that text tokens can attend to all visual information through causal attention — the standard autoregressive setup where each text token conditions on all tokens that precede it in the sequence, which includes all visual tokens.

Why concatenation rather than cross-attention: In encoder-decoder architectures (like CapPa or AIMv2), the vision encoder and text decoder are separate modules, and cross-modal interaction happens only in the decoder's cross-attention layers. By concatenating tokens into a single sequence and processing them through a shared transformer, GenLIP enables cross-modal attention at every layer — a visual token can attend to a text token (in the bidirectional visual attention region), and vice versa — from the very first transformer block. This is "early fusion," and the paper's premise is that it produces richer cross-modal representations than late fusion through a separate decoder.

Why visual-before-text ordering: This ordering respects the autoregressive constraint — text tokens should condition on the image to predict the caption P(T|I). The reverse ordering (text then visual) would require predicting visual tokens from text, which is a different task (image generation, not captioning). The Prefix-LM attention mask (explained next) implements this asymmetry precisely.

Architecture: Unified Transformer with Prefix-LM Attention and MRoPE

The GenLIP architecture consists of a standard transformer encoder (not encoder-decoder) with three modifications, all designed to make a single transformer stack work effectively for concatenated multimodal sequences (Section 3.1, "Architecture").

Modification 1: Prefix-LM Attention Masks. Standard autoregressive language models use causal attention, where each token attends only to itself and all preceding tokens. Standard ViTs use bidirectional (full) attention, where every token attends to every other token. GenLIP uses a hybrid called Prefix-LM attention (introduced by Raffel et al., 2020, and used in models like T5): the visual tokens (the "prefix") attend bidirectionally among themselves, while the text tokens (the "suffix") attend causally to all preceding tokens (which includes all visual tokens and all preceding text tokens) but not to future text tokens.

Concretely, for the sequence S = [v_0, ..., v_M, t_0, ..., t_L]:

For any visual token v_i, the attention mask allows it to attend to all other visual tokens v_j for all j (full bidirectional within the visual prefix), but not to any text tokens (since text tokens come after, and attending to them from visual tokens would create information leakage in the autoregressive setup — the visual tokens shouldn't "see" the caption before generating it).
For any text token t_k, the attention mask allows it to attend to all visual tokens v_j for all j (the full image prefix) and to all preceding text tokens t_i for i ≤ k (causal within the text suffix), but not to future text tokens t_i for i > k.

This attention pattern directly instantiates the conditional probability model P(T|I) — the text tokens condition on the full image but generate sequentially, while the visual tokens process the image independently of the caption. The paper implements this using PyTorch's flex-attention, which allows per-sample attention masks with variable sequence lengths, essential because the concatenated sequences have different lengths depending on both the image (number of patches) and the caption (number of tokens).

Modification 2: Multimodal Rotary Position Encoding (MRoPE). Standard ViTs typically use learned absolute position embeddings — a lookup table of position vectors added to patch embeddings before the transformer. Standard LLMs typically use Rotary Position Encoding (RoPE), which encodes relative position by rotating query and key vectors in the attention computation. GenLIP adopts MRoPE (introduced in Qwen2-VL) and discards absolute position embeddings entirely.

MRoPE works by decomposing the position encoding into separate components for different modalities. In the GenLIP setting, this means injecting position information into the query (Q) and key (K) vectors of the attention computation with awareness of which tokens are visual versus textual. The paper states: "we use multimodal rotary position encoding (MRoPE) and discard the absolute position embeddings for image patches" (Section 3.1). The RoPE base frequency (RoPE Theta) is set to 10,000, as listed in Table 2.

Why MRoPE over absolute position embeddings: In a concatenated multimodal sequence, the position indices have different meanings for visual and textual tokens. Position 5 in the visual prefix might refer to a specific image patch location, while position 5 in the text suffix refers to the fifth word. MRoPE can encode these differently because it applies modality-conditioned rotations, whereas absolute position embeddings would assign the same vector to both positions, conflating the semantics. Additionally, MRoPE's relative encoding property means the model can generalize to different numbers of visual patches (different image resolutions) more naturally than absolute position embeddings, which is important for the diverse-resolution adaptation stage.

Modification 3: Gated Attention (explained in detail in its own subsection below).

Beyond these three modifications, the architecture is standard: each transformer block consists of multi-head self-attention followed by a feed-forward network (FFN) with residual connections and layer normalization. The configurations across three model scales are specified in Table 1:

Model	Parameters	Layers	Hidden Dims	Attention Heads	FFN Width
GenLIP-L	0.3B	24	1024	16	2816
GenLIP-So	0.4B	27	1152	16	3072
GenLIP-g	1.1B	40	1536	24	4096

After the final transformer block, a Layer Normalization (LN) layer normalizes the output representations, and then a language modeling (LM) head — a linear projection from the hidden dimension to the vocabulary size (151,936) — produces logits over the token vocabulary only for the text positions. No LM head predictions are made for visual token positions.

Training Objective: Autoregressive Language Modeling on Text Tokens Only

GenLIP uses a single objective: standard autoregressive language modeling applied exclusively to the textual portion of the concatenated sequence (Section 3.1, "Objective"). The model is trained to predict each text token t_k conditioned on all preceding visual tokens {v_j}_{j=0}^M and all preceding text tokens {t_i}_{i=0}^{k-1}.

The objective is the negative log-likelihood of the text sequence, summed over all text positions:

$\mathcal{L}_{\text{LM}} = -\sum_{k=0}^{L} \log P(t_k | \{v_j\}_{j=0}^{M}, \{t_i\}_{i=0}^{k-1}; \theta)$

where θ represents all model parameters (patch embedding, text embedding, transformer blocks, LN, and LM head), {v_j}_{j=0}^{M} are the M+1 visual token embeddings, {t_i}_{i=0}^{k-1} are the text tokens preceding position k, and P(t_k | ...) is the model's predicted probability for the ground-truth token t_k at position k, obtained by applying softmax to the LM head's logits at that position.

What it computes: For each text position k in the caption sequence, the model takes the full image prefix (all M+1 visual tokens) and the partial caption up to position k-1, processes this entire sequence through the transformer, produces a logit vector over the 151,936-token vocabulary at position k via the LM head, computes the log-probability of the correct token t_k, and accumulates the negative of this log-probability across all text positions. Intuitively, the model is rewarded for assigning high probability to the actual caption tokens and penalized for assigning high probability to incorrect tokens. The loss is averaged over the text positions and over the training batch.

Why this form: This is the canonical maximum-likelihood objective for autoregressive sequence modeling — it trains the model to approximate the true conditional distribution P(T|I). Important design choices:

Loss applied only to text tokens: Visual tokens contribute to the forward pass (they are processed by the transformer and provide the conditioning information for text prediction) and receive gradients through backpropagation (since the transformer processes them), but they have no explicit prediction targets. The only learning signal for visual representations comes indirectly through the text prediction loss: the visual features must encode information useful for predicting the caption. This is the "let ViT speak" philosophy — the vision encoder learns by being forced to produce features that a language model head can decode into text.
No contrastive loss, no masked modeling: Unlike CLIP (which adds a contrastive objective comparing image and text representations in a batch), AIMv2 (which adds pixel reconstruction), or VL-BEiT (which adds masked image modeling), GenLIP uses only this single autoregressive loss. The paper argues this is sufficient because the autoregressive objective already provides a strong cross-modal learning signal: to predict the next word in a detailed caption, the model must extract fine-grained visual information (object identities, attributes, spatial relationships, text in the image), which implicitly aligns visual and textual representations.
Full autoregressive, not masked: Unlike masked language modeling (which predicts randomly masked tokens given surrounding context), the autoregressive objective predicts each token left-to-right, matching the inference procedure of downstream LLMs exactly. This is deliberate — the paper wants the vision encoder's pretraining objective to match the LLM's operating mode as closely as possible.

Extracting GenLIP as a Frozen Vision Encoder for MLLMs

When GenLIP is deployed as a vision encoder in a modular MLLM (Section 3.1, "Using GenLIP as a Vision Encoder"), only the vision-relevant components are retained:

Retained components: The convolutional patch embedding layer and all transformer blocks with their gated attention mechanisms remain. The LayerNorm after the final transformer block remains.
Discarded components: The text tokenizer, the text token embedding lookup table, and the LM head are removed because there are no text inputs at inference time (the MLLM receives only an image; the text query goes directly to the LLM).
Attention pattern change: The Prefix-LM attention mask is degraded to standard full bidirectional attention. Since only visual tokens are present in the sequence, the "prefix" is the entire sequence, and there are no text tokens that need causal masking. Every visual token attends bidirectionally to every other visual token — this is identical to a standard ViT's attention pattern at inference time.
Feature extraction point: Vision features are extracted from the output of the LN layer following the last transformer block — not from an intermediate layer, and not from a special CLS token (GenLIP has no CLS token). The output is a set of M+1 patch-level feature vectors, each of dimension d.
Projection to LLM input space: These M+1 features are fed through a 2-layer MLP projector (standard in LLaVA-style MLLMs) that transforms them from the ViT's hidden dimension to the LLM's input embedding dimension. The MLP is trained during the instruction-tuning phase (along with, optionally, fine-tuning the LLM), while the vision encoder weights remain frozen.

The critical point is that the pretrained GenLIP model already produces visual features that are "language-aligned" because they were trained to support autoregressive text generation. The 2-layer MLP only needs to perform a dimension transformation and minor adaptation — the semantic alignment between visual features and the LLM's representational space is already baked into the pretrained weights.

Gated Attention: Mechanism, Motivation, and Impact

The gated attention mechanism is the paper's solution to the attention sink problem that arises when training a single transformer on concatenated visual-textual sequences with Prefix-LM attention (Section 3.2).

The attention sink phenomenon in GenLIP. Under the Prefix-LM attention pattern, text tokens attend causally, meaning the earliest tokens in the sequence are accessible to all subsequent tokens. The first visual token v_0 is special: it is accessible to every single text token through causal attention (since it's at position 0, before all text tokens). The model discovers a shortcut during training — compress globally useful visual information into v_0 (and potentially a few other early visual tokens) and let all text tokens attend primarily to this compact representation. This produces good language modeling loss (the text predictions are accurate) but catastrophically degrades the visual representations: the remaining visual tokens become information-poor, carrying little spatially localized content.

The consequences, as observed by the authors:

"Obvious loss spikes during pretraining": The training dynamics become unstable because the model oscillates between distributing information across patches (good for visual representation, harder for language modeling) and compressing into sink tokens (easier for language modeling, bad for visual representation).
"Attention distributions where the first token absorbs most of the attention mass": The attention weights from text tokens to visual tokens concentrate heavily on v_0, meaning text tokens extract almost all their visual information from a single patch, losing access to spatial details.
Failure at discriminative tasks: Table 8 shows that GenLIP-So/16 without gated attention achieves only 76.2% top-1 accuracy on ImageNet-1K linear probing, compared to 84.3% with gated attention — a drop of over 8 points. This demonstrates that the sink behavior destroys the spatial discriminability of visual features.
"Unstable scaling behavior": As model size increases, the sink phenomenon worsens, making larger models unexpectedly perform worse.

The gated attention mechanism. The solution is a per-token, input-dependent gate applied to the attention output in every transformer block. Given input hidden states X ∈ R^{n × d} where n is the sequence length and d is the hidden dimension, the standard attention computation produces A = Attn(X) (the multi-head self-attention output after concatenating heads and projecting). Gated attention computes:

$G = \sigma(X W_g + b_g)$

$\tilde{A} = G \odot A$

where σ(·) is the sigmoid function, W_g ∈ R^{d × d} and b_g ∈ R^{d} are learnable parameters (one set per transformer layer), and ⊙ denotes element-wise multiplication.

What it computes: For each token at each layer, an independent gate value between 0 and 1 is computed by passing the token's hidden state through a linear layer followed by sigmoid. This scalar gate is broadcast across the hidden dimension and multiplied element-wise with the attention output for that token. The gated attention output \tilde{A} then enters the standard residual pathway: X' = X + \tilde{A}, followed by the FFN block as usual.

Why this form: The gate acts as a learnable information flow regulator. For sink-prone tokens (like early visual tokens that would normally absorb disproportionate attention from text tokens), the model can learn to down-weight the attention output (gate value close to 0), preventing them from becoming information bottlenecks. For tokens that should carry rich spatial information, the model can learn to up-weight the attention output (gate value close to 1). The key properties:

Per-token granularity: Each token position has its own gate value, allowing the model to suppress sink behavior at specific positions (e.g., the first visual token) without affecting other positions.
Input-dependent: The gate is computed from the token's own hidden state, so it can adapt based on the token's content. A token that the model learns is becoming a sink can have its output gated down.
Sigmoid gating (not a simple scaling factor): The sigmoid constrains gate values to (0, 1), which matters for two reasons: (i) it prevents the gate from exploding or going negative, which would destabilize training; (ii) it allows the model to learn to completely block certain attention outputs (gate ≈ 0) while passing others through unchanged (gate ≈ 1).
Applied to attention output, not attention weights: The gate modulates the output of the attention computation, not the attention weights themselves. This means the sink behavior in the attention distribution can still exist (early tokens can still have high attention weights from text tokens), but the information carried by those high-attention positions is suppressed, forcing the model to extract information from other positions whose gated outputs are less suppressed. This is more effective than directly modifying attention weights because it allows the model to use attention patterns for routing information while separately controlling how much information each position contributes to the residual stream.

The paper credits prior work for inspiration: "Inspired by [54]" (citing Qiu et al., 2025, on gated attention for LLMs). However, the application to vision-language pretraining and specifically to the attention sink problem in Prefix-LM multimodal sequences is novel.

Empirical evidence for gated attention's effectiveness. Figure 3 visualizes the attention distribution of the first token in the input sequence, showing that "without gated attention, the first token absorbs most of the attention mass." Figure 6 plots data scaling curves comparing GenLIP with and without gated attention across 1.0B to 8.0B pretraining samples, showing "consistent advantages of gated attention across data scales" — the gap is especially large in the low-data regime (1.0B–2.0B), indicating gated attention improves data efficiency. Table 8 directly shows the discriminative quality difference: 84.3% ImageNet top-1 with gated attention vs. 76.2% without (So/16 scale). The paper also notes that gated attention "alleviates loss spikes, accelerates convergence, and stabilizes scaling behavior."

Two-Stage Pretraining: Fixed Resolution Then Diverse Resolution Adaptation

GenLIP's pretraining is split into two stages with distinct datasets, resolutions, and objectives (Section 3.3).

Stage 1: Fixed-Resolution Pretraining on Recap-DataComp-1B.

Dataset: Recap-DataComp-1B, a large-scale dataset of 1.0 billion unique image-text samples collected from the web and recaptioned using LLaMA-3. The recaptioning is significant: rather than using raw alt-text (which is often noisy, short, or irrelevant), each image is paired with a high-quality synthetic caption generated by a strong LLM. This provides the dense, descriptive language supervision that GenLIP needs to learn fine-grained visual-textual alignment.
Resolution: All images are processed at a fixed 224 × 224 resolution. With a patch size of 16, this yields 14 × 14 = 196 visual tokens per image.
Training scale: The model is trained for 8 billion samples total, corresponding to 8 epochs over the 1B-image dataset. The paper notes that this is a deliberate choice based on data scaling experiments (Figure 6) showing that performance improvements become flatter when scaling from 4B to 8B samples, particularly for VQA and captioning tasks.
Data preprocessing: Only "resize and crop operations without additional augmentations" — no RandAugment, color jittering, or other data augmentation commonly used in contrastive VLMs. The paper's reliance on the captioning objective (rather than contrastive matching) means that heavy augmentation (which could distort the visual information needed to predict detailed captions) is unnecessary and potentially harmful.

Stage 2: Diverse Resolution Adaptation on Long-Caption Data.

Datasets: Two public open-source caption datasets: the caption subset of Infinity-MM (Stage 1) (10 million samples) and BLIP3o-Long-Caption (27 million samples), totaling 37 million image-text samples. Both contain long, detailed captions and higher-resolution images than the web-crawled Recap-DataComp-1B. The long captions provide richer supervision for fine-grained visual details.
Resolution strategy: Unlike Stage 1's fixed 224×224, Stage 2 processes images at their native aspect ratios and resizes them so that the number of visual tokens falls within the range [16, 1024]. This is crucial: for an image with a 1:1 aspect ratio, the maximum resolution that fits 1024 tokens is roughly 1024 = 32 × 32 patches at patch size 16, i.e., 512 × 512 pixels. For an image with a 3:1 aspect ratio (e.g., a wide panorama or a document), the resolution might be, say, 960 × 320 to stay within the token budget while preserving the native shape. This variable-resolution, native-aspect-ratio processing teaches the model to handle the diverse image sizes and shapes it encounters in real MLLM applications.
Training scale: Only 1 epoch over the 37M samples, making this a lightweight adaptation stage rather than a full second pretraining. The rationale is that the foundational visual representations are already learned in Stage 1; this stage only adapts the model to higher resolutions, varied aspect ratios, and finer-grained captions.
Key differences from Stage 1 (explicitly listed in Section 3.3):
1. Global batch size reduced from 32K–48K (Stage 1) to 3.6K (Stage 2) "because the average sample length increases from 270 tokens to about 1200 tokens." With longer sequences, each GPU can fit fewer samples.
2. Peak learning rate reduced from 1e-3 to 1e-4. Lower learning rate in the fine-tuning stage prevents catastrophic forgetting of Stage 1 representations.
3. Images processed at native aspect ratios rather than square-cropped to 224×224.
All other training settings remain identical to Stage 1, including the optimizer (AdamW), momentum parameters (β_1 = 0.9, β_2 = 0.95), learning rate schedule (cosine decay to 1e-6), weight decay, gradient clipping (1.0), warmup ratio (0.007 for L/16 and So/16, 0.02 for g/16), and maximum packing length (16,384 tokens).

Pretraining Implementation Details and Hyperparameters

The paper provides detailed hyperparameters in Table 2 and implementation notes in Section 3.3 ("Pretraining Implementation").

Packing strategy: To maximize hardware utilization, samples of variable lengths are packed into long sequences with a maximum length of 16,384 tokens. Packing means that multiple image-text pairs are concatenated into a single training sequence, with careful handling of attention masks so that samples don't attend across packing boundaries. The attention implementation uses PyTorch's flex-attention, which allows variable sequence lengths and arbitrary per-sample attention masks within a batch. Specifically, the Prefix-LM attention mask is constructed per-sample: for a packed sequence containing multiple [image_patches, text] blocks, the mask ensures that within each block, visual tokens attend bidirectionally, text tokens attend causally to preceding tokens in the same block, and tokens from different blocks cannot attend to each other.

Optimization: The optimizer is PyTorch AdamW with β_1 = 0.9, β_2 = 0.95. The learning rate follows a cosine decay schedule from a peak (1e-3 in Stage 1, 1e-4 in Stage 2) to a minimum of 1e-6, with a linear warmup period (warmup ratio 0.007 for L/16 and So/16, 0.02 for g/16). Gradient clipping is set to 1.0 to stabilize training. The global batch size is 32K for L/16 and So/16, 48K for g/16 in Stage 1, and 3.6K in Stage 2 due to longer sequences.

Regularization: Two regularization techniques are applied:

Layer Scale: An initialization of 0.1, which is a learnable scalar multiplied with the output of each transformer sub-layer before the residual addition. This stabilizes training of deep transformers by initially down-weighting the contribution of each sub-layer, effectively making the model behave like a shallower network early in training.
Drop Path: A ratio of 0.1 for L/16 and So/16, 0.2 for g/16. Drop path randomly drops entire residual sub-layers (attention or FFN blocks) during training, which acts as structured dropout and is particularly important for training very deep models like GenLIP-g (40 layers). The higher drop path ratio for g/16 reflects its greater depth and higher risk of overfitting or training instability.

The paper notes that these regularization techniques are "mainly used to stabilize training and prevent divergence when training deeper models, but found less impact on the final GenLIP performance" — implying they are stability mechanisms, not primary drivers of downstream accuracy.

Image preprocessing: "We use only resize and crop operations without additional augmentations on Recap-DataComp-1B" (Section 3.3). This is a deliberate departure from contrastive VLP methods like CLIP or SigLIP, which typically use heavy data augmentation (random cropping, color jittering, RandAugment) to prevent the contrastive objective from exploiting low-level shortcuts. GenLIP's autoregressive objective — predicting a detailed caption from visual features — naturally discourages such shortcuts because the caption often describes specific visual details that would be destroyed by aggressive augmentation.

Key Design Distinctions from Prior Work

The paper explicitly contrasts GenLIP with three categories of prior methods in Section 3.4 ("Discussion").

Vs. encoder-decoder generative pretraining (CapPa, AIMv2, OpenVision2):

No separate text decoder: GenLIP's LM head is a single linear projection applied directly to the transformer's output, rather than a full transformer decoder stack. This eliminates the parameters and computation of a separate text decoder module. When the pretrained model is used as a vision encoder, only the ViT backbone is kept; in encoder-decoder methods, the text decoder (which was needed for training) is discarded.
No auxiliary objectives: AIMv2 additionally trains with "masked image modeling or pixel reconstruction" alongside the captioning loss. GenLIP uses only the language modeling objective.
Direct optimization: In encoder-decoder methods, the vision encoder receives gradients only through the text decoder's cross-attention layers. In GenLIP, the vision encoder directly produces token predictions — the LM head sits on top of the same transformer that processes visual tokens. This means gradients for text prediction flow directly through the visual token representations without an intermediate decoder.

Vs. single-transformer pretraining (SAIL, NEO, VL-BEiT):

Purpose: GenLIP pretrains a vision encoder from scratch for modular MLLMs, while SAIL and NEO build native MLLMs by leveraging pretrained LLMs. This difference drives different design choices: SAIL starts from a pretrained LLM and adds vision capabilities; GenLIP starts from scratch and trains a standalone vision encoder.
Training data: GenLIP uses only image-caption pairs, whereas SAIL and NEO use large-scale instruction-tuning data with complex multimodal prompts and responses.
Training from scratch: GenLIP does not rely on a pretrained LLM initialization, making it a true vision encoder pretraining method. SAIL's representations inherit from the pretrained LLM, which already encodes language knowledge.
Gated attention: GenLIP introduces gated attention specifically to address the attention sink problem in the Prefix-LM multimodal setting — an architectural innovation not present in SAIL or NEO.

Vs. VL-BEiT and other multi-objective single-tower methods:

Single objective: VL-BEiT uses a combination of masked language modeling, masked image modeling, and image-text matching losses. GenLIP uses only autoregressive language modeling. The paper argues that the single objective is sufficient because the autoregressive task inherently requires the model to learn both visual understanding (to condition text generation on image content) and cross-modal alignment (to map visual features to linguistically meaningful representations).

The overarching design philosophy is captured in the paper's statement: "Remove unnecessary modules and train the vision backbone as directly as possible." Every architectural decision — concatenation rather than cross-attention, direct LM head rather than text decoder, single objective rather than multi-task, no data augmentation, gated attention rather than register tokens — follows from this minimalist principle.

4. Key Insights and Innovations

Innovation 1: Generative Pretraining Without Generative Architecture

The paper's most intellectually distinctive move is a conceptual decoupling: it shows that generative pretraining — training a vision encoder to produce features useful for autoregressive text generation — does not require a generative architecture (i.e., an encoder-decoder setup with a separate text decoder). The field had implicitly conflated these two things. Prior generative VLP methods like CapPa, AIMv2, and OpenVision2 all adopted the encoder-decoder pattern: a vision encoder feeds into a separate autoregressive text decoder, and the vision encoder learns indirectly through gradients flowing backward through that decoder's cross-attention layers. This architecture was assumed necessary because language generation requires autoregressive decoding, and it seemed natural to place a dedicated autoregressive module on top of the vision encoder.

GenLIP's key reframing is that the autoregressive decoding capability can be a thin layer on top of the same transformer that processes visual tokens, not a separate network. The language modeling head — a single linear projection — sits directly on the unified transformer's output, and the autoregressive constraint is enforced through the attention mask (Prefix-LM), not through a separate decoder stack. This means the entire transformer learns to produce representations that are simultaneously good visual features and good conditioning context for language generation. There is no architectural separation between "vision processing" and "language conditioning" — they happen in the same layers.

This matters beyond implementation convenience. In encoder-decoder methods, the vision encoder's representations are optimized to be decodable by a separate language module — the encoder learns to produce features that the decoder can use, but the encoder itself has no direct language generation capability. In GenLIP, the vision encoder's representations are optimized to directly support token prediction, meaning the features themselves become "language-shaped." When these features are later fed into an MLLM's LLM (which is also an autoregressive next-token predictor), the alignment is more natural — the vision encoder was trained under the same generative pressure that the LLM applies. This is a representational alignment argument, not just an architectural simplification argument.

The evidence for this decoupling working is in the main results tables (Tables 3 and 4): GenLIP matches or surpasses SigLIP2 (a contrastive method) and OpenVision2 (an encoder-decoder generative method) across 14 multimodal benchmarks, despite using 8B pretraining samples versus 40B (SigLIP2) and 12.8B (OpenVision2). If the encoder-decoder architecture were necessary for generative pretraining to work, GenLIP should underperform. It doesn't. This is a fundamental finding: the generative objective matters more than the generative architecture.

Innovation 2: Attention Sink as the Critical Failure Mode of Single-Tower Multimodal Pretraining

The paper identifies and diagnoses a previously uncharacterized failure mode: when a single transformer processes concatenated visual-then-textual sequences with Prefix-LM attention, the model discovers a shortcut where it compresses globally useful visual information into the first few visual tokens (especially the first one, v_0), which are accessible to all subsequent text tokens through causal attention. The remaining visual tokens become information-poor. This attention sink phenomenon — known in pure language models and pure vision transformers — had not been documented in the mixed-modality Prefix-LM setting, and its specific impact on visual representation quality had not been measured.

What makes this a genuine diagnostic contribution rather than a minor implementation fix:

First, it explains why naive single-tower multimodal pretraining fails as a vision encoder pretraining method. The model achieves reasonable language modeling loss (it can predict captions) but produces visual features that are "undesirable for our target usage as a vision encoder for MLLMs" because the spatial diversity of representations collapses. Without this diagnosis, one might conclude that single-tower generative pretraining simply doesn't work for producing good vision encoders — and abandon the approach.

Second, it provides a precise mechanism: under Prefix-LM attention, text tokens attend causally and therefore have access to all preceding visual tokens. The earliest visual tokens become natural aggregation points because they are visible to every text token. The model exploits this structural property to simplify the language modeling task, at the expense of visual representation quality. This is not a generic "training instability" problem — it's a structural consequence of the interaction between the Prefix-LM attention pattern and the autoregressive objective in multimodal sequences.

Third, the quantifiable impact is stark: GenLIP-So/16 without gated attention achieves 76.2% ImageNet-1K linear probing accuracy versus 84.3% with gated attention (Table 8) — an 8-point drop that would make the vision encoder unusable for many discriminative transfer tasks. The attention distribution visualization in Figure 3 shows this is not a subtle effect; the first token absorbs overwhelming attention mass.

The solution (gated attention) is a per-token sigmoid gate that allows the model to learn to suppress the output of sink-prone tokens. But the innovation is less the specific mechanism (which adapts prior work on gated attention for LLMs) and more the identification of attention sink as the primary obstacle to single-tower multimodal generative pretraining for vision encoders, and the demonstration that addressing it transforms an approach that fails at discriminative tasks into one that succeeds. This finding is likely to transfer to other single-tower multimodal architectures and informs future design of unified vision-language models.

Innovation 3: Vision Encoder Pretraining Through Direct Token Prediction Is More Data-Efficient Than Contrastive or Encoder-Decoder Methods

The paper's most empirically consequential finding is not that GenLIP works — it's that GenLIP works with substantially less pretraining data than the dominant paradigms. GenLIP trained on 8B samples from Recap-DataComp-1B matches or outperforms SigLIP2 trained on 40B samples (5× more data) and OpenVision2 trained on 12.8B samples (1.6× more), across three model scales and 14 benchmarks. The 2B-sample controlled comparison in Table 7 shows GenLIP outperforming both contrastive (SigLIP) and encoder-decoder generative (OpenVision2) methods trained on the exact same data budget.

This is a strong claim about data efficiency, not just performance. It suggests that direct autoregressive token prediction from visual tokens — making the ViT "speak" — provides a denser learning signal per image-text pair than either contrastive matching (which reduces the entire caption to a single binary match/no-match signal) or encoder-decoder captioning (which routes the learning signal through an intermediate decoder). The intuition is that predicting a detailed caption token-by-token forces the vision encoder to extract and represent a rich set of visual details (objects, attributes, spatial relationships, text) that the caption mentions, producing a more comprehensive learning signal from the same amount of data.

Why this is intellectually significant rather than just a nice empirical result:

It provides evidence for the paper's core philosophical claim: that the objective matters more than the data scale. The field has been scaling contrastive pretraining to ever-larger datasets (12B, 40B, even larger) under the assumption that more data compensates for objective limitations. GenLIP's results suggest that a better-aligned objective can achieve equivalent or better performance with a fraction of the data, which reframes the scaling discussion from "how much data do we need?" to "what kind of learning signal is most efficient?"
It challenges the implicit assumption in encoder-decoder generative methods that the text decoder is necessary for efficient learning. The decoder adds parameters, computation, and architectural complexity. GenLIP shows it's not needed — the vision encoder can learn directly from the autoregressive loss without a mediating decoder. This means all the parameters invested in training the decoder (which is discarded when the vision encoder is deployed) are effectively wasted in architectures like CapPa and OpenVision2, at least for the purpose of learning a strong vision encoder.
The data efficiency advantage is most pronounced on Doc&OCR benchmarks (Table 3: GenLIP-g/16 averages 53.2 on seven Doc&OCR tasks vs. SigLIP2's 47.3, a 5.9-point gap). These tasks demand fine-grained visual-textual alignment — recognizing characters, reading text, understanding spatial layouts. The fact that GenLIP excels here with less data supports the paper's argument that direct token prediction provides a learning signal that is particularly effective for detail-sensitive visual understanding, which contrastive methods (optimizing for global alignment) struggle to capture.

This is an incremental advance in the sense that autoregressive pretraining was already known to work, but a fundamentally important empirical finding because it establishes that the minimalist approach is not just conceptually elegant but practically superior in data efficiency — a claim that was not obvious before this paper and that, if replicated, has direct implications for how vision encoders for MLLMs should be pretrained.

Innovation 4: Native-Aspect-Ratio Adaptation as a Lightweight Alternative to Full Multi-Resolution Pretraining

Prior work on improving vision encoder resolution handling (DINOv2, SigLIP2, LLaVA-NeXT's "anyres") typically either trains at multiple resolutions from the start or uses complex post-hoc processing (dynamic patching, tile-based encoding). GenLIP introduces a simpler paradigm: a two-stage pipeline where the model first learns foundational visual representations at a fixed low resolution (224×224, 8B samples), then undergoes a lightweight adaptation stage (37M samples, 1 epoch) at native aspect ratios with variable token budgets.

What makes this distinctive:

It decouples the learning of visual semantics from the learning of resolution invariance. Stage 1 handles the heavy lifting of learning what objects, attributes, and relationships look like — this benefits from large-scale data and stable training dynamics at a fixed resolution. Stage 2 only needs to teach the model to handle varied resolutions and aspect ratios, which is a much simpler adaptation task that can be done with far less data (37M vs. 8B samples) and a single epoch.
The design choice to use native aspect ratios rather than square-cropping or padding in Stage 2 is principled: real-world MLLM inputs include documents, charts, and photographs with widely varying aspect ratios. Square-cropping destroys spatial structure (e.g., a wide table gets cropped to a square, losing columns). Padding wastes visual token budget on empty regions. Native-aspect-ratio processing preserves the image's original structure, which is crucial for tasks like document understanding and chart interpretation where spatial layout carries semantic meaning.
The visual token budget approach (resize so that the number of patches falls in [16, 1024]) is a practical mechanism for handling arbitrary aspect ratios within a fixed computational budget. It means the model sees images at their natural shapes but with a token count that fits in GPU memory, analogous to how LLMs handle variable-length text sequences with a maximum context length.

The evidence that this matters: Figure 7 shows that Stage 2 adaptation improves performance across OCR, VQA, and Caption tasks compared to Stage 1 only, particularly at higher evaluation resolutions. The qualitative examples in Figure 4 and Figure 8 show Stage 2 models producing longer, more detailed, and more accurate descriptions. Table 6 (comparing S1 vs. S2 performance) shows consistent gains across model scales, with the gap being particularly large for g/16 (60.0 → 65.2 ALL AVG).

This is an incremental contribution to the multi-resolution vision encoder literature, but an important one because it provides a clean, reproducible recipe that other practitioners can adopt. The two-stage pipeline is simpler than the alternatives (no dynamic tiling, no multi-resolution training schedule balancing, no complex post-processing) and works effectively with only 37M adaptation samples — making it practical for labs that don't have the compute to pretrain at multiple resolutions from scratch.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation is conducted on a broad suite of 14 multimodal understanding benchmarks grouped into three categories. For Doc&OCR: ChartQA [50], OCRBench [47], DocVQA [51], TextVQA [59], AI2D [32], InfoVQA [52], and SEED-Bench-2-Plus [36]. For General VQA: MME [24], GQA [29], VQAv2 [26], and ScienceQA [48]. For Captioning: NoCaps [2], COCO [49], and TextCaps [58]. All evaluations are conducted using the LMMS-Eval toolkit [80]. Additionally, discriminative ability is assessed on ImageNet-1K [17] for classification and ADE20K [82] for semantic segmentation.
Base model(s). Three GenLIP model scales are evaluated: GenLIP-L (0.3B parameters, 24 layers, 1024 dims, 16 heads), GenLIP-So (0.4B parameters, 27 layers, 1152 dims, 16 heads), and GenLIP-g (1.1B parameters, 40 layers, 1536 dims, 24 heads). All use a patch size of 16. The models are pretrained on Recap-DataComp-1B (8B samples in Stage 1) and further adapted on 37M long-caption samples in Stage 2. For downstream MLLM evaluation, the vision encoder is paired with two LLM backbones: Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct [55].
Metrics. Multimodal understanding performance is evaluated using benchmark-specific metrics as implemented by LMMS-Eval. For overall comparison, an ALL AVG score is computed as the unweighted mean across all 14 benchmarks. MME-P scores (originally on a 0–2000 scale) are rescaled to [0, 100] by dividing by 2000 before averaging, ensuring numerical comparability with other percentage-based metrics. Captioning performance is reported using CIDEr. For discriminative evaluation, ImageNet-1K is assessed via top-1 accuracy (%) and ADE20K via mean Intersection over Union (mIoU).
Baselines. The paper compares against representative methods from three categories. Contrastive methods: CLIP [56] (ViT-L/14, trained on 12.8B pairs), SigLIP [78] (ViT-L/16 and ViT-So/14, 40.0B pairs), SigLIP2 [68] (ViT-L/16, ViT-So/16, ViT-g/16, all 40.0B pairs). Encoder-decoder generative methods: AIMv2 [23] (ViT-L/14, 12.0B pairs), OpenVision2 [46] (ViT-L/16, 12.8B pairs). For the standard LLaVA-NeXT evaluation: MLCD [4], RICE-ViT [73], SigLIP (So/14, 40.0B), and SigLIP2 (So/14, 40.0B). For the controlled 2B-sample comparison: SigLIP (So/16, 2.0B) and OpenVision2 (So/16, 2.0B).
Generation budget / compute accounting. The primary evaluation protocol is frozen visual representation evaluation, where the pretrained vision encoder is kept frozen and only the language model and 2-layer MLP projector are fine-tuned on the LLaVA-OneVision [35] instruction-tuning dataset (3M+ supervised fine-tuning samples). This protocol directly measures the quality of visual features learned by different VLP methods without the confounding effect of further fine-tuning the vision encoder. For a fair comparison, all vision encoders are configured to produce the same number of visual tokens (patches). An additional standard LLaVA-NeXT evaluation is conducted where the vision encoder is unfrozen and fine-tuned jointly with the LLM, tested at two patch budgets (576 and 729 patches). In the controlled 2B-sample ablation (Table 7), GenLIP is evaluated at 384×384 resolution directly after Stage 1, while SigLIP and OpenVision2 receive an additional high-resolution adaptation stage (384×384 for 0.2B samples) to ensure fair comparison.
Cross-validation / statistical protocol. No explicit cross-validation or statistical significance testing is reported. Results are single-point evaluations on each benchmark's standard test set. The controlled comparison in Table 7 ensures internal validity by training all methods on the exact same 2.0B samples from Recap-DataComp-1B under comparable conditions, but the broader main results compare against publicly available pretrained models with potentially different data sources, training recipes, and hyperparameters.

Main Quantitative Results

Frozen Visual Representation Evaluation with Qwen2.5-1.5B

Table 3 reports the primary frozen-feature evaluation using Qwen2.5-1.5B-Instruct as the LLM backbone. GenLIP demonstrates consistent performance advantages across all three model scales despite using substantially less pretraining data.

All-metric average (ALL AVG). GenLIP-L/16 achieves 61.5, outperforming SigLIP2-L/16 (58.7) by 2.8 points and SigLIP-L/16 (56.9) by 4.6 points. GenLIP-So/16 achieves 62.6 versus SigLIP2-So/16 at 60.6, a 2.0-point improvement. GenLIP-g/16 achieves 65.2, the highest overall score in the table, surpassing SigLIP2-g/16 (61.5) by 3.7 points and all other baselines.

Doc&OCR tasks. This is where GenLIP's advantages are most pronounced. Averaged across the seven Doc&OCR benchmarks (ChartQA, OCRBench, DocVQA, TextVQA, AI2D, InfoVQA, SEED-2), GenLIP achieves 49.3 (L/16), 50.1 (So/16), and 53.2 (g/16). The corresponding SigLIP2 scores are 45.0, 46.8, and 47.3, yielding gaps of 4.3, 3.3, and 5.9 points respectively. The largest single-benchmark gains for GenLIP-g/16 over SigLIP2-g/16 are on ChartQA (45.0 vs. 35.3, +9.7), OCRBench (55.6 vs. 47.3, +8.3), and DocVQA (57.0 vs. 47.6, +9.4). GenLIP also outperforms the encoder-decoder generative baseline OpenVision2-L/16 substantially on OCRBench (45.6 for OpenVision2 vs. 51.1 for GenLIP-L/16, +5.5).

General VQA tasks. GenLIP shows more modest but consistent advantages. GenLIP-g/16 achieves 53.3 on VQAv2 versus SigLIP2-g/16's 49.6 (+3.7), and 49.1 on GQA versus 50.1 (−1.0, a rare case where GenLIP trails). On ScienceQA, GenLIP-L/16 (41.5) and GenLIP-So/16 (44.0) modestly outperform SigLIP2 counterparts (42.6 and 43.5), while GenLIP-g/16 (45.5) matches SigLIP2-g/16 (45.2). MME-P scores are mixed: GenLIP-L/16 (1258) beats SigLIP2-L/16 (1165) by 93 points, but GenLIP-g/16 (1256) trails SigLIP2-g/16 (1284) by 28 points.

Captioning tasks. GenLIP shows particular strength on TextCaps (which requires reading text in images): GenLIP-g/16 achieves 135.4 CIDEr versus SigLIP2-g/16's 134.5. On NoCaps, GenLIP-L/16 (82.6) and So/16 (87.5) exceed SigLIP2 counterparts (82.9 and 84.3). On COCO captions, GenLIP-So/16 reaches 81.5 CIDEr, substantially outperforming SigLIP2-So/16's 77.1 (+4.4).

Comparison against generative baselines. GenLIP consistently outperforms AIMv2 and OpenVision2. Against AIMv2-L/14 (ALL AVG 55.7), GenLIP-L/16 achieves 61.5 (+5.8). Against OpenVision2-L/16 (ALL AVG 58.7), GenLIP-L/16 achieves 61.5 (+2.8), and GenLIP-So/16 achieves 62.6 (+3.9). OpenVision2 was pretrained on the stronger Recap-DataComp-1B v2 corpus with a longer training schedule, making GenLIP's advantage notable despite the data disadvantage.

Model scaling behavior. GenLIP shows notably better scaling with model size than SigLIP2. Moving from L/16 to g/16, SigLIP2 improves ALL AVG from 58.7 to 61.5 (+2.8), while GenLIP improves from 61.5 to 65.2 (+3.7). The gap between GenLIP and SigLIP2 widens at larger scales: 2.8 points at L/16, 2.0 at So/16, and 3.7 at g/16. This pattern is interpreted by the authors as evidence that "simplifying both the architecture and the objective can enable more efficient scaling."

Frozen Visual Representation Evaluation with Qwen2.5-7B

Table 4 replicates the frozen-feature evaluation with the larger Qwen2.5-7B-Instruct LLM backbone. The trends largely mirror the 1.5B setting, with some important differences.

All-metric average. GenLIP-L/16 achieves 69.0 versus SigLIP2-L/16 (not directly compared at L/16 scale with 7B — SigLIP L/16 scores 64.5 and GenLIP L/16 69.0, a 4.5-point gap). For So/16, GenLIP achieves 71.8 versus SigLIP2's 69.4 (+2.4). For g/16, GenLIP achieves 73.6 versus SigLIP2's 68.9 (+4.7). The gap at g/16 is larger with 7B (4.7 points) than with 1.5B (3.7 points), suggesting GenLIP's visual features scale more effectively with LLM capacity.

Doc&OCR dominance maintained. Averaging over the seven Doc&OCR tasks: GenLIP-L/16 (52.5) versus CLIP-L/14 (49.0, since SigLIP2 L/16 not reported for 7B), GenLIP-So/16 (56.3) versus SigLIP2-So/16 (51.8, gap of 4.5), and GenLIP-g/16 (60.7) versus SigLIP2-g/16 (52.3, gap of 8.4). Individual benchmarks: ChartQA at g/16 shows GenLIP 57.1 vs. SigLIP2 47.2 (+9.9), OCRBench 65.9 vs. 55.6 (+10.3), DocVQA 69.0 vs. 56.3 (+12.7), TextVQA 66.8 vs. 63.5 (+3.3).

General VQA comparison. GenLIP-g/16 achieves 61.1 on VQAv2 compared to SigLIP2-g/16's 56.4 (+4.7), 64.4 on GQA vs. 62.7 (+1.7), 54.5 on SQA vs. 49.3 (+5.2). MME-P: GenLIP-g/16 at 1483 vs. SigLIP2-g/16 at 1422 (+61 points). On SEED-2, GenLIP-g/16 achieves 43.6 vs. SigLIP2's 36.4 (+7.2).

Captioning with larger LLM. GenLIP-g/16 on NoCaps: 85.0 vs. SigLIP2-g/16's 82.0 (+3.0). TextCaps: 144.8 vs. 142.7 (+2.1). COCO: 75.5 vs. 72.3 (+3.2).

Cross-LLM comparison. Comparing Table 3 (1.5B) and Table 4 (7B) shows that both GenLIP and baselines benefit from a larger LLM, but GenLIP's gap over SigLIP2 often widens. For example, the ALL AVG gap at g/16 grows from 3.7 points (1.5B) to 4.7 points (7B), and the Doc&OCR gap at g/16 grows from 5.9 to 8.4 points. This suggests GenLIP's visual features contain richer information that a more capable LLM can better exploit — consistent with the paper's claim of better alignment with generative LLM operation.

Standard LLaVA-NeXT Evaluation (Unfrozen Vision Encoder)

Table 5 evaluates GenLIP under the standard LLaVA-NeXT setting where the vision encoder is unfrozen and jointly fine-tuned with the LLM during instruction tuning. This protocol tests both the pretrained representation quality and the model's adaptability to downstream fine-tuning. Evaluations are conducted at two patch budgets (576 and 729 patches) under the LLaVA-NeXT dynamic resolution (anyres) configuration.

At 576 patches. GenLIP-So/16 achieves an ALL AVG of 68.5. This outperforms CLIP L/14 (64.8) by 3.7 points, MLCD L/14 (66.3) by 2.2 points, and AIMv2 L/14 (66.5) by 2.0 points. It matches RICE-ViT L/14 (68.1, trained on 13.0B samples). On Doc&OCR specifically: GenLIP-So/16 achieves 79.3 on ChartQA vs. CLIP's 75.2 (+4.1), 75.2 on DocVQA vs. AIMv2's 72.7 (+2.5), 68.5 on TextVQA vs. CLIP's 62.5 (+6.0), 59.7 on OCRBench vs. RICE-ViT's 57.5 (+2.2). AI2D: 78.6 vs. RICE-ViT's 77.9 (+0.7). MMBench: 77.7 vs. AIMv2's 78.6 (−0.9).

At 729 patches. GenLIP-So/16 achieves an ALL AVG of 70.3, the highest in the table. This surpasses SigLIP So/14 (66.1) by 4.2 points, SigLIP2 So/14 (67.5) by 2.8 points, and RICE-ViT L/14 (68.6) by 1.7 points. Doc&OCR highlights: ChartQA 83.0 vs. SigLIP2's 79.1 (+3.9), DocVQA 76.9 vs. RICE-ViT's 75.1 (+1.8), TextVQA 69.6 vs. CLIP's 65.9 (at 729 patches with So/14 architecture) — a direct comparison difficult since architectures differ. OCRBench: 64.7 vs. SigLIP2's 58.7 (+6.0). MMBench: 78.1 vs. SigLIP2's 77.1 (+1.0). POPE: 80.1 vs. SigLIP2's 80.4 (−0.3). MMStar: 53.2 vs. SigLIP2's 52.8 (+0.4).

Key observation about unfrozen evaluation. GenLIP's advantages on Doc&OCR persist in the unfrozen setting, confirming that the pretrained representations provide a strong initialization even when fine-tuned. The ALL AVG improvements over SigLIP2 at 729 patches (70.3 vs. 67.5) are comparable to the frozen-feature advantages, indicating that GenLIP's representations are not merely "fragile" features that collapse under fine-tuning but rather robust initializations that support further adaptation. The consistent ranking across both frozen and unfrozen protocols strengthens the paper's claim that the pretraining objective produces genuinely better visual representations, not just features that happen to work well with a specific downstream LLM configuration.

Controlled Data Budget Comparison (2B Samples)

Table 7 presents the most internally valid comparison in the paper: SigLIP, OpenVision2, and GenLIP are all trained on the exact same 2.0B samples from Recap-DataComp-1B, with the same model architecture (So/16), and evaluated under identical protocols. This isolates the effect of the pretraining method from confounding factors like data scale, caption quality, and training duration.

OCR tasks. GenLIP achieves 44.2 average across seven OCR benchmarks versus SigLIP's 40.2 (+4.0) and OpenVision2's 42.0 (+2.2). Individual scores: GenLIP leads on ChartQA (35.0 vs. SigLIP's 26.1, +8.9; vs. OpenVision2's 27.8, +7.2), DocVQA (46.0 vs. 38.6 and 41.2, +7.4 and +4.8), TextVQA (47.1 vs. 44.3 and 44.7, +2.8 and +2.4), and InfoVQA (29.3 vs. 25.8 and 26.8, +3.5 and +2.5). However, GenLIP trails OpenVision2 on OCRBench (36.9 vs. 43.2, −6.3). The authors attribute this gap to "the known difficulty of dense-text recognition with low-resolution pretraining" and the absence of a high-resolution adaptation stage in GenLIP under this controlled setting.

General VQA tasks. GenLIP achieves 55.2 average versus SigLIP's 53.6 (+1.6) and OpenVision2's 54.3 (+0.9). VQAv2: 45.4 vs. 42.7 and 44.2. GQA: 42.0 vs. 39.8 and 40.3. SQA: 75.6 vs. 75.1 and 74.8. MME-P: 1156 vs. 1132 and 1158.

Interpretation. Under the strictest controlled setting, GenLIP's minimalist generative approach outperforms both contrastive (SigLIP) and encoder-decoder generative (OpenVision2) pretraining on the majority of benchmarks, confirming that the superiority is attributable to the pretraining methodology rather than to differences in data quality or scale. The OCRBench exception suggests that GenLIP benefits particularly from the high-resolution adaptation stage (absent here) for dense text recognition, a point the paper acknowledges explicitly.

Scalability Analysis

Data scaling (Figure 6). GenLIP-So/16 is pretrained on Recap-DataComp-1B with varying sample counts from 1.0B to 8.0B, and frozen visual representations are evaluated on Doc&OCR, VQA, and Caption tasks. Performance improves consistently from 1.0B to 4.0B samples across all three task categories. From 4.0B to 8.0B, the improvement curve flattens: VQA and Caption tasks show "only minor improvements," while Doc&OCR continues to benefit modestly. The variant with gated attention outperforms the variant without across the entire data range, with the gap being most pronounced at low data scales (1.0B–2.0B), supporting the claim that gated attention improves data efficiency. Based on these curves, the authors select 8.0B samples as the default Stage 1 pretraining budget.

Model scaling (Table 6). Results are reported after Stage 1 (fixed-resolution only) and after Stage 2 (with native-aspect-ratio adaptation). After Stage 1: GenLIP-L/16 achieves ALL AVG of 55.2, So/16 reaches 58.9 (+3.7), and g/16 reaches 60.0 (+1.1 over So/16). After Stage 2: L/16 reaches 61.5, So/16 reaches 62.6 (+1.1), and g/16 reaches 65.2 (+2.6). The Stage 2 adaptation disproportionately benefits larger models: the g/16 gains 5.2 points from S1 to S2, while So/16 gains 3.7 and L/16 gains 6.3. After Stage 1 only, GenLIP-L/16 (55.2) still trails SigLIP2-L/16 (58.7), but after Stage 2 adaptation, GenLIP-L/16 (61.5) surpasses SigLIP2-L/16 (58.7) by 2.8 points. The paper notes that "GenLIP-L/16 lags behind GenLIP-So/16 and GenLIP-g/16 only with fixed low-resolution pretraining," suggesting that the adaptation stage is important for smaller models to catch up.

Discriminative Ability Assessment

Table 8 evaluates the quality of GenLIP's visual representations on standard discriminative tasks using a frozen-backbone protocol (features extracted from the final layer, no multi-layer fusion, no test-time augmentation). On ImageNet-1K linear probing, GenLIP-L/16 achieves 83.9% top-1, GenLIP-So/16 achieves 84.3%, and GenLIP-g/16 achieves 85.2%. These trail SigLIP2-So/14 (88.9%) but outperform CLIP-L/14 (85.1%) at the g/16 scale and SigLIP-So/14 (86.7%) at no scale — though direct architectural comparison is confounded by different patch sizes (16 vs. 14). On ADE20K semantic segmentation, GenLIP-So/16 achieves 42.8 mIoU versus SigLIP-So/14's 40.8 and CLIP-L/14's 39.0. GenLIP-g/16 achieves 44.5 mIoU versus SigLIP2-So/14's 45.4 — a small gap but notable given SigLIP2 introduces explicit dense supervision. The variant without gated attention (GenLIP-So/16 w/o GA) achieves only 76.2% ImageNet top-1 — a catastrophic 8.1-point drop that the paper attributes directly to attention sink degrading spatial discriminability.

Ablation Studies and Robustness Checks

Gated attention ablation (Figure 6): Comparing GenLIP with and without gated attention across 1.0B to 8.0B pretraining samples, gated attention consistently yields higher performance on Doc&OCR, VQA, and Caption tasks. The advantage is most pronounced at lower data scales (1.0B–2.0B), where the variant without gated attention lags substantially. Gated attention also enables more stable training: the paper reports that it "alleviates loss spikes, accelerates convergence, and stabilizes scaling behavior." The mechanism's impact on discriminative quality is quantified in Table 8: GenLIP-So/16 without gated attention achieves only 76.2% ImageNet-1K top-1 accuracy versus 84.3% with gated attention, confirming that the attention sink phenomenon catastrophically degrades spatial feature quality for discriminative tasks.

Native-aspect-ratio adaptation (Figure 7): GenLIP-So/16 is evaluated after Stage 1 (fixed 224×224) and after Stage 2 (native aspect ratio adaptation) across multiple evaluation resolutions on OCR, VQA, and Caption tasks. Stage 2 consistently improves performance across all three task categories at all evaluation resolutions. The improvement is most significant at higher resolutions, where Stage 1 models (trained only on 224×224) struggle to effectively utilize additional tokens, while Stage 2 models benefit from the diverse-resolution training. The paper interprets this as evidence that the adaptation stage teaches the model to handle variable-resolution inputs effectively.

Model scale ablation (Table 6): Comparing L/16, So/16, and g/16 at both Stage 1 and Stage 2 reveals that performance improves monotonically with model size in all settings. The gap between g/16 and So/16 is relatively small after Stage 1 (60.0 vs. 58.9 ALL AVG, +1.1) but widens after Stage 2 (65.2 vs. 62.6, +2.6), suggesting that larger models benefit more from the diverse-resolution adaptation. The L/16 model shows the largest absolute gain from Stage 2 (55.2 → 61.5, +6.3), indicating that smaller models may be more resolution-bottlenecked in Stage 1 and therefore benefit disproportionately from adaptation.

Pretraining method comparison under controlled data budget (Table 7): As discussed in the Main Results, training SigLIP, OpenVision2, and GenLIP (all So/16) on identical 2.0B samples from Recap-DataComp-1B shows GenLIP achieves 55.2 ALL AVG versus OpenVision2's 54.3 and SigLIP's 53.6. GenLIP leads on 6 of 7 OCR benchmarks (the exception being OCRBench, trailing OpenVision2 by 6.3 points) and all General VQA benchmarks. This ablation is the cleanest test of the paper's central claim that the minimalist generative objective is more data-efficient than alternatives.

LLM scale robustness (Tables 3 vs. 4): The consistent GenLIP advantage over baselines persists when scaling the downstream LLM from Qwen2.5-1.5B to Qwen2.5-7B, with the gap often widening at larger LLM scales (e.g., Doc&OCR advantage over SigLIP2 at g/16 grows from 5.9 to 8.4 points). This suggests the quality improvement is in the visual representations themselves, not in a fortuitous interaction with a specific LLM configuration.

Attention sink visualization (Figure 3): The attention distribution of the first token in the input sequence is plotted, showing that "without gated attention, the first token absorbs most of the attention mass." This provides qualitative evidence for the mechanism that gated attention is designed to address.

Direct caption generation quality (Figure 4): While not a formal ablation, the qualitative examples demonstrate that GenLIP's generative capability improves with model scale (g/16 correctly identifies "Bulbasaur" while L/16 and So/16 mistake it for "Charmander") and with Stage 2 adaptation (longer, more detailed descriptions). These examples provide face validity for the pretraining approach — the model genuinely learns to describe images, supporting the premise that the generative objective produces meaningful visual-textual alignment.

Patch semantics readout (Figure 5): The ability to "translate" individual image patch features into language concepts via the LM head emerges spontaneously, with Stage 2 models showing stronger alignment. This is not a trained capability — it's a probe of what the model learned — and provides evidence that the pretraining produces localized visual-semantic alignment without explicit region-level supervision.

Critical Assessment

Claim from executive summary: "GenLIP matches or surpasses strong baselines like SigLIP2 (pretrained on 40B pairs) while using only 8B pretraining samples."

This claim is well-supported for the specific configurations tested. Tables 3 and 4 show GenLIP achieving higher ALL AVG than SigLIP2 at all three model scales (L/16, So/16, g/16) under both the 1.5B and 7B LLM backbones. The gaps range from 2.0 to 4.7 points. However, several qualifications are necessary:

The data comparison is not strictly controlled for data quality. SigLIP2 was trained on a different dataset (40B web-crawled image-text pairs, presumably with noisier captions than the LLaMA-3-recapped data GenLIP uses). If SigLIP2's training data had lower average caption quality, GenLIP's advantage is partly attributable to better supervision signal quality, not just the pretraining objective. The controlled 2B-sample comparison in Table 7 partially addresses this by training all methods on the same Recap-DataComp-1B data, but that comparison only covers So/16 scale at 2B samples — far below SigLIP2's 40B training scale.
The architecture comparison confounds several factors. SigLIP2 uses different patch sizes (14 vs. 16), different training recipes (optimizer settings, data augmentation, etc.), and possibly different image preprocessing. GenLIP's Stage 2 adaptation uses long-caption data (Infinity-MM, BLIP3o) that may provide qualitatively different supervision than whatever SigLIP2 used for its high-resolution adaptation. These are unavoidable in comparisons against publicly available pretrained models, but they weaken the claim that the objective alone drives the improvement.
The ALL AVG metric weights all 14 benchmarks equally. GenLIP's advantage is concentrated on Doc&OCR benchmarks; on General VQA and Caption tasks, the gap is modest or occasionally reversed (e.g., GQA at g/16 under 1.5B: GenLIP 49.1 vs. SigLIP2 50.1). An unweighted average makes GenLIP look stronger because its Doc&OCR advantage is large, but for applications where General VQA is the primary concern, the benefit is less clear.

Claim from executive summary: "GenLIP's advantages are most pronounced on Doc&OCR benchmarks, where it outperforms SigLIP2 by up to 5.9 points averaged over seven tasks."

Strongly supported. Table 3 shows the 5.9-point Doc&OCR advantage (GenLIP-g/16 53.2 vs. SigLIP2-g/16 47.3), and this pattern is consistent across model scales and LLM backbones. Table 5 (unfrozen evaluation) confirms the Doc&OCR advantage persists under standard LLaVA-NeXT fine-tuning. The mechanism is plausible — direct autoregressive token prediction from visual features forces the model to preserve fine-grained text-like visual patterns that contrastive methods, optimized for global alignment, may discard. However:

The paper does not provide a diagnostic experiment proving the mechanism. It would be valuable to see, for instance, whether GenLIP's visual features exhibit higher mutual information with character-level OCR labels compared to SigLIP2's features, or whether attention map visualizations show GenLIP attending more precisely to text regions.
OCRBench (Table 7) shows GenLIP trailing OpenVision2 by 6.3 points under the controlled 2B-sample setting. This suggests GenLIP's OCR advantage may be partially dependent on the high-resolution adaptation stage, which the 2B-sample controlled comparison didn't include for GenLIP (it was evaluated at 384×384 without native-aspect-ratio adaptation).

Claim from executive summary: "gated attention mechanism... to suppress attention sink behavior which would otherwise degrade discriminative visual representations."

Strongly supported by specific evidence. Figure 3 visualizes the attention sink. Table 8 quantifies the discriminative impact: 76.2% vs. 84.3% ImageNet top-1. Figure 6 shows consistent data scaling advantages. However, the paper does not investigate whether alternative solutions to attention sink (e.g., register tokens as in DINOv2, or a different positional encoding scheme) would be equally or more effective. The improvement from no-gate to gate is demonstrated, but the claim that gated attention is the best solution among alternatives is not tested. Additionally, the gated attention mechanism adds parameters (W_g, b_g per layer) and computation (a sigmoid and element-wise multiply per token per layer) — the paper doesn't report the overhead relative to the base model.

Claim from executive summary: "two-stage native-aspect-ratio adaptation procedure that improves detail-sensitive OCR and chart understanding."

Supported by Table 6 (comparing S1 vs. S2), Figure 7 (resolution scaling curves), and qualitative examples (Figures 4, 8). However, the evidence has limitations:

The adaptation stage changes two things simultaneously: (1) native aspect ratios, and (2) higher-quality long-caption data from different sources. The improvement cannot be cleanly attributed to native aspect ratios alone; the caption quality improvement is a confound.
An ablation comparing native-aspect-ratio adaptation versus fixed-aspect-ratio high-resolution adaptation (e.g., square-cropping at 384×384 with the same long-caption data) would isolate the contribution of native aspect ratios specifically. The paper does not run this ablation.
The adaptation uses only 37M samples and 1 epoch, which is lightweight, but the paper doesn't explore whether more adaptation data or epochs would further improve performance, or whether similar gains could be achieved by simply increasing Stage 1 resolution.

Missing experiments that would strengthen the paper:

Scaling beyond 8B samples. Figure 6 suggests performance is plateauing, but extrapolating from a flattening curve is risky. Training at 16B or 32B samples would clarify whether GenLIP's data efficiency advantage over SigLIP2 (40B) is about better use of data or about a lower asymptotic ceiling. It's possible that SigLIP2 would continue improving at 40B while GenLIP saturates earlier.
Validation on alternative MLLM architectures beyond LLaVA-NeXT. The paper acknowledges this limitation explicitly: "our validation experiments are conducted on an academic-scale MLLM setting, LLaVA-NeXT, and the generalizability to cutting-edge ones remains to be verified." Testing GenLIP as the vision encoder in InternVL, Qwen-VL, or Cambrian-1 would substantially strengthen the claim that it is a "strong foundation for vision encoders in MLLMs."
Comparison against other single-tower generative methods at matched scale. The paper compares against SAIL conceptually but not empirically. A direct comparison against SAIL or NEO at the same model scale and data budget would clarify whether GenLIP's specific design choices (gated attention, MRoPE, two-stage training) provide advantages over the broader single-tower generative approach.
Fine-grained diagnostic experiments on Doc&OCR advantage. What specific visual capabilities drive the OCR improvement? Is it better character-level feature preservation? Better spatial layout encoding? Better handling of small text? Attention map analysis on text-containing images, or probing experiments that decode character identities from intermediate features, would provide mechanistic insight.
Impact of caption quality on pretraining. GenLIP uses high-quality LLaMA-3-recapped captions. An ablation comparing GenLIP trained on raw web alt-text versus recapped captions (matched data volume) would quantify how much of the data efficiency advantage comes from caption quality versus the pretraining objective.
Computational cost accounting. The paper doesn't report training FLOPs or wall-clock time for GenLIP pretraining. The comparison against SigLIP2 (40B samples) claims data efficiency, but if GenLIP's single-tower Prefix-LM training is substantially more expensive per sample than contrastive training (due to processing concatenated multimodal sequences), the FLOPs efficiency may not track the sample efficiency. This is particularly relevant for the scaling claims.

Conditional nature of the findings:

The OCR advantage is partly dependent on the adaptation stage. Table 7 shows GenLIP without adaptation trails OpenVision2 on OCRBench. The strong Table 3/4 Doc&OCR results include Stage 2 adaptation. Practitioners adopting GenLIP should expect to need the adaptation stage for OCR-heavy applications.
The data efficiency advantage is demonstrated at 8B samples versus 40B. Whether GenLIP remains more efficient at other scales (e.g., 1B vs. 5B, or 32B vs. 160B) is unknown. Figure 6 shows GenLIP's improvement flattening from 4B to 8B, which could indicate an approaching ceiling. Extrapolating beyond 8B is speculative.
The evaluation is on academic benchmarks with existing answer keys. GenLIP's performance on open-ended visual reasoning, multimodal dialogue, or real-world deployment scenarios (where image distributions differ from web-crawled data) is not tested.
GenLIP is a vision encoder pretraining method, not an MLLM. The results show that GenLIP features work well when fed into a separate LLM, but the paper doesn't evaluate GenLIP as a standalone model for tasks that don't involve an LLM (beyond ImageNet and ADE20K). The "Let ViT Speak" qualitative examples are compelling but not systematically evaluated against captioning baselines.

Overall assessment. The experiments provide strong support for the paper's central empirical claims within the tested regime: GenLIP's minimalist generative pretraining produces vision encoders that are highly competitive with and often outperform contrastive and encoder-decoder generative baselines on multimodal understanding benchmarks, with particularly strong performance on document and OCR tasks, and the gated attention mechanism is empirically essential for achieving this performance. The data scaling and model scaling experiments provide internal evidence of healthy scaling behavior. However, the paper does not establish that GenLIP fundamentally scales better than alternatives — only that it performs well at the tested scales — and several confounds (data quality, adaptation stage contributions, architectural differences) prevent attributing all gains purely to the pretraining objective. The missing experiments (especially scaling beyond 8B, alternative MLLM architectures, and FLOPs-matched comparisons) represent genuine gaps that limit the strength of the scalability and generality claims.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted For in Efficiency Claims

The paper's headline efficiency gains — outperforming SigLIP2 (40B training samples) with only 8B pretraining samples — implicitly frame GenLIP as a more data-efficient and therefore computationally cheaper pretraining paradigm. However, GenLIP's per-sample training cost is significantly higher than contrastive methods because it processes concatenated multimodal sequences (visual tokens + text tokens) through a Prefix-LM attention pattern, whereas contrastive methods like SigLIP2 process visual and text encoders independently and compute a lightweight contrastive loss over a shared embedding space.

The consequence. The paper does not report training FLOPs, wall-clock time, or GPU-hours for any GenLIP pretraining run. This means a practitioner cannot determine whether GenLIP's 8B-sample training is actually cheaper, comparable, or more expensive than SigLIP2's 40B-sample training. The sample-efficiency advantage might be partially or fully offset by higher per-sample compute cost. In the autoregressive setting, the model must process the full concatenated sequence (196 visual tokens + typically 50-300 text tokens for captions) through all transformer layers, and the Prefix-LM attention mask requires computing different attention patterns for visual and textual positions, which may reduce hardware efficiency compared to the uniform bidirectional attention used in contrastive vision encoders. Furthermore, the language modeling head produces logits over a 151,936-token vocabulary at every text position, adding computation that has no counterpart in contrastive pretraining.

What evidence exists in the paper. None. The paper reports no FLOPs counts, no training throughput numbers, no GPU-hour estimates, and no cost comparison between GenLIP and baselines. Table 2 provides hyperparameters (batch size, optimizer settings, sequence length) but these are insufficient for a practitioner to estimate total training cost. The controlled comparison in Table 7 (all methods trained on 2.0B samples) reports only downstream accuracy, not the compute cost to reach those accuracies.

Mitigation status. The limitation is not acknowledged in the paper's Limitations section, which focuses on other constraints ("academic-scale MLLM setting," "pretraining dataset limited to 1.0B scale," "reliance on high-quality captions"). The omission of compute accounting is a significant gap in a paper whose central thesis is about efficiency and scalability. A FLOPs-matched comparison (analogous to the pretraining vs. inference compute analysis in Hoffmann et al., 2022) would substantially strengthen the claims, but is absent.

The 8B-Sample Scaling Ceiling Is Not Characterized

The data scaling experiment (Figure 6) shows GenLIP's performance improving from 1.0B to 4.0B pretraining samples, then flattening noticeably from 4.0B to 8.0B, particularly for VQA and Caption tasks. The paper acknowledges this explicitly: "we observe steeper gains when scaling from 1.0B to 4.0B, while the improvement curve becomes flatter when further scaling to 8.0B. In particular, the average performance on VQA and caption tasks shows only minor improvements when scaling from 4.0B to 8.0B."

The consequence. This flattening raises the possibility that GenLIP is approaching a performance ceiling at or near 8B samples, while SigLIP2 (trained on 40B samples, 5× more) may not have saturated. If GenLIP's curve plateaus while SigLIP2's continues to improve with more data, then GenLIP's current advantage at 8B vs. 40B may be an artifact of comparing two methods at different points on their respective scaling curves, rather than evidence that direct token prediction is fundamentally more data-efficient. A practitioner deciding whether to adopt GenLIP needs to know whether training beyond 8B samples would yield meaningful further improvements, or whether the method has inherent scaling limits (e.g., the captioning objective may saturate because captions, unlike contrastive batches, provide limited diversity — there are only so many ways to describe an image).

What evidence exists in the paper. Figure 6 provides the primary evidence, showing GenLIP-So/16 at 1.0B, 2.0B, 4.0B, and 8.0B samples. The VQA and Caption curves are nearly flat from 4.0B to 8.0B. However, this is only one model scale (So/16) and one dataset (Recap-DataComp-1B). The paper does not report data scaling curves for L/16 or g/16, nor does it test whether GenLIP's scaling behavior differs by task difficulty or caption quality. The paper's Limitations section acknowledges this obliquely — "the pretraining dataset is limited to 1.0B scale, the scaling behavior at even larger volumes is yet to be explored" — but this frames it as an unexplored opportunity rather than a potential ceiling.

Mitigation status. The limitation is acknowledged as a scope constraint, but not treated as a potential weakness of the method. The paper does not attempt to diagnose why scaling slows down (e.g., is it data diversity exhaustion? model capacity saturation? objective limitations?) or to project what performance might be achievable at larger scales. The authors select 8.0B as the default based on the flattening curve, which is pragmatic but leaves open the question of whether GenLIP's advantage is sustainable at the data scales where contrastive methods are typically trained (12B–40B+).

The Strongest Doc&OCR Results Depend on a Separate High-Quality Adaptation Dataset That Contrastive Baselines May Not Have Access To

GenLIP's most distinctive performance advantage — the 5.9-point Doc&OCR average improvement over SigLIP2 at g/16 scale (Table 3) — is measured after Stage 2 adaptation, which uses two specific long-caption datasets: the caption subset of Infinity-MM (Stage 1, 10M samples) and BLIP3o-Long-Caption (27M samples). These datasets contain long, detailed captions paired with higher-resolution images, and their captions are qualitatively different from the web-crawled alt-text or synthetic captions used in SigLIP2's pretraining.

The consequence. The Stage 2 adaptation changes two variables simultaneously: (1) image resolution and aspect ratio handling, and (2) caption quality and length. The improvement from Stage 1 to Stage 2 (Table 6: GenLIP-g/16 improves from 60.0 to 65.2 ALL AVG, +5.2 points) cannot be cleanly attributed to native-aspect-ratio training versus better caption supervision. A practitioner cannot determine whether the adaptation stage works because of how images are presented (native aspect ratios, variable token budgets) or because of what the captions contain (denser, more detailed descriptions), or both. This matters because if the caption quality is the primary driver, then contrastive methods might also benefit from similar long-caption data, potentially closing the gap. If the native aspect ratio adaptation is the driver, then the technique is more broadly applicable but needs to be validated independently.

What evidence exists in the paper. The controlled comparison in Table 7 partially illuminates this: without Stage 2 adaptation, GenLIP (So/16, 2.0B samples) still outperforms SigLIP and OpenVision2 on most benchmarks, but trails OpenVision2 on OCRBench by 6.3 points. The paper attributes this gap to "the known difficulty of dense-text recognition with low-resolution pretraining" and the absence of high-resolution adaptation in GenLIP under this setting. This suggests that for certain OCR tasks, the resolution/adaptation stage is indeed important — but the experiment doesn't isolate whether native aspect ratios specifically (versus any form of higher-resolution training with square crops) drive improvement. The paper does not ablate the adaptation stage by running it with fixed-aspect-ratio higher-resolution images on the same long-caption data.

Mitigation status. The paper does not attempt to isolate the contribution of native aspect ratio processing from caption quality in the adaptation stage. An ablation using the same long-caption data but with fixed square-cropped higher-resolution images would clarify whether native aspect ratios are necessary or merely helpful. The paper's Limitations section does not address this confound.

Generalization to Non-Academic MLLM Architectures Is Unverified

The paper evaluates GenLIP exclusively under the LLaVA-NeXT framework with Qwen2.5 LLM backbones, using a 2-layer MLP projector and the LLaVA-OneVision instruction-tuning dataset. While LLaVA-NeXT is a widely used academic MLLM architecture, it represents only one point in the design space of modular MLLMs. Other architectures use different projectors (e.g., Q-Former in InstructBLIP, perceiver resamplers in Flamingo, cross-attention-based connectors), different vision feature extraction strategies (e.g., multi-layer feature fusion, dynamic resolution tiling like in InternVL or Qwen-VL), and different instruction-tuning recipes.

The consequence. It is unknown whether GenLIP's advantages transfer to these other architectures. Specifically:

Projector sensitivity: GenLIP features are extracted from the final LN layer output. The 2-layer MLP projector used in the evaluation may be well-suited to GenLIP's representations but suboptimal for contrastive baselines (or vice versa). A more expressive projector (e.g., a Q-Former or perceiver) might narrow or widen the gap, and a practitioner using a different MLLM framework cannot assume GenLIP will retain its advantage without empirical validation.
Multi-resolution handling: MLLMs like InternVL2 and Qwen2-VL use dynamic resolution strategies (tiling images into multiple sub-images, encoding each separately, and concatenating features) that are different from GenLIP's native-aspect-ratio approach. It's unclear whether GenLIP's Stage 2 adaptation provides complementary benefits or becomes redundant when the MLLM already handles resolution through tiling.
LLM backbone dependence: The evaluation uses Qwen2.5-1.5B and Qwen2.5-7B. Whether GenLIP's representations transfer equally well to other LLM families (LLaMA, Mistral, Gemma, InternLM) is untested. Different LLMs may have different sensitivities to visual feature characteristics (dimensionality, token count, representational structure).

What evidence exists in the paper. The paper acknowledges this limitation explicitly: "our validation experiments are conducted on an academic-scale MLLM setting, LLaVA-NeXT, and the generalizability to cutting-edge ones remains to be verified." However, the acknowledgment understates the scope of the issue — the concern is not just about "cutting-edge" MLLMs but about any MLLM architecture that differs from LLaVA-NeXT in projector design, resolution handling, or training recipe, which includes many widely used open-source MLLMs.

Mitigation status. The limitation is acknowledged but not addressed. No experiments test GenLIP with alternative MLLM architectures, projectors, or LLM backbones beyond Qwen2.5. The paper frames this as future work rather than an immediate concern, but for a practitioner deciding whether to adopt GenLIP for their specific MLLM stack, the lack of cross-architecture validation is a genuine adoption barrier.

The Gated Attention Solution Is Validated Only Against a No-Gate Baseline, Not Against Alternative Attention Sink Remedies

The paper identifies attention sink as a critical failure mode and proposes gated attention as the solution. However, the attention sink problem in transformers is not new — it has been documented in pure language models (StreamingLLM proposed retaining initial tokens as "attention sinks" to enable long-context inference; Xiao et al., 2023) and in pure vision transformers (DINOv2 introduced dedicated register tokens to absorb sink behavior; Darcet et al., 2023). The paper does not compare gated attention against these established alternatives.

The consequence. A practitioner seeking to address attention sink in a single-tower multimodal transformer has several options: gated attention (this paper), register tokens (DINOv2), initial token retention strategies, or alternative positional encoding schemes that reduce the structural advantage of early tokens. The paper provides no evidence that gated attention is the most effective or most efficient choice among these alternatives. It demonstrates only that gated attention is better than nothing. The register token approach, in particular, is architecturally simpler (add a few learnable tokens to the input sequence, no per-layer gating mechanism) and was designed specifically to address the same problem in vision transformers. Whether GenLIP with register tokens would achieve comparable or better performance — and with what tradeoffs in parameters and computation — is unknown.

Furthermore, gated attention adds learnable parameters (W_g, b_g) and computation (a sigmoid activation and element-wise multiplication) at every transformer layer. For GenLIP-g/16 with 40 layers, this means 40 additional linear transformations and 40 sigmoid operations per forward pass. The paper does not report the parameter overhead relative to the base model or the throughput impact. If the overhead is non-trivial, a simpler alternative with lower overhead might be preferable even if slightly less effective.

What evidence exists in the paper. Figure 6 compares GenLIP with and without gated attention across data scales, and Table 8 shows the discriminative quality difference (84.3% vs. 76.2% ImageNet top-1 for So/16). But no experiment compares gated attention against register tokens, against other gating mechanisms (e.g., hard attention gating, learnable temperature scaling), or against architectural modifications that might prevent attention sink without per-layer gating (e.g., changing the positional encoding to reduce the privileged status of the first token).

Mitigation status. The paper does not acknowledge this as a limitation or compare against alternative sink remedies. The gated attention mechanism is motivated by reference to prior work on gated attention for LLMs (Qiu et al., 2025) but the paper does not discuss why gating was chosen over register tokens or other alternatives, nor does it provide ablations testing different gating designs (e.g., per-head vs. per-token gating, different activation functions, gating attention weights vs. attention outputs). A practitioner cannot determine from the paper whether gated attention is the recommended approach or simply the first approach that was tried and worked.

The Vision Encoder Produces Token Sequences Without a CLS Token, Requiring Attentive Probing for Discriminative Tasks

GenLIP has no CLS token — its architecture processes only patch tokens through the transformer. This is a natural consequence of the generative pretraining objective, which requires patch-level features to predict text tokens and has no need for a global image representation. However, it creates a practical friction: standard transfer learning for discriminative tasks (classification, retrieval) typically relies on a CLS token or global average pooling of patch features to produce a fixed-dimensional image representation.

The consequence. To evaluate GenLIP on ImageNet-1K (Table 8), the paper uses "attentive probing" — a learned attention-weighted combination of patch features — rather than the simpler linear probe on a CLS token that standard ViTs and contrastive VLMs support. This adds implementation complexity for practitioners who want to use GenLIP for both generative MLLM tasks and discriminative vision tasks. Furthermore, attentive probing introduces additional learned parameters and may be more prone to overfitting on small downstream datasets compared to CLS token probing. The ADE20K segmentation evaluation (Table 8) uses only a linear layer on patch features, which is standard but may underutilize GenLIP's representations compared to methods that were designed with dense prediction in mind.

More consequentially, the absence of a CLS token means GenLIP cannot be directly plugged into MLLM frameworks that expect a CLS token as the image-level representation (though this is uncommon in modern MLLMs, which typically use patch features). The paper's extraction procedure — taking all patch features from the final LN layer and feeding them through an MLP projector — is compatible with most current MLLMs, but requires the MLLM to handle variable numbers of visual tokens (which LLaVA-NeXT does natively).

What evidence exists in the paper. Table 8 demonstrates that GenLIP achieves competitive discriminative performance despite the lack of a CLS token: 85.2% ImageNet top-1 (g/16) and 44.5 ADE20K mIoU (g/16). The attentive probing method is mentioned but not described in detail. The paper does not compare attentive probing against alternative aggregation methods (e.g., global average pooling, max pooling, learned query tokens) to establish whether the reported numbers are near-optimal for GenLIP's representations.

Mitigation status. The paper treats the absence of a CLS token as a natural consequence of the architecture rather than a limitation, and does not discuss its implications for discriminative transfer or compare against CLS-token-based probing on the same representations. A practitioner who wants a single vision encoder for both MLLMs and standard vision tasks would need to maintain two task-specific heads (attentive probing for classification, MLP projector for MLLMs) or design a unified extraction method, adding engineering overhead that contrastive VLMs with CLS tokens avoid.

7. Implications and Future Directions

How This Work Changes the Landscape

GenLIP makes a concrete methodological argument that shifts the conversation around vision encoder pretraining for MLLMs: the pretraining objective matters more than the pretraining architecture, and aligning the objective with the downstream LLM's operating mode (autoregressive next-token prediction) can be more impactful than scaling data volume under a misaligned objective. This is not a paradigm shift — contrastive pretraining remains dominant, and the results are limited to one benchmark family (LLaVA-NeXT) and one pretraining data scale (8B samples) — but it is a substantive reframing of the efficiency question. The field has largely accepted that scaling contrastive pretraining to ever-larger datasets (12B, 40B, and beyond) is the primary path to better vision encoders. GenLIP provides counter-evidence: a generative objective on 8B recapped samples can match or exceed contrastive training on 40B samples, with the largest gaps on document understanding and OCR tasks (5.9 points averaged across seven Doc&OCR benchmarks at g/16 scale, Table 3).

The conceptual shift is that generative pretraining does not require a generative architecture. Prior generative VLP methods (CapPa, AIMv2, OpenVision2) coupled a vision encoder with a separate text decoder, implicitly treating the decoder as necessary for the autoregressive objective to work. GenLIP shows that a single transformer with a thin LM head — the same transformer that processes visual tokens — can learn directly from the language modeling loss. This decoupling (generative objective without generative decoder) reframes the design space: the question is not "how do we build an encoder-decoder that generates captions?" but "what is the simplest way to make a ViT produce features that support token prediction?" The answer GenLIP provides — concatenated sequences, Prefix-LM attention, gated attention, MRoPE — is one point in this space, but the framing opens it to further simplification.

The work also diagnoses a previously uncharacterized failure mode specific to single-tower multimodal pretraining: attention sink under Prefix-LM attention. By showing that the first visual token absorbs disproportionate attention mass when text tokens attend causally over a visual prefix, and that this catastrophically degrades discriminative visual representations (76.2% vs. 84.3% ImageNet top-1 for So/16 without vs. with gated attention, Table 8), the paper identifies a structural obstacle that any future single-tower multimodal architecture must address. This diagnosis makes naive single-tower approaches less attractive (they will hit the sink problem) while making gated or otherwise sink-mitigated approaches more attractive — a concrete redirection of research effort.

The reconciliation of conflicting intuitions is subtle but present. One intuition from contrastive VLP is that "more data and bigger batches produce better alignment." Another intuition from LLM pretraining is that "autoregressive objectives produce representations that are naturally suited for autoregressive downstream tasks." GenLIP provides evidence that, at least for vision encoders in MLLMs, the second intuition can dominate: better objective alignment can compensate for dramatically less data (8B vs. 40B samples). This doesn't resolve the tension — contrastive methods continue to dominate on retrieval and classification — but it clarifies that the "best" pretraining paradigm is downstream-task-dependent, and that MLLM vision encoders have specific requirements (fine-grained visual-textual alignment, text preservation, generative compatibility) that contrastive methods may not efficiently satisfy.

Follow-Up Research This Work Enables

Scaling GenLIP beyond 8B samples to identify whether the performance ceiling is a method limitation or a data limitation. Figure 6 shows GenLIP's VQA and Caption performance flattening from 4B to 8B samples, while Doc&OCR continues to improve modestly. A critical open question is whether GenLIP would continue improving at 16B, 32B, or 64B samples, or whether the captioning objective fundamentally saturates because captions — unlike the batch-constructed negatives in contrastive learning — provide limited diversity per sample. A scaling experiment extending GenLIP-So/16 to at least 16B samples, with intermediate checkpoints at 10B and 12B, would reveal whether the 4B–8B plateau is temporary (the model is still learning, just more slowly) or structural (the objective has exhausted the information in the captions). If GenLIP plateaus while SigLIP2 continues to improve at 40B, the paper's data efficiency advantage would be reframed as a lower asymptotic ceiling rather than a fundamental efficiency gain. If GenLIP continues improving, even slowly, the efficiency argument strengthens. This experiment is newly tractable because the paper provides the full training recipe (Tables 1 and 2) and demonstrates training stability at 8B samples.

Controlled comparison against register tokens and alternative attention sink remedies at matched model scale. The paper establishes gated attention as effective but does not compare it against DINOv2-style register tokens — the dominant approach to attention sink in vision transformers — or against other gating designs (per-head gating, gating of attention weights rather than outputs, learnable temperature scaling). A systematic ablation training GenLIP-So/16 (with the same 2.0B-sample budget as Table 7's controlled comparison) with: (a) gated attention (the paper's method), (b) 4–16 learnable register tokens prepended to the visual sequence (DINOv2 approach), (c) a fixed retention of the first visual token as an explicit sink (StreamingLLM approach), and (d) no sink mitigation (baseline), and evaluating both discriminative quality (ImageNet linear probe, ADE20K mIoU) and downstream MLLM performance (the 14-benchmark suite from Tables 3/4) would clarify whether gated attention is the best solution or simply the first one tested. The key question is whether the performance overhead of per-layer gating (additional parameters, sigmoid computation, element-wise multiplication) is justified compared to the architectural simplicity of register tokens. This experiment is directly enabled by the paper's identification of attention sink as the failure mode and its controlled evaluation protocol.

Isolating the contribution of native-aspect-ratio processing from caption quality in the adaptation stage. The Stage 2 adaptation improves performance substantially (GenLIP-g/16: 60.0 → 65.2 ALL AVG, Table 6) but simultaneously changes two variables: image processing (native aspect ratios, variable token budgets) and caption quality (long, detailed captions from Infinity-MM and BLIP3o). A clean ablation would train three Stage 2 variants on the same 37M long-caption samples: (a) native aspect ratios with variable token budgets [16, 1024] (the paper's method), (b) square-crop at fixed 384×384 resolution (preserving caption quality, removing native aspect ratio handling), and (c) square-crop at 224×224 with the same long captions (controlling for caption quality while keeping Stage 1 resolution). Comparing downstream Doc&OCR performance across these variants would reveal whether GenLIP's OCR advantage in Tables 3/4 is driven by better resolution handling (supporting the paper's architectural argument) or by better caption supervision (suggesting that contrastive baselines might also benefit from similar data). The OCRBench gap in Table 7 (GenLIP trails OpenVision2 by 6.3 points without adaptation) makes this experiment particularly important for understanding GenLIP's limitations on dense text recognition.

Validating GenLIP as a vision encoder in non-LLaVA MLLM architectures with different projector designs. The paper evaluates exclusively under LLaVA-NeXT with a 2-layer MLP projector and Qwen2.5 LLM backbones. The claim that GenLIP is "a strong foundation for vision encoders in MLLMs" requires testing in at least one alternative MLLM framework — for example, using GenLIP features in InternVL2's dynamic resolution pipeline (where images are tiled into sub-images and features are concatenated), or in a Q-Former-based architecture like InstructBLIP (where a learnable query transformer compresses visual tokens before the LLM). The key question is whether GenLIP's advantages (particularly on Doc&OCR) are contingent on the specific way LLaVA-NeXT processes visual features (simple MLP projection, no compression, full patch tokens) or whether they transfer to architectures that compress, re-weight, or restructure visual tokens before feeding them to the LLM. This experiment is enabled by the paper's open-source release of pretrained checkpoints and would directly address the acknowledged limitation that "generalizability to cutting-edge [MLLMs] remains to be verified."

Probing what specific visual capabilities drive GenLIP's Doc&OCR advantage. The paper demonstrates strong Doc&OCR performance but does not diagnose why GenLIP's features are better for text-heavy tasks. A mechanistic study could: (a) train linear probes to decode character identities, spatial positions, and font attributes from GenLIP and SigLIP2 patch features on text-containing images (e.g., from TextOCR or SynthText), measuring whether GenLIP preserves finer-grained text information; (b) visualize and compare attention maps from the downstream LLM when answering OCR questions, to see whether GenLIP features cause the LLM to attend more precisely to text regions; (c) evaluate GenLIP's per-patch OCR accuracy by using the LM head to decode patch features into characters (extending the "patch semantics readout" from Figure 5) and measuring character-level precision and recall against SigLIP2 features decoded through an equivalent probing method. This would convert the paper's empirical observation ("GenLIP is better at OCR") into a mechanistic understanding ("GenLIP preserves character-level visual features that contrastive pretraining discards"), which would inform future pretraining objective design beyond whether GenLIP specifically is adopted.

Negative result: testing whether GenLIP's advantage persists when contrastive baselines are given equal-quality captions. The paper's data efficiency claim compares GenLIP on Recap-DataComp-1B (LLaMA-3-recapped captions, high quality) against SigLIP2 on WebLI (web-crawled alt-text, lower quality). A critical stress test is to train SigLIP2 on the exact same 8B recapped samples from Recap-DataComp-1B that GenLIP uses, at matched model scale (So/16 or g/16), and compare downstream performance. If SigLIP2 matches or exceeds GenLIP under equal data quality, the paper's data efficiency claim would be reframed as primarily a data quality effect rather than an objective alignment effect — the contrastive objective works fine when given good captions; the performance gap was driven by noisy training data, not objective mismatch. If the gap persists or widens, the objective alignment argument is strengthened. This experiment is resource-intensive (training SigLIP2 from scratch at 8B scale) but essential for disentangling data quality from objective quality in the paper's central empirical claim.

Practical Applications and Downstream Use Cases

Document understanding and OCR pipelines where detail preservation is critical. GenLIP's strongest empirical advantage is on Doc&OCR benchmarks — at g/16 scale with Qwen2.5-7B, it outperforms SigLIP2 by 8.4 points averaged across seven tasks (Table 4), with individual gaps of 12.7 points on DocVQA and 10.3 points on OCRBench. For organizations building MLLM-based document processing systems (invoice extraction, contract analysis, scientific figure interpretation, form understanding), replacing a CLIP or SigLIP vision encoder with GenLIP-g/16 could meaningfully improve accuracy on tasks where missing a single digit or misreading a table cell produces an incorrect answer. The 8B-sample pretraining cost, while not characterized in FLOPs, is a one-time investment that can be amortized across all downstream document applications. The practical deployment path is straightforward: drop in the GenLIP checkpoint in place of an existing ViT in any LLaVA-NeXT-compatible pipeline, retain the 2-layer MLP projector, and fine-tune the LLM on task-specific instruction data. The frozen-feature evaluation protocol (Tables 3/4) already simulates this use case, meaning the reported numbers are close to what a practitioner would achieve without further vision encoder fine-tuning.

Data-efficient vision encoder pretraining for teams with limited compute budgets. GenLIP's 8B-sample pretraining — while the FLOP cost is not reported — uses substantially fewer image-text pairs than the 40B used by SigLIP2 and the 12–13B used by CLIP, AIMv2, and OpenVision2. For academic labs or smaller companies that cannot acquire or process 40B image-text pairs, GenLIP provides a recipe for training a competitive vision encoder on a more manageable data budget. The Recap-DataComp-1B dataset is publicly available, the pretraining hyperparameters are fully specified (Table 2), and the two-stage pipeline reduces the resolution adaptation cost to 37M samples. A team could replicate GenLIP-So/16 pretraining and expect, based on Tables 3 and 4, to achieve vision encoder quality comparable to or exceeding publicly available SigLIP2-L/16 or OpenVision2-L/16 checkpoints, despite using a fraction of the training data. The practical benefit is not just cost savings but also data sovereignty: smaller teams can train on datasets they curate or trust, rather than relying on black-box pretrained encoders trained on undisclosed web data.

Multi-purpose vision encoders that serve both MLLMs and standard discriminative vision tasks. GenLIP's discriminative evaluation (Table 8) shows 85.2% ImageNet top-1 and 44.5 ADE20K mIoU at g/16 scale — competitive with CLIP (85.1% and 39.0) and SigLIP (86.7% and 40.8), though trailing SigLIP2 (88.9% and 45.4) which incorporates dense supervision. For applications requiring both generative multimodal understanding (e.g., visual question answering, captioning) and discriminative vision tasks (e.g., image classification, retrieval, segmentation), GenLIP provides a single pretrained checkpoint that performs competitively on both fronts. This eliminates the need to maintain separate vision encoders for different task families — a practical simplification for deployment pipelines. The attentive probing method required for classification (since GenLIP lacks a CLS token) adds implementation overhead but is a one-time engineering cost. A practical deployment pattern: use GenLIP-g/16 as the shared vision backbone, attach an MLP projector for MLLM tasks and a separately trained attentive probe for classification/retrieval, and deploy both heads in the same service.