PaliGemma 2: A Family of Versatile VLMs for Transfer

🎯 Pitch

Fine-tuning a vision-language model for document tasks? Skipping higher resolution in favor of a larger language model is a costly mistake: PaliGemma 2 reveals that text-heavy tasks strictly demand increased image size, while larger LLMs only help with tasks needing world knowledge. Detailed analysis across 30+ tasks dissects where to spend your compute budget.

1. Executive Summary

PaliGemma 2 introduces a family of open-weight Vision-Language Models spanning three sizes (3B, 10B, 28B parameters) and three resolutions (224px², 448px², 896px²), all built by pairing the SigLIP-So400m vision encoder with the Gemma 2 language model family and trained through the same three-stage recipe as the original PaliGemma. The paper studies how model size and resolution affect transfer performance across 30+ academic benchmarks plus several new specialized tasks—text detection and recognition, table structure recognition, molecular structure recognition, optical music score recognition, long fine-grained captioning, spatial reasoning, and radiography report generation—obtaining state-of-the-art results on many of them. Through systematic analysis, the work identifies distinct task clusters: some benefit primarily from increased resolution (e.g., document understanding tasks where native image resolution exceeds 224px²), others from larger language models (e.g., multilingual captioning and advanced visual reasoning requiring world knowledge), and a third group from both—while also establishing that larger models consistently prefer lower transfer learning rates, and that PaliGemma 2 3B at 896px² outperforms specialized task-specific architectures on OCR benchmarks without relying on any task-specific components.

2. Context and Motivation

The Core Problem: Understanding What Matters for VLM Transfer Performance

The fundamental question this paper tackles is what factors drive effective transfer learning in Vision-Language Models, and how should practitioners choose among model variants for their specific tasks? This is not merely an academic curiosity — it directly shapes deployment decisions for anyone building on top of VLMs. If you need to fine-tune a VLM for document understanding, should you invest your compute budget in a larger language model or in processing images at higher resolution? The answer has direct implications for hardware requirements, latency budgets, and training costs.

The problem is particularly pressing because the VLM landscape has evolved rapidly over the last few years, as the authors note in Section 2: VLMs transitioned "from simple dual-encoder (contrastive) or encoder-decoder (captioning) designs trained from scratch, to more capable designs combining a pretrained vision encoder with a pretrained language model." This architectural convergence means that component choice — specifically, which vision encoder and which language model to pair — has become a central design decision, but one that is poorly understood in systematic terms.

The gap is this: prior work on VLM scaling has investigated various axes independently — some studies vary the vision encoder (Kar et al., 2024), others vary training data and compute (McKinzie et al., 2024; Deitke et al., 2024), others vary resolution (Laurençon et al., 2024), and still others vary model size by borrowing language models from different labs with different architectures and training recipes (Tong et al., 2024; Karamcheti et al., 2024). The authors explicitly call out this fragmentation:

"We are not aware of prior work which jointly studies the effect of the image resolution and the size of the language models on transfer via fine-tuning. In particular, prior works relying on different language model sizes often use models with different architecture and training recipes from different labs."

This is the critical gap: no prior study has varied both resolution and language model size in a controlled setting where everything else — vision encoder architecture, training recipe, data mixture — is held constant, and then evaluated the downstream effects on a wide range of transfer tasks through fine-tuning. Without such a controlled study, observed performance differences could be attributed to the wrong factor. Does a bigger language model help because it's bigger, or because its pretraining recipe was better? Does higher resolution help all tasks equally, or only tasks with fine-grained visual detail?

Why This Problem Matters: Practical Deployment Decisions and Scientific Understanding

The importance of this problem operates on two levels.

Practical impact on deployment decisions. When fine-tuning VLMs for production use, practitioners face a combinatorial choice space: which base model (size, resolution) should they use for their specific task? The paper's Table 1 quantifies the stakes: training cost per example increases roughly 4.6× when going from 224px² to 448px² (due to the quadratic growth in vision tokens from 256 to 1024), and roughly 3.7× when going from a 2B to a 9B language model. These are not trivial multipliers — they translate directly to training time, cloud computing costs, and hardware requirements. A practitioner choosing the wrong combination could waste substantial resources, or worse, deploy an unnecessarily large model when a smaller one at higher resolution would perform better.

The paper also quantifies this for CPU-only deployment scenarios (Section 4.9), where latency constraints are severe and model size directly determines runtime. Understanding whether a task can be handled by a smaller model at higher resolution rather than requiring a larger model has direct implications for on-device deployment feasibility.

Scientific significance for transfer learning theory. Beyond the practical engineering concerns, the question touches on a fundamental issue in multimodal learning: where do capabilities reside? Does visual reasoning emerge primarily from the vision encoder's ability to extract detailed features, or from the language model's ability to reason over those features? By holding the vision encoder (SigLIP-So400m) constant while varying the language model (Gemma 2 at 2B, 9B, 27B) and the input resolution, the study provides a clean decomposition of these effects. Tasks that benefit primarily from resolution are those where the bottleneck is input information content — the relevant visual detail simply cannot be captured at 224px². Tasks that benefit primarily from model size are those where the bottleneck is reasoning or world knowledge — the visual information is accessible but the model needs more sophisticated language understanding to process it correctly.

Where Prior Approaches Fall Short

The paper identifies several specific limitations in prior work:

Conflation of architecture, training recipe, and model size. Several recent works investigating VLM scaling use language models from different families to span different sizes. For example, studies like Cambrian-1 (Tong et al., 2024) and Prismatic VLMs (Karamcheti et al., 2024) compare models built with LLaMA, Vicuna, Gemma, and other language models — each with its own architecture, pretraining data, and optimization procedure. When performance changes with "model size," it is impossible to attribute the change purely to parameter count, since the training recipe and architecture also change. The authors note this limitation explicitly in Section 2, citing LLaVA-NeXT (Li et al., 2024) as a "notable exception" that conducted ablations within a single model family.

Isolated investigation of scaling axes. Research on resolution scaling (Laurençon et al., 2024) investigates how image resolution affects performance, but doesn't simultaneously vary model size in a controlled way. Research on model size scaling (McKinzie et al., 2024) does the opposite. The interaction effects — for instance, whether higher resolution helps a 10B model differently than a 3B model — remain unexplored. Without joint investigation, practitioners are left guessing whether to invest their compute budget in model size or resolution for a new task.

No systematic guidance on transfer learning rates across model sizes. The original PaliGemma (Beyer et al., 2024) established a standard hyperparameter sweep range for fine-tuning, but did not investigate whether the optimal learning rate shifts with model size. Prior scaling work largely ignores this practical question, meaning that practitioners upgrading from a 3B to a 10B model might unknowingly use a suboptimal learning rate, leaving performance on the table through a hyperparameter choice rather than a model capability issue.

Benchmark scope limitations. Most prior scaling studies evaluate on a standard set of captioning and VQA benchmarks. PaliGemma 2 extends this to specialized domains — OCR, molecular structure recognition, music score recognition, table structure recognition, radiography — where the resolution-vs-size tradeoff manifests differently. Document understanding tasks, for instance, have images whose native resolution vastly exceeds typical VLM input sizes, making them a natural testbed for whether higher resolution can substitute for larger models.

How This Paper Positions Itself Relative to Existing Work

The paper explicitly positions itself as a direct upgrade of the PaliGemma (Beyer et al., 2024) approach, not a novel architecture or training paradigm. The authors state in the introduction:

"We now upgrade PaliGemma by replacing its language model component with the more recent and more capable language models from the Gemma 2 family, producing new PaliGemma 2 base VLMs at 3 different sizes and 3 different resolutions."

The paper's contribution is therefore not a new method, but rather a controlled experimental framework enabled by a family of models that are "all derived from comparable building blocks and are trained according to the same recipe" (Section 1). This design choice is deliberate: by fixing the vision encoder architecture (SigLIP-So400m), the training recipe (same three stages, same data mixture up to reweighting at higher resolutions), and the pretraining compute scaling (all models trained on Cloud TPUv5e with the same optimizer), the paper can make clean causal attributions about the effects of language model size and input resolution on transfer performance.

The paper positions its scaling analysis as addressing a specific gap in the existing literature while building on the transfer learning evaluation framework established by the original PaliGemma. The 30+ academic benchmarks from PaliGemma serve as a retrospective baseline, allowing direct comparison: for the same 3B model size and 224px² resolution, PaliGemma 2 "slightly outperform[s]" PaliGemma (an average improvement of 0.65 points on 224px² and 0.85 on 448px², per Section 4.1.3). The additional specialized tasks (Sections 4.2–4.8) extend the evaluation scope to stress-test the resolution-vs-size tradeoff in domains where the bottleneck is clearly knowable: OCR tasks bottlenecked by resolution, reasoning tasks bottlenecked by language understanding.

The paper also positions its findings on transfer learning rate (Section 4.1.2, Figure 5) as practical guidance emerging from the controlled comparison: the observation that larger models consistently prefer lower learning rates is a finding that would be difficult to establish without comparing models of different sizes from the same family with the same training recipe.

Finally, the paper subtly positions the PaliGemma 2 family as a practical alternative to the prevailing "instruction tuning" paradigm (Dai et al., 2023; Liu et al., 2023) for transferring VLMs to new tasks. Rather than relying on large-scale instruction tuning to create a general-purpose chat interface, PaliGemma 2 models are designed for per-task fine-tuning — a more compute-efficient approach for practitioners who know their target task in advance and want the best possible performance on that specific task rather than broad conversational ability.

3. Technical Approach

3.1 Reader Orientation

The PaliGemma 2 "system" is a family of transferable vision-language models—parameterized functions that take an image and a text prompt as input and autoregressively generate a text answer as output, where the specific behavior is determined by fine-tuning on a target task's training data. The paper solves the problem of understanding how to choose between model size and input resolution when allocating a compute budget for transfer learning: by training a grid of models at three sizes (3B, 10B, 28B) and three resolutions (224px², 448px², 896px²) using an identical recipe, the experiments reveal which types of tasks benefit from more visual detail (higher resolution) versus more sophisticated language understanding (larger language model), and provide concrete guidance—such as the finding that larger models require lower transfer learning rates.

3.2 Big-Picture Architecture (Diagram in Words)

The PaliGemma 2 architecture comprises three major components connected in a feedforward pipeline:

SigLIP-So400m Vision Encoder — a pre-trained contrastive vision model that processes the input image (at 224px², 448px², or 896px²) and outputs a sequence of patch embeddings (256, 1024, or 4096 tokens respectively, for a 14px² patch size). This component is frozen in its architecture across all model variants—every PaliGemma 2 model, regardless of size or resolution, uses the identical SigLIP-So400m checkpoint.
Linear Projection — a learned linear transformation that maps the SigLIP-400m embedding dimension into the Gemma 2 language model's input space. This is the only component trained from scratch to connect the vision and language modalities.
Gemma 2 Language Model — a pre-trained autoregressive decoder that receives the concatenated sequence of projected image tokens followed by text prompt tokens as a prefix, and generates the text answer token-by-token. This component comes in three sizes: 2B, 9B, and 27B parameters, determining the 3B, 10B, and 28B total model sizes respectively.

Information flows as follows: an image is resized and padded to a square of the target resolution → the SigLIP encoder tiles it into non-overlapping 14×14 pixel patches and produces one embedding per patch → the linear projection maps these embeddings to the language model dimension → the projected image tokens are prepended to the text prompt → Gemma 2 processes the combined sequence (the "prefill" phase) → the model autoregressively samples output tokens (the "extend" phase) until an end-of-sequence token is produced.

The training process unfolds in three sequential stages, with Stage 0 being the independent unimodal pretraining of the vision encoder and language model (referenced from prior work), Stages 1–2 being multimodal pretraining at progressively higher resolutions, and Stage 3 being task-specific fine-tuning.

3.3 Roadmap for the Deep Dive

First, the three-stage training recipe (Stages 1–3), since it establishes what knowledge the base models possess before any transfer occurs and explains how progressively higher resolutions are introduced. Understanding this is essential to interpreting the scaling analysis—the models are pretrained at different resolutions, not just fine-tuned at them.
Second, the architecture details and computational cost model, explaining why increasing resolution costs roughly as much as increasing language model size (the quadratic token growth) and establishing the FLOPs context for the size-vs-resolution comparisons in Section 4.1.
Third, the transfer (fine-tuning) setup—the hyperparameter sweep protocol, the learning rate selection procedure, and how tasks are formatted through PaliGemma 2's text interface—since this is the mechanism by which base models become specialized task solvers and the source of the paper's practical guidance on learning rates.
Fourth, the specialized task pipelines (OCR, table recognition, molecular recognition, music score recognition, DOCCI captioning, spatial reasoning, radiography), each with their own data sources, output formats, preprocessing steps, and evaluation protocols, since these represent the new applications beyond PaliGemma's original scope and demonstrate the versatility of the text-only interface for structured prediction.
Fifth, the quantization and CPU inference pathway, since it is a distinct deployment mode with its own quality-tradeoff analysis (gemma.cpp's 8-bit switched-floating-point quantization compared against Jax/f32).
Finally, the experimental design behind the scaling analysis—the model grid, the selection of which resolution-trained checkpoints to fine-tune, the cross-task consistency of hyperparameters, and how relative improvements are computed (Figure 3, Figure 13)—since this is the methodological backbone for the paper's central claims about task clustering.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a controlled empirical scaling study whose core idea is that by fixing the vision encoder, training recipe, and data mixture while systematically varying the language model size and input resolution, one can isolate their effects on transfer performance and derive actionable guidance for practitioners choosing among model variants.

The Three-Stage Training Recipe

PaliGemma 2 inherits the same three-stage training approach as the original PaliGemma (Beyer et al., 2024), with Stage 0 being unimodal pretraining of the individual components using their original recipes—SigLIP-So400m trained via sigmoid contrastive loss (Zhai et al., 2023) and Gemma 2 trained via standard language modeling (Gemma Team, 2024). Stages 1–3 are multimodal and are the focus here.

Stage 1: Low-Resolution Multimodal Pretraining (224px²). The pretrained SigLIP-So400m and Gemma 2 checkpoints are combined and trained jointly on a multimodal task mixture of 1 billion examples. The key architectural detail: no parameters are frozen during this stage—both the vision encoder and the language model are updated. The authors emphasize that they use the "raw checkpoints, without post-training steps" for both components, meaning the Gemma 2 weights taken at the end of language model pretraining but before any instruction tuning or RLHF. The image resolution is 224px² (256 vision tokens). The task mixture is designed to "enable transferability to a wide range of tasks via fine-tuning" and includes captioning, grounded captioning (where the output includes bounding box coordinates), OCR, machine-generated visual question answering, object detection, and instance segmentation. Many of the training labels are machine-generated using publicly available specialist models—the paper explicitly states this is done "without using a large commercial VLM as common among other open VLMs such as LLaVA," positioning PaliGemma 2's training data as more transparent and reproducible.

The learning rate schedule scales with model size. The original PaliGemma used a learning rate of 2·10⁻⁵ for its 3B model. For PaliGemma 2, this base rate is multiplied by 0.5 for the 3B model (1·10⁻⁵) and by 0.25 for the 10B and 28B models (5·10⁻⁶). This size-dependent adjustment reflects empirical stability concerns: larger models typically require smaller learning rates to train stably, particularly in multimodal settings where the vision encoder and language model start from different pretraining optima.

The optimizer is Adam (Kingma and Ba, 2017) with default hyperparameters throughout all stages. The authors use fully-sharded data-parallel training (FSDP; Zhao et al., 2023) on Cloud TPUv5e Pod slices of 256 to 1024 chips, except for the 28B model at 896px² which uses TPUv5p. Stage 1 training for the 3B model takes approximately 3 days using 256 chips, which the authors note is "roughly the same training cost as PaliGemma."

Logits soft-capping (from Gemma 2). Following the Gemma 2 language model design (Gemma Team, 2024), the authors apply logits soft-capping to the attention logits and output logits in the Gemma 2 component during Stages 1 and 2. This technique, originally proposed for neural combinatorial optimization (Bello et al., 2016), caps the magnitude of logits before the softmax operation to prevent the model from becoming overconfident—a form of regularization that stabilizes training. The cap values follow the Gemma 2 defaults. Critically, logits soft-capping is disabled in Stage 3 (the transfer/fine-tuning stage) because "this led to worse results for some transfer tasks." This is a non-obvious design choice: the regularization that helps pretraining stability can hurt fine-tuning adaptability. When soft-capping is enabled, the model's output distribution is prevented from becoming too peaked, which is useful during multi-task pretraining where the model needs to maintain broad capability, but limiting during transfer where the model should be able to confidently specialize to the target distribution.

Stage 2: High-Resolution Multimodal Pretraining. This stage introduces higher resolutions sequentially. The model first trains for 50 million examples at 448px², then for 10 million examples at 896px². The task mixture retains the same components as Stage 1 but with two adjustments: tasks that benefit from higher resolution (primarily OCR and document understanding) are upweighted in the sampling distribution, and the output sequence length is increased. The longer output sequence length is specifically motivated: OCR tasks at high resolution require the model to transcribe long strings of visual text, which at 224px² would span a handful of output tokens but at 896px² might require many more tokens for detailed transcriptions.

The progressive resolution introduction—448px² before 896px²—is a curriculum learning strategy. Jumping directly from 224px² to 896px² would require the model to adapt to a 16× increase in vision tokens (256 → 4096) in a single step, which could cause training instability. The intermediate 448px² checkpoint (1024 vision tokens, 4× Stage 1) provides a smoother transition.

The relative training cost per example across configurations is quantified in Table 1. For the 3B model, a single example at 448px² costs roughly 4.6× what it costs at 224px², and at 896px² costs roughly 23.5×. This non-linear cost growth arises because the vision encoder processes (H/patch_size) × (W/patch_size) patches, and each of these patches becomes a token in the Gemma 2 sequence. Since the Gemma 2 self-attention cost is quadratic in sequence length, the total cost grows faster than the linear token count increase. The paper notes that "increasing resolution incurs a similar additional cost as increasing the language model size"—for reference, going from 3B to 10B (224px²) costs 3.7× per example, comparable to the 4.6× cost of going from 224px² to 448px² with the 3B model.

Stage 3: Task-Specific Fine-Tuning (Transfer). This is the stage where base models become specialized task solvers. From the checkpoints produced by Stage 1 (for 224px² models) or Stage 2 (for 448px² and 896px² models), the model is fine-tuned on the target task's training data. The paper distinguishes between two categories of Stage 3 transfer:

Standard academic benchmarks (30+ tasks from PaliGemma): Following the hyperparameter sweep space from PaliGemma (Beyer et al., 2024, Section 3.2.4), the authors explore learning rates in {0.03, 0.06, 0.1, 0.3, 0.6, 1.0, 3.0} × 10⁻⁵. A critical methodological decision: for most tasks, the original PaliGemma used identical hyperparameters for 224px² and 448px², so the authors "only sweep at 224px² resolution and reuse the selection for both resolutions." This reduces the computational burden but assumes that the optimal learning rate does not shift significantly with resolution—an assumption that could mask resolution-dependent learning rate effects. The best learning rate is selected based on the validation split for each model size and task independently, then the model is retrained with that learning rate and evaluated on the test split.
New specialized tasks (Sections 4.2–4.8): Each task uses its own hyperparameter sweep tailored to the task's characteristics, detailed in Appendix B. These typically involve sweeps over learning rate, sometimes label smoothing, weight decay, and dropout, with specific batch sizes, training step counts, and hardware configurations.

Architecture Details and Computational Cost Model

Vision Encoder: SigLIP-So400m. The vision encoder is the SigLIP-So400m model (Alabdulmohsin et al., 2023; Zhai et al., 2023), a vision transformer designed via scaling laws for compute-optimal shape. It operates with a patch size of 14×14 pixels. This fixed patch size means the number of output tokens grows quadratically with resolution:

224px² → (224/14) × (224/14) = 16 × 16 = 256 tokens
448px² → (448/14) × (448/14) = 32 × 32 = 1024 tokens
896px² → (896/14) × (896/14) = 64 × 64 = 4096 tokens

The SigLIP-So400m parameter count is relatively small—the exact count is not specified but is implied to be much smaller than the language models (the 3B model with a 2B language model and SigLIP-400m totals 3.0B parameters, suggesting the vision encoder + projection contribute roughly 400M + projection parameters).

Linear Projection. The projection is a simple linear transformation with no activation function. It maps from the SigLIP-400m embedding dimension to the Gemma 2 input dimension. The projection is applied independently to each token position—there is no cross-token communication in this step. The weights are trained from random initialization in Stage 1 and updated throughout all stages.

Gemma 2 Language Model Component. The Gemma 2 models (Gemma Team, 2024) are decoder-only transformers with several architectural improvements over the original Gemma. The specific sizes used are:

Gemma 2 2B → total PaliGemma 2 size: 3.0B parameters
Gemma 2 9B → total PaliGemma 2 size: 9.7B parameters
Gemma 2 27B → total PaliGemma 2 size: 27.7B parameters

The authors note an important asymmetry in the language model family: the 2B and 9B Gemma 2 models were trained using distillation from a larger teacher model (Gemma Team, 2024, Section 6.1), while the 27B model was "trained from scratch." The paper speculates that this difference may explain why "the largest PaliGemma 2 can thus be useful if one wants to get the best possible performance and has no compute or latency constraints" but also why the 10B→28B scaling often shows diminishing returns compared to 3B→10B: the 27B language model, being trained from scratch without distillation, may have inferior per-parameter quality compared to the distilled 9B model, partially offsetting the benefits of its larger capacity.

Prefill and Extend Phases. The autoregressive generation process has two computationally distinct phases:

Prefill: The combined sequence of image tokens (256/1024/4096) + text prompt tokens is processed in parallel through all transformer layers to produce a key-value cache. This phase is compute-bound and its cost scales with the total prefix length (image + text tokens).
Extend: One new token is sampled at a time, with each new token attending to all prefix tokens (via the KV cache) but only requiring one forward pass through the output layers. The cost per token is proportional to the total sequence length (prefix + previously generated tokens). Since image tokens dominate the prefix for high-resolution models, the per-token generation cost is substantially higher for 896px² models than for 224px² models.

The CPU inference measurements in Table 9 provide concrete wall-time numbers for the 3B 224px² model: prefill with 260 tokens (256 image + 4 text) takes 1.6 seconds on an Apple M1 Max, while each subsequent extend call takes 0.9 seconds for 11 output tokens (generating "A large building with two towers on the water").

Training Infrastructure. Models are trained on Cloud TPUv5e (for most configurations) and TPUv5p (for the 28B model at 896px²). TPUv5p is assumed to provide a 2.3× speed-up per chip over TPUv5e, which is factored into the relative cost estimates in Table 1. The training uses fully-sharded data-parallel (FSDP) across 256 to 1024 chips.

Transfer (Fine-Tuning) Setup and Hyperparameter Protocol

Task Formatting Through the Text Interface. PaliGemma 2 handles all tasks through a unified text-to-text interface, regardless of whether the output is natural language (captioning), a structured string (HTML for table recognition), or coordinates (bounding boxes for detection). The key design principle is that no task-specific architectural components are added—the same model architecture serves all tasks, with task specialization achieved entirely through the training data format and fine-tuning.

Examples of task-specific formatting include:

Captioning: Input is the image + a prefix indicating the task; output is the free-form caption.
VQA: Input is the image + a text question; output is a short answer.
Detection and segmentation: The output vocabulary includes special location tokens (<locDDDD>, where DDDD encodes a quantized image coordinate in 0000–1023) and segmentation tokens (<segDDD>) that define binary masks inside bounding boxes. Figure 2 shows an example: for referring segmentation of puffins in an image, the model outputs a sequence of segmentation and location tokens interspersed with text labels to indicate which mask corresponds to which object.
Table structure recognition: The output is an HTML string with cell bounding box coordinates encoded as <td coords="<locXMIN><locYMIN><locXMAX><locYMAX>">cell text</td>.
Molecular structure recognition: The output is a SMILES string (a linear notation for molecular graphs).
Music score recognition: The output is a string in **kern format, which encodes pitch, duration, articulation, and barlines.
Spatial reasoning (VSR): Formulated as a QA task with True/False answers.
Radiography report generation: The model receives the "INDICATIONS" section as a prefix and generates the "FINDINGS" and "IMPRESSIONS" sections.

This unified interface is the architectural enabler for the paper's central claim about versatility: "PaliGemma 2's versatile interface" is what allows it to handle OCR, table recognition, and music score recognition without "task-specific architecture components as common in the OCR literature."

Learning Rate Selection Protocol. For the 30+ academic benchmarks from PaliGemma, the learning rate is swept over {0.03, 0.06, 0.1, 0.3, 0.6, 1.0, 3.0} × 10⁻⁵ at 224px² for each model size independently. The best learning rate is selected per (task, model size) pair based on the validation split, and the same learning rate is then applied to the 448px² version. This protocol has three important implications:

Model-size-dependent learning rates are explicitly studied. The sweep is conducted separately for 3B, 10B, and 28B, enabling the analysis in Figure 5 and Section 4.1.2.
The resolution-independent learning rate assumption. By reusing the 224px²-optimal learning rate for 448px², the protocol cannot detect if the optimal learning rate shifts with resolution. This is a pragmatic decision to limit computational cost but means that the 448px² results might be slightly suboptimal if resolution interacts with learning rate.
Cross-validation integrity. The learning rate is selected on the validation split and evaluated on the test split, with results reported as the mean and standard deviation over 5 fine-tuning runs (Table 13). This provides both a measure of central tendency and an estimate of run-to-run variability.

For the new specialized tasks (Sections 4.2–4.8), learning rate sweeps follow task-specific ranges detailed in Appendix B. For instance, text detection and recognition sweeps {0.01, 0.05, 0.1, 0.5, 1.0} × 10⁻⁴, finding an optimum of 10⁻⁵, while spatial reasoning (VSR) sweeps {0.1, 0.2, 1.0, 3.0} × 10⁻⁶.

Training Length and Batch Size. For the academic benchmarks, training length and batch size follow the PaliGemma defaults. For new tasks, Appendix B specifies individual configurations:

Text detection: 15k steps, batch size 256 on 256 TPUv5e.
Table recognition: same setup as text detection, but with maximum output length 4096 (to accommodate long HTML strings) and no label smoothing.
Molecular recognition: 30k steps, batch size 256, 256 TPUv5e, learning rate 10⁻⁴, label smoothing 0.1, maximum output length 256.
Music score recognition: same as molecular but with maximum output length 1024.
Radiography: batch size determined by hardware, fine-tuned for 8 epochs with learning rate 5·10⁻⁶, no label smoothing/dropout/weight decay for the initial configuration, then sweeping learning rates from {0.03, 0.1, 0.3, 1.0, 5.0} × 10⁻⁴ for higher resolutions and model sizes.

Stage 1 and Stage 2 Pretraining Data Mixture

While the paper does not provide the exact mixture proportions (referencing PaliGemma, Section 3.2.5 for details), it characterizes the mixture qualitatively as involving:

Captioning: Image → natural language description.
Grounded captioning: Image → descriptions with inline bounding box coordinates for mentioned objects, following the LocCa approach (Wan et al., 2024).
OCR tasks: Image containing text → transcription of the text.
Machine-generated VQA: Images paired with question-answer pairs generated by specialist models (Changpinyo et al., 2022; Piergiovanni et al., 2022).
Object detection: Image → list of {bounding box, class label} tuples, formatted using Pix2Seq-style sequence generation (Chen et al., 2022).
Instance segmentation: Image → per-instance masks with class labels, formatted using the same text interface with segmentation tokens (Chen et al., 2023).

The critical design philosophy is the use of machine-generated labels from publicly available specialist models rather than large commercial VLMs. The paper makes this an explicit positioning point: "none uses a large commercial VLM as common among other open VLMs such as LLaVA." This means PaliGemma 2's training data can be reproduced by anyone with access to the same specialist models, whereas many competing open VLMs are effectively distilled from proprietary models like GPT-4V, making their training pipelines not fully reproducible.

During Stage 2 (high-resolution pretraining), the mixture is reweighted to "upweight" tasks benefiting from higher resolution—primarily OCR and document understanding tasks where text at 224px² may be illegible but at 448px²/896px² becomes readable. The exact reweighting factors are not specified.

Specialized Task-Specific Pipelines

Each new transfer task requires its own data preprocessing, output formatting, and evaluation protocol. These are documented in Appendix B and in the task-specific experiment sections.

Text Detection and Recognition (Section 4.2, Appendix B.1). The task is to output pairs of {transcription, bounding_box} for all word instances in an image. Training data combines ICDAR'15, Total-Text, MLT17, MLT19, HierText, TextOCR, and IntelOCR. The bounding box output format uses PaliGemma 2's built-in location tokens. The evaluation protocol follows the HierText competition: a prediction counts as a true positive if the IoU between the predicted and ground-truth bounding boxes is ≥0.5 AND the transcription exactly matches (with no case normalization, punctuation filtering, or text length filtering—a stricter protocol than some prior work). Training runs for 15k steps with batch size 256, sweeping learning rates {0.01, 0.05, 0.1, 0.5, 1.0} × 10⁻⁴ (optimum: 10⁻⁵), with label smoothing of 0.1 and resolution 896px² producing the best results.

Table Structure Recognition (Section 4.3, Appendix B.2). The task is to output an HTML string representing the table structure, with cell bounding boxes and text content. Trained on PubTabNet (516k medical tables) and FinTabNet (113k financial tables). Preprocessing is detailed: images are resized preserving aspect ratio and padded to square with white pixels, cell bounding boxes are encoded using <locDDDD> tokens scaled to 0000–1023, and corrupted ground-truth examples are removed (e.g., boxes extending outside the image). Additional refinements from Smock et al. (2023) are applied to FinTabNet to improve annotation quality. Maximum output length is 4096 tokens, no label smoothing, learning rate 10⁻⁴. The evaluation metrics are Tree Edit Distance Similarity (TEDS) and Grid Table Similarity (GriTS), which independently measure cell text content, cell topology/structure, and bounding box quality.

Molecular Structure Recognition (Section 4.4, Appendix B.3). The task is to output a SMILES string given an image of a molecular drawing. Training data: 1 million molecules from PubChem, rendered using the Indigo toolkit with diverse drawing styles and random perturbations (following MolScribe; Qian et al., 2023). This data augmentation strategy is crucial because molecular drawings in the wild exhibit varied rendering conventions (font choices, bond line thickness, label positioning). Training: 30k steps, batch size 256, learning rate 10⁻⁴, label smoothing 0.1, max output length 256, images padded to square and resized. Evaluation metric: exact match percentage on 5.7k synthetic molecules rendered with ChemDraw.

Optical Music Score Recognition (Section 4.5, Appendix B.4). The task is to output a **kern encoding (pitch, duration, articulation, barlines) from an image of a pianoform score. Dataset: GrandStaff (53.7k images), with the official train/validation/test splits. During training, both original and synthetically augmented images are used; evaluation is on original images only. The metrics operate at three granularities: Character Error Rate (CER) on individual characters, Symbol Error Rate (SER) on multi-character symbols, and Line Error Rate (LER) on full **kern lines. Training follows the molecular recognition setup but with max output length 1024. The authors note that increasing resolution from 448px² to 896px² improves all error rates, but increasing model size from 3B to 10B does not—suggesting this task is bottlenecked by visual detail rather than language understanding.

Long, Fine-Grained Captioning (DOCCI; Section 4.6, Appendix B.5). The task is to generate detailed image descriptions where every sentence must be factually grounded in the image content. Dataset: DOCCI (Descriptions of Connected and Contrasting Images; Onoe et al., 2024), containing 15k images with human-annotated descriptions averaging 7.1 sentences (639 characters, 136 words). The evaluation protocol is notable for its rigor: raters are presented with each generated sentence highlighted and asked to classify the relationship as "Entailment," "Neutral," "Contradiction," or "Nothing to assess." Five raters evaluate each sentence, and the majority vote determines the label. The final metric is the percentage of Non-Entailment Sentences (NES), where lower is better. The protocol derives from the Attribution framework of Rashkin et al. (2023), adapted for visual grounding.

Model selection occurs in two rounds: first, perplexity on the test split is used to narrow down candidate models; second, human evaluation on the 100-image qual_dev split selects the most factually aligned model, which is then retrained on the union of train and test splits before final evaluation. This two-stage selection is necessary because perplexity—a standard language modeling metric—does not directly measure factual accuracy, and human evaluation is too expensive to run on all candidate models.

Spatial Reasoning (VSR; Section 4.7, Appendix B.6). The VSR benchmark tests whether a statement about spatial relationships (e.g., "the cat is to the left of the dog") is true or false for a given image. The benchmark is specifically designed to overcome issues with machine-generated annotations by including challenging cases like negations. PaliGemma 2 is fine-tuned on VSR's training split with a hyperparameter sweep covering learning rates {0.1, 0.2, 1.0, 3.0} × 10⁻⁶, weight decay {0.1, 0.3, 1.0} × 10⁻⁶, dropout {0.0, 0.1, 0.2}, and training epochs {1, 3, 5, 10, 15, 30}, with batch size 1024 on 64 chips.

Radiography Report Generation (Section 4.8, Appendix B.7). The task is to generate the Findings and Impressions sections of a radiology report given a chest X-ray image and the Indications section as a prefix. Dataset: MIMIC-CXR (377k images from 228k studies). The authors apply an important preprocessing step: using Gemini 1.5 Pro to remove mentions of prior X-rays from the reports, since the model has no access to prior images—this prevents the model from hallucinating comparisons to unavailable data.

The primary evaluation metric is RadGraph F1 score (Jain et al., 2022), which extracts clinical entities and their relationships from both the reference and generated reports, then computes F1 between the extracted graphs. This metric is specifically designed to capture clinically meaningful content (presence/absence of findings and their anatomical relationships) rather than surface-level string similarity. The initial exploration at 448px² finds that 8 epochs with learning rate 5·10⁻⁶, no label smoothing/dropout/weight decay, and greedy decoding works well; subsequent sweeps for higher resolutions and model sizes explore learning rates in {0.03, 0.1, 0.3, 1.0, 5.0} × 10⁻⁴.

Object Detection Evaluations (Appendix C)

While not one of the main new tasks, the paper includes an important ablation in Appendix C: testing how PaliGemma 2 performs on classical object detection (MS COCO and DocLayNet document layout detection) using a Pix2Seq-style sequence augmentation approach.

The method works as follows. The model receives the prefix "detect all classes\n". The target sequence lists all ground-truth bounding boxes with coordinates and class names in random order. The target is then padded to the maximum sequence length with "noise boxes"—random coordinates with a special <noise> class token. During training, loss is applied to the class tokens of noise boxes but not to their coordinate tokens. This trains the model to output a fixed number of boxes and to learn that <noise> tokens indicate "no object here." During inference, the model samples autoregressively while suppressing the <noise> and <EOS> tokens from sampling; the likelihood assigned to class tokens (vs. <noise>) serves as a confidence score.

Results in Table 11 show that while PaliGemma 2 10B at 896px² achieves 43.6 mAP on COCO—roughly on par with prior sequence-based approaches like Pix2Seq—it substantially lags behind specialized detection architectures like ViTDet. On DocLayNet, the gap to specialized detectors like YOLOv11 (79.5 mAP vs. PaliGemma 2's 66.0) is even larger. The authors hypothesize that "the limiting factor is not the model's intrinsic object understanding" (since it performs well on VQA and referring expressions) but rather "a mismatch between the Average Precision metric, which rewards large numbers of predictions and accurate confidence scores, and the language modeling objective." This is an important negative result that establishes boundaries on the "versatile interface" claim: general-purpose VLMs without task-specific architectural modifications can handle structured prediction tasks, but they are not yet competitive with purpose-built detectors on detection-specific metrics.

Quantization and CPU Inference (Section 4.9)

The paper evaluates PaliGemma 2's feasibility for on-device CPU inference using gemma.cpp, a lightweight C++ inference engine supporting 8-bit switched-floating-point quantization. The evaluation has two components: quality preservation and inference speed.

Quality after Quantization (Table 10). A PaliGemma 2 3B (224px²) model fine-tuned on five datasets (COCOcap, TextCaps, AI2D, OKVQA, DocVQA) is evaluated in Jax with 32-bit floating-point weights and in gemma.cpp with 8-bit quantization. The metric values are virtually identical across all five tasks—relative metric values of 99.9–100.2%—indicating that the 8-bit quantization introduces no practical quality degradation. The memory footprint shrinks from 12.1 GB (f32) to 4.0 GB (quantized), making the model feasible for consumer devices.

CPU Inference Speed (Table 9). Measurements on four processor architectures (Apple M1 Max, Apple M3 Pro, AMD Milan, AMD Genoa) with varying thread counts (4–32) show:

Prefill (260 tokens): 0.17–1.6 seconds depending on hardware.
Extend (per token): 0.27–0.9 seconds.
Total throughput: 12–41 tokens/second, with the AMD Genoa at 32+1 threads achieving the fastest speeds (323 tokens/sec for prefill, 41 tokens/sec for extend).

These measurements are for a specific prompt/image/response combination and batch size 1, providing a rough estimate of what users can expect on different hardware tiers. The authors are careful to note that these numbers are for "this particular setting" and that alternative CPU inference engines (llama.cpp, XNNPack) may offer different performance characteristics.

Experimental Design for the Scaling Analysis

The scaling analysis in Section 4.1 relies on a specific experimental protocol designed to produce clean, interpretable comparisons.

Model grid. The analysis considers a 3×2 grid: three model sizes (3B, 10B, 28B) × two resolutions (224px², 448px²). Each cell in this grid is fine-tuned independently on each of the 30+ academic benchmarks. The 896px² resolution is not included in the systematic grid comparison, likely because the computational cost of running all 30+ tasks × 5 runs × learning rate sweeps at 896px² would be prohibitive.

Relative improvement computation. Figure 3 and Figure 13 express improvements as relative changes rather than absolute differences. For a metric M, the relative improvement from configuration A to configuration B is computed as (M_B - M_A) / M_A × 100%. Since metrics have different scales and directions (CIDEr ranges 0–200+, accuracy ranges 0–100), this normalization enables cross-task comparison on a common scale. Figure 13 also shows "error reduction": (E_A - E_B) / E_A × 100% where error E = (max_metric - M). The error reduction perspective is more appropriate for saturated benchmarks—for example, ScienceQA's 2.2% relative improvement from 3B to 10B at 224px² (96.1 → 98.2) corresponds to a 53.8% error reduction, revealing that the small absolute gain represents a substantial reduction in remaining mistakes.

Task clustering. Figure 3 groups tasks into three clusters based on whether relative improvement is larger from model size increase (blue, yellow markers above the diagonal) or from resolution increase (yellow markers below the diagonal) or both (green markers near the diagonal). The clustering is not derived algorithmically but through visual inspection of where markers fall relative to the parity line. The text- and document-focused tasks (DocVQA, TextVQA, ST-VQA, InfoVQA, ChartQA) cluster in the resolution-sensitive group; multilingual and visual reasoning tasks (XM3600, xGQA, AI2D, CountBenchQA, NLVR2) cluster in the model-size-sensitive group.

Variance reporting. All results in Table 13 include standard deviations over 5 fine-tuning runs with different random seeds. This is important because fine-tuning variance can be substantial for some tasks—for instance, MARVL (avg5) at 3B 448px² has a mean of 82.7 with a standard deviation of 0.3, while at 10B 448px² it is 89.1 with 0.0 standard deviation, indicating not only improved mean performance but also more stable convergence for the larger model.

Checkpoint selection for transfer. A critical detail: for 224px² transfer, the model is fine-tuned from the Stage 1 checkpoint. For 448px² and 896px² transfer, the model is fine-tuned from the Stage 2 checkpoint (which includes both the Stage 1 pretraining and the high-resolution continuation). This means the higher-resolution models have seen more total training data (Stage 1's 1 billion examples + Stage 2's 50–60 million), which could confound the "pure resolution" effect with a "data quantity" effect. The paper does not ablate this, but the Stage 2 data is relatively small compared to Stage 1 (50M vs. 1B), suggesting the resolution effect primarily comes from the architectural change (more vision tokens) rather than the incremental data.

4. Key Insights and Innovations

Innovation 1: A Controlled Scaling Framework That Disentangles Model Size from Resolution

The paper's most fundamental scientific contribution is not any single result, but the experimental framework itself—a grid of models spanning three sizes and three resolutions, all sharing an identical vision encoder, training recipe, and data mixture. This enables causal attribution of performance differences to specific factors in a way that prior work could not achieve.

Prior to PaliGemma 2, the literature on VLM scaling suffered from a fundamental confound. Studies that varied language model size typically did so by borrowing models from different labs—Cambrian-1 (Tong et al., 2024) compared VLMs built with LLaMA, Vicuna, and Gemma backbones, each with its own architecture, pretraining data, and optimization procedure. When performance changed with "model size," it was impossible to determine whether the improvement came from parameter count or from, say, better pretraining data in one model family. Similarly, studies on resolution scaling (Laurençon et al., 2024) typically held the language model constant but could not disentangle resolution effects from the coincident changes in training data distribution that high-resolution training inevitably introduces. The authors explicitly call out this gap: "We are not aware of prior work which jointly studies the effect of the image resolution and the size of the language models on transfer via fine-tuning."

PaliGemma 2 resolves this by construction. Every model in the family uses the same SigLIP-So400m vision encoder, the same three-stage training recipe, the same optimizer, the same data mixture (up to resolution-dependent reweighting in Stage 2), and the same transfer protocol. The only variables are the Gemma 2 language model size (2B, 9B, 27B) and the input resolution (224px², 448px², 896px²). When Figure 3 reveals that TextVQA benefits far more from increasing resolution (4.6× cost) than from increasing model size from 3B to 10B (3.7× cost), we can attribute this to the nature of the task—text at 224px² is simply illegible—rather than to confounding differences in training recipes. Conversely, when AI2D and NLVR2 benefit more from model size than resolution, we can attribute this to their demands on reasoning and world knowledge.

The significance of this framework extends beyond the paper's specific findings. It establishes a methodology for future VLM scaling studies: if you want to make claims about what drives performance, you must control for everything except the factor you are studying. The paper's task clustering (Figure 3)—identifying resolution-sensitive tasks, model-size-sensitive tasks, and tasks sensitive to both—is a direct product of this controlled framework and provides actionable guidance that practitioners can use when choosing model variants for new tasks: document understanding tasks should prioritize resolution; reasoning tasks should prioritize model size.

This is a fundamental methodological advance, not an incremental refinement. It transforms the study of VLM scaling from a collection of loosely controlled comparisons into a systematic experimental science. The cost of this rigor is that the findings are specific to the SigLIP-So400m + Gemma 2 combination—different vision encoders or language model families might exhibit different scaling patterns—but the template is transferable.

Innovation 2: The Identification of Task-Dependent Compute Allocation as the Central Deployment Decision

Prior work on VLMs for transfer treated the choice of model as a univariate optimization problem: "which model performs best on average across benchmarks?" PaliGemma 2 reframes this as a task-conditional resource allocation problem where the optimal choice depends on the specific bottleneck of the target task. This is a conceptual shift from "which model is best?" to "given my task's characteristics, where should I spend my compute budget?"

The evidence for this reframing comes from the systematic asymmetry in Figure 3 and Figure 4. When the 3B model at 224px² is upgraded to 10B at 224px² (3.7× more FLOPs), performance on AI2D improves from 74.7 to 83.1 (an 11.2% relative gain). But when the same 3B model is instead upgraded to 448px² (4.6× more FLOPs), AI2D improves only to 76.0 (a 1.7% relative gain). For DocVQA, the pattern reverses: the resolution upgrade yields 73.6 (from 39.9, an 84.5% gain), while the model size upgrade yields only 43.9 (a 10.0% gain). The difference is stark and task-defining: AI2D's bottleneck is reasoning over visual relationships (which benefits from a stronger language model), while DocVQA's bottleneck is the legibility of small text (which requires higher resolution to capture).

This framing has direct practical consequences that the paper makes explicit. A practitioner working on document understanding should, given a fixed compute budget, prefer to increase resolution rather than language model size. The paper quantifies the tradeoff: going from 3B 224px² to 3B 448px² costs 4.6× per example and provides massive gains on OCR tasks; going from 3B 224px² to 10B 224px² costs 3.7× and provides gains primarily on reasoning tasks. The decision is not academic—it translates directly to training time, hardware costs, and deployment latency.

The paper further refines this insight by showing that the 10B to 28B jump often yields diminishing returns (Figure 4). Many benchmarks plateau or show only marginal improvement from 10B to 28B, particularly those already saturated. This introduces a second-order consideration: even among tasks that benefit from model size, the marginal return on additional parameters may not justify the cost. The paper speculates that the distilled training of the 9B Gemma 2 model (compared to the from-scratch training of the 27B) may partially explain this pattern, but regardless of the cause, the practical implication is that 10B represents a sweet spot for many applications.

This is an incremental conceptual advance built on the controlled framework—it would not be possible without the disentanglement described in Innovation 1—but it shifts the practitioner's mental model from "buy the biggest model you can afford" to "buy the model whose scaling dimension matches your task's bottleneck." It provides a diagnostic vocabulary for reasoning about VLM deployment decisions that did not exist before.

Innovation 3: The Discovery That Optimal Transfer Learning Rate Decreases with Model Size

This finding is deceptively simple but has significant practical and theoretical implications. Figure 5 visualizes a consistent diagonal pattern across nearly all 30+ academic benchmarks: the optimal transfer learning rate for the 28B model is systematically lower than for the 10B model, which is systematically lower than for the 3B model. The paper summarizes: "As a general trend we observe that the optimal learning rate for larger models tends to be lower than for smaller models."

Prior to this work, the original PaliGemma established a standard learning rate sweep range for fine-tuning, but did not investigate size-dependence. Practitioners upgrading from a 3B to a 10B or 28B model might carry forward the same learning rate sweep range, unknowingly using suboptimal hyperparameters and leaving performance on the table. The paper demonstrates that this is not a minor effect: in Figure 5, many tasks show substantial performance degradation when the learning rate is too high for the model size (the lighter-colored cells in the upper-right of each heat map).

The significance extends beyond the practical recommendation ("sweep smaller learning rates when increasing model size"). It suggests a deeper principle about how model scale interacts with transfer dynamics. Larger models, having been pretrained with smaller learning rates (recall from Section 3 that Stage 1 uses 5·10⁻⁶ for 10B/28B vs. 1·10⁻⁵ for 3B), may have weight configurations that are more sensitive to perturbation—their pretrained features are more precisely tuned and require gentler adaptation during fine-tuning. Alternatively, larger models may have lower intrinsic loss landscape curvature, making aggressive optimization steps more likely to overshoot good minima. The paper does not resolve the mechanism, but the empirical regularity is strong enough to warrant inclusion as a finding.

This is an incremental but practically important discovery. It does not change the theoretical understanding of transfer learning, but it provides actionable guidance that could save practitioners from systematic underperformance when scaling up their VLM deployments. The fact that it emerges from the controlled framework—where model size is the only variable changing, not architecture or recipe—lends it credibility that prior anecdotal observations lacked.

Innovation 4: General-Purpose VLMs Can Match or Exceed Specialized Architectures—With Sharp Boundaries

PaliGemma 2 demonstrates that a VLM with no task-specific architectural components can achieve state-of-the-art results on specialized structured prediction tasks that have traditionally required purpose-built models. The 3B model at 896px² outperforms the specialized HTS model (Long et al., 2024) on ICDAR'15 Incidental text detection (75.9 vs. 74.5 F1) and Total-Text (74.2 vs. 72.4 F1). On table structure recognition, it sets a new state of the art on most metrics for both FinTabNet and PubTabNet. On molecular structure recognition, it exceeds the specialized MolScribe (94.8% vs. 93.8% exact match). On music score recognition, it dramatically reduces error rates compared to the specialized Sheet Music Transformer. On radiography report generation, it achieves the highest RadGraph F1 score (29.5 for the 10B model).

This is significant because the dominant paradigm in the OCR, table recognition, and document understanding communities has been to design task-specific architectures with custom detection heads, region proposal networks, or structured output decoders. The paper emphasizes this contrast: "We emphasize that this result is obtained simply by fine-tuning a general-purpose VLM which does not rely on task-specific architecture components as common in the OCR literature." The implication is that the versatility of the text-to-text interface—combined with sufficient pretraining on diverse multimodal tasks—can substitute for architectural specialization, at least for tasks where the output format can be serialized as text (HTML, SMILES, **kern).

However, the paper identifies a critical boundary condition that qualifies this insight. In Appendix C, PaliGemma 2 is evaluated on classical object detection (MS COCO, DocLayNet) and substantially underperforms specialized detectors like ViTDet and YOLOv11. On COCO, the 10B 896px² model achieves 43.6 mAP—respectable for a general-purpose VLM but far below the state of the art. On DocLayNet, the gap is even larger: 66.0 mAP vs. 79.5 for YOLOv11. The authors provide a specific diagnosis: "the limiting factor is not the model's intrinsic object understanding" but rather "a mismatch between the Average Precision metric, which rewards large numbers of predictions and accurate confidence scores, and the language modeling objective."

This negative result is as informative as the positive ones. It reveals that the versatility of the text interface breaks down when the task demands fine-grained confidence calibration over many predictions—language models are trained to maximize likelihood of the correct sequence, not to output well-calibrated confidence scores for structured outputs. Task-specific architectures that directly optimize for detection metrics maintain an advantage in this regime. The boundary is not about output format (bounding boxes CAN be serialized as location tokens) but about the alignment between the training objective and the evaluation metric.

This is a fundamental empirical finding with theoretical implications. It delineates the scope of the "versatile interface" claim: general-purpose VLMs can dominate specialized models on tasks where the output is a single structured object (a table, a molecule, a music score) or a set of predictions evaluated by exact match (text detection under HierText protocol), but they falter on tasks requiring dense predictions with calibrated confidence scores evaluated by ranking-based metrics. This provides a diagnostic for practitioners: if your task's evaluation protocol aligns with exact-match-style comparison, a fine-tuned VLM is likely competitive; if it relies on precision-recall curves over confidence-ranked predictions, a specialized architecture is probably still necessary.

5. Experimental Analysis

Evaluation Methodology

Dataset. The evaluation spans two categories of tasks. The first category consists of 30+ academic benchmarks inherited from PaliGemma (Beyer et al., 2024), covering captioning (COCOcap, NoCaps, TextCaps, SciCap, Screen2Words, WidgetCap), visual question answering (VQAv2, OKVQA, AOKVQA, GQA, VizWizVQA, ScienceQA, TallyQA, CountBenchQA), document/infographic VQA (DocVQA, InfoVQA, ChartQA, ST-VQA, TextVQA, OCR-VQA), referring expression comprehension and segmentation (RefCOCO, RefCOCO+, RefCOCOg), multilingual tasks (XM3600, xGQA, MARVL), video tasks (ActivityNet-QA/CAP, MSRVTT-QA/CAP, MSVD-QA, VATEX), remote sensing VQA (RSVQA), and visual reasoning (AI2D, NLVR2). All tasks use established train/validation/test splits; the exact splits, preprocessing, and metrics follow the PaliGemma protocol (Beyer et al., 2024, Section 3.2.4). The second category consists of 7 new specialized tasks introduced in this paper: text detection and recognition (ICDAR'15 Incidental, Total-Text test sets), table structure recognition (FinTabNet and PubTabNet test splits), molecular structure recognition (5.7k ChemDraw-rendered molecules from MolScribe; Qian et al., 2023), optical music score recognition (GrandStaff test split; Ríos-Vila et al., 2023), long fine-grained captioning (DOCCI qual_dev split of 100 images; Onoe et al., 2024), spatial reasoning (VSR zeroshot and random test splits; Liu et al., 2023), and radiography report generation (MIMIC-CXR test split; Johnson et al., 2019).

Base model(s). All experiments use PaliGemma 2 base models—the family of VLMs built by pairing the SigLIP-So400m vision encoder (Alabdulmohsin et al., 2023; Zhai et al., 2023) with Gemma 2 language models (Gemma Team, 2024) at three sizes: 2B (yielding 3.0B total), 9B (9.7B total), and 27B (27.7B total). For the 30+ academic benchmarks, all six combinations of model size (3B, 10B, 28B) and resolution (224px², 448px²) are evaluated, with the 224px² models fine-tuned from Stage 1 checkpoints and the 448px² models from Stage 2 checkpoints. For new specialized tasks, the evaluation primarily focuses on the 3B model at 896px² (for tasks where resolution is the bottleneck, like OCR and table recognition) or sweeps across sizes and resolutions to find the optimum. The choice of this model family is deliberate: it provides controlled variation in language model size (2B → 9B → 27B) while holding the vision encoder, training recipe, and data mixture constant, enabling causal attribution of performance differences to model scale rather than confounding factors.

Metrics. Task-specific metrics are used throughout, with no single headline number aggregating across tasks. For the 30+ academic benchmarks: CIDEr (captioning tasks: COCOcap, NoCaps, TextCaps, SciCap, Screen2Words, WidgetCap), accuracy (VQA tasks: VQAv2, OKVQA, AOKVQA, GQA, VizWizVQA, ScienceQA, TallyQA, CountBenchQA, DocVQA, InfoVQA, ChartQA, ST-VQA, TextVQA, OCR-VQA, RSVQA), accuracy with IoU thresholding (referring expression tasks: RefCOCO, RefCOCO+, RefCOCOg), and task-specific metrics following PaliGemma for video and multilingual tasks. For new tasks: word-level precision, recall, and F1 with IoU ≥ 0.5 and exact transcription match for text detection (following the HierText competition protocol; Long et al., 2023); Tree Edit Distance Similarity (TEDS) and Grid Table Similarity (GriTS) with sub-metrics for topology and content for table structure recognition (Zhong et al., 2020; Smock et al., 2022); exact match percentage for molecular structure recognition (SMILES string comparison; Qian et al., 2023); Character Error Rate (CER), Symbol Error Rate (SER), and Line Error Rate (LER) for music score recognition (Ríos-Vila et al., 2024); percentage of Non-Entailment Sentences (NES) from human evaluation for DOCCI long captioning, where five raters per sentence classify the relationship as Entailment, Neutral, Contradiction, or Nothing to assess (following Rashkin et al., 2023); accuracy on True/False classification for VSR spatial reasoning; and RadGraph F1 score for radiography report generation, plus standard NLG metrics (CIDEr, BLEU-4, ROUGE-L) as supplementary (Jain et al., 2022). For the object detection analysis in Appendix C, mean Average Precision (mAP) is used for COCO and DocLayNet.

Baselines. For the 30+ academic benchmarks, the primary baseline is PaliGemma 3B (Beyer et al., 2024) at matching resolutions (224px² and 448px²), since PaliGemma 2 is an upgrade using the same architecture with an improved language model. For new specialized tasks, baselines are state-of-the-art task-specific models from the literature:

Text detection: HTS (Hierarchical Text Spotter; Long et al., 2024), which uses a specialized detection and recognition architecture.
Table structure recognition: The state of the art varies by metric and dataset, with baselines drawn from VAST (Huang et al., 2023), MCL (Kawakatsu, 2024), TED-Net (Ly and Takasu, 2023), and the alignment-refined baselines from Smock et al. (2023), each achieving the best reported number on specific metric-dataset combinations.
Molecular structure recognition: MolScribe (Qian et al., 2023), a task-specific image-to-graph model trained on the same PubChem data.
Music score recognition: Sheet Music Transformer (Ríos-Vila et al., 2024), a specialized transformer for optical music recognition.
Long captioning (DOCCI): MiniGPT-4 (7B), mPLUG-Owl2 (8B), InstructBLIP (7B; Dai et al., 2023), LLaVA-1.5 (7B; Liu et al., 2023), VILA (7B), PaliGemma 3B (448px²), and PaLI-5B (448px²). These baselines span general-purpose instruction-tuned VLMs and PaliGemma-family models fine-tuned on DOCCI.
Spatial reasoning (VSR): InstructBLIP in zero-shot (Dai et al., 2023) and LXMERT fine-tuned (Tan and Bansal, 2019), plus the human performance ceiling of 95.4% accuracy (Liu et al., 2023).
Radiography report generation: Flamingo-CXR (Tanno et al., 2024) and Med-Gemini-2D (Yang et al., 2024), both specialized medical VLMs.

Generation budget / compute accounting. The paper measures compute in two distinct regimes. For the scaling analysis (Sections 4.1, Figures 3–4), the relevant "compute budget" is the relative training cost per example during the multimodal pretraining stages, quantified in Table 1 as multiples of the 3B 224px² baseline: 3B at 224px² costs 1.0×, 3B at 448px² costs 4.6×, and 3B at 896px² costs 23.5× per example. These costs capture the full FLOPs spent on vision encoding and language model forward/backward passes, incorporating the quadratic complexity of self-attention over the vision token sequence. For the transfer (fine-tuning) stage, the budget is implicit in the training steps and batch sizes specified per task (e.g., 15k steps × batch size 256 for text detection), but the paper does not normalize transfer costs across tasks—each task is treated independently, and the focus is on final performance rather than transfer efficiency. For inference (Section 4.9), the metric is wall-clock time and tokens/second on specific CPU hardware, not FLOPs, with measurements taken for a specific prompt/image/response configuration on gemma.cpp.

Cross-validation / statistical protocol. For the 30+ academic benchmarks, the protocol is: (1) sweep learning rates at 224px² for each model size independently over {0.03, 0.06, 0.1, 0.3, 0.6, 1.0, 3.0} × 10⁻⁵; (2) select the best learning rate based on the validation split performance for each (task, model size) pair; (3) reuse the selected learning rate for the 448px² version (no additional sweep); (4) retrain with the selected learning rate and report the test metric as the mean and standard deviation over 5 fine-tuning runs with different random seeds. This protocol provides estimates of both central tendency and run-to-run variability. For the new specialized tasks, learning rate selection is done on the respective validation splits, but the number of runs is typically not specified beyond the single best configuration. For the DOCCI human evaluation, each generated sentence is rated by 5 independent raters, and the majority vote determines the label (entailment vs. non-entailment); the overall binary agreement among raters is 0.8407, meaning all 5 raters agree on the "Entailment" category 84.07% of the time. The two-fold cross-validation protocol described in the original PaliGemma is not explicitly reused here; for the scaling analysis, since strategies are not being selected per-bin (unlike the compute-optimal allocation in the reference paper), cross-validation across difficulty bins is not applicable—the model grid itself serves as the experimental design, with each cell independently fine-tuned and evaluated.

Main Quantitative Results

Scaling Analysis: Effect of Model Size and Resolution on Transfer Performance

Headline finding: Tasks cluster into three groups based on whether they benefit more from increased resolution or increased language model size, and the 10B model represents a practical sweet spot before diminishing returns set in at 28B.

Figure 3 presents the relative improvement when upgrading PaliGemma 2 3B (224px²) in two directions: increasing the language model to 10B while keeping 224px² resolution (3.7× more FLOPs per example, x-axis, plotted on a log scale from 0.1% to 100%), versus increasing resolution to 448px² while keeping the 3B model (4.6× more FLOPs, y-axis). Each marker is a task; its position relative to the diagonal parity line (where the two upgrades provide equal relative benefit) determines its cluster assignment.

Resolution-sensitive tasks (yellow markers, falling below the diagonal—meaning the vertical 448px² gain is larger than the horizontal 10B gain) include: DocVQA (val), TextVQA (val), ST-VQA (val), InfoVQA (val), ChartQA (human), WidgetCap, and TallyQA (simple). For DocVQA, the resolution upgrade yields an 84.5% relative improvement (39.9 → 73.6, Table 13), while the 10B upgrade yields only 10.0% (39.9 → 43.9). For TextVQA, the resolution upgrade gives a 26.2% gain (59.6 → 75.2) versus 7.4% for the 10B upgrade (59.6 → 64.0). The paper notes that images in these benchmarks "often have a native resolution significantly larger than 224px²," making the bottleneck visual rather than linguistic—small text that is illegible at 256 vision tokens becomes readable at 1024.

Model-size-sensitive tasks (blue markers, above the diagonal—horizontal 10B gain exceeds vertical resolution gain) include: AI2D (11.2% relative gain from 10B vs. 1.7% from 448px²), CountBenchQA (3.7% vs. 1.2%), NLVR2 (2.7% vs. 0.2%), XM3600 (avg35) (4.0% vs. 0.9%), and xGQA (avg7) (4.8% vs. 3.1%). The paper interprets this cluster as tasks requiring "advanced visual reasoning" (AI2D, CountBenchQA, NLVR2) or "multilingual data" (XM3600, xGQA)—capabilities that benefit from a larger language model's increased world knowledge and reasoning capacity. Notably, some of these benchmarks are already quite saturated at 3B 224px² (ScienceQA: 96.1%, NLVR2: 91.4%), so the absolute gains are small, but Figure 13 (right) shows that ScienceQA's 2.1% absolute improvement (96.1 → 98.2) represents a 53.8% error reduction—a substantial proportional reduction in remaining mistakes.

Tasks sensitive to both (green markers, near the diagonal) form the largest cluster and include: AOKVQA-DA, RefCOCO, RefCOCO+, RefCOCOg, VQAv2, GQA, VizWizVQA, OCR-VQA, and several captioning benchmarks. For these tasks, the two upgrades provide roughly comparable relative gains, suggesting no single bottleneck dominates.

Figure 4 provides per-task detail by plotting the absolute metric values for all six combinations (3B, 10B, 28B) × (224px², 448px²), with lighter lines indicating higher resolution. The key pattern is visible across most tasks: the 3B → 10B jump (at either resolution) yields substantial gains, while the 10B → 28B jump often yields marginal or no improvement, with the 28B lines frequently overlapping or only slightly exceeding the 10B lines. For example, on AI2D: 3B 224px² = 74.7, 10B 224px² = 83.1, 28B 224px² = 83.2—a 0.1 point gain from nearly 4× the parameters. On ChartQA (aug): the 28B model at 224px² (68.9) actually underperforms the 10B (74.2) and even the 3B (74.4). At 448px², the 28B (85.1) underperforms the 10B (90.1). The paper speculates this may be related to the fact that "the underlying Gemma 2 27B model is trained from scratch, as opposed to the 2B and 9B models, which are distilled"—the distilled models may have higher per-parameter quality that partially offsets their smaller size, making the 10B a practical sweet spot where additional parameters provide diminishing returns.

Headline finding: PaliGemma 2 3B slightly outperforms PaliGemma 3B at matching resolution, with an average improvement of 0.65 points at 224px² and 0.85 points at 448px².

Table 15 provides the per-task comparison. The improvements are small but consistent: the PaliGemma 2 models achieve better scores on the majority of tasks at both resolutions. However, there are regressions on some tasks—for instance, Screen2Words drops from 117.6 (PaliGemma) to 113.3 (PaliGemma 2) at 224px², a decrease of 4.3 points. At 448px², COCOcap drops from 144.6 to 143.4, and TextCaps drops from 153.9 to 152.1. The paper does not investigate the cause of these regressions, but they indicate that the Gemma 1 → Gemma 2 upgrade is not universally beneficial across all fine-tuning tasks—the architecture and pretraining changes in Gemma 2 (logits soft-capping, different distillation procedure, updated training data) may interact differently with different downstream tasks.

Text Detection and Recognition

Headline finding: PaliGemma 2 3B at 896px² outperforms the specialized HTS model on both ICDAR'15 Incidental and Total-Text, with F1 scores of 75.9 (vs. 74.5) and 74.2 (vs. 72.4) respectively.

Table 2 breaks down the results. On ICDAR'15 Incidental, PaliGemma 2 achieves precision 81.9, recall 70.7, F1 75.9; HTS achieves 81.9, 68.4, 74.5. The improvement is driven entirely by recall (+2.3 points) with precision matched exactly—PaliGemma 2 finds more true positives without increasing false positives. On Total-Text, the pattern differs: PaliGemma 2 achieves lower precision (73.8 vs. HTS's 75.7) but substantially higher recall (74.5 vs. 69.4), yielding a net F1 improvement of 1.8 points. This suggests PaliGemma 2 is more aggressive in proposing word candidates, recovering text instances that the specialized detector misses, but at a cost of slightly more false positives on the Total-Text benchmark specifically.

The paper emphasizes that this result is obtained using a resolution of 896px² (4096 vision tokens); reducing resolution "led to substantially lower prediction quality," while increasing model size "did not lead to improvements." This confirms that text detection at these benchmark scales is bottlenecked by visual detail rather than reasoning—once the text is legible, the 2B language model in the 3B variant is sufficient for transcription, and adding more parameters does not help.

Table Structure Recognition

Headline finding: PaliGemma 2 3B at 896px² sets a new state of the art on most evaluation metrics for both FinTabNet and PubTabNet.

Table 3 presents the results against the prior state of the art, where the SOTA numbers are composites from multiple systems (each achieving the best reported result on that specific metric-dataset combination). On FinTabNet, PaliGemma 2 achieves: S-TEDS 99.2 (vs. SOTA 98.9), TEDS 98.9 (vs. 98.2), GriTS-Top 99.4 (vs. 99.0), GriTS-Con 99.2 (vs. 98.6). On PubTabNet: S-TEDS 97.6 (vs. SOTA 97.9), TEDS 97.3 (vs. 96.9), GriTS-Top 98.0, GriTS-Con 97.8 (GriTS SOTA not reported for PubTabNet in the referenced works). PaliGemma 2 exceeds or matches the best prior results on 7 out of 8 reported metric-dataset combinations, with PubTabNet S-TEDS being the sole exception (97.6 vs. 97.9).

The paper notes that using a larger model size "did not lead to additional benefits" while lower resolution "led to a small regression in quality." Like text detection, this task is bottlenecked by the legibility of fine-grained visual detail (table cell text, grid lines at small font sizes), and 896px² provides sufficient resolution. The HTML output format—containing cell bounding boxes encoded as <locDDDD> tokens—demonstrates that structured output can be serialized effectively through the text interface without task-specific decoders. However, the Appendix B.2 notes that "correction of cell bounding box annotations and cell text annotations are applied to FinTabNet training examples using information from the source PDFs, following a similar approach as [86]," meaning the training data quality is enhanced through preprocessing that goes beyond simple fine-tuning on raw dataset annotations.

Molecular Structure Recognition

Headline finding: PaliGemma 2 10B at 448px² achieves 94.8% exact match on ChemDraw molecules, outperforming the specialized MolScribe (93.8%).

Table 4 shows the single-number comparison. The 1.0 percentage point improvement is modest but meaningful: it demonstrates that a general-purpose VLM can match and slightly exceed a task-specific model that uses graph generation with chemical validity constraints. The result is obtained at 448px² resolution; "further increasing the resolution did not lead to a higher exact match percentage," suggesting that the bottleneck at 448px² is no longer visual—molecular structures at this resolution are already sufficiently detailed—but may be in the language model's ability to correctly generate syntactically valid SMILES strings. The use of the 10B model rather than the 3B (which might be expected if the task is purely visual) hints that the 9B language model's stronger sequence modeling capabilities provide a small but measurable benefit, potentially in maintaining SMILES syntax consistency over long sequences.

The training data preparation is worth noting: 1 million PubChem molecules are rendered with "a variety of drawing styles and random perturbations" using Indigo, following the MolScribe augmentation protocol. This augmentation is likely critical—molecular drawings in the real world exhibit substantial stylistic variation (font choices, bond line thickness, label positioning conventions), and training on diverse renderings prevents the model from overfitting to a specific visual style. The evaluation set, however, consists exclusively of ChemDraw-rendered molecules (a specific, professional rendering style), which may be more uniform than the training data, potentially making the evaluation somewhat easier than a truly in-the-wild test.

Optical Music Score Recognition

Headline finding: PaliGemma 2 3B at 896px² reduces the Character Error Rate from 3.9% (Sheet Music Transformer) to 1.6%, a 59% relative reduction, with similar improvements across SER and LER.

Table 5 shows the error rates at three granularities: CER drops from 3.9% to 1.6%, SER from 5.1% to 2.3%, and LER from 13.1% to 6.7%. The error reduction is largest at the character level (59%) and narrows at the line level (49%), consistent with the interpretation that character-level recognition benefits most directly from higher resolution, while line-level errors may involve longer-range structural dependencies (e.g., getting the barline structure correct across an entire musical phrase) that are harder to resolve purely through better visual input.

The paper notes that error rates "decrease with increasing resolution, with the best error rates obtained at 896px² resolution" and that "increasing the model size from 3B to 10B did not lead to further error reduction." This is another clear case of a resolution-bottlenecked task: reading small musical notation symbols (note heads, articulation marks, dynamic markings) requires high pixel density, and once the symbols are legible, even a 2B language model can produce correct **kern encodings.

Long, Fine-Grained Caption Generation (DOCCI)

Headline finding: PaliGemma 2 10B at 448px² achieves the lowest Non-Entailment Sentence rate (20.3%) among all tested models, substantially better than instruction-tuned VLMs like LLaVA-1.5 (40.6%) and VILA (28.6%), and better than PaliGemma 3B (34.3%).

Table 6 compares models along two dimensions: average caption length (in characters and sentences) and the percentage of Non-Entailment Sentences (NES, where lower is better). PaliGemma 2 3B generates captions averaging 529 characters / 7.7 sentences, with an NES of 28.4%—an improvement over PaliGemma 3B (535 characters / 8.9 sentences, NES 34.3%), generating slightly shorter but more factually accurate captions. The 10B model reduces NES further to 20.3% while maintaining similar length (521 characters / 7.5 sentences), indicating that the larger language model improves factual grounding without inducing verbosity.

The comparison against instruction-tuned baselines is instructive: VILA (7B) generates the longest captions (871 characters / 8.6 sentences) with 28.6% NES, while LLaVA-1.5 (7B) generates shorter captions (395 characters / 4.2 sentences) with 40.6% NES. PaliGemma 2's advantage likely stems from being fine-tuned specifically on DOCCI (15k human-annotated examples emphasizing factual precision) rather than relying on general instruction-tuning—the DOCCI training data explicitly rewards describing what is in the image without hallucination, while instruction-tuned models are optimized for broad conversational ability and may prioritize fluency over strict factual accuracy. The human evaluation protocol described in Appendix B.5 makes this explicit: each sentence is judged independently for entailment with the image, with 5 raters per sentence and majority voting, meaning the model is penalized for any sentence that a majority of raters considers not entailed, regardless of how fluent or natural the caption reads as a whole.

A notable detail: the model selection occurs in two rounds. First, perplexity on the test split narrows candidate models; then, human evaluation on a 100-image qual_dev split selects the best. The need for this two-stage process reveals that perplexity—a language modeling metric that measures fluency—does not correlate perfectly with factual accuracy. A model could generate extremely fluent text that is factually ungrounded (hallucination) and achieve low perplexity, or it could generate slightly less fluent text that is strictly accurate and achieve higher perplexity. The human evaluation filters for the latter.

Spatial Reasoning (VSR)

Headline finding: PaliGemma 2 10B at 224px² achieves 79.8% on the zeroshot split and 86.8% on the random split, substantially outperforming the fine-tuned LXMERT (70.1%/61.2%) and the zero-shot InstructBLIP (65.6%).

Table 7 compares PaliGemma 2 3B and 10B against baselines. On the zeroshot split (which tests generalization to unseen spatial relation types), PaliGemma 2 3B achieves 74.8% and 10B achieves 79.8%—a 5.0 point improvement from scaling the language model. On the random split (standard i.i.d. train/test), the jump is larger: 81.6% (3B) to 86.8% (10B). The paper notes "significant benefits from larger model size, indicating benefits from improved language understanding, whereas going beyond resolution 224 did not lead to improvements." This cleanly positions VSR as a model-size-sensitive task: the visual features needed to identify objects and their spatial layout are already accessible at 224px² (objects are typically large and clearly visible), but determining whether statements like "the cat is to the left of the dog" are True or False requires compositional reasoning over spatial relations—a capability that scales with language model capacity.

The gap between PaliGemma 2 10B (86.8%) and human performance (95.4%) suggests substantial room for improvement. The 8.6-point gap implies that even the 10B model struggles with certain types of spatial reasoning, possibly negations or complex multi-object relationships that humans resolve effortlessly.

Radiography Report Generation

Headline finding: PaliGemma 2 10B at 896px² achieves a RadGraph F1 score of 29.5%, outperforming Med-Gemini-2D (24.4%) and Flamingo-CXR (20.5%).

Table 8 provides the full comparison, including standard NLG metrics as supplementary measures. On RadGraph F1—the primary clinical metric that captures the presence/absence of findings and their anatomical relationships—PaliGemma 2 3B achieves 28.8% and the 10B reaches 29.5%. The improvement from 3B to 10B is modest (0.7 points), while the improvement over Med-Gemini-2D (5.1 points) is substantial.

The relationship between standard NLG metrics and the clinical metric is noteworthy: PaliGemma 2 3B scores higher CIDEr (19.9 vs. Med-Gemini-2D's 17.5) but lower BLEU-4 (14.6 vs. 20.5), illustrating that these are not perfectly correlated. A model can score well on CIDEr (which rewards n-gram overlap with the reference) while still missing clinically important findings, or vice versa. The RadGraph metric explicitly addresses this by extracting clinical entities and their relationships before computing similarity, making it more aligned with what radiologists care about.

Both increasing resolution (to 896px²) and increasing model size (to 10B) provide "modest improvements." The 896px² resolution is likely necessary because chest X-rays contain fine-grained visual features (small opacities, subtle line patterns) that benefit from high pixel density, while the language model must generate coherent, clinically accurate prose spanning multiple sections (Findings, Impressions).

Object Detection (Appendix C)

Headline finding: PaliGemma 2 matches prior sequence-based detection approaches but substantially underperforms specialized detectors, highlighting the boundary between general-purpose VLMs and task-specific architectures.

Table 11 reports mAP for COCO and DocLayNet across PaliGemma and PaliGemma 2 variants at three resolutions. On COCO, PaliGemma 2 10B at 896px² achieves 43.6 mAP, up from 30.4 for PaliGemma 2 3B at 224px²—a 43% improvement from scaling both resolution and model size. On DocLayNet, the pattern is similar: going from 46.7 mAP (PaliGemma 2 3B at 224px²) to 66.0 mAP (PaliGemma 2 10B at 896px²). Resolution is the dominant factor—the jump from 224px² to 448px² roughly doubles mAP for both datasets—while model size improvements are smaller and inconsistent (on DocLayNet, PaliGemma 3B at 224px² achieves 50.8 vs. PaliGemma 2 3B's 46.7, a rare case where PaliGemma outperforms PaliGemma 2).

These results, while not matching specialized detectors (YOLOv11: 79.5 mAP on DocLayNet), demonstrate that the text-to-text interface can handle dense prediction tasks. The Pix2Seq-style noise box augmentation (Appendix C) is the key enabler, training the model to output a fixed number of predictions with <noise> tokens acting as confidence proxies. However, the AP metric fundamentally rewards models that produce many predictions with well-calibrated confidence scores—something language models trained with cross-entropy loss are not optimized for, as the authors note: "the limiting factor is not the model's intrinsic object understanding... [but] a mismatch between the Average Precision metric, which rewards large numbers of predictions and accurate confidence scores, and the language modeling objective."

CPU Inference and Quantization

Headline finding: 8-bit quantization via gemma.cpp introduces no practical quality degradation (relative metric values of 99.9–100.2%) while reducing memory from 12.1 GB to 4.0 GB for the 3B model.

Table 10 compares five fine-tuned PaliGemma 2 3B (224px²) checkpoints evaluated in Jax with 32-bit weights and in gemma.cpp with 8-bit quantization. The relative metric values are: COCOcap 99.9%, TextCaps 100.2%, AI2D 100.1%, OKVQA 100.1%, DocVQA 99.9%. The tiny deviations (≤ 0.2%) are within the noise of quantization and greedy decoding differences.

Table 9 provides CPU inference speed benchmarks. The fastest configuration (AMD Genoa, 32 threads for prefill + 1 for extend) achieves prefill at 323 tokens/second (0.17 seconds for 260 tokens) and extend at 41 tokens/second (0.27 seconds per token for 11 output tokens). This translates to roughly 1.1 seconds total for the example prompt ("describe this image" → "A large building with two towers on the water"). At the slower end (Apple M1 Max, 4+1 threads), prefill takes 32 tokens/second (8.2 seconds) and extend at roughly 1.1 tokens/second (~12 seconds total). These speeds are for a specific prompt/image; the authors caution that these are representative measurements for "this particular setting" and that alternative inference engines may yield different performance.

Ablation Studies and Robustness Checks

Effect of logits soft-capping in Stage 3 (vs. disabling it): The paper states that logits soft-capping is "not [used] in Stage 3, as this led to worse results for some transfer tasks" (Section 3). This is not a formal ablation with numbers—it is a design decision based on preliminary experiments. The implication is that the regularization that prevents overconfidence during multi-task pretraining actually constrains the model's ability to specialize during fine-tuning, but no quantitative comparison (soft-capping on vs. off across tasks) is provided.

Resolution-dependence of specialized tasks (Sections 4.2–4.5): Each specialized task section includes a brief statement on whether increasing model size or resolution beyond the reported optimum provided further benefit. For text detection: "reducing the resolution... led to substantially lower prediction quality, while increasing the model size did not lead to improvements." For table recognition: "increasing the model size... did not lead to additional benefits, and using a lower image resolution led to a small regression in quality." For molecular recognition: "further increasing the resolution did not lead to a higher exact match percentage." For music score: "error rates decrease with increasing resolution, with the best error rates obtained at 896px². Increasing the model size from 3B to 10B did not lead to further error reduction." These are not controlled ablations in the sense of varying one factor while holding all else equal (since training data, steps, and learning rates differ across configurations), but they serve as informal sensitivity analyses confirming the resolution-vs-size tradeoff identified in the systematic scaling analysis: OCR-style tasks are resolution-bottlenecked, and additional language model capacity does not compensate for insufficient visual detail.

Effect of distillation vs. from-scratch training on Gemma 2 language model quality (Section 4.1.1): The paper speculates that the diminishing returns from 10B to 28B may be due to the 27B Gemma 2 model being "trained from scratch, as opposed to the 2B and 9B models, which are distilled" (referencing Gemma Team, 2024, Section 6.1). This is an observation about a design choice rather than a controlled ablation—there is no 27B model trained with distillation to compare against—but it functions as a caveat for the claim that model size scaling alone drives performance gains. If the 27B model's per-parameter quality is lower due to the training methodology, then the "model size" variable is partially confounded with "training methodology" within the Gemma 2 family.

PaliGemma 1 vs. PaliGemma 2 at matching size and resolution (Table 15): This comparison isolates the effect of upgrading the language model from Gemma 1 to Gemma 2 while holding all else constant (same vision encoder, same training recipe, same data mixture). Across 30+ benchmarks, PaliGemma 2 3B outperforms PaliGemma 3B by an average of 0.65 points (224px²) and 0.85 points (448px²), confirming that the Gemma 2 language model provides a small but consistent benefit. However, the presence of regressions on individual tasks (Screen2Words: −4.3 at 224px²; TextCaps: −1.8 at 448px²) indicates that the upgrade is not universally beneficial and that task-specific interactions with the language model architecture and pretraining matter.

Object detection sequence augmentation approach (Appendix C): The Pix2Seq-style noise box augmentation is ablated implicitly by being the chosen method, but no alternative detection transfer approach (e.g., adding a detection head, using DETR-style set prediction) is explored. The negative result—that general-purpose VLMs substantially underperform specialized detectors on AP-based metrics—is itself the finding, establishing a boundary condition for the "versatile interface" claim.

CPU quantization quality across diverse downstream tasks (Table 10): Five fine-tuning datasets spanning captioning (COCOcap, TextCaps), VQA (AI2D, OKVQA), and document VQA (DocVQA) are evaluated in both f32 and 8-bit quantization. The consistency of relative metric values (all between 99.9% and 100.2%) demonstrates that quantization robustness is not an artifact of a single easy task—it generalizes across task types with different output formats, sequence lengths, and required capabilities. This is a stronger result than if only captioning tasks had been tested, since VQA tasks require more structured outputs (short answers, True/False) that might be more sensitive to quantization noise.

Critical Assessment

The experimental design in PaliGemma 2 is fundamentally sound for its primary goal: characterizing how model size and resolution affect transfer performance when all other factors are held constant. The 3 × 2 grid of systematically varied models, combined with per-task learning rate selection and 5-run variance estimation, provides a level of experimental control that prior VLM scaling studies lacked. However, the paper's claims rest on several assumptions and design choices that warrant scrutiny, and the evidence supporting different claims varies in strength.

Claim: "Increasing image resolution and increasing LM size both lead to an increase in FLOPs... we generally expect most tasks to benefit from both these changes" (Section 4.1.1). This claim is supported with qualifications that the paper makes explicit through its task clustering analysis. The "general expectation" is confirmed for the large cluster of tasks (the green markers in Figure 3) that benefit comparably from both upgrades. However, the resolution-sensitive and model-size-sensitive clusters reveal that the expectation does not hold universally—and these exceptions are the paper's most interesting findings. The qualification is that the "general expectation" would be wrong for practitioners working on document understanding tasks (where model size alone provides minimal benefit) or advanced reasoning tasks (where resolution alone provides minimal benefit). The clustering analysis itself provides the evidence, but the assignment of tasks to clusters is done through visual inspection of Figure 3 rather than through a statistical clustering procedure, leaving some ambiguity—for instance, tasks near the diagonal could be assigned to any of the three clusters depending on threshold choices, and the paper does not specify what proximity to the parity line qualifies a task as "sensitive to both."

Claim: "PaliGemma 2 3B slightly outperforms PaliGemma at the same resolution and model size, and obtains substantial improvements at larger model sizes" (Section 1). The "slight outperformance" part is supported by Table 15: the average improvement of 0.65 (224px²) and 0.85 (448px²) across 30+ tasks is consistent with "slight." The "substantial improvements at larger model sizes" part is supported with mixed strength. For model-size-sensitive tasks, the 3B → 10B jump does provide substantial gains (AI2D: 74.7 → 83.1, +8.4 points; AOKVQA-DA: 64.2 → 68.9, +4.7 points). However, the 10B → 28B jump often provides negligible or zero improvement: AI2D goes from 83.1 to 83.2, COCOcap from 143.7 to 144.0 (+0.3), VQAv2 from 84.3 to 84.5 (+0.2). On several tasks, the 28B model underperforms the 10B (ChartQA aug: 68.9 vs. 74.2; RefCOCO val: 75.0 vs. 75.9). The claim of "substantial improvements at larger model sizes" is true primarily for the 3B → 10B transition, not for the 10B → 28B transition. A reader might interpret "larger model sizes" as referring to all variants larger than 3B, which overstates the evidence for the 28B model specifically.

Claim: The task clustering in Figure 3 identifies which tasks "benefit more from compute due to increased resolutions, and which from compute due to a larger, more capable language model" (Section 1). This claim is supported by the data, but with two important caveats. First, the resolution upgrade (4.6× cost) and the model size upgrade (3.7× cost) are not FLOPs-matched—they burn slightly different amounts of pretraining compute, meaning the comparison slightly favors the cheaper upgrade (10B at 224px²). A reader comparing the two directions should keep in mind that the 10B upgrade costs ~20% less per training example, which could modestly affect the parity line. Second, the clustering is purely correlational—it shows which tasks benefit from which upgrade, but it does not provide mechanistic evidence for why. The paper's interpretations (resolution-sensitive tasks have high native resolution images; model-size-sensitive tasks require reasoning and world knowledge) are plausible inferences supported by qualitative task characteristics, but they are not tested through direct manipulation (e.g., downscaling document images and measuring whether the resolution-sensitivity disappears, or adding world knowledge probes to reasoning tasks).

Claim: "PaliGemma 2 obtains state-of-the-art results on many of those tasks" (Section 1, referring to the new specialized tasks). This claim is supported for the tasks where PaliGemma 2 is directly compared against the prior state of the art: text detection (Table 2: +1.4 F1 on ICDAR'15, +1.8 on Total-Text), table recognition (Table 3: new SOTA on 7/8 metric-dataset combinations), molecular recognition (Table 4: +1.0 exact match), music score recognition (Table 5: substantial error reductions), and radiography report generation (Table 8: +5.1 RadGraph F1 over Med-Gemini-2D). However, the strength of this evidence varies. For text detection, the comparison is against a single prior SOTA model (HTS) on two test sets, and the margins (1.4–1.8 F1) are modest—comfortably above zero but not overwhelming. For table recognition, the "state of the art" is a composite of the best result from different systems on each metric-dataset pair, meaning PaliGemma 2 is compared against an unrealistically strong baseline that no single system achieves simultaneously. For DOCCI captioning, PaliGemma 2 is evaluated against instruction-tuned VLMs that were not specifically trained on DOCCI, making the comparison partially about training data (DOCCI-specific fine-tuning vs. general instruction tuning) rather than purely about model architecture. The human evaluation protocol adds rigor, but the baseline models may have been disadvantaged by not being fine-tuned on the target distribution.

The paper does not report statistical significance tests for any of the SOTA claims. For text detection, where the F1 difference is 1.4–1.8 points, it is unclear whether this exceeds test-retest variability, since only a single fine-tuning run is reported (the 5-run protocol used for academic benchmarks is not applied to the new tasks). For the table recognition S-TEDS on PubTabNet where PaliGemma 2 (97.6) underperforms the prior SOTA (97.9) by 0.3 points, it is similarly unclear whether this is a meaningful regression or noise.

Missing experiments and analysis. Several experiments would have strengthened the paper's claims but were not conducted:

No ablation on the effect of Stage 1 data quantity vs. resolution. The 448px² model receives Stage 1 (1B examples, 224px²) + Stage 2 (50M examples, 448px²), while the 224px² model receives only Stage 1. Performance differences could reflect the additional 50M high-resolution examples rather than resolution per se. An ablation where a 224px² model receives 50M additional 224px² examples would disentangle data quantity from resolution.
No transfer learning rate sweep at 448px². The paper assumes the optimal learning rate at 224px² transfers to 448px² without verification (Section 4.1: "for most tasks the earlier work used the same hyperparameters for 224px² and 448px², we only sweep at 224px² resolution and reuse the selection for both resolutions"). This pragmatic decision is understandable given computational constraints but means the 448px² results may be systematically suboptimal—if the learning rate interacts with resolution, the performance gap between 224px² and 448px² could be understated.
No investigation of why some tasks regress from PaliGemma to PaliGemma 2. Screen2Words drops by 4.3 points at 224px² (Table 15). The paper does not analyze possible causes (diffusion of the Gemma 2 architecture, logits soft-capping effect on the pretrained weights, changes in tokenization, interaction with the optimization setup), leaving the regression as an unexplained anomaly.
No direct Flops-matched comparison of model size vs. resolution at inference time. While Table 1 compares pretraining costs, the CPU inference measurements only cover the 3B 224px² model under gemma.cpp. A comparison of the 3B at 448px² vs. 10B at 224px² on CPU would inform the practitioner's deployment tradeoff directly—is it better to serve high-resolution input to a small model or low-resolution input to a larger model, given the same hardware?
No confidence intervals for the SOTA claims on specialized tasks. The 5-run protocol used for academic benchmarks (Table 13) is not applied to the new tasks in Tables 2–8. Without variance estimates, it is unknown whether PaliGemma 2's advantage over HTS on ICDAR'15 (75.9 vs. 74.5) is reliable or could disappear with a different random seed.
No analysis of the difficulty estimation cost. The paper does not address this, which is reasonable since PaliGemma 2 is not a compute-optimal allocation system that needs difficulty estimates—the scaling analysis is retrospective and the specialized tasks use fixed configurations.

Strengths of the experimental design that are worth noting explicitly:

The 5-run protocol with standard deviation reporting (Table 13) is genuinely valuable and not universal in the VLM literature. It reveals that fine-tuning variance can be non-trivial (e.g., MARVL at 10B 448px²: mean 89.1, std 0.0, but at 3B 448px²: mean 82.7, std 0.3), and it allows readers to assess which differences are likely noise vs. signal.
The use of human evaluation with 5 raters per sentence for DOCCI (Appendix B.5, Figure 12) sets a high standard for caption quality assessment that goes beyond automated metrics. The inter-annotator agreement (0.8407 for "Entailment" agreement) is reported, providing transparency about the reliability of the human evaluation.
The decision to report both relative improvement and error reduction (Figure 13) is methodologically sound, since relative improvement can be misleading for saturated benchmarks. Presenting both metrics lets the reader assess scaling effects from complementary perspectives.
The negative result on object detection (Appendix C) is valuable precisely because it tempers the enthusiasm around general-purpose VLMs. By showing that task-specific detectors still dominate on AP-based metrics, the paper defines the scope of its own versatility claims and prevents overgeneralization.

Overall, the experiments provide compelling evidence for the paper's main scaling insights—that tasks cluster into resolution-sensitive, model-size-sensitive, and both-sensitive groups, and that the 10B model represents a practical sweet spot—while the SOTA claims on specialized tasks are credible but would benefit from variance estimates. The controlled framework is the paper's primary contribution, and the experimental design faithfully executes on that framework.

6. Limitations and Trade-offs

The 896px² Configurations Are Not Included in the Systematic Scaling Analysis

The assumption or constraint. The paper's central scaling analysis in Section 4.1—spanning 30+ academic benchmarks across all model sizes—considers only the 224px² and 448px² resolutions. The 896px² resolution is evaluated exclusively on the new specialized tasks (Sections 4.2–4.8) and on object detection (Appendix C), but is absent from the grid that produced Figures 3–5 and Table 13. The paper never explicitly states why, but the cost is clearly prohibitive: Table 1 shows that 896px² training costs 23.5× the 224px² baseline for the 3B model, and approximately 155.6× for the 28B model. Running the full benchmark suite—30+ tasks × 5 runs × learning rate sweeps—at 896px² across all model sizes would represent a massive computational investment on top of the already-substantial grid evaluated.

The consequence. The paper's key insights about task clustering (resolution-sensitive vs. model-size-sensitive) and the diminishing returns at 28B are derived from the 224px² and 448px² data only. We cannot know whether the clustering pattern generalizes to 896px². For instance, tasks classified as "resolution-sensitive" at the 224px² → 448px² transition (DocVQA, TextVQA, InfoVQA) may continue to benefit from further resolution increases to 896px², or they may saturate at 448px² and become model-size-sensitive thereafter, or they may show qualitatively different behavior. Similarly, the finding that "the 10B to 28B jump often only leads to moderate improvements, or no improvements at all" (Section 4.1.1) may partially reflect that these comparisons are made at 224px² and 448px²—it is possible that the 28B model's advantages become apparent only at 896px², where the increased vision token count places greater demands on the language model's capacity to process long sequences and reason over dense visual information. Without 896px² data, the paper's scaling recommendations are truncated: a practitioner considering the full resolution range cannot extrapolate with confidence from the 224px² → 448px² trends.

What evidence exists in the paper. The specialized tasks (text detection, table recognition, music score recognition) provide anecdotal evidence that 896px² matters for resolution-intensive applications—the best results on these tasks are consistently achieved at 896px². However, these tasks are selected precisely because they benefit from resolution; they represent a biased sample. The 30-task clustering in Figure 3 includes several tasks (COCOcap, VQAv2, OKVQA) that show roughly equal sensitivity to resolution and model size at 224px² → 448px², and their behavior at 896px² is unknown. The paper provides no data to characterize where diminishing returns to resolution set in, or whether some tasks that appear model-size-sensitive at lower resolutions become resolution-sensitive at 896px².

Mitigation status. The paper does not address this gap directly. The absence of 896px² from the scaling analysis is a pragmatic decision driven by computational constraints, but the paper never acknowledges it as a limitation on the generality of the scaling findings. A reader could easily overlook that the headline clustering results and model-size recommendations are derived from only two-thirds of the available resolution range. Future work could mitigate this by evaluating a representative subset of benchmarks at 896px² across model sizes—not the full 30+, but perhaps 8–10 tasks spanning the three clusters—to determine whether the scaling patterns observed at lower resolutions extend to the highest resolution.

The 224px²-to-448px² Comparison Is Not a Pure Resolution Effect

The assumption or constraint. When comparing 224px² and 448px² model performance (Figure 3, Table 13), the paper interprets the difference as the effect of "increasing the resolution" (Section 4.1.1: "Increasing image resolution and increasing LM size both lead to an increase in the FLOPs"). However, two confounding factors are present. First, the 448px² models are fine-tuned from Stage 2 checkpoints, which have seen 50 million additional training examples at 448px² beyond the Stage 1 data (1 billion examples at 224px²). The 224px² transfer models are fine-tuned from Stage 1 checkpoints only. Performance differences therefore conflate the architectural change (more vision tokens, enabling finer visual detail) with a data quantity difference (the Stage 2 models have seen 5% more training data, specifically curated to upweight resolution-benefiting tasks). Second, the transfer learning rate for 448px² is not independently selected—the paper "only sweep[s] at 224px² resolution and reuse[s] the selection for both resolutions" (Section 4.1), assuming no interaction between the optimal learning rate and resolution. If the optimal learning rate differs at 448px² (as it does across model sizes, per Figure 5), the 448px² transfer results may be systematically suboptimal.

The consequence. The clustering of tasks into "resolution-sensitive" and "model-size-sensitive" groups rests on the measured gap between 224px² and 448px² performance for each task. If part of that gap is attributable to the additional 50M high-resolution training examples (rather than to resolution per se), the asymmetry between tasks may be partly a data effect: the tasks that benefit most from "resolution" might actually be benefiting from the upweighted OCR and document tasks in the Stage 2 mixture, which provide domain-relevant pretraining that transfers to similar downstream evaluations. This would mean a practitioner who only increases input resolution at fine-tuning time (without the benefit of Stage 2-style high-resolution pretraining) might see smaller gains than Figure 3 predicts. Similarly, if the 448px² learning rate is systematically suboptimal (because the 224px²-optimal rate was reused), the resolution gain is understated—the true benefit of 448px² at its optimal learning rate might be larger than reported, which would push some tasks from the "both-sensitive" cluster into the "resolution-sensitive" cluster or shift the parity line in Figure 3.

What evidence exists in the paper. The Stage 2 data quantity confound is visible from the training recipe description (Section 3, Stages 1–2) but is not acknowledged or analyzed anywhere in the paper's scaling discussion. There is no ablation where a 224px² model receives an additional 50M examples of 224px² training to isolate the data quantity effect from the resolution effect. The learning rate confound is explicit in Section 4.1: "Since for most tasks the earlier work used the same hyperparameters for 224px² and 448px², we only sweep at 224px² resolution and reuse the selection for both resolutions." While pragmatic, this is an untested assumption. Figure 5 shows that the optimal learning rate shifts with model size for nearly every task, demonstrating that optimal hyperparameters are not invariant to scale—it would not be surprising if they also shift with resolution, which also changes the effective scale of the input representation. The paper provides no learning rate sweep data at 448px² to test this.

Mitigation status. Not addressed. The paper does not flag either confound as a limitation. The Stage 2 data quantity confound is inherent to the training recipe—the higher-resolution models must be trained at that resolution, and the paper's choice to do so on additional data (rather than, e.g., downscaling the Stage 2 mixture to 224px² for a control) means the confound cannot be resolved retroactively from the reported experiments. The learning rate confound could be partially mitigated post-hoc by noting that PaliGemma (the predecessor) did use identical hyperparameters at both resolutions, suggesting the assumption may hold approximately in practice—but the paper does not make this argument explicitly. Future work could run a controlled ablation: take the Stage 1 checkpoint, fine-tune directly at 448px² without Stage 2 pretraining, and compare against the Stage 2 → 448px² transfer. This would isolate the contribution of high-resolution pretraining data from the contribution of increased inference-time resolution.

Generalization Is Limited to a Single Model Family Trained on a Proprietary Data Mixture

The assumption or constraint. All experiments use the SigLIP-So400m vision encoder paired with Gemma 2 language models, trained on Google's internal multimodal data mixture (Section 3). The paper characterizes this mixture qualitatively—captioning, grounded captioning, OCR, machine-generated VQA, detection, instance segmentation—but does not release the data, specify mixture proportions, or provide the specialist models used for label generation. This means the findings are tied to a specific combination of vision encoder, language model family, and training data that cannot be independently reproduced or verified. The paper explicitly positions this as a design choice: the family of models is "all derived from comparable building blocks and are trained according to the same recipe" (Section 1), which enables the controlled comparison but also means the comparison is confined to one ecosystem.

The consequence. Practitioners using other model families (LLaVA, Cambrian, Qwen-VL, Molmo) or building VLMs from different components (CLIP vision encoders, LLaMA language models, different training data distributions) cannot assume that the scaling patterns observed here transfer. The task clustering in Figure 3—which tasks are resolution-sensitive vs. model-size-sensitive—may be specific to the SigLIP-So400m encoder's resolution-accuracy characteristics (how much visual information it can extract at each resolution), the Gemma 2 models' per-parameter reasoning capability, and the pretraining mixture's coverage of different task types. A vision encoder with different patch size, compression ratio, or pretraining objective might exhibit different resolution scaling behavior. A language model from a different family might have different scaling characteristics—the paper itself notes that the Gemma 2 27B model's from-scratch training (vs. distillation for 2B and 9B) may explain its anomalous scaling behavior (Section 4.1.1), suggesting that scaling patterns are sensitive to language model training methodology. More fundamentally, the paper's central recommendations ("sweep smaller learning rates when increasing model size"—Section 4.1.2; choose resolution over model size for document tasks—Figure 3) may be specific to the Gemma 2 architecture and its interaction with the SigLIP encoder, making their transferability to practitioners using different VLM stacks uncertain.

What evidence exists in the paper. The paper acknowledges the single-family limitation implicitly by its choice of title and framing ("PaliGemma 2: A Family of Versatile VLMs for Transfer"), focusing on characterizing one family rather than making cross-family claims. Section 2 explicitly notes that prior scaling work "often use[s] models with different architecture and training recipes from different labs," and positions PaliGemma 2's controlled comparison as a methodological advance over that fragmented landscape. However, the paper does not state the reverse: that its own findings are specific to the PaliGemma 2 family and may not generalize. The closest the paper comes is the speculation about distillation vs. from-scratch training in the Gemma 2 language models (Section 4.1.1), which acknowledges that a language model training detail can meaningfully affect scaling behavior. This creates tension: if a single difference within the same language model family (distillation) can alter scaling patterns, then the patterns should be even less stable across different families with different architectures, training data, and objectives.

Mitigation status. Not addressed. The paper does not compare against other VLM families or discuss the generalizability of its scaling insights. The proprietary data mixture presents a particularly challenging barrier to independent verification—even if other labs wished to replicate the study with the same open-weight SigLIP and Gemma 2 models, they would need to assemble a comparable multimodal pretraining mixture, and the paper provides only a qualitative sketch of its contents. The paper's value to the broader community is therefore primarily in its methodology (the controlled grid approach to scaling analysis) and in its qualitative insights (task clustering, learning rate scaling), rather than in the specific numbers or model choices. Future work could apply the same controlled-grid methodology to fully open VLM stacks (e.g., SigLIP + LLaMA, DFN + Gemma) with openly documented training data, enabling cross-family comparisons that would test the generality of the PaliGemma 2 findings.

No Statistical Significance Testing for the State-of-the-Art Claims on Specialized Tasks

The assumption or constraint. The paper reports state-of-the-art results on text detection and recognition (Table 2, +1.4 F1 on ICDAR'15, +1.8 on Total-Text), table structure recognition (Table 3, new SOTA on 7/8 metric-dataset combinations), molecular structure recognition (Table 4, +1.0 exact match), music score recognition (Table 5, approximately halved error rates), and radiography report generation (Table 8, +5.1 RadGraph F1). All of these results are reported as single numbers—the mean and standard deviation over multiple fine-tuning runs (used for the 30+ academic benchmarks in Table 13) are not provided for any of the specialized tasks. The paper does not report whether the observed improvements exceed test-retest variability, nor whether they are robust to different random seeds, data splits, or hyperparameter choices.

The consequence. The strength of the SOTA claims is indeterminate. For text detection on ICDAR'15 Incidental, the F1 margin is 1.4 points (75.9 vs. 74.5). Without knowing the variance of PaliGemma 2's performance across multiple runs, it is impossible to assess whether this margin represents a reliable improvement or a favorable draw from overlapping distributions. For table structure recognition on PubTabNet, PaliGemma 2 achieves S-TEDS 97.6 while the prior SOTA is 97.9—a 0.3-point deficit. If the variance on this metric is, say, 0.2 points (comparable to the standard deviations observed for some academic benchmarks in Table 13), this apparent regression could be noise. For music score recognition (Table 5), the error rate reductions are dramatic (CER: 3.9 → 1.6, a 59% relative reduction), making them less susceptible to noise concerns, but without variance estimates even large improvements cannot be assigned a confidence level.

This limitation is most consequential for the paper's central narrative about versatility: "PaliGemma 2 obtains state-of-the-art results on many of those tasks" (Section 1). This statement implies a level of reliability that the paper's reported data cannot fully support, since the specialized-task results lack the statistical rigor applied to the academic benchmarks. A practitioner choosing between PaliGemma 2 and a specialized architecture like HTS for OCR would want to know whether the 1.4 F1 advantage reflects a genuine performance difference or run-to-run noise, particularly since the specialized system (HTS) is a single-point result without reported variance either.

What evidence exists in the paper. Table 13 demonstrates that the authors can produce variance estimates when they choose to—the 5-run protocol with standard deviations is applied consistently to the 30+ academic benchmarks. The specialized tasks in Tables 2–8 lack this treatment entirely. The training configurations for specialized tasks (Appendix B) specify batch sizes, learning rates, and step counts, but do not mention multiple runs or seed variation. The discrepancy between the two categories of results suggests either that the specialized tasks were evaluated with a single run (matching typical practice in the OCR and document understanding communities) or that multiple runs were conducted but the variance is not reported.

Mitigation status. Not addressed. The paper does not explain why the 5-run protocol used for academic benchmarks is not applied to the specialized tasks, nor does it acknowledge the absence of variance estimates as a limitation on the SOTA claims. For the text detection and table recognition tasks, where the margins over prior work are relatively narrow (1–2 points), this is a significant gap in the evidence. Future work could close this gap by running even 3–5 seeds on a subset of the specialized tasks to characterize variance and confirm that the SOTA improvements are reliable. Given that the 3B model at 896px² is used for text detection and table recognition, and that the 3B model training at 896px² costs 23.5× the 224px² baseline (Table 1), running multiple seeds on these specialized tasks would be computationally expensive but necessary for robust SOTA claims.

The DOCCI Long Captioning Comparison Mixes Training Data and Model Architecture Effects

The assumption or constraint. The DOCCI evaluation in Table 6 compares PaliGemma 2 models (fine-tuned specifically on the DOCCI training set) against instruction-tuned VLMs (MiniGPT-4, mPLUG-Owl2, InstructBLIP, LLaVA-1.5, VILA) that were not fine-tuned on DOCCI. The comparison therefore conflates two variables: (1) whether the model is fine-tuned on DOCCI's specific data distribution of detailed, factually-grounded image descriptions, and (2) the underlying capability of the model architecture and pretraining. The paper states that the instruction-tuned baselines are "often instruction-tuned on 10−100× larger high-quality captioning sets than PaliGemma 2" (Section 4.6), implying they have seen more captioning data overall, but this data was not from DOCCI and was not optimized for the factual entailment metric used in evaluation.

The consequence. The claim that PaliGemma 2 "produces more factually aligned sentences than many popular VLMs" (Section 4.6, referring to Table 6) overstates the evidence. The more accurate claim—which the data does support—is that fine-tuning on DOCCI's training data produces more factually aligned sentences than zero-shot or instruction-tuned deployment of other VLMs on DOCCI. We cannot determine from Table 6 whether PaliGemma 2's advantage comes from better architecture/pretraining or simply from being trained on the target data distribution. If the comparison VLMs were also fine-tuned on DOCCI (or if PaliGemma 2 were evaluated zero-shot), the gap might narrow substantially or reverse. The paper acknowledges this indirectly by listing PaLI-5B and PaliGemma 3B as DOCCI-fine-tuned baselines ("Pali* models are models fine-tuned on DOCCI at 448px²"), but the primary comparison that draws attention is against the popular instruction-tuned models (VILA: 28.6% NES, LLaVA-1.5: 40.6% NES vs. PaliGemma 2 10B: 20.3%), which are not DOCCI-fine-tuned.

This is particularly important because the DOCCI evaluation protocol (human evaluation of per-sentence entailment) is specifically what DOCCI was designed to measure (Onoe et al., 2024). A model fine-tuned on DOCCI's training set is being optimized for the exact evaluation criterion used to assess it, while instruction-tuned models are being evaluated on a distribution and metric for which they were not optimized. The comparison is therefore partially circular: DOCCI fine-tuning → better DOCCI evaluation scores, which measures the effectiveness of DOCCI fine-tuning more than it measures the underlying capability of the PaliGemma 2 architecture.

What evidence exists in the paper. The inclusion of PaliGemma 3B (34.3% NES) and PaLI-5B (32.9% NES) as DOCCI-fine-tuned baselines provides some control—PaliGemma 2 3B (28.4% NES) and 10B (20.3% NES) improve over these, isolating the architecture/scale effect from the data effect within the fine-tuned group. The jump from PaliGemma 3B (34.3%) to PaliGemma 2 3B (28.4%) is a meaningful architecture improvement (5.9 points), and the further jump to 10B (20.3%) is a meaningful scale improvement (8.1 points). However, the paper never reports PaliGemma 2 zero-shot on DOCCI (without fine-tuning), which would be needed to directly compare against the instruction-tuned baselines on equal footing. The two-stage selection process (perplexity on test → human eval on qual_dev) further embeds the DOCCI-specific optimization, since the model selected is the one that performs best on DOCCI's validation splits—a process not applied to the comparison baselines.

Mitigation status. Partially addressed by the inclusion of DOCCI-fine-tuned PaliGemma-family baselines, but not by providing a zero-shot PaliGemma 2 evaluation. The paper does not explicitly acknowledge the confound between training data and model architecture. The description in Section 4.6 notes that "We first fine-tune PaliGemma 2 on DOCCI's train split" and that baseline models "are often instruction-tuned," making the data difference visible to attentive readers but without drawing attention to it as a limitation on the strength of the comparison. A fairer comparison would include either (a) PaliGemma 2 evaluated zero-shot on DOCCI (to match the instruction-tuned baselines) or (b) at least one instruction-tuned baseline also fine-tuned on DOCCI (to match PaliGemma 2's training). Without either of these controls, the DOCCI results primarily demonstrate that fine-tuning on DOCCI improves DOCCI evaluation scores—a finding that is true by construction—with PaliGemma 2's specific contribution being the improvement over other DOCCI-fine-tuned PaliGemma-family models.

High-Resolution Inference Latency Is Not Characterized, Constraining Deployment Guidance

The assumption or constraint. The paper's CPU inference analysis (Section 4.9, Table 9) measures latency only for the 3B model at 224px² resolution. No latency measurements are provided for the 448px² or 896px² models, for any model size larger than 3B, or for GPU/TPU inference (the hardware on which most practitioners would deploy these models). The paper's scaling analysis conclusively demonstrates that many tasks—particularly OCR and document understanding—derive their largest gains from increasing resolution to 448px² or 896px². But the paper provides no data on what latency cost this incurs. For the 3B model at 224px², prefill processes 260 tokens (256 image + 4 text); at 448px², the prefill would process 1028 tokens (1024 image + 4 text), and at 896px², 4100 tokens. Since the Gemma 2 self-attention cost is quadratic in sequence length, the prefill latency grows faster than linearly with token count, and the per-token extend latency also scales with prefix length.

The consequence. A practitioner who reads the paper and concludes—correctly, based on the scaling analysis—that their document understanding task requires 896px² resolution has no information about the deployment implications. The 3B 224px² model achieves prefill at 32–323 tokens/sec on CPU (Table 9), with the fastest hardware processing the 260-token prefix in 0.17 seconds. At 896px², the prefix is 4100 tokens, roughly 16× longer. If prefill latency scales even linearly (which is optimistic—self-attention is quadratic), the fastest hardware would require ~2.7 seconds just for prefill, and the slowest (Apple M1 Max) would require over two minutes. In practice, the quadratic scaling of attention means the actual penalty is worse than linear. The paper's silence on this point means a practitioner cannot make an informed tradeoff between the accuracy gains from higher resolution (which the paper quantifies well) and the latency cost (which it does not quantify at all).

This is particularly significant given the paper's framing of PaliGemma 2 as practical for "on-device deployment" (Section 4.9). The CPU inference benchmarks are positioned to demonstrate feasibility, but they cover only the smallest model at the lowest resolution—the configuration least likely to be useful for the resolution-intensive tasks where PaliGemma 2's advantages over the original PaliGemma are most pronounced. If 896px² inference is impractically slow on CPU, a practitioner might need to use GPU/TPU inference for high-resolution tasks, but the paper provides no GPU/TPU latency numbers either.

What evidence exists in the paper. Table 9 provides detailed CPU latency measurements for one specific configuration (3B, 224px², specific prompt/image/response). Table 1 provides relative pretraining cost per example across all configurations, showing that 896px² is 23.5× more expensive than 224px² for the 3B model during training. However, training cost does not directly translate to inference latency—training involves forward and backward passes with gradient computation, while inference requires only forward passes, potentially with KV caching. The paper does not estimate inference FLOPs or measure wall-clock time for any configuration beyond the single one in Table 9.

Mitigation status. Not addressed. The paper does not discuss inference latency as a limitation on the practicality of high-resolution or large-model variants, nor does it suggest latency characterization as future work. The gemma.cpp CPU benchmarks in Section 4.9 are presented as a positive demonstration of on-device feasibility but are so narrowly scoped (one model size, one resolution, one example) that they provide almost no guidance for the deployment decisions the rest of the paper motivates. A practitioner who wants to deploy PaliGemma 2 3B at 896px² for OCR, or 10B at 448px² for DOCCI-style captioning, will need to measure latency independently. This is a significant omission given that the paper's central contribution is a framework for choosing between model sizes and resolutions—a decision that in practice depends jointly on accuracy and latency, not accuracy alone.

7. Implications and Future Directions

How This Work Changes the Landscape

PaliGemma 2 does not introduce a new architecture, training paradigm, or loss function. Its contribution is methodological: it establishes a controlled experimental framework for studying the drivers of VLM transfer performance and uses that framework to produce actionable, empirically-grounded guidance about how to allocate compute between model size and input resolution. This reframes the practitioner's decision from "which model is best?" to "given my task's characteristics, where should I spend my compute budget?"

The magnitude of this shift is best characterized as a diagnostic reframing with practical teeth rather than a paradigm shift. The paper does not overturn the prevailing approach to VLM design—the encoder-decoder architecture combining a pretrained vision model with a pretrained language model remains the dominant framework—but it changes how practitioners should choose among such models. Before PaliGemma 2, the literature offered only fragmented guidance: some papers studied resolution scaling in isolation (Laurençon et al., 2024), others studied model size scaling but with confounded architectures and training recipes (Tong et al., 2024; Karamcheti et al., 2024). The paper's explicit statement—"We are not aware of prior work which jointly studies the effect of the image resolution and the size of the language models on transfer via fine-tuning" (Section 2)—identifies the gap precisely, and the 3×3 model grid fills it.

The most consequential landscape change is the task-dependent compute allocation mental model that emerges from Figure 3. The paper divides transfer tasks into three clusters based on whether they respond more to resolution increases, model size increases, or both, and quantifies the asymmetry: on DocVQA, going from 224px² to 448px² yields an 84.5% relative improvement, while going from 3B to 10B at 224px² yields only 10.0%. On AI2D, the pattern reverses: 11.2% from the model upgrade vs. 1.7% from the resolution upgrade. This asymmetry is not subtle—it represents a qualitative difference in where the bottleneck for each task lies. The practical consequence is that a practitioner building a document understanding system should invest their compute budget in resolution, not model size, while someone working on visual reasoning should do the opposite. This replaces the default "use the biggest model you can afford" heuristic with a diagnostic: identify your task's bottleneck first, then scale the relevant axis.

The paper also resolves an implicit tension in prior work between general-purpose VLM interfaces and task-specific architectures. The specialized-task results in Sections 4.2–4.5 demonstrate that a general-purpose VLM with no task-specific architectural components can match or exceed specialized state-of-the-art models on OCR, table recognition, molecular recognition, and music score recognition—tasks where custom detection heads, graph decoders, or structured output layers were previously considered essential. The paper emphasizes this explicitly: "We emphasize that this result is obtained simply by fine-tuning a general-purpose VLM which does not rely on task-specific architecture components" (Section 4.2). This should shift research attention away from architecturally specialized solutions for structured prediction tasks that can be serialized as text, and toward improving the base VLM's versatility and its ability to handle diverse output formats through the text interface alone.

However, the negative result on object detection (Appendix C, Table 11) establishes a critical boundary condition that narrows the scope of the "versatile interface" claim and redirects research toward understanding when and why general-purpose VLMs fall short. PaliGemma 2 achieves only 43.6 mAP on COCO and 66.0 mAP on DocLayNet, substantially below specialized detectors (YOLOv11: 79.5 mAP on DocLayNet). The authors' diagnosis—that "the limiting factor is not the model's intrinsic object understanding" but "a mismatch between the Average Precision metric, which rewards large numbers of predictions and accurate confidence scores, and the language modeling objective" (Appendix C)—points to a specific research direction: aligning VLM training objectives with evaluation metrics that reward calibrated confidence over many predictions. This finding makes it clear that the universal text interface is not a panacea, and that certain task structures (dense prediction with ranking-based evaluation) remain better served by specialized architectures, at least for now.

The learning rate scaling finding (Section 4.1.2, Figure 5) introduces a practically important dimension to the transfer learning protocol that had been overlooked. The consistent diagonal pattern—larger models prefer lower transfer learning rates—is not merely an engineering detail; it implies that practitioners upgrading from a 3B to a 10B or 28B model who carry forward the same learning rate sweep range will systematically underperform, potentially by margins comparable to the gains they expected from the larger model. This finding is actionable immediately and requires no new tooling, just a shift in hyperparameter sweep ranges.

Finally, the paper's methodological contribution—the controlled model grid—raises the bar for future VLM scaling studies. Prior work that compares models of different sizes by borrowing language models from different labs is now, by implication, confounded. The paper does not state this polemically, but the contrast is clear: PaliGemma 2's approach of varying one factor at a time within a single training framework makes it possible to attribute performance differences causally, while cross-family comparisons can only report correlations. Future scaling studies that do not adopt a similar controlled framework will need to justify why their comparisons are valid despite confounding variables.

Follow-Up Research This Work Enables

Extending the controlled scaling grid to 896px² for a representative subset of benchmarks to determine whether the resolution-sensitivity curve saturates or continues.

The paper's scaling analysis (Figure 3, Figure 4) covers only 224px² and 448px², leaving the 896px² resolution unexplored for the 30+ academic benchmarks. The specialized tasks (Sections 4.2–4.5) demonstrate that 896px² is crucial for OCR and document understanding, but these were selected precisely because they benefit from resolution—we do not know whether tasks classified as resolution-sensitive at the 224px²→448px² transition (DocVQA, TextVQA, ST-VQA) continue to benefit from 896px², or whether they saturate at 448px². Similarly, tasks classified as model-size-sensitive at lower resolutions might become resolution-sensitive at 896px² if the increased vision token count places greater reasoning demands on the language model. A follow-up study evaluating 8–10 representative benchmarks spanning all three clusters from Figure 3—at all three model sizes and all three resolutions—would characterize the shape of the resolution-sensitivity curve and determine whether diminishing returns to resolution set in universally or at task-specific thresholds. The computational cost would be substantial (23.5× per example at 896px² for the 3B model, per Table 1), but a carefully chosen subset of benchmarks could be tractable. This would also test whether the 28B model's advantages, which appear marginal at 224px² and 448px², become more pronounced at 896px² where the language model must process 4096 vision tokens instead of 256 or 1024—a sequence length where capacity differences may matter more.

Disentangling resolution effects from data quantity effects through a controlled ablation on Stage 2 pretraining.

The paper's comparison of 224px² and 448px² transfer performance conflates two factors: the architectural change (more vision tokens, enabling finer visual detail) and the additional 50 million high-resolution training examples in Stage 2. The 448px² models are fine-tuned from Stage 2 checkpoints that have seen 50M more examples than the Stage 1 checkpoints used for 224px² transfer, and these additional examples are specifically curated to upweight resolution-benefiting tasks like OCR. A controlled ablation would take the Stage 1 checkpoint and (a) fine-tune it directly at 448px² without any Stage 2 pretraining, and (b) continue training it at 224px² for 50M additional examples with the same mixture reweighting as Stage 2, then fine-tune at 224px². Comparing (a) vs. the standard Stage 2→448px² transfer would isolate the contribution of high-resolution pretraining; comparing (b) vs. the standard Stage 1→224px² transfer would isolate the data quantity effect. If the resolution-sensitive cluster in Figure 3 is partially driven by the Stage 2 data rather than resolution per se, the gap between clusters would narrow, and the practical recommendation would shift from "use higher resolution for document tasks" to "include diverse OCR data in pretraining for document tasks." This experiment would also inform whether practitioners who cannot afford Stage 2-style high-resolution pretraining can achieve similar gains through resolution-aware fine-tuning strategies alone.

Testing the transferability of the task clustering findings to other VLM families to determine whether resolution-sensitivity and model-size-sensitivity are architectural or universal.

The paper's task clustering (Figure 3) is derived from a single combination of vision encoder (SigLIP-So400m), language model family (Gemma 2), and training data mixture (Google's internal multimodal data). We do not know whether the same clusters would emerge with a different vision encoder (e.g., DFN, CLIP-L, or a ViT with a different patch size), a different language model family (e.g., LLaMA, Qwen), or a different pretraining data distribution. The paper's own speculation about the Gemma 2 27B model's from-scratch training vs. distillation (Section 4.1.1) suggests that language model training methodology alone can alter scaling behavior within the same family, implying that cross-family scaling patterns may differ. A systematic replication using a fully open VLM stack—for example, SigLIP-So400m + LLaMA-3 at 2B/8B scales, or DFN + Gemma 2 at multiple resolutions—trained on openly documented data (e.g., a combination of open captioning, VQA, OCR, and detection datasets), would test whether the task clustering is a general property of VLM transfer or specific to the PaliGemma 2 recipe. If the clusters replicate, the findings gain substantially in generality and the SigLIP+Gemma combination is validated as a representative experimental platform; if they do not—if, say, document tasks become model-size-sensitive in a different setup—then the field learns that scaling behavior is deeply recipe-dependent, and the paper's specific recommendations are bounded. This study would also help establish open benchmarks for controlled VLM scaling research, addressing the reproducibility gap created by the paper's proprietary training data.

Investigating whether the optimal transfer learning rate interacts with resolution, not just with model size.

The paper demonstrates that optimal transfer learning rates decrease with model size (Figure 5), but it assumes no interaction with resolution—the 224px²-optimal learning rate is reused at 448px² without independent verification (Section 4.1: "for most tasks the earlier work used the same hyperparameters for 224px² and 448px², we only sweep at 224px² resolution and reuse the selection for both resolutions"). Given that increasing resolution changes the number of vision tokens by 4× (224px²→448px²) or 16× (224px²→896px²), the effective "scale" of the model's input representation changes substantially, and it would be surprising if the optimization dynamics were completely invariant. A study replicating the learning rate sweep from Table 14 at 448px² for a subset of tasks (perhaps 10 tasks spanning the three clusters in Figure 3, at all three model sizes) would either validate the paper's assumption—in which case the resolution-insensitivity of the learning rate is an interesting finding in its own right—or reveal interactions that would require practitioners to sweep learning rates along both axes. If interactions exist, the reported 448px² performance in Table 13 may be systematically suboptimal (since the learning rate was selected at 224px²), meaning the true benefit of resolution could be larger than reported, potentially strengthening the paper's resolution-sensitivity claims.

Training a VLM with a combined objective that jointly optimizes for language modeling and detection-style confidence calibration to close the gap with specialized detectors.

The negative result on object detection (Appendix C) identifies a specific mechanism for the performance gap: the Average Precision metric rewards models that produce many predictions with well-calibrated confidence scores, while language models trained with cross-entropy loss are optimized for sequence correctness, not confidence calibration. The <noise> token mechanism provides a rudimentary confidence proxy (by measuring the probability the model assigns to a prediction being noise vs. a real object), but this is an emergent property of the sequence augmentation strategy, not a directly optimized quantity. A follow-up could explore adding an auxiliary detection loss during the transfer stage—for instance, a contrastive or ranking loss that explicitly penalizes the model when it assigns higher confidence to a false positive than to a true positive—while preserving the text-to-text output format. This would test whether the versatility of the text interface can be preserved while closing the gap to specialized detectors, or whether the text interface inherently constrains the model's ability to represent fine-grained confidence rankings. The experiment would fine-tune PaliGemma 2 on COCO with the standard Pix2Seq-style sequence augmentation augmented by a detection-specific loss term, and measure both mAP and the model's performance on other tasks (VQA, captioning) to check for regression—if adding detection-specific training hurts general versatility, that would define another boundary of the "versatile interface" claim.

Practical Applications and Downstream Use Cases

Document digitization pipelines for enterprises managing large volumes of scanned forms and tables.

Many organizations—insurance companies, financial institutions, government agencies, healthcare providers—process millions of scanned documents annually, extracting structured data from forms, tables, and text-heavy pages. The traditional approach uses a pipeline of specialized models: a document layout analyzer, a table detection model, a table structure recognizer, and an OCR engine. PaliGemma 2 3B at 896px² achieves state-of-the-art results on table structure recognition (Table 3: TEDS 98.9 on FinTabNet, 97.3 on PubTabNet) and text detection (Table 2: 75.9 F1 on ICDAR'15 Incidental) using a single model with no task-specific components. This suggests that a single fine-tuned PaliGemma 2 checkpoint could replace multiple specialized models in a document processing pipeline, dramatically reducing system complexity, maintenance burden, and the need for integration between models with different interfaces and failure modes. The 3B model's 4.0 GB quantized footprint (Table 10) and CPU inference speeds of 32–323 tokens/sec for prefill on modern hardware (Table 9) make on-premise deployment feasible even for organizations with data residency requirements that preclude cloud inference. The key operational benefit is simplification: instead of maintaining and updating five specialized models, a team maintains one, fine-tuned on their specific document types. The paper's finding that these tasks are resolution-bottlenecked rather than model-size-bottlenecked (increasing model size "did not lead to additional benefits" for table recognition and text detection—Sections 4.2–4.3) means the 3B variant at 896px² is the optimal cost-performance point, avoiding the higher training and inference costs of the 10B or 28B models.

Medical imaging report generation in clinical settings with domain-specific fine-tuning.

The radiography report generation results (Section 4.8, Table 8) demonstrate that PaliGemma 2 10B at 896px² achieves a RadGraph F1 of 29.5%, outperforming the specialized Med-Gemini-2D (24.4%) and Flamingo-CXR (20.5%). The RadGraph metric specifically measures clinical accuracy—the presence and relationships of findings—rather than surface-level text similarity, making the improvement clinically meaningful. A hospital or radiology practice could fine-tune PaliGemma 2 on their own report database (following the MIMIC-CXR protocol with an LLM to preprocess indications and remove references to unavailable prior studies) to generate draft Findings and Impressions sections that radiologists review and edit, rather than composing from scratch. The workflow benefit is accelerated report turnaround without sacrificing clinical accuracy. The paper's finding that the model can condition on the Indications section as a prefix (Appendix B.7: "during prediction emulate the clinical workflow by providing the indications as a prefix to the model") means the system integrates naturally with existing clinical workflows where indications are already recorded before image review. The modest improvement from 3B to 10B (28.8% to 29.5% RadGraph F1) and the benefit of 896px² resolution suggest that practitioners should prioritize resolution over model size if compute is constrained—the 3B at 896px² would likely provide most of the clinical benefit at lower inference cost.

On-device image captioning and accessibility applications for visually impaired users.

The combination of PaliGemma 2 3B at 224px² quantized to 8-bit (4.0 GB, Table 10) with CPU inference speeds of 0.17–1.6 seconds for prefill and 0.27–0.9 seconds per token on modern processors (Table 9) makes on-device image description feasible for accessibility applications. The captioning quality from the DOCCI evaluation (Table 6) shows that PaliGemma 2 3B generates captions with 28.4% Non-Entailment Sentences—fewer factual errors than much larger instruction-tuned models like VILA (28.6% NES) or LLaVA-1.5 (40.6% NES), even though those models have 7B+ parameters and access to cloud-scale compute. The 10B model reduces NES further to 20.3%, but at the cost of a larger memory footprint and slower inference. A smartphone accessibility app could run PaliGemma 2 3B locally (avoiding cloud latency and privacy concerns) to describe a scene, read text in an image, or identify objects, with per-image latency of a few seconds on a modern phone processor. The quantization quality preservation (relative metrics of 99.9–100.2% across five diverse tasks) means the user gets essentially the same accuracy as the 32-bit model, but on hardware that fits in a pocket. The key insight from the paper that enables this application is that for general scene understanding tasks (near the "both-sensitive" cluster in Figure 3), the 3B model at moderate resolution provides strong performance, making it viable for deployment scenarios where larger models or cloud inference are impractical.

Specialized scientific data extraction where output formats can be serialized as structured text.

The molecular structure recognition (Section 4.4: 94.8% exact match on SMILES) and music score recognition (Section 4.5: CER 1.6%, SER 2.3%, LER 6.7% at 896px²) results demonstrate a pattern that extends beyond these specific domains: whenever a scientific or technical output format can be serialized as structured text (SMILES for chemistry, **kern for music, potentially InChI for molecules, MathML for equations, chemical reaction notation, or domain-specific markup), PaliGemma 2 can be fine-tuned to perform the extraction without developing task-specific architectures. A chemistry research group could fine-tune PaliGemma 2 on their own rendered molecule dataset to extract structures from scanned journal figures or hand-drawn molecules, using the MolScribe-style data augmentation strategy (multiple rendering styles, random perturbations) to achieve robustness. A music digitization project could fine-tune on their specific notation conventions. The practical benefit is that a single model architecture, fine-tuning protocol, and inference pipeline serves all these tasks—the research group does not need to become computer vision specialists to solve their domain-specific extraction problem. The paper's finding that these tasks are consistently resolution-bottlenecked (increasing resolution helps; increasing model size beyond 3B does not—Sections 4.2–4.5) provides clear guidance: invest in 896px² resolution and fine-tuning data quality; the 3B model is sufficient and keeps inference costs manageable.

When to Prefer This Method

The paper does not present PaliGemma 2 as a method to be preferred over named alternatives in a competitive sense—it is not proposing a new training algorithm, loss function, or inference strategy against which other approaches should be compared. Rather, PaliGemma 2 is a family of base models that practitioners choose among based on their task and resource constraints. The relevant decision is not "should I use PaliGemma 2 vs. some other approach?" but rather "given that I am using a transfer learning workflow for a vision-language task, how should I choose among the available PaliGemma 2 configurations?"

The paper provides explicit guidance for this within-family decision:

Prefer the 3B model at 896px² for resolution-bottlenecked tasks: text detection and recognition, table structure recognition, music score recognition, and document VQA. The evidence is that increasing model size beyond 3B provided no improvement on these tasks (Sections 4.2, 4.3, 4.5), while increasing resolution provided substantial gains (text detection F1 improves from substantially lower at 448px² to state-of-the-art at 896px²; music CER drops from higher error rates at lower resolutions to 1.6% at 896px²).
Prefer the 10B model at moderate resolution (448px²) for reasoning-intensive or knowledge-intensive tasks: AI2D, NLVR2, spatial reasoning (VSR), long fine-grained captioning, and multilingual tasks. The 3B→10B jump provides large gains on these tasks (AI2D: +8.4 points at 224px², Table 13; VSR: +5.0 points on zeroshot split, Table 7), while the 10B→28B jump often provides marginal or no improvement (AI2D: +0.1 points; NLVR2: +0.3 points, Table 13), and resolution beyond 448px² did not help for spatial reasoning (Section 4.7). The 10B model represents a practical sweet spot.
Prefer the 28B model only when accuracy at the absolute frontier is required and compute/latency constraints are absent. The paper is candid that "using the largest PaliGemma 2 can thus be useful if one wants to get the best possible performance and has no compute or latency constraints" (Section 4.1.1), but the evidence shows that the 28B advantage over 10B is often negligible, and on some tasks the 28B model underperforms the 10B (e.g., ChartQA aug at 448px²: 85.1 vs. 90.1, Table 13). The paper speculates this may be due to the 27B Gemma 2 model's from-scratch training vs. distillation for the 2B and 9B models (Section 4.1.1), and suggests that the diminishing returns are not fully understood.
Sweep lower transfer learning rates when using larger models. Figure 5 provides systematic evidence across 30+ tasks that the optimal learning rate decreases with model size. Practitioners fine-tuning the 10B or 28B models should shift their learning rate sweep range downward relative to what they would use for the 3B model, and should verify that their sweep range extends low enough to capture the optimum.

For on-device CPU deployment, the paper provides specific guidance: the 3B model at 224px² with 8-bit quantization (via gemma.cpp) achieves no practical quality degradation (Table 10) with a 4.0 GB memory footprint and CPU inference speeds of 12–41 tokens/second depending on hardware (Table 9). Higher resolutions and larger models are not characterized for CPU inference, so practitioners targeting CPU-only deployment should start with this configuration and test their specific latency requirements before investing in higher-resolution or larger-model deployments.