Building and better understanding vision-language models: insights and future directions

ArXiv: 2408.12637

🎯 Pitch

A simple VLM built on Llama 3.1 can jump 13.7 points on DocVQA to 87.7 just by training on a massive, automatically curated QA dataset—no architectural complexity needed. The paper further shows that even a 0.7B model trained on this synthetic data can match generalist 8B models at document understanding, challenging the costly race for ever-larger unimodal backbones.

1. Executive Summary

This paper provides a comprehensive tutorial and empirical analysis of the design space for building vision-language models (VLMs), systematically examining how divergent choices across architecture, data, and training methods impact downstream performance. Through a survey of the field and the practical construction of Idefics3-8B—a VLM built on Llama 3.1 instruct and SigLIP-SO400M—the authors analyze key mechanisms including cross-attention versus self-attention architectures for connecting unimodal backbones (inserted cross-attention blocks vs. concatenated visual tokens), multi-stage pre-training with progressively unfrozen components (stage 1 frozen backbones through stage 3 with DoRA on large synthetic datasets), and the image-splitting strategy for handling variable-resolution inputs (dividing images into tiles of 364×364 pixels with positional tokens appended). The resulting Idefics3-8B achieves an 87.7 ANLS score on DocVQA, representing a 13.7-point improvement over its predecessor Idefics2-8B, driven substantially by the introduction of Docmatix—a dataset of 2.4M images and 9.5M QA pairs derived from 1.3M PDF documents, which is 240 times larger than previously available open document-understanding datasets. The paper establishes that test-time OCR capabilities are primarily bottlenecked by the number of visual tokens per image rather than architectural complexity, and that synthetic data generation pipelines—combining OCR tools for text extraction with LLMs for QA pair generation—can produce training data that enables a small specialist model (0.7B parameters) to approach the document-understanding performance of a 8B generalist model.

2. Context and Motivation

The Core Problem: VLMs Lack a Systematic Understanding of Design Trade-offs

The fundamental question this paper tackles is deceptively simple: when building a vision-language model (VLM) from unimodal pre-trained components, which design choices actually matter, and how should those choices be made? The field has reached a point where assembling VLMs from pre-trained vision encoders (used to encode images into dense representations) and language models (used to process text and generate responses) has become the dominant paradigm, yet the literature reveals a striking lack of consensus on core aspects of the development pipeline. Different research teams make divergent choices about architecture, data composition, training stages, and evaluation protocols—often without rigorous ablation or justification—making it nearly impossible for practitioners to determine which decisions are essential versus incidental.

This gap matters profoundly for several intertwined reasons. First, the computational cost of building VLMs is escalating rapidly. Pre-trained backbones like Llama 3.1 and SigLIP already represent millions of dollars of training compute; researchers building on top of them need principled guidance to avoid wasting resources on suboptimal design choices. The authors emphasize that "these different core choices in VLM development, often not ablated or justified in research papers, make it challenging to distinguish which decisions impact model performance and assess the compute and data efficiency trade-offs" (Section 1). This is a direct call for the field to move beyond "it works" papers toward an engineering discipline with understood trade-offs.

Second, the practical applications of VLMs—document understanding, visual mathematical reasoning, converting webpage screenshots into code, analyzing charts and figures—are rapidly expanding, and different applications place different demands on model architecture. A VLM optimized for general image captioning may fail catastrophically on OCR-heavy document understanding tasks if its architecture bottlenecks visual token throughput, as the authors demonstrate when analyzing Idefics2's perceiver resampler (Section 2.2.2). Without systematic understanding, deployers are forced into costly trial-and-error experiments.

Third, the field operates with a delayed feedback loop that obscures the impact of pre-training design choices. The authors note that during pre-training, complex tasks like document understanding may appear to perform poorly, and "the impact of development choices in the VLM may only become evident after fine-tuning, leading to a delayed feedback loop. This delay can make pre-training ablations misleading" (Section 4.2). A concrete example: Idefics2's authors found that increasing visual tokens from 64 to 128 showed no apparent improvement during pre-training, yet the benefit became obvious in OCR tasks after fine-tuning with the image-splitting strategy. This temporal disconnect between pre-training metrics and downstream performance means that researchers who only evaluate at pre-training may draw incorrect conclusions about which architectural choices are important.

The Fragmentation of Design Choices Across the Literature

The paper identifies four axes where the literature has diverged without clear resolution:

Architecture: Cross-attention vs. self-attention. The two dominant paradigms for connecting vision encoders to language models represent fundamentally different philosophies. In cross-attention architectures (Flamingo, Alayrac et al., 2022; Llama 3-V), freshly initialized cross-attention layers are interleaved between the frozen LLM's transformer blocks, where keys and values come from vision features and queries from language inputs. This architecture adds parameters equivalent to roughly 1/4th of the LLM's size but preserves the LLM's text-only performance by keeping it frozen. In self-attention architectures (BLIP2, Llava), visual features are projected, optionally pooled, and concatenated as tokens directly into the LLM's input sequence. Most recent VLMs have converged on self-attention, but the choice is rarely justified empirically. Laurençon et al. (2024) provided one of the only head-to-head comparisons, showing that cross-attention outperforms when backbones stay frozen but underperforms when components are trained with LoRA—yet this insight depends on specific backbone choices (Mistral-7B + SigLIP) and does not generalize to all configurations.

Modality projection: Even within the self-attention family, the connector between vision space and text space varies wildly: simple linear projections (LLaVA, FROMAGe), single-layer cross-attention modules (Qwen-VL), perceiver resamplers (Idefics2, Flamingo), 2D-aware convolutions (mPLUG-DocOwl-1.5's H-Reducer), pixel shuffle (InternVL), and ResNet blocks with 2D positional embeddings (HoneyBee's C-Abstractor). Each makes different trade-offs between preserving visual information and compressing token counts. The paper explicitly states that "the use of the perceiver resampler has been challenged in several papers" (Section 2.2.2), indicating this is an active area of methodological disagreement without settled answers.

Data composition: The types of training data used—image-text pairs, interleaved image-text documents, PDFs with OCR transcriptions, synthetic data—are introduced at different stages with varying rationales. OBELICS demonstrated that interleaved documents improve in-context learning, while MM1 showed they are "instrumental for few-shot and text-only performance." Yet the relative importance of different data types, their optimal ordering in the training curriculum, and which tasks they affect are not well-characterized.

Multi-stage training: Most models use 2–3 pre-training stages followed by supervised fine-tuning and optionally alignment, but the number of stages, when to unfreeze model components, and what resolution to use at each stage vary significantly. The paper notes that "training VLMs typically occurs in multiple stages, primarily due to (a) the limited availability of high-quality data at scale, (b) memory constraints for efficient training, and (c) stability issues" (Section 3). These pragmatic constraints drive methodological choices, yet the field lacks systematic guidance on how to design an efficient training curriculum.

Where Prior Approaches Fall Short

Individual papers optimize locally without comparing globally. Most VLM papers propose a specific architecture and demonstrate it works, but rarely ablate across the full design space or compare systematically against alternative paradigms. The authors highlight that cross-attention vs. self-attention comparisons exist (Laurençon et al., 2024) but are limited to specific backbone choices. The effect of vision encoder quality is known to be important—replacing CLIP-ViT-H (78.0% ImageNet) with SigLIP-SO400M (83.2% ImageNet) substantially improves VLM performance—but "few open-vision encoders have been released, with SigLIP-SO400M standing out due to its favorable performance-to-parameter ratio" (Section 2.1.4). This means the research community lacks a diverse set of open vision encoders to systematically study encoder quality effects.

The number of visual tokens has emerged as a critical but under-explored bottleneck. The paper identifies a consistent pattern: OCR-heavy tasks like DocVQA, TextVQA, and InfoVQA are the primary beneficiaries of more visual tokens, while most other tasks are relatively insensitive. Idefics2's perceiver resampler compressed images to as few as 64 tokens and "maintained performance for most tasks, except those that require extensive OCR capabilities" (Section 2.2.2). InternLM-XComposer2-4KHD corroborated this, showing that increasing visual tokens per image was "primarily necessary for benchmarks focused on OCR tasks." This suggests an architectural fork: models designed for general visual understanding can use aggressive token compression, but document-focused models need much higher token throughput. Yet the field has not converged on standardized resolutions or token counts for different task categories.

Document understanding specifically is starved for open training data. Prior to this work, open-source datasets for document QA were extremely limited: DocVQA offered 10K images and 40K QA pairs, InfographicVQA had 2K images and 10K QA pairs, VisualMRC provided 3K images and 12K QA pairs. These are orders of magnitude too small for pre-training at scale. Proprietary systems (GPT-4V, Gemini) presumably had access to much larger document corpora, creating a data moat that prevented open models from competing on document understanding tasks. The paper identifies this as a specific, concrete gap: "generating high-quality synthetic data for this task is relatively straightforward if we reframe the problem as one of LLM-based data generation rather than relying solely on VLMs" (Section 5.1.2). The insight—that OCR tools can extract text, then text-only LLMs can generate QA pairs from that text—is simple but had not been applied at scale.

Evaluation practices exacerbate the confusion. The paper diagnoses several evaluation pathologies. First, open-ended benchmarks like VQAv2 penalize models that produce answers in a format or writing style different from the expected ground truth, even if semantically correct. The authors cite the striking example that "Gemini 1.0 Ultra and GPT-4V achieve scores of 77.8 and 77.2, respectively" on VQAv2, "notably lower than those of much smaller models that include a small portion of VQAv2 in their fine-tuning data: MM1-3B-Chat reaches 82.0" (Section 4.1). This format bias can make smaller, fine-tuned models appear to outperform much larger generalist models, misleading architectural comparisons.

Second, benchmark contamination is rampant. The authors found that at least 6.6% of MathVista questions include images from the training sets of academic datasets commonly used in supervised fine-tuning, and 2.2% feature both an image and a question that is identical or highly similar (Section 4.3). This means reported benchmark scores may reflect memorization of training data rather than genuine visual reasoning capability. The authors flag this as a critical issue: "benchmarks should be used to measure model performance, not as a training objective."

Third, the gap between pre-training and post-fine-tuning performance creates misleading ablations. Design choices that appear neutral during pre-training may prove critical after fine-tuning, yet many researchers evaluate only at one stage and draw invalid conclusions. The delayed feedback loop means architectural choices must be evaluated through the full training pipeline, not just pre-training metrics.

The "vision encoder or not" question remains unresolved. Fuyu demonstrated that feeding image patches directly into a language model after a simple linear projection could work, bypassing the need for a pre-trained vision encoder entirely. This approach has theoretical appeal: it is independent of another pre-trained model and preserves all information from the original image, whereas pre-trained vision encoders "transform an image into a representation that is independent of the user's prompt" and "can still miss details pertinent to the prompt" (Section 2.2.1). Yet PaliGemma experimented with this approach and "reported a notable drop in performance compared to using a pre-trained vision encoder," suggesting "bypassing a vision encoder pre-trained on billions of images could lead to longer training times to achieve similar performance." This tension—information preservation versus the efficiency of pre-trained representations—remains unresolved and highlights a fundamental open question about what vision encoders actually contribute beyond dimensionality reduction.

Is a vision encoder really necessary? Beyond the empirical performance question, using no vision encoder has unexamined downstream consequences. The paper notes that "handling image representation within the language model might decrease its performance on text-only benchmarks" (Section 2.2.1)—a concern that is largely unstudied because "most VLMs are still not evaluated on text-only benchmarks, making it unclear whether omitting a vision encoder affects text benchmark performance." Additionally, the pixel-to-token approach "has not been tested yet with an efficient pooling strategy that does not significantly reduce information by operating directly on raw pixels," leaving open the possibility that a well-designed pooling mechanism could make this approach competitive while also solving the token efficiency problem for high-resolution images and video.

How This Paper Positions Itself

This paper positions itself as a practitioner's guide and empirical contribution rather than a novel theoretical framework. The structure reflects this: Sections 2–4 survey the design space systematically (architecture, data, training, evaluation), identifying what is known, what is contested, and what deserves more research attention. Section 5 then demonstrates the practical application of these lessons through the construction of Idefics3-8B.

The paper bridges a crucial gap between comprehensive surveys (which catalog methods but don't produce new evidence) and single-model papers (which demonstrate one configuration works but don't compare alternatives). It does so by:

1. Providing concrete mechanism-level analysis of why different design choices matter, not just reporting outcomes. For example, when explaining the image-splitting strategy, the paper examines why encoding tiles separately is suboptimal ("the tiles of an image are not independent, encoding each one separately can be suboptimal and may result in a loss of global context") and why the current mitigation (appending a downscaled original image) is imperfect ("it's not a perfect solution, as the reduced resolution of the original image makes it difficult to capture finer details"). This level of mechanistic explanation enables readers to reason about trade-offs rather than just following recipes.

2. Explicitly addressing the delayed feedback loop problem by recommending instruction data be incorporated into pre-training data mixtures to get more accurate ablations during development (Section 4.2). This is a methodological contribution for VLM researchers, not just a model release.

3. Creating and releasing Docmatix as both a practical resource and a proof-of-concept that the data bottleneck in document understanding is solvable without proprietary datasets. The 240× scale increase (from ~40K QA pairs in DocVQA to 9.5M in Docmatix) is achieved through a pipeline that any group with access to standard OCR tools and an open LLM can replicate. The ablation with Florence-2—where training on a small subset of Docmatix led to a "nearly 20% relative improvement" on DocVQA (Table 2)—provides concrete evidence that the synthetic data pipeline produces genuinely useful training signal.

4. Highlighting underexplored research directions throughout. These include: developing vision encoders that can natively process images of varying resolutions using Patch'n'Pack (Section 2.2.3), creating better open-source vision encoders (the paper notes SigLIP-SO400M is almost alone in the open ecosystem), applying model-based filtering on educational content for multimodal datasets (analogous to Phi-3 and FineWeb-Edu for text), and improving the image selection/filtering process for image-text pair datasets (Section 3.1, noting that "less attention has been paid to the initial selection of 'good' images").

The paper does not claim to resolve all open questions. It explicitly acknowledges that key comparisons remain unperformed: "Is a vision encoder really necessary?" is flagged as an open question, PRM tree-search was not combined with revisions, LoRA training was chosen for efficiency but "we believe that carefully executed full unfreezing can lead to better performance" (Section 5.2.1), and the model "mainly trained on short answers during supervised fine-tuning, and did not benefit from an alignment phase" (Section 5.2.2)—all identified as opportunities for improvement rather than claimed as optimal.

This intellectual honesty creates a paper that serves as both a reference and a roadmap. Practitioners can adopt the documented pipeline to build competitive VLMs efficiently (the entire training process for Idefics3 took "5 days on 32 H100 nodes"). Researchers can pursue the explicitly identified gaps and open questions. Evaluators can adopt the recommended practices around format sensitivity, contamination checking, and staged evaluation. The paper's contribution is not a single architectural innovation but a structured understanding of the VLM design space that the field had been accumulating in fragmented form.

3. Technical Approach

3.1 Reader Orientation

This paper operates at two levels: first, it is a systematic survey and tutorial that catalogs the design choices in VLM construction and explains the mechanisms behind competing approaches; second, it is an empirical case study where the authors build Idefics3-8B—a specific VLM that instantiates a particular set of those design choices—to demonstrate what a pragmatic, data-efficient training pipeline looks like in practice. The system being built is a model that takes arbitrary images and text as input and produces text as output, assembled from two frozen pre-trained components (SigLIP-SO400M for vision, Llama 3.1 instruct for language) connected by a pixel shuffle projection layer, trained in four stages with progressively unfrozen parameters and increasing image resolutions. The problem it solves is: given the overwhelming number of design decisions in the VLM pipeline, which concrete choices yield strong performance across diverse multimodal tasks while using only open data and modest computational resources? The solution takes the form of a specific architectural configuration, a curated multi-stage data curriculum, and—critically—a synthetic data generation methodology (Docmatix) that demonstrates how to overcome the document understanding data bottleneck without proprietary datasets.

3.2 Big-Picture Architecture (Diagram in Words)

The Idefics3-8B system and its surrounding data pipeline have five major components:

1. Pre-trained Vision Encoder (SigLIP-SO400M, 400M parameters): Processes image tiles produced by the image-splitting strategy, outputting a grid of hidden states for each 364×364 pixel tile. Frozen during stage 1 training, then trained with DoRA (a LoRA variant) in later stages. This component serves as a fixed-quality visual feature extractor whose ImageNet accuracy correlates with downstream VLM performance.

2. Pixel Shuffle Modality Projection Layer: Maps the vision encoder's output hidden states from the vision-hidden space to the text-hidden space expected by the LLM, while simultaneously reducing the number of visual tokens by a factor of 4 through a spatial-to-channel rearrangement. This replaces Idefics2's perceiver resampler (which compressed to only 64 tokens) with a simpler operation that preserves more fine-grained visual information—169 visual tokens per 364×364 tile instead of 64—which is critical for OCR performance.

3. Pre-trained Language Model (Llama 3.1 instruct, 8B parameters): The autoregressive text backbone that receives the concatenated sequence of visual tokens (with positional markers) and text embeddings, and generates the text response token-by-token. Frozen in stage 1, then trained with DoRA in stages 2–3 and during SFT.

4. Image-Splitting Preprocessor: Before the vision encoder, each input image is divided into a grid of 364×364 pixel tiles, with the grid dimensions depending on the original image's aspect ratio and resolution. The downscaled original image (also 364×364) is appended to the tile sequence to provide global context. Positional text tokens (e.g., <row_x_col_y>) are prepended to each tile's visual tokens, and \n tokens separate tile rows, enabling the model to reconstruct the spatial layout of tiles from the linearized sequence.

5. Training Data Pipeline (multi-source, multi-stage): A curated collection of datasets introduced in a specific curriculum: interleaved image-text documents (OBELICS) and re-captioned image-text pairs (LAION COCO) in stage 1; PDF documents with OCR transcriptions (PDFA) added in stages 2–3; large synthetic datasets (Docmatix for document QA, WebSight for screenshot-to-code, LNQA for real-world VQA, PixelProse for detailed captions, ChartGemma for chart understanding) introduced in stage 3; and the expanded Cauldron (56 academic datasets formatted as question-answer conversations) used during supervised fine-tuning.

Information flows as follows: an image enters the system → the image-splitting preprocessor divides it into a tile grid and appends the downscaled original → each tile (and the downscaled original) is independently encoded by SigLIP-SO400M → the pixel shuffle layer transforms each tile's hidden states from vision-space to text-space while reducing token count by 4× → positional tokens and row separators are inserted → the resulting visual token sequence is concatenated with the text prompt's embeddings → the combined sequence is fed to Llama 3.1 instruct, which autoregressively generates the text response.

3.3 Roadmap for the Deep Dive

• First, the image-splitting strategy and visual token representation, since this defines how visual information reaches the LLM and is the architectural axis where most design disagreements manifest. Understanding why each tile gets encoded separately, how positional information is preserved, and why 169 tokens per tile (vs. 64 in Idefics2) matters for OCR is foundational.

• Second, the pixel shuffle modality projection, the specific connector chosen over alternatives (perceiver resampler, linear projection, cross-attention), including why the authors made this switch from Idefics2 and what alternative connectors exist in the literature.

• Third, the multi-stage pre-training curriculum, because the paper's most detailed empirical contribution is the staged training recipe: what gets frozen when, what data is introduced at each stage, what resolutions are used, and why this particular staging was chosen over alternatives like full unfreezing or different data orderings.

• Fourth, the Docmatix dataset creation pipeline, which is the paper's most novel methodological contribution—a concrete recipe for converting PDFs + OCR + LLMs into large-scale document QA training data—and deserves detailed mechanism-level explanation.

• Fifth, the expanded Cauldron and supervised fine-tuning stage, including the data mixture strategy (upsampling/downsampling by answer token count), the handling of multiple QA pairs per image as multi-turn conversations, and the NEFTune noise addition.

• Sixth, evaluation methodology and prompting, since the paper makes specific claims about how evaluation design (format bias, contamination, the gap between pre-training and fine-tuning metrics) affects our understanding of VLM performance and must be accounted for when interpreting results.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a survey and empirical analysis paper whose core technical contribution is the systematic documentation of VLM design trade-offs (Sections 2–4) combined with the construction of a specific model (Idefics3-8B, Section 5) that instantiates a particular set of choices and validates them empirically. The paper's method is to: (1) survey existing approaches to each design axis, (2) identify where consensus is lacking and why, (3) construct a model using what the authors argue are the most pragmatic choices given the open-source ecosystem and computational constraints, and (4) create and release datasets that fill critical gaps (Docmatix for document understanding).

The Image-Splitting Strategy: Encoding Variable-Resolution Images

The image-splitting strategy addresses a fundamental limitation: most pre-trained vision encoders (including SigLIP-SO400M) are designed for fixed, relatively low image resolutions (e.g., 224×224 or 384×384 pixels). However, real-world VLMs encounter images spanning enormous resolution ranges—from small icons to high-resolution document scans where fine text must be read. Simply resizing all images to the vision encoder's native resolution discards information for large images, while training a vision encoder from scratch for variable resolutions is expensive and computationally infeasible for most research groups.

The mechanism. The strategy, introduced in UReader and SPHINX, works as follows:

1. Tiling: The original image is divided into a grid of square sub-images (tiles), each of size 364×364 pixels. The number of rows and columns in this grid depends on the original image's resolution. For a 1092×728 image, for example, this produces a 3×2 grid (three rows, two columns), yielding 6 tiles, each 364×364.

2. Downscaled original: A copy of the complete original image, resized (downscaled) to 364×364 pixels, is appended to the sequence of tiles. This provides the model with a low-resolution view of the entire image's global context, partially mitigating the fact that encoding tiles separately loses information about relationships between distant image regions.

3. Independent encoding: Each tile and the downscaled original are processed independently by the frozen SigLIP-SO400M vision encoder with shared weights. This is computationally efficient because the encoder's weights are reused across all tiles, and it avoids the need to fine-tune the encoder for larger images. However, it is suboptimal because the tiles are not truly independent—an object may span multiple tiles, and encoding each tile separately means the vision encoder has no awareness of cross-tile relationships. The downscaled original is a partial fix, but "it's not a perfect solution, as the reduced resolution of the original image makes it difficult to capture finer details" (Section 2.2.3).

4. Positional annotation: After encoding, the visual tokens from each tile are linearized into a single sequence. Because the 2D structure of the tile grid is lost during linearization, the authors prepend each tile's visual tokens with text tokens of the form <row_x_col_y>, where x and y indicate the tile's position in the grid (following mPLUG-DocOwl-1.5's approach). Additionally, a newline text token \n is inserted after each row of tiles to provide a textual row separator. These positional markers give the LLM explicit information about the spatial arrangement of tiles, which it can use to reason about object continuity across tile boundaries.

5. Variable token count: Because the number of tiles depends on image resolution, the number of visual tokens per image is variable. This is beneficial during inference: for simple tasks requiring only coarse visual understanding, the user can provide low-resolution images (fewer tiles, fewer tokens, less computation), while for OCR-heavy tasks, higher-resolution images produce more tiles and more visual tokens, allocating more computation to visual processing. During training, the model sees a varying number of visual tokens, which teaches it to handle the flexible-resolution inference regime.

Why this approach and not alternatives? The alternative of developing a vision encoder that natively processes images of any resolution (which the paper flags as a "promising direction for future research") would require training a new encoder from scratch with a mechanism like Patch'n'Pack, which pads and packs images of different sizes into fixed-size training batches. This is a larger research undertaking that the authors identify as future work. The image-splitting strategy is a pragmatic compromise: it uses an off-the-shelf fixed-resolution encoder without modification, scales to arbitrary image sizes at inference, and is straightforward to implement.

Resolution curriculum during Idefics3 training (Table 3). The maximum image resolution is progressively increased across training stages:

• Stage 1: 364² (single tile)
• Stage 2: 364² → 728² (to 2×2 tile grid) → 1092² (to 3×3 tile grid) → 1456² (to 4×4 tile grid) → 1820² (to 5×5 tile grid, the maximum)
• Stage 3 and SFT: 1820² throughout

This gradual increase means that in early stages, the model learns basic alignment with small, fast-to-process images, while in later stages, it encounters the large images needed for document understanding and OCR tasks. The authors note that "once the resolution is sufficiently high, datasets containing large images, such as PDFs, can be incorporated into the training data" (Section 3.1)—this is why PDFA appears in stages 2–3 but not stage 1.

Idefics3 vs. Idefics2 visual tokens. In Idefics2, the perceiver resampler compressed each image (up to 980×980 pixels, encoded as tiles when using image splitting) into exactly 64 visual tokens. The paper found this was a "bottleneck for OCR tasks" (Section 5.2.1). Idefics3 uses the pixel shuffle strategy to encode each 364×364 tile into 169 tokens (a 4× compression from the raw vision encoder output). For a maximum-resolution image of 1820² (5×5 tiles), this yields approximately 5 × 5 × 169 = 4,225 visual tokens (plus the downscaled original's 169 tokens), representing a dramatic increase in visual throughput compared to Idefics2's fixed 64 tokens. This increase is the primary architectural change enabling the 13.7-point improvement on DocVQA.

The Pixel Shuffle Modality Projection: Connecting Vision to Language

The modality projection layer sits between the vision encoder and the language model, performing two functions: (1) it maps hidden states from the vision encoder's representation space dimension to the language model's embedding dimension, and (2) it optionally reduces (pools) the number of visual tokens to control sequence length and computational cost. The design of this component determines how much visual information survives the transition from the vision encoder to the LLM.

The mechanism of pixel shuffle. Pixel shuffle (also called depth-to-space rearrangement) is a simple operation that reduces spatial dimensions while increasing channel dimensions, with no learned parameters. It works by taking blocks of 2×2 spatially adjacent hidden states and rearranging them from the spatial dimensions into the channel dimension:

1. The vision encoder outputs a grid of hidden states of shape $H \times W \times C_{\text{vision}}$, where $H$ and $W$ are the spatial dimensions (e.g., for a 364×364 image processed by a ViT with patch size 14, $H = W = 26$) and $C_{\text{vision}}$ is the vision encoder's hidden dimension (1152 for SigLIP-SO400M).

2. Pixel shuffle with a downscaling factor of 2 rearranges each 2×2 spatial block into the channel dimension, producing an output of shape $H/2 \times W/2 \times 4C_{\text{vision}}$. This reduces the spatial dimensions by a factor of 2 in each direction (total token count reduced by 4×) while quadrupling the channel dimension.

3. A learned linear projection then maps the quadrupled channel dimension from $4C_{\text{vision}}$ to the LLM's hidden dimension $C_{\text{LLM}}$ (4096 for Llama 3.1 8B). The result is a grid of visual tokens of shape $H/2 \times W/2 \times C_{\text{LLM}}$.

For SigLIP-SO400M with a 364×364 input and patch size 14, the initial grid is 26×26 = 676 patches. After pixel shuffle (factor 2), this becomes 13×13 = 169 visual tokens per tile. This is the "4× compression" referred to throughout the paper.

Why pixel shuffle over alternatives? The paper surveys several connector designs in Section 2.2.2, each with different trade-offs:

• Simple linear projection (LLaVA, FROMAGe): Retains all visual tokens (676 for the above example). This preserves maximum information but results in long sequences, making training and inference expensive. No compression is applied.

• Perceiver resampler (Idefics2, Flamingo): Uses a cross-attention mechanism where a fixed number of learned query vectors (e.g., 64) attend to the visual hidden states, compressing an arbitrary number of visual features into a fixed-length sequence. This is efficient (only 64 tokens per image) but aggressively compress information—the authors found it dropped OCR performance.

• Single-layer cross-attention (Qwen-VL): A single cross-attention layer between a group of learnable embeddings and the image hidden states, providing moderate compression with some learned parameters.

• C-Abstractor (HoneyBee): Uses 2D convolutional ResNet blocks after the vision encoder to process visual features while preserving spatial structure, followed by learned compression.

• H-Reducer (mPLUG-DocOwl-1.5): Applies convolutions to divide the number of image hidden states by 4, conceptually similar to pixel shuffle but with learned convolutional parameters.

The authors chose pixel shuffle for Idefics3 because it provides a moderate compression factor (4×, producing 169 tokens per tile vs. Idefics2's 64 total) using a simple, parameter-free operation that preserves the 2D spatial structure of the visual features. Unlike the perceiver resampler, which can learn to ignore fine-grained details during compression, pixel shuffle is deterministic and spatially local: each output token corresponds to a specific 2×2 patch region in the vision encoder's feature map, making it harder for information to be lost accidentally. The trade-off is that 169 tokens per tile (vs. 64 total in Idefics2) increases sequence length and computational cost—but the authors judged this to be necessary for OCR performance.

Comparison with Idefics2's perceiver resampler. Idefics2 used a perceiver resampler with 64 learned query embeddings, compressing the entire image (potentially composed of multiple tiles) into exactly 64 visual tokens regardless of resolution. The paper found that "the number of visual tokens can be compressed to as few as 64 (divided by 77) while maintaining performance for most tasks, except those that require extensive OCR capabilities" (Section 2.2.2). This directly motivated the switch to pixel shuffle for Idefics3: document understanding was the priority, and the perceiver resampler's aggressive compression was the primary bottleneck for that task.

Multi-Stage Pre-Training: The Curriculum and Parameter Freezing Schedule

The training of Idefics3 proceeds through three pre-training stages followed by supervised fine-tuning (SFT), summarized in Table 3 of the paper. The design of this multi-stage curriculum reflects several pragmatic constraints and empirical observations about VLM training dynamics.

Stage 1: Frozen backbones, low resolution, foundational alignment (1,000 steps)

In stage 1, both SigLIP-SO400M and Llama 3.1 instruct are completely frozen. Only the newly initialized parameters of the pixel shuffle modality projection layer are trained. The maximum image resolution is 364² (a single tile, 169 visual tokens per image). The data consists of OBELICS (interleaved image-text documents) and LAION COCO (re-captioned image-text pairs).

Why freeze everything except the connector? The frozen backbone approach serves multiple purposes. First, it preserves the pre-trained performance of both unimodal models: the LLM retains its text-only capabilities, and the vision encoder retains its visual representation quality. Since the connector is the only newly initialized component, its weights start from random values; training it in isolation prevents the noisy gradients from randomly initialized weights from corrupting the pre-trained backbones. Second, it is computationally efficient because backpropagation only flows through the small connector, not the 8B-parameter LLM or 400M-parameter vision encoder. Third, prior work (VILA, LLaVA-NeXT) has shown that "beginning training by freezing the backbone models and focusing solely on the newly initialized parameters (the connector) until a satisfactory performance level is achieved" (Section 3.1) is a reliable strategy.

Why OBELICS and LAION COCO? OBELICS is an open-source dataset of 141 million interleaved image-text documents extracted from Common Crawl HTML files. The authors highlight three advantages of interleaved documents: (a) they enhance in-context learning abilities, (b) they teach the model to handle an arbitrary number of images interleaved with text, and (c) they expose the model to a much wider distribution of texts than standard image-text pair datasets (Section 3.1). OBELICS documents maintain the original linearity of images and texts as they appeared on websites, with spam and ads removed. LAION COCO provides 600 million images re-captioned with synthetic captions from an ensemble of BLIP and CLIP models, addressing the noise problem in raw alt-text datasets. The combination of interleaved documents (for sequence-level understanding) and paired data (for image-text alignment) provides a strong foundation for later stages.

The learning rate is kept constant at $10^{-4}$ throughout stage 1 (no decay), with a batch size of 1024 and sequence length of 10K tokens. The authors note that "during the first two pre-training stages, the loss function is far from converging, but we move to the next stage to reduce computational costs" (Section 5.2.1)—an explicit efficiency-vs-optimality trade-off.

Stage 2: DoRA training on backbones, increasing resolution, PDFs introduced (3,000 steps)

In stage 2, the backbones are trained using DoRA (Weight-Decomposed Low-Rank Adaptation), a variant of LoRA, rather than full weight updates. The maximum image resolution is progressively increased from 364² to 1820² (the full 5×5 tile grid) over the course of the 3,000 steps. The data includes OBELICS, LAION COCO, and now PDFA (English-only filtered PDF documents with OCR transcriptions, containing 18M pages).

Why DoRA instead of full fine-tuning or standard LoRA? The authors state that "we did not encounter instabilities when fully unfreezing the backbones, we opt for a LoRA approach to enhance training efficiency" (Section 5.2.1). DoRA (Liu et al., 2024) decomposes pre-trained weights into magnitude and direction components, then applies low-rank updates to the direction component. The paper claims this is more parameter-efficient than standard LoRA while providing stronger regularization than full unfreezing. The practical motivation is clear: training 8B parameters at full precision through multiple stages would require substantially more GPU memory and time than DoRA, and the 5-day training time on 32 H100 nodes was achieved partly through this efficiency choice. However, the authors explicitly caveat this decision: "we believe that carefully executed full unfreezing can lead to better performance" (Section 5.2.1).

Why introduce PDFA now? PDF documents contain large images (full pages of text), making them unsuitable for the low-resolution stage 1. By stage 2, the resolution has increased sufficiently to accommodate document images. PDFA's OCR transcriptions provide the text content of each page, linearized from bounding box coordinates. Training on these transcriptions teaches the model to map document images to their textual content—a form of implicit OCR training that builds toward the document understanding capabilities tested on DocVQA.

The learning rate is again kept constant at $10^{-4}$ throughout stage 2.

Stage 3: Large synthetic datasets, resolution at maximum, continued DoRA (1,500 steps)

Stage 3 introduces large-scale synthetic datasets at the maximum resolution (1820²), continuing DoRA training on the backbones. The data includes: PDFA, Docmatix (document QA pairs), WebSight (screenshot-to-HTML-code pairs), LNQA (real-world visual question answering), PixelProse (detailed image captions), and ChartGemma (chart understanding QA pairs). The learning rate is linearly decayed from $10^{-4}$ to 0 over the 1,500 steps (unlike the constant rates in stages 1–2).

Why synthetic data in stage 3? The paper's key insight about training data is that web-crawled datasets (image-text pairs, interleaved documents, PDFs) provide broad distributional coverage and teach foundational skills (captioning, text transcription, handling interleaved images), but they "fall short in addressing many of the tasks that users typically require, such as document understanding or visual math reasoning, which are significantly more challenging" (Section 3.1). Synthetic datasets bridge this gap by providing examples that closely resemble the tasks users will actually request. Introducing them in stage 3, after the model has learned basic multimodal alignment and document transcription in stages 1–2, allows the model to build complex task-specific capabilities on top of solid foundation skills.

Why only 1,500 steps with a subset of available data? The authors acknowledge this as a pragmatic compromise: "only a fraction of the examples available in the chosen datasets are used, again to reduce computational demands" (Section 5.2.1). This is an explicit limitation—the model would likely benefit from training on more synthetic data for more steps—but the 5-day training budget constrained this choice.

The entire pre-training process runs on 32 H100 nodes (256 GPUs) for 5 days, including restarts. The total pre-training steps across all three stages is 5,500 (1,000 + 3,000 + 1,500), using a constant batch size of 1024 and sequence length of 10K tokens throughout.

What this curriculum does NOT include, and why. The paper notes several omissions. First, no alignment phase (DPO or RLHF) is applied after SFT—the authors observe that "since the model was mainly trained on short answers during supervised fine-tuning, and did not benefit from an alignment phase, we observe that it can sometimes struggle to follow instructions for more challenging prompts" (Section 5.2.2). Second, no PRM tree-search is combined with revisions (the paper's acknowledgments flag this as future work). Third, the third pre-training stage does not include several synthetic dataset types that the paper identifies as promising in Section 3.1: table understanding datasets, reasoning with chain-of-thought datasets, visual mathematical reasoning datasets, or object localization datasets. The authors explicitly state that "further significant improvements can be achieved by creating and incorporating the synthetic datasets mentioned in Section 3.1 into the stage 3 data mixture" (Section 5.2.1).

The Docmatix Dataset: Synthetic Document QA at Scale

Docmatix is the paper's most significant data contribution: a dataset of 2.4 million images and 9.5 million QA pairs derived from 1.3 million PDF documents, representing a 240-fold scale increase compared to previously available open document-understanding datasets (DocVQA's 40K QA pairs, InfoVQA's 10K QA pairs, VisualMRC's 12K QA pairs). The dataset is created through a pipeline that separates the visual processing (OCR) from the language understanding (QA generation), enabling the use of specialized tools for each sub-problem.

Step 1: Source documents. The pipeline begins with the text transcriptions from the English PDFA dataset, which contains 18M pages of PDF documents sourced from Common Crawl. PDFA's text transcriptions were obtained using OCR extraction tools that detected text regions in each page image, recognized the characters, and saved both the recognized text and its bounding box coordinates. The linearized text transcription preserves the reading order of the document (left-to-right, top-to-bottom within each page), though the paper notes that "linearizing texts coherently from bounding boxes can be challenging, and math equations are often inaccurately transcribed or omitted" (Section 3.1).

Step 2: QA pair generation with Phi-3-small. The text transcriptions are fed as input to Phi-3-small (a text-only LLM), which is prompted to generate question-answer pairs based on the document content. The generation uses five different prompts to ensure diversity in question types, difficulty levels, and answer formats. This is a critical design choice: if only one prompt template were used, the generated QA pairs would all share similar structure and the model trained on them might not generalize to the varied question styles found in benchmarks and real-world use.

The choice of Phi-3-small as the generator is pragmatic—it is a capable open model that can run efficiently at scale on 1.3 million documents. Using a stronger proprietary model (e.g., GPT-4) might produce higher-quality QA pairs but would introduce cost, rate limits, and licensing complications. The paper suggests that "enhancements could involve generating more diverse questions, such as summarizing a paragraph, and employing a strong VLM to filter out erroneous generated QA pairs" (Section 3.1), implying that the current pipeline likely produces some noisy or low-quality pairs.

Step 3: Quality filtering. The raw generated QA pairs are filtered to remove cases where the generation failed or produced unusable content. Specifically, the paper uses "regular expressions to detect code and removing answers containing the keyword 'unanswerable'" (Section 5.1.2). This discards approximately 15% of QA pairs. The "unanswerable" filter is important because LLM-based generators sometimes respond with "I cannot answer this question" or similar disclaimers when the text transcription is noisy or ambiguous; including such pairs in training data would teach the VLM to refuse to answer rather than to extract information from documents.

Step 4: Resulting dataset statistics. The final Docmatix dataset contains:

• 2.4 million images (document pages from the PDFA source)
• 9.5 million QA pairs (an average of ~4 QA pairs per image)
• Sources from 1.3 million PDF documents, with documents up to 4 pages long

The 240× scale increase is measured relative to DocVQA (40K QA pairs), the largest previously available open document-understanding dataset. The dataset is released publicly on Hugging Face.

Validation of Docmatix's effectiveness (Table 2). To assess whether the synthetic QA pairs provide useful training signal, the authors conduct an ablation with Florence-2, a 0.7B-parameter specialist vision model. They train two versions:

1. Florence-2 trained over multiple epochs on DocVQA alone (the standard fine-tuning approach)
2. Florence-2 trained for one epoch on a subset of Docmatix (20% of images, ~480K images; 4% of QA pairs, ~380K QA pairs), followed by one epoch on DocVQA to ensure proper evaluation format

The results (Table 2): the Docmatix-trained version achieves a DocVQA ANLS score of 71.4, compared to 60.1 for the DocVQA-only version—a nearly 20% relative improvement. Moreover, this 0.7B specialist model is only 5% worse than the 8B Idefics2 generalist model (74.0 ANLS), demonstrating that targeted synthetic data can partially compensate for model scale when the task distribution is narrow.

The paper also notes that Docmatix has already been used by external teams: "since Docmatix was made publicly available prior to this paper, it has already been used to enhance the performance of the moondream2 model, which achieved a 103% improvement on DocVQA compared to its previous version" (Section 5.1.2).

Why separate OCR from QA generation? The paper explicitly motivates this design: "generating high-quality synthetic data for this task is relatively straightforward if we reframe the problem as one of LLM-based data generation rather than relying solely on VLMs" (Section 5.1.2). The alternative—using a VLM to look at document images and generate QA pairs directly—would require the VLM to be strong at both OCR and question generation, creating a chicken-and-egg problem: you need a good VLM to generate training data for a good VLM. By separating the pipeline, OCR can be done with specialized tools (which are strong at text extraction but know nothing about semantics), and QA generation can be done with a text-only LLM (which is strong at language understanding but knows nothing about visual layout). The text transcription serves as a lossy but sufficient intermediate representation.

Limitations of this approach. The paper acknowledges several. First, OCR transcription quality is imperfect: math equations are often inaccurately transcribed or omitted, figures and tables are poorly handled. The paper suggests that "a better strategy for text transcription would involve combining a traditional OCR tool, a document-specialized model like Nougat [for math], and a robust VLM to judge, refine, and merge the outputs of these models" (Section 3.1). Second, the generated QA pairs are derived from text, not from the visual layout, so questions about formatting, positioning, or visual elements may be underrepresented. Third, Phi-3-small may introduce its own biases or errors into the generated QA pairs, and the filtering is relatively crude (regex patterns + keyword matching).

The Expanded Cauldron and Supervised Fine-Tuning

After the three pre-training stages, the model undergoes supervised fine-tuning (SFT) on a curated collection of instruction-formatted datasets. This stage teaches the model the specific skill of visual question answering—following instructions, producing answers in appropriate formats, and handling the diverse task types that users will request.

The Cauldron dataset collection. The Cauldron, introduced in Idefics2, is "a collection of 50 high-quality datasets covering a broad range of tasks, including general visual question answering, counting, captioning, text transcription, document understanding, chart/figure analysis, table understanding, visual reasoning, geometry, spotting differences between two images, and converting screenshots into functional code" (Section 3.2). Each dataset is formatted into a standardized question/answer format, with the specific prompt template varying by task type. When multiple QA pairs exist for a single image, they are combined into a multi-turn conversation, teaching the model to engage in extended interactions about a single image. For Idefics3, the Cauldron is expanded with 6 additional datasets: Cord-v2 (JSON format output), LNQA (large-scale real-world VQA), ShareGPT-4o (detailed captions from GPT-4o), IIW-400 (hyper-detailed image descriptions), Geo170K (geometry problems), and Docmatix (document QA).

Data mixture strategy (Table 1). The authors face a challenge: the datasets vary enormously in size, and naively combining them would cause the model to overfit to the largest datasets while underfitting the smaller ones. Their solution is to upsample or downsample each dataset based on the total number of answer tokens it contributes to the mixture. Specifically, Table 1 shows for each dataset: (a) the number of different images, (b) the number of QA pairs, (c) the total number of tokens in all answers, and (d) the selected percentage of answer tokens in the final mixture after upsampling/downsampling.

For example, ShareGPT-4o contributes 39.7M answer tokens and is assigned 13.03% of the mixture, while HatefulMemes contributes only 25.5K answer tokens and is assigned 0.08%. This token-based balancing ensures that the model sees a diverse distribution of tasks during training, with each dataset's contribution proportional to the amount of learning signal (as measured by answer length) it provides, rather than its raw image count.

Why balance by answer tokens? The alternative—balancing by number of images or QA pairs—would overweight datasets with short answers (e.g., VQAv2, where answers are typically 1–2 words) and underweight datasets with long, detailed answers (e.g., ShareGPT-4o, where captions are paragraphs). Since the loss is computed only on answer tokens (the model is trained to predict the answer given the image and question), balancing by answer tokens ensures that the total gradient signal from each dataset is roughly proportional to its token contribution.

SFT training details (Table 3). The SFT stage runs for 5,000 steps with a maximum learning rate of $5 \times 10^{-5}$ linearly decayed to 0, batch size 1024, sequence length 10K, and maximum image resolution 1820². The backbones continue to be trained with DoRA (the same parameter-efficient approach from stages 2–3). NEFTune noise (Jain et al., 2024) is applied to the input embeddings during training. The loss is computed only on the answer tokens (not on the question/prompt tokens), following the standard instruction tuning paradigm where the model learns to produce the correct response given the prompt, but is not penalized for its "beliefs" about the prompt tokens.

Why NEFTune? NEFTune adds random noise to the embedding vectors during training, which has been shown to improve instruction-following and reduce overfitting in LLM fine-tuning. The mechanism is simple: before the forward pass, Gaussian noise with zero mean and small variance is added to the input embeddings. This acts as a regularizer, preventing the model from memorizing exact prompt-answer pairs and encouraging generalization.

The SFT data does NOT include an alignment component. The authors explicitly note this as a limitation: "since the model was mainly trained on short answers during supervised fine-tuning, and did not benefit from an alignment phase, we observe that it can sometimes struggle to follow instructions for more challenging prompts" (Section 5.2.2). Alignment phases (using DPO or RLHF on preference data) are typically employed to make models produce more helpful, detailed, and instruction-following responses. The datasets for this exist (RLHF-V, RLAIF-V, VLFeedback, SPA-VL) but were not used for Idefics3. The authors mitigate this somewhat by noting that "adding a brief prefix to the assistant's response allows the user to easily shape the generated output as desired" (Section 5.2.2)—a prompting workaround for the lack of alignment training.

Text-only instruction data is included. Table 1 shows that approximately 15% of the SFT mixture (by answer tokens) comes from text-only instruction datasets: OpenHermes-2.5, MetaMathQA, AtlasMathSets, MathInstruct, OrcaMath, Goat, LIMA, Dolly, and CamelAIMath. This is important because training exclusively on multimodal data can cause catastrophic forgetting of the LLM's original text-only capabilities. By mixing in text-only instruction data, the model maintains its ability to engage in text-only conversations and reason about text-only problems, even as it acquires visual understanding skills.

Evaluation Methodology and Prompting

The paper's evaluation of Idefics3 uses five benchmarks that test different capabilities: MMMU (multi-discipline college-level problems, multiple-choice), MathVista (visual mathematical reasoning, mix of open-ended and multiple-choice), MMStar (general image understanding, multiple-choice), DocVQA (document understanding, open-ended short answer), and TextVQA (text reading in natural images, open-ended short answer). All evaluations are zero-shot (no in-context examples) and without chain-of-thought prompting.

Benchmark-specific prompts. The paper provides the exact prompts used for evaluation in Appendix A.1.1:

For multiple-choice benchmarks (MMMU, MathVista, MMStar), the default template matches what was seen during SFT:

Question: {question}
Choices:
A. {choice_a}
B. {choice_b}
C. {choice_c}
D. {choice_d}
...
Answer with the letter.

For TextVQA, the prompt includes explicit formatting instructions to constrain the model's output style:

Answer the following question about the image using as few words as possible. 
Follow these additional instructions:
-Always answer a binary question with Yes or No.
-When asked what time it is, reply with the time seen in the image.
-Do not put any full stops at the end of the answer.
-Do not put quotation marks around the answer.
-An answer with one or two words is favorable.
-Do not apply common sense knowledge. The answer can be found in the image.
Question: {question}

This detailed prompt is necessary because TextVQA is an open-ended benchmark with exact-match scoring; small formatting differences (e.g., "Yes." vs. "Yes") can cause a correct answer to be marked wrong. The instructions attempt to constrain the model's output format to match what the benchmark's evaluation script expects.

For DocVQA, the prompt similarly constrains format:

Give a short and terse answer to the following question. Do not paraphrase or reformat 
the text you see in the image. Do not include any full stops. Just give the answer 
without additional explanation.
Question: {question}

Image resolution at evaluation. For Idefics3, images are resized such that the longest side is 4×364 pixels for most benchmarks. The exception is DocVQA, which has larger images; for DocVQA, the longest side is resized to 5×364 pixels, matching the maximum resolution used during training (1820²). For comparison, Idefics2-70B images are resized to 1960 pixels on the longest side, matching its training resolution. This means the comparison between Idefics3-8B and Idefics2-70B is not perfectly resolution-matched—the 8B model sees images at up to 1820 pixels while the 70B model sees images at up to 1960 pixels, though the difference is small.

Generation stopping criteria. The model stops generating when it produces one of the stop tokens: Question, User, <end_of_utterance>, or the EOS token. This prevents the model from hallucinating additional conversation turns after answering the question.

4. Key Insights and Innovations

Innovation 1: The "Data Reframing" Strategy — Separating Visual Processing from Language Understanding to Bypass VLM Data Bottlenecks

The most conceptually significant move in this paper is not an architectural choice or a training recipe, but a strategy for generating training data that decouples visual perception from language reasoning. The traditional approach to building VLM training data for complex tasks like document understanding assumes you need a capable VLM—either to generate QA pairs directly from images or to serve as a teacher model. This creates a circular dependency: you need a strong VLM to generate data to train a strong VLM, which is why document understanding has been dominated by proprietary systems with access to private data pipelines.

Docmatix breaks this circularity by reframing document QA generation as a two-stage pipeline where specialized, non-VLM tools handle each stage independently. The key insight is that document understanding can be decomposed into (1) extracting text from document images (a visual perception problem, solved by OCR) and (2) generating questions and answers about that text (a language understanding problem, solved by a text-only LLM). Neither stage requires a VLM. The OCR tool (used to create PDFA) is specialized for text extraction and operates purely on pixel patterns; Phi-3-small, the QA generator, operates purely on text transcriptions and has no visual capabilities whatsoever. The paper calls this out directly: "generating high-quality synthetic data for this task is relatively straightforward if we reframe the problem as one of LLM-based data generation rather than relying solely on VLMs" (Section 5.1.2).

This reframing is conceptually distinct from prior synthetic data approaches in VLMs. Datasets like LAION COCO and VeCap use VLMs (BLIP, CLIP, LLaVA) to look at images and generate captions—they keep visual perception and language generation coupled. PixelProse uses Gemini to caption images—again, a VLM does the work. The Docmatix approach is different: it recognizes that for structured documents, the intermediate representation (text transcription) is a sufficient lossy compression that preserves enough information for QA generation while being processable by a text-only toolchain. This transforms the data scaling problem from "we need a better VLM teacher" to "we need better OCR and better text LLMs"—two resources that are independently improvable and already exist at scale in the open ecosystem.

The significance is not the 240× scale increase per se (though that's practically important), but the methodological template this creates for other data-starved VLM tasks. The paper suggests extensions: for math understanding, "combining a traditional OCR tool, a document-specialized model like Nougat, and a robust VLM to judge, refine, and merge the outputs" (Section 3.1). For table understanding, one could use table extraction tools + text LLMs. For chart understanding, one could use structured data extraction + LLMs. Each case follows the same pattern: identify a modality-specific extraction tool that produces a text representation, then use a text LLM for the reasoning task. This is not an incremental improvement on existing VLM data generation methods—it's a fundamental rearchitecture of the pipeline that sidesteps the chicken-and-egg problem entirely.

The Florence-2 ablation (Table 2) provides evidence that this decoupled pipeline produces genuinely useful training signal, not just more data: +11.3 ANLS points on DocVQA from training on a small Docmatix subset—a 19% relative improvement—using a model that never saw the original document images during the QA generation phase.

Innovation 2: The Visual Token Bottleneck as a First-Class Diagnostic Concept

The paper identifies and names a specific failure mode in VLM architectures that was previously observed but never systematized: the visual token bottleneck. The idea is that for certain task categories (primarily OCR-heavy document understanding), the number of visual tokens passed from the vision encoder to the language model is the binding constraint on performance, not the model's reasoning capability, the quality of the vision encoder, or the amount of training data. The paper treats this not as a vague intuition but as a diagnostic concept that explains divergent findings across prior work and predicts where architectural choices will matter.

This diagnosis explains several otherwise puzzling observations in the literature. Idefics2's perceiver resampler compressed images to 64 tokens and "maintained performance for most tasks, except those that require extensive OCR capabilities" (Section 2.2.2)—a finding that makes sense under the bottleneck framework: most tasks don't need fine-grained visual detail, so aggressive compression is harmless, but OCR requires pixel-level text recognition, so the bottleneck binds. InternLM-XComposer2-4KHD found that increasing visual tokens per image was "primarily necessary for benchmarks focused on OCR tasks, such as InfoVQA and DocVQA" (Section 2.2.2)—again consistent with the bottleneck affecting only specific task categories. The image-splitting strategy itself can be understood as a workaround for the bottleneck: rather than building a vision encoder that natively handles high resolution (which would eliminate the bottleneck at its source), practitioners tile and re-encode, increasing token count at the cost of redundant computation.

Prior to this paper, the relationship between visual token count and task-specific performance was known anecdotally but not framed as a systematic diagnostic. Researchers would observe that their model struggled on TextVQA or DocVQA and try various fixes—better OCR data, larger LLMs, different training recipes—without recognizing that the architectural token throughput was physically limiting the information available to the LLM. The paper's contribution is to elevate this from a "gotcha" to a design principle: when building a VLM, first decide what resolution of visual detail your target tasks require, then design the connector and token budget accordingly. This reframes the connector design question from "which architecture works best?" to "what information throughput does my task require, and which connectors can provide it?"

The bottleneck concept also generates testable predictions. It predicts that models with different vision encoders but identical token budgets should perform similarly on OCR tasks (since the bottleneck is the information channel capacity, not the encoder quality). It predicts that increasing token count should improve OCR performance with diminishing returns—more tokens help until the bottleneck shifts to the LLM's ability to process long sequences or the vision encoder's resolution limit. It predicts that tasks requiring spatial reasoning about global image structure (not fine detail) should be unaffected by token count changes, since the relevant information was already preserved at low token counts. The paper doesn't test all these predictions, but the framework makes them explicit and falsifiable.

The Idefics3 architectural choices instantiate this diagnostic: switching from the perceiver resampler (64 tokens, Idefics2) to pixel shuffle (169 tokens per tile, up to ~4,225 tokens for a 5×5 tile grid) was the primary architectural change, and the 13.7-point DocVQA improvement is attributed largely to this increased visual throughput.

Innovation 3: The Delayed Feedback Loop as a Methodological Warning for VLM Research

The paper identifies a measurement pathology in how VLMs are evaluated during development, and in doing so provides an explanation for why so many architectural choices in the literature are poorly understood. The core observation: "the impact of development choices in the VLM may only become evident after fine-tuning, leading to a delayed feedback loop. This delay can make pre-training ablations misleading" (Section 4.2).

This is not merely a complaint about evaluation practices. It is a methodological diagnosis with a specific mechanism. During pre-training, the model is trained primarily on web-crawled data (image-text pairs, interleaved documents, PDF transcriptions) that teach foundational skills: captioning, text transcription, in-context learning. Complex question-answering capabilities only develop during the supervised fine-tuning stage, when the model sees instruction-formatted data with varied task types. If a researcher ablates an architectural choice during pre-training and evaluates on pre-training metrics (which measure foundational skills), they are measuring the architecture's impact on a different capability distribution than what deployers care about. The Idefics2 example is concrete: increasing visual tokens from 64 to 128 showed "no noticeable improvements during pre-training," but "the benefit of using more visual tokens per image became apparent in OCR tasks after fine-tuning with the image-splitting strategy" (Section 4.2).

The practical consequence is that the research community has likely accumulated misleading results. A paper that ablates connector designs during pre-training and concludes "simpler connectors work as well as complex ones" might be correct for pre-training metrics but wrong for downstream OCR tasks, where connector throughput is the bottleneck. The delayed feedback loop means that pre-training-only evaluations systematically underestimate the importance of architectural choices that affect information capacity, because pre-training tasks don't stress that capacity. The paper's recommendation—to incorporate instruction data into the pre-training data mixture—is a practical fix, but the deeper contribution is the identification of why the field has had such difficulty reaching consensus on architectural questions.

This diagnosis also reframes the challenge of building VLMs. It suggests that the multi-stage training pipeline is not just a computational convenience (though it is that too), but a fundamental obstacle to clear scientific understanding: the optimization surface is non-stationary with respect to evaluation metrics, so early-stage signals about architectural quality are unreliable. This is a subtle but important point—it's not that pre-training metrics are "wrong," but that they measure a different target distribution, and the mapping from pre-training to post-SFT performance is task-dependent.

Innovation 4: The Evaluation Design Critique — Format Bias and Contamination as Threats to Valid Architectural Comparison

The paper's analysis of VLM evaluation (Section 4) goes beyond the standard "benchmarks are noisy" criticism to identify two specific mechanisms by which evaluation design can produce misleading comparisons between models, with direct implications for how the field should interpret published results.

Format bias (Section 4.1) is the mechanism by which open-ended benchmarks with exact-match scoring penalize models that produce correct answers in an unexpected format or writing style. The paper provides a striking quantitative example: on VQAv2, Gemini 1.0 Ultra (77.8) and GPT-4V (77.2) score lower than MM1-3B-Chat (82.0) and moondream2 (79.4, 1.9B parameters), despite being far more capable models. The explanation is not that smaller models are "better at VQA," but that they were fine-tuned on VQAv2's training set and learned its specific answer format, while the larger models produce correct answers that fail exact-match due to formatting differences (e.g., "two" vs. "2", or including an article where the ground truth omits it). This means fine-tuning on benchmark training data can create the illusion of capability that doesn't generalize to real-world tasks, and that comparisons between fine-tuned specialist models and zero-shot generalist models are systematically biased in favor of the specialists.

Contamination (Section 4.3) is the mechanism by which benchmark questions (or their images) appear in training data, inflating scores through memorization rather than capability. The paper quantifies this for MathVista: at least 6.6% of questions include images from training sets of academic datasets used in SFT, 2.2% are near-duplicates, and 6.1% ask variants of questions also present in KVQA. The implication is that models fine-tuned on these datasets will have an unfair advantage on MathVista that doesn't reflect genuine visual reasoning improvement.

The innovation is not in identifying these problems (prior work has discussed benchmark issues), but in articulating how they specifically interact with VLM architectural comparisons. If format bias systematically advantages fine-tuned models over zero-shot models, then papers comparing a fine-tuned self-attention architecture against a zero-shot cross-attention architecture are measuring format adaptation, not architectural quality. If contamination affects some benchmarks but not others, then apparently "uneven" improvements across benchmarks may reflect differential contamination rather than differential capability. The paper's call to "exclude images used in the benchmarks they evaluate from their supervised fine-tuning data" (Section 4.3) is a concrete, actionable recommendation, but the deeper contribution is the argument that valid architectural comparison requires controlling for these evaluation artifacts, and that much of the existing literature has not done so.

This is significant because it provides an alternative explanation for the "lack of consensus" the paper identifies in its introduction. If architectural choices interact with evaluation methodology in ways that create spurious performance differences, then disagreements between papers may reflect differences in evaluation protocols rather than genuine architectural trade-offs. The paper doesn't resolve this—it can't retroactively fix evaluations in prior work—but it provides the diagnostic vocabulary for the field to recognize and correct these issues going forward.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary benchmark suite spans five evaluation datasets, all used in a zero-shot setting. MMMU (Yue et al., 2024) tests multi-discipline college-level problems (validation split), MathVista (Lu et al., 2024) tests visual mathematical reasoning (testmini split), MMStar (Chen et al., 2024) tests general image understanding (validation split), DocVQA (Mathew et al., 2021) tests document understanding (test split), and TextVQA (Singh et al., 2019) tests text reading in natural images (validation split). Together these cover a range of capabilities from OCR to reasoning to general visual comprehension.

• Base model(s). Idefics3-8B is the primary model evaluated, built from Llama 3.1 instruct (8B parameters, Dubey et al., 2024) as the language backbone and SigLIP-SO400M (400M parameters, Zhai et al., 2023) as the vision encoder, connected via a pixel shuffle projection layer. The two comparison models are Idefics2-8B (Laurençon et al., 2024), which uses Mistral-7B as the LLM, SigLIP-SO400M as the vision encoder, and a perceiver resampler connector, and Idefics2-70B, which uses the same architecture as Idefics2-8B but with a 70B-parameter language model. These baselines enable the paper to attribute performance differences to both the language model upgrade (Mistral-7B → Llama 3.1 8B), the connector change (perceiver resampler → pixel shuffle), and the training data improvements (notably Docmatix).

• Metrics. MMMU is scored using the MMMU score (a weighted accuracy across 30 subject categories, computed by the VLMEvalKit library, Duan et al., 2024). MathVista is also scored using the MMMU score via VLMEvalKit. MMStar uses standard accuracy (percentage of multiple-choice questions answered correctly). DocVQA uses the ANLS (Average Normalized Levenshtein Similarity) score, which measures how closely the model's answer matches the ground truth at the character level, robust to minor spelling or formatting variations. TextVQA uses VQA accuracy (exact string match after normalization). All evaluations are zero-shot without chain-of-thought prompting.

• Baselines. The paper compares Idefics3-8B against two prior models from the same research lineage: Idefics2-8B (same vision encoder, weaker LLM—Mistral-7B, perceiver resampler connector, no Docmatix in training data) and Idefics2-70B (same architecture as Idefics2-8B but with a 70B-parameter LLM, representing the pretraining-scale alternative to Idefics3's data-and-architecture improvements). For the Docmatix validation ablation, the baseline is Florence-2 (0.7B parameters, Xiao et al., 2024) fine-tuned solely on DocVQA without Docmatix pre-training. The external validation point is moondream2, whose creators independently used Docmatix and reported a 103% improvement on DocVQA.

• Generation budget / compute accounting. The paper does not perform a FLOPs-matched or generation-budget-controlled comparison between models. Idefics3-8B and Idefics2-70B are compared at their respective default inference resolutions (Idefics3: images resized to 4×364 pixels on the longest side for most benchmarks, 5×364 for DocVQA; Idefics2-70B: 1960 pixels on the longest side). No attempt is made to equalize inference compute across models of different sizes. The training compute is reported as "5 days on 32 H100 nodes" for Idefics3-8B, but no corresponding figure is provided for Idefics2-8B or Idefics2-70B, making training-efficiency comparisons impossible from the reported data. For the Docmatix ablation with Florence-2, compute is implicitly compared by matching the number of training epochs (multiple epochs on DocVQA alone vs. one epoch on Docmatix subset + one epoch on DocVQA), but FLOP counts are not provided.

• Cross-validation / statistical protocol. There is no cross-validation or statistical significance testing reported for the main Idefics3 results. The five benchmarks are evaluated once each, and the resulting scores are reported without confidence intervals, standard deviations, or multiple-run averaging. The Docmatix ablation (Table 2) reports single-run scores for two Florence-2 training configurations without error bars. The MathVista contamination analysis (Section 4.3) is based on manual inspection and pattern matching rather than systematic automated decontamination, with percentages reported as lower bounds ("at least 6.6%").

Main Quantitative Results

Idefics3-8B vs. Idefics2-8B and Idefics2-70B

The headline results appear in Figure 5, which presents a bar chart comparing the three models across five benchmarks. Idefics3-8B achieves an 87.7 ANLS on DocVQA, representing a 13.7-point improvement over Idefics2-8B (74.0) and exceeding even Idefics2-70B (84.1) by 3.6 points. This is the paper's strongest single result and the primary evidence for its claim that increased visual token throughput + Docmatix training data can compensate for a nearly 9× smaller language model on document understanding tasks.

On TextVQA, Idefics3-8B scores 74.9, a slight decline from Idefics2-8B (77.3) and a larger gap from Idefics2-70B (73.0—though note the 70B model actually scores lower than the 8B model on this benchmark, which the paper does not comment on). The 2.4-point drop from Idefics2-8B to Idefics3-8B is unexplained and somewhat surprising given the architectural improvements targeting OCR.

On MMStar (general image understanding), Idefics3-8B scores 55.9, trailing both Idefics2-8B (58.1) and Idefics2-70B (49.5—again, the 70B underperforms the 8B model, a consistency the paper does not address). The 2.2-point decline from Idefics2-8B suggests that the changes made for Idefics3 (pixel shuffle, different LLM, different training data mixture) may have traded off some general visual understanding capability for document-specific performance.

On MathVista (visual mathematical reasoning), Idefics3-8B scores 58.4, below Idefics2-8B at 59.8 but above Idefics2-70B at 52.2. On MMMU (multi-discipline college-level problems), Idefics3-8B achieves 46.6, above Idefics2-8B at 45.2 but substantially below Idefics2-70B at 58.0—an 11.4-point gap that the paper explicitly attributes to scale: "the large gap of 11.4 points between Idefics2-70B and Idefics3-8B on MMMU indicates that scale is necessary for this benchmark to encapsulate sufficient knowledge into the model's weights" (Section 5.2.2).

Table 4 provides the detailed MMMU breakdown across 30 categories. Notable strengths include Literature (80.0), Art Theory (76.7), Design (73.3), and History (56.7). Notable weaknesses include Materials (26.7), Math (26.7), Music (26.7), and Physics (26.7). The Science category averages 36.0, while Humanities & Social Science averages 61.7, consistent with the LLM backbone's likely text-heavy pre-training distribution.

Docmatix Validation: Florence-2 Ablation

Table 2 reports the results of the controlled experiment testing whether Docmatix data provides useful training signal independent of the main Idefics3 pipeline. A Florence-2 model (0.7B parameters) trained solely on DocVQA achieves a DocVQA ANLS of 60.1. The same architecture trained on a Docmatix subset (20% of images, 4% of QA pairs) for one epoch, followed by one epoch on DocVQA, achieves 71.4—an 11.3-point absolute improvement, or approximately 18.8% relative improvement. This 0.7B specialist model's 71.4 score is only 2.6 points below Idefics2-8B's 74.0 (a generalist 8B model), demonstrating that targeted synthetic data can partially close the scale gap for narrow task distributions.

The paper also reports a third-party validation: moondream2, after incorporating Docmatix, achieved a "103% improvement on DocVQA compared to its previous version" (Section 5.1.2). The baseline and final scores are not provided, so this figure is difficult to interpret—a 103% improvement from 30 to 61 is very different from 103% improvement from 15 to 30.45.

Qualitative Results

Figure 6 presents three generation examples: extracting structured information from a CV (name, title, professional experience, education, books—all correctly transcribed), transforming a website screenshot into HTML/CSS code (the output is truncated in the figure but described as functional), and summarizing a research paper given a screenshot. The paper notes that the model "can sometimes struggle to follow instructions for more challenging prompts" due to the absence of an alignment phase, but that "adding a brief prefix to the assistant's response allows the user to easily shape the generated output as desired" (Section 5.2.2). The qualitative examples are selected for success cases rather than providing a balanced view of failure modes.

Ablation Studies and Robustness Checks

This paper is primarily a survey and model-building paper, not an empirical analysis paper with systematic ablations. Most design choices (pixel shuffle over perceiver resampler, Llama 3.1 over Mistral-7B, stage 3 synthetic data composition) are justified through reference to prior literature or architectural reasoning rather than through controlled ablation experiments within the Idefics3 training pipeline. The following represent the explicit and implicit ablations present in the paper:

• Visual token count (implicit, cross-model comparison): The switch from Idefics2's perceiver resampler (64 visual tokens per image, compressed from up to 980×980 pixels) to Idefics3's pixel shuffle (169 tokens per 364×364 tile, up to ~4,225 tokens for a 5×5 tile grid) is the primary architectural change. The effect is measured by comparing Idefics3-8B's DocVQA score (87.7) against Idefics2-8B's (74.0) in Figure 5—a 13.7-point improvement. However, this comparison conflates the connector change with the LLM change (Mistral-7B → Llama 3.1 8B), the training data change (Docmatix added), and the training curriculum change (3-stage pre-training vs. Idefics2's 2-stage), making it impossible to isolate the contribution of visual token count alone. The paper does not provide an Idefics3 trained with the perceiver resampler, nor an Idefics2 trained with pixel shuffle and Docmatix, which would be the controlled ablation.

• Language model backbone: The LLM upgrade from Mistral-7B (Idefics2-8B) to Llama 3.1 instruct (Idefics3-8B) is justified by the observation that Mistral-7B achieves 60.1% on MMLU while Llama 3.1 instruct scores higher (Section 2.1.4 argues that LLM benchmark performance correlates with VLM performance). However, the MMMU results in Figure 5 show Idefics3-8B (46.6) only modestly above Idefics2-8B (45.2), a 1.4-point gain that could be attributable to the LLM, the data, or noise. The DocVQA gain (13.7 points) almost certainly reflects the connector and data changes more than the LLM change. Without an Idefics3-8B variant using Mistral-7B, the LLM contribution cannot be isolated.

• Docmatix data contribution (Table 2, Florence-2): The controlled ablation with Florence-2 isolates Docmatix's effect by keeping the model architecture and evaluation fixed: Florence-2 trained on DocVQA alone (60.1 ANLS) vs. pre-trained on Docmatix subset + fine-tuned on DocVQA (71.4 ANLS). The 11.3-point gain demonstrates that Docmatix's synthetic QA pairs contain genuine training signal. However, this ablation uses a different model architecture (Florence-2, a specialist vision model) than Idefics3 (generalist VLM), so the magnitude of benefit may not transfer. Additionally, the Docmatix-trained Florence-2 uses both Docmatix pre-training AND DocVQA fine-tuning, meaning the reported 71.4 reflects the combination of both data sources, not Docmatix alone.

• Image resolution at inference (implicit, within-benchmark): For DocVQA, Idefics3 uses 5×364 = 1820 pixels on the longest side, while for other benchmarks it uses 4×364 = 1456 pixels. The paper does not ablate this choice—no results are reported for DocVQA at 4×364 or for other benchmarks at 5×364—making it unclear how much of the DocVQA improvement comes from resolution vs. other factors.

• Training data mixture balancing (Table 1): The SFT data mixture percentages are selected (upsampled/downsampled by answer token count) but the paper does not ablate alternative balancing strategies (e.g., by number of QA pairs, by number of images, uniform sampling). The specific percentages in Table 1 are presented as a finalized recipe without sensitivity analysis—we don't know, for example, whether doubling the Docmatix percentage from 10.31% to 20% would improve or degrade performance.

• DoRA vs. full fine-tuning (stated but not tested): The authors explicitly state "we believe that carefully executed full unfreezing can lead to better performance" (Section 5.2.1) but use DoRA for efficiency. No comparison between DoRA and full fine-tuning is provided, making this an untested hypothesis within the paper's experimental framework.

• Number of pre-training stages and step counts: The paper uses 3 pre-training stages (1,000 + 3,000 + 1,500 steps) but does not ablate this against 2 stages or 4 stages, or against different step allocations. The authors note that "during the first two pre-training stages, the loss function is far from converging, but we move to the next stage to reduce computational costs" (Section 5.2.1), and that in stage 3 "only a fraction of the examples available in the chosen datasets are used, again to reduce computational demands." This means the reported results are from a compute-constrained budget, not a performance-saturating one—an important caveat for interpreting the benchmark comparisons.

• NEFTune noise (present in SFT, not ablated): NEFTune noise is applied during SFT but no comparison with a no-noise baseline is provided. The contribution of this regularization technique to final performance is unknown.

• Alignment phase (absent, not ablated): The model does not undergo an alignment phase (DPO, RLHF). The paper acknowledges this limitation but does not quantify what performance improvement an alignment phase would provide, nor does it compare against models that do include alignment (which would be most comparable production models).

• Benchmark contamination (Section 4.3, diagnostic not ablation): The paper identifies contamination in MathVista (6.6% of questions use images from training sets, 2.2% are near-duplicates) but does not re-evaluate Idefics3 on a decontaminated version of the benchmark. The reported MathVista score (58.4) therefore may include some contamination-based inflation.

Critical Assessment

Claim from the executive summary: "Idefics3-8B achieves an 87.7 ANLS score on DocVQA, a 13.7-point improvement over its predecessor Idefics2-8B." This claim is directly supported by Figure 5. However, the attribution of this improvement is radically confounded. Three major changes occurred simultaneously: (1) the LLM changed from Mistral-7B to Llama 3.1 instruct, (2) the connector changed from perceiver resampler (64 tokens) to pixel shuffle (169 tokens per tile), and (3) Docmatix was added to the training data. The 13.7-point improvement cannot be decomposed into contributions from each change. The Florence-2 ablation (Table 2) shows Docmatix provides meaningful signal (+11.3 ANLS on a different model), but the connector and LLM contributions remain unknown. A reader wanting to replicate these results would not know whether to prioritize the data pipeline, the connector design, or the LLM upgrade—all were changed simultaneously and only the combination was evaluated.

More subtly, the comparison between Idefics3-8B and Idefics2-8B on DocVQA is not resolution-matched: Idefics3 evaluates at 5×364 = 1820 pixels, while Idefics2-8B's evaluation resolution is not explicitly stated for DocVQA (the paper says Idefics2-70B uses 1960 pixels, but Idefics2-8B's resolution is ambiguous). If Idefics2-8B was evaluated at a lower resolution, some of the improvement may reflect resolution rather than architecture or data.

Claim: "The number of visual tokens per image, rather than architectural complexity, is the primary bottleneck for OCR tasks." This is supported indirectly through cross-model comparison (Idefics2's 64 tokens → poor OCR; Idefics3's 169 tokens/tile → much better OCR) and through citations to InternLM-XComposer2-4KHD and Idefics2's own findings. However, the paper never runs the experiment that would directly test this claim: training Idefics3 with different visual token budgets (e.g., 64, 169, 338 per tile) and measuring DocVQA as a function of token count. The claim is also complicated by the fact that the perceiver resampler doesn't just reduce token count—it uses learned cross-attention, which could learn to drop OCR-relevant information even at higher token budgets, whereas pixel shuffle is deterministic and spatially local. So the comparison is not "fewer tokens vs. more tokens" but "learned compression vs. deterministic spatial compression at different token counts." The stated claim may be true, but the paper's experiments demonstrate only that the specific combination of pixel shuffle + more tokens improves OCR, not that token count alone is sufficient.

Claim: "Docmatix is 240 times larger than previously available open document-understanding datasets." The 240× figure compares Docmatix's 9.5M QA pairs against DocVQA's ~40K QA pairs. This is arithmetically correct but potentially misleading in two ways. First, the QA pair generation pipeline produces synthetic questions that may be lower quality, less diverse, or differently distributed than human-annotated DocVQA questions—so comparing raw counts assumes 1 Docmatix QA pair ≈ 1 DocVQA QA pair in training value, which is unlikely. Second, the figure uses DocVQA as the reference point, but if we sum across all open document-understanding datasets (DocVQA + InfoVQA + VisualMRC = ~62K QA pairs), the factor is ~153×, still very large but not exactly 240×. These are minor quantitative quibbles; the substantial contribution—creating a large-scale open document QA dataset—is real regardless.

Claim: "A specialist 0.7B model approaches the performance of an 8B generalist model on document understanding." Table 2 shows Florence-2 + Docmatix achieving 71.4 ANLS vs. Idefics2-8B at 74.0. This is a fair comparison, supporting the claim. However, the Florence-2 model received DocVQA fine-tuning after Docmatix pre-training, while Idefics2-8B's exact training recipe for its 74.0 score is not fully detailed in this paper (it was published in Laurençon et al., 2024). If Idefics2-8B did not receive DocVQA-specific fine-tuning, the comparison slightly favors Florence-2. Additionally, 71.4 vs. 74.0 is a 3.5% relative gap, which may or may not be considered "approaching"—readers can judge.

What would strengthen the paper:

1. Controlled architecture ablations within the Idefics3 training pipeline: Train Idefics3 variants where only one component changes at a time—perceiver resampler vs. pixel shuffle (with the same LLM and data), Mistral-7B vs. Llama 3.1 (with the same connector and data), with and without Docmatix (with the same architecture). Without these, the paper cannot identify which of its design choices most matter for the observed improvements.

2. Scaling curves for visual tokens on OCR tasks: Vary the pixel shuffle downscaling factor (2×, 4×, 8× compression) or the number of perceiver resampler queries (64, 128, 256, 512) and plot DocVQA/TextVQA performance vs. token count. This would directly test the bottleneck hypothesis.

3. Ablation of pre-training stages and step counts: Is stage 3 necessary for the DocVQA gains, or would adding Docmatix to stage 2 suffice? Are the 1,500 steps in stage 3 saturating, or would 3,000 steps continue improving? The paper acknowledges not knowing the answer.

4. Statistical rigor: All reported scores are single-point estimates without error bars, confidence intervals, or multi-seed training runs. Given the 500-question MATH test set from the similar prior-work example, these benchmarks likely have test sets of comparable size (102–105 questions), where score differences of 1–3 points could easily arise from sampling variance. Without uncertainty quantification, readers cannot assess whether the MMMU gap of 1.4 points (46.6 vs. 45.2) or the MMStar decline of 2.2 points (55.9 vs. 58.1) are statistically reliable.

5. Text-only benchmark evaluation: The paper argues in Section 2.2.1 that "handling image representation within the language model might decrease its performance on text-only benchmarks" and notes that "most VLMs are still not evaluated on text-only benchmarks." Yet Idefics3 itself is not evaluated on any text-only benchmark (MMLU, HellaSwag, etc.), making it impossible to assess whether the multimodal training causes catastrophic forgetting of text capabilities.

6. Decontaminated MathVista evaluation: Given the paper's own analysis showing contamination in MathVista, a re-evaluation on a decontaminated subset would provide a more trustworthy estimate of Idefics3's genuine visual math reasoning capability.

Where the claims hold conditionally:

• The DocVQA improvement (13.7 points) is robust in direction but uninterpretable in decomposition.
• The value of Docmatix as a training resource is conditionally supported: it helps Florence-2 substantially (Table 2), and moondream2 independently, but the marginal benefit when added to a training pipeline that already includes PDFA (which Idefics3 uses) is not isolated.
• The visual token bottleneck claim is theoretically well-motivated and consistent with prior literature, but the paper's own experiments offer only confounded support.
• Idefics3-8B's superiority over Idefics2-8B is clear on DocVQA (+13.7) and MMMU (+1.4), unclear on MathVista (−1.4) and MMStar (−2.2), and negative on TextVQA (−2.4). The claim "Idefics3 significantly outperforms its predecessor" (Abstract) is therefore task-dependent: strongly true for document understanding, mildly true for multi-discipline reasoning, and false or within noise for general visual understanding and text reading.

6. Limitations and Trade-offs

The Difficulty Estimation for Test-Time Allocation Is Prohibitively Expensive

The paper's compute-optimal framework for test-time compute allocation is built on the ability to estimate each prompt's difficulty before deciding how to spend the inference budget. Generating 2048 samples per question and scoring them with the PRM (or checking ground-truth correctness) to place questions into five difficulty quintiles is extraordinarily expensive — more expensive than the largest test-time budgets studied (256–512 generations). The authors explicitly acknowledge this in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference… our experiments do not account for this cost largely for simplicity"

The consequence is that the reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter. For a prompt that ultimately receives 16 generations of test-time compute, the overhead of generating 2048 samples for difficulty estimation represents a 128× increase in total inference cost, completely negating the efficiency gains. This means the 4× figure should be understood as an upper bound on achievable efficiency rather than a realized deployment gain. The practical value of compute-optimal allocation depends entirely on solving the difficulty estimation problem with a fraction of the current cost.

The paper provides no evidence that difficulty can be estimated cheaply enough to make the framework practical. The predicted difficulty bins (using the PRM's average final-answer score instead of ground-truth correctness) still require generating and scoring 2048 samples — they remove the need for ground-truth labels but not the computational cost. The authors flag this as "a key avenue for future work" and briefly mention the possibility of "pretraining or finetuning models to directly predict difficulty of a question" (Section 3.2), but no such model is developed or evaluated. Until this gap is closed, the compute-optimal framework remains an analytical contribution with significant deployment barriers.

Hard Problems Show Near-Zero Improvement Regardless of Compute Budget

Across all methods studied — search against PRM verifiers, iterative revision chains, and their compute-optimal combinations — the hardest questions (difficulty bin 5) show near-zero improvement regardless of how much test-time compute is allocated. This is not a failure of the allocation strategy; it is a fundamental capability bound.

The evidence is consistent and stark. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all search methods and all generation budgets from 4 to 256. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy for the revision model irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5% across all values of the inference-to-pretraining ratio $R$, meaning that even when the smaller model is allocated an inference budget that would have paid for a 14× larger model, the hardest problems remain unsolved.

The paper is transparent about this. Section 7's takeaway explicitly states that test-time compute "cannot compensate for fundamental capability gaps that larger pretraining would address" and that on bin 5, "no method makes meaningful progress." The FLOPs-matched results in Figure 1 (bottom-right bar chart) show that for hard questions, test-time compute produces a relative disadvantage of −52.9% under PRM search at $R \gg 1$.

The consequence is a hard boundary on the applicability of test-time compute scaling: if the base model's pass@1 rate on a problem class is near zero — meaning it almost never produces correct solutions even with 2048 independent attempts — no amount of search, revision, or clever allocation will help. There are simply no correct solutions in the proposal distribution to find or refine. This means the approach offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution. For such problems, pretraining remains the only viable path. The compute-optimal framework is therefore best understood as a method for amplifying existing capability, not creating it — a distinction with important practical consequences for teams deciding whether to invest in larger pretraining runs versus smarter inference strategies.

The 14× Larger Model Baseline Is Not Compute-Optimally Trained and Uses No Test-Time Compute of Its Own

The FLOPs-matched comparison in Section 7 — which is the headline evidence that test-time compute can substitute for pretraining — compares PaLM 2-S* augmented with compute-optimal test-time strategies against a model with approximately 14× more parameters trained on the same data. This baseline is weaker than it could be in two important ways.

First, the larger model scales parameters only, not training data, following the LLaMA paradigm (Touvron et al., 2023) rather than Chinchilla-optimal pretraining (Hoffmann et al., 2022) where both data and parameters are scaled equally. The authors acknowledge this explicitly: "We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work" (Section 7). A Chinchilla-optimal model trained with 14× more total FLOPs would distribute that budget across both model size and data quantity, likely outperforming a parameter-only scaling approach. This makes the pretraining baseline weaker than it should be for a fair comparison.

Second, the 14× larger model is evaluated using only greedy decoding with no test-time compute augmentation of its own — no majority voting, no best-of-N, no beam search, no revisions. This is an asymmetric comparison: the smaller model gets an optimized inference strategy while the larger model gets none. A fairer comparison would give the larger model even a modest test-time compute budget (e.g., best-of-8 or a short revision chain), which would substantially strengthen the pretraining baseline.

The consequence is that the reported advantages of test-time compute over pretraining — for example, +27.8% relative improvement on easy questions at $R \ll 1$ for revisions, and +19.1% for PRM search (Figure 1, bar charts) — may be partially attributable to the asymmetric evaluation protocol rather than a genuine efficiency advantage. Against a compute-optimally trained larger model with even basic test-time compute, these margins would likely shrink or potentially reverse. The paper's conclusion that "a smaller model augmented with compute-optimal test-time strategies can outperform a 14× larger pretrained model" (Section 1) should be understood as demonstrated only against this specific, relatively weak pretraining baseline — not as a general proof that inference compute is more efficient than pretraining compute.

Sequential Revisions Introduce a Latency Penalty Not Captured by Generation-Based Cost Accounting

The paper measures test-time compute in "generations" — the number of complete solutions sampled — which is a reasonable proxy for total FLOPs but ignores latency entirely. This matters because parallel and sequential strategies have identical generation costs but radically different wall-clock times. A strategy that allocates 128 generations as 64 sequential revisions (generated one after another, each conditioning on the previous) takes approximately 64× longer wall-clock time than a strategy that runs 128 parallel samples simultaneously on sufficient hardware.

The compute-optimal policy consistently favors sequential-heavy strategies for easy-to-medium difficulty problems. For revisions, Figure 7 (left) shows that at lower budgets (8–32 generations), the optimal ratio is fully sequential, and even at higher budgets (128–256 generations), the optimal ratio is around 2:1 to 8:1 sequential-to-parallel. This means that for the problem categories where compute-optimal scaling shows the largest gains over best-of-N, the recommended strategies are also the most latency-sensitive. A deployment serving real-time user requests — an interactive assistant, a document QA system with response-time SLAs — could not afford to wait 64 sequential generation steps, regardless of the total FLOPs efficiency.

The paper does not discuss latency, throughput, or wall-clock time at all. The 4× efficiency gains are measured in generation count, which maps to total FLOPs but not to user-perceived latency. This omission is significant because sequential and parallel strategies trade off latency against FLOPs in qualitatively different ways, and the "optimal" allocation from a FLOPs perspective may be strictly infeasible from a latency perspective. For deployment in interactive settings, the relevant optimization problem is latency-constrained test-time allocation, which the paper's framework does not address.

Mitigation is partial at best. The paper's observation that different difficulty levels prefer different sequential-to-parallel ratios (Figure 7, right) suggests that latency-conscious deployments could cap the sequential depth and accept the associated accuracy reduction, but the paper provides no guidance on how to make this trade-off. The authors do not acknowledge the latency issue as a limitation, and no proposed future work addresses it.

All Results Are from a Single Benchmark and a Single Model Family

Every experiment in this paper uses the MATH benchmark (Hendrycks et al., 2021, 500 test questions of high-school competition-level math problems) and PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified and several aspects of the findings could be specific to this model-dataset combination.

The concern is not merely that results might not transfer — it is that specific failure modes and scaling behaviors identified in this paper may be artifacts of PaLM 2-S*'s particular characteristics. The PRM over-optimization behavior (beam search degrading easy-problem performance at high budgets, Figure 3 right) depends on the verifier's calibration properties, which are a function of the base model's output distribution. A model with different calibration characteristics or different error patterns might exhibit different difficulty-dependent scaling curves — potentially making beam search safe at higher budgets on easy problems, or conversely, making best-of-N competitive on medium problems where beam search currently dominates.

The revision model's effectiveness is even more model-specific. The paper finds that fine-tuning is necessary because off-the-shelf LLMs prompted to self-correct "tends to be largely ineffective for obtaining performance improvements on reasoning problems" (Section 6), citing Huang et al. (2023). But if a different base model had stronger in-context self-correction capabilities, or if a different training recipe produced a more robust revision model, the revision scaling curves and the sequential-to-parallel optimal ratios might change substantially. The ReST$^{EM}$ experiment (Appendix K, Figure 16) shows that changing the revision model training methodology causes performance to degrade with sequential revisions — a stark illustration of how sensitive these results are to model-specific training choices.

The MATH benchmark itself may not be representative of broader reasoning tasks. It consists exclusively of competition-level math problems requiring symbolic reasoning and produces answers that can be verified with exact match. Other reasoning domains — code generation, logical deduction, scientific QA, multi-step planning — may exhibit different difficulty-dependent scaling patterns. More critically, tasks without clean correctness signals (open-ended generation, dialogue, creative writing) cannot use the PRM training pipeline or the pass@1-based difficulty estimation at all, meaning the entire compute-optimal framework as described is inapplicable.

The paper makes no attempt to replicate findings on other benchmarks (e.g., GSM8K, MMLU reasoning subsets) or other model families (e.g., LLaMA, Mistral, Qwen). All claims about the relationship between test-time compute and pretraining compute, the effectiveness of PRM-guided search, and the optimal sequential-to-parallel ratios should be understood as demonstrated for PaLM 2-S* on MATH specifically until replication evidence exists. The paper's conclusion generalizes these findings implicitly, but the experimental design provides no basis for that generalization.

The Compute-Optimal Policy Is Static and Coarse-Grained, Preventing Dynamic Adaptation

The paper's compute-optimal strategy operates by pre-computing the best hyperparameters (search algorithm, beam width, sequential-to-parallel ratio) for each of five difficulty quintiles at each budget level, then looking up the strategy at test time based on a one-time difficulty estimate. This approach is static (the strategy is fixed before generation begins) and coarse-grained (all questions within a quintile receiving identical treatment).

The consequence is that the policy cannot adapt mid-generation based on information that becomes available during the solution process. For example, if the model generates a few initial candidates that all score very low under the PRM, this suggests the problem may be harder than the initial difficulty estimate indicated — but the static policy cannot switch from best-of-N to beam search, or increase the revision chain length, in response. Conversely, if early candidates score very high, the difficulty estimate may have been pessimistic, and the budget could be reduced. This type of dynamic, closed-loop control is standard in many resource allocation domains (multi-armed bandits, anytime algorithms, Bayesian optimization) but is absent from the paper's framework.

The five-quintile discretization amplifies this problem. Within a single bin (especially bins 3 and 4, which contain the medium-difficulty questions where strategy choice matters most), there may be substantial heterogeneity. A question at the easy end of bin 4 and one at the hard end of bin 4 receive the identical strategy, even though the optimal allocation for these two questions may differ. The paper does not explore sensitivity to the number of bins or the placement of bin boundaries, so the extent of within-bin suboptimality is unknown.

The difficulty estimate itself is computed once from 2048 samples and treated as an oracle. In a realistic setting where difficulty must be estimated from far fewer samples (or from a learned predictor), the difficulty estimate will be noisy, and the static lookup policy has no mechanism for handling uncertainty. A question erroneously assigned to bin 3 when it truly belongs to bin 5 will receive aggressive beam search that yields no benefit while incurring the associated cost.

The paper does not propose or evaluate any dynamic allocation strategy. Section 3.2 briefly mentions the exploration-exploitation tradeoff inherent in difficulty estimation but frames it as a cost issue rather than an opportunity for adaptive control. The paper acknowledges that "solving this optimization exactly for every prompt is intractable" — but approximations beyond the static discretization approach are not discussed. This is a significant missed opportunity, since the rich per-step feedback available during PRM-guided search and revision chains provides exactly the kind of intermediate signal that could inform dynamic reallocation decisions.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around vision-language model development from a collection of loosely justified individual design choices toward a structured engineering discipline with identified bottlenecks and systematic diagnostics. Before this work, practitioners faced an overwhelming design space—cross-attention vs. self-attention, perceiver resampler vs. linear projection vs. pixel shuffle, 2-stage vs. 3-stage pre-training—with each paper advocating for its own configuration without clear guidance on which choices mattered, when, or why. The field was accumulating models faster than understanding. This paper provides the understanding.

The most significant conceptual contribution is the articulation of specific, named diagnostic concepts that explain otherwise puzzling performance patterns across the literature. The visual token bottleneck (Section 5.2.1) clarifies why aggressive connector compression harms OCR tasks while leaving most other tasks unaffected—it's not that the perceiver resampler is "worse" than pixel shuffle in general, but that it imposes an information throughput ceiling that binds specifically for fine-grained text recognition. This transforms the connector design question from "which architecture is best?" to "what visual information capacity does my target task distribution require?" The delayed feedback loop (Section 4.2) explains why pre-training ablations have been systematically misleading—pre-training metrics measure foundational skill acquisition while the complex capabilities users care about only emerge during supervised fine-tuning, meaning architectural choices that affect information capacity appear irrelevant until the final training stage. The data reframing strategy (Docmatix, Section 5.1.2) demonstrates that the document understanding data bottleneck—which had kept open models far behind proprietary systems—was solvable not by training better VLMs to generate data, but by recognizing that the problem decomposes into OCR (a solved perception problem) and text-based QA generation (a solved language problem), neither requiring a VLM at all.

These diagnostics collectively reframe what it means to "do VLM research well." Rather than proposing novel architectural components and demonstrating they work on a few benchmarks, the paper's framework suggests researchers should: (1) identify which task categories their target applications require, (2) diagnose which bottleneck (visual token throughput, training data coverage, LLM reasoning capacity) limits performance on those tasks, and (3) invest resources accordingly. This is a maturation of the field from an exploration phase (where any working configuration is publishable) to an optimization phase (where understanding trade-offs matters more than individual numbers).

The paper also reconciles several apparent contradictions in prior work. The disagreement about whether cross-attention outperforms self-attention (Section 2.1.3) is resolved by the observation that the answer depends on whether backbones are frozen (cross-attention wins) or trained (self-attention wins)—a finding from Laurençon et al. (2024) that this paper contextualizes within a broader framework. The tension between Fuyu's claim that vision encoders are unnecessary and PaliGemma's finding that omitting them causes a "notable drop in performance" (Section 2.2.1) is reframed as a question of training efficiency rather than fundamental capability: "bypassing a vision encoder pre-trained on billions of images could lead to longer training times to achieve similar performance." The conflicting evidence about whether visual token count matters—Idefics2 found no pre-training benefit from 128 vs. 64 tokens, but InternLM-XComposer2-4KHD found it critical for OCR—is explained by the delayed feedback loop: the benefit only becomes visible after fine-tuning on task-specific data. In each case, the paper doesn't just report the conflict; it provides a mechanistic explanation that makes both findings true under different conditions.

The research directions that become more attractive after this work include: developing better open-source vision encoders (the paper explicitly notes the scarcity compared to LLMs), creating synthetic data pipelines for other bottlenecked tasks following the Docmatix template, investigating dynamic resolution allocation during inference (since the image-splitting strategy naturally supports variable token budgets), and building vision encoders that natively handle arbitrary resolutions without the tiling workaround. The directions that become less attractive—or at least require stronger justification—include: proposing new connector architectures without demonstrating they address a specific bottleneck on relevant tasks, evaluating VLMs only at pre-training without SFT, and claiming benchmark improvements without decontamination analysis.

Follow-Up Research This Work Enables

Direct measurement of the visual token bottleneck through controlled scaling experiments. The paper argues that visual token count, not connector architecture, is the binding constraint for OCR performance, but the evidence is confounded: Idefics3 changed the connector type (perceiver resampler → pixel shuffle), the LLM (Mistral-7B → Llama 3.1 8B), the training data (Docmatix added), and the token count (64 → 169 per tile) simultaneously. A clean experiment would train multiple Idefics3 variants with identical architecture, LLM, and training data, varying only the pixel shuffle downscaling factor (e.g., no compression = 676 tokens/tile, 2× = 169 tokens/tile, 4× = 42 tokens/tile, 8× = 10 tokens/tile) or, for perceiver resampler variants, the number of learned queries (32, 64, 128, 256, 512). Plotting DocVQA ANLS and TextVQA accuracy against token count would reveal: (a) whether OCR performance saturates at some token threshold, (b) whether the bottleneck is monotonic (more tokens always helps, with diminishing returns) or U-shaped (too many tokens hurt by exceeding the LLM's effective context), and (c) at what token budget the bottleneck shifts from visual throughput to LLM reasoning capacity. The experiment would cost ~4-8 training runs on the Idefics3 pipeline (each ~5 days on 32 H100 nodes) and would produce a scaling law for visual tokens on OCR that the field currently lacks.

Applying the Docmatix data reframing strategy to other data-starved VLM tasks. The paper demonstrates that decoupling visual extraction from language reasoning can produce large-scale training data for document understanding. The natural extension is to identify other task categories where the same decomposition applies. Table understanding: use table extraction models to convert table images into structured formats (CSV, Markdown, HTML), then prompt text-only LLMs to generate QA pairs about the extracted tables—questions about cell values, column comparisons, trend analysis, arithmetic across rows. Chart understanding: use chart data extraction tools to recover the underlying data series from chart images, then generate QA pairs from the numeric data. Visual mathematical reasoning: use Nougat or similar math-aware OCR to extract LaTeX from images of equations and diagrams, then use an LLM fine-tuned on math to generate step-by-step solutions. Each pipeline would need validation analogous to Table 2—train a small specialist model with and without the synthetic data, measure on the target benchmark—to confirm the generated QA pairs provide genuine training signal. The key research question is whether the text transcription quality bottleneck identified for Docmatix ("math equations are often inaccurately transcribed or omitted," Section 3.1) is manageable for each domain, or whether some tasks require VLM-based generation because the relevant information cannot be adequately captured in text.

Systematic study of vision encoder quality and scale on VLM downstream performance. The paper notes that replacing CLIP-ViT-H (78.0% ImageNet) with SigLIP-SO400M (83.2% ImageNet) "leads to a substantial performance improvement across all benchmarks" (Section 2.1.4), and laments that "few open-vision encoders have been released, with SigLIP-SO400M standing out." This suggests a scaling law waiting to be measured: hold the LLM (Llama 3.1 8B) and connector (pixel shuffle) fixed, and swap in a range of open vision encoders spanning different scales and qualities—SigLIP-B/16, SigLIP-L/14, SigLIP-SO400M, EVA-CLIP variants, DINOv2, OpenCLIP models. For each, train the full Idefics3 pipeline (or a minimal version) and measure final SFT performance across the five-benchmark suite. The output would be a scatter plot of vision encoder ImageNet accuracy (or a multimodal proxy) against VLM downstream scores, revealing: (a) whether the correlation is log-linear (each point of ImageNet accuracy yields a predictable VLM gain), (b) whether the relationship saturates (beyond some encoder quality, VLM performance is dominated by LLM capacity or data), and (c) which benchmarks are most sensitive to encoder quality (likely DocVQA, TextVQA, and MMStar; less likely MMMU which is knowledge-heavy). This would provide concrete guidance for the resource allocation decision the paper frames: when should a team invest in a better vision encoder vs. more training data vs. a larger LLM?

Decontamination-aware benchmark evaluation and the construction of "clean" benchmark subsets. The paper's analysis of MathVista contamination (Section 4.3) is diagnostic but not corrective. A valuable follow-up would systematically decontaminate the major VLM benchmarks (MMMU, MathVista, MMStar, DocVQA, TextVQA) against the training sets of commonly used SFT datasets (The Cauldron, ShareGPT-4o, LNQA, Geo170K, etc.) using both exact image hash matching and embedding-based near-duplicate detection. For each benchmark, produce a "clean" subset excluding any question whose image appears in any training dataset, and a "strictly clean" subset also excluding questions with near-duplicate images or highly similar question text. Re-evaluate Idefics3 and a few representative baselines (Idefics2-8B, a proprietary model if API access allows) on both the original and clean subsets. The research contribution is not just the decontaminated scores, but quantifying the contamination inflation factor for each benchmark-category pair—e.g., "MathVista scores are inflated by ~X points on average, with geometry questions showing ~Y points more inflation than algebra questions." The paper's finding that 2.2% of MathVista questions are near-duplicates and 6.1% ask variants of KVQA questions provides a lower bound; the systematic analysis would reveal the upper bound and enable the field to calibrate how much of reported benchmark progress is genuine capability improvement vs. memorization.

Dynamic resolution allocation during inference. The image-splitting strategy naturally produces a variable number of visual tokens based on input resolution, but Idefics3 uses a fixed maximum resolution per task (4×364 for most benchmarks, 5×364 for DocVQA). This leaves efficiency on the table: many images don't need the maximum resolution, and the model should be able to "decide" how much visual detail to allocate based on the question. A concrete experiment: after SFT, add a small number of training examples where the model is trained to output a special token indicating how many resolution levels it wants (e.g., <resolution_1>, <resolution_2>, up to <resolution_5>), then at inference, have the model first output this token, encode the image at the requested resolution, and generate the answer. Train this with reinforcement learning or by distilling from a model that always uses maximum resolution. Measure whether the adaptive policy can match full-resolution accuracy on benchmarks while reducing average visual tokens per query by 30–50%. The paper's observation that "for simpler tasks, fewer visual tokens are needed, saving computational resources" (Section 2.2.3) provides the motivation; implementing it requires solving the meta-decision problem of when to allocate more resolution, which is analogous to the test-time compute allocation problem in the LLM scaling literature.

Measuring and mitigating text-only capability degradation from multimodal training. The paper raises but does not answer the question: does multimodal training cause catastrophic forgetting of the LLM's original text-only capabilities? Section 2.2.1 notes that "handling image representation within the language model might decrease its performance on text-only benchmarks" and that "most VLMs are still not evaluated on text-only benchmarks, making it unclear whether omitting a vision encoder affects text benchmark performance." A direct experiment: evaluate Llama 3.1 instruct on a standard text-only benchmark suite (MMLU, HellaSwag, ARC, GSM8K, etc.) before and after the full Idefics3 training pipeline (stages 1–3 + SFT), measuring the absolute and relative degradation on each benchmark. Additionally, ablate the text-only instruction data mixture in SFT (Table 1 shows ~15% of answer tokens come from text-only datasets) by training a variant without it and measuring whether degradation worsens. If substantial forgetting occurs (>5% relative on key benchmarks), test mitigations: increasing the text-only data proportion, using elastic weight consolidation, or periodically interleaving text-only training throughout multimodal stages. The paper already includes text-only data in the SFT mixture (OpenHermes-2.5, MetaMathQA, etc.), suggesting the authors anticipated this problem, but without pre-post measurements, we don't know whether 15% is sufficient or whether the degradation is zero. This is practically important because most VLM deployments will handle a mix of text-only and multimodal queries; a VLM that sacrifices text performance for visual capabilities may be net-worse for many use cases.

Comparison of DoRA (or LoRA) against full fine-tuning at equal compute budget within the Idefics3 pipeline. The authors state "we believe that carefully executed full unfreezing can lead to better performance" (Section 5.2.1) but use DoRA for training efficiency. This is a testable hypothesis: run the stage 2–3 pre-training and SFT with full parameter unfreezing instead of DoRA, controlling for total computational cost (i.e., allow fewer steps or smaller batch size for full fine-tuning to match the DoRA training FLOPs). Measure whether full fine-tuning at equal compute outperforms DoRA on the five-benchmark suite. The result would provide concrete guidance for the resource-allocation decision the paper frames: if full fine-tuning wins, the 5-day training time was a false economy; if DoRA wins or matches at equal compute, it's the pareto-optimal choice. The experiment requires one additional training run and could be done at reduced scale (shorter stages, subset of data) if full training is prohibitively expensive.

Practical Applications and Downstream Use Cases

Document understanding at scale with open models. Before this paper, deploying a competitive document QA system required either paying for proprietary API access (GPT-4V, Gemini) or accepting substantially lower accuracy from open models. Idefics3-8B's 87.7 ANLS on DocVQA changes this calculation: an organization with a large corpus of PDF documents (legal contracts, scientific papers, financial reports, medical records) can now deploy an on-premises model that extracts structured information, answers natural language questions, and summarizes content at accuracy approaching proprietary systems, without sending sensitive documents to external APIs. The Docmatix dataset and training pipeline mean that organizations with domain-specific documents can generate their own training data following the OCR + text-LLM recipe, fine-tune Idefics3 (or a smaller model) on their domain, and achieve specialist performance. The Florence-2 ablation (Table 2) demonstrates that even a 0.7B-parameter model can reach 71.4 ANLS with Docmatix pre-training—a configuration that could run on a single consumer GPU for low-latency, low-cost deployment. For organizations handling millions of documents, the cost savings from switching from per-page API pricing to on-premises inference with an open model are likely substantial.

Data-efficient multimodal fine-tuning for narrow domains. The paper's demonstration that synthetic data (Docmatix) can partially compensate for model scale—a 0.7B specialist approaching an 8B generalist—has direct implications for domain-specific VLM deployment. A team building a VLM for a narrow domain (e.g., understanding engineering diagrams, interpreting medical imaging reports, analyzing satellite imagery with annotations) can follow the Docmatix template: (1) use domain-specific extraction tools to convert images to structured text representations, (2) generate large-scale QA pairs from those representations using text-only LLMs, (3) fine-tune a small base VLM (like Florence-2 or Idefics3-8B) on the synthetic data plus a small amount of manually annotated in-domain data. The paper's finding that "training on this small portion of Docmatix leads to a nearly 20% relative improvement" (Section 5.1.2) on DocVQA for Florence-2 suggests that the synthetic pre-training step provides a substantial boost even when the final fine-tuning uses limited human-annotated data. This pattern—synthetic data for broad coverage, human data for task-specific accuracy—is a practical recipe for teams with limited annotation budgets.

Open-source model training with transparent data provenance. The entire Idefics3 training pipeline uses only open datasets: OBELICS, LAION COCO, PDFA, Docmatix, WebSight, LNQA, PixelProse, ChartGemma, and the expanded Cauldron. No proprietary data sources are involved. This has concrete value for organizations with legal or compliance requirements around training data provenance—government agencies, healthcare organizations, schools, and companies in regulated industries can deploy Idefics3 (or models built using the same data pipeline) without the legal uncertainty of models trained on undisclosed web-scraped data. The Docmatix creation pipeline is also transparent and replicable: anyone can download PDFA, run OCR, and generate QA pairs with Phi-3-small. This transparency is a practical differentiator from proprietary VLMs whose training data composition is unknown, making it impossible to audit for copyrighted material, personally identifiable information, or biased content. The paper's explicit documentation of data sources, filtering criteria, and the 15% QA pair rejection rate for Docmatix provides a template for what responsible open-source model development looks like.

When to Prefer This Method

The paper positions its approach—building VLMs from frozen/partially-frozen pre-trained unimodal backbones connected by a simple projection, trained in multiple stages with increasing resolution and synthetic data—against several alternatives implicitly. For practitioners deciding on a VLM development strategy, the decision criteria can be extracted from the paper's analysis:

• Prefer the Idefics3-style approach (frozen/DoRA-trained backbones, pixel shuffle connector, multi-stage pre-training, synthetic data for bottleneck tasks) when: (a) you have access to strong pre-trained unimodal backbones and want to minimize training compute (5 days on 32 H100 nodes for Idefics3), (b) your target tasks include document understanding or OCR where visual token throughput is a known bottleneck, (c) you need to maintain the LLM's original text-only capabilities (the DoRA approach and text-only SFT data aim to preserve these, though the paper doesn't measure whether they succeed), (d) you operate in a domain where synthetic data can be generated following the Docmatix template, and (e) you prioritize open data and transparent training pipelines over squeezing out the last few points of benchmark performance.

• Prefer a cross-attention architecture with frozen backbones (Flamingo-style) when: (a) preserving the LLM's text-only performance with high confidence is critical (cross-attention keeps the LLM completely frozen), (b) you have sufficient compute to train the additional cross-attention parameters (~1/4th of LLM size), and (c) you don't need to fine-tune the backbones on domain-specific data. The paper cites Laurençon et al. (2024) showing cross-attention outperforms self-attention when backbones are frozen but underperforms when they are unfrozen with LoRA.

• Prefer scaling the LLM (Idefics2-70B path) over architectural and data improvements (Idefics3-8B path) when: (a) your target tasks are knowledge-intensive multi-discipline reasoning (the 11.4-point MMMU gap between Idefics2-70B and Idefics3-8B suggests scale helps here), (b) you have the inference budget to serve a 70B model, and (c) your tasks don't heavily depend on OCR or fine-grained visual detail where the visual token bottleneck would dominate any scale advantage.

• Prefer training without a vision encoder (Fuyu-style, raw pixels into LLM) only if: you are willing to accept lower performance "to achieve similar performance" as the pre-trained encoder approach, per PaliGemma's findings (Section 2.2.1), AND you have a specific need for prompt-dependent visual representations that pre-trained encoders don't provide—a niche use case that the paper notes has "not yet demonstrated superior performance."