Open-Qwen2VL: Compute-Efficient Pre-Training of Fully-Open Multimodal LLMs on Academic Resources

ArXiv: 2504.00595

🎯 Pitch

A 2B-parameter multimodal LLM outperforms Qwen2-VL-2B on multiple benchmarks while using only 0.36% of the original's pre-training tokens—just 220 A100 GPU hours on 8 academic GPUs. The key insight is that careful data filtering (especially using an MLLM-based rater) matters far more than data volume, and multimodal sequence packing eliminates wasted computation from padding. The authors open-source everything, including the data, code, and filtering scripts, making true reproduction possible for the first time.

---

1. Executive Summary

Open-Qwen2VL introduces a compute-efficient pipeline for pre-training a fully-open 2B-parameter multimodal LLM on academic resources, using only 29M image-text pairs and 220 A100-40G GPU hours. The approach combines low-to-high dynamic image resolution (compressing 729 visual patches to 144 tokens during pre-training, then scaling to full resolution at fine-tuning) with multimodal sequence packing (a First-fit-decreasing bin-packing algorithm that groups variable-length image-text pairs into sequences near the 4096 context length, eliminating padding waste) and MLLM-based data filtering (specifically MLM-Filter with semantic understanding scoring at an 85/100 threshold, unioned with DFN-CLIP-filtered data) to achieve remarkable data efficiency. The final instruction-tuned model outperforms the partially-open Qwen2-VL-2B-Instruct on MMBench (80.9 vs. 68.8), SEEDBench (72.5 vs. 72.0), MMStar (49.7 vs. 46.3), and MathVista (53.1 vs. 48.0) despite being trained on only 5B multimodal tokens—0.36% of Qwen2-VL's 1.4T pre-training tokens—establishing that SOTA-competitive MLLM pre-training is feasible on academic 8×A100-40G hardware when high-quality data filtering substitutes for data scale, though the approach shows weaker OCR-VQA performance on AI2D (66.3 vs. 72.3) and TextVQA (63.3 vs. 78.8) due to the absence of OCR-specific pre-training data.

2. Context and Motivation

The Core Problem: SOTA Multimodal LLM Pre-Training Is Inaccessible to Academia

The fundamental tension this paper addresses is stark: state-of-the-art multimodal LLMs exist, but the ability to reproduce, study, and innovate on their pre-training pipelines is concentrated entirely within a handful of large industrial labs. The paper identifies a multi-dimensional accessibility crisis that goes well beyond the familiar complaint about closed-source model weights.

Table 1 lays out the landscape explicitly. The authors evaluate seven recent SOTA MLLMs—VILA [28], MM1 [54], Ideflics [24], BLIP-3 [49], Llama-3.2-Vision [13], Phi-3.5-Vision [1], and Qwen2-VL [44]—across eight dimensions of openness. The pattern is damning:

• Only one model (Ideflics) releases data filtering techniques, sequence packing scripts, and pre-training data alongside its model weights.
• Four of seven (MM1, BLIP-3, Phi-3.5-Vision, Qwen2-VL) keep their data filtering techniques closed—meaning the method by which they selected high-quality training data from noisy web-scale sources is essentially a trade secret.
• Four of seven (MM1, Llama-3.2-Vision, Phi-3.5-Vision, Qwen2-VL) do not release their sequence packing scripts—the engineering machinery that determines how variable-length multimodal training examples are assembled into efficient batches. This is not a trivial omission; as Section 2.3 of this paper demonstrates, packing algorithms directly impact training throughput and compute waste.
• Most critically, only VILA and Ideflics release their pre-training data at all. The rest provide model checkpoints—often just the instruction-tuned versions—leaving researchers with a black box they can probe but cannot scientifically examine.

The authors' definition of "fully open" crystallizes this: they require complete release of (1) the training codebase, (2) detailed data filtering techniques, and (3) all pre-training and SFT data. By this metric, every prior SOTA MLLM fails on at least one dimension, and most fail on several.

Why This Gap Matters: Beyond "Open Weights" as Openness

The paper's framing of openness goes deeper than the familiar weights-release debate. The authors identify specific scientific and practical consequences of the current closedness regime:

Reproducibility is impossible. If a research group cannot see the data filtering techniques, they cannot determine whether a model's performance derives from architecture, data quality, or data scale—variables that are fundamentally confounded. The paper quotes, implicitly, a growing concern in the ML reproducibility community: pre-training data curation is increasingly the "secret sauce" that determines model quality, and when that process is entirely opaque, published results become non-reproducible by design.

Pre-training research is gated by compute assumptions. The closed models cited in Table 1 train on enormous token budgets. DeepSeek-VL2 uses 8.1T multimodal tokens; Qwen2-VL uses 1.4T. These scales implicitly communicate that pre-training research requires industrial-scale compute. The paper directly pushes back on this assumption: "We wish to demonstrate that the research on pre-training is not only a game for giant tech companies and encourage the academic community to work on pre-training data and pipeline research even with very limited computing resources" (Section 1, final paragraph). This is not just rhetoric—it is a methodological claim that compute-efficient pre-training is possible if the pipeline is optimized for efficiency rather than brute-force scaling, and that academic researchers can contribute meaningfully to this optimization if given access to the full stack.

The academic community is locked out of pipeline innovation. Even when model weights are released, academics cannot study how to make pre-training more efficient because they cannot modify or re-run the pipeline. This creates a self-reinforcing cycle: industrial labs iterate on their closed pipelines and publish high numbers; academics, lacking access to these pipelines, cannot propose alternative data filtering strategies, packing methods, or training curriculum designs; the frontier of pipeline research migrates entirely into proprietary settings.

Where Prior Approaches Fall Short

The paper identifies specific failure modes in the status quo, organized into three categories:

1. Data Filtering Is Treated as an Artisanal Process Rather Than a Reproducible Science

Prior work had established that data quality matters enormously for MLLM pre-training (MM1 [34] provides a systematic analysis), but the actual filtering techniques used by SOTA models are almost never disclosed. When they are partially described, critical details are missing:

• DFN [14] releases a selected subset (top-15% data UIDs from DataComp-Medium) but not the model checkpoint used for filtering. This means other researchers cannot adjust the filtering threshold (e.g., to study the quality-vs-quantity tradeoff at top-5% vs. top-15% vs. top-30%) because they cannot reproduce the filtering scores. The paper explicitly notes this: "it is impossible to change the retained data fraction based on the quality scores generated by DFN-model" (Section 2.1).

• MLM-Filter [46] provides a methodology for using efficient MLLMs as data quality evaluators across four dimensions (image-text matching, object detail fulfillment, caption text quality, and semantic understanding), but prior to this paper, no fully-open MLLM demonstrated end-to-end deployment of this approach with the resulting pre-training data released for scrutiny.

• ATIQE [18] identifies semantic understanding (SU) as the most impactful quality metric among MLM-Filter's four dimensions, but this finding existed in isolation—no subsequent work showed how to integrate it into a complete, reproducible pre-training pipeline on academic compute.

The paper positions itself as filling this gap: it takes the best available filtering science (DFN-CLIP for CLIP-based filtering, MLM-Filter with SU scoring for MLLM-based filtering) and deploys them in a transparent, documented, released pipeline where every threshold, every data source, and every filtering decision is specified.

2. Pre-Training Efficiency Techniques Exist but Are Not Integrated or Openly Documented

Two specific efficiency techniques exist in the literature but are not combined in an open, reproducible system:

Multimodal sequence packing. The concept of packing variable-length sequences to minimize padding tokens is well-established in text-only LLM training (e.g., the First-fit-decreasing bin packing algorithm dates to Johnson's 1973 thesis [20]), but its application to multimodal training—where each example contains both text tokens and a variable number of images—introduces additional complexity. The closed models likely use some form of packing (MM1 [34] discusses efficient batching strategies), but the actual implementation details—how to handle multiple images per sequence, how to manage the <image> placeholder tokens across packed examples, what separator token to use—are not released. The paper provides these details (Algorithm 1, Section 2.3) and releases the packing scripts, making multimodal sequence packing a reproducible technique rather than an unspecified industrial practice.

Low-to-high dynamic image resolution. Prior work has established that higher image resolution improves visual understanding (Qwen2-VL [44] uses naive dynamic resolution to process images at their native sizes), but processing high-resolution images during pre-training is computationally expensive—Qwen2-VL processes up to 729 visual tokens per image from the start. The paper's approach of using an Adaptive Average-Pooling layer to compress 729 visual patches to 144 during pre-training, then restoring full resolution during fine-tuning, is not claimed as a novel architectural contribution. Rather, the paper identifies it as an under-documented efficiency technique that, combined with sequence packing, enables academic-scale pre-training without a fundamental accuracy penalty (the SFT-stage resolution restoration recovers high-resolution visual understanding capability).

3. The Compute-Feasibility Assumption Has Gone Unchallenged

The paper addresses a sociological gap as much as a technical one. The prevailing narrative in the MLLM community—reinforced by the token budgets in Table 4 (DeepSeek-VL2: 8.1T, Qwen2-VL: 1.4T, InternVL2.5-2B-MPO: 277B)—is that competitive MLLM pre-training requires data scales measured in hundreds of billions to trillions of tokens. This creates an implicit barrier: academic groups without access to industrial compute clusters self-select out of pre-training research, focusing instead on fine-tuning or evaluation.

The paper systematically dismantles this assumption by demonstrating that a 2B-parameter MLLM trained on 5B tokens (0.36% of Qwen2-VL's budget) can match or exceed the larger-budget model on specific benchmarks. The mechanism is not magical—it is substitution of data quality for data quantity, enabled by aggressive filtering (19.9M pairs selected from a pool of ~140M candidates across all source datasets) and compute-efficient training infrastructure.

How This Paper Positions Itself Relative to Existing Work

The paper's positioning has four distinct facets:

It is not proposing novel architectural components. Unlike Qwen2-VL [44], which introduces 2D-Multimodal RoPE and naive dynamic resolution, or InternVL-2.5 [9], which introduces Mixed Preference Optimization, Open-Qwen2VL deliberately uses simple, established components: a standard SigLIP-SO-400M vision encoder, a two-layer MLP projector with adaptive average pooling, and a Qwen2.5-1.5B-Instruct LLM backbone. The architecture description (Section 2.2) explicitly states: "Open-Qwen2VL does not adopt advanced designs of 2d-Multimodal RoPE and naive dynamic resolution to save computes and ensure the training efficiency." This is a feature, not a bug—the goal is accessibility and reproducibility.

It positions data filtering quality as the primary driver of compute efficiency. The core experimental claim is not about a better model architecture but about a better data pipeline. The ablation study in Table 3 demonstrates this: switching from CLIP-only filtered data to a mixture that includes MLLM-filtered data (DataComp-MLM-Filter & DFN) yields a +0.5 average benchmark improvement with fewer total training pairs (28.4M vs. 38.5M in the three-dataset mixture). This is the empirical spine of the paper's thesis—that compute-efficient pre-training is fundamentally about knowing which data to train on, not just having more data.

It defines openness as a scientific standard, not a marketing claim. Table 1 is a pointed intervention. By evaluating prior models across eight specific, falsifiable openness criteria, the paper creates a taxonomy that makes openness a measurable property rather than an ambiguous label. This framing is distinct from prior work: VILA [28] describes itself as "open," but the paper's analysis reveals it does not release data filtering techniques or sequence packing scripts. The "fully open" redefinition is a challenge to the field: release the complete pipeline or stop claiming openness.

It targets the academic community as its primary audience. The paper's emphasis on 8×A100-40G hardware, 220 GPU-hour budgets, and 200GB CPU memory constraints (Section 3.1) is deliberate. These are the resource profiles of university labs, not industrial research divisions. The technical decisions flow from this constraint: resizing images to 512 pixels on the smaller side during download to save disk space; converting 10M-instruction JSON files to individual JSON files with an index to fit in limited CPU memory; choosing FSDP over DeepSpeed-Zero3 because it empirically gives 17% faster training steps on the same hardware. These are not publication-motivated optimizations—they are practical accommodations that make the pipeline run on real academic hardware.

3. Technical Approach

3.1 Reader Orientation

Open-Qwen2VL is a fully-reproducible pipeline that takes raw web-scale image-text pairs and produces a competitive 2B-parameter multimodal LLM through a sequence of data filtering, efficient pre-training, and supervised fine-tuning stages. The system solves the problem of academic inaccessibility to SOTA MLLM pre-training—not by inventing new architectures, but by demonstrating that careful data curation (MLLM-based filtering that selects only the highest-quality 19.9M pairs from ~140M candidates) combined with compute-efficient training techniques (sequence packing, low-to-high image resolution) can substitute data quality for data quantity, achieving competitive performance on only 5B tokens (0.36% of Qwen2-VL's budget) using 8×A100-40G GPUs.

3.2 Big-Picture Architecture (Diagram in Words)

The Open-Qwen2VL system has five major stages connected in a linear pipeline:

1. Multi-Source Data Download and Preparation — Raw image URLs from four public datasets (CCS-CLIP, DataComp-DFN, LAION-CLIP, DataComp-MLM-Filter & DFN) are downloaded, resized to 512 pixels on the smaller side, and stored in WebDataset format. This component produces raw image-text pairs with associated quality metadata.

2. High-Quality Data Filtering and Union — Two complementary filtering strategies are applied: CLIP-based filtering (DFN's top-15% selection, CLIP cosine similarity thresholds) and MLLM-based filtering (MLM-Filter's semantic understanding scoring at 85/100 threshold). The filtered subsets are unioned and deduplicated, yielding the final 19.9M-pair training dataset.

3. Multimodal Sequence Packing — The filtered image-text pairs are regrouped into sequences using First-fit-decreasing bin packing, with each bin targeting the 4096 context length. Images are converted to 144 visual tokens each via the adaptive average-pooling projector. This component eliminates padding waste and incidentally creates pseudo-interleaved sequences.

4. MLLM Pre-Training — A frozen SigLIP-SO-400M vision encoder extracts 729 patch features per image, an adaptive average-pooling layer compresses them to 144 visual tokens, a trainable two-layer MLP projector maps them to the LLM's embedding space, and a Qwen2.5-1.5B-Instruct LLM backbone is trained for one epoch on the packed sequences using FSDP. Only the projector and LLM parameters are updated.

5. Supervised Fine-Tuning — The pre-trained base model is fine-tuned on up to 10M single-image instruction-following examples from MAmmoTH-VL-10M, with the visual token count restored to 729 and the vision encoder optionally unfrozen.

Information flows sequentially: raw URLs → downloaded/resized images → filtered pairs → packed sequences → pre-trained base model → instruction-tuned model. Each stage's output is the next stage's input, with the key decision points being filtering thresholds, packing algorithm parameters, and training hyperparameters.

3.3 Roadmap for the Deep Dive

• First, the training objective and loss function, because the entire pipeline exists to minimize this loss over the chosen data, and understanding what is being optimized grounds all subsequent design decisions.
• Second, the data preparation and filtering pipeline, because the paper's central claim is that data quality—not architecture—is the primary driver of compute efficiency, and every filtering decision (CLIP vs. MLLM-based, threshold choices, dataset union strategy) must be understood precisely.
• Third, the multimodal sequence packing algorithm, because it is the key engineering innovation that eliminates padding waste and enables the 4096-context-length training to run efficiently on academic GPUs.
• Fourth, the model architecture with low-to-high dynamic image resolution, because this is where the token compression strategy (144 during pre-training, 729 during SFT) is implemented, directly impacting pre-training throughput.
• Fifth, the training infrastructure (FSDP, WebDataset format, memory management for 10M SFT examples), because reproducing the 220 GPU-hour budget requires these specific engineering choices.
• Sixth, the ablation methodology for data mixture selection, because it demonstrates the empirical process by which the final dataset was chosen and validates the claim that MLLM-based filtering provides complementary value to CLIP-based filtering.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical pipeline paper whose core idea is that competitive MLLM pre-training is achievable on academic hardware if and only if (1) aggressive data filtering substitutes quality for quantity, (2) sequence packing eliminates padding waste, and (3) token compression during pre-training defers high-resolution processing to the cheaper SFT stage.

---

Training Objective and Loss Function

The Open-Qwen2VL pre-training uses a standard causal language modeling loss (next-token prediction) applied to multimodal sequences. The model receives a sequence of tokens where image regions are represented by learned visual embeddings interleaved with text tokens, and it is trained to predict each text token given all preceding tokens (both visual and textual).

Formally, for a packed sequence containing $K$ image-text pairs, the input consists of interleaved visual and text tokens:

$\text{input} = [V_1, T_1, V_2, T_2, \ldots, V_K, T_K]$

where $V_i$ represents the 144 visual tokens for the $i$-th image (obtained by compressing the 729 SigLIP patch features through adaptive average pooling and the MLP projector), and $T_i$ represents the text tokens of the $i$-th caption.

The training objective is:

$\mathcal{L}_{\text{CE}} = -\sum_{t \in \mathcal{T}} \log P_\theta(x_t \mid x_{<t})$

where $\mathcal{T}$ is the set of positions corresponding to text tokens, $x_t$ is the ground-truth token at position $t$, $x_{<t}$ represents all preceding tokens (both visual and text), and $P_\theta$ is the probability distribution output by the LLM backbone parameterized by $\theta$ (which includes the projector and LLM parameters but excludes the frozen vision encoder).

What it computes: the standard autoregressive cross-entropy loss summed over all text token positions in the packed sequence. The model processes the visual tokens as a prefix, then generates the caption text autoregressively; for the next image-text pair in the packed sequence, the second image's visual tokens are appended after the first caption's text tokens, and the model generates the second caption conditioned on all preceding content. The loss is computed only on text tokens—visual tokens are input-only and do not contribute to the loss.

Why this form: this is the standard causal LM objective used by virtually all MLLMs built on decoder-only LLM backbones (GPT-style). The key design choice is that visual tokens are treated as a prefix that conditions the text generation rather than as targets to be predicted. This reflects the asymmetry of the task: the model learns to describe images in text, not to generate images from text. An alternative would be to also predict visual tokens (making it a full multimodal generative model), but this would require a vision decoder head and dramatically increase computational cost—exactly what the paper's resource constraints preclude.

The loss is computed on packed sequences where multiple image-text pairs are concatenated with <|im_end|> separator tokens between them. During pre-training, the LLM backbone and projector parameters $\theta$ are updated via gradient descent, while the SigLIP vision encoder parameters remain frozen. During SFT, the loss is the same autoregressive cross-entropy, but applied to instruction-following sequences (image + instruction → response) rather than caption sequences, and optionally with the vision encoder unfrozen.

A critical implementation detail: because the sequences are packed to length 4096, the model sees multiple independent image-text pairs in a single forward pass. The causal attention mask ensures that tokens from pair $i$ can attend to tokens from pairs $j < i$ (earlier in the sequence) but not to tokens from pairs $j > i$. This is not a bug—it means the model is effectively trained on pseudo-interleaved sequences where earlier image-text pairs serve as additional multimodal context for later pairs, which the authors leverage to demonstrate emergent few-shot multimodal in-context learning (Section 4.1, Table 5).

---

Data Preparation and Filtering Pipeline

The data pipeline transforms raw image URLs from four public datasets into a curated, deduplicated set of 19.9M high-quality image-text pairs. Each source dataset and filtering strategy is described below.

Source Dataset 1: CCS-CLIP (8.5M Pairs)

The CCS dataset is the union of CC3M, CC12M, and SBU captions, curated using CLIP-based filtering by the BLIP-1 authors [26]. Concretely, BLIP-1 downloaded images from the Conceptual Captions 3M, Conceptual Captions 12M, and SBU datasets, then applied CLIP image-text similarity scoring to filter out pairs with low alignment. The exact filtering threshold is not re-specified in this paper, but the process is inherited from BLIP-1's release.

The paper downloads these images from the released URLs using img2dataset [7]. The critical preprocessing choice: images are resized such that the smaller side is 512 pixels while preserving the aspect ratio. This is explicitly justified by disk space constraints ("For academic computing resources, downloading images and saving in original resolution require huge disk space, which is unavailable in most of academic institutions"—Section 2.2). The tradeoff is that the model never sees full-resolution images during pre-training; the 512-pixel resizing is a deliberate compromise between visual fidelity and storage feasibility.

Why CCS is included: CCS is the most widely-used open-source caption dataset for vision-language model pre-training. It provides a baseline of CLIP-filtered quality that has been validated by numerous prior works. The 8.5M size is manageable for academic compute while providing broad visual coverage (general web images with descriptive captions).

Source Dataset 2: DataComp-DFN (15M Pairs)

The DataComp-Medium dataset [15] originally contains 128M image-text pairs pooled from Common Crawl. The Data-Filtering-Network (DFN) [14] method trains a CLIP-style filtering model at larger scale and selects the top-performing subset.

The DFN authors release only the indices (UIDs) of the top-15% selected data, not the DFN model checkpoint itself. The paper uses the official DataComp resharder script to extract this subset from the full DataComp-Medium pool. Of the original 128M pairs, 99.8M were successfully downloaded; applying the top-15% filter yields approximately 15M pairs.

The non-reproducibility of DFN's threshold: because DFN does not release the model checkpoint, the paper cannot adjust the retained fraction (e.g., to experiment with top-5% or top-30% filtering). The 15% threshold is fixed by what DFN chose to release. This is explicitly noted as a limitation ("it is impossible to change the retained data fraction based on the quality scores generated by DFN-model"—Section 2.1). The paper must work with the released subset as-is, which constrains the data mixture ablation studies.

Why DataComp-DFN is included: DFN is the top-performing independent filter on the DataComp medium filtering track leaderboard, establishing it as the best available CLIP-based filtering method. Its inclusion provides a strong CLIP-filtered baseline that represents the state-of-the-art in contrastive-model-based data curation.

Source Dataset 3: LAION-CLIP (15M Pairs)

LAION-400M [39] is a web-scale image-text dataset filtered using OpenAI's CLIP model with a strict 0.3 cosine similarity threshold. The paper downloads only 15M pairs from LAION-400M (not the full 400M), explicitly to "perform controlled-size data mixture ablation studies" (Section 2.1). This controlled subsetting is critical: it ensures that when comparing data mixtures in Table 3, the total training data volume is held roughly constant, isolating the effect of data composition rather than data quantity.

The filtering mechanism: for each image-text pair in the LAION-400M pool, the OpenAI CLIP ViT-B/32 model computes the cosine similarity between the image embedding and text embedding. Pairs with similarity below 0.3 are discarded. The threshold 0.3 is a design choice from LAION-400M's original release and is inherited without modification.

Why LAION-CLIP is included: LAION-400M is the most widely-used open-source large-scale image-text dataset. Including a subset enables direct comparison with models that use LAION data and tests whether LAION's CLIP-filtering produces meaningfully different data quality compared to DataComp-DFN's CLIP-filtering.

Source Dataset 4: DataComp-MLM-Filter & DFN (19.9M Pairs)

This is the paper's primary methodological contribution to data filtering. It combines the DFN-CLIP-filtered 15M subset with an additional 8M pairs selected by MLM-Filter [46], an MLLM-based filtering method.

MLM-Filter's quality dimensions. MLM-Filter evaluates each image-text pair along four axes, each producing a score from 0 to 100:

1. Image-Text Matching (ITM): Does the caption accurately describe the visual content of the image? This is the most direct quality signal and is analogous to what CLIPScore measures, but uses an MLLM's more sophisticated visual understanding rather than a contrastive embedding similarity.

2. Object Detail Fulfillment (ODF): Does the caption mention specific objects present in the image, and does it describe them with sufficient detail (attributes, quantities, spatial relationships)? This catches captions that are vaguely correct but missing important visual elements.

3. Caption Text Quality (CTQ): Is the caption well-formed natural language? This evaluates fluency, grammaticality, and whether the caption is a coherent description rather than a collection of keywords or a mangled alt-text.

4. Semantic Understanding (SU): Does the caption demonstrate understanding of the image's semantic content—the relationships between objects, the scene type, the activity or event depicted—rather than just listing objects?

Why SU is used exclusively. Based on findings from ATIQE [18], the Semantic Understanding (SU) metric yields the best downstream MLLM performance when used as the sole filtering criterion. The paper does not combine all four metrics or use a weighted ensemble; it applies SU scoring alone. The rationale: a caption that demonstrates semantic understanding inherently satisfies the other criteria to some degree (you cannot demonstrate semantic understanding without some level of image-text matching, object detail, and text quality), while the inverse is not true (a caption can have high ITM but low SU by accurately describing objects without understanding the scene).

The filtering model. The paper uses mlm-filter-qwen2.5-1.5b-gpt4o, a Qwen2.5-1.5B model fine-tuned on GPT-4o-generated quality annotations. This model is an efficient MLLM in its own right—it processes each image-text pair and outputs a numerical SU score between 0 and 100.

The filtering threshold. Pairs with SU score ≥ 85 out of 100 are retained. This is an aggressive threshold: it selects only the highest-quality pairs where the MLLM judge is confident that the caption demonstrates genuine semantic understanding. From the pool of DataComp-Medium pairs, this yields approximately 8M pairs.

Union and deduplication. The 8M MLM-Filter-selected pairs are unioned with the 15M DFN-CLIP-selected pairs, then deduplicated. The deduplication removes pairs that appear in both subsets—pairs that are rated highly by both CLIP-based and MLLM-based filters. The final dataset contains 19.9M unique pairs.

Why union rather than intersection: the paper explicitly hypothesizes that "MLLM-based data filter may introduce a different data distribution into the pre-training set, which brings new knowledge for enhancing the MLLM capabilities" (Section 2.5). An intersection (keeping only pairs selected by both DFN and MLM-Filter) would reduce data volume and miss pairs that the MLLM filter identifies as high-quality but CLIP misses—exactly the pairs that provide complementary training signal. The union strategy maximizes data diversity while still enforcing quality through at least one rigorous filter.

The key design insight: CLIP-based filtering evaluates alignment between image and text embeddings, which is a shallow, distributional signal. MLLM-based filtering evaluates genuine semantic understanding, which catches qualitatively different failure modes (e.g., a caption that is distributionally similar to good captions but factually hallucinates object relationships). By combining both, the dataset covers two complementary quality axes.

Data Storage Format: WebDataset

All filtered image-text pairs are stored in WebDataset format, where each tar file contains exactly 10,000 pairs. This format choice has two motivations:

1. Efficient sequential access: WebDataset stores data as tar archives that can be streamed sequentially during training, avoiding the need to load a massive index or JSON metadata file into memory.

2. Natural sharding for distributed training: With exactly 10k pairs per tar file, the dataset is naturally sharded across distributed data-parallel workers—each GPU can load a different tar file, eliminating coordination overhead.

The WebDataset format choice directly enables the sequence packing pipeline: the packing algorithm operates on each 10k-pair tar file independently (Section 2.3), grouping the pairs within that file into packed sequences of length 4096.

---

Multimodal Sequence Packing

Sequence packing is the engineering mechanism that eliminates padding waste during pre-training. Without packing, the standard approach is to batch image-text pairs of similar lengths together and pad all sequences in the batch to the length of the longest sequence. Because image-text captions vary dramatically in length—a short caption might be 20 tokens, a detailed description might be 500 tokens—padding can easily account for 30–50% of all tokens processed, which are wasted computation.

The multimodal sequence packing algorithm (Algorithm 1 in the paper) solves this by regrouping variable-length image-text pairs into fixed-length bins that exactly fill the 4096 context window. The algorithm has three phases:

Phase 1: Compute Multimodal Lengths

For each image-text pair, the total multimodal length is:

$\ell_d = |T_d| + |V_d|$

where $|T_d|$ is the number of text tokens in the caption (after tokenization with the Qwen2.5 tokenizer) and $|V_d| = 144$ is the fixed number of visual tokens per image during pre-training.

The fixed visual token count is crucial: because every image is compressed to exactly 144 tokens via the adaptive average-pooling projector, the visual token contribution to sequence length is constant. This makes the bin-packing problem purely a function of text length variation—without this compression, the visual token count would depend on image resolution, and the packing problem would be substantially more complex.

Additionally, an <image> placeholder token is prepended to each caption (following LLaVA convention), which adds 1 token per image-text pair. The effective length per pair is therefore $144 + 1 + |T_d|$.

Phase 2: First-Fit-Decreasing Bin Packing

The packing uses the First-Fit-Decreasing (FFD) algorithm, a classic approximation algorithm for the bin packing problem [20]. The procedure:

1. Sort descending: All image-text pairs are sorted by their multimodal length in descending order (longest pairs first).

2. Initialize empty bins: Start with no bins.

3. Iterate through sorted pairs: For each pair, attempt to place it into an existing bin where the total length of all pairs already in that bin plus the new pair's length does not exceed the context length $L = 4096$. If such a bin exists, place the pair in the first one found ("first fit"). If no existing bin can accommodate the pair, create a new bin.

4. Continue until all pairs are placed.

Why FFD rather than a greedy online approach: sorting in descending order and using first-fit is a well-known $\frac{11}{9}$-approximation algorithm for bin packing—it is guaranteed to use at most 11/9 times the optimal number of bins. This near-optimality is important because it means the packing is highly efficient: across the 10k pairs in each tar file, the algorithm produces bins that are nearly full on average, minimizing the number of sequences that need padding.

Why 4096 context length: this is the native context window of Qwen2.5-1.5B-Instruct during its text-only pre-training. Using the same context length for multimodal pre-training avoids the need to extend positional embeddings or implement length-extrapolation techniques, simplifying the training infrastructure. It also means the model processes tokens in chunks similar to what it was pre-trained on, reducing distribution shift.

Phase 3: Concatenation and Storage

After bin assignment, each bin becomes one training sequence:

1. Concatenate input_ids: The text token IDs from all pairs in the bin are concatenated, with <|im_end|> tokens inserted as separators between pairs. The <image> placeholder token is inserted at the beginning of each pair's caption.

2. Concatenate images: The PIL image objects from all pairs in the bin are concatenated into a list, preserving the order matching the text tokens.

3. Pad to length L: If the total token count is less than 4096, pad with padding tokens to exactly 4096. Because FFD packs bins tightly, this padding is minimal—ideally zero for most bins.

4. Store as pickle: Each packed sequence is stored as a pickle file containing a dictionary with three keys:

   • "images": a list of PIL image objects
   • "input_ids": a torch LongTensor of length 4096 with image placeholder tokens at the positions where images appear
   • "lengths": a list of integers recording the multimodal length of each original image-text pair in the sequence

The lengths field is a crucial implementation detail that enables correct loss computation during training. Because the model should only compute the cross-entropy loss on text tokens (not visual tokens), and because multiple pairs are packed into one sequence, the training loop needs to know where each pair's text tokens begin and end to mask the loss appropriately. The lengths field provides exactly this information, enabling per-pair loss masking within a single packed sequence.

Incidental pseudo-interleaving: because FFD packs multiple unrelated image-text pairs into one sequence, the resulting training examples structurally resemble image-text interleaved documents (like those in the M3W dataset from Flamingo [3]). The model sees sequences like "Image A → Caption A → Image B → Caption B → Image C → Caption C," which trains it to process multimodal context where earlier images can inform later captions. The authors exploit this in Section 4.1, showing that the pre-trained base model exhibits few-shot multimodal in-context learning capabilities despite never being explicitly trained on interleaved data.

Scale: for a 10k-pair tar file, FFD typically produces a few thousand packed sequences (the exact number depends on the text length distribution). Across 19.9M total pairs, the total number of training sequences is approximately 19.9M ÷ (average pairs per bin), where the average pairs per bin depends on the typical caption length. With captions averaging roughly 50–100 tokens and 144 visual tokens per image, each bin can hold approximately 20–40 pairs, yielding roughly 0.5–1M packed sequences in total—all of which are processed exactly once (one-epoch training).

---

Model Architecture with Low-to-High Dynamic Image Resolution

The Open-Qwen2VL architecture uses three components: a frozen vision encoder, a trainable projector, and a trainable LLM backbone. The key innovation is the two-stage resolution strategy: compress images to 144 tokens during pre-training for efficiency, then restore to 729 tokens during SFT for high-resolution understanding.

Vision Encoder: SigLIP-SO-400M

The SigLIP-SO-400M model [53] is a Vision Transformer (ViT) pre-trained with a sigmoid loss for image-text alignment. Given an input image resized to 384×384 pixels (the native resolution of this ViT variant), it produces a grid of patch features. With a patch size of 14 pixels, a 384×384 image yields $384/14 \approx 27$ patches per side, producing $27 \times 27 = 729$ visual patch tokens.

Each patch token is a $D_{\text{vit}}$-dimensional vector (specifically 1152 for SigLIP-SO-400M). These 729 vectors encode the visual content of the image at a relatively fine spatial granularity.

Why SigLIP: compared to the more common OpenAI CLIP, SigLIP's sigmoid loss formulation provides better calibration for image-text similarity scores and has been shown to produce stronger visual representations for downstream tasks. The SO-400M variant is a good size-efficiency tradeoff: it provides strong visual features while being small enough to run on academic GPUs (the vision encoder is frozen, so its forward pass cost is incurred once per image during data preprocessing or training).

Why frozen: freezing the vision encoder during pre-training saves substantial GPU memory and compute. The encoder's 400M parameters would require gradients, optimizer states, and activation storage during training if trainable. By freezing it, only the forward pass activations need to be stored, and only for the current batch. This is explicitly justified by compute constraints: "For both the pre-training and SFT stages, we freeze the parameters of vision encoder and make the parameters of projector and LLM backbone trainable to save more computes" (Section 2.2).

Visual Projector: Adaptive Average-Pooling + MLP

The visual projector serves two functions: dimensionality mapping (from the vision encoder's output dimension to the LLM's embedding dimension) and token compression (reducing the 729 patch tokens to a smaller number of visual tokens).

Architecture:

1. 2D Adaptive Average-Pooling layer: This layer takes the $27 \times 27$ grid of patch features and applies adaptive average pooling to resize it to a configurable target grid size. During pre-training, the target size is $12 \times 12 = 144$ tokens. The pooling operation computes the mean feature vector within each spatial bin, effectively downsampling the visual features by a factor of $27/12 = 2.25$ per spatial dimension.

2. Two-layer MLP: The pooled features (now 144 vectors of dimension $D_{\text{vit}}$) are projected to the LLM's embedding dimension $D_{\text{llm}}$ through a two-layer MLP with a non-linear activation function (architecture details from prior work [50]—the DeCo paper on decoupling token compression from semantic abstraction). The MLP has shape $D_{\text{vit}} \rightarrow D_{\text{hidden}} \rightarrow D_{\text{llm}}$.

The compression design space: the adaptive average-pooling approach is chosen over alternatives for specific reasons:

• vs. learned query pooling (e.g., Q-Former): Learned query approaches like BLIP-2's Q-Former use a fixed number of learnable query tokens that cross-attend to visual features. This is more expressive but adds trainable parameters and computational overhead. Average pooling is parameter-free and deterministic, making it computationally cheaper and more stable during training.

• vs. simple linear projection with all 729 tokens: Processing 729 visual tokens per image during pre-training would make each image-text pair consume $729 + |T_d|$ tokens instead of $144 + |T_d|$, a 5× increase in visual token count. This would reduce the number of image-text pairs that can fit in the 4096 context window from roughly 20–40 to roughly 4–8, making sequence packing far less effective and requiring more training steps to see the same number of images.

• vs. learnable spatial downsampling (convolutions): A convolutional downsampling layer (e.g., stride-2 convolutions) would introduce trainable parameters in the compression step. The adaptive average-pooling is parameter-free, reducing the projector's parameter count and making it faster to train.

Why average pooling specifically: average pooling computes the mean of features within each spatial region. This has the property of being a linear operation that preserves the expected magnitude of features—unlike max pooling, which amplifies the most active features and discards weaker ones. For visual features that represent a mix of object presence, textures, and spatial relationships, averaging is more likely to retain information from all regions of the image rather than focusing on the most salient patches. However, this is an architectural choice inherited from DeCo [50] rather than validated through ablation in this paper.

Low-to-high resolution strategy: during pre-training, the adaptive average-pooling target size is $12 \times 12$, producing 144 visual tokens per image. During SFT, the target size is restored to $27 \times 27$, producing the full 729 visual tokens. The MLP projector is shared between stages, meaning it must project features at both spatial resolutions—the adaptive pooling layer handles the resolution change, while the MLP operates on fixed-dimensional vectors regardless of how many tokens are produced.

Why defer high resolution to SFT: pre-training processes 19.9M images—processing each at 729 tokens instead of 144 would multiply the visual token budget by 5×, dramatically increasing pre-training cost. SFT processes only 10M examples, making high-resolution processing cheaper in absolute terms. Moreover, the paper's hypothesis (validated by the results) is that high-resolution visual understanding is primarily needed for fine-grained tasks (reading text, counting objects, understanding diagrams) that are better learned during instruction tuning with task-specific supervision, while pre-training can focus on coarse visual-semantic alignment at lower resolution.

LLM Backbone: Qwen2.5-1.5B-Instruct

The LLM backbone is the instruction-tuned variant of Qwen2.5-1.5B, a 1.5-billion-parameter decoder-only transformer. Using the Instruct variant (rather than the base model) is a deliberate choice: the instruction-tuned model already possesses strong language understanding and instruction-following capabilities, which the paper leverages during both pre-training and SFT.

Why 1.5B parameters: this is the smallest practical LLM size for competitive multimodal performance. Smaller LLMs (e.g., Qwen2.5-0.5B) would reduce training cost further but likely sacrifice too much language understanding capability to serve as effective multimodal backbones. The 1.5B size sits at a "sweet spot" where the total model (vision encoder + projector + LLM) is approximately 2B parameters—large enough to be competitive with other 2B-class MLLMs (Qwen2-VL-2B, InternVL2.5-2B, DeepSeekVL-2-Tiny), small enough to train on 8×A100-40G GPUs.

Why not train from scratch: the LLM backbone is pre-trained on text-only data before multimodal pre-training begins. This ensures the model already possesses strong language capabilities; the multimodal pre-training only needs to teach it to condition on visual inputs, not to learn language from scratch. This is the standard MLLM construction approach and is critical for compute efficiency—jointly learning language and vision would require orders of magnitude more compute.

Complete Forward Pass

For a single image-text pair during pre-training:

1. Vision encoding: The 512-pixel image is resized to 384×384 and fed through the frozen SigLIP-SO-400M encoder, producing a $27 \times 27 \times 1152$ feature grid.

2. Token compression: The adaptive average-pooling layer with target size $12 \times 12$ reduces this to $12 \times 12 \times 1152$, i.e., 144 vectors of dimension 1152.

3. Projection: Each of the 144 vectors is passed through the two-layer MLP, producing 144 vectors of dimension $D_{\text{llm}}$ (the embedding dimension of Qwen2.5-1.5B, which is 1536 for this model size).

4. Token embedding insertion: The 144 projected visual tokens are inserted at the position of the <image> placeholder token in the text sequence, replacing it. The text tokens are embedded via the LLM's standard token embedding layer.

5. LLM forward pass: The combined sequence of visual embeddings and text embeddings is processed by the Qwen2.5-1.5B transformer, which applies standard causal self-attention and feed-forward layers. The model produces logits over the vocabulary at each text token position.

6. Loss computation: Cross-entropy loss is computed at text token positions only, using the lengths field from the packed sequence to identify which positions correspond to text tokens.

During SFT, the process is identical except the adaptive average-pooling target size is $27 \times 27$ (729 visual tokens) and the sequences are not packed (each SFT example is a single image-instruction-response triplet).

---

Training Infrastructure and Codebase

The training codebase is built on Prismatic-VLM [21], with modifications to support multimodal packed sequences and FSDP (Fully-Sharded Data Parallel) training.

FSDP vs. DeepSpeed-Zero3

The paper uses PyTorch's FSDP implementation rather than the more commonly-used DeepSpeed-Zero3. Both implement the same core idea—shard model parameters, gradients, and optimizer states across GPUs to reduce per-GPU memory—but the paper reports that FSDP achieves "approximately 17% faster for each training step than the DeepSpeed implementation, consistent with findings reported by Karamcheti et al." (Section 2.4).

Why the speed difference: FSDP is natively integrated into PyTorch's autograd engine, allowing it to overlap communication (parameter gathering before each layer's forward pass, gradient scattering after each layer's backward pass) with computation more efficiently than DeepSpeed, which hooks into PyTorch at a higher level. For the relatively small 2B-parameter model on 8 GPUs, the communication overhead is modest, and FSDP's tighter integration yields measurable throughput improvements.

Dataloader Modifications for Packed Sequences

The original Prismatic-VLM dataloader only supports single-image SFT examples. The paper modifies it to handle:

1. Multiple images per sequence: The packed sequences contain a list of PIL images rather than a single image. The dataloader's batch preparation code is extended to process variable numbers of images per sequence.

2. Length-based loss masking: The lengths field is used to construct a boolean mask that identifies text token positions, ensuring the loss is only computed on text tokens within each packed pair.

3. WebDataset streaming: The dataloader streams tar files from disk, unpacks them on-the-fly, and feeds packed pickle files to the training loop. This avoids loading the entire dataset into memory.

Pre-Training Hyperparameters

From Appendix Table 7, the pre-training configuration:

The single-epoch design choice: training for exactly one epoch means each image-text pair is seen exactly once. This is unusual—most pre-training runs use multiple epochs—but is motivated by two factors: (1) the dataset is relatively small (19.9M pairs), so multiple epochs risk overfitting, and (2) compute budget is tightly constrained, and a single epoch is sufficient duration for the model to learn visual-semantic alignment given the high data quality.

Global batch size 256: with 8 GPUs, each GPU processes a local batch of 256 / 8 = 32 sequences per step. Each sequence is 4096 tokens, so the total tokens per step is 256 × 4096 = 1,048,576 tokens. Over the full dataset of packed sequences, this determines the total training steps (approximately 19.9M pairs ÷ average pairs per sequence ÷ 256 sequences per step).

BF16 precision: Brain Floating Point 16 uses the same exponent range as FP32 but with reduced mantissa precision. This prevents the gradient underflow/overflow issues that can occur with FP16 while still providing memory and speed benefits. BF16 is natively supported on A100 GPUs.

Peak learning rate 5e-5: relatively standard for LLM fine-tuning on new modalities. The linear warmup over the first 3% of steps helps prevent gradient spikes early in training when the randomly initialized projector produces poor visual embeddings. Cosine decay to near-zero ensures smooth convergence.

SFT Hyperparameters

From Appendix Table 7, the SFT configuration differs from pre-training in several key ways:

Why lower learning rate for SFT: by the SFT stage, the model has already learned visual-semantic alignment. The SFT objective is to fine-tune the model for instruction-following behavior, which requires more careful optimization to avoid catastrophic forgetting of pre-trained capabilities. The lower learning rate (2e-5 vs. 5e-5) reflects this.

Why higher weight decay for SFT: weight decay of 0.1 is relatively high and likely serves as regularization against overfitting to the SFT data distribution. With 10M SFT examples, there is a risk of memorizing specific instruction patterns; stronger weight decay encourages the model to stay closer to its pre-trained weights.

Why no sequence packing during SFT: SFT examples are instruction-response pairs that typically use most of the 4096 context window (instructions plus model responses). Packing would put multiple distinct conversations into one sequence, confusing the model about conversation boundaries—the model might learn to continue responding after a conversation should have ended.

Memory Management for 10M SFT Examples

The MAmmoTH-VL-10M dataset [16] contains 10M single-image instruction-following examples. The standard LLaVA dataloader loads the entire dataset (as a JSON file) into memory at once, which requires over 200GB of CPU memory—exceeding typical academic server capacity.

The paper's solution: store each of the 10M examples as an individual JSON file on disk, and load only an index file (containing file paths, data types, and pre-computed lengths) into memory. The dataloader uses this index to fetch individual examples on-the-fly during training, streaming from disk rather than pre-loading everything.

The index file contains for each example:

• Path to the individual JSON file
• Boolean flag: text-only or image-text data
• Pre-computed sequence length (for dynamic batching)

This approach reduces CPU memory usage from 200+GB to the size of the index file (approximately 10M × ~200 bytes ≈ 2GB, plus in-flight batches), making SFT on 10M examples feasible on academic servers with 256GB or even 128GB of RAM.

---

Ablation Methodology for Data Mixture Selection

The paper evaluates four data mixture configurations to select the optimal pre-training dataset, with all experiments using a controlled protocol to isolate the effect of data composition.

Controlled Protocol

For each data mixture, the following protocol is followed exactly:

1. Pre-train the MLLM for exactly one epoch on the packed multimodal sequences from that mixture.
2. Fine-tune the resulting base model on LLaVA-665k instruction data [30] using identical SFT hyperparameters.
3. Evaluate on eight multimodal benchmarks: AI2D-test, TextVQA-val, POPE, MMMU-val, MMBench-v1.0-dev, SEEDBench-img-dev, MMStar, and MathVista-test-mini.

Why LLaVA-665k as the controlled SFT dataset: using a fixed, relatively small SFT dataset (665k examples, well below the eventual 10M SFT scale) ensures that differences in final performance reflect differences in pre-training data quality, not differences in SFT scale or composition. LLaVA-665k is the standard open-source visual instruction dataset, making results comparable to prior work.

The Four Data Mixtures

The four configurations are constructed by starting with CCS-CLIP (8.5M pairs) as a fixed base, then incrementally adding other datasets. The naming convention is additive—each mixture includes all datasets from previous mixtures plus the new one:

1. CCS + DataComp-DFN (23.5M pairs): CCS-CLIP (8.5M) + DataComp-DFN (15M)
2. CCS + LAION (23.5M pairs): CCS-CLIP (8.5M) + LAION-CLIP (15M)
3. CCS + DataComp-DFN + LAION (38.5M pairs): CCS-CLIP (8.5M) + DataComp-DFN (15M) + LAION-CLIP (15M)
4. CCS + DataComp-MLM-Filter & DFN (28.4M pairs): CCS-CLIP (8.5M) + DataComp-MLM-Filter & DFN (19.9M)

Why not all 16 combinations: the paper explicitly states "there is 16 combinations between the four datasets, we only consider 4 combinations." The selection is structured to test specific hypotheses:

• Mixture 1 vs. 2: Tests whether DataComp-DFN and LAION-CLIP produce equivalent models (both are CLIP-filtered web data). Result: nearly identical performance (55.3 vs. 55.4 average), confirming data homogeneity.
• Mixture 3 vs. 1/2: Tests whether adding more CLIP-filtered data helps. Result: marginal improvement (55.5 vs. 55.3/55.4), suggesting diminishing returns from scaling within the same filtering paradigm.
• Mixture 4 vs. 1: Tests whether replacing 5M CLIP-filtered pairs with 5M MLLM-filtered pairs improves quality despite smaller total data volume. Result: +0.5 average improvement with 4.9M fewer pairs, supporting the quality-over-quantity thesis.

The key result: Mixture 4 (CCS + DataComp-MLM-Filter & DFN, 28.4M pairs) achieves the highest average benchmark score (56.0) while being the second-smallest mixture (only Mixture 1/2 at 23.5M are smaller, but they score lower). This validates the paper's central claim: MLLM-based filtering introduces a complementary quality signal that CLIP-based filtering alone cannot capture.

Compute Budget Accounting

The pre-training on the best mixture (CCS + DataComp-MLM-Filter & DFN, 28.4M pairs → packed into sequences → one epoch) takes "about 220 A100-40G GPU hours" (Section 2.5). The SFT on LLaVA-665k (for the ablation models) takes "48 A100-40G GPU hours." The final model's SFT on MAmmoTH-VL-10M takes longer (the paper reports checkpointing every 2M examples, suggesting 10M total), but the exact SFT time is not specified.

What 220 GPU hours means operationally: on 8×A100-40G GPUs, this is 220 / 8 = 27.5 hours (just over one day) of continuous training. This is the key number for academic reproducibility—a lab with a single 8-GPU server can complete the full pre-training in roughly a day, making it feasible to iterate on data filtering strategies, packing algorithms, or training hyperparameters.

4. Key Insights and Innovations

Innovation 1: Openness as a Falsifiable Property Redefines the MLLM Research Landscape

The paper's most conceptually ambitious contribution is not a technical method but a redefinition of what "fully open" means for multimodal LLMs, converting a vague, easily-abused label into a set of eight specific, falsifiable criteria. Prior work in the MLLM space exhibited what might be called openness theater: model weights were released (often only the instruction-tuned variants), and this was sufficient to claim openness, even when the data filtering techniques, sequence packing scripts, pre-training data, and training codebase remained completely closed.

Table 1 makes this visible in a way that no prior MLLM paper had attempted. By evaluating seven SOTA models—VILA, MM1, Ideflics, BLIP-3, Llama-3.2-Vision, Phi-3.5-Vision, and Qwen2-VL—across eight specific openness dimensions, the paper transforms openness from a marketing claim into an empirically measurable property. The result is damning in its specificity: only Ideflics qualifies as fully open under these criteria; VILA releases pre-training data and code but not filtering techniques or packing scripts; MM1 is closed on every dimension; Qwen2-VL releases only model checkpoints (both base and instruct) and nothing else.

The significance of this move extends beyond shaming industrial labs. It creates an accountability framework that the academic community can use to evaluate future model releases. A lab claiming their model is "open" can now be asked: do you release your data filtering details, or just the weights? The criteria are specific enough to prevent motte-and-bailey rhetorical moves—no one can claim "we're open because we released the instruct checkpoint" while withholding the training pipeline. This is a fundamentally different kind of contribution than a new architecture or loss function; it is a sociological intervention in how the research community evaluates claims of reproducibility.

The paper's own compliance with this framework—releasing training codebase, data filtering details, all pre-training and SFT data, both base and instruct checkpoints—is not merely virtue signaling. It is a proof of existence: a demonstration that SOTA-competitive MLLM pre-training can be achieved without any secret sauce. By meeting all eight criteria, the paper challenges the implicit assumption that industrial labs must keep their pipelines closed to maintain competitive advantage. If a model trained on 0.36% of Qwen2-VL's token budget can outperform it on multiple benchmarks using only publicly documented techniques, then the secrecy surrounding filtering methods and data composition in closed models is revealed as a strategic choice rather than a technical necessity.

This innovation is fundamental rather than incremental: it changes what the field considers a valid contribution to MLLM research. A paper that releases model weights but hides its data pipeline is now, by this paper's framework, making a partially reproducible claim—and the community has a vocabulary for exactly which parts are missing.

---

Innovation 2: The Data Filtering Complementarity Hypothesis—CLIP and MLLM-Based Filters Capture Qualitatively Different Quality Signals

The ablation study in Table 3 produces a result that is easy to overlook because it is modest in absolute magnitude (+0.5 average benchmark improvement for the best mixture vs. the second-best, and only +0.7 vs. the worst), but its conceptual implications are substantial. The paper demonstrates that unioned CLIP-filtered and MLLM-filtered data outperforms larger datasets filtered by only one method, even when the total training pairs are fewer.

The dominant assumption in prior data curation work was that filtering methods exist on a quality spectrum—DFN outperforms basic CLIP filtering on DataComp leaderboards, so it is "better"—and the goal is to find the single best filter and apply it. The DataComp benchmark [15] itself, which the paper explicitly engages with, is structured as a competition to find the best filtering method for CLIP pre-training. Within that framing, you pick the winner and use it.

The Open-Qwen2VL result challenges this scalar quality assumption. The best mixture (CCS + DataComp-MLM-Filter & DFN, 28.4M pairs, 56.0 average) outperforms the three-way CLIP-filtered mixture (CCS + DataComp-DFN + LAION-CLIP, 38.5M pairs, 55.5 average) despite having 10.1M fewer pairs. If CLIP-based filtering and MLLM-based filtering captured the same quality signal, adding more CLIP-filtered data (moving from 23.5M to 38.5M) would at least maintain performance, and the unioned mixture would be equivalent to just using more of whichever filter was stronger. Instead, the unioned mixture with fewer pairs performs better.

The paper hypothesizes a specific mechanism for this complementarity: "MLLM-based data filter may introduce a different data distribution into the pre-training set, which brings new knowledge for enhancing the MLLM capabilities" (Section 2.5). This is the data distribution diversity hypothesis—CLIP-based filtering selects for image-text embedding alignment (a shallow, distributional signal), while MLLM-based filtering via semantic understanding selects for genuine comprehension (a deeper signal about whether the caption demonstrates understanding of the scene). These are qualitatively different quality axes, and a dataset that includes pairs selected by both covers a broader range of "high-quality" than a dataset twice as large but selected by only one method.

The evidence for this is in the comparison between Mixture 1 (CCS + DataComp-DFN, 55.3) and Mixture 2 (CCS + LAION, 55.4): they perform nearly identically, confirming that DataComp-DFN and LAION-CLIP—both CLIP-based filters—select data from similar distributions. Adding LAION to DataComp-DFN (Mixture 3, 55.5) yields only marginal improvement despite 15M additional pairs. The jump from 55.5 (Mixture 3, 38.5M pairs, all CLIP-filtered) to 56.0 (Mixture 4, 28.4M pairs, CLIP + MLLM-filtered) is small but directionally significant: removing 10M CLIP-filtered pairs and adding 5M MLLM-filtered pairs improves performance. This is not explainable by scale; it requires complementarity.

This innovation is conceptual rather than architectural. The paper does not propose a new filtering algorithm—it uses existing DFN and MLM-Filter methods. The contribution is the diagnostic finding that these methods are not competitors on a quality ladder but complementary tools that sample from different regions of the data quality space. This reframes the data filtering research agenda: rather than searching for a single best filter, researchers should investigate filter ensembles where different quality criteria capture non-overlapping aspects of data quality.

---

Innovation 3: Multimodal Sequence Packing as an Incidental In-Context Learning Mechanism

The sequence packing algorithm described in Section 2.3 (Algorithm 1) is presented primarily as a compute-efficiency technique—eliminating padding waste to maximize throughput on academic GPUs. But Section 4.1 reveals that packing has a functionally significant side effect: by concatenating unrelated image-text pairs into single training sequences, the model is effectively trained on pseudo-interleaved multimodal data, and it acquires few-shot multimodal in-context learning capabilities as an emergent property.

This is a diagnostic finding that changes how to think about in-context learning in MLLMs. Prior work, particularly Flamingo [3], had established that explicitly constructed interleaved image-text data (Flamingo's MultiModal MassiveWeb dataset) could elicit multimodal in-context learning. The implicit assumption was that this required carefully designed data—images and text from the same web page, preserving genuine semantic relationships between adjacent items. Open-Qwen2VL demonstrates that random concatenation of unrelated image-text pairs produces the same emergent behavior. The 8-shot results in Table 5 show gains of +3% to +12% across VQA datasets compared to 0-shot, with no interleaving-specific training data whatsoever.

The mechanism is straightforward but the implications are not. During packed pre-training, the model sees sequences like "Image of a dog → 'A brown dog running in a park' → Image of a cat → 'A cat sleeping on a couch' → ..." and learns to condition each caption on all preceding images and captions in the sequence, even though they are semantically unrelated. This trains the model's attention mechanism to use earlier image-text pairs as demonstrations—it learns that when an image is followed by a descriptive caption, subsequent images should also be followed by descriptive captions in a similar style. The model does not need the demonstrations to be relevant (a dog image does not help describe a cat); it needs them to establish the task format.

This is a reinterpretation of what in-context learning requires. The Flamingo view emphasized semantic coherence (images and text from the same source document). The Open-Qwen2VL result suggests that task format consistency alone is sufficient, and that semantic coherence is unnecessary for eliciting the behavior. This has practical implications: if random packing produces in-context learning, then any MLLM pre-trained with packed sequences—including the closed models that hide their packing implementations—likely possesses this capability as an untested side effect. The capability is not designed; it is emergent from the efficiency optimization.

The innovation here is not the packing algorithm itself (FFD bin packing dates to 1973) but the demonstration that a purely efficiency-motivated training choice produces a qualitatively new model capability, and the implication that this capability is likely latent in many existing MLLMs that use packing but have never been evaluated for in-context learning. This is a fundamental finding about training dynamics rather than an incremental engineering contribution.

---

Innovation 4: The Low-to-High Resolution Strategy Demonstrates That Visual Token Compression During Pre-Training Is Lossless for Final Performance

The paper's architecture uses adaptive average pooling to compress 729 visual patch tokens to 144 during pre-training, then restores full resolution during SFT. This is not presented as a novel technique—the DeCo paper [50] introduced the general approach of decoupling token compression from semantic abstraction. But the paper provides the first empirical demonstration that this strategy enables competitive final performance against models that never compressed, establishing a new efficiency-vs-quality Pareto frontier for MLLM pre-training.

The comparison with Qwen2-VL-2B (Table 4) is the critical evidence. Qwen2-VL processes images at full native resolution using naive dynamic resolution throughout its 1.4T-token pre-training—it never compresses. Open-Qwen2VL compresses during its 5B-token pre-training, then decompresses during SFT. The final models are compared on identical benchmarks, and Open-Qwen2VL outperforms Qwen2-VL on MMBench (80.9 vs. 68.8), SEEDBench (72.5 vs. 72.0), MMStar (49.7 vs. 46.3), and MathVista (53.1 vs. 48.0).

This is surprising. The natural assumption is that compressing visual information during pre-training (144 tokens instead of 729, a ~5x reduction in visual information bandwidth) would permanently limit the model's visual understanding—that high-resolution visual features need to be learned alongside the initial visual-semantic alignment, and that compressing them during pre-training creates an information bottleneck that cannot be fully recovered during SFT. The results falsify this assumption.

Why does decompression during SFT recover performance? The likely mechanism is that coarse visual-semantic alignment (mapping images to language at a gist level—"this is a dog in a park") can be learned effectively from compressed visual features, while fine-grained visual understanding (reading text in images, counting objects, identifying small details) is task-specific and can be acquired during instruction tuning when the model has explicit supervision for these capabilities. The pre-training phase teaches the model that visual tokens carry semantic information that should influence text generation; the SFT phase teaches it to extract detailed information from high-resolution features for specific tasks.

This innovation is an empirical validation of a previously theoretical efficiency strategy. The DeCo paper proposed the compression-decompression approach; Open-Qwen2VL demonstrates that it works at scale, against a strong baseline (Qwen2-VL), enabling a 5x reduction in visual token processing during pre-training without sacrificing final performance. The implication is that future MLLM pre-training should not process full-resolution images by default—high resolution can be deferred to the cheaper SFT stage, making pre-training dramatically more compute-efficient. This is a fundamental efficiency insight that changes the default architecture choices for resource-constrained MLLM training.

---

Innovation 5: The Fully-Open Pipeline as a Capability Demonstration—SOTA-Competitive Performance on 0.36% of the Data Budget

The final innovation is the existence proof itself. Prior to Open-Qwen2VL, the dominant narrative—reinforced by the token budgets of closed models (Qwen2-VL: 1.4T, DeepSeek-VL2: 8.1T, InternVL2.5-2B-MPO: 277B)—was that competitive MLLM pre-training requires hundreds of billions to trillions of tokens, and therefore industrial-scale compute. Open-Qwen2VL achieves competitive or superior performance to Qwen2-VL-2B on multiple benchmarks using only 5B tokens (0.36% of Qwen2-VL's budget) and 220 A100-40G GPU hours (roughly one day on 8 GPUs).

This is not a claim that data scale is irrelevant—the model underperforms on OCR-VQA tasks (AI2D: 66.3 vs. 72.3, TextVQA: 63.3 vs. 78.8) where the absence of OCR-specific pre-training data is clearly limiting. Rather, the claim is that data quality can substitute for data quantity along specific capability axes, and that the substitution is dramatic enough to make pre-training research accessible to academic labs.

The significance is sociological as much as technical. This paper is an argument, backed by reproducible evidence, that the academic community should not cede pre-training research to industry. The argument has specific, falsifiable components: (1) aggressive data filtering (selecting 19.9M pairs from ~140M candidates using complementary CLIP and MLLM-based methods) can produce training data whose quality-per-token is orders of magnitude higher than web-scale unfiltered data; (2) sequence packing and visual token compression can make training on this data computationally feasible on academic hardware; (3) the resulting model can be competitive with industrial models trained on 280x more data. If these claims were false—if Open-Qwen2VL significantly underperformed Qwen2-VL across the board—the paper would be a negative result demonstrating that data quality cannot substitute for data quantity. Instead, the mixed results (competitive on general benchmarks, weaker on OCR) precisely characterize where the substitution works and where it fails, providing a nuanced picture that future work can build on.

This is a fundamental reframing of the pre-training research agenda. The question shifts from "how much data can we afford to train on?" to "how do we select the right data such that each training token maximally improves model capability?" The paper's fully-open release enables the community to answer this question through iterative experimentation—trying different filtering thresholds, different data sources, different MLLM-based quality metrics—using the released pipeline as a baseline. This is the paper's most enduring contribution: not the specific model or the specific 56.0 average benchmark score, but the infrastructure for academic pre-training research that makes data filtering science a reproducible, incremental discipline rather than a trade secret.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary pre-training evaluation uses four image-text caption datasets: CCS-CLIP (8.5M pairs from CC3M-CC12M-SBU filtered by BLIP-1's CLIP-based method), DataComp-DFN (15M pairs, the top-15% subset of DataComp-Medium selected by DFN), LAION-CLIP (15M pairs from LAION-400M filtered with CLIP cosine similarity threshold 0.3), and DataComp-MLM-Filter & DFN (19.9M pairs, union of DFN's top-15% and MLM-Filter's SU ≥ 85 selection). For supervised fine-tuning, the paper uses LLaVA-665k [30] for ablation studies and MAmmoTH-VL-10M [16] (single-image subset, 10M examples) for the final model. All final benchmark evaluations are conducted on standard multimodal benchmarks: MMBench-dev [31], SEEDBench-Img-dev [25], MMStar [8], MMMU-val [52], AI2D-test [22], TextVQA-val [42], MathVista-test-mini [32], and POPE [27].

• Base model(s). The model uses Qwen2.5-1.5B-Instruct [43] as the LLM backbone and SigLIP-SO-400M [53] as the frozen vision encoder. Qwen2.5-1.5B-Instruct is chosen because it provides strong language understanding and instruction-following capabilities in a compact 1.5B parameter footprint suitable for academic GPUs. SigLIP-SO-400M is selected over alternatives like OpenAI CLIP for its stronger visual representations from sigmoid-loss pre-training. The total model size is approximately 2B parameters after adding the trainable projector.

• Metrics. All evaluations use accuracy as the primary metric, with exact matching against ground-truth answers via automated grading scripts from the respective benchmarks. For POPE, the paper reports average F1 score (as shown in Figure 2 and Table 3, where POPE values of 79–86 are labeled "Avg-F1" rather than "Accuracy"). For data mixture ablations in Table 3, the paper computes an unweighted average across all eight benchmarks to produce the "Average" metric (e.g., 56.0 for the best mixture). Multi-modal in-context learning experiments in Table 5 average performance over 5 random seeds for the 8-shot setting.

• Baselines. The paper compares against four partially-open 2B-parameter MLLMs in Table 4: InternVL2.5-2B-MPO [45] (277B pre-training tokens, mixed preference optimization), DeepSeekVL-2-Tiny [48] (8.1T pre-training tokens), Qwen2-VL-2B-Instruct [44] (1.4T pre-training tokens, naive dynamic resolution and 2D-Multimodal RoPE), and the LLaVA-665k fine-tuned model as a controlled SFT baseline in the ablation studies. For data mixture ablations (Table 3), the baselines are the three alternative mixture configurations (CCS + DataComp-DFN, CCS + LAION, CCS + DataComp-DFN + LAION). For in-context learning experiments (Table 5), the 0-shot setting serves as the baseline against 8-shot performance.

• Generation budget / compute accounting. Pre-training compute is measured in A100-40G GPU hours (220 for the full pre-training on the best mixture, 48 for LLaVA-665k SFT) and multimodal tokens processed (5B for the full pre-training, compared to 1.4T for Qwen2-VL and 8.1T for DeepSeek-VL2). Training throughput is indirectly measured via the 17% FSDP-vs-DeepSpeed speed comparison. All experiments use 8×A100-40G GPUs with BF16 precision, global batch size 256 for pre-training and 128 for SFT, and exactly one training epoch for both stages. Sequence packing efficiency is measured implicitly by the elimination of padding tokens — no explicit packing efficiency metric (e.g., average bin fullness) is reported.

• Cross-validation / statistical protocol. No cross-validation is used for benchmark evaluations; results are reported as single-run scores on standard test/dev splits. For few-shot in-context learning experiments (Table 5), the paper averages over 5 random seeds for shot selection in the 8-shot setting to account for demonstration order sensitivity. For the SFT scaling experiments (Figure 2), checkpoints are evaluated every 2M training examples (every 15,625 steps at batch size 128), producing 5 data points per benchmark from 2M to 10M. The data mixture ablation in Table 3 uses a single pre-training and SFT run per mixture configuration with no reported variance estimates or statistical significance tests.

Main Quantitative Results

Data Mixture Ablation: MLLM-Based Filtering Provides Complementary Quality to CLIP-Based Filtering

Table 3 reports benchmark performance for MLLMs pre-trained on four different data mixtures and fine-tuned on LLaVA-665k. The headline finding: Mixture 4 (CCS + DataComp-MLM-Filter & DFN, 28.4M pairs) achieves the highest average benchmark score of 56.0, outperforming the three-way CLIP-filtered mixture (Mixture 3, 38.5M pairs, 55.5) by +0.5 points despite using 10.1M fewer training pairs.

The per-benchmark breakdown reveals where the MLM-Filter data contributes most:

• MMBench: Mixture 4 achieves 77.3, compared to 75.9 for Mixture 2 (the next best) and 75.6 for Mixture 1 — a +1.4 to +1.7 point gain.
• MMStar: Mixtures 2, 3, and 4 all score similarly (41.7, 41.7, 41.3) vs. Mixture 1 at 39.6 — a persistent ~2 point gain that appears with any addition of non-CCS data.
• TextVQA: Mixtures 2, 3, and 4 cluster at 57.0–57.4 vs. Mixture 1 at 55.1 — a consistent ~2 point gain.
• MMMU: Performance actually degrades slightly as more data is added: Mixture 1 scores 38.9, while Mixtures 2, 3, and 4 score 37.2, 36.7, and 38.0 respectively — the MLM-Filter data partially recovers this degradation.
• POPE: Mixtures 2 and 4 both score 80.1 vs. Mixture 1 at 79.2 and Mixture 3 at 77.7 — the MLM-Filter data helps avoid the POPE degradation seen in Mixture 3.

The critical comparison is Mixture 1 (CCS + DataComp-DFN, 23.5M, 55.3 average) vs. Mixture 2 (CCS + LAION, 23.5M, 55.4 average): near-identical performance across virtually all benchmarks, confirming the paper's hypothesis that DataComp-DFN and LAION-CLIP select data from similar distributions (both are CLIP-filtered web data). The further comparison of Mixture 3 (adding LAION to Mixture 1, expanding to 38.5M pairs, 55.5 average) vs. Mixture 4 (replacing 10M LAION pairs with 5M MLM-Filter-selected pairs, 28.4M total, 56.0 average) demonstrates that data composition matters more than data volume: fewer total pairs with complementary filtering outperforms more pairs from homogeneous filtering.

SFT Scaling: Most Benchmarks Saturate at 8M Examples, While OCR and Math Continue Improving

Figure 2 tracks benchmark performance across SFT checkpoints from 2M to 10M examples on MAmmoTH-VL-10M, with LLaVA-665k performance shown as a horizontal reference line. The key findings:

Saturating benchmarks: MMBench improves from 76.4 (2M) to 81.8 (8M) then drops to 80.9 (10M); SEEDBench improves from 68.1 (2M) to 73.0 (8M) then drops to 72.5 (10M); MMMU improves from 37.4 (2M) to 41.5 (8M) then drops to 39.8 (10M); MMStar improves from 42.0 (2M) to 51.3 (8M) then drops to 49.7 (10M). The consistent peak at 8M followed by degradation at 10M suggests overfitting or distribution shift in the final 2M SFT examples for general knowledge-based benchmarks.

Non-saturating benchmarks: TextVQA steadily improves from 51.7 (2M) to 63.3 (10M); MathVista steadily improves from 30.5 (2M) to 53.1 (10M). The paper attributes this to "the lack of such pre-training math or OCR data in our curated pre-training caption dataset, making the visual math reasoning and text-based VQA become out-of-distribution tasks" (Section 3.2). The model continues to benefit from more SFT examples in these domains because the pre-training data provides insufficient coverage.

AI2D and POPE show intermediate patterns: AI2D peaks at 8M (68.3) then drops slightly at 10M (66.3); POPE peaks at 4M (85.3) then fluctuates between 83.3–84.4.

The LLaVA-665k baseline (horizontal line) is substantially below the MAmmoTH-VL-10M curves for all benchmarks except the lowest data points, confirming that scaling SFT data from 665k to 10M provides significant gains. The magnitude varies: MMBench gains ~4 points (from ~77 to ~81), MathVista gains ~24 points (from ~29 to ~53), TextVQA gains ~8 points (from ~55 to ~63).

Final Model Comparison: Competitive on General Benchmarks, Weaker on OCR Tasks

Table 4 compares the final Open-Qwen2VL (trained on 5B pre-training tokens + 10M SFT examples) against four 2B-parameter SOTA MLLMs. The results are mixed:

Where Open-Qwen2VL excels:

• MMBench: 80.9, substantially ahead of InternVL2.5-2B-MPO (72.5), DeepSeekVL-2-Tiny (68.3), and Qwen2-VL-2B-Instruct (68.8). This is a +12.1 to +12.6 point advantage over the nearest competitor.
• SEEDBench: 72.5, essentially tied with DeepSeekVL-2-Tiny (72.5), slightly ahead of Qwen2-VL-2B (72.0) and behind InternVL2.5-2B-MPO (73.2).
• MathVista: 53.1, competitive with DeepSeekVL-2-Tiny (54.5) and InternVL2.5-2B-MPO (55.3), ahead of Qwen2-VL-2B (48.0) by +5.1 points.

Where Open-Qwen2VL is competitive but not leading:

• MMStar: 49.7, slightly ahead of Qwen2-VL-2B (46.3) and DeepSeekVL-2-Tiny (49.9, essentially tied), behind InternVL2.5-2B-MPO (54.3).
• MMMU: 39.8, slightly ahead of DeepSeekVL-2-Tiny (39.6), behind Qwen2-VL-2B (41.1) and InternVL2.5-2B-MPO (41.2).
• POPE: 84.4, behind all three competitors (87.6–89.8), with a 5.4-point gap to the best (InternVL2.5-2B-MPO at 89.8).

Where Open-Qwen2VL significantly underperforms:

• AI2D: 66.3 vs. 72.3–75.3 for competitors — a 6.0 to 9.0 point deficit.
• TextVQA: 63.3 vs. 77.2–80.5 for competitors — a 13.9 to 17.2 point deficit.

The paper explicitly attributes the OCR-VQA weakness to the absence of OCR-specific pre-training data (SynthDoG, LAIONCOCO-OCR): "Simply introducing such OCR-related pre-training data will significantly enhance the OCR-VQA task performance of MLLMs" (Section 3.2).

The comparison with Qwen2-VL-2B-Instruct — the most relevant baseline given the shared "Qwen2" naming and similar architecture — shows Open-Qwen2VL outperforming on MMBench (+12.1), SEEDBench (+0.5), MMStar (+3.4), and MathVista (+5.1), underperforming on MMMU (-1.3), AI2D (-6.0), TextVQA (-15.5), and POPE (-3.2). With only 0.36% of Qwen2-VL's pre-training tokens (5B vs. 1.4T), this represents a dramatic data efficiency advantage on the benchmarks where it wins, though the losses on OCR tasks reveal boundaries of the quality-over-quantity strategy.

Multimodal In-Context Learning: Packed Pre-Training Elicits Few-Shot Learning Without Interleaved Data

Table 5 evaluates the pre-trained base model (before any SFT) on five VQA datasets in 0-shot and 8-shot settings. The base model is the one pre-trained on the CCS + DataComp-MLM-Filter & DFN mixture (from Table 3 ablation studies, before any instruction tuning):

• GQA: 0-shot 27.1 → 8-shot 35.4 (+8.3)
• VQA-v2: 0-shot 40.2 → 8-shot 51.8 (+11.6)
• VizWiz: 0-shot 26.1 → 8-shot 31.2 (+5.1)
• OKVQA: 0-shot 24.7 → 8-shot 27.1 (+2.4)
• TextVQA: 0-shot 30.4 → 8-shot 30.6 (+0.2, essentially no improvement)

The 8-shot setting provides meaningful gains on GQA, VQA-v2, and VizWiz (+5.1 to +11.6 points), modest gains on OKVQA (+2.4 points), and no gain on TextVQA (+0.2 points). The paper attributes this emergent capability to the pseudo-interleaved structure created by multimodal sequence packing during pre-training: "The multimodal sequence packing also constructs similar pseudo image-text interleaving data structure" to Flamingo's M3W dataset (Section 4.1).

The failure on TextVQA is consistent with the final model's weak OCR-VQA performance in Table 4 — the base model has no OCR capabilities from pre-training, and in-context demonstrations cannot teach it to read text if it lacks the fundamental visual-text recognition ability.

Ablation Studies and Robustness Checks

Data mixture composition (CLIP-only vs. CLIP+MLLM filtering): Table 3 serves as the primary data mixture ablation. The key finding is that mixing CCS-CLIP, DataComp-DFN, and LAION-CLIP (all CLIP-filtered) yields 55.5 average, while replacing LAION with MLM-Filter data yields 56.0 average with 10.1M fewer total pairs. This validates the complementarity hypothesis — MLLM-based filtering selects for a qualitatively different data distribution that improves model performance even at smaller total data volume. The near-identical performance of Mixture 1 (DataComp-DFN) and Mixture 2 (LAION) at 55.3 vs. 55.4 provides a robustness check confirming that CLIP-based filters from different sources produce similar data distributions.

SFT data scale (2M to 10M): Figure 2 tracks checkpoint performance every 2M SFT examples. The key robustness insight is that most benchmarks peak at 8M and degrade at 10M (MMBench, SEEDBench, MMMU, MMStar, AI2D all show this pattern), while TextVQA and MathVista continue improving through 10M. This suggests that the optimal SFT data scale is task-dependent and that naively scaling to the maximum available SFT data may slightly degrade general benchmark performance while improving specialized capabilities.

Vision encoder frozen vs. trainable during SFT: Table 6 compares SFT on LLaVA-665k with the vision encoder frozen (the default) vs. trainable. With a trainable vision encoder: AI2D improves (56.8 → 57.4), TextVQA improves (57.0 → 57.6), POPE improves (80.1 → 82.3), SEEDBench improves (68.7 → 69.7), MMStar improves slightly (41.3 → 41.4), MathVista improves (28.6 → 29.3), but MMMU degrades (38.0 → 36.1) and MMBench degrades (77.3 → 76.5). The average across all benchmarks is 56.0 (frozen) vs. 56.3 (trainable) — a marginal +0.3 improvement that is likely within noise, with the MMMU degradation (-1.9) as a notable negative result. The paper does not adopt trainable vision encoder for the final model, suggesting the marginal gains do not justify the additional compute cost.

FSDP vs. DeepSpeed-Zero3 training speed: Section 2.4 reports that the FSDP-based implementation "achieves approximately 17% faster for each training step than the DeepSpeed implementation, consistent with findings reported by Karamcheti et al. [21]." This is a practical engineering finding rather than a model quality ablation — it validates that the choice of distributed training framework meaningfully impacts training time on academic hardware.

Pre-training data volume (implicit, via mixture comparison): The comparison between Mixture 3 (38.5M pairs, 55.5 average) and Mixture 4 (28.4M pairs, 56.0 average) in Table 3 serves as an implicit ablation on data volume vs. data composition. More data with homogeneous filtering (38.5M CLIP-filtered pairs) underperforms less data with complementary filtering (28.4M CLIP+MLLM-filtered pairs), demonstrating that data quality/composition can outweigh data quantity by a margin large enough to overcome a ~35% reduction in training pairs.

MLM-Filter threshold (implicit, fixed at 85/100): The paper does not ablate the MLM-Filter SU score threshold. The 85/100 threshold is inherited from prior work [18] and applied without experimentation. This is a notable missing ablation — varying the threshold (e.g., 75, 80, 90, 95) would characterize the quality-vs-quantity tradeoff for MLLM-based filtering and is flagged as a limitation in the critical assessment below.

DFN retained fraction (non-ablatable): The paper explicitly cannot ablate the DFN filtering threshold because "DFN only releases the top-15% curated data rather than the DFN model checkpoint" (Section 2.1). This is a constraint of the open-source landscape rather than an experimental choice, but it means the paper cannot determine whether top-5% or top-30% DFN filtering would perform differently.

Critical Assessment

Claim: "Compute-efficient pre-training is achievable on academic resources (220 GPU hours, 5B tokens)"

What was demonstrated: The paper pre-trained a 2B-parameter MLLM from scratch (frozen vision encoder, trainable projector and LLM) on 19.9M image-text pairs packed into 4096-length sequences in 220 A100-40G GPU hours, and the resulting model after SFT achieves competitive performance on several benchmarks. This is a genuine demonstration that the end-to-end pipeline runs within the stated budget.

Caveats and bounding conditions:

• The 220 GPU hours covers only the pre-training stage. The SFT stage on MAmmoTH-VL-10M is reported as taking additional time (the 48 GPU-hour figure is specifically for LLaVA-665k, not the 10M SFT). The total compute for the final model (pre-training + 10M SFT) is not stated as a single number. Based on the batch size and data scale, 10M SFT examples at batch size 128 on 8 GPUs requires approximately 78,125 steps; at a similar per-step time to pre-training, this would add significant GPU hours that are not explicitly totaled.
• The 220-hour figure is for the best mixture (28.4M pairs). The ablation study required pre-training four separate models (on Mixtures 1–4), totaling 4 × 220 = 880 GPU hours for pre-training alone, plus SFT on each. The paper's claim is about the feasibility of reproducing the final model, not about the cost of the research program that produced it. This is a reasonable distinction, but the ablation cost should be noted when evaluating the paper's accessibility thesis — an academic group starting from scratch would need to run the ablations (or trust the paper's mixture recommendation) to produce a competitive model.
• The 5B token count is directly tied to the specific data mixture and one-epoch training. If a different data source were used or if multiple epochs were needed, the token count and GPU hours would scale proportionally. The paper does not demonstrate that the one-epoch, 5B-token budget generalizes to other data configurations.

Claim: "Data quality (via MLLM-based filtering) can substitute for data quantity"

What was demonstrated: The ablation in Table 3 shows that Mixture 4 (28.4M pairs, CLIP + MLLM filtered) achieves 56.0 average vs. Mixture 3 (38.5M pairs, CLIP-only) at 55.5 average, a +0.5 improvement with 10.1M fewer pairs. This supports the claim that better filtering enables competitive performance with less data.

What was not demonstrated and limits the claim:

• The absolute improvement is small (+0.5 average) and likely within variance. The paper reports no error bars, standard deviations, or confidence intervals for any benchmark scores. With test sets ranging from a few hundred to a few thousand examples, the variance of a single-run accuracy estimate could easily be ±1–2 points for many of these benchmarks. The +0.5 average could be noise rather than signal. The claim would be substantially strengthened by multiple training runs with different random seeds for data shuffling and/or initialization.
• The claim is tested at exactly one data scale. The paper compares 28.4M pairs (high-quality) to 38.5M pairs (medium-quality). What about 50M high-quality pairs? 10M high-quality pairs? Without a data scaling curve plotting performance vs. dataset size for both filtering strategies, the claim "quality substitutes for quantity" is a point estimate rather than a characterized relationship. The paper does not establish whether the quality advantage is constant, diminishing, or growing with scale.
• The complementary filtering claim rests on a single data point. The assertion that CLIP and MLLM-based filters capture different quality signals is supported by Mixture 4 outperforming Mixture 3. But this could also be explained by the MLM-Filter-selected data simply being higher quality along the same axis that CLIP measures — if the MLM-Filter threshold of 85/100 is stricter than the effective CLIP threshold, then the unioned set could be higher quality overall without any "complementarity." The paper does not ablate different MLM-Filter thresholds or compare to a CLIP-filtered set at an equivalent selectivity rate to rule out this alternative explanation.

Claim: "Open-Qwen2VL outperforms the partially-open Qwen2-VL-2B on multiple benchmarks"

What was demonstrated: Table 4 shows Open-Qwen2VL achieving higher scores than Qwen2-VL-2B-Instruct on MMBench (80.9 vs. 68.8), SEEDBench (72.5 vs. 72.0), MMStar (49.7 vs. 46.3), and MathVista (53.1 vs. 48.0). This is a genuine out-performance on these benchmarks.

Caveats:

• The comparison is asymmetric in SFT data. Open-Qwen2VL is fine-tuned on MAmmoTH-VL-10M (10M examples), while Qwen2-VL-2B-Instruct's SFT recipe is not fully disclosed (the Qwen2-VL technical report [44] describes a multi-stage SFT process but exact data scales are not specified). If Open-Qwen2VL's SFT data is larger or higher-quality than Qwen2-VL's, then the pre-training efficiency claim is partially confounded by SFT data differences. The paper's own Figure 2 shows that scaling SFT from LLaVA-665k to 10M examples dramatically improves performance — if Qwen2-VL used less SFT data, the comparison is not purely about pre-training efficiency.
• Open-Qwen2VL underperforms on 4 of 8 benchmarks (MMMU, AI2D, TextVQA, POPE), including a large 15.5-point deficit on TextVQA. The paper frames this as "outperforms on various benchmarks" (Abstract) and "demonstrates competitive performance across benchmarks" (Section 3.2), which is accurate but selective. A reader focused on OCR applications would reasonably conclude that Qwen2-VL-2B is the superior model.
• The Qwen2-VL-2B comparison model may not be the strongest available. Qwen2-VL-2B was released in September 2024; Qwen2.5-VL [6] was released later and may represent improved performance at the same scale. The paper compares against the model that shares the "Qwen2-VL" naming convention but does not establish that this is the strongest available 2B baseline.
• The 0.36% token comparison (5B vs. 1.4T) is eye-catching but potentially misleading. Qwen2-VL's 1.4T tokens include multi-stage pre-training (vision-language alignment, multi-task pre-training) and may include interleaved data, video data, and multi-resolution processing that Open-Qwen2VL does not attempt. Not all tokens are equivalent — Qwen2-VL's token budget covers a broader training curriculum. The 0.36% figure captures the scale difference but not the scope difference.

Claim: "Multimodal sequence packing enhances multi-modal in-context learning"

What was demonstrated: Table 5 shows that the base model (pre-trained with packed sequences, no SFT) achieves +2.4 to +11.6 point improvements in 8-shot vs. 0-shot settings on 4 of 5 VQA datasets. This is evidence that the model can leverage multimodal demonstrations despite never being explicitly trained on interleaved data.

Missing controls:

• No comparison to a model trained without packing. The paper attributes the in-context learning capability to sequence packing, but the experiment has no control group — a model pre-trained on the same data without packing (individual image-text pairs, padded to context length) would establish whether packing is causal. It is possible that the base LLM's text-only in-context learning capabilities transfer to multimodal settings regardless of packing.
• The TextVQA failure (0-shot 30.4, 8-shot 30.6) is unexplained. If in-context learning emerges from packing-induced pseudo-interleaving, why does it fail specifically on TextVQA? The paper does not analyze this null result, which could be informative about the mechanism — perhaps in-context learning only transfers for visual-semantic tasks but not for OCR tasks where the model lacks fundamental text-recognition capability.
• The base model is not compared to other MLLMs on this capability. There is no external baseline for what "good" multimodal in-context learning looks like at this scale. The 8-shot improvements are directionally positive, but without comparison to a known in-context learner (e.g., a Flamingo-style model with explicit interleaved training), the absolute quality of the in-context learning is uncalibrated.

Missing Experiments That Would Strengthen the Paper

• Data scaling curves for CLIP-only vs. CLIP+MLLM filtering. Training models on 5M, 10M, 20M, 40M, and 80M pairs from each filtering strategy would characterize the quality-vs-quantity tradeoff across orders of magnitude, rather than at a single point. This would establish whether the complementarity advantage grows, shrinks, or remains constant with scale — critical for predicting whether the approach would remain competitive at larger budgets.

• MLM-Filter threshold ablation. The 85/100 threshold is a critical hyperparameter that controls the quality-vs-quantity tradeoff for MLLM-based filtering. Training models with thresholds of 70, 75, 80, 85, 90, and 95 would characterize the optimal operating point and reveal whether the current threshold is near-optimal or could be improved.

• Multiple training runs with error bars. Every benchmark score in Tables 3, 4, 5 and Figure 2 is a single run. With test sets of varying sizes and model training subject to random seed effects (data order, initialization), reporting mean ± std over 3–5 runs would allow readers to assess whether the reported differences (especially the +0.5 average in Table 3) exceed noise.

• Ablation on the adaptive average-pooling compression ratio. The paper compresses 729 patches to 144 (a 5× reduction) during pre-training and restores 729 during SFT. Training a model with 256 compressed tokens (3× reduction) or 64 compressed tokens (11× reduction) would characterize the sensitivity of final performance to the compression ratio, revealing how much compression is "safe" before pre-training visual quality is permanently degraded.

• Packing efficiency metrics. The paper claims sequence packing eliminates padding waste but reports no metrics on achieved efficiency — what fraction of tokens in a packed sequence are padding vs. content, average number of image-text pairs per packed sequence, distribution of bin fullness. These metrics are essential for other researchers to implement and evaluate their own packing implementations.

• Comparison to Qwen2-VL-2B fine-tuned on MAmmoTH-VL-10M. To isolate the effect of pre-training efficiency from SFT data scale, fine-tuning Qwen2-VL-2B on the same 10M SFT data would establish a fair comparison. If Qwen2-VL-2B + MAmmoTH-VL-10M matches or exceeds Open-Qwen2VL, then the pre-training efficiency claim would be substantially weakened.

• Ablation on the LLM backbone. The paper uses Qwen2.5-1.5B-Instruct as the LLM backbone. Training with a different 1.5B-class LLM (e.g., SmolLM2-1.7B, a Llama-3.2-1B variant) would test whether the efficiency results are specific to the Qwen family or generalize across LLM architectures.

6. Limitations and Trade-offs

6.1 The Difficulty Estimation Cost Vastly Exceeds the Reported Compute Savings

The assumption or constraint. The compute-optimal framework in Section 3.2 and Figures 4, 8 depends on accurately binning each test question into one of five difficulty quintiles before selecting a strategy. The paper's method requires generating 2048 complete solutions per question and scoring them with the PRM (for predicted difficulty) or checking against ground truth (for oracle difficulty). The authors acknowledge this cost explicitly:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. A single question's difficulty estimation (2048 generations) costs anywhere from 4× to 512× the budget of the actual problem-solving stage (which uses 4–512 generations in the paper's experiments). The headline $4\times$ efficiency gains over best-of-N (e.g., "16 generations matching 2048") are computed after difficulty is already known, without amortizing the estimation cost. In a deployment setting where difficulty is unknown, the total cost would be 2048 + N generations rather than N, and the $4\times$ advantage would invert to a massive disadvantage for all but the hardest problems. The claimed efficiency is therefore an upper bound that is not realizable without a fundamentally cheaper difficulty estimation method.

What evidence exists in the paper. The 2048-sample procedure is described in Section 3.2. The $4\times$ claims appear in Figure 4 (search) and Figure 8 (revisions). Nowhere does the paper include difficulty estimation cost in any total compute accounting, compute-optimal scaling curve, or FLOPs-matched comparison. The acknowledgment of the cost in Section 3.2 is qualitative only.

Mitigation status. Not mitigated. The paper suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8) but develops no such model and provides no evidence that difficulty can be predicted cheaply with the fidelity needed to reproduce the compute-optimal gains. The predicted-difficulty curves in Figures 4 and 8 use the same 2048-sample procedure with PRM scoring substituted for ground-truth checking—this removes the need for labels but does not reduce the generation cost.

---

6.2 Hard Problems Receive Essentially Zero Benefit from Any Test-Time Strategy

The assumption or constraint. All test-time compute methods studied—beam search, best-of-N, lookahead search, sequential revisions, and their compute-optimal combinations—presuppose that the base model's proposal distribution already contains correct solutions at some non-trivial rate. When the base model's pass@1 on a problem class is near zero, no amount of search or revision can find a correct answer, because there is no correct answer in the search space to find.

The consequence. On difficulty bin 5 (the hardest quintile of MATH problems), accuracy remains at 1–3% regardless of method or compute budget (Figure 3, right panel; Figure 7, right panel). The FLOPs-matched comparison in Figure 9 confirms that on hard problems, test-time compute with the smaller model is always worse than simply training a ~14× larger model, across all values of the inference-to-pretraining ratio R. For organizations whose problem distribution skews toward genuinely novel or out-of-distribution reasoning, the entire compute-optimal framework provides no path forward—pretraining remains the only viable route to improved performance.

What evidence exists in the paper. Figure 3 (right) shows bin 5 accuracy for beam search and best-of-N flatlining near 1–3% from 4 to 256 generations. Figure 7 (right) shows bin 5 accuracy at 2–3% across all sequential-to-parallel ratios at 128 generations. Figure 9 shows the bin 5 scaling curve essentially horizontal near 0–5% accuracy with the ~14× larger model's greedy performance (stars) consistently above it. The paper is transparent about this: the Section 7 takeaway explicitly states that test-time compute "provides essentially zero benefit" on the hardest problems.

Mitigation status. Not mitigated—and the paper does not claim otherwise. The failure on hard problems is presented as a fundamental boundary condition: test-time compute amplifies existing capability but does not create it from nothing. This is a clean negative result that is useful for practitioners deciding when to invest in test-time compute vs. pretraining, but it means the approach offers no solution for the regime where improvement is most needed.

---

6.3 All Results Are on a Single Benchmark with a Single Model Family

The assumption or constraint. Every experiment in the paper uses the MATH benchmark (500 test questions from Hendrycks et al., 2021) and PaLM 2-S* as the base model. The PRM is trained on PaLM 2-S* outputs using Monte Carlo rollouts; the revision model is fine-tuned from PaLM 2-S*; the FLOPs-matched comparison uses a second PaLM 2 model with ~14× more parameters. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but provide no cross-model or cross-benchmark validation.

The consequence. Several findings could be model-specific or benchmark-specific in ways that critically affect their generality:

• The difficulty quintile boundaries (pass@1 rates defining bins 1–5) depend on PaLM 2-S*'s specific capability profile on MATH. A different base model—even one with similar aggregate MATH accuracy—might distribute problems differently across difficulty bins, altering which strategies the compute-optimal policy selects and potentially changing the magnitude of the $4\times$ efficiency gain.
• The PRM's quality and over-optimization behavior (the point at which beam search degrades performance, Figure 3 right) depends on PaLM 2-S*'s solution distribution. A model with different error patterns (e.g., more arithmetic errors vs. more reasoning errors) would produce different Monte Carlo rollouts during PRM training, yielding a PRM with different reliability characteristics and a different over-optimization threshold.
• The revision model's behavior depends on PaLM 2-S*'s in-context learning ability when conditioned on incorrect answers. Models with different in-context learning characteristics might benefit more or less from sequential revision, or might show different correct-to-incorrect reversion rates.
• MATH is exclusively competition-level math problems requiring symbolic reasoning. It is unknown whether the difficulty-dependent patterns (beam search hurting easy problems, sequential revisions dominating on easy problems, balanced parallelism on hard problems) transfer to code generation, logical reasoning, scientific QA, or tasks requiring factual recall rather than inference.

What evidence exists in the paper. None—this is a scope limitation, not a measured failure. The paper acknowledges it implicitly by not claiming generality, but the "representative" assertion in Section 4 is unsubstantiated. All figures, tables, and conclusions are conditioned on MATH + PaLM 2-S* without caveats about model or domain dependence in the figure captions or results discussions.

Mitigation status. Not mitigated. The paper does not replicate any experiment on a second model family or a second benchmark. The authors do not claim to have done so, and Section 8 does not identify cross-model or cross-domain validation as future work (though "Extension to other domains" is implied by the general research direction, it is not stated as a specific next step).

---

6.4 The PRM Is Trained on Base Model Outputs and Does Not Transfer to the Revision Model's Distribution

The assumption or constraint. The PRM is trained on solutions sampled from the few-shot prompted base PaLM 2-S* model (Section 5.1). The revision model, however, produces solutions from a different distribution—it conditions on its own previous (incorrect) answers, which changes the statistical properties of generated solutions compared to independent samples from the base model. The paper implicitly assumes the PRM remains a reliable verifier when scoring revision chain outputs.

The consequence. As Figure 15a (Appendix J) demonstrates, the base-LM PRM underperforms a revision-specific ORM when scoring revision model outputs. Specifically, sequential revisions + base-LM PRM achieves approximately 40% at 64 generations vs. sequential revisions + revision ORM at ~42%. This ~2 percentage point gap might seem modest, but it has a compounding effect: the compute-optimal policy in Section 6 selects strategies based on estimated difficulty, which itself relies on PRM scores. If the PRM is miscalibrated on revision outputs, both the difficulty estimation and the final answer selection degrade, potentially altering which sequential-to-parallel ratio is optimal and reducing the realized efficiency gain.

More broadly, this distribution-shift problem is a fundamental tension in the paper's framework: the PRM is the only verifier shared across both search and revision pipelines, yet its training distribution (base model outputs) matches only one of those pipelines. The paper's compute-optimal policy assumes a single verifier works well for both, but the evidence suggests otherwise.

What evidence exists in the paper. Figure 15a (Appendix J) directly measures the degradation: the base-LM PRM (solid blue line, sequential) tracks below the revision-specific ORM (solid orange line, sequential) at all generation budgets. Figure 15b shows that including revision history in the ORM's context provides a small additional improvement (~1–2 points at 64 generations), confirming that revision-model outputs have statistical properties that the base-PRM is not trained to handle. Section 6.2 acknowledges the distribution shift implicitly by training a separate ORM for revisions.

Mitigation status. Partially mitigated—the paper trains a separate ORM specifically on revision model outputs for the revision experiments. However, the compute-optimal policy that combines search and revisions (the vision articulated in Section 8) would need a single verifier that works across both distributions. The paper does not develop such a unified verifier, nor does it characterize how much the compute-optimal gains depend on using a revision-specific ORM vs. the base-PRM.

---

6.5 The Sequential Revision Strategy Imposes a Latency Penalty Not Captured by the Generation-Based Compute Model

The assumption or constraint. The paper measures test-time compute in "generations"—the total number of complete solutions sampled, regardless of whether they are generated in parallel or sequentially. The compute-optimal policy in Figure 8 often allocates a substantial fraction of the budget to sequential revisions (e.g., on easy problems, fully sequential is optimal; on medium problems, a balanced ratio like $2^2$ = 4:1 sequential-to-parallel is optimal). Sequential revisions are inherently serial: each revision depends on the previous one and cannot be generated until the previous step completes.

The consequence. A strategy that allocates 64 generations as 32 sequential × 2 parallel chains takes roughly 16× longer wall-clock time than a strategy that uses 64 parallel samples, assuming sufficient hardware to run all parallel samples simultaneously. For latency-sensitive applications—interactive assistants, real-time decision-making, any user-facing system—the sequential-heavy strategies favored by the compute-optimal policy on easy and medium problems may be unacceptable regardless of their accuracy advantages. The paper's compute-optimal framework optimizes a single objective (accuracy) subject to a single constraint (total generations), ignoring the latency dimension entirely.

What evidence exists in the paper. None—the paper never discusses latency, wall-clock time, or the practical implications of serial vs. parallel execution. The generation-based cost model treats one sequential generation as equivalent to one parallel generation, which is accurate for total FLOPs but not for time-to-answer.

Mitigation status. Not addressed. The paper does not acknowledge this tradeoff, does not report wall-clock times for any strategy, and does not propose latency-aware allocation policies. For practitioners, this means the compute-optimal policy must be manually adjusted for latency constraints: a deployment engineer would need to impose a maximum chain length and re-optimize within that constraint, which the paper provides no guidance for.

---

6.6 The Revision Model Degrades When Trained with On-Policy Data, Limiting the Self-Improvement Loop the Paper Envisions

The assumption or constraint. The revision model in Section 6.1 is trained with an offline data construction procedure: independently sampled correct and incorrect solutions are paired post-hoc, with the incorrect solution in each pair selected to minimize character-level edit distance to the correct solution. This procedure approximates the on-policy multi-turn trajectories used by Qu et al. (2024) but avoids the computational expense of actually generating revision chains during training. The paper envisions (Section 8) an iterative self-improvement loop where the outputs of test-time compute are distilled back into the base model.

The consequence. When the authors attempt to improve the revision model using ReST$^{EM}$ (Singh et al., 2024)—an on-policy reinforcement learning approach that collects training data from the model's own revision trajectories—performance collapses. Appendix K (Figure 16) shows that with the ReST$^{EM}$-trained model, additional sequential revisions substantially hurt performance: at 256 generations, fully sequential performance drops to ~33.5% vs. ~38.5% at the optimal ratio, and the curve is monotonically decreasing with more sequential iterations. This is a reversal of the pattern observed with the offline-trained revision model (Figure 6, left), where more revisions monotonically improved pass@1.

The authors hypothesize that "on-policy data collection in ReST$^{EM}$ exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly." This is a significant liability: the self-improvement loop that the paper identifies as a key future direction (Section 8) depends on the model being able to generate high-quality revision trajectories that can then serve as training data for the next iteration. The ReST$^{EM}$ result suggests that when the model generates its own revision data, the data quality is so poor that fine-tuning on it destroys the revision capability.

What evidence exists in the paper. Figure 16 (Appendix K) provides the direct comparison between the standard revision model and the ReST$^{EM}$-trained variant. The degradation is large and consistent across all sequential-to-parallel ratios. The paper discusses this result briefly in Appendix K, attributing it to spurious correlations but not deeply analyzing the mechanism.

Mitigation status. Not mitigated. The paper does not attempt alternative on-policy training methods, does not diagnose which spurious correlations cause the failure, and does not propose a solution. The offline training procedure works but is a one-shot process—it cannot be iterated to create the self-improvement loop. This means the revision model is effectively a fixed component rather than part of a bootstrapping pipeline, limiting the long-term scaling potential of the approach.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper causes a sociological and methodological shift more than a technical one. Its primary intervention is not a novel architecture or loss function but the demonstration—backed by a fully-reproducible artifact—that competitive multimodal LLM pre-training is possible on academic hardware when data quality is prioritized over data quantity. The magnitude of this shift depends on whether the community adopts the paper's "fully open" standard as a norm rather than an aspiration.

The paper reframes the pre-training research agenda from scale-driven to quality-driven. Prior to Open-Qwen2VL, the dominant narrative—reinforced by the token budgets in Table 4 (DeepSeek-VL2: 8.1T, Qwen2-VL: 1.4T, InternVL2.5-2B-MPO: 277B)—was that competitive MLLM pre-training requires hundreds of billions to trillions of tokens, implicitly excluding academic labs from the research frontier. The paper falsifies this assumption: a 2B-parameter model trained on 5B tokens (0.36% of Qwen2-VL's budget) outperforms Qwen2-VL-2B on MMBench (+12.1 points), SEEDBench (+0.5), MMStar (+3.4), and MathVista (+5.1). This is not a marginal efficiency improvement—it is a demonstration that the relationship between data scale and model capability can be dramatically reshaped by data filtering quality. The implication is that pre-training research should focus at least as much on which data to use as on how much data to use, a reframing that puts data filtering science at the center of the field rather than treating it as pre-processing engineering.

The paper creates an accountability framework for openness claims. Table 1 is a pointed intervention that converts "open source" from a vague, easily-abused label into eight specific, falsifiable criteria: data filtering techniques, sequence packing scripts, pre-training data, pre-training codebase, base model checkpoint, SFT data, and instruct model checkpoint. By evaluating seven SOTA models against these criteria—revealing that only Ideflics qualifies as fully open, while Qwen2-VL is open on just two of eight dimensions—the paper establishes that most "open" MLLMs are actually partially-closed systems where the most scientifically important components (data pipeline, training infrastructure) remain proprietary. This is a fundamental shift in what the community can reasonably demand from model releases. A lab claiming their model is "open" can now be asked: do you release your data filtering scripts, or just the weights? The criteria are specific enough to prevent rhetorical ambiguity.

The paper reconciles a latent tension in the open-source MLLM community. Prior to this work, there was an unspoken assumption that "open weights" constituted openness, and that the data pipeline and training code were optional extras. The paper demonstrates that this assumption has concrete consequences: without access to data filtering techniques, researchers cannot determine whether a model's performance derives from architecture, data quality, or data scale—variables that are fundamentally confounded. The paper's fully-open release provides a baseline against which these confounds can be studied: by releasing the exact data, filtering scripts, and training code, the paper enables the community to run controlled experiments that isolate the effect of each component. This makes pre-training research a falsifiable, incremental science rather than an artisanal practice where each lab's "secret sauce" prevents cross-lab comparison.

The paper redirects research attention toward data filtering as a first-class research problem. The finding that Mixture 4 (CCS + DataComp-MLM-Filter & DFN, 28.4M pairs) outperforms Mixture 3 (CCS + DataComp-DFN + LAION, 38.5M pairs) despite having 10.1M fewer training pairs (Table 3) establishes that data composition can outweigh data volume. This is not a marginal result—it demonstrates that switching from CLIP-only filtering to CLIP+MLLM filtering provides enough of a quality boost to overcome a ~35% reduction in training data. The implication is that improving data filtering (better MLLM-based quality metrics, more sophisticated filter ensembles, dynamic filtering thresholds) should be treated as a core research contribution with as much significance as architectural innovations. This shifts the incentive structure: a paper that improves data filtering by 2 points on average benchmarks is now a meaningful advance, not merely "data cleaning."

Research directions that become more attractive:

• Data filtering science: The paper's demonstration that MLLM-based and CLIP-based filtering are complementary (rather than MLLM-based simply being "better") opens a research program on filter ensembles, multi-dimensional quality metrics, and adaptive filtering thresholds.
• Academic-scale pre-training research: The 220 GPU-hour budget means a lab with a single 8×A100-40G server can run a full pre-training experiment in roughly one day, making iterative experimentation on data composition, packing strategies, and training curricula feasible for PhD students.
• Efficiency techniques for multimodal training: Sequence packing, visual token compression, and low-to-high resolution strategies are validated as effective efficiency multipliers, encouraging further work on training-time optimizations.

Research directions that become less attractive (or are revealed as incomplete):

• Brute-force scaling of pre-training data without quality control: The paper's results suggest that adding more CLIP-filtered data (Mixture 1/2 vs. Mixture 3 in Table 3) provides diminishing returns, while adding complementary-filtered data provides gains even at smaller volumes. This undermines the "just collect more data" approach that dominates industrial MLLM development.
• Purely architectural innovations without data pipeline transparency: The paper's openness framework implies that an architectural contribution evaluated on an opaque, unreleased data pipeline is not fully reproducible, reducing its scientific value relative to contributions evaluated on open data.

Follow-Up Research This Work Enables

Characterizing the CLIP-vs-MLLM filtering complementarity through controlled ablation of the filtering threshold. The paper's central claim—that MLLM-based and CLIP-based filtering capture complementary quality signals—rests on a single data point: Mixture 4 (union of DFN and MLM-Filter data) outperforms Mixture 3 (all CLIP-filtered data) by +0.5 average. But the MLM-Filter threshold of 85/100 is inherited from prior work without ablation, and the DFN threshold of top-15% is fixed by the released data. A strong follow-up would train models on datasets filtered at multiple thresholds for both methods (e.g., MLM-Filter SU scores at 70, 75, 80, 85, 90, 95; DFN-style CLIP filtering at top-5%, top-10%, top-15%, top-20%, top-30%) and measure whether the complementarity advantage grows or shrinks with threshold strictness. If the advantage is largest at intermediate thresholds—where CLIP and MLLM filters disagree most—this would validate the distinct-distribution hypothesis. If the advantage disappears when both filters are set to equivalent selectivity rates, it would suggest the "complementarity" is actually just stricter filtering.

Testing whether multimodal in-context learning from packed pre-training transfers to non-VQA tasks and scales with sequence length. Table 5 demonstrates 8-shot multimodal in-context learning on VQA datasets as an emergent property of packed pre-training, but the evaluation is limited to five VQA benchmarks and a single 8-shot setting. A follow-up would test: (1) whether the in-context learning extends to more complex tasks (image captioning, visual reasoning, OCR) where the model from Table 5 shows no improvement on TextVQA; (2) whether performance scales with the number of shots (0, 2, 4, 8, 16, 32) to characterize the in-context learning curve; (3) whether packing with different numbers of pairs per sequence (achievable by varying the image token count or context length) produces different in-context learning capabilities; and (4) most critically, whether a model trained without packing on the same data exhibits any in-context learning, establishing causality. The null result—no in-context learning without packing—would confirm the paper's attribution; the alternative—in-context learning emerges regardless—would indicate that the base LLM's text-only in-context learning transfers to multimodal settings independent of packing, a finding with different implications for training design.

Evaluating the data efficiency ceiling: how far can quality-over-quantity be pushed before the model hits capability walls? The paper demonstrates competitive performance at 5B tokens (0.36% of Qwen2-VL's budget), but this is a single point on the quality-vs-quantity curve. A systematic follow-up would train models on progressively larger datasets selected by the same pipeline—10M, 20M, 50M, 100M, 500M, 1B pairs—and measure how performance scales. This would reveal whether the quality advantage is constant (each MLLM-filtered token is worth ~280× a Qwen2-VL token, and the gap persists), diminishing (the advantage shrinks as models saturate on available data quality), or actually growing (the quality advantage compounds with scale because cleaner data enables more efficient learning). The OCR-VQA deficit on AI2D and TextVQA (Table 4) suggests that some capabilities require specific data types (OCR text in images) that quality filtering alone cannot provide—a scaling curve would reveal whether these deficits persist or close with more data. This experiment would directly inform whether the paper's approach is a viable path to 7B- and 13B-scale MLLMs or whether it hits a quality ceiling that requires industrial-scale data diversity.

Combining the Open-Qwen2VL pipeline with OCR-specific pre-training data to close the OCR-VQA gap. The paper explicitly attributes the 15.5-point TextVQA deficit (63.3 vs. 78.8 for Qwen2-VL-2B) and 6.0-point AI2D deficit (66.3 vs. 72.3) to the absence of OCR-specific pre-training data like SynthDoG [23] or LAIONCOCO-OCR [38]. A natural follow-up would integrate these datasets into the pre-training mixture, apply the same filtering pipeline (CLIP + MLM-Filter), and measure whether OCR-VQA performance reaches competitive levels without sacrificing general benchmark performance. The key question is whether OCR data can be added in proportion (e.g., 5M OCR pairs among 25M total) or whether it requires a dedicated OCR pre-training phase, which would increase the token budget. If a modest OCR data addition closes most of the gap while maintaining the 220 GPU-hour budget, it would validate the paper's claim that "simply introducing such OCR-related pre-training data will significantly enhance the OCR-VQA task performance" and demonstrate the composability of the filtering pipeline across data types.

Stress-testing the pipeline on a non-Qwen LLM backbone to establish architecture independence. Every experiment uses Qwen2.5-1.5B-Instruct as the LLM backbone. The paper's efficiency claims—that data quality enables competitive performance at 5B tokens—could be partially confounded with Qwen2.5's specific pre-training quality or its multimodal transferability. A strong stress-test would replicate the entire pipeline (same data, same filtering, same packing, same training recipe) with a different 1.5B-class LLM—for example, SmolLM2-1.7B, Llama-3.2-1B, or a Gemma-2-2B variant—and compare the resulting multimodal performance. If the pipeline produces similar relative gains (competitive with the corresponding LLM family's 2B-class MLLM at 0.36% of tokens), it validates that the efficiency is driven by data quality rather than Qwen-specific architecture properties. If performance degrades substantially, it would reveal that the paper's results are partially architecture-dependent, narrowing the generality of the efficiency claims. The null result (substantial degradation) would be equally informative, identifying that compute-efficient multimodal pre-training requires LLM backbones with specific pre-training characteristics.

Practical Applications and Downstream Use Cases

Academic research on multimodal pre-training data curation. The paper's primary downstream use case is as a research infrastructure. An academic lab with a single 8×A100-40G server can now run a full end-to-end MLLM pre-training experiment—from data download through filtering, packing, pre-training, and SFT—in roughly one week (220 GPU-hours for pre-training + estimated ~100–200 GPU-hours for 10M SFT). This enables PhD students to conduct controlled experiments on data filtering strategies: they can swap in new MLLM-based quality metrics (beyond MLM-Filter's SU score), test different dataset union strategies, experiment with progressive filtering (applying stricter thresholds as training progresses), or study the interaction between pre-training data composition and downstream task specialization. Prior to this release, such experiments required either industrial-scale compute (making them inaccessible to academia) or were limited to fine-tuning existing models (confounding pre-training effects with base model properties). The released WebDataset-format data, FSDP training codebase, and documented hyperparameters make this a turnkey research platform rather than a theoretical possibility.

Cost-efficient fine-tuning of specialized multimodal models for niche domains. Organizations that need multimodal models for domain-specific applications (medical image understanding, engineering diagram interpretation, legal document analysis) can use the Open-Qwen2VL base model as a starting point and fine-tune on domain-specific data, avoiding the compute cost of pre-training from scratch. The base model's 5B-token pre-training on general image-text data provides visual-semantic alignment at a fraction of the cost of training from Qwen2-VL or DeepSeek-VL2 checkpoints (which required 1.4T and 8.1T tokens respectively). A medical imaging startup, for example, could take the released Open-Qwen2VL base model, fine-tune it on a curated dataset of radiology images with reports, and deploy a specialized model within a modest GPU budget—something that would be computationally prohibitive if they needed to first pre-train a general multimodal model on billions of tokens. The paper's demonstrated success with MAmmoTH-VL-10M fine-tuning (Figure 2) provides a template: specialized SFT on domain data can substantially improve task-specific capabilities even when the pre-training data lacks domain coverage.

Teaching and reproducibility in graduate-level ML courses. The paper's fully-open release—training codebase, data filtering details, sequence packing scripts, all pre-training and SFT data, and model checkpoints—makes it an ideal teaching tool for courses on multimodal machine learning. Students can reproduce the full training pipeline on a university cluster, run the ablation studies from Table 3 to understand how data composition affects model performance, modify the MLM-Filter threshold to observe the quality-vs-quantity tradeoff, or extend the model to new modalities (adding a video encoder following the same projector design). The 220 GPU-hour budget means a semester-long course with access to a shared 8-GPU server could support multiple student projects, each running a complete pre-training experiment. This was not possible with prior SOTA MLLMs, where the closed data pipelines and industrial-scale compute requirements made reproduction infeasible. The paper's explicit "fully open" standard also serves as a pedagogical benchmark: students can evaluate new model releases against the eight criteria from Table 1, learning to critically assess claims of openness in the ML literature.