Multimodal2025~133 minarXiv:2501.14818

Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models

Zhiqi Li, Guo Chen, Shilong Liu +24Jan 20, 2025cs.CV

🎯 Pitch

A systematic data strategy—not model scale—lets a 9B-parameter model outperform GPT-4V on ChartQA, OCRBench, and MathVista. The paper reveals the full post-training data recipe that open-weight frontier models typically keep secret, showing how 'diversity first, then quality' collection and balanced data packing drive these gains.

1. Executive Summary

This paper develops a comprehensive data-centric post-training strategy for building frontier vision-language models (VLMs), resulting in the Eagle 2 model family. Using the Qwen2.5-7B-Instruct backbone and a tiled mixture of vision encoders (SigLIP + ConvNeXt) on diverse multimodal benchmarks, the work systematically introduces a series of data strategies — "diversity first, then quality" collection from 180+ sources (passive gathering and proactive error-analysis-driven searching), data filtering (rule-based removal of mismatched QA pairs, irrelevant image-question pairs, and formatting errors), advanced subset selection (K-means clustering on SSCD image embeddings to balance data distributions rather than random sampling), data augmentation (CoT generation, rule-based QA expansion, short-to-long answer conversion), and data formatting (removing template artifacts and appending task-specific instructions) — alongside a three-stage training recipe (Stage-1 connector alignment, Stage-1.5 full-model pretraining on 21.6M diverse samples, Stage-2 high-quality fine-tuning on 4.6M samples) and a balanced-aware data packing algorithm that distributes long and short samples evenly across training packs. Eagle2-9B achieves state-of-the-art results, notably outperforming InternVL2-8B and MiniCPM-V2.6 on all 14 evaluated benchmarks and surpassing GPT-4V on ChartQA (86.4 vs. 78.5), OCRBench (868 vs. 656), and MathVista (63.8 vs. 54.7), while matching models with up to 70B parameters, establishing that careful data orchestration — rather than model scale alone — can close the gap to proprietary frontier VLMs when the data strategy systematically addresses both diversity and quality across visual instruction domains.

2. Context and Motivation

The Core Problem: Frontier VLMs Exist, but Their Data Recipes Are Opaque

The paper addresses a specific and urgent problem in the vision-language model community: the critical details of post-training data strategy remain largely hidden, even for models whose weights are publicly available. This is not a theoretical gap — it is a practical barrier to reproduction, improvement, and scientific understanding.

To understand why this matters, we need to recognize the three-tier landscape of VLM transparency that the paper maps out in Section 1:

Tier 1 — Commercially closed-source models (GPT-4V/4o, Claude, Gemini-1.5-Pro). These models release neither weights nor training details. The community can only speculate about their data strategies, architectures, and training recipes. They serve as performance targets but offer no transferable knowledge.

Tier 2 — Weight-available frontier models (Qwen2-VL, InternVL2, Llama 3.1, MiniCPM-V). These models release model weights — a significant step toward reproducibility — but their technical documentation is "often insufficient for reproduction." As the paper notes, "the technical details provided by these models are often insufficient for reproduction." The weights tell you what the model learned, but not how it was taught. This is the critical gap: you can use these models for inference or fine-tuning, but you cannot replicate their training pipeline, understand which data decisions mattered most, or systematically improve upon them.

Tier 3 — Fully open-source models (Cambrian-1, LLaVA family). These models release weights, training code, and data recipes. They are transparent and reproducible. However, as the paper observes, "most of the open source models still lag behind their frontier counterparts." The transparency exists, but the performance does not match Tier 1 or Tier 2.

This creates a frustrating situation for researchers and practitioners: the best-performing models (Tiers 1 and 2) are black boxes with respect to their most crucial ingredient — data. The transparent models (Tier 3) are behind in capability. The paper's central question, stated directly in Section 1, is: "What could help the community to develop more competitive open-source frontier VLMs?"

Why This Problem Is Important

The significance of this gap extends beyond academic curiosity. The paper identifies several reasons why hidden data strategies are a substantial bottleneck for the field:

1. Data is the decisive factor, not architecture. The paper states this position clearly in Section 1.1: "Assuming the same pre-trained LLM backbone, we posit that data is the most decisive factor to obtain high-quality models." With the rapid convergence of VLM architectures — the paper notes in Section 1.2 that "simple MLP connector is still the most popular choice" and that "model structure is no longer a primary factor driving performance differences among models" — the differentiating factor between models is increasingly what data they are trained on and how that data is curated, sequenced, and formatted. When this information is hidden, the community cannot learn from the most successful models.

2. Resource constraints limit exploration. Fully open-source models often operate under computing and data constraints that prevent them from exploring the full range of data sources that Tier 1 and Tier 2 models can access. The paper notes in Section 1.1 that "various constraints such as computing resources may limit the study on more dataset sources, despite their intention to make the data recipe publicly available." This means that even well-intentioned open-source efforts cannot fully explore the data strategy design space, making the lack of shared knowledge from frontier models even more costly.

3. Data strategy drives real capability gains. The paper's Figure 2 illustrates this dramatically: the baseline model (Cambrian-1 data with LLaVA's two-stage training) achieves an average score of 58.8 across 13 benchmarks. Through systematic data strategy improvements alone — adding diverse data sources, introducing Stage-1.5, filtering, formatting, advanced selection, augmentation, re-updating Stage-1.5, and adding the mixture of vision encoders — the average score climbs to 73.5. That is a 14.7 percentage point improvement, almost entirely driven by data and training recipe decisions rather than model scaling or architectural innovation. This empirical trajectory makes the case that data strategy is not a marginal optimization but the primary lever for VLM capability.

4. The field needs "fish," not "fishing lessons." The paper explicitly frames its contribution around this metaphor in Section 1.4: "we aim to 'teach readers to fish than to give them a fish,' by sharing our data strategies, model design and training recipe, detailing the iterative development process than merely displaying the final benchmark results." The hidden-data problem means the community currently receives only fish (model weights) without learning how to fish (the data curation and training methodology).

Where Prior Approaches Fall Short

The paper identifies specific limitations in prior work along several dimensions that motivate its contributions:

Limitation 1: Data strategy documentation is shallow where it matters most.

Prior fully open-source models like Cambrian-1 and the LLaVA family provide extensive documentation, but the paper argues these are insufficient for reaching frontier performance. The gap is not just about scale — Cambrian-1 used 7M training samples — but about the systematicity of the data strategy. Prior work tends to describe what data was used but not why specific curation decisions were made, how data quality was assessed and maintained, or what iterative process led to the final data composition. The paper cites the observation that "LLaVA-OneVision-72B still ranks slightly behind InternVL2-40B despite having a stronger LLM backbone" (Section 1), suggesting that data strategy — not model scale — explains the performance gap, and that prior documentation has not adequately captured the nuances of effective data strategy.

Limitation 2: Two-stage training is insufficient for rapid data iteration.

The paper identifies a practical bottleneck: the standard LLaVA two-stage training recipe (connector alignment followed by full-model SFT) makes it difficult to iterate on data strategy. As they explain in Section 2.3: "While efficient, this approach proved unsuitable for quick SFT data updates, as the expanding SFT data makes it harder to track the impact of new data and reduces the experimental efficiency." When all data is loaded into a single SFT stage, adding new data sources can interact in unexpected ways with existing data, making ablation studies unreliable and iteration slow. The paper observes that "the main limitation of the two-stage strategy is the lack of robust pre-training," which motivated their introduction of Stage-1.5.

Limitation 3: Existing data selection is crude and distribution-unaware.

Prior work on data subset selection, such as Cambrian-1, used simple approaches: limit each data source to a maximum number of samples (e.g., 350K) and randomly select from the remainder. The paper identifies two problems with this. First, "random selection is suboptimal" because it does not ensure balanced representation across the subtypes within a dataset — for example, in chart data, "histogram samples are more frequent than other types like line charts or pie charts, and random sampling wouldn't ensure balance across these types." Second, maximum-threshold approaches treat all datasets uniformly regardless of their original size, quality, and diversity, whereas the paper argues that "data source diversity and distribution determine the sample quantity" and that "auto-labeled sources are featured by larger sizes, but often contain errors and lack diversity" while "manually labeled datasets are often smaller."

Limitation 4: Public datasets contain systematic quality issues.

The paper documents specific, recurring quality problems in commonly used public VLM datasets that go beyond typical noise. These include:

Mismatched question-answer pairs (Figure 5a): The answer does not correspond to the question asked, often because synthetic generation pipelines lose coherence.
Irrelevant image-question pairs (Figure 5b): The question references content not present in the paired image, an error from dataset assembly.
Repeated texts (Figure 5c): Answers contain loops of identical text, a known failure mode of LLM-based data generation.
Numeric formatting issues (Figure 6): Answers contain excessive decimal precision (e.g., "5336666.666666667") that cannot be extracted from the image, revealing that generation pipelines often do not validate numerical consistency.

The paper argues that these issues are not merely cosmetic — they degrade model training in measurable ways, as demonstrated by the 45-point improvement on OCRBench after data formatting and filtering (Table 6). Prior work had not systematically characterized these failure modes or provided concrete filtering strategies.

Limitation 5: Data diversity is underexplored beyond common benchmarks.

The paper observes that existing open-source models often cluster around a relatively narrow set of well-known academic datasets. "For fully open-source models, various constraints such as computing resources may limit the study on more dataset sources, despite their intention to make the data recipe publicly available." This creates a "bucket effect" where model capabilities are bounded by the diversity of the training data. The paper's aggressive collection from 180+ sources — spanning chart understanding, handwriting recognition, medical imaging, landmark recognition, meme analysis, music sheets, and more — represents a deliberate expansion of data diversity that prior open-source efforts had not systematically pursued or documented.

Limitation 6: Data packing introduces training artifacts.

The paper identifies a subtle but consequential issue with data packing, a standard technique for accelerating training by concatenating shorter sequences. The "naive greedy knapsack algorithm" used in frameworks like LLaMA-Factory tends to group samples of similar lengths together, producing packs with "uneven length distributions" (Figure 9). This causes imbalanced loss weighting between long and short samples during training. The paper notes that existing packing methods like SPFHP prioritize packing efficiency over balanced length distribution, and that this choice "is not desirable to model training." This is a concrete example of how implementation details in training infrastructure — not just data content — affect final model quality.

How This Paper Positions Itself

The paper positions itself at the intersection of two communities — weight-available frontier models and fully open-source models — aiming to combine the performance of the former with the transparency of the latter. Its positioning can be understood along several axes:

1. Data strategy as the primary contribution, not model architecture.

The paper explicitly states in Section 1.1 that "data is the most decisive factor to obtain high-quality models." Unlike work that advances new architectures (Q-Former, Hybrid-Attention) or scaling recipes, Eagle 2's primary contribution is a systematic, documented, reproducible data strategy. The architecture — tiled mixture of vision encoders — is an important component but is presented as an enabler of the data strategy rather than the main novelty. The paper's Figure 2, which tracks performance improvements through successive data and training recipe changes, reinforces this positioning: most of the gain comes from data decisions, with the vision encoder change adding a final increment.

2. "Diversity first, then quality" as organizing principle.

The paper introduces and operationalizes a specific philosophical stance: collect as broadly as possible first, then filter and refine. This is not the only possible approach — one could prioritize quality from the start and accept narrower coverage — but the paper argues this ordering is essential: "A model's capability is strongly correlated with the diversity of data" (Section 2.2). The similarity score metric (Equation 1) provides a quantitative tool for assessing whether new data sources add meaningful diversity rather than redundancy, grounded in embedding-space comparisons of both images (via SSCD) and text (via all-mpnet-base-v2).

3. Iterative, error-analysis-driven development rather than single-pass curation.

The paper emphasizes that data strategy is not a one-time design decision but an iterative process. Figure 3 illustrates a cycle: collect data, run experiments, identify weaknesses through error analysis, search for targeted new data to address those weaknesses, and repeat. This positions data development as an engineering discipline with feedback loops, similar to how software testing drives development. The three-stage training recipe (Stage-1 → Stage-1.5 → Stage-2) creates an infrastructure for this iteration: Stage-2 enables rapid experimentation on high-quality data (6 hours on 256 H100s, per Appendix A), conclusions from Stage-2 inform updates to Stage-1.5, and the updated Stage-1.5 provides a stronger foundation for subsequent Stage-2 experiments. As the paper describes it: "Stage-1.5 accelerates Stage-2 and Stage-2 gives Stage-1.5 feedback" (Figure 8).

4. Full transparency as a deliberate choice, not an afterthought.

The paper shares not just the final dataset composition but the decision process: the similarity score metric for assessing data overlap, the K-means clustering approach for balanced subset selection, the specific filters applied to remove low-quality samples, the data augmentation prompts and procedures, and even the data packing algorithm with code. This level of detail is unusual for a model that achieves frontier-level performance and represents a philosophical commitment to transparency that the paper explicitly frames as its purpose: "Through sharing the journey of exploration, we aim to 'teach readers to fish than to give them a fish'" (Section 1.4).

5. Acknowledging tradeoffs and limitations in data strategy.

The paper does not present its data strategy as universally optimal. It acknowledges specific tensions: the cost of difficulty estimation (though in a different context), the trade-off between data diversity and experimental iteration speed, the fact that dataset-level ablation is impractical ("we avoid performing ablation for each dataset individually," Section 2.2), and the need to balance detailed instructions against generalization (overusing "Please answer yes or no" can impair the model's ability to answer correctly without such prompts, Section 2.2). This honesty about tradeoffs strengthens the paper's credibility and distinguishes it from work that presents only polished final results.

6. Complementing, not competing with, the scaling laws approach.

The paper builds on the Cambrian-1 philosophy of systematic data exploration but extends it significantly. Cambrian-1 established the importance of data ratios and source balancing for VLMs. Eagle 2 adds: (a) a much broader data collection from 180+ sources versus Cambrian-1's 7M samples from fewer sources, (b) detailed filtering and formatting strategies that Cambrian-1 did not systematically address, (c) K-means-based subset selection replacing random selection, (d) a three-stage training recipe replacing the standard two-stage approach, and (e) explicit data augmentation pipelines. The paper also connects to the broader VLM training literature (LLaVA-OneVision's multi-stage training, InternVL2's progressive scaling) but argues that these works "tend to use more knowledge-related data, such as captioning data" in intermediate stages, whereas Eagle 2's Stage-1.5 "adds all data sources intended for visual instruction" — a distinctly different data allocation philosophy.

In summary, the paper addresses a specific pain point in the VLM community: the opacity of data strategies in frontier models, which prevents reproduction, systematic improvement, and scientific understanding. By sharing the complete iterative development process of a competitive model, the paper aims to demonstrate that open documentation of data strategy is both possible and essential for advancing the field — and that careful data orchestration, rather than model scale or architectural novelty alone, is the primary driver of VLM capability.

3. Technical Approach

3.1 Reader Orientation

The paper builds a comprehensive post-training data pipeline and training recipe for vision-language models — essentially, a systematic methodology for collecting, filtering, selecting, augmenting, and formatting multimodal training data, then feeding it through a carefully designed three-stage training process. The system solves the problem of how to transform a large pool of heterogeneous, often-noisy public datasets into a high-quality training corpus that produces frontier-level VLM performance, with the core insight being that data strategy decisions (what to include, what to filter, how to balance, how to format) are more consequential than architectural innovations when starting from a capable pretrained LLM backbone.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components that interact in a specific pipeline:

Data Collection Engine — gathers candidate training samples from 180+ public sources through passive monitoring of arXiv and HuggingFace and proactive, error-analysis-driven targeted searching. This component produces a raw, unfiltered candidate pool organized by category (captioning, OCR, charts, math, science, etc.).
Data Refinement Pipeline — applies four sequential operations to the raw pool: (a) rule-based filtering to remove low-quality samples (mismatched QA pairs, irrelevant image-question pairs, repeated text, excessive decimal precision), (b) subset selection via K-means clustering on SSCD image embeddings to ensure balanced representation across data subtypes, (c) data augmentation using auxiliary VLMs and rule-based methods to add chain-of-thought reasoning, generate new QA pairs, and expand short answers into detailed responses, and (d) data formatting to standardize task representations and remove harmful template artifacts.
Similarity Scoring Module — computes a quantitative diversity metric between candidate new data sources and the existing data pool using image embeddings (SSCD) and text embeddings (all-mpnet-base-v2). This guides decisions about whether a new source adds meaningful diversity or merely duplicates existing coverage.
Three-Stage Training Recipe — executes the actual model training in a specific sequence: Stage-1 aligns the vision encoder outputs with the LLM's embedding space by training only the MLP connector on 1.2M alignment samples; Stage-1.5 performs full-model pretraining on a large, diverse 21.6M-sample dataset to build broad multimodal capabilities; Stage-2 fine-tunes the full model on a carefully curated 4.6M high-quality subset to specialize for target tasks.
Balanced-Aware Data Packing — concatenates short training samples into fixed-length packs for training efficiency, but using a modified greedy knapsack algorithm that distributes samples to achieve uniform length distributions across packs rather than the naive approach that segregates long and short samples.

Information flows through these components in a feedback loop: raw data is collected → refined through filtering, selection, augmentation, and formatting → packed into balanced training sequences → used to train through three stages → the trained model's error patterns inform targeted data collection for the next cycle. Stage-2 provides rapid feedback (6 hours on 256 H100 GPUs) that guides updates to Stage-1.5, which in turn provides a stronger foundation for subsequent Stage-2 experiments.

3.3 Roadmap for the Deep Dive

First, the baseline configuration and initial conditions — what the authors started from before applying their innovations — because understanding the starting point is essential for appreciating the magnitude and nature of each subsequent improvement.
Second, the data collection philosophy and mechanism — the "diversity first" principle, the two collection strategies (passive and proactive), and the similarity score metric — because this defines the raw material that all downstream operations work with.
Third, the data filtering strategies — the specific quality problems identified, the rule-based detection methods, and the rationale for each — because filtering is the first refinement step and has a disproportionate impact on downstream model behavior.
Fourth, the subset selection approach — K-means clustering on image embeddings, the principles governing subset size determination, and why random selection is insufficient — because this addresses the "every data comes with a price" tradeoff.
Fifth, the three data augmentation methods — CoT generation, rule-based QA expansion, and short-to-long answer conversion — because these add training signal that the original annotations lack.
Sixth, data formatting — the principle of "same task, similar format; different tasks, clearly distinct formats" and the concrete example of LaTeX template removal — because formatting errors can catastrophically distort model behavior.
Seventh, the three-stage training recipe — the rationale for adding Stage-1.5, the data allocation strategy across stages, the "large wheel drives small wheel" feedback dynamic, and the specific hyperparameters — because this is where the refined data meets the model.
Eighth, the balanced-aware data packing algorithm — the problem with naive greedy knapsack, the modified algorithm with pseudocode, and why uniform pack lengths matter — because this is a subtle infrastructure decision with measurable performance impact.
Ninth, the tiled mixture of vision encoders — the SigLIP + ConvNeXt configuration, the PixelShuffle downsampling, and the channel concatenation — because this architectural choice complements the data strategy.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a data engineering and training methodology paper whose core idea is that frontier VLM performance can be achieved through systematic, documented data strategy applied to a capable pretrained LLM backbone, without architectural novelty or model scaling beyond the single-digit billions of parameters.

Baseline Configuration

Before introducing any innovations, the paper establishes a concrete starting point (Table 1). The baseline uses the SigLIP-400M vision encoder, a simple MLP connector (no Q-Former, no hybrid attention, no cross-attention — just a learned linear projection), and the Qwen2.5-7B-Instruct LLM backbone. Images are processed at dynamic resolutions through tiling: the base resolution is $448 \times \{(1,1), (1,2), (2,2), ..., (1,12)\}$ , meaning images are split into tiles according to aspect ratios that produce between 1 and 12 tiles total. The training data comes from the Cambrian-1 dataset, but with certain low-quality subsets removed: ShareGPT-4V, GPT-77K, and Data-Engine-161K are discarded, resulting in a filtered subset of 5.2M samples. Training follows the standard LLaVA two-stage recipe: Stage-1 trains only the MLP connector on 1.2M alignment samples (from ALLaVA), and Stage-2 trains the full model on the 5.2M SFT data.

Why start here? The authors chose Cambrian-1 as the baseline because it represents the state of open-source VLM data efforts at the time, and because its documented data composition provides a transparent starting point. Removing known low-quality subsets establishes a clean foundation. The choice of Qwen2.5-7B-Instruct as the LLM backbone reflects the paper's goal of showing that data strategy — not model scale — drives performance: this is a 7B-parameter model, and the paper demonstrates it can match models with up to 70B parameters.

Data Collection: The "Diversity First" Principle

The paper's data collection strategy is organized around a single maxim stated in Section 2.2: "diversity is the key." The authors operationalize this through two complementary mechanisms and a quantitative filtering metric.

Passive gathering means monitoring the latest related datasets from arXiv manuscripts and HuggingFace Datasets and adding them to a candidate list. This is ongoing surveillance, not a one-time scrape — as new datasets appear, they are evaluated for inclusion. The paper notes that this approach is enabled by using "pre-organized dataset collections" from prior work (specifically citing Cambrian-1, LLaVA-OneVision, and the Idefics3/Molmo data compilation referenced as [151]) to speed up preparation, but with "careful inspection to prevent issues like test data leakage." The paper explicitly flags one discovered leakage: "The test split of AI2D is used in Cambrian-1 training data" (Section 2.2 footnote).

Proactive searching addresses what the paper calls the "bucket effect": the tendency for models to develop uneven capabilities because training data is unevenly distributed across domains. After each update to the data pool, the authors generate error analysis on model outputs, identify specific weaknesses (e.g., poor performance on handwriting recognition, weak spatial reasoning, or inability to read music notation), and perform targeted searches for new datasets that specifically address those gaps. This is a feedback-driven data acquisition loop: train → evaluate → identify errors → search for data that covers those error categories → retrain.

A critical operational detail: the authors explicitly state that they do NOT perform per-dataset ablation studies. The paper explains: "To reduce training costs, we avoid performing ablation for each dataset individually. Instead, datasets with similar domains are added in batches to the data pool when meeting the following criteria: (1) Maintaining overall accuracy without noticeable regression for every considered benchmark, and (2) Introducing meaningful diversity to the current domains." This batch-addition strategy is a pragmatic response to the combinatorics problem: with 180+ sources, individual ablations would be computationally prohibitive. The trade-off is that the contribution of any single dataset cannot be precisely quantified, but the paper argues that maintaining overall accuracy while expanding diversity is sufficient evidence of positive contribution.

Non-QA data conversion: the paper also describes a process for converting public non-QA datasets into VQA format. Datasets like Google Landmark (instance-level image recognition) contain images but no question-answer pairs. The authors convert these "into VQA data using specific rules or auto-labeling tools." The rules-based approach is used for classification datasets (converting labels into multiple-choice questions, as shown in Appendix Figure B for the SpatialScene dataset), while auto-labeling tools (third-party VLMs) generate image descriptions for datasets containing only images, which become captioning training samples.

The resulting data pool spans 180+ sources across categories including Captioning & Knowledge, Mathematics, Science, Chart & Table, Naive OCR, OCR QA, Grounding & Counting, General VQA, and Text-only. The complete enumeration of every dataset with exact sample counts appears in Appendix Tables B through J, providing full transparency about the data composition.

Similarity Score: Quantifying Data Diversity

To operationalize the diversity principle, the paper introduces a quantitative metric called the Similarity Score (Equation 1) that measures how much a new data source overlaps with the existing data pool:

$S_k = \frac{1}{N} \sum_{i=1}^{N} \max_{1 \leq j \leq M_k} \left( \text{Sim}(I_i, I_j) \times \text{Sim}(T_i, T_j) \right)$

where:

$S_k$ is the similarity score for a new data source within category $k$
$N$ is the number of samples in the new data source
$M_k$ is the number of samples in the existing pool within the same category $k$
$i$ indexes samples in the new source, $j$ indexes samples in the existing pool
$\text{Sim}(I_i, I_j)$ is the cosine similarity between the SSCD image embeddings of the new sample $i$ and existing sample $j$
$\text{Sim}(T_i, T_j)$ is the cosine similarity between the all-mpnet-base-v2 text embeddings of the new sample $i$ and existing sample $j$

What it computes: for each sample in the new data source, the metric finds the most similar sample in the existing pool (within the same category) by computing the product of image similarity and text similarity, then averages these maximum similarities over all $N$ new samples. The product means that both the image AND the text must be similar for a high score — a dataset with similar images but different questions would get a low score because the text similarity term would be small. Image embeddings come from SSCD (a self-supervised copy detection model designed to be sensitive to near-duplicate images), and text embeddings come from all-mpnet-base-v2 (a sentence transformer model that maps semantically similar questions to nearby vectors).

Why this form: the product of image and text similarity ensures that only samples that are similar in BOTH modalities contribute to a high score. A dataset with identical images but completely different questions (e.g., the same COCO images used for different VQA tasks) would have high image similarity but low text similarity, correctly yielding a low product. Conversely, a dataset with similar question types but on entirely different images would also yield a low product. The $\max$ operation over the existing pool means the metric captures the existence of any highly similar counterpart, not just average similarity — if even one sample in the existing pool is very similar to a new sample, the new sample's contribution to $S_k$ will be high. The category restriction (only comparing within the same category) reflects the practical observation that "inter-category similarity is generally low" — a chart dataset will not be similar to a captioning dataset regardless of embeddings.

How it is used: the paper provides a concrete example in Appendix B.1. When the existing pool is Cambrian-7M and three new datasets are evaluated: UReader (a collection containing DocQA, InfoQA, and ChartQA — all already present in Cambrian) gets a similarity score of 0.45 with a maximum value of 0.95, indicating substantial overlap. COCO-Colors (using COCO images but with new color-related questions) gets 0.10, indicating the images are familiar but the task is new. Textbooks-QA (internal data) gets 0.02, indicating essentially no overlap. The paper's operational threshold: "sources with a score below 0.3 are considered different from the existing data pool. Data with a score above 0.3 may also be retained or removed based on specific considerations." This metric enables rapid screening of candidate datasets without manual inspection of every sample.

Data Filtering: Removing Systematic Quality Failures

The paper identifies specific, recurring categories of low-quality samples in public datasets and develops rule-based filters to remove them. The filtering philosophy is captured in the aphorism: "A rotten apple spoils the barrel" — a small number of bad samples can disproportionately degrade model behavior.

Category 1: Mismatched question-answer pairs. In Figure 5a, a sample from ShareGPT4o shows an image of a man in mid-20th-century attire, but the answer describes "the photograph of the man appears to be from the mid-20th century due to the black-and-white color and classic attire" — an answer that seems related to the image but is actually responding to a request to "describe the situation in the picture as clearly as possible" by generating an inference-based analysis that does not directly answer the question. The question-answer coherence has broken. The paper notes that such errors "often present characteristics making them distinguishable for removal through rule-based filtering" because the synthetic generation process leaves detectable patterns.

Category 2: Irrelevant image-question pairs. In Figure 5b, from the Cambrian-1 Data Engine, an image shows a mosquito anatomy diagram, but the question asks "What is the structural organization of the heart of the mosquito Anopheles gambiae, and how does it differ from vertebrate hearts?" — a question that cannot be answered from the visible image content. The image and question were incorrectly paired during dataset assembly.

Category 3: Repeated texts. In Figure 5c, from ShareGPT-4V, the answer repeats the phrase "The image does not contain any elements that suggest a particular style or genre" multiple times in a loop. This is a known failure mode of LLM-generated responses where the model gets stuck in a repetitive pattern, and it creates training data that would teach the model to produce degenerate outputs.

Category 4: Numeric formatting issues. Figure 6 shows two examples: an answer of "5336666.666666667" for a question about net bilateral aid flow, and "0.2700346484855259" for a ratio calculation. The paper explains: "Public datasets often do not rigorously handle numerical precision, resulting in high decimal precision impossible to directly extract from the image." These answers come from computational pipelines that perform division without rounding, producing floating-point artifacts that no human would write and that cannot be verified from visual inspection. Training on such data teaches the model to produce nonsensical precision.

Additional filtering rules described in Appendix B.2 include keyword-based detection of refusal responses: samples where the "answer" is some variant of "I cannot answer this" or "Sorry, I cannot." While appropriate for safety-sensitive questions, such responses also appear in contexts where they make no sense — for example, a text-only question asking "Can you help me describe this image" receiving "Sorry, I cannot" as the answer. The paper applies keyword filtering to exclude these samples.

Why rule-based filtering rather than learned filtering: the paper argues that "most low-quality data are generated from synthesis, they often present characteristics making them distinguishable for removal through rule-based filtering." The errors are systematic and detectable through pattern matching (repeated n-grams, excessive decimal places, known refusal templates), making learned filtering unnecessary. Rule-based filtering is also deterministic and auditable — each removed sample can be justified by a specific, inspectable rule, which aligns with the paper's transparency goals.

Impact: Table 6 shows that after applying data formatting and filtering to an 8.2M-sample dataset, performance improves on 8 out of 14 benchmarks, including a dramatic 45-point gain on OCRBench (from 798 to 843). The paper interprets this as evidence that "carefully reviewing and utilizing existing data, as well as exercising caution with publicly available data" is essential and that the gains from cleaning existing data can exceed those from collecting new data.

Subset Selection: K-Means Clustering for Balanced Representation

Once the data pool is filtered, the next challenge is selecting which samples to actually use for training. The paper's philosophy is captured in the statement: "every data comes with a price" — larger datasets are not necessarily better if the additional samples are redundant or low-quality.

The problem with random selection: the paper identifies a concrete failure mode. In chart data, "histogram samples are more frequent than other types like line charts or pie charts, and random sampling wouldn't ensure balance across these types." Random selection preserves the original distribution, which means over-represented subtypes dominate the training data and under-represented subtypes may be essentially absent. This matters because "data source diversity and distribution determine the sample quantity" — the goal is not to faithfully reproduce the source distribution but to create a training distribution that covers all subtypes.

Subset quantity determination: the paper describes two principles that interact to determine how many samples to select from each source. First, "auto-labeled sources are featured by larger sizes, but often contain errors and lack diversity" — automatically generated datasets (from model outputs, web scraping, or rule-based synthesis) tend to be large but have correlated errors and limited variety per sample. Second, "manually labeled datasets are often smaller" but each sample carries more information because humans provide diverse, accurate annotations. This leads to a general rule: "datasets with larger original sizes are generally applied with smaller sampling ratios." For Stage-2 data, "the average size per source is around 20K, with the largest subset VisualWebInstruct having 263K samples." The paper provides specific heuristics in Appendix B.3: "For datasets with fewer than 20,000 samples, we do not perform subset selection. If we perform subset selection, we remove at least half of the data. For datasets originally exceeding 100,000 samples, in most cases, we limit the subset to no more than 50,000 samples."

K-means clustering selection: once the target subset size is determined, the paper uses unsupervised K-means clustering on SSCD image embeddings to select which specific samples to include. The procedure is as follows:

For a given dataset, compute SSCD image embeddings for all samples. SSCD is a self-supervised copy detection model that produces embeddings sensitive to visual content but trained to be invariant to transformations like cropping, resizing, and compression — meaning visually similar images (same chart type, same document layout) map to nearby points.
Run K-means clustering on these embeddings with $K$ equal to the target subset size. This partitions the dataset into $K$ clusters of visually similar samples.
From each cluster, select one representative sample (typically the sample closest to the cluster centroid). This ensures that the selected subset covers the full diversity of visual patterns in the original dataset, with each cluster contributing one representative.

The paper notes a limitation: "K-means using SSCD image embeddings performs poorly on natural scene images." For such datasets, clustering does not improve over random selection because natural images are highly varied and SSCD embeddings — designed for copy detection — do not capture the semantic similarity structure needed for meaningful clustering. However, the method "excels with mathematical, medical, and document-based data" where visual patterns are more structured (chart types, document layouts, medical image modalities).

Impact: Table 6 shows that switching from naive subset selection (random sampling with maximum thresholds, 8.2M samples) to advanced subset selection (K-means clustering, 4.6M samples) improves the average score from 70.6 to 71.8 — better performance with nearly half the data. This validates the principle that balanced representation matters more than total sample count.

Data Augmentation: Mining Rich Information from Images

The paper describes three data augmentation strategies, all designed to "mine the rich information from input images that is not fully present in the existing QA annotations." These transformations add training signal without requiring new image collection.

Augmentation 1: Adding Chain-of-Thought (CoT) explanations. Many existing VQA datasets provide only short final answers — a letter choice, a number, or a brief phrase — without the reasoning steps that lead to those answers. The paper uses "existing SOTA VLMs" (third-party models, not the model being trained) to rewrite answers into detailed step-by-step explanations. Figure 7a shows an example: an original question about planets with rings with answer "2" gets expanded into a multi-step reasoning process ("Identify Planets with Rings," "List Planets with Observed Rings," "Counting Planets with Rings," "Mathematical Principle," "Final Answer"). The prompt used for this generation is provided in Listing 1 of the Appendix and specifies a rigid structure: "Problem Restatement → Step-by-Step Process (Explicit Steps with Justifications) → Mathematical Principles → Final Answer."

A critical quality control step: the generated CoT answers can contain reasoning errors even when the final answer is correct, because the auxiliary VLM may produce a plausible-sounding but incorrect chain of reasoning. To address this, the paper uses "another LLM to compare the generated answers with the original answers, filtering out some erroneous sample" — a verification step where a separate model checks whether the generated CoT answer's final conclusion matches the known correct answer. The prompt for this verification is provided in Listing 2 and simply asks: "Please evaluate if the correctness of my answer based on the provided question and the correct answer... Please only return 'True' if my answer is correct, or 'False' if it is incorrect."

The effectiveness of CoT augmentation is validated through a targeted experiment (Appendix Table M): models trained without CoT data show no improvement or even degradation when prompted with "Think it step-by-step" at inference time — they produce CoT-formatted output but the reasoning quality does not improve. Models trained with the augmented CoT data show a boost when the CoT prompt is added (MathVista-Mini scores improve from 61.0/60.5 to 63.2/63.5 depending on the evaluation prompt). This demonstrates that the CoT training data teaches the model to generate useful reasoning chains, not just to mimic the formatting.

Augmentation 2: Rule-based QA generation. For datasets with rich visual content but limited question diversity, the paper applies rule-based transformations to generate new QA pairs. Figure 7b shows an example from chart data: given an original question "What is the maximum number of word titles?" with answer "16 from the Ferrari," a rule generates a new question "What is the mean number of word titles?" with answer "4.2" by computing the mean of the values visible in the chart. This is not an LLM generation — it is a deterministic computation that extracts values from the chart's underlying data and computes statistics. The paper also describes OCR-specific rule-based augmentation (Appendix Figure C): taking datasets like IIIT5K and WordArt that contain images of text, and generating questions like "Is 'life' written on the image? No, it is 'life'" (testing whether the model correctly reads the text) or "What is the word in the right-top corner?" (testing spatial localization of text). These are automatically generated by extracting bounding box coordinates from the original dataset annotations and constructing template questions.

Augmentation 3: Expanding short answers into longer responses. Some datasets (VQAv2, GQA, VSR) provide very short answers — single words or brief phrases. The paper uses an LLM prompted with "Given the question {question}. The original answer is {answer}. Please reply with a more specific answer based on the existing answer, as detailed as possible" (Listing 3) to expand these into detailed, fluent responses. This teaches the model to produce natural language explanations rather than one-word answers, which the paper argues prevents the model from becoming "an 'answering machine' that is used to giving short answers."

Impact: Table 6 shows that adding data augmentation to the training pipeline improves MMMU (from 52.5 to 53.6), MathVista (from 60.5 to 63.5 — a 3.0 point gain), and ChartQA (from 84.1 to 85.1), with the average score increasing from 71.8 to 72.1.

Data Formatting: Standardizing Task Representations

The paper identifies data formatting as a crucial but often-overlooked step in data preparation, organized around the principle: "same task, similar format; different tasks, clearly distinct formats." Two examples illustrate the importance.

Example 1: LaTeX template removal (Table 3). Two datasets contain samples for the task "Extract LaTeX from image." One sample has the LaTeX formula S=4\pi R^2=\frac{9}{2}\pi — clean, just the formula. Another sample has \begin{align*}F_c=m_{J}\frac{c^2}{R}\end{align*} — the formula wrapped in an align* equation environment. The paper reports that "even with limited use of such data, the model consistently outputs this fixed template" — the model learns to always wrap LaTeX output in \begin{align*}...\end{align*}, even when the task does not call for it, because the template was present in the training data. "After removing the fixed equation environment, the model returns to normal behavior, leading to a significant improvement on OCRBench." This is a concrete example of how a seemingly minor formatting inconsistency can cause a model to learn a spurious pattern that degrades performance on a major benchmark.

Example 2: Task-specific instruction appending. The paper describes a nuanced trade-off in adding instructions to training samples. Appending specific instructions based on the response type — for example, "Provide a short answer" for samples with brief responses — helps prevent the model from over-generalizing to always producing verbose answers. It also helps the model learn to modulate its response style based on explicit instructions, which is valuable for controllable generation. However, the paper warns: "overusing this can also hinder generalization. For instance, adding 'Please answer yes or no' to every yes-or-no question may impair the model's ability to answer correctly without such prompt during inference." The model can become dependent on the instructional cue and fail when it is absent. This reveals a design tension: instructions improve controllability but can reduce robustness if applied too broadly. The paper navigates this by applying instruction appending selectively rather than uniformly.

Impact: Table 6 shows that data formatting and filtering together improve OCRBench from 798 to 843 (45 points) and the average score from 70.6 to 71.2. The OCRBench improvement is attributed primarily to the LaTeX template removal described above.

Three-Stage Training Recipe

The paper introduces a three-stage training strategy that departs from the standard LLaVA two-stage recipe in both structure and data allocation philosophy.

Stage-1: Vision-Language Alignment (1.2M samples, connector only). This stage trains only the MLP connector that bridges the vision encoder outputs to the LLM's embedding space, using 1.2M samples from ALLaVA. The data consists of simple image-caption pairs and basic VQA that teach the model to map visual features into the LLM's token embedding space. All other model parameters (vision encoder, LLM) are frozen. The batch size is 1024, learning rate is $2 \times 10^{-4}$ , and maximum sequence length is 4096 tokens. This stage takes approximately 2.5 hours on 128 H100 GPUs for the 9B model.

Why Stage-1 is connector-only: training only the connector prevents catastrophic forgetting in the pretrained LLM while the visual representations are being aligned. The connector has relatively few parameters (since it's just an MLP projection), so it can learn the mapping quickly without disrupting the LLM's language capabilities.

Stage-1.5: Diverse Full-Model Pretraining (21.6M samples, full model). This is the paper's key innovation over the standard two-stage recipe. Stage-1.5 trains the full model (vision encoder, connector, and LLM) on a large-scale diverse dataset of 21.6M samples. Critically, this stage includes all data sources intended for visual instruction — not just captioning or knowledge data as in prior work. As stated in Section 2.3: "Other works tend to use more knowledge-related data, such as captioning data, at this stage. In this work, we add all data sources intended for visual instruction to Stage-1.5, simultaneously introducing several other datasets as shown in Tab. 2b."

The additional datasets in Tab. 2b are larger-scale sources that are included in Stage-1.5 but not Stage-2: CC3M (2.2M captioning samples), TextCaps (110K captioning samples with text reading), ShareGPT-4V (767K captioning samples), DenseFusion-1M (1.17M captioning samples), Object365 (1.18M object detection samples converted to grounding format), and OpenMathInstruct (1.58M text-only math samples). These datasets are valuable for building broad capabilities but may be too large, redundant, or noisy for the high-quality Stage-2 fine-tuning.

The data distribution for Stage-1.5 (Figure 4a) shows Text-Only data at 22.0%, Captioning & Knowledge at 22.2%, General VQA at 13.8%, OCR QA at 8.6%, with the remaining categories at 4-7% each. The paper makes a point of ensuring "text-only data occupy over 20%" to prevent catastrophic forgetting of language capabilities during multimodal training.

Training hyperparameters for Stage-1.5: batch size 1024, learning rate $2 \times 10^{-5}$ (or $4 \times 10^{-5}$ for smaller models), maximum sequence length 8192 tokens. This stage takes approximately 28 hours on 256 H100 GPUs for the 9B model.

Why Stage-1.5 uses all instruction data: the paper argues that limiting intermediate pretraining to captioning/knowledge data — as done in LLaVA-OneVision and InternVL2 — creates a gap between what the model learns in the intermediate stage and what it needs to do in final SFT. By including all visual instruction data in Stage-1.5, the model builds capabilities across all task types (OCR, chart reading, math reasoning, spatial reasoning, etc.) in a single stage, providing a more comprehensive foundation for Stage-2 fine-tuning. Table 6 shows that the Stage-1.5 checkpoint by itself achieves an average score of 69.7 — competitive with some fully trained models — and that Stage-2 training on top of Stage-1.5 improves this to 70.9.

Stage-2: High-Quality Fine-Tuning (4.6M samples, full model). This stage continues training the full model on a carefully curated, high-quality subset of 4.6M samples. The data distribution shifts significantly from Stage-1.5 (Figure 4b): Text-Only increases to 26.7%, Captioning & Knowledge drops to 14.2% (compared to 22.2% in Stage-1.5), and OCR QA increases to 12.2%. The reduction in captioning data is due to "concerns about the overly monotonous instructions" — captioning tasks all follow the same "describe this image" pattern, and too many such samples can bias the model toward producing descriptive rather than analytical responses.

Training hyperparameters for Stage-2: batch size 256 (reduced from 1024 to accommodate longer sequences), learning rate $2 \times 10^{-5}$ (or $4 \times 10^{-5}$ for smaller models), maximum sequence length 16384 tokens. This stage takes approximately 6 hours on 256 H100 GPUs for the 9B model. The shorter training time compared to Stage-1.5 is intentional: it enables rapid iteration, with the paper noting that "in actual development, we rarely iterate the Stage-1 model. Usually, we iterate Stage-1.5 once after iterating Stage-2 > 10 times" (Appendix A).

The "large wheel drives small wheel" feedback dynamic. Figure 8 illustrates the relationship between Stage-1.5 and Stage-2. Stage-1.5 ("Large-scale Data, Excellent Foundation, Slow Iteration") provides a strong base model that Stage-2 ("Small-scale Data, Rapid Iteration") can rapidly fine-tune. The conclusions from Stage-2 experiments — which data filtering, formatting, augmentation, and selection strategies work — can then be applied to update Stage-1.5 data, creating a virtuous cycle. Table 6 confirms this: "Re-Update Stage 1.5" (applying the data strategies discovered in Stage-2 to refresh the Stage-1.5 data) improves the average score from 72.1 to 72.4, with particularly notable gains on ChartQA (85.1 → 85.9), MMVet (60.1 → 61.3), and MathVista (63.5 → 65.2).

Why three stages instead of two: the paper argues that the standard two-stage recipe (connector alignment + full-model SFT) is unsuitable when SFT data is expanding and evolving. As they explain: "the expanding SFT data makes it harder to track the impact of new data and reduces the experimental efficiency." In a two-stage setup, all data goes into a single SFT stage, making it difficult to isolate the effects of new data sources. The three-stage setup decouples broad capability building (Stage-1.5) from targeted fine-tuning (Stage-2), allowing rapid experimentation in Stage-2 while maintaining a stable foundation from Stage-1.5.

Cross-stage data flow and the test leakage concern: the paper acknowledges one specific data contamination issue: "The test split of AI2D is used in Cambrian-1 training data" (Section 2.2). This means some AI2D test images may have been seen during training, which could inflate AI2D benchmark scores. The paper flags this but does not quantify its impact on reported results.

Balanced-Aware Data Packing

Data packing is a standard technique for accelerating LLM training: instead of processing one sample at a time (which wastes computation on padding tokens when samples are shorter than the maximum sequence length), multiple short samples are concatenated into a single training sequence that fills the maximum length. The paper identifies a problem with the standard implementation and proposes a solution.

The problem with naive greedy knapsack. The paper begins by examining the packing algorithm used in LLaMA-Factory, a popular open-source fine-tuning framework. The "naive greedy knapsack algorithm" works as follows, using the terminology from Figure 10: given a list of sample lengths (in tokens) and a maximum pack length $L$ , sort samples in descending order, then greedily fill knapsacks (packs) with the largest remaining samples that fit. This produces the behavior visible in Figure 9a: the first knapsack contains the longest samples, the last knapsack contains the shortest samples, and intermediate knapsacks have varying lengths. The result is "uneven length distributions" — some packs are nearly full while others are nearly empty.

Why uneven pack lengths matter: the paper argues that this pattern is "not desirable to model training" because it creates imbalanced loss weighting. In standard language model training with packing, the loss is typically averaged over all tokens in a pack. Packs that are nearly full contribute more tokens to the gradient update than packs that are mostly padding (if padding loss is masked) or packs that are short. This means the model effectively sees long samples more often (they appear in nearly-full packs) and short samples less often (they appear in mostly-empty packs), creating an unintended bias in the training distribution.

The balanced greedy knapsack algorithm. The paper proposes a modified algorithm, presented as pseudocode in Figure 10, with the full Python implementation in Listing 4. The algorithm works as follows:

Sort samples in descending order by length.
Compute the minimum number of knapsacks needed: min_knapsacks = (total_length + L - 1) // L + delta, where delta is a small redundancy parameter (set to 20 in Listing 4) to avoid creating new knapsacks mid-loop.
Initialize min_knapsacks empty knapsacks, each with length 0.
Iterate through samples from longest to shortest. For each sample, attempt to place it in the current knapsack (indexed by ks_index). If it fits, add it and increment the sample index. If it does NOT fit, create a new empty knapsack and set ks_index to the index of the current shortest knapsack (using argmin(knapsack_lengths)).
Continue until all samples are placed.

The key difference from the naive algorithm lies in Step 4: when a sample does not fit in the current knapsack, the naive algorithm would increment ks_index to the next knapsack (meaning long samples all go into early knapsacks). The balanced algorithm resets ks_index to the shortest knapsack, which means the next sample (still relatively long) will be placed in a mostly-empty pack rather than starting a new pack. This distributes long and short samples across all packs, producing the uniform length distribution visible in Figure 9b.

Why delta redundancy is needed: without the delta parameter, the initial allocation of min_knapsacks would be exactly the ceiling of total_length/L, and the algorithm might need to create additional knapsacks during the loop when samples don't fit. The delta parameter pre-allocates slightly more knapsacks than strictly needed, avoiding the need to dynamically add knapsacks and ensuring the balancing logic works correctly from the start.

Impact: Table N in the Appendix compares the naive greedy packing against the balanced packing method. The balanced method improves average score from 72.8 to 73.5, with particularly notable gains on ChartQA (84.7 → 86.4), InfoVQA (76.5 → 77.2), MMMU (54.7 → 56.1), HallusionBench (47.9 → 49.3), and MathVista (61.6 → 63.8). These are substantial improvements from a change that does not affect what data the model sees — only the order in which it is presented.

Runtime impact: the paper notes that packing accelerates training by 2-3 times compared to processing samples individually with padding. The balanced method achieves this acceleration while also improving final model quality, making it a pure win over both no-packing and naive-packing baselines.

Tiled Mixture of Vision Encoders (MoVE)

The paper's architectural contribution is the combination of two previously separate design ideas — dynamic image tiling and mixture of vision encoders — into a single unified design.

Individual components. The vision backbone consists of two encoders:

SigLIP-400M: a Vision Transformer trained with sigmoid loss for contrastive language-image pretraining. It processes images at $448 \times 448$ resolution and produces a feature grid. After applying PixelShuffle (a $2\times$ spatial downsampling operation that rearranges channel dimensions into spatial dimensions, converting a $H \times W \times 4C$ tensor into a $2H \times 2W \times C$ tensor — but used here in reverse to downsample), the features are reduced to $16 \times 16$ spatial dimensions.
ConvNeXt-XXLarge: a convolutional neural network based on the ConvNeXt architecture. It processes images at $512 \times 512$ resolution and naturally produces a $16 \times 16$ feature grid (since ConvNeXt uses $32\times$ total downsampling).

Why different input resolutions produce matching output sizes: SigLIP at $448 \times 448$ with PixelShuffle downsampling by $2\times$ yields a $16 \times 16$ grid (since $448 / 28 = 16$ at the standard ViT patch size). ConvNeXt at $512 \times 512$ with $32\times$ total downsampling yields $16 \times 16$ (since $512 / 32 = 16$ ). The mismatch in input resolutions is intentional: SigLIP is designed for contrastive pretraining at moderate resolution, while ConvNeXt benefits from higher input resolution for fine-grained feature extraction. The PixelShuffle operation aligns the spatial dimensions post-hoc.

Tiling for dynamic resolution: following InternVL-1.5, images are split into tiles according to a set of allowed aspect ratios: ${(i, j) | i, j \in \mathbb{Z}^+, i \times j \leq 12}$ . This means an image can be split into, for example, $1 \times 1$ (one tile, no splitting), $1 \times 2$ (two horizontal tiles), $2 \times 2$ (four tiles), up to a maximum of 12 tiles total (e.g., $3 \times 4$ , $2 \times 6$ , $1 \times 12$ ). Each tile is independently encoded by both vision encoders, and the features from all tiles are concatenated to form the visual representation. This allows the model to process arbitrarily high-resolution images without being constrained by a fixed input size.

Channel concatenation of MoVE: for each image tile, both encoders produce a $16 \times 16$ feature grid. These grids are concatenated along the channel dimension: the SigLIP features (with however many channels after PixelShuffle) and the ConvNeXt features (with their channel count) are stacked, creating a richer per-position representation that combines the complementary strengths of both encoders. The SigLIP encoder, trained with contrastive language-image objectives, captures semantic and alignment-relevant features. The ConvNeXt encoder, trained with supervised or self-supervised objectives on visual recognition tasks, captures fine-grained spatial and textural features. The concatenation gives the subsequent MLP connector access to both types of information.

Total token count: the paper specifies the vision token count as (i × j + 1) × 256 tokens, where $i \times j$ is the number of tiles and the $+1$ accounts for a thumbnail/global view of the entire image (a standard practice in tiling approaches). Each tile contributes 256 tokens ( $16 \times 16 = 256$ ). The total tokens can thus range from $(1+1) \times 256 = 512$ for a single-tile image to $(12+1) \times 256 = 3328$ for a maximally-tiled image.

Design rationale (why tiled MoVE instead of SigLIP alone or ConvNeXt alone): Table 6 shows that adding the mixture of vision encoders improves performance on 12 out of 14 benchmarks, with the largest gains on document, chart, and OCR tasks: DocVQA (91.3 → 92.6), ChartQA (85.9 → 86.4), InfoVQA (73.3 → 77.2, a 3.9-point gain), TextVQA (81.9 → 83.0), OCRBench (842 → 868, a 26-point gain). The paper states that this "clearly demonstrates that the mixture of vision encoders significantly enhances the model's understanding to visual spaces." The tiling component is already present in the baseline (Eagle and InternVL-1.5 both use tiling with a single encoder), so the gains are specifically attributable to combining the two encoder types.

Why this design rather than alternatives: the paper positions tiled MoVE as inheriting "the benefits from both worlds" — the robust semantic perception from SigLIP (trained on billions of image-text pairs) and the fine-grained visual feature extraction from ConvNeXt (trained on large-scale visual recognition). Compared to using a single larger encoder (e.g., a larger ViT), the mixture approach provides complementary feature types (transformers capture long-range dependencies differently than convolutions). Compared to other MoVE designs in prior work (Sphinx, Prismer, Prismatic VLMs, Mini-Gemini, LLaVA-HR), the paper claims to be "the first to explore the tiled mixture of vision encoder (MoVE) design" — combining both the multi-encoder and tiling innovations that had previously been studied separately.

Training detail for small models: for the 1B and 2B model variants (using Qwen2.5-0.5B and Qwen2.5-1.5B LLM backbones), the paper uses only the SigLIP encoder, not the full MoVE configuration. This is noted in Table 4: "For small scale model with 0.5/1.5B LLM, we only use SigLIP as visual encoder." The paper does not elaborate on why, but the likely reason is that the smaller LLMs have limited capacity to utilize the richer visual representations from two encoders, making the additional computational cost unjustified. Additionally, the learning rate for smaller models in Stage-1.5 and Stage-2 is increased to $4 \times 10^{-5}$ (from $2 \times 10^{-5}$ for the 9B model), suggesting that smaller models benefit from more aggressive updates.

4. Key Insights and Innovations

Innovation 1: Data Strategy as a First-Class, Documentable Research Artifact — Not an Opaque Implementation Detail

The paper's most distinctive intellectual contribution is not any single data technique, but rather the elevation of post-training data strategy from an opaque, under-documented implementation detail to a systematically studied, fully transparent research artifact that can be shared, critiqued, and reproduced. This is a conceptual move about what constitutes a scientific contribution in VLM research, not merely a practical recipe.

To appreciate what makes this distinctive, consider the dominant norms in the field. Tier 1 models (GPT-4V/4o, Claude, Gemini-1.5-Pro) treat data strategy as proprietary trade secrets — the community knows nothing about their training data composition, filtering criteria, or augmentation pipelines. Tier 2 models (Qwen2-VL, InternVL2, Llama 3.1) release weights but provide only high-level sketches of their data approach. As the paper notes, "the technical details provided by these models are often insufficient for reproduction." Tier 3 models (Cambrian-1, LLaVA) release data recipes but lag in performance, creating an implicit assumption in the community that frontier performance requires hidden, industrial-scale data operations that cannot be openly documented.

Eagle 2 breaks this assumption. The paper demonstrates that a model achieving frontier-competitive performance — matching or exceeding closed-source models like GPT-4V on ChartQA (86.4 vs. 78.5), OCRBench (868 vs. 656), and MathVista (63.8 vs. 54.7), and outperforming all weight-available models at the 7-8B scale — can be built with a fully documented data strategy. The level of disclosure is unusually granular: exact sample counts per dataset across two training stages (Appendix Tables B-J), the specific CoT augmentation prompts (Listing 1), the correctness verification prompt (Listing 2), the balanced packing algorithm with Python code (Figure 10, Listing 4), the similarity score metric with operational thresholds (Equation 1 and the 0.3 cutoff), and the K-means clustering procedure for subset selection.

This is not just about transparency for its own sake. The paper argues that hidden data strategies are a bottleneck for the entire field: "This limitation often affects their capability compared to models that can access and utilize a wider range of data sources." When data strategy is opaque, every research group must independently rediscover effective practices through costly trial and error. By making data strategy a first-class, documentable artifact, the paper enables the community to build cumulatively — future work can understand why specific decisions were made, challenge specific choices, and propose improvements to the documented pipeline rather than starting from scratch.

The paper's framing of this contribution is explicit: "we aim to 'teach readers to fish than to give them a fish,' by sharing our data strategies, model design and training recipe, detailing the iterative development process than merely displaying the final benchmark results." This is a statement about scientific norms, not just model performance. The contribution is the methodology of systematic, documented data development — the fishing lesson — not merely the resulting model weights — the fish. The iterative development narrative in Section 3.1 (Table 5 → Table 6 → Table 7, tracking changes from baseline 58.8 to final 73.5) is itself the primary research output, showing how each data decision contributed to the final result in a way that others can replicate and extend.

Significance beyond performance: this reframes what "open-source" means for VLMs. The paper argues that releasing model weights alone (Tier 2) is insufficient for scientific progress because weights are opaque artifacts — they reveal what was learned but not how it was taught. Full openness requires disclosing the data strategy, the training recipe, and the development process. Eagle 2 demonstrates that this level of disclosure is compatible with frontier performance, countering any argument that competitive pressure requires secrecy. This is a normative claim about research practice, supported by the empirical result that systematic, documented data development produces competitive models.

Comparison to prior work: Cambrian-1 made important strides toward data transparency by analyzing data ratios and source balancing, but its data strategy documentation was primarily about what data was used rather than why specific curation decisions were made or how the iterative development process unfolded. LLaVA-OneVision disclosed its multi-stage training but did not provide the granular per-dataset accounting, the filtering rules, or the augmentation procedures that Eagle 2 shares. The paper's Appendix alone — with its exhaustive per-dataset sample counts across two stages, similarity score examples, and packing algorithm code — represents a step change in documentation thoroughness for a frontier-competitive model.

Innovation 2: "Diversity First, Then Quality" as an Operationalized Principle with a Quantitative Diversity Metric

The paper introduces and systematically operationalizes a specific data development philosophy: "Diversity first, then quality" — push coverage breadth to the extreme before optimizing sample quality within that coverage. While the importance of data diversity is widely acknowledged in VLM literature, the paper's contribution is making this principle quantitative and actionable through a specific metric (the Similarity Score) and a specific workflow (passive monitoring + proactive error-analysis-driven searching + batch addition with diversity checking).

What the field did before: prior open-source VLM data efforts typically started with a curated set of known high-quality datasets and expanded cautiously, prioritizing quality control over coverage breadth. Cambrian-1's approach of limiting each source to a maximum of 350K samples exemplifies this quality-first mindset — the assumption being that too much data from any single source risks introducing noise that degrades performance. The paper argues this constrains capability: "various constraints such as computing resources may limit the study on more dataset sources, despite their intention to make the data recipe publicly available. This limitation often affects their capability."

The conceptual shift: Eagle 2 inverts the priority order. The first phase is aggressive, broad collection from 180+ sources spanning domains that prior work often ignored — Google Landmark recognition, music sheet reading, meme sentiment analysis, handwritten mathematical expressions, movie poster identification, CAPTCHA reading, weather question answering, and many others. The justification is that capability boundaries are set by data coverage, not data quality — if the model has never seen sheet music during training, no amount of filtering or augmentation on standard VQA data will teach it to read musical notation. Quality refinement (filtering, selection, augmentation) happens after coverage is established, not as a gate that prevents diverse data from entering the pipeline.

The Similarity Score as an enabling technology: the paper's similarity metric (Equation 1, Section 2.2) operationalizes the diversity principle by providing a quantitative answer to "does this new data source add meaningful coverage or merely duplicate what we already have?" Computing the maximum product of image similarity (via SSCD embeddings) and text similarity (via all-mpnet-base-v2 embeddings) within the same data category gives a scalar between 0 and 1. The paper provides operational guidance: scores below 0.3 indicate genuinely new content, scores above 0.3 may indicate overlap but can still be retained if the data quality warrants it. This metric enables a systematic, inexpensive screening process — the paper explicitly states that similarity scores "can quickly help us assess the overlap between new data and the existing data pool" — replacing the manual inspection that would be infeasible across 180+ sources and millions of samples.

Why the product form is conceptually important: the paper emphasizes that multiplying image similarity and text similarity ensures that both modalities must overlap for a high score. A dataset that reuses the same COCO images but asks entirely new questions (different text) gets low similarity despite high image overlap — correctly recognized as adding diversity. A dataset with similar question types but entirely different images also gets low similarity. This specificity matters because the paper's collection includes many datasets that share image sources but differ in task (e.g., COCO-Colors using COCO images for color-focused questions) — the metric correctly distinguishes these from true duplicates.

The proactive searching feedback loop: beyond passive monitoring of new datasets, the paper's error-analysis-driven proactive searching is a concrete instantiation of the diversity-first philosophy in an iterative development process. After each training run, the team identifies specific capability gaps (e.g., poor handwriting recognition, weak spatial reasoning) and actively searches for or creates datasets that specifically address those gaps. This transforms data collection from a one-time upfront effort into a continuous, measurement-driven engineering process — analogous to how software testing drives development through bug reports that prompt targeted fixes.

Evidence: Figure 2 shows the step-by-step accumulation of capability as diverse data sources are added: starting from the baseline 58.8 average score, each addition of a new data category (Naive OCR, Chart/Table/OCR QA, Science/Math, Captioning, Grounding/Counting, Text-Only, General VQA) produces monotonic improvements, reaching 67.0 before the training recipe and data refinement strategies kick in. The absolute gains from diversity expansion (58.8 → 67.0, +8.2 points) exceed the gains from any subsequent single refinement step except the introduction of Stage-1.5. This empirical trajectory validates the diversity-first ordering: establishing broad coverage first provided the raw material that downstream refinement could then optimize.

Comparison to prior work: Instruct-BLIP approached data balancing by devising optimal data ratios across sources, but did so within a relatively narrow set of established academic datasets. Cambrian-1 systematically analyzed data source mix ratios for VLMs but operated at smaller scale (7M samples from fewer sources) and did not have the similarity score as a quantitative screening tool. The paper's similarity score is conceptually related to the duplication detection in Llama3 (which used SSCD for removing near-duplicate images) and Molmo (which used human annotation for quality), but Eagle 2 uses it for diversity assessment rather than removal — the metric guides whether to add new sources, not whether to delete existing ones. This reframing from deduplication to diversity quantification is a subtle but meaningful conceptual shift.

Innovation 3: The Three-Stage Training Recipe as a Deliberate Feedback Architecture — Not Merely "More Stages"

The paper introduces a three-stage training recipe (Stage-1 connector alignment → Stage-1.5 diverse full-model pretraining → Stage-2 high-quality fine-tuning) that might initially appear to be an incremental extension of the standard LLaVA two-stage recipe or the multi-stage approaches in LLaVA-OneVision. However, the paper's distinctive contribution is not the number of stages but rather the deliberate design of the stages as a feedback architecture — a system where Stage-2 provides rapid experimental feedback that drives improvements to Stage-1.5, which in turn provides a stronger foundation for subsequent Stage-2 experiments. This is qualitatively different from simply inserting an additional training phase.

What the field did before: the standard LLaVA two-stage recipe (connector alignment + full-model SFT) treats training as a one-pass pipeline: align, then fine-tune, then you are done. The data for SFT is fixed, and if you want to improve the model, you change the SFT data and retrain the whole SFT stage from scratch. This makes data iteration slow and expensive because every change requires retraining the full model on the entire dataset. As the paper notes: "this approach proved unsuitable for quick SFT data updates, as the expanding SFT data makes it harder to track the impact of new data and reduces the experimental efficiency." Multi-stage approaches like LLaVA-OneVision added intermediate pretraining stages to improve final performance, but did not conceptualize the stages as a bidirectional feedback system — they were purely sequential, with data flowing forward but experimental insights not flowing backward.

The conceptual shift: the paper designs Stage-1.5 and Stage-2 to have fundamentally different properties that make them complementary in an iterative development cycle. Stage-1.5 uses large-scale, diverse data (21.6M samples) and takes substantial compute (28 hours on 256 H100 GPUs) — it is slow to iterate but provides a robust, broadly capable foundation. Stage-2 uses curated, high-quality data (4.6M samples) and trains quickly (6 hours on 256 H100 GPUs) — it enables rapid experimentation to test data strategies (filtering rules, augmentation methods, formatting changes, subset selection approaches). The paper explicitly quantifies this asymmetry: "in actual development, we rarely iterate the Stage-1 model. Usually, we iterate Stage-1.5 once after iterating Stage-2 > 10 times" (Appendix A).

This is more than an engineering convenience. It creates a virtuous cycle that the paper visualizes in Figure 8: Stage-1.5 ("Large wheel") provides the stable foundation, Stage-2 ("Small wheel") enables fast iteration, and "high-quality data back" from Stage-2 discoveries feeds into Stage-1.5 updates. The underlying insight is that data strategy development itself needs an efficient experimental infrastructure — without rapid iteration capability, data scientists cannot explore the design space thoroughly and are forced to make decisions based on intuition rather than evidence. The three-stage recipe is thus not just a training strategy for the model but a meta-strategy for the research process that produced the model.

The data allocation philosophy: another distinctive aspect is what data goes where. Prior multi-stage approaches (LLaVA-OneVision) tend to use captioning and knowledge data in intermediate stages, reserving task-specific instruction data for the final stage. Eagle 2 takes the opposite approach: "we add all data sources intended for visual instruction to Stage-1.5." The rationale is that limiting intermediate pretraining to captioning creates a capability gap — the model learns to describe images but not to reason about charts, read text, or solve math problems — and that gap must be closed entirely in the final SFT stage, making that stage's job harder. By including all task types in Stage-1.5, the model builds broad capabilities early, and Stage-2 can focus on refinement rather than teaching fundamentally new skills.

Evidence for the feedback architecture: Table 6 shows the cycle in action. The Stage-1.5 checkpoint alone achieves 69.7 average score — already competitive with some fully trained models. Stage-2 on top of this reaches 70.9. Then, applying data strategies discovered through Stage-2 experimentation back to Stage-1.5 ("Re-Update Stage 1.5") improves the average to 72.4, with notable jumps on ChartQA (85.1 → 85.9), MMVet (60.1 → 61.3), and MathVista (63.5 → 65.2). This confirms that the feedback loop is not just theoretical — it produces measurable improvements that could not be achieved by simply extending Stage-2 or adding more data to a single SFT stage.

Significance beyond performance: this contribution is fundamentally about research methodology for VLM development. The paper is arguing, through its architecture, that building frontier VLMs requires designing the training process itself for rapid experimentation and knowledge accumulation — not just designing the model or the data. This is a meta-level insight that applies regardless of the specific data content or model architecture. Future VLM projects can adopt the three-stage feedback architecture with different data compositions, different model scales, and different tasks, because the structural insight (decouple broad capability building from targeted experimentation, create a feedback loop between them) is general.

Innovation 4: Discovering and Systematically Characterizing the Hidden Costs of "Small" Data Errors

The paper makes a distinctive empirical contribution by identifying, categorizing, and quantifying the impact of systematic quality failures in publicly available VLM datasets that prior work either overlooked or considered too minor to address. The key intellectual move is showing that these errors — which might each affect only a small fraction of samples — have disproportionate effects on model behavior because they create spurious patterns that the model learns and reproduces, sometimes catastrophically.

What the field did before: prior work on VLM data quality focused on broad statistical properties — removing entire datasets with known issues, balancing source ratios, deduplicating near-identical samples, or filtering based on CLIP similarity scores. Cambrian-1 removed subsets like ShareGPT-4V and Data-Engine-161K entirely based on quality concerns. Llama3 used SSCD for image deduplication. Molmo incorporated human-annotated images for quality assurance. But these approaches operate at the dataset or sample level — they do not identify specific, recurring patterns of errors that span multiple datasets and require targeted, pattern-based filtering rules.

The four error categories: the paper characterizes four distinct failure modes (Section 2.2, Figures 5 and 6) that recur across publicly available datasets:

Mismatched question-answer pairs: the answer does not correspond to the question, often because synthetic generation pipelines lose coherence between the question prompt and the generated response.
Irrelevant image-question pairs: the question references visual content not present in the image, an error from incorrect dataset assembly or overly general question templates applied to arbitrary images.
Repeated texts: LLM-generated responses that get stuck in loops, producing degenerate training examples that would teach the model to repeat itself.
Numeric formatting issues: excessive decimal precision from computational pipelines that do not handle floating-point output appropriately, producing answers like "0.2700346484855259" that no human would write and that cannot be verified from visual inspection.

What makes this contribution distinctive is not the existence of these errors — practitioners have long suspected that public datasets contain noise — but the systematic characterization and the demonstration that fixing them produces large, measurable improvements. The paper's most striking evidence is the LaTeX template example (Table 3): a seemingly minor formatting inconsistency (wrapping LaTeX output in \begin{align*}...\end{align*} in some training samples but not others) caused the model to "consistently output this fixed template" even when inappropriate. After removing the template, the model "returns to normal behavior, leading to a significant improvement on OCRBench" — a 45-point gain (798 → 843, Table 6). This is a remarkable sensitivity: a formatting artifact affecting a subset of LaTeX extraction training samples cascaded into a 45-point degradation on a major OCR benchmark.

Why this matters conceptually: the LaTeX example reveals a general principle about VLM training that is underappreciated in the literature: VLMs are extremely sensitive to format-level inconsistencies in training data, and these inconsistencies can cause disproportionate downstream harm because they create spurious correlations that the model latches onto. A single incorrectly formatted training sample does not just contribute noise to the gradient — it can teach the model a specific, incorrect output pattern that it then reproduces systematically. This is not a data quality issue in the traditional sense of noisy labels reducing statistical efficiency; it is a pattern-learning issue where the model learns a harmful production rule from a small number of examples.

Evidence beyond OCRBench: Table 6 shows that data formatting and filtering together improve performance on 8 out of 14 benchmarks, not just OCRBench. The broad impact suggests that format-level errors are pervasive across public datasets and that systematic attention to formatting is a high-leverage intervention — more impactful per unit of engineering effort than collecting additional data or scaling model parameters.

Comparison to prior work: this contribution is closest in spirit to the "data cascades" concept from the data-centric AI literature (Sambasivan et al., 2021) — the idea that small data quality issues early in a pipeline can compound into large downstream effects. But Eagle 2 provides a concrete, quantified instantiation in the VLM domain with actionable remediation strategies. The paper's contribution is not a new filtering algorithm (the filters are rule-based and straightforward) but rather the diagnostic framework — identifying the specific error categories that matter, demonstrating their impact, and providing the evidence to justify the engineering effort of fixing them. This diagnostic knowledge is transferable: future VLM projects can use the same error categories as a checklist when evaluating their own training data, even if their specific filtering implementations differ.

Innovation 5: Balanced-Aware Data Packing as a Demonstration That Training Infrastructure Details Are Not "Just Engineering"

The paper identifies a subtle but consequential issue in a standard training infrastructure component — the greedy knapsack algorithm used for data packing in LLaMA-Factory — and shows that fixing it produces improvements comparable to adding substantial new training data. The intellectual contribution is not the specific algorithm (the balanced greedy knapsack) but rather the demonstration that training infrastructure decisions, typically treated as "just engineering" below the level of scientific interest, can have effects on final model quality that rival or exceed those of data strategy decisions.

What the field did before: data packing is a standard technique in LLM training, used to accelerate training by concatenating multiple short samples into fixed-length sequences. The dominant approach, implemented in frameworks like LLaMA-Factory, uses a naive greedy knapsack algorithm: sort samples by length, fill packs with the longest remaining samples that fit, and move to the next pack when the current one is full. This algorithm is simple, fast, and produces valid packs — which is why it is the default. The community has generally treated packing as an implementation detail that affects training speed but not model quality, as long as packing is done correctly (no cross-contamination between samples, proper attention masking).

The conceptual shift: the paper shows that the naive packing algorithm introduces a systematic bias in the training distribution. Because long samples tend to get grouped together in early packs and short samples in later packs (Figure 9a), the loss weighting across samples becomes imbalanced — long samples in nearly-full packs contribute many more tokens to the gradient than short samples in mostly-empty packs (even with proper loss masking, the distribution of effective training signal is skewed). The paper argues this is "not desirable to model training" and proposes a balanced alternative that distributes long and short samples evenly across all packs (Figure 9b), ensuring uniform effective loss weighting.

Why this is intellectually significant, not "just engineering": the performance impact is substantial. Table N shows that switching from naive to balanced packing improves the average score from 72.8 to 73.5, with gains of 1.7 points on ChartQA, 1.4 points on MMMU, 2.2 points on MathVista, and 1.4 points on HallusionBench. These improvements come from a change that does not affect what data the model sees — only the order and grouping in which it is presented. This demonstrates that training dynamics (loss weighting across samples, gradient contribution distributions) can be as consequential as data content for final model quality.

The broader implication is that the boundary between "training infrastructure" and "training methodology" is a false one. Decisions made in data loaders, packing algorithms, and sequence ordering — typically delegated to engineering teams and treated as implementation details — encode assumptions about how training should work that can have first-order effects on model quality. The paper's contribution is making this point empirically visible and providing a concrete, measurable example. The balanced packing algorithm itself is a straightforward modification of the greedy approach (Figure 10, Listing 4), but the key contribution is the diagnostic — identifying that the naive algorithm's length distribution bias matters for training — and the framework — showing that infrastructure optimization deserves the same systematic attention as data curation or architecture design.

Comparison to prior work: the SPFHP (Shortest-Pack-First Histogram Packing) algorithm prioritized packing efficiency — minimizing the number of packs and thus training steps. The paper explicitly distinguishes its approach: "Unlike SPFHP, our method prioritizes balanced length distribution over packing efficiency, helping balance loss weights between long and short samples." This is a different optimization objective — training quality rather than training speed — applied to the same infrastructure component. The paper's finding that this trade-off is worthwhile (balanced packing sacrifices some efficiency for quality, but packing still provides 2-3× speedup over no packing) challenges the default prioritization of throughput over all other considerations in training infrastructure.

Scope of the insight: while the paper demonstrates this in the context of VLM post-training, the principle is general to any setting where sequence packing is used — LLM pretraining, instruction tuning, multimodal training. The paper's balanced algorithm and the evidence of its importance can be directly adopted by any project using packed training sequences, making this a readily transferable contribution.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation is conducted on 14 diverse multimodal benchmarks, each targeting a specific VLM capability. These are: DocVQA (Test split, document visual question answering), ChartQA (Test split, chart reasoning), InfoVQA (Test split, infographic understanding), TextVQA (Val split, text reading in natural images), OCRBench (full benchmark score, optical character recognition), MMStar (Test split, multimodal reasoning), RWQA (Test split, real-world visual QA), AI2D (Test split, diagram understanding), MMMU (Val split, multi-discipline multimodal understanding), MMBench v1.1 (Test split, general multimodal understanding), MMVet (Test split, visual reasoning and explanation), HallusionBench (Test split, hallucination detection), MathVista (Test-Mini split, mathematical visual reasoning), and OpenCompass (aggregated score across Eagle benchmarks: OCRBench, MMStar, AI2D, MMMU, MMB1.1, MMVet, HallusionBench, and MathVista). Benchmarks are drawn from standard academic sources, with test splits used where available to avoid training contamination. The paper also reports scores on the MATH benchmark split from Lightman et al. (2022) consisting of 12,000 training and 500 test questions, though this is used for internal validation rather than the main comparison tables.
Base model. The Eagle 2 model family is built on the Qwen2.5 LLM backbone in three scales: Qwen2.5-0.5B-Instruct, Qwen2.5-1.5B-Instruct, and Qwen2.5-7B-Instruct, producing Eagle2-1B, Eagle2-2B, and Eagle2-9B respectively. The 7B variant is the primary model for comparison against state-of-the-art VLMs. For the vision encoders, the 9B model uses a tiled mixture of SigLIP-400M and ConvNeXt-XXLarge, while the 1B and 2B models use SigLIP alone (Table 4). The paper argues this model scale is "representative of contemporary VLM capabilities" and positions the 9B model against competitors spanning 7B to 90B parameters to demonstrate that data strategy, not scale, drives performance.
Metrics. The primary metric throughout is accuracy — the fraction of test questions for which the model's selected answer matches the ground truth — reported as a percentage for each benchmark individually and as an average score across all benchmarks. The average score is computed as the arithmetic mean of the 13 benchmark scores (DocVQA through MathVista in Tables 5 and 6, with 14 benchmarks including OpenCompass in Table 7). For OpenCompass, the paper uses a specific subset of 8 benchmarks (OCRBench, MMStar, AI2D, MMMU, MMB1.1, MMVet, HallusionBench, MathVista) and averages them. All benchmarks use their standard evaluation protocols; answers are graded using the official grading functions released by each benchmark's authors.
Baselines. The paper compares against an extensive set of models spanning all three transparency tiers:

Closed-source commercial models:
- GPT-4o-0513 [184] — OpenAI's multimodal flagship model
- GPT-4V [11] — the predecessor to GPT-4o
- Gemini-1.5-Pro [185] — Google's multimodal model
Weight-available frontier models (public weights, limited training details):
- InternVL2 family [14] at 1B, 2B, 8B, and 26B scales
- Qwen2-VL [25] at 2B and 7B scales
- MiniCPM-V2.6 [186]
- Llama-3.2-90B-Vision [15]
- LLaVA-OneVision [17] at 0.5B, 7B, and 72B scales
Fully open-source models (public weights and training details):
- Cambrian-1-7B [16] — serves as the starting point for Eagle 2's baseline
- LLaVA-OneVision family (also listed above, as it provides full training details)
The paper also includes an internal Eagle2-Baseline (Table 1): the Cambrian-1 5.2M filtered subset with LLaVA's two-stage training recipe and SigLIP-400M encoder, which serves as the ablation starting point.
Generation budget / compute accounting. The paper does not use a unified "generation budget" in the sense of inference-time scaling papers, since this is a post-training methodology paper rather than a test-time compute study. Instead, compute is accounted for in terms of training data volume (number of samples in millions, e.g., 21.6M for Stage-1.5, 4.6M for Stage-2) and GPU-hours (Table A: 2.5 hours on 128 H100s for Stage-1, 28 hours on 256 H100s for Stage-1.5, 6 hours on 256 H100s for Stage-2, for the 9B model). For the FLOPs-matched comparison in the FLOPs-matched comparison described in Section 2.3, the paper reports total training compute but does not perform a separate inference-time scaling analysis analogous to the comparison against pretraining in the referenced paper. All comparisons across models are at fixed benchmark evaluation protocols — each model answers each question once under its default inference configuration.
Cross-validation / statistical protocol. The paper does not employ formal cross-validation for benchmark evaluation. Instead, standard benchmark test splits are used with official evaluation protocols. The data strategy development uses an informal validation process: "we avoid performing ablation for each dataset individually. Instead, datasets with similar domains are added in batches to the data pool when meeting the following criteria: (1) Maintaining overall accuracy without noticeable regression for every considered benchmark, and (2) Introducing meaningful diversity to the current domains" (Section 2.2). This is a practical heuristic rather than a statistical protocol. For internal development, the paper uses the MATH benchmark (500 test questions, Section 4) and refers to validation splits of various benchmarks, but these are not systematically cross-validated. The paper does not report confidence intervals or statistical significance tests on any of the benchmark results, which is a limitation.

Main Quantitative Results

Step-by-Step Ablation of Data Strategy Improvements

The paper's central experimental narrative is the progressive improvement of the model through successive data strategy and training recipe changes, documented in Figure 2, Table 5, and Table 6. The journey from baseline to final model spans a 14.7-point average score improvement across 13 benchmarks.

Starting point (Table 5, Eagle2-Baseline). Using the Cambrian-1 5.2M filtered subset with LLaVA's two-stage training and SigLIP-400M encoder, the baseline achieves an average score of 58.8 across 13 benchmarks. Individual benchmark scores include DocVQA 77.0, ChartQA 65.9, InfoVQA 50.0, TextVQA 69.9, OCRBench 589, MMStar 47.3, RWQA 60.3, AI2D 74.7, MMMU 49.7, MMBench v1.1 71.2, MMVet 45.9, HallusionBench 44.8, and MathVista 49.2. For reference, Cambrian-1-7B (the original model using the full 7M dataset) achieves an average that is not directly computed in the paper but shows individual benchmark scores broadly similar to or slightly below the Eagle2-Baseline on several metrics. Notably, the baseline already improves on some benchmarks by removing low-quality subsets (ShareGPT-4V, GPT-77K, Data-Engine-161K) from the Cambrian-7M pool.

Data scaling in two-stage training (Table 5). The paper first explores scaling the Stage-2 SFT data under the standard two-stage recipe. Starting from the 5.2M baseline, they add data in batches organized by category:

+ Naive OCR (0.8M samples): Adds datasets like SynthDoG, MTWI, LVST, SROIE, FUNSD, Latex-Formula, IAM, Handwriting-Latex, ArT, CTW, ReCTs, COCO-Text, and others (detailed in Appendix Table C). Average score improves from 58.8 to 61.3. The largest individual gains are on OCRBench (589 → 719, a 130-point jump) and InfoVQA (50.0 → 57.0, +7.0 points). This dramatic improvement on OCR metrics demonstrates that the baseline had essentially no OCR training data beyond what was incidentally present in general VQA datasets.
+ Chart, Table & OCR QA (2.0M samples): Adds ChartQA, MMC-Inst, DVQA, PlotQA, LRV-Instruction, TabMWP, UniChart, Vistext, TAT-DQA, VQAonDB, FigureQA, Chart2Text, RobuT, MultiHiertt (Appendix Table H) plus OCR QA datasets like DocVQA, InfoVQA, TextVQA, ArxivQA, ScreenQA, DocReason, Ureader, FinanceQA, DocMatrix, A-OKVQA, and many others (Appendix Table I). Average score reaches 65.0. The largest gains are on DocVQA (77.0 → 88.0, +11.0 points), ChartQA (65.9 → 78.3, +12.4 points), InfoVQA (57.0 → 65.0, +8.0 points), TextVQA (73.7 → 77.6, +3.9 points), and OCRBench (719 → 783, +64 points). This is the single largest improvement step, adding 3.7 points to the average score and highlighting that chart and document understanding capabilities are almost entirely determined by the presence of domain-specific training data.
+ Science & Math (0.7M samples): Adds AI2D (12K repeated 4 times), ScienceQA, TQA, PathVQA, SciQA, VQA-RAD, VisualWebInstruct, TextBooks-QA for science (Appendix Table E) plus GeoQA+, MathQA, CLEVR, CLEVR-Math, MAVIS-math-rule-geo, MAVIS-math-metagen, InterGPS, Raven, GEOS, and UniGeo for math (Appendix Table F). Average score reaches 65.2. Gains are modest and concentrated on MMBench v1.1 (72.7 → 75.6, +2.9 points) and MMVet (50.1 → 51.5, +1.4 points), while some benchmarks show slight regressions (InfoVQA 65.0 → 64.0, OCRBench 783 → 777, HallusionBench 39.9 → 38.9). This pattern suggests that the science and math data provide incremental reasoning capability improvements, but the effect is less dramatic than the OCR and chart additions.
+ Captioning (0.5M samples): Adds ShareGPT4o, KVQA, Movie-Posters, Google-Landmark, WikiArt, Weather-QA, Coco-colors, music-sheet, SPARK, SAM-caption, Tmbd-Celeb-10K, Image-Textualization (Appendix Table G). Average score reaches 65.7. Gains are primarily on MMMU (50.7 → 52.9, +2.2 points), MMBench v1.1 (75.6 → 77.9, +2.3 points), and MMVet (51.5 → 55.8, +4.3 points). The MMVet improvement is notable since this benchmark evaluates detailed visual reasoning and explanation quality — capabilities that captioning data directly supports by teaching the model to produce rich, detailed descriptions.
+ Grounding & Counting (0.2M samples): Adds TallyQA, OODVQA, RefCOCO/+/g, GroundUI (Appendix Table D). Average score reaches 66.0. Gains are small and distributed: TextVQA (78.0 → 77.9, slight decline), ChartQA (79.0 → 79.4), OCRBench (784 → 792), MMStar (53.7 → 54.3), MMBench v1.1 (77.9 → 77.6, slight decline). The modest impact suggests that grounding data provides marginal improvements but is not a critical bottleneck at this stage.
+ Text-Only (2.3M samples): Adds Orca, Orca-Math, MathInstruct, WizardLM, TheoremQA, OpenHermes2.5, NuminaMath-CoT, Python-Code-25k, Infinity-Instruct, Python-Code-Instructions-18k-Alpaca, Ruozhiba, InfinityMATH, StepDPO, TableLLM, UltraInteract-sft (Appendix Table J). Average score reaches 66.6. The most notable individual gain is on MMVet (55.8 → 58.8, +3.0 points), with smaller improvements on MMStar (54.3 → 55.5) and RWQA (61.7 → 65.4). The large text-only component (2.3M out of the total ~10M at this stage, or 23%) is deliberate — the paper explicitly states that "the quality of text-only data still matters for multi-modal LLMs" (Appendix Table J caption) and ensures "text-only data occupy over 20%" in the final data distribution to prevent catastrophic forgetting of language capabilities.
+ General VQA (1.0M samples): Adds the remaining general VQA datasets: LLaVA-150K (EN and CN versions), LVIS-Instruct4V, ALLaVa (laion and vflan splits), Laion-GPT4V, LLAVAR, SketchyVQA, VizWiz, IDK, AlfworldGPT, LNQA, Face-Emotion, SpatialSense, Indoor-QA, Places365, MMinstruct, DriveLM, YesBut, WildVision, LLaVA-Critic-113k, RLAIF-V, VQAv2, MMRA, KONIQ, MMDU, Spot-The-Diff, Hateful-Memes, COCO-QA, NLVR2, Mimic-CGD, Datikz, Chinese-Meme, IconQA, Websight (Appendix Table B). Average score reaches 67.0. Gains are modest improvements on ChartQA (78.5 → 80.0), RWQA (65.4 → 65.9), AI2D (76.5 → 76.9), MMMU (51.4 → 52.2), MMVet (58.8 → 56.8, a 2.0-point decline), HallusionBench (37.3 → 40.8, +3.5 points). The mixed results suggest that adding more general VQA data at this scale provides diminishing returns — the model's major capability gaps have already been addressed by the domain-specific data additions.

The overall trend from data scaling in the two-stage setting is consistent improvement from 58.8 to 67.0 (+8.2 points), with the largest jumps coming from OCR and chart data additions. However, the paper notes that "reaching the performance of frontier VLMs like Qwen2-VL would be difficult" with this approach alone (Section 3.1), motivating the transition to the three-stage recipe.

The transition to three-stage training represents the paper's core methodological shift, and Table 6 documents the progressive improvements through data refinement strategies applied within this new framework.

Introducing Stage-1.5 (Table 6, rows 1-2). The Stage-1.5 checkpoint — trained on 21M diverse samples without any Stage-2 fine-tuning — achieves an average score of 69.7 by itself. This is already 2.7 points higher than the best two-stage model (67.0) trained with extensive Stage-2 data. Individual benchmark scores include DocVQA 89.9, ChartQA 82.7, InfoVQA 71.1, TextVQA 79.5, OCRBench 812, MMStar 58.5, RWQA 69.4, AI2D 78.6, MMMU 50.9, MMBench v1.1 81.7, MMVet 55.7, HallusionBench 47.1, and MathVista 60.1. The fact that a pretraining-only checkpoint competes with or exceeds a fully fine-tuned two-stage model is a strong endorsement of the Stage-1.5 philosophy — broad, diverse pretraining is more effective than narrow, high-quality fine-tuning for building foundational capability.

Adding Stage-2 on top of Stage-1.5 (training on 12.7M samples, which is the full data pool before subset selection) yields an average of 70.9. The Stage-2 fine-tuning improves DocVQA (89.9 → 91.0), ChartQA (82.7 → 84.0), InfoVQA (71.1 → 72.5), TextVQA (79.5 → 81.0), AI2D (78.6 → 81.0), and most benchmarks by 1-2 points. Notably, HallusionBench shows a slight decline (47.1 → 46.4), and MathVista improves only modestly (60.1 → 61.4). This confirms that Stage-2 fine-tuning on curated data provides incremental specialization beyond the broad capabilities built in Stage-1.5.

Naive Subset Selection (Table 6, row 3). Reducing the Stage-2 data from 12.7M to 8.2M using naive random selection with maximum thresholds (the Cambrian-1 approach) causes a performance decline: average drops from 70.9 to 70.6. Individual regressions include DocVQA (91.0 → 90.4), ChartQA (84.0 → 83.7), TextVQA (81.0 → 79.7), and OCRBench (825 → 798, a 27-point drop). The paper speculates that "the randomly selected data have inadvertently excluded some valuable samples, while also failing to adequately ensure a balanced data distribution." This is a critical negative result: naive subset selection can undo the gains from scaling up data, and the method of selection matters as much as the quantity.

Data Formatting & Filtering (Table 6, row 4). Applying rule-based filtering (removing mismatched QA pairs, irrelevant image-question pairs, repeated texts, numeric formatting issues, and refusal responses) and data formatting (standardizing task representations, removing LaTeX template artifacts, appending task-specific instructions) to the 8.2M subset improves the average to 71.2, exceeding the previous best of 70.9 achieved with 12.7M unfiltered data. The most dramatic improvement is on OCRBench: 798 → 843, a 45-point gain from formatting and filtering alone. Other gains include DocVQA (90.4 → 91.1), ChartQA (83.7 → 84.5), InfoVQA (72.0 → 72.3), TextVQA (79.7 → 81.2), MMMU (51.7 → 53.2), and MathVista (61.0 → 62.0). Eight out of 14 benchmarks show improvement. The paper explicitly attributes the OCRBench gain to the LaTeX template removal described in Section 2.2: removing the fixed \begin{align*}...\end{align*} environment from training samples stopped the model from incorrectly reproducing this template at inference time. This is a striking example of how a formatting artifact affecting a small subset of LaTeX extraction training data can cascade into a 45-point degradation on a major OCR benchmark — validating the paper's claim that "carefully reviewing and utilizing existing data, as well as exercising caution with publicly available data" is essential.

Advanced Subset Selection (Table 6, row 5). Switching from naive random selection to K-means clustering on SSCD image embeddings — which balances representation across data subtypes within each source — further reduces the Stage-2 data to 4.6M samples while improving the average score to 71.8. This means performance improves while using only 4.6M / 8.2M = 56% of the naive subset's samples, and only 4.6M / 12.7M = 36% of the original unfiltered Stage-2 data. Gains are concentrated on InfoVQA (72.3 → 73.3), MMStar (62.1 → 62.7), RWQA (67.7 → 68.9), AI2D (83.0 → 84.1), MMBench v1.1 (82.2 → 82.5), MMVet (55.8 → 59.3, a 3.5-point jump), and HallusionBench (46.6 → 49.2, a 2.6-point jump). Some benchmarks show slight declines (DocVQA 91.1 → 90.8, ChartQA 84.5 → 84.1, MMMU 53.2 → 52.5, MathVista 62.0 → 60.5), suggesting that the balanced subset trades off some domain-specific performance for broader capability improvement. The net effect is positive (71.2 → 71.8) with substantially less data.

Data Augmentation (Table 6, row 6). Adding CoT-augmented training data, rule-based QA generation, and short-to-long answer expansion improves the average to 72.1. The most notable gains are on MMMU (52.5 → 53.6, +1.1 points), MathVista (60.5 → 63.5, +3.0 points), MMVet (59.3 → 60.1), and ChartQA (84.1 → 85.1, +1.0 point, attributed specifically to rule-based augmentation on chart data). OCRBench shows a slight decline (843 → 839), and MMStar declines slightly (62.7 → 61.4). The MathVista improvement is particularly significant because this benchmark tests mathematical reasoning with visual inputs — a capability that CoT augmentation directly supports by teaching the model to produce step-by-step reasoning chains rather than just final answers. Appendix Table M provides targeted evidence: models trained without CoT data show no benefit from "Think it step-by-step" prompts at inference time (scores of 61.0 and 60.5 with and without the prompt for a non-CoT-trained model), while models trained with CoT data show clear improvement when the CoT prompt is used (63.2 and 63.5 with and without). This demonstrates that the CoT augmentation teaches genuine reasoning capability, not just formatting mimicry.

Re-Update Stage-1.5 (Table 6, row 7). Applying the data strategies discovered through Stage-2 experimentation (filtering, formatting, augmentation) back to the Stage-1.5 data pool — expanding it from 21M to 22M samples — and retraining the Stage-1.5 checkpoint improves the average to 72.4. The gain comes primarily from ChartQA (85.1 → 85.9, +0.8 points), MMVet (60.1 → 61.3, +1.2 points), and MathVista (63.5 → 65.2, +1.7 points). Several benchmarks show slight regressions (DocVQA 91.8 → 91.3, MMStar 61.4 → 61.7 is a gain, RWQA 68.4 → 68.2, AI2D 83.9 → 83.5). This step validates the "large wheel drives small wheel" feedback architecture (Figure 8): effective data strategies identified through rapid Stage-2 iteration can be propagated back to improve the Stage-1.5 foundation, creating a compounding improvement cycle.

Mixture of Vision Encoders (Table 6, row 8). The final architectural change — replacing SigLIP alone with the tiled MoVE (SigLIP + ConvNeXt) — yields the largest single-step improvement in the entire development trajectory: average score jumps from 72.4 to 73.5, a +1.1 point gain. Improvements appear on 12 out of 14 benchmarks: DocVQA (91.3 → 92.6, +1.3), ChartQA (85.9 → 86.4, +0.5), InfoVQA (73.3 → 77.2, +3.9 — the largest individual gain), TextVQA (81.9 → 83.0, +1.1), OCRBench (842 → 868, +26 points), MMStar (61.7 → 62.6), RWQA (68.2 → 69.3), AI2D (83.5 → 83.9), MMMU (53.6 → 56.1, +2.5 points), MMVet (61.3 → 62.2), HallusionBench (49.0 → 49.3), and MathVista (65.2 → 63.8, a 1.4-point decline). MMBench v1.1 shows a slight decline (82.4 → 81.9). The paper emphasizes that the improvements are particularly pronounced on "benchmarks related to documents, charts, and OCR" — the domains where fine-grained visual feature extraction from ConvNeXt complements the semantic understanding from SigLIP.

Comparison with State-of-the-Art Models (Table 7)

The final Eagle2-9B model is compared against an extensive set of frontier VLMs across all 14 benchmarks plus the aggregated OpenCompass score.

Eagle2-9B vs. same-scale weight-available models (7-8B parameter class):

vs. InternVL2-8B [14]: Eagle2-9B outperforms InternVL2-8B on all 14 benchmarks. The margins range from narrow (MMStar 62.6 vs. 60.9, +1.7; MMVet 62.2 vs. 54.2, +8.0; HallusionBench 49.3 vs. 45.2, +4.1) to substantial (ChartQA 86.4 vs. 83.3, +3.1; OCRBench 868 vs. 794, +74 points; MathVista 63.8 vs. 58.3, +5.5). On OpenCompass, Eagle2-9B scores 68.2 vs. 64.1.
vs. Qwen2-VL-7B [25]: Eagle2-9B leads on 9 out of 14 benchmarks. Eagle2-9B wins on DocVQA (92.6 vs. 94.6 — actually loses here, since Qwen2-VL has 94.6; a 2.0-point gap), ChartQA (86.4 vs. 83.0, +3.4), InfoVQA (77.2 vs. 74.3, +2.9), OCRBench (868 vs. 845, +23), MMStar (62.6 vs. 60.7, +1.9), MMMU (56.1 vs. 54.1, +2.0), HallusionBench (49.3 vs. 50.5 — loses by 1.2 points), MathVista (63.8 vs. 58.2, +5.6), and OpenCompass (68.2 vs. 67.0). Qwen2-VL-7B wins on TextVQA (83.0 vs. 84.3), RWQA (69.3 vs. 70.1, -0.8), AI2D (83.9 vs. 83.0 — actually Eagle2 wins here at 83.9 vs. 83.0, +0.9), MMBench v1.1 (81.9 vs. 81.0, +0.9), and MMVet (62.2 vs. 62.0, +0.2). The comparison is close overall, with Eagle2-9B showing particular strength in OCR and chart domains.
vs. MiniCPM-V2.6 [186]: Eagle2-9B outperforms on all 13 reported benchmarks (MiniCPM-V2.6 does not report InfoVQA). Notable margins include OCRBench (868 vs. 852, +16), MMStar (62.6 vs. 57.5, +5.1), MMMU (56.1 vs. 49.8, +6.3), and MathVista (63.8 vs. 60.6, +3.2).
vs. LLaVA-OneVision-7B [17]: Eagle2-9B leads on all 11 benchmarks with available comparisons (LLaVA-OneVision does not report TextVQA or MMBench v1.1 in this table). The margins are particularly large on OCRBench (868 vs. 622, +246 points), DocVQA (92.6 vs. 87.5, +5.1), and ChartQA (86.4 vs. 80.0, +6.4).

Eagle2-9B vs. larger weight-available models:

vs. InternVL2-26B [14] (a model with approximately 2.9× more parameters): Eagle2-9B competes closely. Eagle2-9B wins on ChartQA (86.4 vs. 84.9, +1.5), InfoVQA (77.2 vs. 75.9, +1.3), TextVQA (83.0 vs. 82.3, +0.7), OCRBench (868 vs. 825, +43), MMStar (62.6 vs. 61.0, +1.6), RWQA (69.3 vs. 68.3, +1.0), MMMU (56.1 vs. 50.7, +5.4), MMVet (62.2 vs. 62.1, +0.1), and OpenCompass (68.2 vs. 66.4). InternVL2-26B wins on DocVQA (92.6 vs. 92.9, -0.3), AI2D (83.9 vs. 84.5, -0.6), MMBench v1.1 (81.9 vs. 81.2, — actually Eagle2 wins at 81.9 vs. 81.2, +0.7), HallusionBench (49.3 vs. 50.7, -1.4), and MathVista (63.8 vs. 59.4, +4.4 — Eagle2 wins). The overall picture is that Eagle2-9B is roughly on par with InternVL2-26B despite having 2.9× fewer parameters.
vs. LLaVA-OneVision-72B [17] (8× parameters): Eagle2-9B wins on ChartQA (86.4 vs. 83.7, +2.7), InfoVQA (77.2 vs. 74.9, +2.3), OCRBench (868 vs. 741, +127 points), and MMVet (62.2 vs. 60.6, +1.6). LLaVA-OneVision-72B wins on DocVQA (92.6 vs. 91.7, -0.9), MMStar (62.6 vs. 66.1, -3.5), RWQA (69.3 vs. 71.9, -2.6), AI2D (83.9 vs. 85.6, -1.7), MMMU (56.1 vs. 56.6, -0.5), MMBench v1.1 (81.9 vs. 84.5, -2.6), HallusionBench (49.3 vs. 47.5, +1.8 — Eagle2 wins), MathVista (63.8 vs. 68.4, -4.6), and OpenCompass (68.2 vs. 68.0, +0.2). The comparison is remarkably balanced given the 8× parameter difference.
vs. Llama-3.2-90B-Vision [15] (10× parameters): Eagle2-9B outperforms on ChartQA (86.4 vs. 85.5, +0.9), OCRBench (868 vs. 783, +85 points), MMStar (62.6 vs. 55.3, +7.3), MMVet (62.2 vs. 64.1, -1.9 — Llama wins), HallusionBench (49.3 vs. 44.1, +5.2), MathVista (63.8 vs. 57.3, +6.5), and OpenCompass (68.2 vs. 63.4). Llama-3.2-90B wins on DocVQA (92.6 vs. 90.1, -2.5), MMMU (56.1 vs. 60.3, -4.2), and MMBench v1.1 (81.9 vs. 77.3, +4.6 — Eagle2 wins). InfoVQA and RWQA are not reported for Llama-3.2-90B. Eagle2-9B is clearly competitive with this 10× larger model.

Eagle2-9B vs. closed-source commercial models:

vs. GPT-4V [11]: Eagle2-9B surpasses GPT-4V on ChartQA (86.4 vs. 78.5, +7.9), OCRBench (868 vs. 656, +212 points), MathVista (63.8 vs. 54.7, +9.1), DocVQA (92.6 vs. 88.4, +4.2), InfoVQA (77.2 vs. 75.1, +2.1), TextVQA (83.0 vs. 78.0, +5.0), MMStar (62.6 vs. 56.0, +6.6), RWQA (69.3 vs. 68.0, +1.3), AI2D (83.9 vs. 78.6, +5.3), MMVet (62.2 vs. 67.5, -5.3 — GPT-4V wins), HallusionBench (49.3 vs. 43.9, +5.4), and OpenCompass (68.2 vs. 63.5). GPT-4V wins on MMMU (56.1 vs. 61.7, -5.6) and MMBench v1.1 (81.9 vs. 79.8, +2.1 — Eagle2 wins). Eagle2-9B comprehensively surpasses GPT-4V across most benchmarks with the notable exceptions of MMMU and MMVet.
vs. GPT-4o-0513 [184]: Eagle2-9B surpasses GPT-4o on ChartQA (86.4 vs. 85.7, +0.7), OCRBench (868 vs. 736, +132 points), and MathVista (63.8 vs. 61.3, +2.5). Eagle2-9B is competitive on DocVQA (92.6 vs. 92.8, -0.2), MMStar (62.6 vs. 63.9, -1.3), AI2D (83.9 vs. 84.6, -0.7), and OpenCompass (68.2 vs. 69.9, -1.7). GPT-4o leads substantially on MMMU (56.1 vs. 69.2, -13.1) and MMVet (62.2 vs. 69.1, -6.9). This comparison shows that Eagle2-9B has closed the gap with GPT-4o on many perceptual and reasoning benchmarks, with the largest remaining gaps in comprehensive multimodal understanding (MMMU) and detailed visual explanation (MMVet).

Eagle2-1B and Eagle2-2B (Table 7). The paper also evaluates the smaller models in the Eagle 2 family. Eagle2-1B outperforms InternVL2-1B on all reported benchmarks, with particularly large margins on MMBench v1.1 (63.0 vs. 59.7, +3.3) and MathVista (45.3 vs. 37.7, +7.6), and substantially outperforms LLaVA-OneVision-0.5B across the board. Eagle2-2B outperforms InternVL2-2B on 11 out of 13 shared benchmarks (missing TextVQA and MMBench v1.1 comparisons because InternVL2-2B values differ slightly from the pattern), and outperforms Qwen2-VL-2B on ChartQA (82.3 vs. 73.0, +9.3), OCRBench (818 vs. 809, +9), MMStar (56.4 vs. 48.0, +8.4), RWQA (63.1 vs. 62.6, +0.5), AI2D (79.3 vs. 78.9, +0.4), MMMU (43.1 vs. 41.1, +2.0), MMBench v1.1 (74.9 vs. 72.2, +2.7), MMVet (53.8 vs. 49.5, +4.3), HallusionBench (45.8 vs. 41.7, +4.1), MathVista (54.7 vs. 43.0, +11.7), and OpenCompass (61.2 vs. 57.2). Qwen2-VL-2B wins on DocVQA (88.0 vs. 90.1, -2.1) and InfoVQA (65.8 vs. 65.5, +0.3 — Eagle2 wins). The consistency of improvements across model scales (1B, 2B, 9B) suggests that the data strategy benefits are not specific to a particular model size.

Overall headline result: Eagle2-9B achieves state-of-the-art performance among models at the 7-9B parameter scale, outperforming InternVL2-8B on all 14 benchmarks, leading Qwen2-VL-7B on 9 of 14, and surpassing GPT-4V on 10 out of 13 reported benchmarks (plus OpenCompass). It matches or exceeds models with up to 10× more parameters (LLaVA-OneVision-72B, Llama-3.2-90B) on a majority of benchmarks, with particularly dominant performance on OCR and chart-related tasks. The remaining gaps with GPT-4o are concentrated in MMMU (comprehensive multimodal reasoning) and MMVet (detailed visual explanation).

Balanced vs. Naive Data Packing (Table N)

The paper provides a direct ablation comparing the naive greedy packing (LLaMA-Factory default) against the proposed balanced-aware packing algorithm, using the Eagle2-9B model. The balanced packing method achieves an average score of 73.5 versus 72.8 for naive packing, a +0.7 point improvement. Individual benchmark improvements include ChartQA (84.7 → 86.4, +1.7), InfoVQA (76.5 → 77.2, +0.7), OCRBench (855 → 868, +13), MMMU (54.7 → 56.1, +1.4), HallusionBench (47.9 → 49.3, +1.4), and MathVista (61.6 → 63.8, +2.2). Several benchmarks show slight declines: TextVQA (83.8 → 83.0, -0.8), MMStar (62.7 → 62.6, -0.1), AI2D (84.0 → 83.9, -0.1), and MMBench v1.1 (81.7 → 81.9, +0.2 — slight gain). DocVQA is unchanged at 92.6, and MMVet is essentially flat (63.0 → 62.2, -0.8). The paper attributes the improvements to "balancing loss weights between long and short samples" — the balanced packing ensures that long, information-rich samples and short, terse samples both contribute properly to the gradient signal during training, rather than having long samples dominate early packs and short samples dominate later packs in an imbalanced way.

Ablation Studies and Robustness Checks

Data scaling under two-stage training (Table 5): Progressive addition of data categories (OCR, Charts, Science/Math, Captioning, Grounding, Text-only, General VQA) produces monotonic average score improvement from 58.8 to 67.0 (+8.2 points total). The largest jumps occur from Naive OCR (+2.5 points, driven by a 130-point OCRBench gain) and Chart/Table/OCR QA (+3.7 points). The diminishing returns in later additions (Captioning +0.5, Grounding +0.3, Text-Only +0.6, General VQA +0.4) suggest that domain-specific data for model weaknesses provides higher marginal benefit than additional data for already-covered domains.
Naive vs. advanced subset selection (Table 6, rows 3 vs. 5): Naive random selection (8.2M samples) achieves 70.6 average score. K-means clustering selection (4.6M samples) achieves 71.8 — better performance with 44% fewer samples. This validates the paper's claim that balanced subtype representation matters more than total sample count. The 27-point OCRBench drop from naive selection (825 → 798) is particularly striking and suggests that random selection disproportionately removes critical OCR samples.
Data formatting and filtering impact (Table 6, row 4): Applying rule-based filtering and formatting to 8.2M samples improves average from 70.6 to 71.2. The 45-point OCRBench gain (798 → 843) is attributed primarily to LaTeX template removal (Table 3). This single formatting correction — removing a fixed equation environment from a subset of LaTeX training samples — accounts for most of the OCRBench improvement, demonstrating that formatting artifacts can have disproportionate effects on benchmark performance.
CoT augmentation effectiveness (Table 6, row 6; Appendix Table M): Adding CoT-augmented training data improves MathVista from 60.5 to 63.5 (+3.0 points). Appendix Table M provides a targeted validation: non-CoT-trained models show no improvement from "Think it step-by-step" prompts (61.0 vs. 60.5 with and without the prompt), while CoT-trained models improve from 63.2 to 63.5 when the prompt is added. This demonstrates that CoT augmentation teaches genuine step-by-step reasoning rather than just formatting mimicry.
Stage-1.5 re-updating (Table 6, row 7): Applying Stage-2-discovered data strategies (filtering, formatting, augmentation) back to Stage-1.5 data improves average from 72.1 to 72.4. Gains are concentrated on ChartQA (+0.8), MMVet (+1.2), and MathVista (+1.7). This validates the "large wheel drives small wheel" feedback architecture (Figure 8).
Mixture of vision encoders (Table 6, row 8): Replacing SigLIP alone with tiled MoVE (SigLIP + ConvNeXt) improves average from 72.4 to 73.5 (+1.1). Gains appear on 12/14 benchmarks, with the largest on InfoVQA (+3.9), OCRBench (+26), and MMMU (+2.5). The paper notes that improvements are concentrated on document, chart, and OCR benchmarks — domains where ConvNeXt's fine-grained spatial features complement SigLIP's semantic features.
Balanced vs. naive data packing (Table N): Balanced packing improves average from 72.8 to 73.5 (+0.7), with notable gains on ChartQA (+1.7), MMMU (+1.4), and MathVista (+2.2). This demonstrates that training infrastructure decisions (packing algorithm design) can have effects on final model quality comparable to data curation decisions, challenging the conventional separation between "training engineering" and "training methodology."
Model scale consistency (Table 7): The Eagle 2 data strategy and training recipe produce consistent improvements across 1B, 2B, and 9B scales. Eagle2-1B outperforms InternVL2-1B on all benchmarks, and Eagle2-2B outperforms InternVL2-2B on 11/13 benchmarks and Qwen2-VL-2B on 11/14. This suggests that the data strategy benefits are not specific to a particular model capacity and transfer across scales.
Internal validation dataset: The paper mentions using the MATH benchmark (500 test questions) for internal validation during development, but does not report systematic MATH ablation results in the main tables. The CoT augmentation validation (Appendix Table M) is performed on MathVista, not MATH.

Critical Assessment

The experiments in this paper collectively make a persuasive case for the effectiveness of systematic data strategy in producing frontier-competitive VLMs, but several important questions about the strength of the evidence, the scope of the claims, and the robustness of the findings deserve careful examination.

Claim from the executive summary: "Eagle2-9B achieves state-of-the-art results across various multimodal benchmarks, matching certain competitive models with up to 70B parameters."

This claim is supported by Table 7, which shows Eagle2-9B outperforming InternVL2-26B on 9/14 benchmarks and competing closely with LLaVA-OneVision-72B (8× parameters) and Llama-3.2-90B (~10× parameters). The claim is specifically about matching — not universally surpassing — larger models, and the data supports this qualified statement: Eagle2-9B wins on OCR-heavy and chart benchmarks but loses on some reasoning benchmarks (MMMU, AI2D) against the 72B model.

However, there is a genuine weakness: the comparison against larger models does not control for total training compute. The paper does not report FLOPs for LLaVA-OneVision-72B or InternVL2-26B, making it impossible to determine whether Eagle2-9B's competitive performance comes from a more efficient training recipe or simply from using more (or less) total compute than the baselines. The claim about matching 70B-parameter models is about parameter efficiency, not compute efficiency — Eagle2-9B uses fewer parameters but the paper provides no evidence that it used fewer total FLOPs to train. It is possible that Eagle2-9B's 28 hours on 256 H100 GPUs for Stage-1.5 is actually more total compute than some larger models' training budgets, or less — we simply cannot tell from the reported data. This is a significant missing piece of the comparison.

Claim from the executive summary: "Eagle2-9B surpasses GPT-4V on ChartQA (86.4 vs. 78.5), OCRBench (868 vs. 656), and MathVista (63.8 vs. 54.7)."

This claim is numerically accurate based on Table 7. The margins are substantial: +7.9 on ChartQA, +212 on OCRBench, +9.1 on MathVista. Eagle2-9B also surpasses GPT-4V on 7 additional benchmarks (DocVQA, InfoVQA, TextVQA, MMStar, RWQA, AI2D, HallusionBench, and OpenCompass). GPT-4V wins clearly on MMMU (61.7 vs. 56.1, -5.6) and MMVet (67.5 vs. 62.2, -5.3). The overall picture is that Eagle2-9B is stronger on perceptual and OCR-heavy tasks while GPT-4V retains an advantage on comprehensive multimodal reasoning (MMMU) and detailed visual explanation (MMVet).

A caveat: GPT-4V's reported numbers come from its 2023 system card [11], and the model has likely been updated since then. The paper uses these as point-in-time comparisons against a known version, which is reasonable practice, but the claim "surpasses GPT-4V" should be understood as referring to the specific 2023 GPT-4V configuration rather than whatever GPT-4V variant is currently deployed. Additionally, GPT-4V's training data and strategy are entirely unknown, so it is impossible to attribute Eagle2-9B's advantages specifically to data strategy rather than to other factors (model architecture, LLM backbone quality, benchmark-specific optimization).

Claim from the executive summary: "Careful data orchestration — rather than model scale alone — can close the gap to proprietary frontier VLMs when the data strategy systematically addresses both diversity and quality across visual instruction domains."

The strongest version of this claim would be that data strategy is the primary driver of the observed improvements, with model scale playing a secondary role. The step-by-step ablation in Figure 2 and Table 6 provides substantial evidence for this: the average score climbs from 58.8 to 73.5 (+14.7 points) almost entirely through data and training recipe changes, with only the final 1.1-point gain coming from the architectural change (tiled MoVE). The data scaling in Table 5 shows a clear 8.2-point improvement from expanding data diversity alone, before any refinement strategies are applied. These trajectories support the paper's core thesis that data decisions are the decisive factor.

However, there is a gap in the evidence: the paper never isolates the effect of the LLM backbone. All experiments use Qwen2.5-7B-Instruct. We cannot know how much of Eagle2-9B's performance comes from Qwen2.5-7B being a particularly strong base model for multimodal post-training, versus the data strategy being universally effective. If the same data pipeline were applied to a weaker LLM backbone, would the improvements be proportionally similar or would they be smaller? The paper does not provide this ablation. The success of the data strategy may be partially contingent on starting from a strong instruction-tuned LLM — an interaction effect that the claims do not address.

Missing experiments:

No ablation of the Qwen2.5 LLM backbone: The paper claims data strategy is the decisive factor, but never tests with a different LLM to show the data strategy works independently of the backbone choice. This leaves open the possibility that Qwen2.5-7B-Instruct has properties (instruction-following ability, reasoning capability, multilingual support) that are essential for the observed performance and that the same data strategy applied to a different LLM would yield different results.
No FLOPs-matched comparison against baselines: The paper claims Eagle2-9B matches 70B models, but never accounts for total training compute. A fairer comparison would report total GPU-hours for Eagle2-9B versus baselines, or — better — conduct a FLOPs-matched comparison where both Eagle2-9B and a scaled-up baseline are trained with equivalent total compute. Without this, the parameter-efficiency claim is strong but the compute-efficiency claim is unverified.
No single-dataset ablation: The paper explicitly states it does not perform per-dataset ablations ("we avoid performing ablation for each dataset individually," Section 2.2). This is a pragmatic choice given 180+ sources, but it means we cannot identify which specific datasets contributed most to the improvements. The category-level ablations in Table 5 provide coarse-grained evidence, but the actual causal contribution of individual datasets is unknown. Some datasets in the collection may be irrelevant or even harmful, but their effects are masked by batch addition.
No test of filtering/formatting strategies in isolation: Table 6 shows "Data Formatting/Filtering" as a single row (improving OCRBench by 45 points), but the paper does not separate the effects of filtering from the effects of formatting. We cannot determine whether the 45-point OCRBench gain came primarily from removing bad samples (filtering) or from fixing format inconsistencies (formatting). The LaTeX template example suggests formatting was important, but the relative contributions are not quantified.
No confidence intervals or statistical testing: All benchmark results are reported as point estimates without any measure of uncertainty — no standard deviations, no confidence intervals, no statistical significance tests. Given that benchmarks like DocVQA and ChartQA have finite test sets (~5,000 questions for DocVQA, ~2,500 for ChartQA), scores have associated sampling variance. The paper reports differences of 0.1-0.3 points between models — these could easily be within random variation. Without uncertainty quantification, many of the fine-grained comparisons between models are not statistically grounded.
No test of the similarity score threshold sensitivity: The paper uses a similarity score cutoff of 0.3 for deciding whether a new data source adds diversity, but never ablates this threshold. Performance could be sensitive to this choice — a higher threshold might admit many near-duplicates, while a lower threshold might reject genuinely useful data.
Limited analysis of failure modes: The paper presents strong benchmark results but provides almost no error analysis or qualitative failure case discussion. For a paper that emphasizes "error analysis" as a key part of the data collection feedback loop (Figure 3, "proactive searching"), the final model's error patterns are not systematically characterized. We do not know what kinds of questions Eagle2-9B still gets wrong, where it hallucinates, or what capability gaps remain — information that would be valuable for future work and would strengthen the paper's methodological contribution.

Conditional nature of the findings:

The paper's findings about data strategy effectiveness are conditional on several factors that are not varied in the experiments:

Starting from a strong instruction-tuned LLM: Qwen2.5-7B-Instruct provides strong language understanding, reasoning, and instruction-following capabilities out of the box. The data strategy builds on these — it does not create them. If the base LLM lacked these capabilities, Stage-1.5 and Stage-2 might not be sufficient to develop them.
Access to 180+ diverse data sources: The paper's data strategy benefits from broad access to publicly available datasets. Research groups with limited data collection resources may not be able to replicate the diversity-first approach, and the paper does not provide guidance on which subsets of the 180+ sources are most essential.
Sufficient compute for Stage-1.5: The 28-hour Stage-1.5 training on 256 H100 GPUs is a substantial compute investment. The paper does not explore how performance scales with Stage-1.5 data volume or training duration — it is possible that similar results could be achieved with less compute, or that further compute would yield further gains.
English-centric evaluation: All 14 benchmarks are in English. The paper mentions multilingual OCR capability (Figure 13) as a demo but provides no quantitative multilingual evaluation. The data strategy's multilingual coverage is concentrated in OCR data (MTWI for Chinese text, MTVQA for multilingual VQA) but is otherwise English-dominant.

Assessment of the benchmark selection:

The 14 benchmarks cover a reasonable range of VLM capabilities (document understanding, chart reading, OCR, visual reasoning, math, hallucination detection), but they are weighted toward text-rich and structured visual understanding tasks. Benchmarks testing fine-grained visual recognition (e.g., fine-grained species classification, medical image interpretation), spatial reasoning (e.g., relative position, 3D understanding), temporal reasoning (video understanding), and open-ended generation quality (beyond MMVet's scoring) are absent. Eagle2-9B's particularly strong performance on OCRBench (+212 over GPT-4V) and consistently dominant chart/document results might partly reflect a benchmark portfolio that favors the paper's data strategy strengths (aggressive OCR and chart data collection) rather than balanced VLM capability. The paper does not discuss whether the benchmark selection might bias the apparent effectiveness of the data strategy.

Overall assessment: The experiments provide strong evidence that the documented data strategy — diversity-first collection, filtering, K-means-based subset selection, CoT augmentation, formatting standardization, three-stage training with a feedback loop between stages, and balanced data packing — produces a 9B VLM that is competitive with much larger models and outperforms same-scale peers across a range of benchmarks. The evidence is weaker for the claims that data strategy is the primary causal factor (rather than the Qwen2.5 backbone) and that the approach is compute-efficient (rather than parameter-efficient). The lack of uncertainty quantification, per-dataset ablations, and backbone variation experiments limits the strength of causal attribution, and the benchmark portfolio's OCR/chart emphasis may favor the paper's specific data strategy choices. These limitations do not undermine the paper's main practical contribution — a reproducible recipe for building competitive VLMs — but they do constrain the generality of the methodological claims about data strategy being universally decisive.

6. Limitations and Trade-offs

6.1 Difficulty Estimation Cost Is Unaccounted for in the Data Collection Feedback Loop

The assumption or constraint. The paper's data collection strategy relies on error analysis to guide proactive searching for new datasets: "for each update of the data pool, we generate error analysis to identify model weaknesses and perform targeted searches for new data" (Section 2.2, Figure 3). This implies running full training cycles, evaluating the resulting model on a broad benchmark suite, and manually inspecting failure cases to determine which capability gaps are most limiting and which new datasets might address them. The paper does not account for this cost in any of its reported training budgets (Table A: 28 hours for Stage-1.5, 6 hours for Stage-2 on 256 H100 GPUs for the 9B model). The development process that produced the final model involved "iterating Stage-2 > 10 times" before each Stage-1.5 update (Appendix A), meaning the true total compute cost — including all intermediate training runs for error analysis, ablations, and data strategy testing — is substantially higher than the final training budget alone. The paper acknowledges a related cost concern in a different context: "estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity" (Section 3.2), but does not extend this transparency to the data development cycle itself.

The consequence. A practitioner attempting to replicate or adapt Eagle 2's methodology for a different domain, language, or model family would discover that the paper provides a description of what data ended up in the final training mixture, but not a cost-effective procedure for arriving at that mixture from scratch. The error-analysis-driven search requires multiple full training runs to identify capability gaps, followed by targeted data collection, followed by retraining to verify improvements. Without the pre-existing knowledge of which datasets matter (which the paper's authors acquired through the iterative process but do not fully codify into a decision procedure), a new practitioner would need to recreate much of this expensive search. The paper's similarity score metric (Equation 1) helps screen candidate datasets for diversity, but it cannot predict whether a diverse new dataset will address a specific model weakness — that requires the error analysis loop the paper describes but does not budget for.

What evidence exists in the paper. The paper is transparent about the iterative development process (Section 3.1, Tables 5 and 6, Figure 2) and the "large wheel drives small wheel" feedback architecture (Figure 8, Appendix A noting ~10 Stage-2 iterations per Stage-1.5 update). However, it never tallies the total GPU-hours spent on all intermediate runs, failed data experiments, or the error analysis process itself. Table A reports only the final training run's compute (28 + 6 = 34 hours on 256 H100 GPUs for Stage-1.5 and Stage-2), not the iterative development cost. The paper also notes that "we avoid performing ablation for each dataset individually" due to training costs (Section 2.2), which partially mitigates the problem but also means the contribution of individual datasets is unknown — a practitioner cannot prioritize which of the 180+ sources to include if operating under tighter compute constraints.

Mitigation status. Not addressed. The paper does not propose a method for reducing the cost of the error-analysis-driven development cycle, nor does it provide a ranked list of dataset importance that would allow practitioners to prioritize data collection. The three-stage recipe with rapid Stage-2 iteration (6 hours per cycle) is presented as an efficiency improvement over two-stage training, but the total number of cycles and their cumulative cost remain unquantified. The paper's framing — "we aim to 'teach readers to fish than to give them a fish'" (Section 1.4) — is admirable, but the fishing lesson comes with an unstated tuition cost that limits its practical accessibility.

6.2 Single LLM Backbone, Single Benchmark Language — No Evidence of Cross-Model or Cross-Lingual Generalization

The assumption or constraint. All experiments use the Qwen2.5-7B-Instruct (or its 0.5B and 1.5B variants) as the LLM backbone (Table 4). The paper explicitly states that "assuming the same pre-trained LLM backbone, we posit that data is the most decisive factor to obtain high-quality models" (Section 1.1). This is a controlled-variable claim: given a fixed strong LLM, data strategy drives performance differences. But the paper never tests this assumption by applying the same data pipeline to a different LLM backbone. Similarly, all 14 evaluation benchmarks are in English (Table 7), and while the paper demonstrates multilingual OCR capability qualitatively (Figure 13, "Multilingual Text Recognition"), it provides no quantitative multilingual evaluation. The data composition includes some Chinese-language datasets (LLaVA-150K-CN, Chinese-Meme, MTWI, MTVQA) and OCR data in multiple scripts, but the training data remains overwhelmingly English-dominant.

The consequence. A practitioner working with a different LLM backbone — Llama 3, Mistral, Gemma, DeepSeek, or a non-English-centric model — cannot predict whether Eagle 2's data strategy will transfer effectively. The Qwen2.5 model family may have specific properties that make it particularly amenable to the three-stage post-training approach: strong instruction-following from the Instruct fine-tuning, robust multilingual capabilities that support the diverse OCR data, or architectural properties that interact favorably with the tiled MoVE design. If a weaker or differently-trained LLM backbone were used, the same data strategy might produce substantially different results, or the data composition might need to be rebalanced (e.g., more text-only data to prevent catastrophic forgetting in a weaker base model). For non-English deployment, the paper provides no evidence that the predominantly English training data produces models that can reason, answer questions, or follow instructions in other languages beyond recognizing text in images.

What evidence exists in the paper. The paper demonstrates consistency across three scales of the same model family (Qwen2.5-0.5B, 1.5B, 7B) in Table 7, showing that Eagle2-1B and Eagle2-2B also outperform same-scale competitors. This provides evidence of within-family scalability but not cross-family transfer. The demo figures (Figures 12-18) include examples of multilingual OCR and mathematical reasoning borrowed from other papers' demo sets, but these are cherry-picked qualitative examples, not systematic evaluations. The paper does not report performance on multilingual benchmarks like MMMB (Multilingual MMBench), xGQA, or multilingual DocVQA variants, despite collecting multilingual training data (MTVQA, Chinese-Meme, LLaVA-150K-CN).

Mitigation status. Not addressed. The paper does not acknowledge the single-backbone limitation or call for cross-model validation in future work. The multilingual limitation is implicitly acknowledged through the absence of multilingual evaluation results, but the paper neither discusses it as a limitation nor proposes multilingual benchmarks for future evaluation. Given the paper's stated goal of benefiting "the development of competitive models for the open-source community" (Abstract), the lack of backbone diversity testing is a significant gap: open-source practitioners use diverse LLM backbones, and knowing whether Eagle 2's data strategy works across them is essential for practical adoption.

6.3 Benchmark Portfolio Skews Toward Text-Rich and Perceptual Tasks — Reasoning and Open-Ended Generation Gaps Remain

The assumption or constraint. The paper's evaluation spans 14 benchmarks (Table 7), but these are weighted toward text-rich visual understanding (DocVQA, ChartQA, InfoVQA, TextVQA, OCRBench — 5 of 14 are directly text/document-focused) and perceptual QA tasks. The model's largest performance advantages over closed-source frontier models appear in these domains: +212 points over GPT-4V on OCRBench, +7.9 on ChartQA, +5.0 on TextVQA. However, on benchmarks requiring deeper multimodal reasoning (MMMU) and detailed open-ended visual explanation (MMVet), Eagle2-9B trails GPT-4V by -5.6 and -5.3 points respectively, and trails GPT-4o by -13.1 and -6.9 points. The benchmark portfolio does not include tasks measuring fine-grained visual recognition, spatial/temporal reasoning, video understanding, or open-ended generation quality beyond MMVet's scoring rubric. MMMU itself is acknowledged as a "challenging benchmark" where the paper "observe[s] considerable performance fluctuations" during data scaling experiments (Section 3.1).

The consequence. The paper's headline claim — "Eagle2-9B achieves state-of-the-art results across various multimodal benchmarks, matching certain competitive models with up to 70B parameters" — is narrowly true within the selected benchmark set, but the selection may be systematically favorable to the paper's specific data strategy. The data collection was explicitly driven by error analysis and proactive searching, which likely identified OCR, chart, and document understanding as weaknesses and directed data collection toward those domains. This is a legitimate development strategy, but it means the final model's capability profile reflects the priorities of the development process, not necessarily balanced multimodal intelligence. A practitioner choosing between Eagle2-9B and Qwen2-VL-7B (which leads on MMMU at 54.1 vs. 56.1 for Eagle2-9B after MoVE — actually Eagle2 leads here, but Qwen2-VL leads on TextVQA 84.3 vs. 83.0 and MMBench v1.1 81.0 vs. 81.9 — Eagle2 leads) might prefer Qwen2-VL if their application emphasizes comprehensive reasoning over OCR, or vice versa. Without evaluation on broader reasoning, spatial, and generation benchmarks, the scope of Eagle2-9B's capabilities is incompletely characterized.

What evidence exists in the paper. The performance profile is clearly visible in Table 7: Eagle2-9B dominates on document/chart/OCR tasks but shows smaller margins or losses on MMMU and MMVet. The ablation in Table 6 shows that the tiled MoVE architecture provides a +2.5 point MMMU boost, suggesting that the paper was aware of MMMU as a weakness and addressed it partially through architecture rather than data. The paper does not report results on common reasoning benchmarks like MMBench-CC (Chinese), SEED-Bench, LLaVA-Bench (In-the-Wild), or video benchmarks, all of which are standard in contemporary VLM evaluations (Qwen2-VL, InternVL2, and LLaVA-OneVision all report on broader sets). The absence of these benchmarks makes it difficult to assess whether Eagle2-9B's competitive performance is genuinely broad or concentrated in the text-rich perceptual domains where its data strategy invested most heavily.

Mitigation status. Partially acknowledged through the data itself — the paper's Figure 4 shows data distributions that are heavy on OCR QA, chart, and text-only data, and Stage-2 specifically reduces captioning data "primarily due to concerns about the overly monotonous instructions" (Section 2.2). This reveals that the data strategy involves explicit tradeoffs that shape the model's capability profile, but the paper does not frame these tradeoffs as limitations or discuss what capabilities may have been sacrificed. The paper does not suggest future work on balancing the benchmark portfolio or evaluating on broader task sets.

6.4 No Isolation of Filtering, Formatting, and Augmentation Contributions — the 45-Point OCRBench Gain Is Not Attributed

The assumption or constraint. Table 6 reports "Data Formatting/Filtering" as a single row, improving the average score from 70.6 to 71.2 and OCRBench from 798 to 843 (+45 points). The paper attributes this gain primarily to LaTeX template removal (Section 2.2, Table 3) — a formatting fix — but also describes several filtering operations (removing mismatched QA pairs, irrelevant image-question pairs, repeated texts, numeric formatting issues, refusal responses) that were applied simultaneously. Similarly, "Data Augmentation" is reported as a single row applying CoT generation, rule-based QA expansion, and short-to-long answer conversion together. No ablation separates the individual contributions of filtering vs. formatting within the combined step, or CoT augmentation vs. rule-based augmentation vs. answer expansion within the augmentation step.

The consequence. A practitioner with limited engineering resources cannot determine which specific data refinement operation to prioritize. If the 45-point OCRBench gain comes entirely from removing one LaTeX template artifact (as the paper's narrative suggests), then the filtering operations (removing bad QA pairs, irrelevant images, etc.) might have contributed much less than the combined row implies. Conversely, if the filtering operations were actually responsible for most of the gain, the LaTeX example may be an illustrative anecdote rather than the primary mechanism. The paper's stated goal of transparency — sharing "detailed insights into the development processes" (Abstract) — is partially undermined when operations with very different costs and complexities (writing a regex to strip \begin{align*} vs. building a multi-rule filtering pipeline vs. training a verification model for CoT correctness) are bundled into single rows. This makes the ablation less actionable than it could be.

For data augmentation specifically, Appendix Table M provides strong evidence that CoT augmentation improves MathVista performance and that the improvement translates to better CoT prompt utilization. But the rule-based QA generation and short-to-long answer expansion are not separately evaluated, leaving their individual contributions unknown. The paper notes that rule-based augmentation "brings 1 point improvement on ChartQA" (Section 3.1), suggesting it was tracked internally but not reported in a disaggregated ablation.

What evidence exists in the paper. The LaTeX template removal example (Table 3, Section 2.2) provides qualitative evidence for formatting's importance. The CoT augmentation validation (Appendix Table M) provides quantitative evidence for that specific augmentation. But no table separates the "Data Formatting/Filtering" row into "Data Formatting only" and "Data Filtering only" sub-rows, nor disaggregates the three augmentation methods.

Mitigation status. Not addressed. The paper does not acknowledge this bundling as a limitation, does not provide disaggregated ablations in an appendix, and does not suggest future work on isolating the contributions of individual data refinement operations. The bundling is understandable given the paper's practical focus on final model quality rather than scientific decomposition of each technique, but it limits the paper's value as a prescriptive guide for practitioners who need to allocate limited engineering effort across these operations.

6.5 The Baseline Comparison Overstates Gains by Starting from a Deliberately Weakened Starting Point

The assumption or constraint. The paper's central improvement narrative — progressing from an average score of 58.8 (Eagle2-Baseline) to 73.5 (Eagle2-9B final) — uses a baseline that the authors constructed by removing known low-quality subsets from Cambrian-7M. Specifically: "We remove some low-quality data from Cambrian-7M, such as ShareGPT-4V, GPT-77K and Data-Engine-161K, ultimately resulting in a subset of 5.2M samples" (Section 2.1). The baseline also uses a different (older) training recipe (LLaVA's two-stage training), a single vision encoder (SigLIP only), and does not include the diverse data sources that the paper later adds. The final Eagle2-9B model uses a three-stage recipe, tiled MoVE, and 180+ data sources. The 14.7-point improvement is therefore the combined effect of (a) adding diverse data, (b) switching training recipes, (c) changing the vision encoder architecture, (d) applying filtering/formatting/augmentation, and (e) implementing balanced packing — but the relative contribution of each is confounded because the baseline differs from the final model on all dimensions simultaneously.

The consequence. The paper's core claim — that data strategy is the decisive factor — is supported by the step-by-step ablation in Figure 2 and Table 6, which shows that most of the 14.7-point gain accumulates before the MoVE architectural change. However, the starting point for this accumulation is itself a product of data strategy decisions (removing known low-quality subsets). If the baseline had started from a stronger data mixture — for example, the full Cambrian-7M or a more competitive open-source baseline like LLaVA-OneVision-7B's training data — the absolute gains from subsequent data additions might have been smaller. The paper never reports what performance would be achieved by simply adding the diverse data sources to the original Cambrian-7M without removing subsets, or what the two-stage baseline would achieve with the full 180+ source data pool. This makes it difficult to determine whether the removal of low-quality data, the addition of diverse new data, or the interaction between them is the primary driver of the observed gains. A practitioner starting from a stronger initial data pool (e.g., LLaVA-OneVision's curated data) might see smaller marginal returns from applying Eagle 2's data strategy.

What evidence exists in the paper. Table 5 provides the initial data scaling from the 5.2M baseline, showing an 8.2-point improvement from adding diverse data in a two-stage setting. This demonstrates that data diversity helps even within the weaker training recipe. But the paper does not report a baseline using the full Cambrian-7M (without removing ShareGPT-4V, etc.) to show whether the subset removal was beneficial, harmful, or neutral. The nearest comparison is Cambrian-1-7B's reported scores in Table 7, which show slightly different numbers than Eagle2-Baseline (e.g., DocVQA 77.8 for Cambrian-1-7B vs. 77.0 for Eagle2-Baseline, MMStar 50.7 vs. 47.3, AI2D 74.6 vs. 74.7), suggesting the subset removal had mixed effects. Cambrian-1-7B used 7M samples (vs. Eagle2-Baseline's 5.2M) and a different LLM backbone, so the comparison is not straightforward.

Mitigation status. Partially addressed by the paper's transparency about the baseline construction. The paper explicitly states what was removed and why (Section 2.1), allowing readers to understand the starting point. The step-by-step ablation (Tables 5 and 6) provides coarse-grained evidence about which changes matter. But the paper does not discuss the possibility that starting from a weaker baseline inflates the apparent effect size of the data strategy, nor does it provide a sensitivity analysis showing how the gains would change if starting from a different initial data pool.

6.6 Data Packing Improvement Is Validated Only on the Final Model — No Cross-Stage or Cross-Scale Validation

The assumption or constraint. The balanced-aware data packing algorithm is evaluated in a single ablation: comparing naive greedy packing against balanced packing using the final Eagle2-9B model configuration (Table N). The ablation shows a +0.7 average score improvement (72.8 → 73.5) from switching packing methods. However, this comparison is performed only at the final stage of development — after all other data strategy refinements, the three-stage recipe, and the MoVE architecture are in place. The paper does not test whether balanced packing provides similar benefits at earlier stages of development (e.g., during two-stage training in Table 5, or during Stage-1.5 pretraining where data volume is 21.6M and packing behavior might differ), or whether the benefit transfers to the smaller model scales (1B, 2B) that use different training configurations (single vision encoder, higher learning rates).

The consequence. A practitioner implementing balanced packing for their own VLM training cannot predict whether the benefit is robust across training stages, data scales, and model sizes, or whether it is specific to the high-quality, carefully curated Stage-2 data distribution. If balanced packing primarily helps by rebalancing loss weights between long reasoning samples (CoT-augmented math and science data) and short direct-answer samples (simple VQA), its benefit might be smaller in Stage-1.5 where the data distribution is different (heavier on captioning, less curated, different length distributions). Conversely, if balanced packing is universally beneficial, the paper's failure to apply it during Stage-1.5 training (which uses 21.6M samples and 28 hours of compute) means the Stage-1.5 checkpoint might have been suboptimal, and the reported Stage-1.5 → Stage-2 improvement trajectory might understate what balanced packing could achieve if applied consistently. The paper also does not compare the balanced packing method against no packing at all (training on individual samples with padding), so the absolute benefit of packing (naive or balanced) versus unpacked training is not measured — only the relative benefit of balanced over naive packing.

What evidence exists in the paper. Table N provides a single data point: Eagle2-9B final model, naive vs. balanced packing. Figure 9 visualizes the difference in pack length distributions between the two methods. The paper does not report an ablation of balanced packing for Stage-1.5 training, for the 1B or 2B models, or for an unpacked baseline. The paper states that packing "accelerates the training by 2-3 times" (Section 2.3), suggesting unpacked training is significantly slower, but does not quantify the quality difference.

Mitigation status. Not addressed. The paper does not acknowledge the limited scope of the packing ablation, does not discuss whether balanced packing was used during Stage-1.5 (the training configuration in Table 4 does not specify the packing method per stage), and does not suggest cross-stage or cross-scale validation as future work. Given that the packing improvement (+0.7 average) is comparable in magnitude to several data augmentation and subset selection gains reported in Table 6, understanding its robustness across training configurations is important for practitioners deciding whether to adopt the method in their own pipelines.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper represents a methodological shift in norms, not a paradigm shift in techniques — it demonstrates that frontier-competitive VLM post-training can be achieved through fully documented, systematic data engineering, challenging the implicit assumption in the field that leading-edge performance requires opaque, proprietary data operations. The significance lies in what kind of contribution the paper makes, rather than in any single novel algorithm or architecture.

The core reframing: data strategy as a first-class, publishable research contribution. Before Eagle 2, the VLM community operated with a tacit understanding: Tier 1 models (GPT-4V/4o, Claude, Gemini) are black boxes whose data recipes are trade secrets; Tier 2 models (Qwen2-VL, InternVL2, Llama 3.1) release weights but provide only high-level sketches of their data approach; and Tier 3 models (Cambrian-1, LLaVA) release data recipes but lag in performance, creating an implicit divide between "competitive" and "transparent." Eagle 2 breaks this divide by demonstrating that a model matching GPT-4V on 10 of 14 benchmarks (and substantially exceeding it on OCRBench by 212 points, ChartQA by 7.9 points, and MathVista by 9.1 points — Table 7) can be built with a fully documented data pipeline: exact per-dataset sample counts across two training stages (Appendix Tables B-J), the CoT augmentation prompts (Listings 1-3), the correctness verification procedure, the similarity score metric with operational thresholds (Equation 1, cutoff of 0.3), the K-means clustering selection protocol, the balanced packing algorithm with Python code (Figure 10, Listing 4), and the iterative development trajectory (Tables 5 and 6, tracking the 14.7-point improvement from baseline 58.8 to final 73.5).

This matters because it changes what the community can reasonably expect from model releases. If Eagle 2 can achieve competitive performance with full transparency, the argument that competitive pressure requires secrecy becomes empirically weaker. The paper demonstrates that there is no inherent trade-off between frontier performance and documentation thoroughness — the bottleneck has been the willingness to document, not any technical limitation that makes documentation and performance incompatible. This is a normative shift: the paper implicitly argues that model weights without data strategy documentation should be viewed as incomplete research contributions, and that the field should reward transparency as a scientific virtue alongside benchmark scores.

Reconciling contradictory prior findings. The paper does not explicitly frame itself as resolving contradictions, but its findings provide a unifying lens for several tensions in the VLM literature:

Why do some open-source models underperform despite strong LLM backbones? The paper cites the observation that "LLaVA-OneVision-72B still ranks slightly behind InternVL2-40B despite having a stronger LLM backbone" (Section 1). Eagle 2's results suggest that data strategy — the diversity, filtering, formatting, and staging of training data — can outweigh LLM backbone quality in determining final VLM performance. InternVL2-40B's advantage over LLaVA-OneVision-72B may reflect superior data orchestration rather than anything architectural. Eagle 2 demonstrates that a 7B-parameter Qwen2.5 backbone with optimized data strategy can match LLaVA-OneVision-72B (8× parameters) on a majority of benchmarks, providing concrete evidence for this interpretation.
Does intermediate pretraining help or does it waste compute? The VLM community has oscillated between two-stage recipes (LLaVA's connector alignment + SFT) and multi-stage recipes (LLaVA-OneVision's additional pretraining stages). Eagle 2 provides a clear resolution: intermediate pretraining (Stage-1.5) helps substantially, but only if it uses all visual instruction data rather than just captioning/knowledge data. The Stage-1.5 checkpoint alone achieves 69.7 average score — already 2.7 points above the best two-stage model (67.0) trained with extensive SFT data (Table 6). And Stage-2 on top of Stage-1.5 reaches 70.9, which improves to 72.4 after the feedback cycle. This demonstrates that Stage-1.5 is not merely an additional training phase — it is specifically the broad, diverse instruction data allocation in Stage-1.5 that makes it effective, resolving the ambiguity in prior work about what kind of data intermediate stages should contain.
Is data quantity or data quality more important? The paper provides a nuanced empirical answer: both matter, but at different stages and for different reasons. Stage-1.5 benefits from quantity and diversity (21.6M samples from 180+ sources, achieving 69.7 alone). Stage-2 benefits from quality and curation (4.6M carefully selected samples achieving 72.4 when combined with a strong Stage-1.5 foundation). Within Stage-2, the advanced subset selection experiment (Table 6, rows 3-5) shows that better curation (K-means clustering selection) with half the data (4.6M vs. 8.2M) produces better results (71.8 vs. 70.6). And the data formatting/filtering experiment shows that fixing quality issues in existing data (45-point OCRBench gain) can be more impactful than adding new data. This multi-faceted answer — quantity for foundation, quality for specialization, formatting as a hidden lever — resolves the false dichotomy between "more data" and "better data" that has characterized prior debates.

Research directions that become more attractive:

Data-centric VLM development as a primary research methodology. Eagle 2 makes data strategy a legitimate, publishable, competitive research contribution. This should encourage more groups to invest in systematic data exploration rather than racing to propose new architectures, and to publish detailed data strategies alongside model weights. The paper's framework (similarity score for diversity screening, K-means for balanced selection, rule-based filtering with documented error categories, CoT augmentation with verification) provides a template that others can adopt and extend.
Understanding the interaction between LLM backbone properties and data strategy effectiveness. The paper's results are specific to Qwen2.5-7B-Instruct. A natural next step — applying the same data pipeline to Llama 3, Mistral, Gemma, or DeepSeek backbones — would reveal whether the data strategy's benefits are universal or backbone-dependent, and what properties of the base LLM (instruction-following, multilingual capability, reasoning depth) are prerequisites for the three-stage recipe to work.
Automated data quality assessment for VLM training. The paper's manual error analysis and rule-based filtering were effective but labor-intensive. The documented error categories (mismatched QA pairs, irrelevant image-question pairs, repeated texts, numeric formatting issues — Figures 5 and 6) provide a labeled taxonomy that could be used to train automated quality classifiers, making the filtering process scalable and reproducible across different data sources.
Data packing as a first-class optimization target. The paper's demonstration that balanced packing provides +0.7 average score improvement (Table N) — comparable to adding data augmentation (+0.3 from 71.8 to 72.1) — elevates packing from "training infrastructure" to "training methodology." This should encourage systematic exploration of packing strategies (beyond greedy variants), potentially including learned packing algorithms that optimize for specific training objectives.

Research directions that become less attractive:

Pure architectural innovation without data strategy documentation. If data strategy is the decisive factor, then proposals for new VLM architectures that do not control for or document their data pipeline become less informative. A new connector design or vision encoder may appear beneficial simply because it was tested with better-curated data. The field should increasingly expect data strategy transparency as a prerequisite for credible architectural comparisons.
Scaling model parameters as the default path to improvement. Eagle2-9B matches internVL2-26B and LLaVA-OneVision-72B on most benchmarks (Table 7), demonstrating that a 7B model with optimized data can compete with models 3-10× larger. This does not make scaling irrelevant — the paper does not test whether scaling with the same data strategy would yield further gains — but it shifts the burden of proof: before scaling parameters, practitioners should first exhaust data strategy improvements, which may provide comparable benefits at lower compute cost.

Follow-Up Research This Work Enables

1. Cross-backbone replication: does Eagle 2's data strategy transfer to Llama 3, Mistral, or Gemma backbones? The paper's claim that "assuming the same pre-trained LLM backbone, we posit that data is the most decisive factor" (Section 1.1) is tested only with Qwen2.5. A strong follow-up would apply the identical data pipeline — same 180+ source collection, same three-stage recipe with identical data allocations, same filtering/formatting/augmentation procedures — to Llama-3.1-8B-Instruct, Mistral-7B-Instruct, and Gemma-2-9B-IT, then evaluate on the same 14-benchmark suite. If all backbones achieve similar relative improvements (e.g., comparable gains over their respective two-stage baselines), that would validate the universality of the data strategy and establish it as backbone-agnostic best practice. If Qwen2.5 benefits disproportionately, that would reveal backbone-specific properties (instruction-following quality, multilingual pretraining, attention architecture) that interact with data strategy effectiveness — equally valuable knowledge for practitioners choosing base models.

2. Disaggregating the 45-point OCRBench gain: what fraction comes from LaTeX template removal vs. general filtering? The paper bundles "Data Formatting/Filtering" into a single ablation row (Table 6), attributing the dramatic 45-point OCRBench improvement (798 → 843) primarily to LaTeX template removal (Section 2.2, Table 3) but without isolating this from the simultaneous removal of mismatched QA pairs, irrelevant image-question pairs, and repeated texts. A targeted follow-up would replicate the formatting/filtering step in three variants: (a) LaTeX template removal only, (b) all other filtering rules only (no template removal), and (c) both combined (the paper's reported configuration). The OCRBench scores for each variant would reveal whether the 45-point gain is genuinely attributable to a single formatting fix (suggesting that targeted format auditing of training data is a high-leverage activity) or is distributed across multiple filtering operations (suggesting that systematic quality filtering is broadly important and the LaTeX case is an illustrative but not dominant example). This matters for practitioners allocating limited data engineering effort: if a single regex fix accounts for most of the gain, the lesson is "audit your data for format artifacts"; if multiple filters each contribute incrementally, the lesson is "invest in comprehensive filtering infrastructure."

3. Scaling the data strategy: what happens if Stage-1.5 data is doubled or halved? The paper fixes Stage-1.5 at 21.6M samples and Stage-2 at 4.6M samples based on iterative development, but never ablates these quantities. A natural follow-up would train Eagle 2 variants with Stage-1.5 data volumes of 10M, 21.6M (the original), and 40M samples — keeping Stage-2 fixed at 4.6M — and measure the performance scaling curve. This would reveal whether the 21.6M point is near saturation (diminishing returns) or whether further data collection would yield substantial gains. Similarly, fixing Stage-1.5 at 21.6M and varying Stage-2 from 2M to 8M samples (with appropriate subset selection) would characterize the specialization scaling behavior. The paper notes that "our experiments' costs have risen sharply, and the efficiency of data iteration has decreased" beyond 10M samples in the two-stage setting (Section 3.1), but does not provide this analysis for the three-stage recipe. Such scaling curves would be practically valuable for groups with different compute budgets, and scientifically valuable for understanding whether the three-stage recipe changes the data scaling exponent compared to two-stage training.

4. Balanced packing across training stages: does it help during Stage-1.5 pretraining or only during Stage-2 fine-tuning? Table N validates balanced packing only on the final Eagle2-9B configuration after all three stages. A targeted ablation would apply balanced vs. naive packing during Stage-1.5 (21.6M samples, 28 hours of training) and evaluate the resulting Stage-1.5 checkpoint on the full benchmark suite — before any Stage-2 fine-tuning. If balanced packing during Stage-1.5 provides comparable benefits (+0.7 average score or more), that would establish it as a general training infrastructure improvement applicable to any large-scale VLM pretraining. If the benefit is negligible during Stage-1.5 but substantial during Stage-2, that would suggest the mechanism is specific to curated, high-quality data distributions (where long CoT samples and short direct-answer samples coexist and loss-weight balancing matters more). Additionally, testing balanced packing on the 1B and 2B model scales — which use different training configurations (single vision encoder, higher learning rates, Table 4) — would test whether the benefit is scale-invariant.

5. The similarity score as a data collection tool: does filtering by similarity score alone match human curation? The paper introduces the similarity score (Equation 1) as a rapid screening metric but uses it alongside manual inspection and batch-addition experiments. A validation study would operate the similarity score in isolation: for each new candidate data source, accept it if its similarity score is below 0.3, reject it otherwise — with no manual review — and train models using this automated data collection protocol. Compare the resulting model against the manually curated Eagle2-9B on the full benchmark suite. If automated collection matches or approaches manual curation, the similarity score becomes a powerful tool for scaling data collection to thousands of sources without proportional increases in human effort. If it underperforms, analysis of the failure cases (datasets incorrectly rejected or incorrectly accepted) would refine the metric — perhaps by category-specific thresholds, multi-modal embedding choices beyond SSCD and all-mpnet-base-v2, or incorporating task-type metadata.

6. Stress-testing the "diversity first, then quality" ordering: does quality-first produce a different capability profile? The paper's organizing principle is to maximize coverage breadth first, then refine quality within that coverage. A counterfactual experiment would reverse this: start with a core set of verified high-quality datasets (the final Stage-2 4.6M subset), apply all filtering/formatting/augmentation, then progressively add diverse data sources in the same order as Table 5 and Table 6. Train models at each step and compare against Eagle 2's reported trajectory. If quality-first reaches similar final performance but with fewer intermediate training runs (because the core model is already strong and new data can be evaluated more efficiently), that would challenge the paper's ordering principle and suggest a more computationally efficient development strategy. If quality-first plateaus lower — because the model's capability boundaries are set early by data coverage and cannot be fully recovered by later diversification — that would strengthen the paper's claim that diversity must come first. This experiment addresses a practical question: for a new domain where no large diverse data pool exists, should practitioners invest first in broad collection or in careful curation of a smaller set?

Practical Applications and Downstream Use Cases

1. Rapid prototyping of domain-specific VLMs through data strategy transfer. An organization needing a VLM specialized for a particular domain — medical image understanding, legal document analysis, architectural diagram interpretation, retail product catalog QA — can adopt Eagle 2's data strategy template rather than designing a training pipeline from scratch. The recipe is concrete: (a) collect domain-specific datasets using the passive + proactive search methodology, screening candidates with the similarity score against existing general-domain data; (b) apply the four documented filtering rules to remove low-quality samples; (c) use K-means clustering on domain-appropriate image embeddings for balanced subset selection (the paper notes this "excels with mathematical, medical, and document-based data"); (d) apply CoT augmentation using a strong VLM with the provided prompts (Listings 1-2) to generate step-by-step reasoning from short answers; (e) train with the three-stage recipe, seeding Stage-1.5 with the paper's general 21.6M data plus the new domain data, then fine-tuning Stage-2 on a curated domain-specific subset. The Eagle 2 checkpoints (available on HuggingFace) provide a strong starting point for Stage-1.5, reducing the compute needed for domain adaptation. The expected benefit is measured in the paper's own trajectory: domain-specific data additions (OCR, charts, science/math) produced large benchmark-specific gains (OCRBench +130 points from Naive OCR addition, ChartQA +12.4 points from chart data addition — Table 5), suggesting that targeted data collection following the paper's protocol can rapidly build capabilities in new domains.

2. Cost-efficient deployment through smaller models with optimized data rather than larger models with generic data. For organizations deploying VLMs at scale where inference cost scales with model size, Eagle 2's results provide a concrete economic argument: Eagle2-9B (based on Qwen2.5-7B) matches InternVL2-26B on 9/14 benchmarks, LLaVA-OneVision-72B on 7/14 benchmarks, and Llama-3.2-90B on a majority of reported benchmarks (Table 7). In a production setting processing millions of images per day, the difference between running a 9B model and a 72B model translates to approximately 8× reduction in inference compute and latency. If the deployment domain aligns with Eagle 2's strengths — document processing, chart interpretation, OCR-heavy applications, visual question answering with factual answers — an organization can achieve frontier-competitive accuracy at a fraction of the serving cost by investing in data strategy rather than model scale. The paper's detailed data documentation means the training recipe can be replicated or adapted, and the availability of Eagle 2 model weights on HuggingFace means practitioners can start from the released checkpoints and fine-tune on proprietary data using the documented Stage-2 protocol (6 hours on 256 H100 GPUs, Table A).

3. Automated training data quality auditing for large-scale VLM data pipelines. The paper's four documented error categories — mismatched QA pairs (Figure 5a), irrelevant image-question pairs (Figure 5b), repeated texts (Figure 5c), and numeric formatting issues (Figure 6) — along with the additional keyword-based refusal detection (Appendix B.2), provide a concrete checklist for automated data quality assessment in any VLM training pipeline. A data engineering team can implement rule-based detectors for each category (regex patterns for repeated n-grams, decimal precision thresholds for numeric anomalies, embedding similarity thresholds for image-question relevance, keyword lists for refusal responses) and integrate them as pre-training validation gates. The paper's evidence that these filters matter is unusually strong: applying them produced a 45-point OCRBench gain (Table 6) and improvements on 8 of 14 benchmarks. The LaTeX template example specifically demonstrates that format-level auditing — checking for unintended structural artifacts in training answers (like fixed equation environments, extraneous markdown, inconsistent punctuation) — can prevent the model from learning spurious output patterns that catastrophically degrade benchmark performance. This is a low-cost, high-impact practice that any team training VLMs on public data can adopt immediately, without requiring the full Eagle 2 data collection and staging infrastructure.

4. Iterative data development as an organizational workflow. The paper's "large wheel drives small wheel" feedback architecture (Figure 8) — where a slow, broad Stage-1.5 (28 hours on 256 H100s) provides a stable foundation, and a fast Stage-2 (6 hours on 256 H100s) enables rapid data experimentation, with discoveries from Stage-2 propagating back to improve Stage-1.5 — provides a template for organizing VLM development teams. The key operational insight is the asymmetry: Stage-2 is iterated ~10× for each Stage-1.5 update (Appendix A). This means a team can test 10 different data filtering rules, augmentation strategies, or subset selection methods in the time it takes to run one full Stage-1.5 retraining. The workflow is: (1) train Stage-1.5 once (or start from a public checkpoint), (2) run many Stage-2 experiments varying data strategies, (3) identify which strategies improve validation metrics, (4) apply those strategies to the Stage-1.5 data and retrain Stage-1.5, (5) repeat. This is a concrete project management pattern, not just a training recipe — it specifies how to allocate engineering effort (most time on Stage-2 experimentation, occasional Stage-1.5 updates) and compute resources (Stage-2 experiments are ~4.7× cheaper than Stage-1.5 training). Organizations building VLMs for multiple domains or languages can parallelize Stage-2 experiments across domains while sharing a common Stage-1.5 foundation, further amortizing the Stage-1.5 cost.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Marks2025· multimodal← in-domain predecessor
- Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos2025· multimodal← in-domain predecessor
- VADER: Towards Causal Video Anomaly Understanding with Relation-Aware Large Language Models2025· multimodal← similar topic
- UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Driving2025· multimodal← similar topic
- 4D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillation2025· multimodal← similar topic
Nearest Neighbor Speculative Decoding for LLM Generation and Attribution2024· llm-systems← topic thread
- Generalization through Memorization: Nearest Neighbor Language Models2019· retrieval← LLM-judged similar
- RecurrentGemma: Moving Past Transformers for Efficient Open Language Models2024· llm-systems← in-domain predecessor
- Clover: Regressive Lightweight Speculative Decoding with Sequential Knowledge2024· inference-optimization← topic thread
- K-Search: LLM Kernel Generation via Co-Evolving Intrinsic World Model2026· llm-systems← similar topic
- The Geometry of LLM Quantization: GPTQ as Babai's Nearest Plane Algorithm2025· llm-systems← similar topic
- Recurrent Drafter for Fast Speculative Decoding in Large Language Models2024· llm-systems← similar topic
Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models2025· multimodal← similar topic
- InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models2025· multimodal← in-domain predecessor
- A Comprehensive Survey on Long Context Language Modeling2025· llm-systems← topic thread
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test2025· inference-optimization← topic thread
- Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context2026· multimodal← similar topic
- PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model2025· multimodal← similar topic

What it influenced

Later papers in the same chain. Direction reversed.

Scaling Pre-training to One Hundred Billion Data for Vision Language Models2025· multimodal→ in-domain predecessor
- Training Video Foundation Models with NVIDIA NeMo2025· training-methods→ LLM-judged similar
- Qwen2.5-VL Technical Report2025· multimodal→ in-domain predecessor
Reward-Guided Speculative Decoding for Efficient LLM Reasoning2025· reasoning→ topic thread
- s1: Simple test-time scaling2025· reasoning→ in-domain predecessor
- EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test2025· inference-optimization→ topic thread
- STaR: Bootstrapping Reasoning With Reasoning2022· reasoning→ similar topic
- Advancing LLM Reasoning Generalists with Preference Trees2024· reasoning→ similar topic
- Offline Reinforcement Learning for LLM Multi-Step Reasoning2024· reasoning→ similar topic
Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Models2025· multimodal→ similar topic
- Kimi-Audio Technical Report2025· multimodal→ in-domain predecessor
- QwenLong-L1: Towards Long-Context Large Reasoning Models with Reinforcement Learning2025· rl-training→ topic thread
- Beat the long tail: Distribution-Aware Speculative Decoding for RL Training2025· training-methods→ topic thread
- PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Model2025· multimodal→ similar topic
- Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context2026· multimodal→ similar topic