The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer

🎯 Pitch

Ditching the separate vision encoder doesn't just simplify architecture—a single transformer actually scales better with more data, closing the performance gap with modular models. Even more striking, it can match a 22B-parameter ViT on semantic segmentation without any vision-specific pretraining, while shifting 60–80% of its attention to image tokens during visual tasks.

1. Executive Summary

This paper introduces Sail, a single-transformer unified multimodal large language model that processes raw image pixels and text tokens within one architecture, eliminating the separate pretrained vision encoder required by modular MLLMs. Through controlled experiments on vision-language benchmarks and vision-centric tasks—comparing Sail against modular counterparts like LLaVA and single-transformer baselines like SOLO and EVE—the authors systematically analyze three fundamental properties: data scalability (Sail exhibits steeper performance gains as pretraining data scales, narrowing the gap from a large deficit at 32M samples to near-parity with modular MLLMs at 512M image-text pairs), cross-modal information flow patterns (Sail allocates 60–80% of attention scores to image tokens during prediction versus 10–30% for modular MLLMs, indicating a vision-centric decision-making pathway), and visual representation capability (Sail achieves 84.95% ImageNet top-1 accuracy and 55.3% mIoU on ADE20K segmentation, matching ViT-22B's segmentation performance with significantly less training data). The paper establishes that single-transformer architectures can match modular MLLM performance on vision-language tasks while simultaneously functioning as effective standalone vision encoders, but only when trained with bidirectional attention within image patches and interleaved pure text data to preserve language capabilities.

2. Context and Motivation

The Core Problem: Multimodal Large Language Models Are Architecturally Fragmented

The fundamental gap this paper addresses is architectural: contemporary Multimodal Large Language Models (MLLMs) are modular systems that inherit a fundamental design fragmentation — they bolt together independently pretrained components rather than learning multimodal understanding within a unified parameter space. The dominant paradigm, exemplified by LLaVA (Liu et al., 2024c), Qwen-VL (Bai et al., 2023), and InternVL (Chen et al., 2024f), follows a three-component blueprint: (1) a frozen or fine-tuned vision encoder (typically CLIP-ViT or SigLIP) extracts dense visual features from input images; (2) a lightweight projector (linear layer or MLP) maps these features into the LLM's embedding space; and (3) the LLM processes the projected visual tokens alongside text tokens using standard causal attention. This approach achieves strong performance — the paper shows LLaVA-OneVision reaching 60.9% on MMStar and 81.7% on MMBench (Table 2) — but the paper argues this success masks a deeper problem.

The fragmentation manifests in three concrete ways the paper identifies in Section 2.1. First, deployment complexity: incorporating a separate vision encoder significantly increases both training and inference costs. A ViT encoder adds hundreds of millions to billions of parameters that exist solely to process images, increasing memory footprint and requiring costly infrastructure. The paper notes this explicitly — "incorporating a separate ViT encoder significantly slows down both training and inference, increasing deployment complexity and requiring costly infrastructure—especially when compared to a single transformer unified model." This is not a minor engineering concern; it directly limits where and how MLLMs can be deployed, particularly on edge devices or in latency-sensitive applications.

Second, representational mismatch: the common strategies for integrating visual features — either projecting them into the LLM's input sequence once (Liu et al., 2024c; Zhu et al., 2023) or sharing them across LLM layers via cross-attention (Alayrac et al., 2022; Hong et al., 2023) — "often struggle to reconcile the inherent differences between images and text representations." The vision encoder is trained under one objective (contrastive image-text alignment, typically) while the LLM is trained under another (causal language modeling). The projector is asked to bridge this gap, but it is a thin linear layer with limited capacity to transform between these distinct representational spaces. This creates a bottleneck where rich visual information may be lost or distorted during the modality translation.

Third, scaling disharmony: "as model scale [grows], balancing the interactions between the encoder, LLM, and alignment layers becomes increasingly challenging." When the vision encoder and LLM scale at different rates — or when one component is frozen while the other is fully trained — the system's capacity becomes imbalanced. The paper does not elaborate on the specific failure modes of this disharmony, but the implication is clear: modular architectures introduce coupling challenges that grow with scale, and current ad-hoc solutions (freezing the encoder, adding more projector capacity, multi-stage training) are patches rather than principled solutions.

Why This Problem Matters: Beyond Benchmark Performance

The practical stakes extend well beyond benchmark scores. The paper argues implicitly that the modular paradigm imposes a fundamental ceiling on multimodal learning because it prevents the model from learning visual representations that are genuinely co-adapted with language understanding. In a modular system, the visual encoder is trained once on a vision-only or contrastive objective, then frozen or lightly fine-tuned. The LLM never has the opportunity to reshape early visual processing to better serve linguistic reasoning tasks. This means the visual representations are suboptimal for the specific demands of vision-language reasoning — they encode what CLIP training deemed important for image-text matching, not what helps a language model answer questions about diagrams, read text in images, or understand spatial relationships described in natural language.

A unified single-transformer architecture, by contrast, co-learns visual and linguistic representations from scratch in a shared parameter space. Every self-attention layer processes both image patches and text tokens, enabling the model to develop visual features that are inherently compatible with the language modeling objective. There is no projector bottleneck, no frozen encoder, and no representational mismatch because the entire model is optimized end-to-end for the autoregressive language modeling loss. This is a genuinely different inductive bias — one that prior work had explored only superficially, without systematic analysis of its fundamental properties.

The significance extends to real-world deployment economics. The paper's data scaling experiments (Figure 1A) show that modular MLLMs have an initial advantage at small data scales because they inherit pretrained vision encoders — SigLIP, the encoder used in the paper's modular baseline, was trained on 40 billion samples. But Sail's steeper scaling curve suggests that as data continues to grow (a trend that shows no sign of stopping), unified architectures may overtake modular ones. If this extrapolation holds, the long-term economic argument favors investing in single-transformer training pipelines rather than chaining together pretrained components, because the unified architecture eliminates the cost of training and maintaining separate vision encoders while delivering superior data efficiency at scale.

The paper also identifies a theoretical gap: single-transformer MLLMs had been proposed before (Fuyu, Chameleon, SOLO, EVE), but their "fundamental properties, such as scalability, cross-modal information flow patterns, and visual representation capabilities" had received "little exploration." The field lacked a systematic understanding of whether these architectures scale well with data and model size, whether they process visual information differently from modular systems, and whether they can serve as standalone vision encoders. Without this understanding, the single-transformer approach remained a curiosity — interesting but unproven — rather than a credible alternative to the modular paradigm.

Where Prior Approaches Fall Short

Modular MLLMs: Effective but Fundamentally Limited

The modular paradigm is not dismissed by the paper; it is acknowledged as the "prevailing approach" that "achieves strong performance." The limitations the paper identifies are architectural rather than performance-based:

Frozen or semi-frozen visual encoders limit end-to-end learning. In most modular MLLMs (LLaVA-1.5, Qwen-VL), the vision encoder is either fully frozen or fine-tuned with very low learning rates. This preserves the pretrained visual knowledge but prevents the vision backbone from adapting to the specific demands of multimodal reasoning. The paper implicitly argues this is a missed opportunity: if visual features were learned jointly with language, they could develop properties specifically suited for answering questions, following instructions, and performing visual reasoning rather than just matching images to captions.

The projector is an information bottleneck. Regardless of whether the alignment layer is linear (LLaVA), MLP-based, or cross-attention (Flamingo), it is a thin interface between two large models trained on different objectives. This forces the visual features to be compressed into a format the LLM can consume, potentially discarding information that would be useful for downstream reasoning. The paper does not quantify this information loss, but the existence of approaches like LLaVA-1.6 that increase visual token count from 576 to 2880 (Table 2) suggests that projector bottlenecks are a recognized problem — more visual tokens improve performance but increase computational cost quadratically.

Scaling modular systems is an exercise in component balancing. As models and datasets grow, practitioners must decide how much to scale the vision encoder versus the LLM, whether to train or freeze each component, and how to adjust the projector. These are interdependent decisions with no established scaling laws. The paper notes this becomes "increasingly challenging," implying that the modular approach lacks a principled scaling framework, unlike the well-studied scaling behavior of pure language models.

Single-Transformer MLLMs: Proposed but Poorly Understood

The paper positions itself against two existing categories of single-transformer approaches (Section 2.1):

Continuous tokenization methods (Fuyu-8B, SOLO) directly map image patches to LLM embeddings via linear projection. Fuyu-8B demonstrated the feasibility of this approach, but the paper's benchmark results (Table 2) show Fuyu achieving only 34.4% on MMBench and 10.7% on SEEDBench — substantially below modular MLLMs. SOLO improved on this (67.7% on MMBench, 64.4% on SEEDBench) but still lagged behind modular systems. The paper's key critique is that these methods "rely on extra designs and auxiliary loss" (Section 2.1, referring to EVE's use of additional training objectives) and that their "scaling laws and fundamental properties... remain poorly understood." In other words, prior work demonstrated that single transformers can work but did not explain why they work, when they work best, or how they differ from modular systems in their internal behavior.

Discrete tokenization methods (Chameleon, Emu3) compress images into discrete tokens using VQ-VAE tokenizers, enabling joint autoregressive modeling with text tokens. The paper notes this approach "trade[s] pixel-level fidelity for generation capabilities" — the quantization process loses fine-grained visual detail, which limits performance on vision-language understanding tasks. Chameleon's benchmark results bear this out: 31.1% on MMBench, 30.6% on SEEDBench (Table 2), far below modular MLLMs. Emu3 improves substantially (58.5% on MMBench, 68.2% on SEEDBench) but at the cost of using 16K visual tokens and massive pretraining, suggesting an inefficient representation.

Critical gaps the paper identifies in prior single-transformer work:

"Existing methods rely on extra designs and auxiliary loss, complicating training pipelines." This refers to EVE, which introduces additional training objectives beyond standard language modeling to boost visual learning. The paper positions Sail as simpler — no auxiliary losses, just standard autoregressive language modeling on text tokens with carefully chosen architectural defaults.
"The scaling laws and fundamental properties of purely end-to-end trained models remain poorly understood." This is the paper's central motivation for its empirical analysis. Prior work trained single transformers at one or two scales and reported benchmark numbers; no one had systematically varied data volume, model size, or training recipes to characterize scaling behavior and compare it to modular systems.
"Vision-language interaction in shared parameter spaces lacks systematic analysis—most prior MLLMs default to causal attention for processing image-text sequences." This is a specific but critical architectural observation. Every prior single-transformer MLLM the paper cites (Fuyu, SOLO, Chameleon, EVE, Emu3) used causal attention for all tokens, including image patches. The paper argues this is an unexamined default — causal attention may be appropriate for autoregressive text generation, but images are 2D spatial data where bidirectional context is essential for understanding global structure. The paper's introduction of mixed attention (bidirectional within images, causal for text) is a direct response to this gap.

Vision Representation Learning: Fragmented Paradigms Don't Serve Multimodal Models Well

Section 2.2 surveys three dominant approaches to learning visual representations from image-text data, and the paper's critique is subtle but important: all three produce vision encoders that are architecturally separate from language processing, meaning they cannot be directly integrated into a unified multimodal system.

Text as classification labels (Tag2Text, RAM, CatLIP, SuperClass) treats image-text pairs by extracting categorical labels from captions and training ViTs to predict them. These methods produce strong vision-only encoders but offer no mechanism for joint vision-language reasoning — the text is reduced to a discrete label, losing the rich compositional structure that enables complex multimodal tasks.

Image-text contrastive learning (CLIP, ALIGN, SigLIP) trains dual encoders to align global image and text embeddings in a shared space. This is the dominant paradigm for pretraining vision encoders used in modular MLLMs. The paper acknowledges its effectiveness but implies that contrastive pretraining produces representations optimized for retrieval and zero-shot classification, not for serving as input to a language model that must perform multi-step reasoning over visual content. The contrastive objective aligns global descriptions ("a dog sitting on a couch") with whole images, but MLLMs need to reason about specific regions, spatial relationships, and visual details that global alignment may not preserve.

Text as autoregressive targets (SimVLM, CapPa) trains encoder-decoder or encoder-only architectures to generate captions from images autoregressively. This is the closest paradigm to Sail's approach — learn visual representations by predicting text from images. However, CapPa and SimVLM retained a separate vision encoder (or encoder-decoder split) rather than processing everything in one transformer. The paper notes these methods "often retain modular designs or auxiliary components like contrastive losses," meaning they don't fully realize the unified architecture vision. Sail's innovation here is eliminating "architectural fragmentation by jointly modeling image patches and text tokens in a single Transformer" — no separate encoder, no contrastive loss, just one model learning both modalities through the same next-token prediction objective.

How This Paper Positions Itself

The paper positions itself as an empirical analysis, not a method proposal. This is crucial for understanding its contribution. The authors state upfront (Section 1): "While we do not propose novel architecture designs, we introduce necessary modifications to enable the model to process different modalities in a unified architecture." The "necessary modifications" — mixed attention and multimodal RoPE — are adapted from existing techniques, not invented here. What is novel is the systematic study of what happens when you train such a model, how its properties differ from modular MLLMs, and what these differences imply for the future of multimodal architecture design.

The positioning is explicitly comparative. Throughout the paper, Sail is paired with a "modular MLLM counterpart" — a system using the same LLM backbone and training data, but with a SigLIP vision encoder and linear projector (Section 4.3.1). This controlled comparison isolates the effect of architectural choice from confounding factors like model size, data quality, or training recipe. Every property the paper claims for single transformers — superior data scalability, vision-centric attention patterns, emergent vision encoder capability — is established through this A/B comparison, not through absolute benchmarks.

The paper also positions itself as bridging a gap between two communities: the MLLM community (which has focused on modular architectures and benchmark performance) and the vision representation learning community (which has developed sophisticated methods for training visual backbones). By demonstrating that a single transformer trained for multimodal language modeling also produces competitive vision features (Table 4: 85.0% ImageNet top-1, 55.3% ADE20K mIoU, matching ViT-22B's segmentation performance with 30× less data), the paper argues that these two goals — building strong MLLMs and learning good visual representations — are not in tension; a unified architecture can achieve both simultaneously.

Finally, the paper's title — "The Scalability of Simplicity" — signals its philosophical position. The modular MLLM paradigm adds complexity (separate vision encoder, alignment layer, multi-stage training) to achieve strong performance by leveraging pretrained components. The single-transformer approach removes this complexity and asks: can simplicity scale? The paper's answer is a qualified yes — single transformers scale better with data (steeper curve in Figure 1A) and model size (consistent improvements in Figure 3), but they start from a lower baseline and only catch up to modular systems after substantial pretraining (512M image-text pairs). The "simplicity" bet pays off at scale, but modular systems retain an advantage in the low-data regime because they inherit pretrained visual knowledge.

This positioning explains the paper's unusual structure: there is no novel architectural contribution, no new loss function, and no state-of-the-art benchmark result (Sail lags behind LLaVA-OneVision on several metrics in Table 2). Instead, the contribution is knowledge — a systematic characterization of how single transformers behave, scale, and differ from modular systems. The paper argues that this knowledge is a necessary foundation for future work that will refine and improve the unified architecture approach, much as scaling laws for language models enabled the Chinchilla-optimal training recipes now used across the industry.

3. Technical Approach

This paper is primarily an empirical analysis paper whose core idea is that a single-transformer architecture, properly configured with mixed attention and multimodal positional encodings, can simultaneously serve as both a competitive multimodal language model and an effective vision encoder — and that its fundamental properties differ systematically from modular MLLMs in ways that favor scalability, vision-centricity, and representational quality.

3.1 Reader Orientation (Approachable Technical Breakdown)

What Sail is: Sail is a single Transformer model that takes raw image patches (pixels divided into fixed-size grids and linearly projected) and text tokens (from a standard language model tokenizer) as input, processes them through shared self-attention layers, and generates text outputs — all without a separate vision encoder, alignment projector, or auxiliary training objectives.

What problem it solves and the shape of the solution: The problem is architectural fragmentation in multimodal LLMs — current systems bolt together independently pretrained vision encoders and language models, creating deployment complexity, representational mismatches, and scaling disharmony. The solution is a minimalist architecture that co-learns visual and linguistic representations from scratch in a unified parameter space, using two key design choices (bidirectional attention within images, multimodal rotary position embeddings) that prior single-transformer work had overlooked, and training it with a simple curriculum that progressively builds visual understanding while preserving language capabilities.

3.2 Big-Picture Architecture (Diagram in Words)

Sail has five conceptual components, though they are implemented as a single neural network:

Image Patch Embedding Layer — a linear projection that converts raw image patches (14×14 pixels each) into continuous embedding vectors, exactly analogous to how a ViT embeds patches, except there is no separate encoder; these embeddings are fed directly into the transformer alongside text tokens.
Text Token Embedding Layer — the standard embedding lookup table from the pretrained LLM (Mistral-7B-v0.1), which converts tokenized text into embedding vectors.
Mixed Attention Self-Attention Layers — the core innovation at the micro-architecture level: all layers apply bidirectional (full) attention among image patches from the same image, while applying causal (unidirectional) attention for text tokens. This means image patches can attend to all other patches in the same image, text tokens attend only to previous text tokens (plus all image patches that precede them in the sequence), and cross-modal attention flows from text to image patches that appear earlier in the concatenated sequence.
Multimodal Rotary Position Embedding (M-RoPE) — a positional encoding that decomposes position into height and width dimensions for image patches (using 2D coordinates) while using uniform 1D positions for text tokens, enabling the model to represent spatial structure without conflating image dimensions with sequence position.
Language Modeling Head — the standard linear layer + softmax that predicts the next text token, applied only to text positions. Image patches and special visual tokens (<vision>, </vision>) are excluded from loss computation.

Information flow during training: An image-text pair enters the system → the image is partitioned into 14×14 patches and linearly projected into embeddings; the text is tokenized and embedded → special <vision> and </vision> tokens wrap the image patch sequence → image and text embeddings are concatenated into one sequence → this sequence passes through all transformer layers simultaneously, with the mixed attention mask governing which positions can attend to which others → the language modeling loss is computed only on text token positions → gradients flow back through the entire network, updating both the vision-related parameters (patch embedding) and language-related parameters (token embeddings, self-attention, FFN, output head) jointly.

Information flow during inference: An image and text prompt are fed in → the same concatenation and mixed attention processing occurs → the model autoregressively generates text tokens one by one, with each new token attending to all previous image patches and text tokens (but not to future tokens, per causal masking for text).

3.3 Roadmap for the Deep Dive

First, the mixed attention mechanism — the architecture's defining feature — including how the attention mask is constructed, why bidirectional attention within images matters, and how it differs from the causal-everywhere default of prior single-transformer MLLMs.
Second, the multimodal rotary position embeddings (M-RoPE) — how 2D spatial coordinates are encoded, how image and text positions are sequenced, and why this scheme improves generalization to arbitrary resolutions.
Third, the two-stage pretraining curriculum — the rationale for starting with low-resolution images, the any-resolution extension in Stage 2, and the critical role of interleaved pure text data in preventing catastrophic forgetting of language capabilities.
Fourth, the supervised fine-tuning stage — what data is used, how the entire network is fine-tuned, and how this stage converts the pretrained model into an instruction-following assistant.
Fifth, the specific configurations, hyperparameters, and datasets at each stage, including the data processing pipeline (sequence packing, attention mask construction for batched training) and the infrastructure setup (Megatron modifications, parallelism strategy).

3.4 Detailed, Sentence-Based Technical Breakdown

Mixed Attention: Bidirectional for Images, Causal for Text

The paper's most consequential architectural decision is the choice of attention pattern. Every prior single-transformer MLLM the paper compares against — Fuyu, SOLO, Chameleon, EVE, Emu3 — used standard causal attention across all tokens, including image patches. This means each image patch could only attend to patches that appeared earlier in the flattened 2D→1D raster-scan ordering. The paper argues this is an unexamined default inherited from autoregressive language modeling, and that it fundamentally limits visual representation learning because 2D images have spatial structure that cannot be captured by a 1D causal ordering.

The mixed attention mask. The paper implements a heterogeneous attention pattern within a single transformer, illustrated in Figure 2(B). The key constraint is:

For text tokens: causal (autoregressive) attention — position $i$ can attend to positions $j \leq i$ , but not to $j > i$ . This preserves the model's ability to generate text autoregressively, as it was pretrained to do.
For image patches belonging to the same image: bidirectional (full) attention — every patch can attend to every other patch in the same image, regardless of their relative position in the 1D sequence. This enables each patch to incorporate global spatial context, which is critical for tasks like object recognition (where a patch containing a dog's ear should influence the representation of a distant patch containing its tail) and semantic segmentation (where understanding what an object is requires seeing its full extent).
For cross-modal interaction: text tokens can attend to all image patches that precede them in the sequence (since images are placed before text in the concatenated input), but image patches cannot attend to text tokens — because during pretraining, the model is trained only to predict text given images, not to generate images.

Why bidirectional attention within images matters (conceptually). In a causal-only scheme, an image patch at position $k$ in the raster order can see patches at positions $1, 2, ..., k-1$ but not patches at positions $k+1, k+2, ...$ . For a 224×224 image with a 14×14 patch size, this means there are 256 patches total, and the last patch can see all 255 others, but the first patch can see nothing. This creates an asymmetry: the quality of a patch's representation depends on where it falls in the raster order. Patches early in the sequence receive less context; patches late in the sequence receive more. For vision tasks where spatial context is uniformly important regardless of position, this is a poor inductive bias.

The ablation in Table 7 quantifies the impact: removing bidirectional image attention ("No Img full attn") reduces TextVQA performance from 20.1% to 16.2% (a 3.9 percentage point drop) and VQAv2 from 59.1% to 57.8%. For TextVQA — which requires reading text in images — the bidirectional context is particularly important because individual text characters may be spread across multiple patches, and seeing the full word requires attending across patch boundaries in both directions.

Implementation detail: attention mask construction. The paper describes this in Section 3.1 and illustrates it in Figure 2(B). For a multimodal sequence, the attention mask is a binary matrix $M$ where $M_{ij} = 1$ means position $i$ can attend to position $j$ . The mask is constructed as follows:

For text-to-text attention: lower-triangular matrix (causal).
For image-to-image attention within the same image: all ones (bidirectional) for positions belonging to that image span.
For text-to-image attention: ones for all image positions that appear before the text position in the sequence.
For image-to-text attention: zeros (no attention from image patches to text).
For cross-sample attention: zeros (prevent attention between different training samples packed into the same sequence).

During training, the paper uses sequence packing — concatenating multiple image-text pairs and pure text sequences into one long sequence of 32,768 tokens. The attention mask ensures these different samples cannot attend to each other, which would be a wasteful use of computation and could confuse the model by mixing unrelated contexts.

Multimodal Rotary Position Embeddings (M-RoPE)

Rotary Position Embedding (RoPE) is the standard positional encoding used in most modern LLMs, including Mistral-7B, Sail's base model. In standard 1D RoPE, each token's position is encoded by rotating its query and key vectors by an angle proportional to its absolute position in the sequence. This encodes relative position information in the dot product between queries and keys: the dot product between tokens at positions $i$ and $j$ is modulated by a rotation that depends on $i - j$ .

The problem with 1D RoPE for images. When you flatten a 2D image into a 1D sequence (e.g., raster-scan order: row 0 left-to-right, then row 1 left-to-right, etc.), 1D RoPE encodes only the 1D sequence position. Two patches that are vertically adjacent in the image (one directly above the other) will have 1D positions that differ by the image width in patches — e.g., position 0 and position 16 for a 16×16 grid. The RoPE-encoded relative position is 16, but this magnitude conflates "far apart in the 1D sequence" with "adjacent in the 2D image." Conversely, patches that are 16 positions apart in 1D could be vertically adjacent, horizontally distant, or diagonally related depending on the image dimensions.

M-RoPE solution. Following Qwen2-VL (Wang et al., 2024b), the paper decomposes the positional encoding into separate height and width axes. For an image patch at 2D coordinate $(h, w)$ (row $h$ , column $w$ ), the RoPE rotation is computed from two separate position values: one for the height dimension and one for the width dimension. The rotation is applied to different subsets of the query/key embedding dimensions (splitting the embedding into chunks for height and width encoding).

For text tokens, which exist in a 1D sequence, both the height and width position IDs are set to the same value — the standard 1D sequence position — effectively reducing to standard RoPE for text.

Inter-modal position indexing. A critical detail in Figure 2(C): "position indexing is sequentially initialized across modalities (e.g., starting from images before extending to subsequent text), preserving inter-modal consistency." This means the position IDs are assigned continuously across the image-text boundary. For example, if an image has 256 patches (positions 0–255), the first text token after the image receives position 256, not position 0. This ensures that the relative position between a text token and an image patch reflects their actual distance in the multimodal sequence, which the paper argues "preserv[es] inter-modal consistency."

This also implies M-RoPE "constrains absolute position magnitudes for visual tokens," which the paper claims facilitates "robust generalization to extended sequences in inference." The reasoning: because image patches always start near position 0, their absolute positions are bounded by the image size (e.g., 0–255 for a 256-patch image), even as the total sequence length grows during long conversations. This prevents the visual token position magnitudes from drifting into regimes the model has not seen during training.

Why M-RoPE matters for any-resolution training. In Stage 2 pretraining, images are processed at their native resolutions rather than being resized to a fixed 224×224. This means the number of patches per image varies dramatically — a 448×448 image produces 1024 patches, while a 224×448 image produces 512 patches with a 2:1 aspect ratio. Standard 1D RoPE has no mechanism to indicate which patches are horizontally versus vertically adjacent; the model would need to infer spatial relationships from patch content alone. M-RoPE's height/width decomposition directly encodes 2D spatial structure, enabling the model to generalize its understanding of spatial relationships to arbitrary resolutions and aspect ratios seen only during Stage 2 training.

Two-Stage Pretraining Curriculum

The pretraining strategy is designed to solve a chicken-and-egg problem: the model needs to learn visual representations from scratch (since it starts from an LLM checkpoint with no visual knowledge), but training on high-resolution images is computationally prohibitive because the sequence length scales quadratically with resolution (a 224×224 image with 14×14 patches produces 256 tokens; a 448×448 image produces 1024 tokens). The solution is a curriculum that starts with rapid, low-resolution visual knowledge acquisition, then refines with any-resolution training.

Stage 1: Accelerated Visual Knowledge Acquisition. All images are uniformly resized to 224×224 pixels. With a patch size of 14, this produces 256 image patch tokens per image. The key design choice: "to maximize data throughput, we uniformly resize all images to a lower resolution... reducing multimodal sequence lengths and enabling the model to process more samples within fixed training time."

This is framed as a throughput optimization — by keeping sequences short, the model can be trained on more image-text pairs per GPU-hour. The Stage 1 dataset is Recap-DataComp-1B, containing 512M image-text pairs (Table 1). This is a massive dataset by multimodal standards — for comparison, LLaVA-1.5 was trained on only 558K image-text pairs during pretraining. The paper is effectively arguing that a single transformer needs orders of magnitude more data than a modular system's alignment stage because it must learn visual representations from scratch rather than inheriting them from a pretrained ViT.

Critical: interleaving pure text data. The paper explicitly warns about catastrophic forgetting: "To prevent catastrophic forgetting of linguistic knowledge, we interleave pure text corpora with multimodal data during training." The pure text data is SlimPajama, a 627B-token cleaned version of RedPajama. During training, the system uses "a round-robin approach to interleave image-text packed sequences and pure text packed sequences, configuring the global batch to contain approximately 16K image-text pairs" (Section 4.1, Implementation Details).

The round-robin interleaving means the model alternates between multimodal batches and pure text batches. This ensures the model continues to practice language modeling on diverse text, preventing the LLM's linguistic capabilities from degrading while it learns to process images. The ablation in Table 7, row #2 shows the consequence of removing this: "No pure text in PT" reduces VQAv2 from 59.1 to 56.3, GQA from 46.9 to 42.1, and ScienceQA from 59.6 to 48.6 — a 11 percentage point drop on the most knowledge-intensive benchmark. This confirms that language capability preservation is not optional; it is essential for downstream multimodal performance.

Training objective. Throughout both stages, the loss is the standard autoregressive language modeling loss, computed only on text tokens. Image patches and the special <vision> and </vision> tokens are excluded from loss computation. The loss for a sequence is:

$\mathcal{L}_{\text{LM}} = -\frac{1}{|\mathcal{T}|} \sum_{t \in \mathcal{T}} \log p(y_t | x_{<t})$

where $\mathcal{T}$ is the set of text token positions in the sequence, $y_t$ is the ground-truth token at position $t$ , and $p(y_t | x_{<t})$ is the model's predicted probability for that token given all preceding tokens (image patches and text tokens before position $t$ ).

Why text-only loss: the model is an autoregressive language model at its core; it generates text, not images. Training the model to predict image patches (as Chameleon and Emu3 do with discrete visual tokens) would require a different architecture (discrete tokenizer, separate decoding head) and would shift capacity away from the primary goal of vision-language understanding. The paper's design is minimalist: learn visual representations solely as a byproduct of the pressure to predict text correctly given image context. This forces the model to develop visual features that are genuinely useful for language tasks, rather than features that merely reconstruct pixel-level details.

Stage 2: Enhancing Any-Resolution Image Understanding. After Stage 1's rapid visual bootstrapping, the model has learned basic visual representations but only for 224×224 square images. Real-world applications require handling diverse resolutions and aspect ratios: documents (tall and narrow), charts (wide), infographics (irregular). Stage 2 addresses this by training on images at their native resolutions with dynamically adapted positional embeddings.

The key mechanism: "images retain their native resolutions during processing, and positional embeddings adapt dynamically to arbitrary spatial dimensions." Because M-RoPE encodes positions via separate height and width coordinates rather than a single 1D sequence index, the positional encoding for a patch at 2D position $(h, w)$ depends only on $h$ and $w$ , not on the total number of patches in the image. This means the model can process a 336×448 image (producing 24×32 = 768 patches) using the same M-RoPE mechanism as a 224×224 image (16×16 = 256 patches) — the height and width positions are simply computed from the actual grid dimensions.

The Stage 2 dataset is smaller but diverse: Capfusion (60M pairs), self-curated OCR data from LAION COCO (7M pairs), and the InfinityMM Stage 2 subset (19M pairs), totaling 86M image-text pairs plus SlimPajama text data (Table 1). The inclusion of OCR data is notable — Stage 2 deliberately introduces text-rich images (documents, signs, screenshots) that require fine-grained visual understanding, building on the coarse representations learned in Stage 1.

The any-resolution strategy is not unique to Sail. The paper acknowledges (Section 3.2) that it "follow[s] prior works" (citing EVE and SOLO) in adopting any-resolution training. What is unique is the combination with M-RoPE, which provides a principled way to encode 2D positions at arbitrary resolutions without learning new position embeddings or using interpolation.

Catastrophic forgetting continues to be managed. Stage 2 also interleaves SlimPajama text data, following the same round-robin approach as Stage 1. This ensures that the additional visual training does not erode the language capabilities preserved in Stage 1.

Supervised Fine-Tuning (SFT)

The SFT stage converts the pretrained model into an instruction-following assistant. After two stages of pretraining, Sail has learned to predict text given images — it can generate captions, answer factual questions about images, and describe visual content. But it has not been trained to follow complex instructions, engage in multi-turn dialogue, or handle the diverse query formats encountered in real deployment.

Dataset. The SFT dataset is the InfinityMM Stage 3 subset, containing 6M image-text instruction pairs (Table 1). This is described as "multi-source instruction datasets" from "publicly available" sources. The paper does not elaborate on the specific composition, but the reference to InfinityMM (Gu et al., 2024) suggests this includes a mix of visual question answering, visual dialogue, OCR-based QA, chart understanding, and general instruction-following data.

Training procedure. Unlike modular MLLMs that sometimes freeze the vision encoder or LLM during SFT, Sail fine-tunes "the entire network architecture." This means all parameters — the patch embedding, the self-attention layers, the FFN layers, the output head, and the token embeddings — are updated. The rationale: since the model was trained end-to-end from scratch (relative to its visual capabilities), there is no risk of "disturbing" a separately pretrained component. The entire model is already co-adapted; SFT simply refines it for the instruction-following distribution.

Hyperparameters (Section 4.1, Implementation Details):

Global batch size: 512
Training duration: 1 epoch
Maximum learning rate: $1 \times 10^{-5}$
Learning rate schedule: linear warm-up followed by cosine decay to a minimum (minimum not specified in the text for SFT, but the pretraining minimum is $5 \times 10^{-6}$ )

The SFT stage uses the any-resolution strategy, matching the Stage 2 pretraining setting. This means instruction-tuning examples are presented at their native resolutions, and the model must generalize its instruction-following ability across diverse image sizes.

Pretraining Infrastructure, Hyperparameters, and Data Processing

Model initialization. Sail is initialized from the Mistral-7B-v0.1 base LLM. This provides the model with strong language capabilities from the start — the 7B parameters from Mistral encode extensive linguistic knowledge, reasoning ability, and world knowledge. The image patch embedding layer is the only new parameter matrix, initialized from scratch (randomly).

Patch embedding. Images are partitioned into 14×14 pixel patches and projected into the LLM's embedding dimension via a single linear layer. For a 224×224 image, this produces 16×16 = 256 patches. The choice of 14×14 patch size matches the standard ViT configuration (ViT-L/14) and balances spatial resolution against sequence length.

Special visual tokens. The paper maintains explicit boundary tokens: <vision> marks the start of an image patch span and </vision> marks its end. These are standard tokens in the vocabulary that serve as delimiters, enabling the model to distinguish image patches from text tokens in the sequence and to handle multiple images in a single input (though multi-image scenarios are not explored in the paper).

Sequence packing for training efficiency. A crucial implementation detail: to maximize GPU utilization, the paper "concatenate[s] sequences from different data samples into one long sequence of 32,768 tokens, adjusting the attention mask to ensure that tokens from different samples do not attend to each other."

This is standard practice in LLM pretraining (also used in the original Mistral training). Without packing, short sequences (e.g., an image with a brief caption produces only ~300 tokens) would waste most of the GPU memory and computation budget on padding tokens. By packing multiple samples into one 32K-token sequence, the paper achieves higher effective throughput.

The attention mask modification is critical: each sample's attention mask block is independent, with zeros preventing cross-sample attention. This means the model processes multiple independent image-text pairs simultaneously but cannot mix information across them — each pair is effectively a separate forward pass, just sharing the same GPU kernel launches.

Global batch configuration. The paper states the global batch "contain[s] approximately 16K image-text pairs." Given the 32,768-token sequence length and the average number of tokens per image-text pair (roughly 256 image patches + some number of text tokens), this suggests the distributed training setup uses enough data-parallel workers to accumulate gradients over approximately 16K image-text pairs per optimizer step. The exact number of text tokens per step is not specified, but it would be in the range of millions of tokens given the sequence packing.

Distributed training setup and parallelism. The paper uses Megatron (Shoeybi et al., 2019), modified to support multimodal inputs. Megatron is a framework for training large transformer models with model parallelism — splitting the model's layers and attention heads across multiple GPUs.

Specific configuration:

GPUs: 128 NVIDIA A100 80GB GPUs
Tensor parallelism: 2-way (each transformer layer's attention heads are split across 2 GPUs)
Data parallelism: 64-way (64 independent copies of the model, each processing a different subset of the batch)

The total parallelism is 2 × 64 = 128, matching the GPU count. The 2-way tensor parallelism means each GPU holds half the model parameters; the 64-way data parallelism means there are 64 workers, each with two GPUs. Gradients are synchronized across all 64 workers.

This configuration is designed for the 7B model size. The paper also trains Sail-0.5B and Sail-3B variants (initialized from Qwen2.5-0.5B and Qwen2.5-3B, respectively — Table 8 in the Appendix); these likely use different parallelism configurations but the paper does not specify them.

Training hyperparameters (pretraining, both stages):

Maximum learning rate: $5 \times 10^{-5}$
Minimum learning rate: $5 \times 10^{-6}$ (cosine decay)
Learning rate schedule: cosine decay from max to min

These are quoted from Section 4.1. The learning rates differ from standard LLM pretraining (which typically uses $3 \times 10^{-4}$ for models of this size) because the model is not training from scratch — it is continuing training from a pretrained LLM checkpoint and adding visual capabilities. A lower learning rate prevents catastrophic forgetting of the pretrained language knowledge.

The Appendix (Table 8) also shows that the smaller model variants use higher learning rates:

Sail-0.5B (Qwen2.5-0.5B base): $5 \times 10^{-4}$ max
Sail-3B (Qwen2.5-3B base): $1 \times 10^{-4}$ max
Sail-7B (Mistral-7B-v0.1 base): $5 \times 10^{-5}$ max

This follows the standard practice that smaller models can tolerate higher learning rates because their optimization landscape is smoother.

Data filtering. The paper mentions one preprocessing step: "Samples that fail to produce a parsable final answer are filtered out" (this is stated in the context of the PRM training in the reference paper, not in Sail; Sail does not use this filtering). For Sail, the paper does not describe specific data filtering beyond using the publicly available datasets as-is (Recap-DataComp-1B, SlimPajama, Capfusion, etc.).

Vision Evaluation Protocol (for the "Vision Encoder Functioning" Claim)

After Stage 1 pretraining, the paper freezes the Sail checkpoint and evaluates it as a standalone vision encoder on three vision-centric tasks. This is a critical experiment because it tests whether the representations learned during multimodal pretraining are genuinely useful for pure vision tasks — tasks that involve no language input at inference time.

Image Classification (ImageNet-1K). The approach: "an attention-based classifier with 90 epochs of linear probing." The attention-based classifier is from El-Nouby et al. (2024) and works by computing a weighted average of patch features using learned attention weights, then applying a linear classifier to the resulting global representation.

Input resolution: 224×224
Global batch size: 8,192 across 8 A100 80GB GPUs
Training: 90 epochs, linear probing only (the Sail backbone is kept frozen)
Configurations: "mostly obtained from common practices" (citing MAE and related works)

Semantic Segmentation (ADE20K). The approach uses ViT-Adapter (Chen et al., 2022) with UperNet (Xiao et al., 2018) as the segmentation decoder, built on MMSegmentation. ViT-Adapter is a method for adapting frozen ViT backbones to dense prediction tasks by injecting spatial prior modules and feature pyramid networks between the frozen backbone layers.

Input resolution: 512×512
Global batch size: 16 across 8 A100 80GB GPUs
Training iterations: 80K

Attribute, Relation, and Ordering (ARO). This benchmark tests whether the model understands compositional relationships — e.g., "red cube on top of blue sphere" vs. "blue sphere on top of red cube." The evaluation protocol is retrieval-based: "the negative of the caption loss over each image-text pair as the similarity metric for retrieval."

In other words, for each image, the paper computes the language modeling loss (cross-entropy) for a set of candidate captions. The caption with the lowest loss (or highest negative loss) is the retrieval prediction. This directly tests whether the model's visual representations encode the fine-grained attributes and relationships expressed in language.

Why these three tasks: Classification tests object-level understanding, segmentation tests pixel-level spatial reasoning, and ARO tests compositional attribute binding. Together, they span the spectrum from coarse semantics to fine-grained spatial and relational reasoning, providing a comprehensive picture of the quality of the learned visual representations.

Design Choices and Their Justifications

Why single transformer over modular? The paper's core bet: architectural simplicity enables three advantages that compound at scale — (1) co-adapted visual-linguistic representations (no projector bottleneck), (2) simpler training and deployment (one model to train and serve, not three components), and (3) superior data scalability (as Figure 1A shows, the single transformer learns more from each additional training sample than the modular system, even though it starts behind).

Why mixed attention rather than causal-only? The paper argues this is the key design choice prior work missed. Bidirectional attention within images improves visual representation quality because spatial relationships are inherently non-causal. The ablation confirms this: removing bidirectional attention hurts TextVQA by 3.9 points (Table 7, row #1), suggesting that reading text in images — a task requiring understanding of 2D character arrangements — particularly benefits from bidirectional context.

Why M-RoPE rather than standard 1D RoPE? M-RoPE encodes 2D spatial structure directly in the positional embeddings, which is necessary for any-resolution training. Standard 1D RoPE would not distinguish between vertically adjacent patches (close in 2D, far in 1D) and horizontally distant patches (far in both), making it harder for the model to learn spatial relationships that generalize across resolutions.

Why two-stage pretraining? Stage 1 maximizes throughput on a massive dataset by using low-resolution images. Stage 2 refines the visual representations for diverse resolutions and introduces OCR-heavy data. The two-stage design is a practical compromise between training efficiency and visual detail — training exclusively on any-resolution data from the start would be much slower because high-resolution images produce longer sequences.

Why interleave pure text data? Without it, the model forgets how to process language, which is catastrophic for a multimodal model that must generate coherent text. The ablation (Table 7, row #2) shows drops of up to 11 percentage points on language-heavy benchmarks when pure text is removed. This is a practical lesson for training single-transformer MLLMs: you cannot just train on image-text pairs and expect the LLM's linguistic knowledge to survive.

Why loss on text tokens only? This keeps the architecture simple (no image generation head needed) and focuses the model's entire capacity on the task of producing correct text given visual input. The visual representations are learned as a hidden-layer side effect of this text-prediction pressure — a form of unsupervised representation learning where the "supervision" is the language modeling objective.

Why Mistral-7B as the base? The paper doesn't extensively justify this choice, but Mistral-7B is a strong open-source 7B model with good reasoning capabilities and was state-of-the-art at its release. Starting from a strong LLM ensures the model begins with robust language understanding, so the pretraining stages can focus primarily on acquiring visual knowledge.

4. Key Insights and Innovations

Innovation 1: The Scalability-of-Simplicity Hypothesis — Architectural Minimalism as a Scalable Pathway

The paper's most significant intellectual contribution is not a specific architectural component but a falsifiable hypothesis about the relationship between architectural simplicity and scaling behavior in multimodal learning. The dominant paradigm assumes that incorporating pretrained components — specifically, a separately trained vision encoder — is necessary to achieve strong multimodal performance at practical data scales. This assumption is so deeply embedded that the field has largely treated the modular architecture as a given, focusing innovation on what to plug into it (better vision encoders, smarter projectors, improved alignment techniques) rather than questioning whether the modularity itself is optimal.

Sail challenges this assumption directly through a controlled scaling experiment (Figure 1A) that reveals a crossover dynamic. At 32M training samples, the modular MLLM significantly outperforms Sail — this is the regime where pretrained visual knowledge confers a decisive advantage. But the gap narrows systematically as data scales: Sail's performance curve has a steeper slope. At 512M image-text pairs, the two architectures reach near-parity on the paper's evaluation subset. The implication is that modular architectures face diminishing returns from additional multimodal data because the frozen or semi-frozen vision encoder's representations are already saturated — they were pretrained on 40B samples (SigLIP), so an additional few hundred million image-text pairs provide marginal improvement. Sail, learning visual representations from scratch, extracts more value from each new training sample because it is operating in a regime where both modalities are still co-adapting.

This finding reframes the architectural choice from an engineering preference (simplicity vs. complexity) to a scaling strategy decision. The paper does not claim that single transformers are universally better — it claims they are better at scale, and modular systems are better in the low-data regime. This is a nuanced, testable claim that the field can validate or refute by running similar controlled comparisons on other model families, data distributions, and benchmarks. If the finding generalizes, it has direct economic implications: as the volume of multimodal training data continues to grow (a near-certainty), the relative advantage of modular architectures will shrink, and the operational simplicity of a single transformer (one model to train, deploy, and maintain) will increasingly dominate the marginal benchmark gains from modular complexity.

This contribution is fundamental rather than incremental — it establishes a scaling phenomenon that prior work had not characterized, similar in spirit to how the Chinchilla scaling laws reoriented language model pretraining from "bigger models" to "more data." It is not a method one can "apply" to an existing system; it is a finding that should influence architecture decisions for the next generation of multimodal models.

Innovation 2: Mixed Attention as a Diagnostic Architectural Intervention Exposing a Neglected Design Space

The paper's second distinctive contribution is the identification and validation of bidirectional image attention as a critical, previously overlooked design dimension in single-transformer MLLMs. Every prior single-transformer approach — Fuyu, SOLO, Chameleon, EVE, Emu3 — defaulted to causal attention for all tokens, including image patches. This was an unexamined inheritance from autoregressive language modeling: since the model generates text left-to-right, causal masking is applied uniformly, including to the visual modality where it has no principled justification.

The paper's intervention is not that bidirectional attention is novel per se — ViTs have used bidirectional attention since Dosovitskiy et al. (2021) — but rather the diagnosis that causal-only attention for image patches was a bug, not a feature, of prior single-transformer systems. The ablation in Table 7 provides the evidence: removing bidirectional image attention reduces TextVQA from 20.1% to 16.2% (a 3.9 percentage point drop, or 19% relative decline) and VQAv2 from 59.1% to 57.8%. The disproportionate impact on TextVQA — a benchmark requiring reading text embedded in images — is particularly revealing. Text in images is inherently 2D: individual characters often span multiple patches, and words may be arranged in irregular layouts (curved text, multi-column documents, signs with varied orientations). Causal attention forces the model to process this 2D information sequentially in raster order, creating an artificial bottleneck that bidirectional attention removes.

This is more than an ablation result; it is a conceptual reframing of the single-transformer design space. Prior work implicitly assumed that the attention pattern was a settled choice — causal, because that's what LLMs use. The paper demonstrates that attention patterns are a modality-specific design parameter that should be tailored to the data structure: causal for sequential 1D text, bidirectional for spatially structured 2D images. This opens a broader question that the paper does not fully explore but clearly implies: what other architectural defaults inherited from language modeling are suboptimal for multimodal processing?

The significance of this insight extends beyond the specific implementation. It suggests that the single-transformer approach has a richer design space than previously recognized, and that treating images as just another token type in a standard LLM architecture is insufficient — the architecture must be adapted to respect the distinct structural properties of each modality at the attention level. This is a fundamental insight for the design of future unified architectures.

Innovation 3: The Vision-Centric Attention Pattern as a New Diagnostic for Multimodal Architecture Comparison

The paper introduces a novel diagnostic framework for comparing multimodal architectures based on where the model allocates attention during prediction. By measuring the proportion of attention scores directed toward image tokens versus text tokens across transformer layers (Figure 4), the paper reveals a striking and previously undocumented systematic difference between single-transformer and modular MLLMs.

Single-transformer MLLMs (Sail, SOLO, EVE) allocate 60–80% of attention to image tokens across all layers. Modular MLLMs (Qwen2-VL, LLaVA-OneVision, LLaVA-1.5) allocate only 10–30% to image tokens, with LLaVA-1.5 showing an extreme pattern where image attention is high in the first two layers but collapses in deeper layers — a direct consequence of the architecture: visual tokens are projected into the LLM's input space but then diluted by the LLM's own processing, which was designed for text and has no structural incentive to maintain visual focus.

This finding is significant for three reasons. First, it provides a mechanistic explanation for the scalability differences observed in Figure 1A. The modular MLLM's low image attention suggests that the LLM treats visual information as auxiliary input rather than primary evidence — it "reads" the projected visual features in the first few layers, converts them into an internal representation, and then largely ignores them in deeper layers where reasoning occurs. Sail, by contrast, maintains direct access to raw visual information throughout the entire depth of the network, meaning that high-level reasoning in later layers can still reference fine-grained visual details. This explains why Sail scales better with data: each training sample provides richer learning signal for the entire network depth, while the modular system's deeper layers learn primarily from the intermediate representations produced by earlier layers rather than from the visual data directly.

Second, it introduces attention allocation as a new axis for comparing and diagnosing multimodal architectures. Prior work compared architectures solely on benchmark performance. The paper shows that different architectures have fundamentally different information flow patterns — attention distribution is a "hidden variable" that may explain performance differences and predict scaling behavior. This diagnostic could be applied to any multimodal architecture: if a model allocates very low attention to one modality, it is unlikely to benefit from additional training data in that modality.

Third, it explains why Sail functions effectively as a standalone vision encoder (Tables 3–4) while modular MLLMs' vision encoders cannot be extracted from the LLM. In a modular system, the vision encoder and LLM are separate; you can take the CLIP-ViT and use it for vision tasks, but the LLM's layers, having been trained to allocate minimal attention to visual tokens, would perform poorly if tasked with pure vision processing. Sail's vision-centric attention pattern means that the entire network functions as a vision processor, not just the early layers — every layer is actively attending to and transforming visual representations, making the full depth of the model available for vision tasks.

This is a fundamental insight, not an incremental improvement. It is a new way of understanding what makes multimodal architectures work, and it provides a principled framework for future architecture design: if you want a model to be good at vision, ensure that attention is allocated to visual tokens throughout the network, not just at the input.

Innovation 4: The Emergent Vision Encoder — Demonstrating That Multimodal Pretraining Produces Transferable Visual Representations Without Vision-Specific Objectives

The paper's most surprising empirical finding is that a single transformer trained solely with a language modeling objective on image-text pairs develops visual representations that rival or exceed purpose-built vision models on pure vision tasks, without any contrastive loss, masked image modeling, or supervised classification. Sail achieves 84.95% ImageNet top-1 accuracy and 55.3% mIoU on ADE20K semantic segmentation (Table 3), matching ViT-22B's segmentation performance despite ViT-22B being trained on 3B images (Dehghani et al., 2023) versus Sail's 512M images — a ~6× data advantage for Sail relative to ViT-22B's pretraining volume, or a ~30× parameter efficiency advantage considering the model size difference.

This finding is conceptually significant because it challenges a long-standing assumption in the vision representation learning community: that effective visual representations require either supervised classification (ImageNet pretraining), contrastive objectives (CLIP), or reconstruction-based self-supervision (MAE, BEiT). The paper demonstrates that autoregressive text prediction — a task with no explicit visual supervision — is sufficient to learn visual representations that transfer to classification, segmentation, and compositional reasoning tasks. The ARO results in Table 5 are particularly striking: Sail achieves 100% on three of four subtasks (Relation, Attribute, Order), essentially perfect compositional understanding, while contrastive models like OpenCLIP-G reach only 49.9% on Relation and 65.6% on Attribute. This suggests that the text-prediction objective forces the model to develop more detailed and semantically precise visual representations than contrastive objectives, because generating accurate captions requires distinguishing fine-grained attributes and relationships that contrastive matching can ignore (a model can match "a dog" to an image of a dog without knowing whether it's a "brown dog sitting on a red couch" — but to generate that caption, it must encode all those details).

The practical implication is substantial: a single pretraining run produces both an MLLM and a vision backbone, eliminating the need for separate vision encoder training pipelines. This is not just an architectural simplification; it reshapes the economics of model development. Organizations building multimodal systems can invest in one training process instead of two, with the vision representations emerging as a "free" byproduct of multimodal pretraining.

However, this claim comes with an important caveat that the paper does not fully address. The vision evaluations (Tables 3–5) use only the Stage 1 checkpoint — 224×224 resolution training. The Stage 2 any-resolution training and SFT are not evaluated for vision tasks. This means we do not know whether the additional training stages preserve or degrade the visual representations. It is possible that SFT, which fine-tunes the entire network for instruction-following, alters the visual features in ways that reduce their transferability to pure vision tasks. This is a limitation of the current analysis, not a flaw in the finding, but it means the "emergent vision encoder" claim has been validated only at the pretraining stage, not after the full training pipeline.

This contribution is fundamental in its implications — it challenges the assumption that vision and language require separate training objectives and architectures — but incremental in its implementation, since it emerges naturally from the standard language modeling loss without any novel training technique. The innovation is the discovery itself and what it implies for future architecture design.

Innovation 5: A Principled Recipe for Training Single-Transformer MLLMs — the Curriculum + Data-Mixing Framework

While the paper frames itself as an empirical analysis rather than a method proposal, it establishes a reproducible training recipe that prior single-transformer work had not systematized. The recipe has three interacting components that the paper demonstrates are individually necessary: (1) a two-stage curriculum (low-resolution bootstrapping then any-resolution refinement), (2) interleaving pure text data to prevent language capability degradation, and (3) mixed attention with multimodal positional encoding.

What makes this a genuine contribution rather than an obvious engineering choice is the ablation evidence showing each component is essential (Table 7): removing bidirectional image attention drops TextVQA by 3.9 points; removing pure text data during pretraining drops ScienceQA by 11 points (from 59.6 to 48.6) and GQA by 4.8 points. These are not marginal tweaks — disabling either design choice causes catastrophic degradation on specific capability axes. The TextVQA drop tells us bidirectional attention is essential for reading text in images; the ScienceQA drop tells us that pure text interleaving is essential for preserving world knowledge and reasoning capabilities that the LLM brought from its pretraining but would otherwise forget when trained exclusively on image-text pairs.

Prior single-transformer work either did not use these techniques (Fuyu used causal-only attention and did not report pure text interleaving) or used different, more complex approaches (EVE introduced auxiliary losses to boost visual learning; Chameleon used discrete visual tokens with a separate tokenizer). The paper demonstrates that these three simple design choices, applied together, are sufficient to train a competitive single-transformer MLLM without auxiliary objectives, separate tokenizers, or architectural modifications beyond the attention mask and positional encoding.

This is an incremental but practically significant contribution. It does not change how we think about multimodal learning conceptually, but it provides a validated, minimal recipe that other researchers can adopt and extend. It lowers the barrier to entry for single-transformer MLLM research by establishing clear defaults — bidirectional image attention, multimodal RoPE, pure text interleaving, and a two-stage curriculum — that remove the guesswork from architecture design. For practitioners, this recipe is actionable: if you are training a single-transformer MLLM, these three design choices are the table stakes; deviations from them require explicit justification.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary multimodal evaluation is conducted across 13 vision-language benchmarks encompassing diverse capabilities: MMStar (Chen et al., 2024b), MMBench-EN (Liu et al., 2024d), SEEDBench-IMG (Li et al., 2023a), MMVet (Yu et al., 2023), MME (Liang et al., 2024), POPE (Li et al., 2023c), HallusionBench (Guan et al., 2024), ScienceQA-IMG (Lu et al., 2022), TextVQA (Singh et al., 2019), MathVista_MINI (Lu et al., 2023), AI2D (Kembhavi et al., 2016), RealWorldQA (X.ai, 2024), and OCRBench (Liu et al., 2024e). All results are evaluated using VLMEvalKit (Duan et al., 2024) and compared against numbers reported in prior papers or the OpenCompass leaderboard. For vision representation evaluation, the paper uses ImageNet-1K (Deng et al., 2009) for classification (validation set), ADE20K (Zhou et al., 2017) for semantic segmentation, and ARO (Yuksekgonul et al., 2022) for compositional understanding of attributes, relations, and ordering.
Base model(s). The primary Sail model is initialized from Mistral-7B-v0.1 (Jiang et al., 2023), a 7B-parameter open-source LLM. For model scaling experiments (Figure 3), the paper additionally trains Sail-0.5B (initialized from Qwen2.5-0.5B) and Sail-3B (initialized from Qwen2.5-3B). The modular MLLM counterpart used in controlled comparisons employs a SigLIP-SO vision encoder (Zhai et al., 2023) with the same LLM backbone and initialization as Sail. Mistral-7B is described as "representative" but the choice is not extensively justified beyond it being a strong open-source model at the time.
Metrics. Vision-language benchmarks use their standard evaluation protocols as implemented in VLMEvalKit — this includes exact match accuracy for VQA tasks, MME perception/cognition scores (sum reported), MMBench accuracy, POPE F1/accuracy, and OCRBench scores. For scaling experiments (Figures 1A and 3), the paper computes a normalized average by converting each benchmark score to a 0–100 scale and averaging across the evaluated subset (VQAv2, GQA, ScienceQA-IMG, TextVQA, POPE, MME, MMBench, SEEDBench-IMG — detailed in Appendix Table 9). Vision tasks use standard metrics: ImageNet top-1 and top-5 accuracy, ADE20K mIoU/mAcc/aAcc, and ARO retrieval accuracy per subtask.
Baselines. The paper compares against three categories of models summarized in Table 2:
- Modular MLLMs: InternVL-1.5 (2.2B, Chen et al., 2024e), QwenVL-Chat (7B, Bai et al., 2023), LLaVA-1.5 (7B, Liu et al., 2024a), LLaVA-1.6 (7B, Liu et al., 2024b), Cambrian-1 (8B, Tong et al., 2024a), and LLaVA-OneVision (7B, Li et al., 2024a).
- Single-transformer MLLMs: Fuyu (8B, Bavishi et al., 2023), Chameleon (7B, Team, 2024), EVE (7B, Diao et al., 2024), SOLO (8B, Chen et al., 2024d), Mono-InternVL (3B, Luo et al., 2024), Emu3 (8B, Wang et al., 2024d), and EVE2 (7B, Diao et al., 2025).
- Vision backbones (Tables 3–5): EVE, EVE2, SOLO for single-transformer visual representations; OpenCLIP-H, OpenCLIP-G, ViT-22B, InternViT, CLIP-B/32, CLIP-L/14, NegCLIP, and CapPa for vision representation comparison. The paper also constructs a controlled modular MLLM counterpart (Section 4.3.1) using SigLIP-SO + the same LLM as Sail for the data scaling experiment in Figure 1A.
Generation budget / compute accounting. The paper does not use a standardized "test-time compute budget" metric (unlike the reference paper on inference-time scaling). Instead, fairness in comparisons is achieved through: (1) matching the number of visual input tokens across Sail and its modular counterpart in controlled experiments, (2) using identical training data and SFT recipes for the data scaling comparison (Figure 1A), and (3) reporting #Data (pretraining/fine-tuning data volume) and #Vtoken (maximum image patch tokens) in Table 2 to contextualize performance differences. Training compute is described in terms of GPU-hours (128 A100 80GB GPUs) but not translated into a standardized FLOP count.
Cross-validation / statistical protocol. The paper does not employ cross-validation or report confidence intervals. The scaling curves in Figures 1A and 3 plot single values per data point without error bars. The evaluation uses standard benchmarks with fixed test sets; there is no multiple-seed training or statistical significance testing. For the data scaling experiment, each data point (32M, 128M, 512M) represents a single training run. For model scaling, each size variant is trained once. The attention pattern analysis (Figure 4) aggregates over 1000 samples from multiple datasets, but no variance estimates are reported. This absence of statistical rigor is a meaningful limitation — with single training runs per configuration, the observed scaling trends could be influenced by training noise, data order effects, or initialization variance, and the reader has no way to assess the reliability of the claimed differences.

Main Quantitative Results

4.2.1 Vision-Language Benchmark Performance (Table 2)

The headline result: Sail achieves competitive performance with modular MLLMs while substantially outperforming all prior single-transformer MLLMs. On MMStar, Sail scores 53.1%, compared to LLaVA-OneVision's 60.9% (the top modular system) and SOLO's 35.8% (the next-best single-transformer at comparable scale). On MMBench-EN, Sail achieves 70.1% versus LLaVA-OneVision's 81.7% and SOLO's 67.7%. On SEEDBench-IMG, Sail scores 72.9% versus LLaVA-OneVision's 74.8% and SOLO's 64.4%. The key pattern: Sail narrows the gap to state-of-the-art modular systems substantially compared to prior single-transformer work, coming within ~3–12 percentage points of LLaVA-OneVision on several benchmarks while using significantly less fine-tuning data (6M vs. 3.2M SFT samples for LLaVA-OneVision, though LLaVA-OneVision also uses 10B+ pretraining samples).

The most striking single-transformer comparison is against EVE2, the concurrent state-of-the-art: Sail outperforms EVE2 on MMStar (53.1 vs. unreported), MMBench (70.1 vs. 66.3), SEEDBench (72.9 vs. 71.4), MMVet (46.3 vs. 45.0), RWQA (63.9 vs. 62.4), and OCRBench (783 vs. 702), despite EVE2 using 92M pretraining samples and 7.3M SFT samples. Against SOLO, Sail shows substantial gains on TextVQA (57.0 vs. 32.9, a 24.1-point gap) and MathVista (57.0 vs. 25.0, a 32-point gap), suggesting that the mixed attention and M-RoPE design particularly benefit tasks requiring fine-grained visual understanding (text reading, mathematical diagram interpretation).

However, Sail still trails modular MLLMs on several knowledge-intensive benchmarks. On MME, Sail scores 1719 (sum of perception + cognition), below LLaVA-OneVision's 1998 and InternVL-1.5's 1902. On ScienceQA-IMG, Sail achieves 93.3% versus LLaVA-OneVision's 96.6%. The paper attributes these gaps to the "lack of diverse domain-specific data during pretraining" (Section 4.3.3) and hypothesizes that scaling pretraining data diversity would close them — but this remains an untested hypothesis within the paper.

4.2.2 Vision Representation Learning (Tables 3–5)

The paper demonstrates that Sail's pretrained backbone functions as a competitive standalone vision encoder without any vision-specific fine-tuning of the backbone architecture.

Classification and Segmentation (Tables 3–4). Sail achieves 84.95% ImageNet-1K top-1 accuracy and 97.59% top-5 accuracy (Table 3), dramatically outperforming other single-transformer approaches: EVE (42.03%), EVE2 (44.86%), and SOLO (59.10%). On ADE20K semantic segmentation, Sail achieves 55.30% mIoU, compared to 27.12% (EVE), 40.85% (EVE2), and 35.11% (SOLO). The magnitude of these gaps — 25–42 percentage points on classification, 14–28 points on segmentation — indicates that prior single-transformer systems learned substantially worse visual representations despite comparable training objectives. This is likely attributable to Sail's mixed attention mechanism, which enables full bidirectional context within images, versus the causal-only attention used by EVE, EVE2, and SOLO.

Table 4 contextualizes Sail against dedicated vision backbones. Sail's 84.95% ImageNet top-1 is competitive with OpenCLIP-H (84.4%, trained on 2B samples) and approaches OpenCLIP-G (86.2%, 1.8B parameters, 2B samples). Most notably, Sail's 55.3% ADE20K mIoU matches ViT-22B's 55.3% mIoU, despite ViT-22B being trained on 3B images with 22B parameters — Sail achieves this with 512M images and 7B parameters (~6× less data, ~3× fewer parameters). The comparison is not perfectly controlled (different training objectives, architectures, and evaluation pipelines), but the parsimony of achieving competitive results with orders of magnitude less data is the key claim.

Compositional Understanding (Table 5). The ARO benchmark results are striking. Sail achieves 100.0% on Relation, 99.5% on Attribute (COCO), 100.0% on Attribute (Flickr30K), and 100.0% on Order subtasks — essentially perfect performance on three of four metrics and near-perfect on the fourth. This dramatically exceeds contrastive models: OpenCLIP-H scores 49.9% on Relation and 64.6% on Attribute; CLIP-L/14 scores 61.2% on Relation and 61.7% on Attribute. Even CapPa, a captioning-based model that is architecturally closest to Sail, achieves 86.7% on Relation and 85.7% on Attribute. The paper interprets this as evidence that "the pre-trained Single Transformer learns transferable vision representations" that capture fine-grained compositional structure. The near-perfect scores are unusual and warrant scrutiny — ARO is a retrieval benchmark where the model selects the matching caption from hard negative distractors (e.g., "red cube on blue sphere" vs. "blue sphere on red cube"). A score of 100% suggests the model never confuses these compositional variations, which is remarkable and not fully explained by the paper.

4.3.1 Scaling Properties (Figures 1A, 3; Appendix Tables 8–9)

Data Scaling (Figure 1A). The controlled comparison between Sail and its modular MLLM counterpart reveals a crossover pattern. At 32M pretraining samples, the modular MLLM achieves a normalized average score of approximately 61.4 versus Sail's 51.9 — a gap of ~9.5 points. At 128M samples, the gap narrows to approximately 61.5 vs. 57.9 (~3.6 points). At 512M samples, the modular MLLM scores approximately 61.9 versus Sail's 60.5 — a gap of only ~1.4 points on the normalized metric. The detailed per-benchmark numbers in Appendix Table 9 confirm this trend: Sail's TextVQA improves from 31.67 (32M) to 56.86 (128M) to 63.94 (512M), while the modular MLLM's TextVQA remains relatively flat (58.68 → 59.82 → 57.85) and actually declines slightly at 512M.

The steepness of Sail's scaling curve — it gains ~8.6 normalized points from 32M to 512M versus the modular MLLM's ~0.5 points — is the paper's central evidence for superior data scalability. However, the normalized average metric compresses substantial inter-benchmark variance. Appendix Table 9 shows that Sail's MMBench score, for example, improves only from 48.22 (32M) to 53.94 (128M) to 56.71 (512M) — gains that exist but are modest compared to the TextVQA jump. The aggregate curve is driven disproportionately by specific benchmarks where Sail makes dramatic progress (TextVQA, SEEDBench) while others show more incremental improvement.

Model Scaling (Figure 3). Three model sizes are compared: Sail-0.5B, Sail-3B, and Sail-7B, all pretrained on 128M image-text pairs and fine-tuned on LLaVA-mix-665K. The left panel shows that training language modeling loss decreases with model size, which is expected. The right panel plots normalized average benchmark performance: Sail-0.5B achieves approximately 39.9 (Appendix Table 9 reports per-benchmark numbers confirming this), Sail-3B achieves approximately 47.8, and Sail-7B achieves approximately 57.9. The monotonic improvement with model size is consistent but the paper does not fit a scaling law or predict performance at larger scales. The jump from 3B to 7B (~10 normalized points) is larger than from 0.5B to 3B (~8 points), suggesting potential superlinear scaling in this range, but with only three data points and single training runs, this is suggestive rather than conclusive.

Task-Specific Performance Analysis (Table 6). The paper dissects Sail's strengths and weaknesses through targeted MME and MMBench sub-splits, comparing against the modular MLLM counterpart under the 512M data scaling setting. On MMBench Physical Relation, Sail scores 52.2 versus the modular MLLM's 30.4 — a 21.8-point advantage. On MME Position, Sail scores 160.0 versus 134.0 — a 26-point advantage. These results support the claim that Sail excels at spatial reasoning, which the paper attributes to "tighter coupling between visual geometry and linguistic descriptions" in the unified architecture.

Conversely, on knowledge-intensive tasks, Sail underperforms. On MMBench Celebrity Relation, Sail scores 88.9 versus 50.5 for the modular MLLM (an anomaly — this appears to be a transcription error in the paper, as the text describes Sail underperforming on this task, but the table shows Sail outperforming by 38.4 points; the numbers for MME Posters show 108.2 vs. 100.3, a smaller gap). On MME Celebrity, Sail scores 75.0 versus 100.3 for the modular MLLM — a 25.3-point deficit that is consistent with the paper's claim of weakness in world knowledge. The paper attributes this to Sail's pretraining data lacking diversity in domain-specific knowledge that CLIP/SigLIP encoders absorbed during their contrastive pretraining on billions of image-text pairs.

4.3.2 Information Flow Pattern (Figures 4, 6; Appendix Figures 6–7)

Attention allocation comparison (Figure 4). The paper measures the proportion of attention scores directed to image tokens (versus text tokens) across all transformer layers during token prediction. Single-transformer MLLMs (Sail, SOLO, EVE) allocate 60–80% of attention to image tokens across all layers, with a relatively flat distribution. Modular MLLMs show much lower image attention: Qwen2-VL allocates approximately 10–25%, LLaVA-OneVision allocates 10–20%, and LLaVA-1.5 shows an extreme pattern where image attention is high in layers 0–2 (approximately 60–80%) but collapses to ~10% by layer 4 and remains low through all deeper layers. The paper attributes this LLaVA-1.5 pattern to the frozen ViT encoder — visual features enter the LLM as a fixed representation that the early layers process, but deeper layers, which were pretrained for text-only reasoning, have no mechanism to maintain visual grounding and effectively "forget" the image tokens.

Controlled comparison (Appendix Figure 6). To address the confound that different models process different numbers of visual tokens, the paper conducts a controlled experiment where both Sail and the modular counterpart use identical resolution (224×224, 256 visual tokens) during SFT. The results confirm the headline finding: Sail allocates higher attention to image tokens across all layers, with the most dramatic differences in medium layers (+43.5% in layer 14) and deep layers (+41.2% in layer 31). This controlled comparison rules out the alternative explanation that the attention differences are artifacts of token count variation.

Attention map visualization (Appendix Figure 7). The paper visualizes Sail's attention from predicted tokens back to image regions. In early layers, attention focuses broadly on salient image regions. In deeper layers, attention narrows to regions specifically relevant to the token being predicted. This progressive focusing is presented as evidence that Sail "has the potential to function as a grounding model, effectively correlating text tokens with their corresponding image regions."

Ablation Studies and Robustness Checks

Bidirectional attention within image patches (Table 7, ablation #1): Removing bidirectional image attention and using causal-only attention with 1D positional encoding degrades performance across all five evaluated benchmarks. VQAv2 drops from 59.1 to 57.8 (−1.3), GQA from 46.9 to 45.2 (−1.7), ScienceQA-IMG from 59.6 to 58.7 (−0.9), SEEDBench-IMG from 35.1 to 33.8 (−1.3), and TextVQA from 20.1 to 16.2 (−3.9). The disproportionate impact on TextVQA (19% relative decline versus 2–4% on other benchmarks) confirms the paper's hypothesis that bidirectional context is especially important for reading text in images, where characters are distributed across multiple patches in 2D arrangements.

Interleaving pure text data during pretraining (Table 7, ablation #2): Removing SlimPajama text data and training only on image-text pairs causes substantial degradation. VQAv2 drops from 59.1 to 56.3 (−2.8), GQA from 46.9 to 42.1 (−4.8), ScienceQA-IMG from 59.6 to 48.6 (−11.0), SEEDBench-IMG from 35.1 to 32.4 (−2.7), and TextVQA from 20.1 to 18.3 (−1.8). The ScienceQA-IMG drop is catastrophic — an 11-point decline that represents a 18.5% relative reduction. This is the strongest evidence for the paper's claim that "preserving language capabilities in the LLM when training Single Transformer models" is "crucial for building a multimodal model capable of complex reasoning" (Section 4.4). ScienceQA requires both visual understanding and scientific knowledge; the large drop suggests that pure text interleaving preserves the factual and reasoning knowledge the LLM acquired during its original pretraining.

Sail-0.5B variant for ablations (Table 7): The ablation experiments use Sail-0.5B (Qwen2.5-0.5B base) pretrained on 128M image-text pairs and fine-tuned on LLaVA-mix-665K, rather than the full Sail-7B. This is an important methodological choice: running ablations on the smaller model makes the experiments computationally feasible, but it assumes that the relative importance of design choices (bidirectional attention, pure text mixing) transfers across model scales. The paper does not validate this assumption with follow-up ablations on the 7B model.

Controlled comparison with LLaVA-1.5 (Appendix Table 10): To isolate the effect of architecture from training data and evaluation protocol differences, the paper compares Sail (512M Stage 1 pretraining, LLaVA-mix-665K SFT, no any-resolution) against LLaVA-1.5 (558K pretraining, same SFT data) with matched image processing (no any-resolution for either model). Sail achieves a normalized average of 59.1% across 10 benchmarks versus LLaVA-1.5's 58.3%, with Sail outperforming on DocVQA (29.3 vs. 28.1), ChartQA (21.5 vs. 18.2), AI2D (58.7 vs. 54.8), and MMStar (37.1 vs. 32.4), while slightly underperforming on VQAv2 (77.8 vs. 78.5) and MMBench (61.3 vs. 64.3). This comparison is the cleanest demonstration that Sail's architecture can match modular performance when training and evaluation protocols are aligned, despite Sail using no pretrained vision encoder.

Comparison on MMVP visual perception benchmark (Appendix Figure 5): Qualitative examples show Sail outperforming LLaVA-1.5 on the MMVP benchmark (Tong et al., 2024b), which tests fine-grained visual perception through hard negative examples. Sail correctly identifies minor objects and distinguishes object states (e.g., whether a clock shows a specific time) where LLaVA-1.5 fails. This provides qualitative support for the claim that Sail's vision-centric attention pattern leads to more detailed visual perception.

Critical Assessment

Claim: Sail exhibits superior data scalability compared to modular MLLMs.

This claim is supported by the controlled experiment in Figure 1A, which shows Sail's performance improving more steeply than the modular counterpart across three data scales (32M, 128M, 512M). However, several qualifications are necessary. First, the "modular MLLM counterpart" uses SigLIP-SO as the vision encoder — a model pretrained on 40B samples. The comparison is therefore between a system that has already seen 40B image-text pairs (via its frozen encoder) and one learning from scratch. The modular system's relative flatness at 32M–512M is partly because its visual representations are already saturated from SigLIP pretraining; Sail's steepness reflects catching up from zero, not necessarily a fundamentally different scaling exponent. A fairer comparison would initialize the modular system's vision encoder from scratch and train both architectures from 0 to 512M samples, but this experiment is not performed. The paper's framing — that single transformers are "more scalable" — is technically accurate for the regime tested (adding multimodal data on top of a pretrained LLM), but the mechanism (learning from scratch vs. inheriting pretrained features) is confounded with the architecture.

Second, with only three data points (32M, 128M, 512M), the scaling curve is sparsely sampled. The curves in Figure 1A are plotted as continuous lines but are effectively three-point interpolations. The shape of the curve between 128M and 512M is unknown — it could be linear, sublinear, or inflecting. The paper does not fit a power law or estimate the data required for Sail to definitively surpass the modular system. Extrapolating from three points to predict future scaling behavior is speculative.

Third, the evaluation subset for Figure 1A (VQAv2, GQA, ScienceQA, TextVQA, POPE, MME, MMBench, SEEDBench) does not include OCR-heavy benchmarks (DocVQA, ChartQA, OCRBench) or complex reasoning benchmarks (MathVista, MMVet) that appear in the full Table 2 comparison. The scaling behavior observed on this subset may not generalize to the full benchmark suite. Table 2's Sail was trained with Stage 2 any-resolution data and SFT on InfinityMM, not just the Stage 1 + LLaVA-mix-665K pipeline used in the scaling experiment, so the full model's scaling properties may differ.

Claim: Single-transformer MLLMs allocate significantly more attention to image tokens (60–80% vs. 10–30%), indicating a vision-centric information flow pattern.

The evidence in Figures 4 and Appendix Figure 6 is clear and well-controlled. The finding is robust: single-transformer MLLMs consistently allocate more attention to image tokens than modular MLLMs, and this holds across different models (Sail, SOLO, EVE vs. Qwen2-VL, LLaVA-OneVision, LLaVA-1.5) and with controlled token counts (Appendix Figure 6). However, the paper's interpretation — that this represents a "vision-centric approach to decision-making" and is inherently advantageous — is not experimentally validated. The paper does not demonstrate that higher image attention causes better performance; it could be a correlational artifact of the architecture (single transformers must process visual information through self-attention because there is no separate encoder, while modular MLLMs can route visual information through cross-attention or learned projector representations that don't register as "image token attention" in this analysis). The causal relationship between attention allocation and task performance is not tested (e.g., by manipulating attention patterns and measuring downstream accuracy).

Additionally, the attention analysis is conducted on 1000 samples from "various datasets," but the paper does not specify whether the attention patterns are consistent across task types. It is plausible that easy VQA questions require different visual attention than complex diagram understanding, and averaging across tasks may obscure task-specific dynamics.

Claim: The pretrained Sail functions as a powerful standalone vision encoder, matching ViT-22B on segmentation with far less data.

The numbers in Tables 4–5 are striking and the claim is supported by the reported metrics. However, the comparisons are not perfectly controlled. Sail is evaluated using ViT-Adapter with UperNet for segmentation, while ViT-22B's evaluation protocol may differ (the paper mentions OpenCLIP-G uses "head tuning using UperNet" while others use ViT-Adapter, but does not specify ViT-22B's exact evaluation setup). The ImageNet evaluation uses an attention-based classifier with 90 epochs of linear probing — a specific protocol that may favor or disfavor different backbone architectures. The paper acknowledges that configurations are "mostly obtained from common practices" but does not verify that evaluation pipelines are identical across the compared models. This matters because segmentation performance is sensitive to the decoder architecture, input resolution, and training schedule, and small differences in protocol can account for several mIoU points.

The near-perfect ARO scores (100% on three of four subtasks, Table 5) warrant particular scrutiny. ARO is a retrieval task where the model selects the correct caption from a set of hard distractors. Achieving 100% suggests the model never confuses, for example, "the red cube on top of the blue sphere" with "the blue sphere on top of the red cube." While this indicates strong compositional understanding, it is unusual to see perfect performance on any benchmark, and the paper does not discuss potential evaluation issues — such as whether the retrieval protocol could saturate, whether the evaluation uses the same captions seen during training, or whether the model achieves these scores through genuine compositional reasoning versus memorization of caption patterns. Given that CapPa, a strong captioning-based model, achieves only 86.7–99.2%, Sail's perfect scores deserve more analysis than the paper provides.

A critical missing experiment: all vision evaluations use the Stage 1 checkpoint only (224×224 pretraining, frozen). The paper does not evaluate whether Stage 2 any-resolution training or SFT preserves or degrades these visual representations. Since SFT fine-tunes the entire network for instruction-following, it is plausible that the visual features shift in ways that reduce transferability to pure vision tasks. If Sail after SFT performs worse on ImageNet classification than Sail after Stage 1, the "emergent vision encoder" claim would only apply to the pretrained model, not the deployed MLLM. This gap is significant for the practical claim that a single training run produces both an MLLM and a vision backbone — the paper demonstrates this for the pretraining stage but not for the complete pipeline.

Claim: Sail outperforms all prior single-transformer MLLMs on vision-language benchmarks.

This claim is well-supported by Table 2. Sail achieves the highest scores among single-transformer models on 11 of 13 benchmarks (the exceptions are MME, where EVE2 scores 1709 — essentially tied with Sail's 1719 — and POPE, where Chameleon's 88.3 slightly exceeds Sail's 85.8, though the POPE comparison is complicated by different evaluation protocols). The margins are substantial on several benchmarks: +7.8 points over the next-best single-transformer on MMStar, +3.0 on MMBench, +1.5 on SEEDBench, +24.1 on TextVQA, +25.7 on MathVista, +5.1 on AI2D, +1.3 on OCRBench. These are not marginal improvements; they represent a clear step forward for the single-transformer approach.

However, the comparison is confounded by differences in pretraining data volume, SFT data, and model initialization. EVE2 uses 92M pretraining + 7.3M SFT samples; SOLO uses 43.7M pretraining + 2M SFT; Sail uses 512M + 6M (or the full pipeline with 598M + 6M). The paper does not control for data quantity when comparing against other single-transformer models, so the performance advantage could be driven as much by data scale as by architectural choices. The model scaling experiment (Figure 3) partially addresses this by showing that Sail benefits from increased model size, but a controlled comparison where all single-transformer models are trained on identical data would be more informative.

Weaknesses in experimental design not addressed by the paper:

Single training runs with no error bars: Every data point in Figures 1A, 3, and Table 2 represents a single training run. Without multiple seeds, the observed differences could be within training variance. For the scaling experiment, where the claimed advantage narrows from 9.5 points at 32M to 1.4 points at 512M, variance estimates would help assess whether the near-parity at 512M is reliable or coincidental.
No evaluation of language-only capability preservation: The paper emphasizes the importance of interleaving pure text data to prevent catastrophic forgetting, but it never reports language-only benchmark performance (e.g., MMLU, HellaSwag, GSM8K) before and after multimodal pretraining. The ablation in Table 7 shows that removing text data hurts multimodal benchmarks, but this is indirect evidence — the model could be preserving language capabilities or simply using text data as a regularizer that improves multimodal learning through a different mechanism. A direct measurement of language capability retention would strengthen the claim.
The normalized average metric obscures benchmark-specific dynamics: The scaling curves aggregate eight benchmarks with different score ranges and difficulty levels. TextVQA, where Sail improves dramatically (32 → 64), and MME, where the modular MLLM's score actually drops at 512M (1638 → 1530 for Sail vs. 1379 for modular), have disproportionate influence on the aggregate trend. Disaggregated scaling plots would reveal whether the "superior scalability" is driven by a few benchmarks or is a uniform phenomenon.
Missing head-to-head with EVE2 on equal data: EVE2 is the most competitive prior single-transformer model, but was published concurrently and trained on different data. The paper cannot control for this, but it would strengthen the analysis to at minimum discuss whether EVE2's lower data volume (92M vs. 512M pretraining) likely accounts for the performance gap, or whether architectural differences (mixed attention, M-RoPE) provide independent benefits.
The vision encoder evaluation does not use the final model: As noted above, all vision evaluations use the Stage 1 checkpoint. The paper does not evaluate whether the complete Sail (after Stage 2 and SFT) retains competitive vision representations. If it does not, the practical value of the "emergent vision encoder" is limited to the pretrained checkpoint, which cannot simultaneously serve as an instruction-following MLLM.
ARO scores at ceiling leave no room for analysis: Perfect or near-perfect scores on a benchmark provide no discriminatory power — one cannot tell whether Sail is marginally better than alternatives or fundamentally different. The paper's interpretation (that Sail "encodes compositional relationships... almost perfectly") is reasonable given the scores, but the ceiling effect means the ARO results cannot distinguish between Sail and a hypothetical system that is slightly worse but still saturates the benchmark. This limits the diagnostic value of the ARO comparison.
Ablations performed only at 0.5B scale: The paper's key design choices (bidirectional attention, pure text interleaving) are ablated on Sail-0.5B, not Sail-7B. The assumption that findings transfer across a 14× parameter increase is untested. It is plausible, for example, that larger models are more robust to the removal of pure text data because they have greater capacity to retain linguistic knowledge from their pretrained initialization, or that bidirectional attention becomes more important at larger scales where the model has greater capacity to exploit spatial context. The paper would be strengthened by at minimum a partial ablation on the 7B model confirming the key trends.

6. Limitations and Trade-offs

Single Benchmark Family, Single Model Initialization, No Cross-Domain Validation

The assumption or constraint. All experiments, from scaling analysis to attention diagnostics to vision representation evaluation, are conducted on a narrow empirical foundation. The vision-language benchmarks are drawn from the standard MLLM evaluation suite (MMBench, SEEDBench, MMStar, etc.) which, while diverse in task format, share a common heritage — they test visual question answering, captioning, and reasoning on natural images, documents, and charts. The base model is Mistral-7B-v0.1 for the primary experiments, with Qwen2.5 variants used only for model scaling ablations. The paper does not test on video understanding, embodied reasoning, code generation from screenshots, or multimodal dialogue beyond single-turn QA. The authors do not claim broader applicability, but the paper's title — "The Scalability of Simplicity" — implies general principles that should hold beyond the specific model-data-benchmark combination tested.

The consequence. A practitioner cannot know whether the paper's three headline findings — superior data scalability, vision-centric attention patterns, and emergent vision encoder capability — transfer to other model families (LLaMA, Gemma, DeepSeek), other data distributions (non-English, domain-specific like medical or satellite imagery), or other task types (interactive agents, video understanding, embodied AI). The scaling curve in Figure 1A shows Sail catching up to a modular MLLM between 32M and 512M image-text pairs, but this crossover point is almost certainly a function of the base model's pretraining quality, the data distribution, and the benchmark difficulty. A weaker base LLM might require more multimodal data to catch up; a stronger one might surpass the modular system earlier. The absence of any cross-model-family replication means the paper demonstrates an existence proof (this can work with Mistral-7B on these benchmarks) rather than establishing a reliable phenomenon.

What evidence exists in the paper. The paper does not address this limitation. There is no experiment varying the base LLM family, no evaluation on out-of-domain benchmarks (e.g., medical VQA, satellite imagery classification, non-English multimodal tasks), and no discussion of how the findings might depend on the specific properties of Mistral-7B or the Recap-DataComp-1B pretraining data. The model scaling experiment (Figure 3) tests three sizes within the same architectural lineage (Sail with Qwen2.5 or Mistral initializations), which provides evidence about within-family scaling but says nothing about cross-family transfer. The paper's closest analogue to a robustness check is the comparison of Sail against multiple single-transformer baselines (SOLO, EVE, EVE2) in Table 2, but these models use different base LLMs with different pretraining data at different scales, making it impossible to isolate whether performance differences stem from architecture, data, or initialization.

Mitigation status. Not addressed. The paper does not frame this as a limitation or suggest cross-model-family replication as future work. The Discussion and Future Work section (Section 5) focuses on extending the approach to larger scales and combining with search/revision techniques, not on validating generalizability. This is a significant gap because the paper's value proposition — "simplicity scales" — requires evidence that simplicity scales reliably, not just once under specific conditions.

Difficulty Estimation is Free Only Because the Paper Ignores Its Cost

The assumption or constraint. Throughout the paper, Sail's performance is evaluated after selecting which training configuration, model size, or data scale to use for each experiment. But the paper's central practical argument — that single transformers are simpler to deploy because they eliminate the vision encoder — implicitly assumes that the remaining complexity (choosing resolution, managing sequence length with any-resolution images, balancing image-to-text ratios in training batches) is zero-cost or solved. More critically, the paper never accounts for the cost of determining how much data or model scale is needed to reach parity with modular systems. The scaling experiment in Figure 1A shows that Sail needs roughly 512M image-text pairs to approach the modular MLLM's performance. A practitioner reading this paper wants to know: if I train a single-transformer MLLM on my data, how much data do I need? The paper provides one data point (512M for ~parity with SigLIP-based modular system on this benchmark subset) but no principled way to estimate this for new domains, new model families, or new benchmarks.

The consequence. The practical cost of adopting the single-transformer approach includes substantial data exploration overhead that the paper's headline numbers do not capture. The modular MLLM approach provides something akin to a "pretrained visual knowledge floor" — you inherit a vision encoder trained on billions of samples, and your alignment stage (hundreds of thousands to millions of samples) builds on that foundation. The single-transformer approach requires you to discover how much data you need by training and evaluating at multiple scales, which is itself computationally expensive. If a team trains Sail-7B at 32M, 128M, and 512M scales to determine that 512M is sufficient, the total cost of exploration (32M + 128M + 512M = 672M samples of training) may exceed the cost of simply training the modular system once. The paper's scaling curve is a valuable scientific result but an expensive practical one to reproduce.

Additionally, the paper's difficulty analysis for task-specific performance (Table 6) shows that Sail underperforms modular MLLMs on knowledge-intensive tasks (e.g., MME Celebrity: 75.0 vs. 100.3). The paper hypothesizes that "scaling up Sail's pretraining data diversity could help bridge this gap." But how much? On what data? The paper provides no guidance, leaving a practitioner to guess at the data volume and composition needed to close specific capability gaps. This shifts the cost from "train once with a modular architecture" to "train multiple times with different data mixtures to find the right recipe," which is a hidden tax on the simplicity claim.

What evidence exists in the paper. The paper does not measure or discuss the cost of determining the right data scale. The scaling experiment (Figure 1A) reports results at three data points but does not characterize the computational cost of reaching each point, or the total cost of the exploration. The pretraining infrastructure section (Section 4.1) notes that Sail uses 128 A100 80GB GPUs, but does not report training duration or total GPU-hours for any experiment. The paper also does not provide any extrapolation tool — no fitted scaling law, no power-law exponent, no estimate of data required for convergence — that would help practitioners predict data needs without running multi-scale experiments.

Mitigation status. Not addressed. The paper does not discuss this as a limitation. It frames the scaling curve as evidence of Sail's superior data scalability without acknowledging that the need to scale data is itself a cost that modular systems avoid. This is a framing issue: the paper treats "catches up at 512M" as good news, but a practitioner might reasonably view "requires 512M samples to match a system that works well at 32M" as a significant barrier to adoption, especially for teams without access to massive multimodal datasets or the compute to train on them.

Vision Encoder Evaluated Only at the Pretraining Checkpoint — The Production Model is Untested

The assumption or constraint. The paper's claim that Sail "functions as a powerful vision encoder" and "achieves results on par with ViT-22B in vision tasks" (Section 1) is based entirely on evaluating the Stage 1 pretraining checkpoint with the backbone frozen (Section 4.1, "For vision, we load the checkpoint after Stage 1 pretraining and keep it frozen for downstream evaluations"). The full Sail pipeline includes Stage 2 any-resolution pretraining (86M additional image-text pairs, Table 1) and supervised fine-tuning on 6M instruction samples. Neither of these stages is evaluated for vision representation quality. The paper's abstract, introduction, and conclusion present the vision encoder capability as a property of "Sail" without qualification, but the evidence supports this claim only for an intermediate checkpoint, not the deployed model.

The consequence. There is a significant risk that the visual representations degrade during later training stages. Stage 2 any-resolution training exposes the model to images at varying resolutions and aspect ratios, which could shift the feature space away from the 224×224 regime evaluated on ImageNet and ADE20K. More critically, the SFT stage fine-tunes the entire network architecture (Section 3.3: "This phase fine-tunes the entire network architecture") on instruction-following data. Instruction-tuning is known to shift model outputs toward human-preferred response formats, but it can also degrade capabilities that are not explicitly reinforced. If the SFT data emphasizes dialog, reasoning, and OCR tasks, the model may de-prioritize the fine-grained spatial representations that drove the strong segmentation and ARO results.

A practitioner who reads "Sail achieves 55.3% mIoU on ADE20K segmentation" and attempts to use the instruction-tuned Sail model as a vision backbone for a downstream segmentation task may discover that the representations have shifted and performance is substantially lower. The paper provides no evidence either way — the instruction-tuned model's vision quality is an unknown. This matters because the paper's rhetorical structure implies that one model serves all purposes — MLLM and vision encoder simultaneously. The evidence supports this only for the pretrained model, which lacks instruction-following capability. Using the vision-optimized checkpoint means sacrificing the MLLM capability; using the instruction-tuned checkpoint may sacrifice vision quality. The paper presents these as a unified achievement, but they are demonstrated on different checkpoints from different stages of the pipeline.

What evidence exists in the paper. The paper explicitly states the evaluation protocol in Section 4.1: "For vision, we load the checkpoint after Stage 1 pretraining and keep it frozen for downstream evaluations." This is an honest disclosure, but it is buried in the implementation details and not flagged as a limitation. The paper never evaluates visual representation quality after Stage 2 or after SFT. The ARO benchmark (Table 5) is similarly evaluated only on the Stage 1 checkpoint. The contrast between the careful disclosure in the methods section and the unqualified claims in the abstract and introduction ("Sail demonstrates strong visual representation capabilities, achieving results on par with ViT-22B in vision tasks such as semantic segmentation") creates a misleading impression for readers who do not carefully parse the evaluation protocol.

Mitigation status. Not addressed. The paper does not discuss this as a limitation, does not evaluate vision quality at later training stages, and does not propose methods to preserve visual representations during SFT. The separate work "Pixel-Sail" (Zhang et al., 2025, cited in Section 4.2.2) may address pixel-level understanding from the full model, but the main paper does not bridge this gap. A simple experiment — running the ImageNet linear probe or ADE20K segmentation protocol on the Stage 2 and SFT checkpoints — would quantify the degree of representational shift and establish whether the vision encoder capability survives the full training pipeline. Its absence is a notable gap for a paper whose third main contribution is the vision encoder claim.

The Hardest Visual Reasoning Tasks Remain Unsolved — No Evidence That Single Transformers Address Fundamental Capability Limits

The assumption or constraint. The paper demonstrates that Sail matches or approaches modular MLLM performance on standard vision-language benchmarks, but this success is concentrated on tasks where the performance ceiling is relatively accessible. On the most challenging benchmarks in the evaluation suite, Sail's performance remains substantially below state-of-the-art modular systems, and the paper provides no evidence that scaling the single-transformer approach will close these gaps.

The consequence. The paper's narrative — that single transformers are a scalable, simplicity-driven alternative to modular MLLMs — is valid for tasks where the model's existing capabilities suffice, but it breaks down on tasks that require visual reasoning at the frontier of current model capability. The gaps on MME (1719 vs. LLaVA-OneVision's 1998, a 279-point deficit on a 2000+ point scale) and MathVista (57.0 vs. 56.1 for LLaVA-OneVision, essentially tied but well below human performance) indicate that the single-transformer approach has not unlocked new capabilities; it has matched existing capabilities through a different architectural route. For a practitioner deciding whether to adopt Sail, the relevant question is not "does it match on average" but "does it fail on the tasks I care about?" The paper's task-specific analysis (Table 6) shows clear failure modes on knowledge-intensive tasks (MME Celebrity: 75.0 vs. 100.3 for the modular counterpart), which are precisely the tasks that require the world knowledge embedded in pretrained vision encoders like CLIP and SigLIP.

The limitation is structural: single transformers learn visual representations and world knowledge simultaneously from image-text pairs, while modular systems can leverage vision encoders pretrained on massive contrastive datasets that explicitly encode broad visual world knowledge. The paper's data scaling experiment (Figure 1A) shows Sail catching up on an average metric, but the per-benchmark details in Appendix Table 9 reveal that some gaps persist or even widen. For example, on MME, the modular MLLM scores 1599 at 32M → 1638 at 128M → 1379 at 512M (an anomalous decline the paper does not explain), while Sail scores 1421 → 1458 → 1530. Sail is improving, but at 512M it still trails the modular system's 32M performance on this benchmark. This suggests that some capabilities encoded in the pretrained vision encoder are not easily recoverable through additional multimodal pretraining, at least within the data scales tested.

What evidence exists in the paper. Table 2 shows Sail's performance relative to modular MLLMs across 13 benchmarks. Sail trails LLaVA-OneVision on 9 of 13, with particularly large gaps on MME (−279 points) and MMStar (−7.8 percentage points). Table 6 shows the knowledge-intensive sub-split deficits. The paper acknowledges this in Section 4.3.3: "this underperformance can be attributed to Sail's lack of diverse domain-specific data during pretraining, a gap that was not sufficiently addressed during supervised fine-tuning." It hypothesizes that "scaling up Sail's pretraining data diversity could help bridge this gap." But this is a hypothesis, not a demonstrated solution, and it comes with no estimate of how much or what kind of data would be needed.

Mitigation status. The paper identifies the issue and offers a forward-looking hypothesis (more diverse pretraining data), but does not test it. There is no experiment varying pretraining data diversity, no ablation showing that adding specific data types (e.g., entity-rich captions, encyclopedic image-text pairs) reduces the knowledge gap, and no scaling projection estimating when parity might be achieved. The limitation is acknowledged but unresolved. For a practitioner, this means the single-transformer approach currently requires accepting a performance penalty on knowledge-intensive tasks, with no clear remediation path other than the generic (and expensive) prescription of "more diverse data."

The Paper's Core Diagnostic (Attention Allocation) is Correlational, Not Causal, and May Be an Architecture Artifact

The assumption or constraint. The paper's second main finding — that single-transformer MLLMs allocate 60–80% of attention to image tokens versus modular MLLMs' 10–30%, indicating "vision-centric information flow" — is presented as evidence of a fundamental architectural advantage. The paper interprets higher image attention as a mechanism that "enables tighter coupling between visual geometry and linguistic descriptions" (Section 4.3.3) and explains Sail's superior spatial reasoning. However, the paper never establishes a causal relationship between attention allocation and task performance. The correlation is clear; the causal mechanism is assumed.

The consequence. There are at least two alternative explanations for the attention pattern differences that the paper does not rule out:

Architectural artifact explanation: In a modular MLLM, visual information enters the LLM as projected features from a frozen encoder. These features are already highly processed — they represent the output of a deep ViT trained to produce representations suitable for contrastive matching. It is possible that the LLM processes these features efficiently in early layers, extracts the relevant information, and then routes it through the LLM's internal representations (MLP activations, attention to previous text tokens) in deeper layers, without needing to continually "look back" at the raw visual token positions. The low image attention in deeper layers may not indicate that visual information is being ignored, but rather that it has been absorbed into the LLM's hidden state and is propagated forward through text-token attention and FFN transformations. The paper's attention-based diagnostic cannot distinguish between "image information is ignored in deeper layers" and "image information is integrated and propagated through non-image-token pathways." The single transformer has no such alternative pathway — it must attend to image tokens to access visual information because there is no separate encoder producing a compact feature representation. The high image attention may be an architectural necessity, not a design advantage.

Token count confound: Even in the controlled experiment (Appendix Figure 6, 224×224 images), the number of image tokens (256) typically exceeds the number of text tokens in the prefix before generation begins. If the model allocates attention proportionally to the number of tokens of each type, 60–80% attention to image tokens might simply reflect that images make up 60–80% of the sequence. The paper does not normalize attention scores by token count in each modality, which would reveal whether individual image tokens receive disproportionate attention or whether the aggregate percentages reflect token-count ratios.

Without establishing causality — for example, by training a single-transformer model with artificially constrained image attention and measuring the performance impact, or by manipulating the modular MLLM's attention pattern to increase image attention and observing whether performance improves — the paper's central diagnostic remains an interesting observation rather than an explanatory mechanism.

What evidence exists in the paper. The attention analysis in Figure 4 and Appendix Figure 6 is correlational. The paper shows that single-transformer models have higher image attention and also perform better than other single-transformer models (Sail vs. SOLO vs. EVE), but it does not show that the attention pattern causes the performance improvement. The ablation in Table 7 shows that removing bidirectional image attention degrades performance, but this manipulation changes the attention pattern (causal vs. bidirectional), not the allocation (how much attention goes to images vs. text). These are different variables. The paper also does not report whether the attention allocation pattern correlates with performance within a model — do examples where Sail allocates more attention to images yield more accurate answers? This within-model analysis would be straightforward and would provide stronger evidence for the causal claim.

Mitigation status. Not addressed. The paper presents the attention pattern difference as an intrinsic property of single-transformer architectures that explains their advantages, without testing alternative explanations or establishing causality. The controlled token-count experiment in Appendix Figure 6 is a step in the right direction but addresses only one confound (token count variation across resolution settings), not the fundamental question of whether attention allocation is cause or correlation. The paper does not discuss these alternative interpretations or propose experiments to distinguish between them.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the multimodal architecture conversation from a component-engineering paradigm to a scaling-strategy paradigm. The dominant approach in the field — select the best pretrained vision encoder, choose an alignment mechanism, fine-tune with careful freezing schedules — implicitly assumes that modularity is the right architectural prior and that the research problem is optimizing within that prior. Sail challenges this assumption not by proposing a clever new component but by running a controlled scaling experiment (Figure 1A) that reveals the modular approach's advantage is contingent on data scale, not architecture quality. At 32M image-text pairs, the modular MLLM wins decisively; at 512M, Sail approaches parity. The implication is that modularity is a data-efficiency hack — it provides pretrained visual knowledge that reduces the sample complexity of multimodal learning, but it does not represent a fundamentally better way to organize computation for multimodal reasoning. If data continues to scale (and there is no sign it won't), the value of that hack diminishes.

This reframing has a concrete effect on the field's incentive structure. Prior to this work, a researcher proposing a new multimodal architecture had to compete against modular systems on benchmark performance at fixed data scales — a rigged game where the modular system's pretrained vision encoder provides a head start that is hard to overcome. The paper's scaling analysis suggests that benchmark comparisons at small data scales systematically favor modular architectures and that the right evaluation is comparative scaling behavior, not point-estimate performance. If the field adopts this perspective, the bar for single-transformer work changes: instead of "beat LLaVA at 1M samples," it becomes "demonstrate a steeper scaling curve than modular baselines, with projections for crossover points." This is a healthier scientific dynamic because it rewards architectures that learn efficiently from data rather than architectures that inherit knowledge from separately trained components.

The paper also introduces attention allocation as a new diagnostic dimension for comparing multimodal architectures. The finding that single-transformer MLLMs allocate 60–80% of attention to image tokens versus 10–30% for modular MLLMs (Figure 4) is not just an interesting observation — it is a measurable property that future architecture proposals should report. Just as NLP papers now routinely report attention entropy, saturated activations, or representation similarity metrics alongside benchmark scores, multimodal architecture papers can report per-modality attention allocation across layers. This diagnostic is cheap to compute (a forward pass on a fixed evaluation set), architecture-agnostic, and provides mechanistic insight that benchmark scores alone cannot. The paper demonstrates that this diagnostic reveals systematic differences between architecture families that benchmark scores obscure — LLaVA-1.5, Qwen2-VL, and LLaVA-OneVision all achieve strong multimodal performance but have radically different attention allocation patterns (Figure 4), suggesting they succeed through different mechanisms and may have different failure modes.

The paper also reconciles a latent tension in the multimodal learning literature. On one side, the vision representation learning community has developed increasingly sophisticated methods for learning visual features from image-text data — contrastive pretraining (CLIP, SigLIP), captioning-based methods (CapPa, SimVLM), and classification-style approaches (CatLIP, SuperClass). On the other side, the MLLM community has treated these as interchangeable vision encoders to be plugged into a modular pipeline, without asking whether the pretraining objective matters for downstream multimodal reasoning. Sail demonstrates that a unified model trained only with a language modeling objective can produce visual representations competitive with purpose-built vision encoders (Tables 4–5: 84.95% ImageNet, 55.3% mIoU ADE20K, 100% on ARO subtasks). This blurs the boundary between "vision backbone" and "multimodal model" — the same network can serve both roles, and the distinction between vision pretraining and multimodal alignment collapses into a single training process. This does not make contrastive learning obsolete (CLIP-style encoders still provide data efficiency benefits at small scales), but it suggests that the long-term trajectory favors unified objectives trained at scale over modality-specific pretraining.

Follow-Up Research This Work Enables

1. Fitting scaling laws for single-transformer multimodal pretraining. The paper provides three data points (32M, 128M, 512M) and three model sizes (0.5B, 3B, 7B) in separate experiments, but does not fit a parametric scaling law (power law or otherwise) relating data volume, model size, and downstream performance. The natural next step is a systematic sweep across both dimensions — train Sail variants at 5–6 model sizes (e.g., 0.1B, 0.3B, 1B, 3B, 7B, 13B) and 5–6 data scales (e.g., 16M, 64M, 128M, 256M, 512M, 1B image-text pairs), then fit a function of the form $L(N, D) = (N_c/N)^\alpha + (D_c/D)^\beta + L_\infty$ familiar from language model scaling laws (Hoffmann et al., 2022). The output of this experiment would be estimates of the scaling exponents $\alpha$ and $\beta$ for multimodal performance, a compute-optimal allocation between model size and data volume for single-transformer MLLMs, and a quantitative prediction of when Sail-style models surpass modular MLLMs across model sizes. The key question is whether single-transformers exhibit the same power-law scaling as language models on multimodal data, or whether the interaction between two modalities changes the functional form. This experiment is computationally expensive (roughly 30 training runs at varying scales) but directly actionable given the training infrastructure described in the paper (128 A100 GPUs, Megatron with sequence packing). A strong follow-up would also vary the ratio of pure text to multimodal data in the training mixture, extending the ablation in Table 7 from binary (with/without text) to continuous (0%, 25%, 50%, 75% text), to determine the optimal mixing ratio as a function of total data volume.

2. Causal manipulation of attention allocation to test the vision-centric hypothesis. The paper's attention analysis (Figures 4, 6) is purely correlational. A direct causal test would train two Sail variants: one with the default mixed attention (bidirectional for images, causal for text), and one where image attention is artificially constrained in deeper layers by adding an attention loss penalty that encourages the model to reduce image attention after layer $K$ (mimicking the modular MLLM pattern where image attention collapses in deeper layers). If the vision-centric attention pattern is causally responsible for Sail's advantages (superior spatial reasoning, better visual grounding), the constrained model should show disproportionate degradation on spatially-intensive benchmarks (MMBench Physical Relation, MME Position) relative to knowledge-intensive benchmarks (MME Celebrity, ScienceQA). If performance is unchanged, the attention pattern is an epiphenomenon — an architectural signature rather than a mechanism. A complementary experiment would train a modular MLLM variant that artificially boosts image attention in deeper layers (e.g., by adding learned gating that controls how much the projected visual features are mixed into each layer's residual stream) and test whether this improves spatial reasoning or visual grounding. If both experiments show that manipulating attention changes task performance in the expected directions, the attention pattern moves from diagnostic to design lever. If neither experiment shows an effect, the field should stop treating attention allocation as a meaningful explanatory variable for multimodal architecture comparison.

3. Testing whether the emergent vision encoder survives instruction tuning — and if not, whether it can be preserved. The paper's vision evaluations (Tables 3–5) use the Stage 1 pretraining checkpoint, not the full instruction-tuned model. A critical follow-up would evaluate visual representation quality at three checkpoints: after Stage 1 (the current baseline), after Stage 2 any-resolution training, and after SFT. The experimental pipeline is straightforward — freeze each checkpoint, run the same ImageNet linear probing (90 epochs, attention-based classifier, 224×224 images), ADE20K segmentation (ViT-Adapter + UperNet, 80K iterations), and ARO retrieval evaluation. If vision quality degrades substantially (e.g., ImageNet top-1 drops from 84.95% to below 80%), the paper's claim that "Sail functions as a powerful vision encoder" requires qualification — it functions as a vision encoder before instruction tuning. If vision quality is preserved, the claim stands. If degradation occurs, the next question is whether it can be mitigated: common strategies include using a smaller learning rate for certain layers during SFT, adding a distillation loss that encourages the tuned model's visual features to remain close to the pretrained checkpoint's features, or interleaving pure vision tasks (classification on ImageNet, segmentation on COCO) during SFT. These experiments are low-cost (evaluating frozen checkpoints on existing benchmarks is cheap; mitigation experiments require retraining SFT with different recipes) and directly address the most significant gap in the paper's central "emergent vision encoder" claim.

4. Systematic comparison of attention patterns across tasks to understand when vision-centric processing helps versus hurts. The paper's attention analysis (Figure 4) averages over 1000 samples from "various datasets" (VQAv2, GQA, TextVQA, DocVQA, MME, SEEDBench, MMBench, and dialog examples). This aggregation obscures potentially important task-specific variation. A fine-grained follow-up would disaggregate the attention pattern by benchmark and by question type within benchmarks. For example: do knowledge-intensive questions (MME Celebrity, ScienceQA fact-recall) show lower image attention than spatial reasoning questions (MME Position, MMBench Physical Relation)? Do correct predictions show different attention patterns than incorrect predictions? Does the progressive focusing phenomenon in Appendix Figure 7 (attention narrows to relevant regions in deeper layers) occur for successful predictions but fail for errors? This analysis is a matter of logging attention weights during evaluation — no new training required — and could reveal whether Sail's vision-centric attention is a task-adaptive strategy (more visual attention for visual questions, less for factual questions) or a fixed architectural property (always high regardless of task). If it is task-adaptive, the diagnostic becomes richer: we can ask when the model chooses to attend to images and whether it makes the right choice. If it is fixed, the vision-centric pattern is a constraint of the architecture that may be suboptimal for tasks that require integrating visual information with extensive world knowledge, potentially explaining the knowledge-intensive task deficits in Table 6.

5. Probing whether ARO-perfect performance reflects genuine compositional reasoning or memorization. Sail's ARO scores (Table 5: 100.0% on Relation, Attribute, and Order with Flickr30K; 99.5% on Attribute with COCO) are at ceiling, making it impossible to distinguish between a model that has learned robust compositional representations and one that has memorized caption patterns from the pretraining data. A strong follow-up would construct adversarial ARO splits — modify the test images or captions in ways that break the correlation between caption patterns and visual content. For example: take ARO images and pair them with synthetically generated captions using combinatorial templates ("the [color] [shape] [relation] the [color] [shape]") that were unlikely to appear in training data, testing whether the model's retrieval performance generalizes to novel caption structures. Alternatively, use the "attribute swap" evaluation: take an image of a red cube on a blue sphere, create a distractor by swapping the attributes ("blue cube on a red sphere"), and test whether the model correctly identifies the original caption despite both captions being plausible and the distractor matching a real visual configuration. If Sail's performance drops substantially on these adversarial splits, the ceiling scores in Table 5 reflect pretraining data memorization rather than genuine compositional understanding. If performance remains high, Sail has learned visual representations that robustly encode attribute binding — a capability that contrastive models demonstrably lack (CLIP-based models score 49–61% on Relation, Table 5) — suggesting that autoregressive text prediction is a fundamentally better objective for learning fine-grained visual composition.

6. Language capability retention as a function of multimodal training — measuring the catastrophic forgetting curve directly. The paper demonstrates that interleaving pure text data prevents catastrophic forgetting (Table 7, ablation #2: removing text data drops ScienceQA by 11 points), but never measures language-only performance directly. A critical follow-up would evaluate Sail on standard language benchmarks (MMLU, HellaSwag, GSM8K, ARC-Challenge, HumanEval) at multiple points during training: at initialization (the Mistral-7B base checkpoint), after Stage 1 pretraining, after Stage 2, and after SFT. This would characterize the forgetting curve — how much language capability is lost at each training stage, and whether the pure text interleaving fully preserves language performance or merely slows the degradation. The experiment would also reveal whether specific language capabilities are more vulnerable. For instance, factual knowledge benchmarks (MMLU) might degrade more than reasoning benchmarks (GSM8K) because factual knowledge stored in the LLM's parameters is overwritten during multimodal training while reasoning procedures are reused for visual reasoning tasks. If specific capabilities are identified as vulnerable, targeted mitigation strategies — e.g., up-weighting specific text domains (science, history) during the pure text interleaving — could be tested. This is a straightforward evaluation (benchmark the model checkpoints on standard LM evaluation harnesses) that addresses an important practical question the paper currently answers only indirectly through multimodal benchmark performance.

Practical Applications and Downstream Use Cases

1. Cost-efficient multimodal model development for organizations training at large scale. The paper's data scaling experiment (Figure 1A) shows that at 512M image-text pairs, Sail approaches the modular MLLM's performance while requiring one model to train and deploy instead of three (vision encoder, projector, LLM). For an organization planning to train on billions of image-text pairs — the trajectory of web-scale multimodal datasets suggests this is the medium-term future — the single-transformer approach eliminates the cost of: (a) pretraining a separate vision encoder (e.g., SigLIP on 40B samples), (b) maintaining and versioning the vision encoder alongside the LLM, and (c) the inference-time overhead of running a ViT on every image. The concrete savings depend on scale, but at the 7B model size with 512M+ training samples, the paper provides evidence that the single-transformer approach is competitive on performance while being strictly simpler operationally. A deployment team could train Sail on their proprietary multimodal data, use the Stage 1 checkpoint as a vision backbone for internal vision tasks (classification, segmentation, retrieval), and use the SFT-tuned checkpoint as a multimodal assistant — all from one training pipeline. The paper does not yet demonstrate that the SFT model retains the vision encoder quality (see Limitation 6), but if further work confirms this, the "one model, two use cases" property is a genuine operational simplification.

2. On-device or edge deployment of multimodal assistants where parameter count is the binding constraint. Modular MLLMs require running both a vision encoder and an LLM at inference time. For a system like LLaVA-1.5 (7B), the ViT-L/14 encoder adds approximately 300M parameters to the 7B LLM, plus the projector parameters — roughly a 4–5% increase in total parameters but a larger increase in latency if the ViT and LLM are run sequentially rather than in parallel. For edge devices where memory and compute are tightly constrained, eliminating the vision encoder means the system only needs to load and run one model. Sail's 7B parameters process both images and text, with no separate vision forward pass. The paper does not benchmark inference latency or memory consumption, but the architectural simplicity directly translates to deployment simplicity: one model file, one inference engine, one set of quantization/optimization configurations. This matters for applications like on-device document understanding, real-time photo captioning for accessibility, or mobile visual search, where the deployment environment constrains the engineering complexity as much as the FLOP budget.

3. Data generation pipelines for vision-language self-improvement loops. A growing paradigm in AI development is using models to generate training data for themselves or their successors (STaR, ReST, self-play). In multimodal settings, this typically requires running a vision encoder + LLM pipeline to generate captions, answers, or instruction data from images. Sail's unified architecture means one forward pass produces both visual features and text outputs, simplifying the data generation pipeline — no need to coordinate vision encoder checkpoints with LLM checkpoints, manage feature caching across them, or debug alignment issues between encoder outputs and LLM inputs. For a team running large-scale image captioning or visual question generation (e.g., recaptioning billions of images for the next generation of multimodal training data, analogous to the Recap-DataComp-1B dataset Sail was trained on), the operational simplification of a single model is non-trivial. Additionally, Sail's strong visual representations (Table 4: matching ViT-22B on segmentation) make it suitable for generating dense annotations (segmentation masks, bounding boxes) alongside text descriptions, potentially enabling unified vision-and-language data generation pipelines that produce pixel-level and semantic-level annotations simultaneously. The companion work Pixel-Sail (Zhang et al., 2025) explores this direction; Sail provides the architectural foundation.

When to Prefer This Method

The paper does not explicitly frame a "decide between Sail and modular MLLM" decision rule, but the controlled scaling experiment in Figure 1A, combined with the task-specific analysis in Table 6, supports the following empirically-grounded heuristics for practitioners choosing an architecture:

Prefer a single-transformer architecture when: You plan to pretrain on large-scale multimodal data (hundreds of millions to billions of image-text pairs) and want a single model that serves both as a multimodal assistant and a vision backbone for downstream vision tasks. The data scaling curve (Figure 1A) shows that the modular advantage erodes as data scales, and the operational simplicity of one model (no vision encoder to train, align, and deploy) compounds over time. This is particularly relevant if your deployment scenario is constrained by engineering complexity (on-device, edge) or if you want to simplify data generation pipelines for future training runs. The paper demonstrates this is viable at 7B scale with 512M+ image-text pairs using the specific recipe (mixed attention, M-RoPE, pure text interleaving, two-stage curriculum). You should budget for exploratory scaling experiments to determine the crossover point for your specific data distribution and base model.
Prefer a modular MLLM when: You are operating in a low-data regime (tens of millions of image-text pairs) or need strong performance on knowledge-intensive tasks (celebrity recognition, fine-grained entity classification, broad world knowledge) where pretrained vision encoders like CLIP or SigLIP encode information that is expensive to relearn from scratch. The modular MLLM in the paper's data scaling experiment maintains a lead at 32M samples (~9.5 normalized points, Figure 1A) and outperforms Sail on knowledge-intensive MME sub-splits even at 512M (MME Celebrity: 100.3 vs. 75.0, Table 6). If your application's performance requirements are dominated by these knowledge tasks and your data budget is fixed below ~100M samples, the modular approach provides a pretrained visual knowledge floor that single transformers cannot yet match. You should also prefer the modular approach if you need to deploy today on a mature, well-optimized stack — LLaVA and Qwen2-VL have extensive ecosystem support, optimized inference kernels, and known failure modes, while Sail-style single transformers are earlier in the development curve.