Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution

ArXiv: 2409.12191

🎯 Pitch

Current vision-language models are crippled by processing all images at a single fixed resolution, but Qwen2-VL’s dynamic resolution approach allocates tokens proportional to image complexity, allowing a 72B model to beat all open-weight competitors on document understanding while consuming fewer visual tokens on average than any single fixed resolution would require.

1. Executive Summary

This paper introduces the Qwen2-VL Series, a family of three open-weight vision-language models scaling from 2B to 72B parameters, evaluated across a comprehensive suite of visual benchmarks including MathVista, DocVQA, MMVet, and Video-MME. The core architectural innovations are Naive Dynamic Resolution (dynamically converting images of arbitrary resolutions into variable numbers of visual tokens, replacing fixed-size encoding) and Multimodal Rotary Position Embedding (M-RoPE) (decomposing rotary embeddings into temporal, height, and width components to fuse positional information across text, images, and video within a single sequence). The 72B model achieves results comparable to GPT-4o and Claude3.5-Sonnet across most multimodal benchmarks—surpassing all open-weight competitors on DocVQA (96.5), InfoVQA (84.5), and OCRBench (877)—while the compute-optimal allocation of dynamic resolution yields the efficiency of consuming fewer tokens on average than any single fixed resolution, establishing that resolution-adaptive visual encoding outperforms predetermined-resolution approaches only when paired with an appropriate compression strategy (2×2 token merging after the ViT) and difficulty-aware resolution thresholds.

2. Context and Motivation

The Core Problem: Fixed-Resolution Visual Encoding Creates an Information Bottleneck

The fundamental problem this paper addresses is deceptively simple: current large vision-language models (LVLMs) process all images at a single, predetermined resolution, and this creates a severe mismatch with how visual information exists in the real world. In practical applications, images arrive at wildly different resolutions and aspect ratios—a document scan might be 2500×3500 pixels, a screenshot 1080×1920, a photograph from a microscope millions of pixels across, and a low-resolution icon only 64×64. Yet standard LVLMs, by design, forcibly resize every input to a fixed resolution (typically 224×224 or 336×336) before the vision encoder ever sees the pixels.

The paper identifies two distinct failure modes of this approach. First, downsampling high-resolution images destroys fine-grained visual information that may be essential for the task. A document with dense text becomes illegible when aggressively downsampled; a chart with small axis labels loses the very information the model needs to answer questions about it; a photograph of a crowded scene loses the ability to distinguish individual objects. Second, upsampling low-resolution images wastes computation and can introduce artifacts that degrade performance. A small icon blown up to 224×224 occupies the same number of visual tokens as a complex photograph, but those tokens encode redundant or interpolated information rather than genuine visual content.

This fixed-resolution constraint is not a minor implementation detail—it is an architectural bottleneck that fundamentally limits what LVLMs can perceive. The paper puts it starkly: such models "fall short of perceiving visual information with the same sensitivity to scale and detail as human vision" (Section 1). Humans dynamically allocate visual attention based on the information density of what they're looking at: we lean in to read fine print, step back to take in a landscape, and scan a page at different levels of granularity depending on our task. Current LVLMs lack any analogous mechanism.

Why This Problem Matters: The Growing Importance of High-Resolution Visual Understanding

The practical significance of this gap has grown rapidly as LVLMs are deployed in real-world applications that demand precise visual perception:

Document understanding and OCR. The paper's experimental results make the stakes concrete. Benchmarks like DocVQA, InfoVQA, and ChartQA specifically test a model's ability to read and reason about textual information embedded in images. These tasks are inherently resolution-dependent: a word rendered in 12-point font on a scanned letter may occupy only a handful of pixels after aggressive downsampling, making it indistinguishable from noise. The paper reports that previous state-of-the-art models achieved only 94.1 on DocVQA (Table 2)—a high number, but one that still leaves room for improvement on documents with particularly dense or small text. In multilingual settings (MTVQA, Table 3), the gap is even larger: previous SoTA reached only 23.2 on MTVQA, suggesting that existing models struggle severely with non-English text recognition, which often involves unfamiliar scripts or character sets that are harder to recognize at low resolution.

Visual reasoning and mathematics. Tasks like MathVista and MathVision require models to read mathematical notation, interpret geometric figures, and reason about quantitative relationships—all from visual inputs. Mathematical expressions are particularly sensitive to resolution because subscripts, superscripts, and special symbols occupy very few pixels. The paper notes that even the best models prior to this work capped out at around 69.0 on MathVista (Table 2), suggesting that resolution-related information loss was a binding constraint on mathematical reasoning performance.

Real-world spatial understanding. RealWorldQA tests whether models can answer practical questions about physical scenes ("Is the chair to the left or right of the table?"). These questions require the model to perceive spatial relationships that may span large portions of an image—information that remains intact at moderate resolutions—but also fine-grained object attributes that require high resolution to distinguish. A model processing everything at 224×224 cannot selectively allocate resolution where it matters most.

Video understanding across timescales. Videos compound the resolution problem with a temporal dimension. A security camera feed and a professionally shot film both contain frames, but at vastly different information densities. Processing every frame at the same resolution treats a blurry, low-information nighttime frame identically to a crisp, text-heavy diagram in a tutorial video. The paper's video benchmarks (MVBench, Video-MME with videos up to one hour) make clear that modern LVLMs need to handle this heterogeneity efficiently—wasting tokens on empty frames while missing details in information-rich ones leads to poor performance on long-form video understanding.

Agent-based interaction with UIs. The paper's agent capabilities (Table 5) add another dimension: an LVLM operating a smartphone needs to read small text on buttons, recognize icons at various sizes, and parse heterogeneous screen layouts. Processing a phone screenshot at the same fixed resolution whether it's a dense settings page or a simple home screen is deeply inefficient, and the ability to dynamically adjust resolution based on content is a prerequisite for practical deployment.

Prior Approaches and Their Shortcomings

The paper situates itself against three main families of prior solutions, each with identifiable weaknesses:

Approach 1: Fixed-Resolution Encoding with Resizing

The most common approach, used by models like LLaVA (Liu et al., 2023b,a), MiniGPT-4 (Zhu et al., 2023), and InstructBLIP (Dai et al., 2023), is to resize all input images to a fixed resolution (typically 224×224 or 336×336) before feeding them to a frozen CLIP-style vision encoder (Radford et al., 2021). The vision encoder then produces a fixed number of visual tokens per image, which are projected into the LLM's embedding space via a simple connector (usually an MLP).

Where this falls short: The paper identifies several specific limitations. First, the fixed token count imposes a uniform information budget regardless of image complexity—a dense document and a simple photograph both get the same representational capacity. Second, the resizing operation itself is lossy: downsampling a high-resolution document scan to 224×224 can render text illegible, while upsampling a small image to 224×224 wastes computation and introduces interpolation artifacts. The paper's ablation study (Table 7) quantifies this: fixed-resolution encoding with 64 tokens (a very aggressive downsample) achieves only 56.47 on RealWorldQA versus 70.07 with dynamic resolution that adapts to content. Third, the use of a frozen vision encoder means the visual representations are optimized for the CLIP pretraining objective (image-text matching), not for the specific reasoning tasks the LVLM needs to perform. This raises the concern the paper articulates about "whether the visual representations produced by such pre-trained models are adequate, particularly for complex reasoning tasks" (Section 1).

Approach 2: Scale-Then-Pad Encoding

A refinement of the fixed-resolution approach, used in some LLaVA variants (Liu et al., 2023a) and other models, is to resize images to fit within a target resolution while preserving aspect ratio, then pad the remaining pixels to reach the fixed size. This avoids the aspect ratio distortion of naive resizing but does not solve the fundamental problem: the effective resolution of the image content is still capped at the target size, and the padding wastes visual tokens on empty space. A very wide or very tall image (a panoramic photo, a long document) still gets the same number of content tokens as a square image, even though it contains more information.

Approach 3: Higher-Resolution Models with More Tokens

Some models, such as Monkey (Li et al., 2023d) and OtterHD (Li et al., 2023a), simply scale up the fixed resolution to capture more detail—using 448×448 or even higher input sizes. Where this falls short: The paper argues this is a brute-force solution that ignores efficiency. Higher resolution everywhere means quadratic growth in visual tokens, which dramatically increases the computational cost of the LLM's self-attention (which scales quadratically with sequence length). Moreover, many images do not need high resolution—a simple photograph or a diagram with large text might be perfectly well-understood at low resolution, and the extra tokens are pure waste. The paper's dynamic resolution approach is explicitly positioned as an alternative that achieves the benefits of high resolution only when needed (Table 7 shows that dynamic resolution achieves 75.89 on InfoVQA while consuming 1924 tokens on average, compared to a fixed 3136 tokens needed to reach 77.27—a substantial efficiency gain).

Approach 4: Interleaved High-Resolution Cropping

Models like SPHINX (Lin et al., 2023) and InternVL2 (Chen et al., 2024c) take multiple crops of the original image at high resolution, encode each crop separately, and then interleave or aggregate the resulting visual tokens. Where this falls short: While this captures fine-grained detail, it creates several problems. First, it loses global context—a crop of a document's corner doesn't know about the document's overall layout. Second, it produces redundant representations of overlapping regions. Third, and most importantly from the paper's perspective, it still uses a predetermined cropping strategy that doesn't adapt to the actual content of the image. A uniform grid of crops treats a blank area of a document the same as a dense paragraph. The paper's naive dynamic resolution is a different approach: rather than cropping and encoding separately, it scales the entire image to an appropriate resolution based on its natural dimensions, encodes it in one pass, and then compresses the resulting tokens. This preserves global context while still capturing detail.

How This Paper Positions Itself: Reframing Resolution as a Dynamic, Learnable Parameter

The paper's positioning is clear and consistent across its narrative: it does not claim to invent the idea of handling multiple resolutions (it cites prior work on this, including Dehghani et al., 2024, and notes that earlier internal Qwen models already had dynamic resolution). Rather, the paper's contribution is a specific, carefully engineered instantiation of dynamic resolution that is shown to work at scale (up to 72B parameters) and to be compatible with video, multilingual OCR, and agent-based use cases within a unified architecture. This is an integration and scaling contribution more than a purely conceptual one.

The paper frames its approach along several axes of improvement over prior work:

From frozen to trained vision encoders. Most prior LVLMs freeze the vision encoder after pretraining. Qwen2-VL trains the ViT during the LVLM training process, and explicitly adopts a three-stage curriculum: first train only the ViT on image-text pairs, then unfreeze everything for comprehensive multimodal training, then lock the ViT and fine-tune only the LLM on instruction data (Section 2.2). This ensures the visual representations are adapted to the specific tasks the model will perform, not just inherited from a generic CLIP-style pretraining. The paper makes a specific architectural choice to support this: replacing the ViT's original absolute position embeddings with 2D-RoPE (Su et al., 2024; Su, 2021), which allows the model to handle images of varying sizes without being tied to a fixed grid of learned positional embeddings.

From separate modalities to a unified position encoding. The paper identifies a deeper architectural problem with how existing models handle videos: "The use of one-dimensional position embeddings in current models significantly limits their ability to model three-dimensional space and temporal dynamics effectively" (Section 1). Most models treat video frames as a sequence of independent images, with the LLM's 1D positional encoding applied to the flattened token sequence. This conflates spatial position (where in the frame) with temporal position (which frame), making it impossible for the model to track an object moving across frames in a principled way. The paper's M-RoPE decomposes the embedding into separate temporal, height, and width components, explicitly modeling the fact that spatial position varies within a frame while temporal position varies across frames (Figure 3). For images, the temporal component is held constant while spatial components vary; for videos, the temporal ID increments per frame. This is a genuinely novel contribution in how it unifies position encoding across modalities.

From independent image and video pipelines to unified training. Prior models typically treat images and videos as separate modalities with separate architectures or training procedures. Qwen2-VL "employs a mixed training regimen incorporating both image and video data" (Section 2.1) and treats an image as a video with two identical frames for architectural consistency. The use of 3D convolutions (depth of two) in the ViT allows the model to process video tubes (spatio-temporal patches) rather than 2D spatial patches, meaning it can handle more video frames without increasing sequence length. This unified paradigm is not entirely new (Flamingo, Alayrac et al., 2022, also unified image and video processing), but the paper's specific combination with dynamic resolution and M-RoPE is novel.

From uniform compute allocation to resolution-adaptive efficiency. The paper's ablation study (Figure 4, Table 7) makes a specific efficiency argument: dynamic resolution achieves top-tier performance across multiple benchmarks while consuming fewer tokens on average than any single fixed resolution that achieves comparable performance. For example, on InfoVQA, dynamic resolution (1924 tokens average) achieves 75.89, while the closest fixed-resolution competitors require 1600 tokens to achieve 74.99 (worse performance) and 3136 tokens to achieve 77.27 (better performance but at 63% higher token cost). This is a compute-optimal allocation argument in the spirit of the training scaling laws literature: there exists a range of resolutions, and the optimal strategy is to choose the right resolution per image rather than a one-size-fits-all compromise. The paper also shows (Figure 4) that there is a task-dependent optimal resolution range—increasing resolution helps on perceptual tasks like InfoVQA and OCRBench but has negligible effect on reasoning-heavy tasks like MMMU, where the bottleneck is the model's reasoning capability rather than visual detail.

The Gap This Paper Fills

To summarize the position: prior to Qwen2-VL, there was no publicly demonstrated model that combined all of the following in a single, open-weight architecture:

1. Truly dynamic resolution handling that processes images at their native aspect ratios with variable token counts, rather than fixed-size crops or single-resolution resizing.
2. A unified position encoding scheme that handles text (1D positions), images (2D positions), and video (3D spatio-temporal positions) within a single rotary position embedding framework.
3. A unified training paradigm for images and video that treats them as points on a spectrum (image = two-frame video) rather than separate modalities with separate architectures.
4. Demonstrated scaling behavior from 2B to 72B parameters, with the largest model matching or exceeding GPT-4o and Claude3.5-Sonnet on most benchmarks.
5. Integration of these capabilities with practical use cases including multilingual OCR, mathematical reasoning, document understanding, and visual agent operation.

The paper does not claim each of these components is novel in isolation. What is novel is the specific integration, the engineering choices that make it work at scale (e.g., the 2×2 token compression after the ViT, the min_pixels/max_pixels thresholds for resolution control, the 3D convolution depth of two for video, the FPS-based frame sampling), and the comprehensive empirical demonstration that this combination achieves state-of-the-art results across a diverse set of benchmarks while maintaining computational efficiency through adaptive resolution allocation.

3. Technical Approach

3.1 Reader orientation (approachable technical breakdown)

The Qwen2-VL series is a family of three vision-language models (2B, 7B/8B, and 72B parameters) that can see, read, reason about, and interact with visual content—images, documents, charts, videos, and user interfaces—and produce text responses in multiple languages. The core problem it solves is that existing vision-language models process all images at a single fixed resolution, losing fine detail in high-resolution images and wasting computation on low-resolution ones, while also treating images and video as fundamentally different types of input requiring separate handling. Qwen2-VL's solution has the shape of a resolution-adaptive visual encoder paired with a unified position encoding scheme that allows text, images, and video to coexist in a single sequence with principled spatial and temporal position tracking, all trained within a three-stage curriculum that progressively builds vision-language alignment, multimodal reasoning, and instruction-following capabilities.

3.2 Big-picture architecture (diagram in words)

The Qwen2-VL architecture consists of five major components connected in a pipeline:

1. Vision Encoder (ViT, ~675M parameters): A Vision Transformer that processes raw images or video frames and produces a grid of visual feature tokens. It uses 2D Rotary Position Embeddings (2D-RoPE) instead of absolute position embeddings, which is the key enabler for handling images of arbitrary resolution. For video, a 3D convolution layer (depth of two) processes spatio-temporal tubes of frames.

2. Token Compression (MLP merger): A simple MLP layer that takes the ViT's output grid and compresses adjacent 2×2 blocks of visual tokens into single tokens. This reduces the token count by approximately 4×, keeping the sequence length manageable for the LLM. For a 224×224 image with patch_size=14, this produces exactly 66 visual tokens.

3. Multimodal Rotary Position Embedding (M-RoPE): A position encoding scheme applied at the LLM's attention layers that decomposes the standard 1D rotary embedding into three orthogonal components—temporal, height, and width—enabling the model to track spatial position within frames and temporal position across frames in a unified way.

4. Large Language Model (Qwen2, varying sizes): The text-generation backbone initialized from Qwen2 (Yang et al., 2024). It receives interleaved sequences of visual tokens (demarcated by <|vision_start|> and <|vision_end|> special tokens) and text tokens, and generates text autoregressively. The LLM comes in three sizes: 1.5B (2B total model), 7.6B (7B/8B total model), and 72B (72B total model).

5. ChatML Interface: A formatting layer that wraps all inputs in a structured dialogue format with <|im_start|> and <|im_end|> tokens, supports multi-turn conversations, visual grounding via bounding box tokens (<|box_start|>, <|box_end|>, <|object_ref_start|>, <|object_ref_end|>), and function-calling for agent tasks.

Information flows as follows: An image enters → its native resolution determines how many ViT patches it gets split into → the ViT with 2D-RoPE encodes all patches into visual features → the MLP compresses 2×2 blocks of features into single tokens → M-RoPE assigns temporal (constant), height, and width position IDs to each visual token → these visual tokens are inserted into the LLM's input sequence between <|vision_start|> and <|vision_end|> → the LLM generates text autoregressively, attending to both visual and text tokens throughout. For video, the same pipeline applies but frames are sampled at 2 FPS, the 3D convolution in the ViT processes spatio-temporal tubes, and M-RoPE increments the temporal ID for each frame.

3.3 Roadmap for the deep dive

The remainder of this section explains the Qwen2-VL technical approach in detail. I will walk through the components in this order, which mirrors the flow of information through the model:

• First, Naive Dynamic Resolution — the mechanism that allows the model to process images at their native resolutions, including the ViT modifications (2D-RoPE), the resolution control parameters (min_pixels, max_pixels), and the token compression strategy. This is the foundation that everything else builds on.

• Second, Multimodal Rotary Position Embedding (M-RoPE) — how position information is encoded for text, images, and video within a unified three-component scheme. This is the key architectural innovation that enables the model to handle mixed-modal sequences.

• Third, the unified image and video processing paradigm — how the ViT with 3D convolutions and a consistent frame sampling strategy treats images as a special case of video, enabling joint training.

• Fourth, the three-stage training procedure — the curriculum that progressively builds the model's capabilities: ViT-only pretraining, comprehensive multimodal pretraining, and instruction fine-tuning. I'll include the specific data compositions, token counts, and training infrastructure details.

• Fifth, the data format and special tokens — the ChatML-based dialogue structure, visual grounding format, and agent action specification that define the model's input/output interface.

• Sixth, the training infrastructure — the parallelism strategy, storage architecture, and software stack used to train models up to 72B parameters.

I'll organize this under Detailed Technical Breakdown (Section 3.4) with dedicated sub-sections for each mechanism. At the end, I'll provide a concise summary of the key design choices and their justifications.

3.4 Detailed, sentence-based technical breakdown

This is primarily an engineering and systems paper whose core contributions are architectural — Naive Dynamic Resolution and M-RoPE — combined with a comprehensive empirical demonstration that these innovations, implemented within a carefully designed training curriculum, enable a single model family to achieve state-of-the-art performance across a diverse set of visual understanding tasks while maintaining computational efficiency through adaptive resolution allocation. The paper does not introduce a new theoretical framework or optimization objective; rather, it shows how specific architectural choices (removing absolute position embeddings, decomposing rotary embeddings into spatial/temporal components, compressing visual tokens after the ViT) resolve concrete limitations of prior vision-language models.

Naive Dynamic Resolution

The most fundamental architectural change in Qwen2-VL is the abandonment of fixed-resolution image encoding in favor of what the paper calls "naive dynamic resolution support" (Section 2.1). The term "naive" refers to the simplicity of the approach: rather than using sophisticated cropping, multi-scale feature extraction, or learned resolution selection, the model simply processes each image at a resolution determined by its natural dimensions, subject only to min_pixels and max_pixels constraints. This simplicity is deliberate — the paper argues that the ViT with 2D-RoPE is naturally capable of handling variable resolutions, and the main engineering challenge is managing the resulting variability in token counts.

The problem with fixed-resolution encoding. In a standard LVLM, every input image is resized to a fixed resolution before being fed to the vision encoder. The ViT splits the image into a grid of patches (typically 14×14 pixels each), producing exactly (H/14) × (W/14) visual tokens per image, where H and W are the fixed input dimensions. For a 224×224 input with patch_size=14, this yields exactly 256 tokens (a 16×16 grid). This fixed token count has two consequences. First, high-resolution images lose information because they are aggressively downsampled — a document scan at 2500×3500 pixels gets compressed to 224×224, destroying fine text. Second, low-resolution images are upsampled, wasting tokens on interpolated pixels that carry no new information.

How Qwen2-VL's dynamic resolution works. The model processes each image at a resolution that respects its native aspect ratio. The procedure is:

1. Determine the target resolution. The image's original dimensions (height h, width w) are scaled such that the total number of pixels falls between min_pixels and max_pixels thresholds. The paper uses min_pixels = 100 × 28 × 28 = 78,400 pixels and max_pixels = 16384 × 28 × 28 = 12,845,056 pixels (Section 3.3.1, Table 7). The factor 28 × 28 is used because the ViT's patch size is 14 and the subsequent 2×2 compression effectively processes 28×28 pixel blocks. The scaling preserves the original aspect ratio — there is no cropping or padding to a square.

2. Encode with ViT + 2D-RoPE. The resized image is split into 14×14 patches and encoded by the ViT, which uses 2D Rotary Position Embeddings (2D-RoPE) instead of learned absolute position embeddings. This is the critical enabler: absolute position embeddings are tied to specific grid positions and cannot generalize to grids of different sizes; 2D-RoPE encodes the relative position of each patch within the image, so the same ViT can handle any grid size. The ViT produces a grid of visual features with dimensions (h'/14) × (w'/14).

3. Compress with MLP. A simple MLP layer takes adjacent 2×2 blocks of ViT output features and projects them into single tokens. This reduces the token count by approximately a factor of 4. The paper states: "to reduce the visual tokens of each image, a simple MLP layer is employed after the ViT to compress adjacent 2 × 2 tokens into a single token" (Section 2.1).

4. Wrap with special tokens. The compressed visual tokens are placed between <|vision_start|> and <|vision_end|> tokens and inserted into the LLM's input sequence.

A concrete example (from Section 2.1). An image with a resolution of 224 × 224 pixels, encoded with a ViT using patch_size=14, would produce a 16 × 16 = 256 token grid from the ViT. After 2×2 compression, this becomes an 8 × 8 = 64 token grid. The <|vision_start|> and <|vision_end|> tokens add 2 more tokens, so the image consumes 66 tokens total before entering the LLM.

Why 2×2 compression? The paper does not explicitly motivate the compression factor, but the rationale is clear from the context: without compression, the token count would scale with (h × w) / 196 (since 14×14 patches yield one token each before compression), which for high-resolution images would produce thousands of tokens. Since the LLM's self-attention cost scales quadratically with sequence length, this would make high-resolution processing prohibitively expensive. The 2×2 compression brings the scaling to approximately (h × w) / 784, a 4× reduction. The choice of 2×2 (rather than, say, 3×3 or 4×4) balances compression against information loss — each compressed token represents a 28×28 pixel region of the original image, which preserves sufficient spatial granularity for most tasks while keeping the sequence length manageable. The paper's Table 7 shows that dynamic resolution with compression produces an average of 1924 tokens per image, compared to benchmark-specific optima ranging from 576 to 3136 for fixed resolution — demonstrating the efficiency gain.

Resolution control parameters: min_pixels and max_pixels. The paper introduces two thresholds that bound the resolution adaptation:

• min_pixels: Images smaller than this threshold are upscaled to reach at least this many pixels. The default value is 100 × 28 × 28. The paper studies the effect of varying min_pixels in the ablation analysis (Figure 4), finding that increasing it improves performance on perceptual tasks like InfoVQA and OCRBench but can degrade performance if set too high on benchmarks with very small images (OCRBench contains many such images, and "excessive enlargement causes these images to deviate from the training data distribution, turning them into out-of-distribution samples" (Section 3.3.1)).

• max_pixels: Images larger than this threshold are downscaled to not exceed this many pixels. The default value is 16384 × 28 × 28. For video, the paper limits "the total number of tokens per video to 16384" (Section 2.1), which controls the computational cost of processing long videos.

The paper explicitly does NOT resize images to a specific height and width, as this "would distort the original aspect ratio" (Section 3.3.1). Preserving aspect ratio is important because many real-world images are far from square — documents are typically portrait, screenshots vary widely, panoramic photos are extremely wide — and distorting the aspect ratio can destroy spatial relationships that are critical for understanding.

Empirical validation of dynamic resolution. The paper's Table 7 provides the key quantitative evidence. For Qwen2-VL-7B, the authors compare fixed-resolution settings (which force a constant number of image tokens — 64, 576, 1600, 3136 — by resizing images accordingly while preserving aspect ratio) against dynamic resolution with min_pixels=100×28×28 and max_pixels=16384×28×28. The dynamic setting achieves: InfoVQA 75.89, RealWorldQA 70.07, OCRBench 866, MMMU 53.44, while consuming an average of 1924 tokens per image. No single fixed resolution achieves optimal performance across all benchmarks — the best fixed resolution for InfoVQA requires 3136 tokens (62.5% more than dynamic), while the best for MMMU requires only 576 tokens. The paper concludes: "the dynamic resolution approach consistently achieves top-tier performance while consuming fewer tokens on average" (Section 3.3.1).

The ViT modification: from absolute to 2D-RoPE position embeddings. The paper replaces the original absolute position embeddings in DFN's ViT (Fang et al., 2023) with 2D Rotary Position Embeddings. Absolute position embeddings are learned vectors associated with each spatial position in a fixed grid — a 16×16 grid has 256 learned position embeddings, and the ViT cannot process images that produce grids of different sizes because those positions have no learned embeddings. 2D-RoPE, by contrast, encodes position through sinusoidal functions applied to the query and key vectors in the attention computation:

For a token at 2D position (h, w) in the image grid, its query and key vectors are rotated by angles that depend on h and w:

$\text{RoPE}_{2D}(q, h, w) = R_h(h) \cdot R_w(w) \cdot q$

where $R_h(h)$ applies a rotation determined by the height position and $R_w(w)$ applies a rotation determined by the width position, and $q$ is the query (or key) vector.

What it computes: the standard 1D RoPE mechanism, applied independently to the height and width dimensions of the 2D grid. RoPE encodes position by multiplying the query and key vectors by rotation matrices whose rotation angles increase with position. For 2D positions, two independent rotations are applied — one for height, one for width — before the query and key vectors are used in the attention dot product.

Why this form: RoPE has the property that the dot product between queries and keys depends only on their relative position, not their absolute positions. For 2D-RoPE, this means the attention between two patches depends on their relative 2D offset (Δh, Δw) — how far apart they are vertically and horizontally — rather than where they are in absolute coordinates. This translation invariance is exactly what's needed for handling variable resolutions, because a patch's semantic role (e.g., "upper-left corner of the image") is preserved by the relative position encoding even when the grid dimensions change. Absolute position embeddings, by contrast, would break entirely when the grid size changes because the model has never seen those position indices during training.

The role of the min_pixels ablation (Figure 4). The paper studies how the choice of min_pixels affects performance by taking small images and upscaling them to meet increasingly high min_pixels thresholds. The results show monotonic improvement on InfoVQA and HallusionBench as min_pixels increases (up to a point), negligible effect on MMMU, and a non-monotonic pattern on OCRBench where very high min_pixels values degrade performance. The paper attributes these differences to task characteristics: "the performance bottleneck in MMMU is more related to the model's reasoning capability rather than image resolution" (Section 3.3.1), while OCRBench contains many very small images that become out-of-distribution when excessively enlarged. This finding motivates the use of a moderate min_pixels threshold rather than always maximizing resolution.

Multimodal Rotary Position Embedding (M-RoPE)

M-RoPE is the second major architectural innovation, extending the concept of rotary position embeddings from 1D (text) and 2D (images) to a unified three-component scheme that handles text, images, and video within a single position encoding framework. The paper describes M-RoPE as "deconstructing the original rotary embedding into three components: temporal, height, and width" (Section 2.1).

The problem with 1D-RoPE for multimodal inputs. Standard language models use 1D-RoPE, which encodes each token's position in a sequence as a scalar index (token 1, token 2, ..., token N). For multimodal inputs where text, image tokens, and video tokens are interleaved in a flat sequence, 1D-RoPE assigns monotonically increasing position IDs across the entire sequence. This conflates completely different types of positional information:

• For text tokens, the position ID correctly represents the token's order in the text sequence.
• For image tokens, the position ID is just a counter that ignores the 2D spatial structure of the image — two tokens from adjacent patches in the same row of the image might get very different position IDs simply because text tokens were inserted between them in the sequence.
• For video tokens, the position ID conflates spatial position (where in the frame) with temporal position (which frame), making it impossible for the model to distinguish between two patches that are spatially adjacent within a frame and two patches that are temporally adjacent across frames.

The paper states this problem explicitly: "The use of one-dimensional position embeddings in current models significantly limits their ability to model three-dimensional space and temporal dynamics effectively" (Section 1).

How M-RoPE works. M-RoPE decomposes the RoPE rotation into three orthogonal components, each with its own position indices:

For a token with temporal position $t$, height position $h$, and width position $w$, the query (or key) vector's dimensions are partitioned into three equal-sized segments, and each segment is rotated by a different component:

$\text{M-RoPE}(q, t, h, w) = [R_t(t) \cdot q^{(0)}, \; R_h(h) \cdot q^{(1)}, \; R_w(w) \cdot q^{(2)}]$

where $q^{(0)}$, $q^{(1)}$, and $q^{(2)}$ are the three segments of the query vector (each of dimension $d/3$ for a $d$-dimensional vector), and $R_t(t)$, $R_h(h)$, and $R_w(w)$ apply 1D RoPE rotations parameterized by the temporal, height, and width position IDs respectively.

What it computes: the standard RoPE mechanism applied with three separate position codes to three parts of the attention head's query/key vectors. The first third of the query vector encodes temporal position, the second third encodes height position, and the final third encodes width position. When attention scores are computed (via dot products), the temporal, height, and width components interact independently — the attention weight between two tokens depends on their relative temporal distance, relative vertical distance, and relative horizontal distance, but these three dimensions do not cross-interact through the rotation operation.

Why this form: the three-way decomposition allows the model to learn different attention patterns for spatial proximity (tokens that are close in the image should attend to each other, regardless of their absolute position) and temporal proximity (tokens from the same video frame should attend to each other differently than tokens from different frames). The separation is important because spatial attention should be translation-invariant (two people next to each other in a photo have the same spatial relationship whether they're in the center or corner of the frame), while temporal attention should distinguish between frames but also preserve the ability to track objects that move across the frame over time. The third-of-dimensions split is the simplest possible decomposition that achieves this separation — more complex schemes (different proportions, learned gating) are possible but were not explored in this paper.

Position ID assignment rules (from Section 2.1 and Figure 3):

• For text tokens: All three components (temporal, height, width) use the same position ID, which is simply the token's position in the text sequence. This makes M-RoPE functionally equivalent to 1D-RoPE for text-only inputs — the three components are identical, so there's no loss of information compared to standard RoPE. The paper states: "For text inputs, these components utilize identical position IDs, making M-RoPE functionally equivalent to 1D-RoPE" (Section 2.1).

• For image tokens: The temporal ID is constant across all tokens within a single image (images have no temporal dimension, so this is a placeholder). The height and width IDs are assigned based on each token's position in the 2D grid after ViT encoding and compression. For example, a token from the 3rd row and 5th column of the compressed grid gets h=3, w=5. The temporal ID is typically set to 0 for the first image, and then incremented for subsequent images based on the maximum position ID used so far.

• For video tokens: The temporal ID increments for each frame — frame 1 gets t=0, frame 2 gets t=1, etc. Within each frame, height and width IDs follow the same pattern as images (the token's row and column in that frame's grid). The temporal ID for frame f would be t = max_position_id_so_far + f, ensuring that frames are ordered in time.

• Cross-modality boundary rule: "In scenarios where the model's input encompasses multiple modalities, position numbering for each modality is initialized by incrementing the maximum position ID of the preceding modality by one" (Section 2.1). This ensures that position IDs are globally unique across the entire sequence, preventing collisions between, say, a text token at position 100 and an image token at position 100.

The length extrapolation benefit. The paper identifies an important consequence of M-RoPE's decomposition: "M-RoPE not only enhances the modeling of positional information but also reduces the value of position IDs for images and videos, enabling the model to extrapolate to longer sequences during inference" (Section 2.1). The key insight is that in 1D-RoPE, a video with many frames would push the position IDs to very large numbers (e.g., frame 100 might get position ID 10,000 if many text tokens preceded it). RoPE's sinusoidal basis functions have known difficulties extrapolating to position IDs far beyond those seen during training. By separating temporal position into its own component, the temporal IDs stay small (just counting frames, not total sequence length), which is within the training distribution even for long videos. Figure 5 validates this: Qwen2-VL-72B, trained with a maximum of 16K tokens per video, maintains robust performance when evaluated at inference lengths up to 80K tokens.

Ablation comparison: M-RoPE vs. 1D-RoPE (Table 8). The paper provides a direct comparison between M-RoPE and 1D-RoPE using Qwen2-1.5B with ViT-L as the backbone, evaluated on a range of image and video benchmarks after pretraining. The results show consistent improvement from M-RoPE, with particularly large gains on video benchmarks: PerceptionTest improves from 46.6 to 47.4, NextQA from 43.9 to 46.0, and STAR from 55.5 to 57.9. Image benchmarks show smaller but still positive effects (e.g., MathVista from 39.2 to 43.4, MMBench from 58.6 to 60.6). The pattern makes intuitive sense: video tasks benefit most from the temporal component of M-RoPE, which explicitly models frame-to-frame dynamics that 1D-RoPE can only capture implicitly through large position ID differences. Image tasks benefit from the spatial decomposition (height and width), which provides a more structured position signal than flat 1D ordering.

Why not use learned position embeddings for 2D? The paper could have used learned 2D position embeddings (separate learned vectors for each (h, w) position in a fixed grid), but this would defeat the purpose of dynamic resolution — the grid dimensions vary per image, and learned embeddings are tied to specific grid sizes. M-RoPE's sinusoidal basis generalizes to any position indices, making it the natural choice for a variable-resolution architecture. This is the same reasoning that motivated the ViT's switch to 2D-RoPE, now extended to the LLM's attention mechanism.

Unified Image and Video Understanding

The paper adopts a unified paradigm for processing images and video: "Qwen2-VL employs a mixed training regimen incorporating both image and video data, ensuring proficiency in image understanding and video comprehension" (Section 2.1). This is enabled by three design choices:

3D convolutions in the ViT for video. Standard ViTs process 2D spatial patches. For video, Qwen2-VL "integrated 3D convolutions (Carreira and Zisserman, 2017) with a depth of two to process video inputs, allowing the model to handle 3D tubes instead of 2D patches, thus enabling it to process more video frames without increasing the sequence length" (Section 2.1; citing Arnab et al., 2021). The 3D convolution with depth two means that the ViT processes spatio-temporal "tubes" that span 2 consecutive frames in time and a 14×14 spatial region. This compresses the temporal dimension at the patch level: rather than producing separate visual tokens for each frame, the model produces tokens that already integrate information across pairs of frames. This reduces the per-video token count by approximately 2× compared to processing frames independently, which is essential for handling videos that may contain thousands of frames.

Image = two-frame video. The paper states: "For consistency, each image is treated as two identical frames" (Section 2.1). This is a pragmatic design choice that avoids creating a separate code path for images. The 3D convolution with depth of two expects input "tubes" spanning two frames. For images, duplicating the frame satisfies this requirement without changing the architecture, and the identical frames mean the temporal convolution learns a no-op for image inputs (the two frames are the same, so there's no temporal signal to extract). This unification means the model can be trained on a mixture of image and video data using exactly the same forward pass, simplifying the training pipeline.

Frame sampling and token budget for video. The paper details a specific video processing protocol:

"To preserve video information as completely as possible, we sampled each video at two frames per second. Additionally, we integrated 3D convolutions... To balance the computational demands of long video processing with overall training efficiency, we dynamically adjust the resolution of each video frame, limiting the total number of tokens per video to 16384." (Section 2.1)

The 2 FPS sampling rate is a deliberate choice: it provides enough temporal resolution to capture most human actions and events (which unfold over seconds), while keeping the frame count manageable. For a 20-minute video, 2 FPS yields 2400 frames. With 3D convolutions compressing pairs of frames, this becomes 1200 "tubes." Each tube produces tokens whose count depends on the frame resolution — with dynamic resolution capping total tokens at 16384, the model can process this 20-minute video within its context window.

The dynamic resolution adjustment per frame means that within a single video, different frames may be processed at different resolutions depending on the overall token budget. This is important because frames within a video vary in information density: a talking-head segment with a static background needs fewer tokens than a rapidly changing scene with text overlays. The 16384 token cap is approximately the context length used during training (the paper mentions "Qwen2-VL supports a 32K context length" in Section 3.2.7, but caps video tokens at 16K to leave room for text and other inputs).

Training approach for joint image-video understanding. The second pre-training stage introduces "a higher volume of mixed image-text content" along with "visual question answering datasets" and "video dialogues" (Section 2.2). By mixing image and video data, the model learns shared visual representations that transfer across modalities — the ability to recognize objects, read text, and understand spatial relationships is common to both still images and video frames. The video-specific capability (understanding motion, action sequences, temporal causality) is learned from the video data and encoded through the temporal dimension of M-RoPE and the 3D convolutions.

Three-Stage Training Procedure

Qwen2-VL follows a three-stage training curriculum adapted from Qwen-VL (Bai et al., 2023b), designed to progressively build the model's capabilities while managing the stability challenges of training a large multimodal model.

Stage 1: ViT-only pretraining. The first stage focuses exclusively on training the Vision Transformer, with the LLM parameters frozen. The objective is to teach the ViT to produce representations that are useful for the LLM's language understanding. The paper states: "In the first stage, we focus exclusively on training the Vision Transformer (ViT) component, utilizing a vast corpus of image-text pairs to enhance semantic understanding within the Large Language Model (LLM)" (Section 2.2). The ViT is initialized from DFN (Fang et al., 2023), but the original absolute position embeddings are replaced with 2D-RoPE.

Design choice: why train ViT separately first? This is a common strategy in vision-language model training (used by LLaVA, BLIP-2, and others). The ViT was pretrained on image-text matching (CLIP-style contrastive learning), which optimizes for semantic alignment — the ViT should produce similar representations for images and their captions. But this objective does not guarantee that the ViT's representations are in the right format or at the right level of abstraction for the LLM to use. Stage 1 adapts the ViT's representations specifically for the LLM, without the risk of degrading the LLM's language capabilities (since the LLM is frozen). The use of image-text pairs (rather than just images) means the ViT learns through the language modeling objective on the text side, with gradients flowing back through the LLM's frozen layers to the ViT — essentially, the ViT is trained to produce representations that make the LLM's next-token prediction easier.

Data composition for Stage 1: The paper mentions the first stage uses "around 600 billion tokens" of which image-text pairs are the primary data, along with OCR data and image classification data (Section 2.2). "Such foundational training is instrumental in enabling the model to develop a robust understanding of core visual-textual correlations and alignments" (Section 2.2).

Stage 2: Comprehensive multimodal pretraining. The second stage unfreezes all parameters (both ViT and LLM) and trains on a much larger and more diverse dataset. The paper states: "In the second stage, we unfreeze all parameters and train with a wider range of data for more comprehensive learning" (Section 2.2). This stage consumes "an additional 800 billion tokens of image-related data" (Section 2.2), bringing the total pretraining to 1.4 trillion tokens.

Data composition for Stage 2: The data includes "image-text pairs, optical character recognition (OCR) data, interleaved image-text articles, visual question answering datasets, video dialogues, and image knowledge datasets" (Section 2.2). The interleaved image-text articles are particularly important for building the model's ability to reason across multiple images and text passages. The paper also notes that "purely textual data continues to play a crucial role in maintaining and advancing the model's linguistic proficiency" (Section 2.2) — without pure text data, the model might forget its language capabilities as it adapts to the multimodal distribution.

Training data sources: The paper states that data "primarily comprise cleaned web pages, open-source datasets, and synthetic data" with a knowledge cutoff date of June 2023 (Section 2.2). The exact datasets are not enumerated, but the benchmarks used for evaluation (Section 3) give an indication of the domains covered: documents (DocVQA, InfoVQA), charts (ChartQA), natural images (TextVQA, RealWorldQA), scientific diagrams (AI2D), mathematics (MathVista, MathVision), and video (MVBench, PerceptionTest, EgoSchema, Video-MME).

Supervision signal: Throughout both pretraining stages, the training objective is next-token prediction on text tokens only. The paper explicitly states: "During the training process, however, we only provide supervision for the text tokens" (Section 2.2). Visual tokens are not supervised — the model learns to produce useful visual representations solely through the indirect signal of making text prediction easier. This is standard practice in LVLM training: the cross-entropy loss is computed only on text tokens, and gradients flow through the visual encoder only via the attention mechanism's influence on text token predictions.

Stage 3: Instruction fine-tuning. The final stage locks the ViT parameters and fine-tunes only the LLM on instruction-following data. The paper states: "In the final stage, we lock the ViT parameters and perform exclusive fine-tuning of the LLM using instructional datasets" (Section 2.2). This is a deliberate asymmetry: the ViT's visual representations are considered "done" after Stage 2, and Stage 3 focuses entirely on teaching the LLM to follow instructions, engage in dialogue, use tools, and perform agent tasks — all of which are primarily language capabilities that build on the now-stable visual representations.

Instruction data format: The instruction fine-tuning data is formatted in ChatML (Openai, 2024), which uses <|im_start|> and <|im_end|> tokens to mark the boundaries of each message in a conversation. The paper provides an example (Section 2.2.1) showing a multi-turn dialogue with both images and text. The training loss is computed only on the assistant's responses (the sections marked in blue in the paper's example), not on user messages.

Instruction data composition (from Section 2.2.1): "This dataset encompasses not only pure text-based dialogue data but also multimodal conversational data. The multimodal components include image question-answering, document parsing, multi-image comparison, video comprehension, video stream dialogue, and agent-based interactions." The inclusion of agent-based interactions (UI operations, tool use, function calling) is notable — these require the model to generate structured action commands rather than natural language, which requires specific training data.

Why lock the ViT in Stage 3? The paper does not explicitly justify this choice, but the rationale is standard in the field: the ViT has already been trained on 1.4 trillion tokens of diverse visual data by the end of Stage 2, and further training on the much smaller instruction dataset (which is primarily text-heavy) risks overfitting the ViT to the instruction distribution or degrading its general visual recognition capabilities. Locking the ViT also reduces memory usage during fine-tuning, which is practical for training large models.

Data Format and Special Tokens

Qwen2-VL uses a carefully designed token vocabulary to demarcate different types of content within the LLM's input sequence. These special tokens serve as control codes that tell the model where visual content begins and ends, where bounding boxes are specified, and where the model should generate function calls versus natural language.

Vision demarcation tokens. The tokens <|vision_start|> and <|vision_end|> are placed at the beginning and end of the visual token sequence for each image or video. The paper states: "Tokens <|vision_start|> and <|vision_end|> are inserted at the start and end of the image feature sequence to demarcate the image content" (Section 2.2.1). This is important for the LLM's attention mechanism — without explicit boundaries, the model would have to infer where the visual tokens end and text tokens begin, which is harder to learn.

ChatML dialogue format. The instruction fine-tuning data uses OpenAI's ChatML format, where each message in a conversation is wrapped with <|im_start|> and <|im_end|> tokens, followed by a role specifier (user, assistant, or system). The paper provides an example (Section 2.2.1):

<|im_start|>user
<|vision_start|>Picture1.jpg<|vision_end|><|vision_start|>Picture2.jpg<|vision_end|>What do the
two pictures have in common?<|im_end|>
<|im_start|>assistant
Both pictures are of SpongeBob SquarePants. <|im_end|>

The user turn contains interleaved vision tokens and text; the assistant turn contains only text. The training loss is computed only on the assistant's tokens — the model learns to generate text conditioned on both the images and the user's question.

Visual grounding tokens. The paper introduces a bounding box format for visual grounding tasks (referring expression comprehension, object detection). Bounding box coordinates are "normalized within [0, 1000) and represented as (Xtop left, Ytop left), (Xbottom right, Ybottom right)" (Section 2.2.1). The tokens <|box_start|> and <|box_end|> demarcate the coordinate string, and <|object_ref_start|> and <|object_ref_end|> demarcate the textual description of the referred object. The paper's example:

<|vision_start|>Picture1.jpg<|vision_end|>
<|object_ref_start|>the eyes on a giraffe<|object_ref_end|><|box_start|>(176,106),(232,160)<|box_end|>

The normalization to [0, 1000) ensures coordinates are integers in a consistent range regardless of image size. The choice of 1000 (rather than, say, 100 or 10000) balances coordinate precision (three-digit numbers provide ~0.1% precision relative to image dimensions, sufficient for most grounding tasks) against token length (coordinates stay short).

Agent and function calling format. For agent tasks, the paper defines a specific format based on the Qwen-Agent framework (Qwen Team, 2024). The format includes a system message that lists all available actions with their parameters, followed by a multi-turn interaction loop where the model generates *FUNCTION*: and *ARGS*: to execute actions, receives *RESULT*: containing new observations, and produces *RETURN*: with natural language reasoning. The paper provides an example (Section 2.2.1) showing a UI operation task where the model interacts with a phone screen across multiple steps. The *FUNCTION*, *ARGS*, *RESULT*, and *RETURN* keywords act as structured control codes that the model learns to generate and interpret, enabling it to participate in the action-observation loop.

Training Infrastructure

The paper provides a detailed account of the infrastructure used to train Qwen2-VL, which is notable given the scale (up to 72B parameter models trained on 1.4 trillion tokens). The infrastructure description covers storage, parallelism, and software.

Hardware platform. The models were trained on Alibaba Cloud's PAI-Lingjun Intelligent Computing Service (Alibaba-Cloud, 2024c), which provides scalable computing with "auto resuming and straggler detection" (Section 2.3). The exact GPU configuration is not specified.

Storage architecture. The paper describes a decoupled storage system:

• Text data: Stored on CPFS (Cloud Parallel File Storage) with mmap for efficient access. mmap maps files directly into virtual memory, allowing the training process to access data without explicit read system calls, which reduces I/O overhead for the high-throughput text data pipeline.

• Vision data: Stored on OSS (Object Storage Service) for persistent storage, accessed concurrently via OSS's Python client during training. "We accessed vision data through OSS's python-client concurrently and tuned the concurrency and retrying parameters to avoid reaching the QPS (queries per second) limit" (Section 2.3). This separation is pragmatic: text data is small per sample (kilobytes) and accessed frequently during training, so high-performance local storage with memory mapping makes sense; vision data (images and especially video) is large per sample (megabytes to gigabytes) and accessed less frequently, so object storage with concurrent retrieval is more cost-effective.

• Video decoding bottleneck: "We also found that video data decoding is a main bottleneck, especially for long videos. After several attempts with open-source (FFmpeg) and in-house software failed, we opted for a caching decoding technique" (Section 2.3). The exact caching mechanism is not detailed, but presumably involves decoding videos once and storing the decoded frames temporarily, avoiding repeated decoding during multiple epochs or when the same video is sampled multiple times.

• Checkpointing: "Checkpointing saves each GPU's optimizer and model states on CPFS" (Section 2.3).

Parallelism strategy. The paper uses "3D parallelism which combines data parallelism (DP), tensor parallelism (TP) and pipeline parallelism (PP)" (Section 2.3), supplemented with DeepSpeed ZeRO-1 (Rajbhandari et al., 2020) for optimizer state sharding and sequence parallelism (SP) (Korthikanti et al., 2023) for memory efficiency.

Specific parallelism details:

• TP sharding policy: "When enabling TP training, we always shard the vision encoder and large language models together but not the vision merger due to its relatively few parameters" (Section 2.3). The vision merger (the MLP that compresses 2×2 token blocks) has few parameters, so sharding it across TP ranks would add communication overhead without meaningful memory savings.

• TP non-determinism issue: "We found the TP training would result in different model shared-weights due to the convolution operator's non-deterministic behavior. We resolved this issue by performing offline reduction of the shared weights, thereby avoiding an additional all-reduce communication step" (Section 2.3). This is a practical engineering detail: the 3D convolutions in the ViT use an operation that is not bitwise-deterministic across different parallel executions, which causes the model weights for shared layers to drift apart across TP ranks. The offline reduction step synchronizes these weights periodically without adding communication during the forward/backward pass, preserving training throughput.

• PP configuration for 72B: "We leverage 1F1B PP (Narayanan et al., 2021) for Qwen2-VL 72B training. We combine the vision encoder, vision adapter and several LLM's decoder layers into one stage, and evenly split the remaining decoder layers" (Section 2.3). 1F1B (one-forward-one-backward) pipeline parallelism schedules the forward and backward passes to keep all GPUs busy while minimizing the memory footprint of activations. Grouping the vision components (ViT + merger) with the first few LLM layers into a single pipeline stage recognizes that the vision components have different computational characteristics than the LLM — they process 2D/3D data rather than 1D sequences, and their activation sizes depend on image resolution rather than sequence length.

• Dynamic sequence length handling in PP: "Note that the vision and text sequence lengths are dynamic for each data point. We broadcast the dynamic sequence lengths before initiating the 1F1B process and access the shape information using batch indices" (Section 2.3). This is necessary because the dynamic resolution mechanism produces variable numbers of visual tokens per image, and different images in the same batch may have different token counts. Broadcasting the lengths upfront allows each pipeline stage to allocate the correct amount of memory.

• Interleaved 1F1B: "We also implemented an interleaved 1F1B PP (Narayanan et al., 2021) but found it is slower than the standard 1F1B setting" (Section 2.3). Interleaved 1F1B schedules micro-batches within each pipeline stage to reduce idle time (the "bubble"), but the paper found that for Qwen2-VL's specific workload, the standard 1F1B was faster — likely because the dynamic sequence lengths make the micro-batch scheduling overhead outweigh the bubble reduction benefit.

Software stack:

• PyTorch 2.1.2 with CUDA 11.8 (Section 2.3).
• Flash-Attention (Dao et al., 2022; Dao, 2024; Shah et al., 2024) for efficient attention computation in both the ViT and LLM. Flash-Attention reduces the memory footprint of attention from O(N²) to O(N) by computing attention in blocks and not storing the full attention matrix, which is essential for handling long sequences with many visual tokens.
• Fused operators from NVIDIA Apex (Nvidia, 2024a): LayerNorm (Ba et al., 2016), RMSNorm (Zhang and Sennrich, 2019), and Adam (Loshchilov and Hutter, 2019). Fused operators combine multiple operations into a single CUDA kernel, reducing the number of GPU memory accesses and improving throughput.
• "We leverage the overlap of communication and computation during matrix multiplication in our training process" (Section 2.3). This refers to the standard technique of overlapping gradient all-reduce (communication) with the next layer's backward pass (computation), hiding communication latency behind useful computation.

Summary of Design Choices and Their Justifications

The Qwen2-VL technical approach can be understood as a series of carefully motivated engineering decisions, each addressing a specific limitation of prior vision-language models:

• Removing absolute position embeddings from the ViT and replacing them with 2D-RoPE: Absolute position embeddings are tied to a fixed grid size and cannot generalize to variable-resolution inputs. 2D-RoPE is translation-invariant and naturally handles arbitrary grid dimensions, which is necessary for dynamic resolution.

• Using 2×2 token compression after the ViT: Without compression, high-resolution images would produce thousands of visual tokens, making the LLM's self-attention cost prohibitive (quadratic in sequence length). The 2×2 compression achieves 4× token reduction while preserving spatial granularity at the 28×28 pixel level, which is sufficient for most visual reasoning tasks.

• Decomposing M-RoPE into temporal, height, and width components with equal-dimensional splits: This is the simplest decomposition that separates spatial and temporal position information, allowing the model to learn distinct attention patterns for within-frame spatial relationships and across-frame temporal dynamics. The equal split ensures no component is starved of representational capacity.

• Treating images as two-frame videos with 3D convolutions: This architectural unification avoids maintaining separate code paths for images and video, simplifies training, and ensures the model's visual representations are shared across modalities. The depth-2 3D convolution processes pairs of frames (or identical frames for images), halving the token count compared to independent frame encoding.

• The three-stage training curriculum (ViT-only → unfrozen all → LLM-only): Stage 1 adapts the ViT's representations for the specific LLM without risking degradation of language capabilities. Stage 2 jointly trains all components on diverse data to build multimodal reasoning. Stage 3 locks the ViT to prevent overfitting on the smaller instruction dataset and focuses on dialogue, instruction-following, and agent capabilities.

• Sampling video at 2 FPS: This balances temporal resolution (capturing actions that unfold over seconds) against computational cost (limiting the number of frames for long videos). For a 20-minute video, 2 FPS yields 2400 frames, which with 3D convolution compression becomes 1200 tubes — manageable within the 16K token video budget.

• Normalizing bounding box coordinates to [0, 1000): This provides integer coordinates at ~0.1% precision relative to image dimensions, which is sufficient for grounding tasks while keeping the coordinate strings short and consistent across images of different sizes.

• Using ChatML with supervised tokens only on assistant responses: This is standard practice for instruction tuning, ensuring the model learns to generate responses (not mimic user queries) and can participate in multi-turn conversations with clear role boundaries.

• Decoupling text and vision storage (CPFS vs. OSS): Text data is small and accessed frequently, so high-performance local storage with memory mapping is appropriate. Vision data (especially video) is large, so cost-effective object storage with concurrent access is more practical.

• Using standard 1F1B pipeline parallelism rather than interleaved 1F1B: The dynamic sequence lengths in Qwen2-VL (due to dynamic resolution) make interleaved scheduling overhead outweigh its benefits for reducing pipeline bubbles, so the simpler standard 1F1B is faster in practice.

4. Key Insights and Innovations

Innovation 1: Reframing Resolution as a Per-Instance Resource Allocation Problem, Not a Fixed Architectural Parameter

The dominant assumption in vision-language model design, from LLaVA through InternVL2, has been that image resolution is a static architectural hyperparameter — choose 224×224, 336×336, or 448×448, and encode all images accordingly. The field's instinct, when higher resolution was needed, was to scale up uniformly (Monkey, OtterHD) or to take multiple fixed-size crops (SPHINX, InternVL2). Both approaches share an implicit assumption: that resolution is a property of the model, not of the input.

Qwen2-VL's distinctive conceptual move is to reclassify resolution as a per-instance resource allocation decision, analogous to how inference-time compute can be allocated adaptively across prompts of varying difficulty. The paper doesn't just say "higher resolution helps" — that's obvious. The insight is that the optimal resolution varies by task and by individual image, and that a model should treat resolution as a budget to be allocated efficiently rather than a constant to be maximized.

What makes this a reframing rather than an incremental improvement: Prior dynamic-resolution approaches (e.g., Dehghani et al., 2024's NaViT, cited by the paper) already showed that ViTs can handle variable resolutions. But those works framed the benefit as architectural flexibility — the model can handle different sizes, which is useful for training efficiency and mixed-dataset training. Qwen2-VL reframes the benefit as perceptual efficiency: the model should handle different sizes because different inputs carry different amounts of visual information, and allocating resolution proportional to information density is more efficient than uniform allocation.

The paper's Table 7 provides the empirical anchor for this reframing: no single fixed resolution achieves optimal performance across all benchmarks. The best resolution for InfoVQA (3136 tokens) is 5.4× larger than what works best for MMMU (576 tokens). Dynamic resolution achieves top-tier performance across all benchmarks while consuming fewer tokens on average than any fixed resolution that matches its worst-case performance. This is not just a "dynamic is better" result — it's evidence that the optimal resolution is genuinely task-dependent, not just "higher is better."

The Figure 4 ablation deepens the insight: increasing min_pixels helps monotonically on InfoVQA and HallusionBench (perceptual tasks), has negligible effect on MMMU (reasoning bottleneck), and actually hurts on OCRBench at very high thresholds (distribution shift for very small images). This shows that resolution allocation is not just about maximizing information capture — it's about matching resolution to both the task's demands and the model's training distribution. Over-resolving can be actively harmful, a finding that would not emerge from a "higher is always better" framing.

Significance beyond performance: This reframing connects Qwen2-VL to the broader scaling laws literature. Just as Chinchilla (Hoffmann et al., 2022) showed that pretraining compute should be allocated between model size and data quantity according to a principled ratio, Qwen2-VL's dynamic resolution suggests that inference-time visual compute should be allocated according to image information density and task requirements. The min_pixels and max_pixels parameters are not just engineering knobs — they are the beginnings of a resolution allocation policy that could, in principle, be learned rather than hand-tuned.

Innovation 2: Decomposing Position Encoding into Orthogonal Modality-Specific Components

Prior to M-RoPE, the default approach for encoding position in multimodal sequences was to flatten everything into a 1D sequence and apply standard 1D-RoPE (or learned position embeddings). Text tokens, image patches, and video frames all shared a single scalar position index. This works in the sense that the model can learn to attend appropriately, but it conflates fundamentally different types of positional relationships. The model must learn that tokens 100-116 are spatially adjacent (they form a row of image patches) while tokens 117-120 are temporally separated (they span video frames), all from a single scalar difference. There's no architectural signal distinguishing "nearby in space" from "nearby in sequence order."

The conceptual move behind M-RoPE is to recognize that position is not a one-dimensional concept in multimodal inputs, and that different dimensions of position (spatial height, spatial width, temporal order) should be encoded through orthogonal channels so the attention mechanism can learn distinct patterns for each. The paper decomposes the rotary embedding's rotation into three components applied to three segments of the query/key vectors:

• Temporal component: encodes frame ordering in video, enabling the model to learn temporal attention patterns (e.g., attend to adjacent frames) independently of spatial proximity.
• Height component: encodes vertical position within a frame, enabling the model to learn that attention should be vertically local (objects near each other vertically should attend to each other).
• Width component: encodes horizontal position, analogously to height but independently — allowing the model to learn different attention ranges for rows versus columns if the task demands it.

What makes this distinctive: This is not simply "2D-RoPE plus time." The key insight is the orthogonal decomposition — giving each position dimension its own independent channel in the query/key vectors. If the three components were simply summed or concatenated, the model would see a combined position signal and would have to disentangle spatial from temporal proximity through learning alone. By applying separate rotations to separate vector segments, the architecture guarantees that the attention dot product separates into independent temporal, height, and width contributions. The model learns attention weights as a function of (Δt, Δh, Δw) rather than some scrambled combination.

The equal-dimensional split (each component gets d/3 dimensions) is the simplest possible decomposition. More complex schemes — learned importance weighting per component, different proportions for different layers — are natural extensions that the paper leaves unexplored. The simplicity is a strength: it establishes that the orthogonal decomposition itself is the valuable concept, not the specific allocation.

Empirical distinction from 1D-RoPE: The Table 8 ablation shows that M-RoPE helps on image benchmarks (MathVista: 39.2 → 43.4, a meaningful gain) but helps substantially more on video benchmarks (PerceptionTest: 46.6 → 47.4, NextQA: 43.9 → 46.0, STAR: 55.5 → 57.9). This pattern is the smoking gun for the orthogonality claim. Image tasks benefit from separated height and width components (better spatial attention), but the gain is modest because 1D-RoPE can approximately capture 2D position through learning. Video tasks benefit more because 1D-RoPE fundamentally conflates spatial and temporal position — the model must learn that a large position delta sometimes means "same place, later time" and sometimes means "different place, same time," and this ambiguity limits performance. M-RoPE resolves the ambiguity architecturally.

The length extrapolation consequence (Figure 5) is a bonus, not the main point. M-RoPE keeps temporal position IDs small (counting frames, not total sequence position), which helps with long videos, but this is a secondary benefit. The primary contribution is the conceptual architecture of orthogonal position dimensions, which enables principled modeling of 3D spatio-temporal attention in a way that 1D encodings fundamentally cannot.

Innovation 3: Unifying Image and Video Processing Through Architectural Consistency Rather Than Separate Modality-Specific Pipelines

Most prior vision-language models treat images and video as distinct modalities requiring separate handling. Flamingo (Alayrac et al., 2022) uses a perceiver resampler for images and a separate video perceiver. Video-LLaMA and similar models use separate video encoders. Even when models can handle both, they typically do so through different forward-pass code paths and different training procedures. The implicit assumption is that the temporal dimension of video is qualitatively different from the spatial dimensions of images, requiring specialized architecture.

Qwen2-VL's move is to recognize that images and video are points on a continuum: a video is just a sequence of images, and an image is a video with one frame. The paper implements this recognition through three architectural decisions that together eliminate modality-specific handling:

1. Images treated as two-frame videos: For architectural consistency with the depth-2 3D convolutions, images are duplicated into identical frame pairs. This means the identical forward pass works for both modalities — no branching, no modality detection, no separate encoders.

2. 3D convolutions with depth two: Rather than a separate video encoder, the ViT's patch embedding is extended to 3D, processing spatio-temporal tubes of 2 consecutive frames. This compresses the temporal dimension at the patch level, making video token counts manageable without changing the architecture.

3. Mixed training with both modalities: Images and videos are interleaved in training data, forcing the model to learn shared visual representations that transfer between modalities.

What makes this distinctive at the idea level: The conceptual contribution is not the 3D convolution (which dates to Carreira and Zisserman, 2017) or joint training (Flamingo did this). It's the recognition that modality unification is an architectural design principle, not a training trick. The paper doesn't just train on images and videos together — it eliminates the distinction between them at the architecture level. An image is not an image; it's a video with identical frames. A video is not a video; it's a sequence of frames processed by the same ViT that handles images. This is a deeper unification than Flamingo's separate perceivers or Video-LLaMA's separate encoders.

The depth-2 choice specifically is an engineering insight: it compresses the temporal dimension by 2×, which for a 20-minute video at 2 FPS (2400 frames) reduces the effective frame count to 1200 — manageable within the 16K token video budget without sacrificing spatial resolution per frame. Deeper 3D convolutions (depth 4 or 8) would compress more but risk temporal smearing — mixing information across frames that should be kept distinct for fine-grained action recognition. Depth 2 is the minimal temporal integration that achieves the computational benefit.

Practical significance: This unification simplifies training infrastructure (one forward pass, one architecture, one optimizer) and ensures that visual representations learned from images transfer to video and vice versa. The strong video results (MVBench 73.6, EgoSchema 77.9, PerceptionTest 68.0 for the 72B model, Table 4) are evidence that the unified approach works: the model's visual understanding, built primarily on image data (which dominates the 1.4T training tokens), transfers effectively to video without architectural modification.

Innovation 4: The Three-Stage Training Curriculum as a Stability-Management Strategy for Large Multimodal Models

The three-stage training procedure (ViT-only → unfreeze all → lock ViT, fine-tune LLM) appears at first glance to be an engineering detail — a schedule for which parameters to update when. But there's a deeper conceptual insight: the training curriculum is a mechanism for managing the conflicting optimization pressures in multimodal models, specifically the tension between adapting visual representations for language tasks (which requires ViT updates) and preserving language capabilities (which benefits from a stable LLM).

Prior work explored variants of this. LLaVA popularized two-stage training (feature alignment → instruction tuning), but with a frozen ViT throughout. BLIP-2 used a Q-Former to bridge frozen vision and language models. Qwen-VL (Bai et al., 2023b) introduced the three-stage curriculum that Qwen2-VL adapts. The conceptual contribution here is not the stages themselves but the progressive shifting of the stability-adaptability tradeoff:

• Stage 1 (ViT only, LLM frozen): The LLM is treated as a fixed critic — it provides a stable training signal for the ViT through next-token prediction loss, but its parameters don't change. This is maximally conservative for language preservation; only the visual representations adapt. The ViT learns to produce features that make the LLM's job easier, but the LLM's language modeling capability is untouched.

• Stage 2 (all parameters unfrozen): With the ViT now producing LLM-compatible representations, both components adapt jointly. The risk of language degradation is mitigated because the ViT's representations are already in the right format, so the LLM doesn't need to radically reorganize its internal representations to accommodate visual input. The paper's inclusion of pure text data during this stage ("purely textual data continues to play a crucial role in maintaining and advancing the model's linguistic proficiency," Section 2.2) is an explicit mechanism to prevent catastrophic forgetting of language skills.

• Stage 3 (ViT frozen, LLM fine-tuned): The ViT is treated as done — its visual representations are fixed. The LLM adapts to the instruction-following distribution, learning dialogue patterns, tool use, and agent behaviors. Freezing the ViT prevents it from overfitting to the instruction dataset's visual distribution (which is narrower than the pretraining distribution) and reduces memory usage.

What distinguishes this from standard curriculum learning: The innovation is the asymmetric freezing schedule — the ViT and LLM are never jointly trained on instruction data. This recognizes that the two components have different learning dynamics and different risks. The ViT, with ~675M parameters, could easily overfit to the instruction dataset's visual biases if trained on instruction data; the LLM, with up to 72B parameters, has enough capacity to learn instruction-following without degrading its visual understanding (since the ViT is frozen). The curriculum is not just about what to train when; it's about which component to trust with which type of adaptation.

The 1.4T token scale makes this a scaling insight, not just a training recipe. At smaller scales, joint end-to-end training might work fine — the risk of language degradation is lower when the model has less to lose. At 72B parameters, the LLM has substantial language capabilities that would be expensive to recover if lost. The three-stage curriculum is a risk-management strategy that becomes increasingly important as model scale grows, which is why it matters for a paper that demonstrates scaling from 2B to 72B.

Empirical validation of the curriculum structure is implicit — the paper doesn't ablate the three-stage schedule against alternatives (e.g., training everything jointly from the start). This is a limitation: the conceptual claim that the stages manage stability-adaptability tradeoffs is reasonable but not directly tested. The strong results across benchmarks (Table 2) are consistent with the curriculum working, but don't isolate its contribution from other factors. Future work that ablates training schedules at scale would strengthen the conceptual argument.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary evaluation spans a broad collection of established multimodal benchmarks. For general visual question answering, the paper uses RealWorldQA (X.AI, 2024a), MMStar (Chen et al., 2024a), MMVet (Yu et al., 2024), MMT-Bench (Ying et al., 2024), MMBench and MMBench-V1.1 (Liu et al., 2023d), and MME (Fu et al., 2023). For document understanding, the paper evaluates on DocVQA (Mathew et al., 2021), InfoVQA (Mathew et al., 2021), ChartQA (Masry et al., 2022), TextVQA (Singh et al., 2019), OCRBench (Liu et al., 2023e), and the multilingual text-centric benchmark MTVQA (Tang et al., 2024). Mathematical reasoning is tested on MathVista (Lu et al., 2024a) and MathVision (Wang et al., 2024). Video understanding is assessed on MVBench (Li et al., 2024), PerceptionTest (Patraucean et al., 2024), EgoSchema (Mangalam et al., 2023), and Video-MME (Fu et al., 2024). College-level multimodal understanding is measured on MMMU (Yue et al., 2023) and MMMU-Pro (Yue et al., 2024). Visual grounding is evaluated on RefCOCO, RefCOCO+, and RefCOCOg (Kazemzadeh et al., 2014; Mao et al., 2016). Agent capabilities use in-house function-calling benchmarks, the AITZ benchmark for UI operations (Zhang et al., 2024b), the RL4VLM card game suite (Zhai et al., 2024), ALFRED for robotic control (Shridhar et al., 2020a), and R2R (Anderson et al., 2018) plus REVERIE (Qi et al., 2020) for vision-language navigation. For multilingual OCR, the paper constructs an internal benchmark covering Korean, Japanese, French, German, Italian, Russian, Vietnamese, and Arabic.

• Base model(s). The Qwen2-VL series comprises three model sizes: Qwen2-VL-2B (675M ViT + 1.5B LLM), Qwen2-VL-7B/8B (675M ViT + 7.6B LLM), and Qwen2-VL-72B (675M ViT + 72B LLM). The ViT is initialized from DFN (Fang et al., 2023) with absolute position embeddings replaced by 2D-RoPE. The LLM component is initialized from Qwen2 (Yang et al., 2024). The scaling from 2B to 72B enables analysis of how performance scales with model size across diverse visual capabilities.

• Metrics. The paper uses accuracy (percentage of correct answers) or equivalent task-specific metrics across all benchmarks. For DocVQA, InfoVQA, ChartQA, AI2D, MMBench, MMStar, MMMU, and MMMU-Pro, the metric is standard accuracy. For MME, a cumulative score across 14 subtasks is reported (Fu et al., 2023). For OCRBench, a composite score is reported (Liu et al., 2023e). For referring expression comprehension (RefCOCO and variants), accuracy is measured by IoU threshold. For agent tasks, specific metrics include: Type Match (correct function selection rate) and Exact Match (correct argument specification rate) for function calling; Success Rate (SR) and Goal-Condition Success (GC) for ALFRED; and Success Rate for navigation tasks. Video benchmarks use their standard published metrics: MVBench uses accuracy, PerceptionTest uses accuracy, EgoSchema uses accuracy, and Video-MME reports accuracy with and without subtitles.

• Baselines. The paper compares against a comprehensive set of both open-source and proprietary models. Key baselines include: GPT-4o (OpenAI, 2024), GPT-4V (OpenAI, 2023), Claude 3.5 Sonnet (Anthropic, 2024), Gemini 1.5 Pro (Team et al., 2023), Previous SoTA models at the time of evaluation (including InternVL2-76B (Chen et al., 2024c), CogVLM (Wang et al., 2023b), Ferret-v2 (Zhang et al., 2024a), and others as listed in Tables 2, 4, 5, and 6), Specialized models for specific tasks (e.g., G-DINO-L (Liu et al., 2023c) and UNINEXT-H (Yan et al., 2023) for referring expression comprehension; ThinkBot (Lu et al., 2023) for robotic control), and Qwen-VL (Bai et al., 2023b), the prior generation from the same team, providing a direct measure of improvement from the architectural innovations.

• Generation budget / compute accounting. For image understanding benchmarks, the generation budget is measured in terms of the number of visual tokens consumed, as controlled by the min_pixels and max_pixels parameters in the dynamic resolution setting. The paper reports average image tokens consumed for dynamic resolution (1924 tokens per image for Qwen2-VL-7B, Table 7). For video benchmarks, the budget is measured by the number of sampled frames (2 FPS) and the total token limit per video (capped at 16384 tokens, Section 2.1). For generation tasks (agent operations, mathematics, visual QA), the model generates text autoregressively up to task completion. The paper does not report FLOPs or wall-clock time for inference, focusing instead on token counts as the primary efficiency metric.

• Cross-validation / statistical protocol. The paper does not describe a specific cross-validation or statistical significance testing protocol for most benchmarks. Standard benchmark evaluation protocols are followed: test set performance is reported using the official metrics and evaluation scripts for each benchmark. For the referring expression comprehension task (Section 3.2.5), results are reported on standard validation and test splits (val, test-A, test-B for RefCOCO and RefCOCO+; val and test for RefCOCOg), following the established evaluation convention (Kazemzadeh et al., 2014; Mao et al., 2016). The agent evaluation uses fixed test splits: AITZ uses a core clean test set derived from AITW (Rawles et al., 2024b; Zhang et al., 2024b), ALFRED reports valid-unseen performance (Shridhar et al., 2020a), and navigation tasks report valid-unseen splits (Anderson et al., 2018; Qi et al., 2020). For the internal function calling benchmark, construction procedures are described (Section 3.2.7) but cross-validation details are not provided.

Main Quantitative Results

The Qwen2-VL series achieves state-of-the-art results across a broad spectrum of visual understanding tasks, with the 72B model generally matching or exceeding GPT-4o and Claude 3.5 Sonnet, particularly on document understanding benchmarks. The results are organized by capability area below.

General Visual Question Answering

Qwen2-VL-72B achieves top-tier performance on most general visual QA benchmarks (Table 2). On RealWorldQA, the 72B model achieves 77.8, surpassing the previous SoTA (72.2, from InternVL2 (Chen et al., 2024c)) and GPT-4o (75.4). On MMStar, Qwen2-VL-72B scores 68.3, outperforming the previous SoTA of 67.1. On MMVet, the model achieves 74.0, substantially ahead of GPT-4V (67.5) and the previous open-source best. On MMT-Bench, Qwen2-VL-72B scores 71.7, a large margin over the previous SoTA of 63.4. On MMBench-EN, the 72B model matches the previous SoTA at 86.5, and on MMBench-CN, it establishes a new SoTA at 86.6. On MME, the cumulative score of 2482.7 exceeds the previous best of 2414.7. The 7B and 2B variants show competitive but lower performance, establishing a clear scaling trend: RealWorldQA scores drop from 77.8 (72B) to 70.1 (7B) to 62.9 (2B); MMStar drops from 68.3 to 60.7 to 48.0.

Document and Diagrams Reading

Qwen2-VL shows particular strength in document understanding tasks (Table 2). On DocVQA, Qwen2-VL-72B achieves 96.5, surpassing both the previous SoTA (94.1, InternVL2) and Claude 3.5 Sonnet (95.2), and well ahead of GPT-4o (92.8). On InfoVQA, the 72B model scores 84.5, exceeding the previous SoTA (82.0). On TextVQA, it achieves 85.5, surpassing the previous SoTA (84.4). On OCRBench, the composite score of 877 exceeds the previous SoTA (852, from MiniCPM-V (Yao et al., 2024)) and significantly outperforms GPT-4o (736) and Claude 3.5 Sonnet (788). On the multilingual MTVQA benchmark, Qwen2-VL-72B achieves 30.9, substantially ahead of GPT-4o (27.8), Claude 3.5 Sonnet (25.7), and the previous SoTA (23.2, Gemini (Team et al., 2023)). The 7B variants also perform competitively: DocVQA 94.5, OCRBench 866, MTVQA 25.6. Notably, the 2B model achieves 90.1 on DocVQA and 809 on OCRBench, demonstrating that even the smallest model retains strong document understanding capabilities.

The internal multilingual OCR benchmark (Table 3) shows Qwen2-VL-72B outperforming GPT-4o on Korean (94.5 vs. 87.8), Japanese (93.4 vs. 88.3), French (94.1 vs. 89.7), German (91.5 vs. 88.3), Italian (89.8 vs. 74.1), Russian (97.2 vs. 96.8), and Vietnamese (73.0 vs. 72.0), but underperforming on Arabic (70.7 vs. 75.9). This pattern across the multilingual OCR evaluation provides further support for the strength of Qwen2-VL's approach to text recognition while also highlighting potential challenges with specific scripts.

Mathematical Reasoning

On MathVista, Qwen2-VL-72B achieves 70.5, establishing a new SoTA (previous best: 69.0, Grok-2 (X.AI, 2024b)), and exceeding GPT-4o (63.8). On MathVision, the 72B model scores 25.9, which is the best open-source result, though it trails GPT-4o (30.4) and Claude 3.5 Sonnet (30.4, as reported in Table 2 with citation to OpenAI (2023) — the paper states "30.3 (OpenAI, 2023)" for MathVision but this may conflate the earlier GPT-4V with GPT-4o; the Table shows GPT-4o at 30.4). The scaling trend is evident: MathVista performance drops from 70.5 (72B) to 58.2 (7B) to 43.0 (2B), and MathVision drops correspondingly from 25.9 to 16.3 to 12.4.

Referring Expression Comprehension

On the RefCOCO benchmarks (Table 6), Qwen2-VL-72B achieves competitive performance among generalist models, with scores that approach or match specialized models. On RefCOCO val, the 72B model scores 93.2, compared to the best generalist (CogVLM at 92.8) and the best specialist (G-DINO-L/UNINEXT-H/ONE-PEACE at 92.6). On RefCOCO test-A, it achieves 95.3. On RefCOCO+ val, the 72B model scores 90.1 (compared to InternVL2-76B at 88.8), and on RefCOCO+ test-A, it achieves 93.8 (vs. InternVL2-76B at 93.1). On RefCOCOg val/test, the 72B model scores 89.9/90.4, comparable to InternVL2-76B (89.5/90.3). The 7B model also performs competitively: 91.7 on RefCOCO val, 87.3/87.8 on RefCOCOg. The paper notes that Qwen2-VL's structured bounding box format (normalized to [0, 1000) with dedicated <box_start>/<box_end> tokens) contributes to these results.

Video Understanding

Qwen2-VL-72B achieves strong video understanding performance (Table 4). On MVBench, it scores 73.6, setting a new SoTA (previous best: 69.6). On PerceptionTest, it achieves 68.0, exceeding the previous SoTA (66.9). On EgoSchema, it scores 77.9, outperforming GPT-4o (72.2) and Gemini 1.5 Pro (63.2). On Video-MME, the model achieves 71.2 without subtitles and 77.8 with subtitles, placing it between Gemini 1.5 Pro (75.0/81.3) and GPT-4o (71.9/77.2). The paper notes that for Video-MME, the maximum number of frames extracted per video was limited to 768 during evaluation (Section 3.2.6), which may impact performance on longer videos. The 7B variants show competitive but lower scores: MVBench 67.0, EgoSchema 66.7.

Visual Agent Capabilities

Qwen2-VL-72B demonstrates robust agent capabilities across diverse task categories (Table 5). In function calling, the 72B model achieves 93.1% Type Match (vs. GPT-4o at 90.2%) and 53.2% Exact Match (vs. GPT-4o at 50.0%). The paper attributes GPT-4o's underperformance to two factors: conservative tool use when uncertain, and weaker OCR capability for Chinese characters (Section 3.2.7). On the AITZ UI operations benchmark, Qwen2-VL-72B achieves 89.6 Type Match (vs. previous SoTA 83.0 and GPT-4o 70.0) and 72.1 Exact Match (vs. previous SoTA 47.7 and GPT-4o 35.3). In card games, it scores 100.0% Success Rate on Number Line (vs. GPT-4o 91.5, previous SoTA 89.4), 100.0% on EZPoint (vs. GPT-4o 85.5), and 42.6% on BlackJack (vs. GPT-4o 34.5). In robotic control (ALFRED valid-unseen), Qwen2-VL-72B achieves 67.8 Success Rate, marginally exceeding the previous specialized model ThinkBot (67.7). In navigation, performance is less competitive: 51.7 on R2R (vs. SoTA 79.0) and 31.0 on REVERIE (vs. SoTA 61.0), with the paper attributing this gap to difficulties in "accurately modeling maps and locations in a 3D environment" (Section 3.2.7).

Cross-Benchmark Scaling Analysis

Figure 6a shows performance scaling across model sizes (2B, 7B, 72B) aggregated by capability dimension. Model performance improves consistently with size across all dimensions: college-level problem-solving (MMMU), mathematical reasoning (MathVista + MathVision average), general visual QA (average of six benchmarks), document/table comprehension (average of six benchmarks), and video comprehension (average of four benchmarks). The strongest scaling effect appears in mathematical reasoning, which shows the steepest improvement with parameter count, while OCR-related tasks show relatively strong performance even at 2B scale. Figure 6b shows training dynamics for Qwen2-VL-7B during the second pretraining stage: performance on AI2D and InfoVQA improves steadily with training tokens, while general VQA performance shows more fluctuation. This suggests that document understanding tasks benefit more consistently from additional multimodal training data than general visual QA tasks, which may be more sensitive to specific data mixtures.

Ablation Studies and Robustness Checks

Dynamic resolution vs. fixed resolution (Table 7): The paper compares Qwen2-VL-7B performance under dynamic resolution (min_pixels=100×28×28, max_pixels=16384×28×28, resulting in 1924 average tokens) against four fixed-token configurations (64, 576, 1600, 3136 tokens). Dynamic resolution achieves InfoVQA 75.89, RealWorldQA 70.07, OCRBench 866, and MMMU 53.44. No single fixed resolution achieves optimal performance across all benchmarks: the best for InfoVQA uses 3136 tokens (77.27), best for RealWorldQA uses 1600 tokens (69.54), best for OCRBench uses 1600 tokens (824), and best for MMMU uses only 576 tokens (53.33). The dynamic approach consistently achieves top-tier or near-top-tier performance while using fewer tokens on average than the highest fixed-resolution setting.

Effect of min_pixels on perceptual vs. reasoning tasks (Figure 4): Ablating the min_pixels threshold for Qwen2-VL-7B reveals task-dependent effects. On InfoVQA, HallusionBench, and OCRBench, increasing min_pixels (upscaling small images to meet higher thresholds) improves performance monotonically up to a point. However, on OCRBench, very high min_pixels values cause severe performance decline — the paper attributes this to OCRBench containing "numerous extremely small images" that become out-of-distribution when excessively enlarged. On MMMU, increasing min_pixels has negligible effect, consistent with the hypothesis that MMMU's bottleneck is reasoning rather than visual detail. This ablation supports the claim that resolution should be allocated adaptively per image and per task, rather than uniformly maximized.

M-RoPE vs. 1D-RoPE (Table 8): Comparing Qwen2-1.5B + ViT-L pretrained models with M-RoPE versus 1D-RoPE, M-RoPE shows consistent improvements. On image benchmarks: MathVista improves from 39.2 to 43.4 (+4.2), MMBench from 58.6 to 60.6 (+2.0), DocVQA from 82.5 to 82.8 (+0.3), ChartQA from 68.0 to 68.4 (+0.4), InfoVQA from 50.8 to 50.3 (-0.5), TextVQA from 71.3 to 71.8 (+0.5). MMStar and RealWorldQA show minimal change (36.7→36.7, 54.5→53.7). On video benchmarks, gains are more pronounced: PerceptionTest improves from 46.6 to 47.4 (+0.8), NextQA from 43.9 to 46.0 (+2.1), and STAR from 55.5 to 57.9 (+2.4). The larger gains on video tasks validate the temporal component of M-RoPE, which provides explicit frame-to-frame position tracking that 1D-RoPE can only approximate.

Length extrapolation with M-RoPE (Figure 5): Qwen2-VL-72B, trained with a maximum of 16K tokens per video, is evaluated on Video-MME medium-length videos at varying inference sequence lengths. The model maintains robust performance from below 16K tokens up to 80K tokens (5× the training length), demonstrating that M-RoPE's decomposition enables extrapolation to longer sequences. The paper attributes this to M-RoPE keeping temporal position IDs small (counting frames rather than total sequence tokens), which keeps them within the range seen during training.

Model scaling across capability dimensions (Figure 6): Figure 6a shows consistent improvement across all five capability dimensions as model size increases from 2B to 7B to 72B. Mathematical reasoning shows the strongest positive correlation with parameter count, while OCR-related tasks show relatively strong performance even at 2B, suggesting that text recognition capabilities saturate earlier with scale than mathematical reasoning. Figure 6b tracks Qwen2-VL-7B performance during Stage 2 pretraining as a function of training tokens. AI2D and InfoVQA show steady improvement, while general VQA tasks show more fluctuation, suggesting that document understanding tasks benefit more consistently from additional multimodal training data.

Robustness to varying image sizes (Table 7, lower section): The paper reports that "adjusting image sizes only results in small perturbations in performance, demonstrating the model robustness to varying image sizes." The fixed-resolution experiments (64, 576, 1600, 3136 tokens) show that performance varies across benchmarks but within a manageable range, and that dynamic resolution achieves results comparable to the best fixed resolution per benchmark while using fewer tokens.

Visual Challenging benchmark (VCR) results (Table 2): On the VCR benchmark (Zhang et al., 2024c), Qwen2-VL-72B achieves 91.9 on VCR_en easy and 65.4 on VCR_zh easy. The VCR_en score exceeds both the previous SoTA (84.7, InternVL2) and GPT-4o (63.9). The VCR_zh score is particularly notable: it dramatically exceeds the previous SoTA (22.1) and GPT-4o (1.0), demonstrating significant advantages in Chinese text-centric visual understanding.

Critical Assessment

The experimental section makes several central claims: (1) Qwen2-VL achieves state-of-the-art performance across diverse benchmarks, (2) the dynamic resolution and M-RoPE innovations contribute meaningfully to this performance, (3) the model scales effectively from 2B to 72B parameters, and (4) the model demonstrates strong video understanding and agent capabilities. The experiments provide substantial evidence for claim (1), conditional support for claim (2) with some important gaps, reasonable support for claim (3) limited by the available data points, and mixed evidence for claim (4) depending on the specific capability.

State-of-the-art performance claim: The breadth of benchmarks and the consistency of competitive results strongly support that Qwen2-VL-72B is among the best general-purpose vision-language models at time of publication. However, several qualifications apply. First, the comparisons against GPT-4o and Claude 3.5 Sonnet rely on the paper's own evaluation pipeline for those models where possible, but some numbers are drawn from published sources and may not be strictly comparable (e.g., different prompting strategies, different image preprocessing). The paper does not detail the exact evaluation protocol used for proprietary models beyond citing their public benchmark results. Second, on MMMU, Qwen2-VL-72B (64.5) trails GPT-4o (69.1) by a meaningful margin, and the paper acknowledges this gap (Section 3.1). This is important because MMMU is one of the few benchmarks that tests complex reasoning across multiple disciplines — it may be a more discriminating test of genuine multimodal understanding than document or chart QA tasks where strong OCR capabilities can drive performance. Third, the specific numbers that place Qwen2-VL ahead of competitors (e.g., DocVQA 96.5 vs. 95.2 for Claude 3.5 Sonnet, MME 2482.7 vs. 2414.7 for InternVL2) are differences of 1-3 percentage points on benchmarks that may have measurement noise at that granularity. The paper does not report confidence intervals or standard deviations, so whether these differences are statistically significant or within noise is unknown.

Architectural innovation contributions: The ablation studies provide evidence that dynamic resolution and M-RoPE improve performance, but the strength of the evidence varies. Table 7 (dynamic resolution vs. fixed resolution) is the most direct test of dynamic resolution, and it convincingly shows that no single fixed resolution is optimal across all benchmarks. However, this comparison is between the full Qwen2-VL model (with all its training) and variants that differ only in the resolution processing — it does not isolate the contribution of dynamic resolution from the effects of training the model to handle variable resolutions. A stronger test would be to train a model with dynamic resolution from scratch and compare it to a fixed-resolution model at the same total training compute, ensuring that the dynamic model's advantages are not confounded with differences in effective training data diversity. The paper's ablation instead uses the same trained model and changes only inference-time resolution, which tests the model's robustness to resolution variation but may not fully characterize the training benefit.

M-RoPE's ablation (Table 8) uses Qwen2-1.5B with ViT-L — a substantially smaller model than the main 72B configuration — and tests only pretrained models, not instruction-tuned ones. The improvements are real (especially on video benchmarks) but moderate in absolute terms (2-4 points). Whether these gains scale to larger models and instruction-tuned settings is untested. The claim that M-RoPE enables length extrapolation (Figure 5) is supported by the Video-MME results, but only one model size (72B) and one benchmark are shown — it is unclear whether the extrapolation benefit generalizes.

Scaling analysis claim: Figure 6a demonstrates scaling from 2B to 72B, but with only three data points, it is impossible to establish the functional form of the scaling relationship. The paper does not fit a scaling law (power law or otherwise) — it simply shows that performance improves with size, which is expected. The more interesting question is whether different capabilities scale at different rates, and while the paper suggests mathematical reasoning scales faster than OCR, this is based on visual inspection of three points. Figure 6b shows training dynamics for only one model size (7B), making it a demonstration of learning curves rather than a scaling law analysis. The paper's claim to "investigate the scaling laws for large vision-language models" (Section 1) overstates what the experiments demonstrate — three model sizes constitute a scaling study, not the identification of scaling laws.

Video and agent capabilities: The video results (Table 4) are strong and cover a range of benchmarks, supporting the claim that the unified image-video architecture works effectively. However, the paper's note that Video-MME evaluation used a maximum of 768 frames (Section 3.2.6) raises a question about whether the model's video understanding is limited by this cap for very long videos. The agent results (Table 5) are more mixed. Function calling and UI operations show dramatic improvements over GPT-4o, but the paper's own analysis identifies that GPT-4o's weaker performance is partly due to conservative tool use and weaker Chinese OCR — confounds that make direct comparison difficult. The navigation results (R2R 51.7 vs. SoTA 79.0; REVERIE 31.0 vs. SoTA 61.0) show a large gap between Qwen2-VL and specialized models, indicating that general-purpose architectures still lag significantly behind specialized systems for 3D spatial reasoning from visual input. The robotic control result (ALFRED 67.8, marginally above ThinkBot 67.7) is competitive but relies on SAM for object localization, making it a hybrid system rather than a pure LVLM capability.

Missing experiments that would strengthen the paper:

1. Ablation of the three-stage training curriculum: The paper claims benefits from the ViT-only → all-unfrozen → LLM-only schedule, but never compares against alternatives (e.g., training everything jointly from the start, or two-stage variants). Without this, the specific contribution of the curriculum design remains conjecture.

2. Ablation of the 2×2 token compression: The compression from ViT output to LLM input is a critical efficiency mechanism, but its impact on accuracy relative to no compression or different compression ratios is never tested. The paper assumes compression is lossless enough, but provides no evidence.

3. Cross-model-family comparisons at matched scale: All results are from the Qwen2-VL family. Comparing against models of similar scale from other families with similar training data would strengthen the case that the architecture, not just scale and data, drives performance.

4. Measurement of statistical significance: The differences between Qwen2-VL-72B and GPT-4o/Claude 3.5 Sonnet on many benchmarks are within 1-3 percentage points. Without confidence intervals, it is unclear whether these differences are meaningful or within measurement noise.

5. Controlled study of min_pixels across diverse image types: Figure 4 shows the effect on a few benchmarks, but a systematic study across image categories (documents, natural images, screenshots, medical images) would better characterize when resolution scaling helps and when it can become harmful due to distribution shift.

6. Video length scaling study: While Figure 5 shows length extrapolation, a more thorough study would test accuracy as a function of video length (from seconds to hours) at different frame sampling rates, to characterize whether the 2 FPS sampling and 16K token budget create performance cliffs for long-form content.

Overall, the experiments provide strong evidence that Qwen2-VL achieves competitive state-of-the-art performance across a broad range of tasks, with particular strengths in document understanding and multilingual OCR. The architectural ablation studies support the contributions of dynamic resolution and M-RoPE, though the evidence would be strengthened by more controlled comparisons (e.g., training from scratch with vs. without the innovations, rather than inference-time comparisons or small-scale pretraining ablations). The scaling analysis is better characterized as a scaling demonstration (three data points showing improvement) rather than a rigorous scaling law investigation. The most robust finding is that adaptive resolution allocation achieves efficiency gains — competing with the best fixed resolution per task while consuming fewer tokens on average — which is well-supported by Table 7.

6. Limitations and Trade-offs

6.1 Dynamic Resolution Efficiency Gains Are Measured Excluding the Cost of Difficulty-Aware Allocation

The paper's efficiency argument — that dynamic resolution "consistently achieves top-tier performance while consuming fewer tokens on average" (Section 3.3.1) — treats resolution as an emergent property of the image's native dimensions constrained by min_pixels and max_pixels thresholds. The model does not learn to allocate resolution strategically; it simply processes images at whatever resolution results from scaling to fit within the pixel bounds. The thresholds (min_pixels = 100 × 28 × 28, max_pixels = 16384 × 28 × 28) are fixed hyperparameters, not learned or adapted per task. This means the resolution allocation policy is uniform across all images and tasks — a dense document gets no more preferential treatment relative to a simple photograph than what their native resolutions happen to dictate.

The consequence: The efficiency gains reported in Table 7 (dynamic resolution achieving InfoVQA 75.89 at 1924 average tokens vs. fixed 3136 tokens for 77.27) are a property of the distribution of image sizes in the benchmark, not a learned optimization. On a benchmark with different image size characteristics — for instance, one dominated by high-resolution medical scans or satellite imagery — the average token count under dynamic resolution would shift substantially, and the efficiency advantage relative to fixed resolution might shrink or disappear. More fundamentally, the approach cannot learn from experience that certain categories of images (e.g., chest X-rays, circuit diagrams, dense spreadsheets) systematically benefit from higher resolution, because there is no mechanism for the model or the allocation policy to adapt thresholds based on image content or task type. The min_pixels and max_pixels parameters are set globally and do not vary per domain, per task, or per image category.

Evidence in the paper: The Figure 4 ablation demonstrates that optimal resolution thresholds are task-dependent — increasing min_pixels helps on InfoVQA and HallusionBench (perceptual tasks) but hurts on OCRBench at high values (due to distribution shift from enlarging very small images) and has negligible effect on MMMU (reasoning-bottlenecked). This shows that a single global threshold cannot be simultaneously optimal for all tasks. Yet the paper's deployed model uses the same min_pixels = 100 × 28 × 28 across all evaluations. The efficiency numbers reported in Table 7 are therefore averages over a heterogeneous benchmark distribution and would not hold if the test distribution shifted toward image categories that require different resolution regimes. The paper does not measure how much additional efficiency could be gained from a task-aware allocation policy vs. the fixed-threshold baseline, so the reported efficiency represents a lower bound on what is achievable with smarter allocation but also overstates the realized efficiency in a deployment where the image distribution may differ from the benchmark mix.

Mitigation status: The paper does not address this limitation. The thresholds were presumably chosen based on validation performance, but the selection process is not described. The paper does not propose or evaluate content-adaptive resolution allocation (e.g., using a lightweight classifier to predict per-image resolution needs) and does not discuss the gap between the current fixed-threshold approach and a truly adaptive policy. Section 8 does not mention this as future work.

6.2 The ViT Produces Variable-Length Outputs, Making Batched Inference Inefficient

Naive Dynamic Resolution produces a variable number of visual tokens per image depending on each image's native dimensions. The paper's Table 7 reports an average of 1924 tokens per image, but the variance around this mean is substantial (the range spans from very small images at the min_pixels floor to high-resolution images near the max_pixels ceiling of 16384 × 28 × 28 pixels). For batched inference — the standard deployment paradigm for transformer models — this variability creates a tensor shape mismatch that must be resolved.

The consequence: In standard batched inference on GPUs, all inputs in a batch must have the same sequence length so they can be stacked into a single tensor. When images produce different numbers of visual tokens, the batch must be padded to the maximum token count in the batch, and attention masks must prevent the model from attending to padding tokens. For a batch where one image produces 8000 tokens (a high-resolution document) and another produces 200 tokens (a small icon), the small image incurs the computational cost of processing 7800 padding tokens — a 39× overhead relative to its actual information content. The paper's statement that "the packed length is controlled to limit GPU memory usage" (Section 2.1) acknowledges that packing is necessary but does not address the computational waste from padding within a batch. The paper's training infrastructure description (Section 2.3) notes that "the vision and text sequence lengths are dynamic for each data point" and that "we broadcast the dynamic sequence lengths before initiating the 1F1B process," confirming that variability is a first-class concern during training. However, training typically uses larger batch sizes and can amortize padding overhead through techniques like sequence packing across samples, whereas inference batches are smaller and less amenable to such optimization.

Evidence in the paper: The paper provides no measurements of batching efficiency, no analysis of the distribution of token counts (only the mean, 1924, is reported in Table 7), and no comparison of throughput (images per second) between fixed-resolution and dynamic-resolution inference. The training infrastructure section discusses how dynamic sequence lengths are handled in pipeline parallelism (broadcasting lengths before 1F1B, accessing shape information via batch indices), but these are solutions for training throughput, not inference latency. For a practitioner deploying Qwen2-VL in a production setting where throughput matters (e.g., processing thousands of images per minute), the absence of any batching efficiency analysis is a significant gap.

Mitigation status: The paper does not directly address this limitation for inference. The "packed length" control mentioned in Section 2.1 suggests that the model can concatenate variable-length image representations into a single sequence (avoiding per-image padding within the sequence), but this does not solve the batch-level shape mismatch across different examples in a mini-batch. Techniques like dynamic batching (grouping images with similar token counts into the same batch) or bucketing (quantizing token counts into bins) are standard solutions not discussed in the paper. The max_pixels = 16384 × 28 × 28 constraint (Section 2.1) caps the worst-case per-image token count, preventing pathological single-image cases from dominating the batch, but this addresses only the extreme tail of the distribution, not the common case of moderate variability.

6.3 The Revision Model Equivalent Is Absent — There Is No Mechanism for Structured Multi-Step Visual Reasoning Within a Single Query

The paper evaluates Qwen2-VL on tasks that require visual reasoning — MathVista (mathematical problem-solving from visual inputs), MMMU (college-level multimodal understanding), ChartQA (chart interpretation), and agent tasks requiring sequential decision-making from visual observations. However, the model architecture provides no built-in mechanism for iterative refinement of visual understanding or multi-step reasoning chains. The model processes an image once (producing a fixed set of visual tokens) and then generates text autoregressively. If the model misreads a number in a chart, misidentifies a geometric relationship in a diagram, or fails to locate a small UI element on a cluttered screen, there is no architectural pathway for it to "look again" at a higher resolution, to attend to a specific region more carefully, or to revise its visual interpretation based on partial reasoning progress.

The consequence: This creates a hard ceiling on performance for tasks where visual perception is the bottleneck. The paper's own results reveal this pattern. On MathVista, Qwen2-VL-72B (70.5) outperforms GPT-4o (63.8), suggesting strong mathematical reasoning from visual inputs. But on MathVision — which the paper describes as containing "3,040 math problems embedded in visual contexts from actual math competitions, covering 16 mathematical disciplines and varying in difficulty across five levels" (Section 3.2.4) — the 72B model scores only 25.9, trailing GPT-4o (30.4) and Claude 3.5 Sonnet (30.4). This gap between MathVista (which tests a broader range of visual-mathematical skills at varying difficulty) and MathVision (which focuses on competition-level problems requiring precise visual extraction and multi-step deduction) is consistent with a model that can reason well when it sees clearly but cannot recover from visual perception errors through iterative refinement. Similarly, on MMMU, the paper acknowledges that the model "still lags behind GPT-4o to some extent, indicating that Qwen2-VL-72B has room for improvement when handling more complex and challenging problem sets" (Section 3.1). The 64.5 vs. 69.1 gap on MMMU may partly reflect GPT-4o's ability (as a proprietary system that may use internal tool use or multi-step visual reasoning capabilities) to engage in more sophisticated visual reasoning chains.

Evidence in the paper: The paper does not directly study the effect of single-pass visual encoding on reasoning depth. The dynamic resolution mechanism (Section 3.3.1) provides a crude form of "look again" by allowing higher-resolution encoding of images that are naturally larger, but this is determined by image dimensions, not by task demands or intermediate reasoning state. The agent framework (Section 3.2.7) does implement multi-step interaction — the model can observe an action's result and decide on the next action — but this is multi-step interaction with the environment, not multi-step reasoning about a static image. Each new observation in the agent loop is a new image (a new screenshot), not a re-examination of the same image at higher resolution or with different attention. The function calling capability (Table 5) enables the model to invoke external tools (e.g., a code interpreter), but again, this is interaction with external computation, not iterative refinement of visual understanding of the original input.

Mitigation status: The paper does not address this limitation as a limitation. The strong performance on document understanding (DocVQA 96.5, InfoVQA 84.5) and chart understanding (ChartQA 88.3) suggests that for many practical tasks, single-pass visual encoding with high resolution is sufficient. For tasks requiring deeper visual reasoning chains, the paper implicitly relies on the LLM's internal chain-of-thought capabilities (generated as text) to compensate for the lack of visual iteration. The paper does not propose or evaluate mechanisms for iterative visual refinement, region-specific re-encoding, or multi-resolution visual reasoning — these are natural extensions that would build on the dynamic resolution foundation but are not explored.

6.4 The 3D Convolution with Depth Two Imposes a Fixed Temporal Compression That Cannot Adapt to Video Content

The paper integrates "3D convolutions with a depth of two to process video inputs, allowing the model to handle 3D tubes instead of 2D patches, thus enabling it to process more video frames without increasing the sequence length" (Section 2.1). This depth-2 3D convolution compresses every pair of consecutive frames into a single spatio-temporal tube representation, reducing the per-video token count by approximately 2× regardless of the video's temporal dynamics. An image is treated as "two identical frames" (Section 2.1) so that the depth-2 convolution can be applied uniformly.

The consequence: The compression rate is uniform across all video content, regardless of how much temporal change occurs between frames. For a video with rapid motion (e.g., a sports clip where every frame differs substantially from the previous one), compressing frame pairs may destroy fine-grained temporal information that is essential for understanding the action — the exact trajectory of a ball, the precise timing of a collision, the subtle motion cues that distinguish one action from another. For a video with static content (e.g., a lecture recording where the speaker and background barely change for minutes at a time), the same 2× compression undersells the true redundancy — much higher compression ratios could be achieved by detecting and exploiting temporal stasis. The fixed depth-2 convolution treats both scenarios identically, sacrificing temporal resolution where it matters and wasting representational capacity where it doesn't.

The depth-2 choice interacts problematically with the 2 FPS sampling rate. Two frames per second means that consecutive frames are 500 milliseconds apart — a long interval in which substantial motion can occur. Compressing pairs of 500ms-separated frames into single tubes means the model's finest temporal granularity is effectively 1 second. For tasks requiring sub-second temporal reasoning (e.g., detecting whether a person picked up or put down an object in a robotic manipulation video, tracking fast-moving objects in driving footage), this coarse temporal resolution may be insufficient.

Evidence in the paper: The video benchmarks (Table 4) show strong overall performance — MVBench 73.6, EgoSchema 77.9, PerceptionTest 68.0 — but these benchmarks test relatively high-level video understanding (action recognition, event comprehension, question answering about video content) rather than fine-grained temporal reasoning. The paper provides no ablation of the 3D convolution depth (e.g., comparing depth-2 against depth-1/no compression or depth-4), no analysis of how temporal compression affects performance on videos with different motion characteristics, and no measurement of whether information loss from the fixed compression rate is a binding constraint on any video task. The FPS ablation (e.g., comparing 1 FPS vs. 2 FPS vs. higher rates) is also absent.

Mitigation status: The paper acknowledges a related limitation — that for Video-MME, "we limited the maximum number of frames extracted per video to 768 during evaluation, potentially impacting performance on longer videos" (Section 3.2.6) — and notes that "Future work will focus on extending Qwen2-VL to support longer sequences, thereby accommodating longer videos." However, this addresses sequence length constraints, not the fixed temporal compression rate. The paper does not discuss adaptive temporal compression (e.g., applying higher compression to static video segments and lower compression to dynamic segments) or learning the optimal compression rate from data. The depth-2 choice appears to be an engineering convenience (it matches the image-as-two-frame convention and provides a clean 2× compression) rather than a principled optimum derived from video understanding requirements.

6.5 Three-Stage Training Curriculum Is Never Ablated — Its Contribution to Final Performance Is Unknown

The paper describes a three-stage training procedure: Stage 1 trains only the ViT on image-text pairs (LLM frozen), Stage 2 unfreezes all parameters for comprehensive multimodal training, and Stage 3 locks the ViT and fine-tunes only the LLM on instruction data (Section 2.2). The paper positions this as an adaptation of the Qwen-VL (Bai et al., 2023b) curriculum. However, the paper never compares this schedule against alternatives — no joint training from scratch, no two-stage variant, no ablation where Stage 1 is skipped or Stage 3 includes ViT updates.

The consequence: The relative contribution of the curriculum design to the model's final performance is unknown. Several plausible alternative curricula could produce equivalent or better results with lower training cost:

• Skip Stage 1 entirely: The ViT is initialized from DFN (Fang et al., 2023), which was already pretrained on a large-scale image-text dataset. Stage 1 adds 600B tokens of additional ViT-only training. If the DFN initialization is already strong, this stage may provide marginal benefit at substantial computational cost.
• Joint training from the start: Stage 1 freezes the LLM to prevent language capability degradation, but with proper learning rate scheduling and data mixing (including pure text data as in Stage 2), it may be possible to train all parameters jointly from the beginning, eliminating the separate ViT-only phase.
• Fine-tune the ViT during instruction tuning: Stage 3 freezes the ViT, treating its representations as fixed. If the instruction dataset contains domain-specific visual content (e.g., UI screenshots for agent tasks, specific document formats), allowing the ViT to adapt could improve performance on those domains at the risk of overfitting.

The paper's claim that Stage 1 "enhances semantic understanding within the Large Language Model (LLM)" (Section 2.2) is not directly tested — the ViT could enhance the LLM's understanding equally well (or better) if trained jointly from the start, because the LLM would provide stronger and more task-relevant gradients to the ViT than the frozen LLM does in Stage 1.

Evidence in the paper: There is none. The paper provides no curriculum ablation of any kind. The scaling analysis (Figure 6b) tracks performance during Stage 2 as a function of training tokens, but this shows learning curves within a single stage, not the marginal benefit of having completed Stage 1 first. The three-stage procedure is presented as a design choice adapted from prior work (Qwen-VL), not as a contribution validated by experiment.

Mitigation status: The paper does not acknowledge this as a limitation or suggest curriculum ablation as future work. For a practitioner seeking to reproduce or adapt Qwen2-VL, the lack of curriculum ablation means there is no guidance on which stages are essential vs. optional, how training cost would change if stages were simplified, or whether the curriculum's benefits are specific to the Qwen2 model family or generalize across LLM architectures. The training cost is substantial — 1.4 trillion tokens total — and any inefficiency in the curriculum directly translates to unnecessary computational expense. A simpler training procedure (e.g., joint training from the start) that achieved comparable performance at lower cost would be a significant practical improvement.

6.6 Agent Navigation Results Reveal a Fundamental Gap in 3D Spatial Understanding — General-Purpose Architectures Lag Far Behind Specialized Models

The paper evaluates Qwen2-VL on vision-language navigation (VLN) tasks: R2R (Anderson et al., 2018) and REVERIE (Qi et al., 2020). On R2R, Qwen2-VL-72B achieves 51.7% Success Rate compared to the specialized SoTA of 79.0% (Chen et al., 2022). On REVERIE, it achieves 31.0% vs. the specialized SoTA of 61.0% (Sigurdsson et al., 2023). These gaps — 27.3 and 30.0 percentage points respectively — are by far the largest performance deficits reported in the paper for any task category. The paper acknowledges the gap candidly: "The performance of Qwen2-VL is comparable to that of GPT-4o, but both models fall significantly behind current specialized VLN models. We attribute this gap to the incomplete and unstructured map information generated by the model from multiple images. Accurately modeling maps and locations in a 3D environment remains a major challenge for multimodal models" (Section 3.2.7).

The consequence: This result exposes a capability boundary that is not specific to Qwen2-VL but applies to the entire paradigm of flat-sequence multimodal architectures. The model processes a sequence of navigation observations (images at each step) and generates the next action. However, it has no explicit 3D spatial representation — no map, no coordinate system, no mechanism for maintaining a consistent spatial model across observations. The M-RoPE decomposition provides temporal, height, and width position encodings, but these encode position within individual frames, not 3D spatial position in the environment. The model must implicitly construct and maintain a spatial understanding through attention over the history of observations, but the 51.7% R2R success rate indicates that this implicit representation is substantially worse than what specialized models achieve with explicit spatial reasoning modules.

This limitation is consequential for the paper's agent vision. Qwen2-VL is positioned as a general-purpose VL-Agent capable of "UI Operations, Robotic Control, Games, and Navigation" (Section 2.2.1). The navigation results suggest that the "general-purpose architecture" approach hits a wall on tasks requiring persistent, accurate 3D spatial modeling — a capability that is central to robotics, autonomous driving, and augmented reality applications. The strong UI operation results (AITZ Exact Match 72.1 vs. previous SoTA 47.7) and robotic control results (ALFRED 67.8, marginally above ThinkBot) may mask the fact that these tasks involve relatively structured environments (a 2D phone screen, a room-scale virtual environment with known object types) where implicit spatial reasoning suffices. Navigation in open 3D environments with complex geometry and ambiguous visual landmarks pushes beyond what the architecture can handle.

Evidence in the paper: The R2R and REVERIE results in Table 5 are the direct evidence. The paper's attribution — "incomplete and unstructured map information generated by the model from multiple images" — is essentially a diagnosis of the limitation: the model cannot construct the kind of structured spatial representation that specialized systems build explicitly (e.g., topological maps, metric maps, graph-based scene representations). The 32K context length support (Section 3.2.7) means the model can attend over many previous observations, which helps, but the results show that raw attention over a flat history of images is not a substitute for structured spatial memory.

Mitigation status: The paper acknowledges the limitation but offers no architectural mitigation. The discussion of future work in Section 4 does not mention 3D spatial reasoning or navigation as a priority area. For practitioners, this limitation implies that Qwen2-VL should be paired with specialized spatial reasoning modules for navigation applications, rather than used as a standalone navigation agent. The function calling capability (Table 5) could, in principle, allow Qwen2-VL to invoke a dedicated SLAM or mapping module as an external tool, but this integration is not explored in the paper. The paper's broader narrative — that Qwen2-VL is a general-purpose multimodal model matching or exceeding GPT-4o across most tasks — should be qualified by the navigation results, which show that "general-purpose" does not mean "uniformly competitive with specialized systems" and that certain capabilities (3D spatial modeling) may require architectural innovations beyond what the current paradigm provides.

7. Implications and Future Directions

How This Work Changes the Landscape

Qwen2-VL's primary contribution is not a single architectural innovation but a demonstration that dynamic resolution and unified spatio-temporal position encoding, when implemented together at scale, allow a single open-weight model to compete with proprietary systems across an exceptionally broad range of visual tasks. The paper's conceptual move — reframing image resolution from a fixed architectural hyperparameter to a per-instance resource determined by native image dimensions — is incremental in isolation (NaViT already showed ViTs can handle variable resolutions; Dehghani et al., 2024). What changes the landscape is the empirical proof that this reframing, combined with M-RoPE, produces state-of-the-art results across document understanding, video, multilingual OCR, and agent tasks without task-specific architectures or modality-specific pipelines.

The methodological shift is subtle but important: the paper demonstrates that architectural unification (one ViT, one LLM, one position encoding scheme, one training procedure for images and video) is not just conceptually elegant but practically competitive with the best proprietary systems — and in some categories (DocVQA, InfoVQA, OCRBench, MTVQA) it surpasses GPT-4o and Claude 3.5 Sonnet. This strengthens the case that the path to general-purpose multimodal intelligence lies in removing architectural assumptions (fixed resolution, separate modalities, separate position encodings) rather than adding specialized components. Each removal of a fixed assumption — "images are 224×224," "images and video are different modalities," "position is one-dimensional" — expands the model's effective capacity without increasing parameter count.

The paper resolves a latent tension in the LVLM literature. Prior work oscillated between two poles: models like LLaVA that froze the ViT and used fixed resolution (simple, efficient, but resolution-limited and potentially suboptimal for visual reasoning), and models like InternVL2 that used large variable-resolution encoders with multi-crop strategies (better visual detail but higher complexity and less principled position handling). Qwen2-VL shows that training the ViT, using dynamic resolution with 2D-RoPE, and applying a simple 2×2 token compression achieves better performance than both approaches while consuming fewer tokens on average. The tension was between simplicity-efficiency and visual-fidelity; the paper resolves it by showing they are not in opposition when resolution allocation is adaptive.

The most significant conceptual reframing concerns position encoding in multimodal models. Prior to M-RoPE, the dominant assumption was that 1D sequence position was a sufficient representation because the model could learn to disentangle spatial and temporal signals from a single scalar. M-RoPE demonstrates that orthogonal decomposition of position into temporal, height, and width components provides a measurable and meaningful improvement, particularly for video understanding (Table 8: NextQA +2.1, STAR +2.4, PerceptionTest +0.8). This is not a paradigm shift — it's a refinement of how position is encoded. But it establishes a design principle with broad applicability: when different modalities have fundamentally different positional structures, orthogonal position encoding channels outperform a single shared channel. This principle extends naturally beyond the three components explored here. Future models with additional modalities (audio, depth, 3D point clouds, tactile) could analogously decompose position into modality-appropriate orthogonal components.

The paper makes certain research directions more attractive:

• Training vision encoders rather than freezing them — Qwen2-VL's success with a trained ViT (vs. the frozen CLIP ViT used by LLaVA and many others) strengthens the case that vision encoder adaptation is worth the computational cost, particularly for tasks requiring fine-grained visual understanding like document parsing and multilingual OCR.
• Unified image-video architectures — the paper's treatment of images as two-frame videos with 3D convolutions provides a template for models that don't need separate image and video pipelines. The strong video results (MVBench 73.6, EgoSchema 77.9, exceeding GPT-4o on both) validate this approach at scale.
• Dynamic compute allocation for visual inputs — the efficiency results (Table 7: competitive performance with 1924 average tokens vs. 3136 for fixed-resolution) open the door to more sophisticated allocation policies that go beyond native resolution and incorporate task type, image content, and available compute budget.

It also makes certain directions less attractive:

• Fixed-resolution multi-crop as the default strategy — the paper shows that adapting resolution to the whole image (with 2×2 compression for efficiency) outperforms or matches fixed-crop approaches while preserving global context, suggesting that grid-crop strategies may be unnecessarily complex for general-purpose models.
• Separate video-specific architectures — the strong video results from a unified architecture (same ViT, same LLM, same training procedure) raise the bar for justifying separate video encoders or modality-specific training pipelines. The burden of proof shifts: a separate video architecture must now demonstrate gains over unified training with 3D convolutions and M-RoPE, not just over image-only baselines.

Follow-Up Research This Work Enables

Content-adaptive resolution allocation that goes beyond native image dimensions. The paper's dynamic resolution uses fixed min_pixels and max_pixels thresholds applied uniformly to all images. Figure 4 demonstrates that optimal resolution thresholds are task-dependent — increasing min_pixels helps on InfoVQA and HallusionBench but can hurt on OCRBench due to distribution shift for very small images. A natural extension would train a lightweight classifier (potentially a small MLP on top of the ViT's [CLS] token or a separate tiny network) that predicts the optimal resolution for a given image based on its content and the downstream task. The training signal could come from the PRM-style approach used elsewhere: process each image at multiple resolutions, observe which resolution produces the correct answer, and train the classifier to predict that resolution. A strong follow-up would measure: (1) whether content-adaptive resolution outperforms the fixed-threshold baseline on a held-out benchmark not used for threshold selection, (2) how much additional efficiency gain (tokens saved) is achievable beyond the fixed-threshold approach, and (3) whether the classifier transfers across domains (e.g., trained on document-heavy data, tested on natural images). The experiment would directly test whether the paper's "naive" approach leaves meaningful efficiency on the table.

M-RoPE decomposition with learned rather than fixed channel allocation. The paper splits the query/key vector dimensions equally into thirds (temporal, height, width). This equal split is the simplest choice but almost certainly suboptimal — video tasks likely benefit from more temporal capacity, while document understanding likely benefits from more spatial (height/width) capacity. A follow-up could replace the fixed equal split with a learned gating mechanism: for each attention head, learn a soft assignment of dimensions to temporal vs. height vs. width components, possibly conditioned on the layer depth (lower layers might need more spatial capacity for fine-grained feature extraction, higher layers more temporal capacity for integrating across frames). The experiment would train Qwen2-VL variants with learned dimension allocation and measure: (a) whether different layers and attention heads learn different allocations, (b) whether the learned allocations correlate with task type (heads that attend to temporal patterns should allocate more dimensions to the temporal component), and (c) whether performance on video benchmarks improves relative to the equal-split baseline. A negative result — equal split being near-optimal — would be equally informative, suggesting that M-RoPE's benefit comes primarily from the orthogonal decomposition itself, not from the capacity allocation across components.

Stress-testing M-RoPE length extrapolation beyond 80K tokens on hour-long videos. Figure 5 shows that Qwen2-VL-72B maintains performance on Video-MME medium videos when inference length extends from 16K (training max) to 80K tokens. The paper notes that "future work will focus on extending Qwen2-VL to support longer sequences, thereby accommodating longer videos" (Section 3.2.6). A systematic stress test would evaluate on videos of increasing duration — 10 minutes, 30 minutes, 1 hour, 2 hours — measuring accuracy as a function of both video length and inference sequence length. The key question is whether M-RoPE's extrapolation property has a soft ceiling (performance gradually degrades with length) or a hard one (performance collapses beyond some threshold). The Video-MME benchmark already includes videos up to one hour; evaluating at full length without the 768-frame cap would test whether the extrapolation holds for the most challenging cases. If performance degrades gracefully, M-RoPE becomes a strong solution for long-form video understanding without architectural modification. If it collapses, it points to the need for complementary mechanisms (memory compression, key-value cache strategies, hierarchical attention).

Ablating the three-stage training curriculum to identify which stages are essential. The paper describes but never validates its training curriculum (ViT-only → all-unfrozen → LLM-only). A controlled study would train Qwen2-VL-7B variants with different curricula at matched total compute: (a) skip Stage 1 (joint training from the start), (b) skip Stage 3 (stop after Stage 2, evaluate with instruction prompts), (c) include ViT updates during Stage 3 (unfreeze all parameters during instruction tuning), and (d) two-stage variant (ViT-only then all-unfrozen, no separate instruction-tuning stage). The comparison would measure final performance across the full benchmark suite (Table 2) and training wall-clock time. This ablation is practically important because the three-stage curriculum adds complexity to the training pipeline and may be partially redundant — if Stage 1 provides minimal benefit given the strong DFN initialization, or if Stage 3's ViT-freezing is unnecessary, simpler training procedures could reduce cost without sacrificing performance. The 1.4T token training budget means even a 20% reduction in training cost through curriculum simplification would be practically significant. A negative result (the three-stage curriculum being essential) would validate the paper's implicit claim that managing the stability-adaptability tradeoff through staged freezing is critical at scale.

Quantifying the information loss from 2×2 token compression across task types. The paper compresses the ViT's output from a 16×16 token grid (for 224×224 images) to 8×8 by merging adjacent 2×2 blocks via an MLP, motivated by the need to keep sequence lengths manageable. The information loss from this compression is never characterized. A direct experiment would compare Qwen2-VL performance with and without compression on task categories where fine-grained spatial information plausibly matters: (a) small-object recognition (e.g., identifying distant objects in high-resolution photographs), (b) dense document reading (e.g., reading footnotes or fine print in scanned legal documents), (c) spatial reasoning requiring precise coordinate estimation (e.g., RefCOCO with very small bounding boxes). For each category, the model would be evaluated at matched token budgets — e.g., compare compression+high-resolution against no-compression+lower-resolution where total token counts are equalized. This would characterize the tradeoff: does compression preserve information better than simply using lower resolution to meet the same token budget, or does the 2×2 merging destroy fine-grained spatial information that lower-resolution encoding preserves through the ViT's own processing? The paper's assumption that MLP compression is "simple" and largely lossless is untested; this experiment would either validate that assumption or identify task categories where compression is a binding constraint, suggesting alternative compression strategies (learned downsampling, cross-attention-based pooling, or task-adaptive compression rates).

Multi-resolution iterative visual reasoning for hard problems. Section 6.3 of the analysis identifies the absence of iterative visual refinement as a limitation: the model processes each image once and cannot "look again" at higher resolution or with focused attention based on partial reasoning progress. A concrete follow-up would implement a simple iterative refinement loop: after the model generates an initial answer, it can request a higher-resolution encoding of specific image regions (indicated via bounding box tokens), which are then re-encoded by the ViT at higher effective resolution (by increasing min_pixels for that region) and inserted into the context for a second reasoning pass. The MathVision benchmark (where Qwen2-VL-72B scores 25.9, trailing GPT-4o at 30.4) would be the testbed — competition math problems often require precise reading of geometric diagrams where a single misread dimension or angle leads to a wrong answer. The experiment would measure: (a) whether iterative refinement improves MathVision accuracy, (b) how many refinement steps are needed before diminishing returns set in, (c) whether the refinement budget (extra tokens from re-encoding) could be spent more effectively on simply encoding the whole image at higher resolution initially. This would test whether the architectural limitation identified in Section 6.3 is real (performance improves with iteration) or merely theoretical (single-pass encoding at high resolution is sufficient, and the MathVision gap has other causes).

Practical Applications and Downstream Use Cases

Multilingual document digitization and information extraction. The paper's results on multilingual OCR — MTVQA 30.9 (vs. GPT-4o 27.8), and the internal benchmark where Qwen2-VL-72B outperforms GPT-4o on 7 of 8 languages (Table 3) — directly enable document processing pipelines for global organizations. A deployment scenario: a multinational corporation receives invoices, contracts, and regulatory filings in 10+ languages. Qwen2-VL-2B (DocVQA 90.1, MTVQA 18.1) can run on-device or in a cost-efficient cloud instance, extracting structured fields (dates, amounts, party names) from scanned documents without per-language OCR fine-tuning. The dynamic resolution mechanism ensures that dense legal documents are processed at sufficient resolution to capture fine print, while simple cover sheets don't waste tokens. The efficiency numbers from Table 7 (1924 average tokens vs. 3136 for fixed-resolution at comparable quality) translate to approximately 38% lower per-document compute costs at scale. The in-house multilingual OCR benchmark (Table 3) provides specific language coverage data to estimate performance for each target language.

Video content indexing and search for media archives. The video understanding results — MVBench 73.6 (new SoTA), EgoSchema 77.9 (exceeding GPT-4o at 72.2), Video-MME 71.2/77.8 (without/with subtitles) — enable automated tagging and search of video archives. A media company with thousands of hours of archival footage could deploy Qwen2-VL-7B (MVBench 67.0, EgoSchema 66.7) to generate dense textual descriptions of each video segment, enabling natural language search ("find all clips where someone is frying an egg in a kitchen with natural lighting"). The 2 FPS sampling rate and depth-2 3D convolution mean that 1 hour of video produces approximately 3600 processed frames (7200 raw frames compressed by 2×), which at the 16K token video budget fits within a single inference pass. For a 10,000-hour archive, processing with Qwen2-VL-7B at approximately 1924 image tokens per frame equivalent (matching the average from Table 7) yields an indexing cost that can be estimated from token counts. The chief value is in search accuracy — the model's strong performance on both action recognition (MVBench, EgoSchema) and text reading in videos (MTVQA results suggest strong multilingual OCR) means queries can find content based on both visual actions and on-screen text in multiple languages.

Mobile UI automation with grounding and function calling. The agent results provide a concrete deployment pathway for on-device UI automation. Qwen2-VL-72B achieves 72.1 Exact Match on AITZ (vs. GPT-4o at 35.3) and 93.1 Type Match for function calling (vs. GPT-4o at 90.2). The bounding box format (normalized to [0, 1000)) and the ChatML-based action loop (Section 2.2.1) provide a complete protocol for integrating Qwen2-VL with mobile automation frameworks. A practical deployment: a smartphone assistant that receives voice commands ("Find me a highly-rated Italian restaurant nearby and make a reservation"), uses Qwen2-VL to interpret screen contents (recognizing app icons, reading search results, identifying reservation buttons), and executes actions through the phone's accessibility API. The AITZ Exact Match of 72.1 suggests approximately 3 in 4 UI operations succeed on the first attempt. The 2B model (which the paper explicitly describes as "designed to run on-device" in Table 1) could handle simpler UI tasks locally, with the 7B or 72B models available via cloud for complex multi-step operations. The function calling capability (Section 3.2.7) enables the model to invoke external APIs (weather, calendar, maps) as part of the automation flow, exactly as demonstrated in the paper's flight arrival + weather example (Figure 26).

Math and science education tools with visual reasoning. The MathVista result (70.5, new SoTA, exceeding GPT-4o at 63.8) and the AI2D result (88.1, new SoTA) demonstrate capability in mathematical reasoning from visual contexts and scientific diagram understanding. An educational deployment: a homework assistance tool that processes photographs of math problems (handwritten or printed) and generates step-by-step solutions with explanations. The dynamic resolution mechanism is crucial here — students photograph problems at varying distances with varying camera quality; the model adapts its token allocation accordingly rather than processing everything at a fixed resolution that might be too low for small handwriting or too high for clear typeset text. The MathVision result (25.9) provides a realistic accuracy ceiling: on competition-level math problems, the model solves approximately 1 in 4, which is appropriate for a supplementary learning tool rather than an automated grader. The OCR capabilities (Table 3, strong multilingual performance) mean the tool works across languages — a student writing in Japanese, Korean, or French gets comparable recognition quality. The visual grounding format (<|box_start|> / <|box_end|>) could be extended to point at specific elements in diagrams ("the angle marked in blue is 45 degrees"), making the tool's explanations more pedagogically useful than text-only responses.