Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

ArXiv: 2308.12966

🎯 Pitch

Qwen-VL achieves fine-grained visual grounding and text reading in an open-source 7B model by simply aligning image-caption-box tuples, allowing it to beat Flamingo-80B on zero-shot captioning. It proves a single model can match or exceed both specialist tools and much larger generalists across conventional vision tasks and precise localization.

1. Executive Summary

This paper introduces the Qwen-VL series, a set of large-scale vision-language models built on the Qwen-7B language model that are designed to perceive and understand both text and images through a three-stage training pipeline—pretraining on 1.4 billion image-text pairs, multi-task pretraining on seven diverse vision-language tasks, and supervised fine-tuning for dialogue. The models incorporate a position-aware vision-language adapter that compresses visual features to a fixed length of 256 using cross-attention with learnable queries, and achieve fine-grained visual understanding—including grounding and text-reading—by aligning image-caption-box tuples with specialized tokens for bounding boxes (<box>, </box>) and referential grounding (<ref>, </ref>). The resulting Qwen-VL and Qwen-VL-Chat models set new records for generalist models at comparable scales on a broad range of benchmarks, including a 85.8 CIDEr score on zero-shot Flickr30K captioning (outperforming Flamingo-80B) and 89.36 accuracy on RefCOCO grounding, while the instruction-tuned Qwen-VL-Chat demonstrates superior dialogue performance on TouchStone, SEED-Bench, and MME, establishing that a single open-source model can match or exceed both specialist models and much larger generalist models across conventional vision-language tasks, text-oriented understanding, and fine-grained localization—though the model's pure-text capability remains comparable to other 7B-scale LLMs, showing no degradation from multimodal training.

2. Context and Motivation

The Gap: Open-Source LVLMs Lag Behind Proprietary Systems in Both Breadth and Precision

The core problem Qwen-VL addresses is that open-source large vision-language models (LVLMs) systematically underperform proprietary counterparts, and critically, most existing open-source models lack fine-grained visual understanding capabilities—specifically object grounding and text reading—that are essential for real-world applications. The paper states this explicitly in the introduction:

"current open-source LVLMs always suffer from inadequate training and optimization, thus lag far behind the proprietary models... which hinders further exploration and application of LVLMs in open-source community."

This is not simply a matter of scale. The gap is qualitative, not just quantitative. The majority of open-source LVLMs at the time of this work perceive images in a coarse-grained manner—they can describe what's in an image or answer questions about it, but they cannot precisely localize objects with bounding boxes or read text within images. As the paper notes:

"only a few attempts had been made toward this direction... the majority of open-source LVLMs remain perceiving the image in a coarse-grained approach and lacking the ability to execute fine-grained perception such as object grounding or text reading."

This distinction matters enormously. A model that can say "there's a dog in the image" is fundamentally different from one that can say "there's a dog at coordinates (120, 55) to (265, 770)." The latter enables downstream applications—robotic manipulation, accessibility tools for visually impaired users, document understanding, autonomous navigation—that the former cannot address.

Why This Problem Matters: Real-World Visual Scenarios Demand Precision

The paper's motivation is rooted in the complexity of real-world visual understanding. Consider what a user might ask an intelligent assistant: "What's the price on the second shelf of that refrigerator?" or "Read the expiration date printed on the bottom of the can." These queries require three interlocking capabilities:

1. Text reading (OCR): extracting textual content from an image.
2. Localization/grounding: mapping natural language descriptions to specific image regions.
3. Integration with reasoning: combining the extracted text and spatial information to answer the query.

A model that can only generate holistic image descriptions—no matter how fluent—cannot satisfy these requests. The paper argues that fine-grained visual understanding is not a niche capability; it's central to making LVLMs genuinely useful as assistants. This is reinforced by the real-world dialogue benchmarks the paper evaluates on (TouchStone, SEED-Bench, MME in Section 4.5), which test precisely these integrated capabilities.

Beyond dialogue, text-oriented visual understanding has direct applications in document digitization, chart interpretation, and accessibility. The paper's emphasis on text-oriented VQA benchmarks (TextVQA, DocVQA, ChartQA, AI2D, OCR-VQA in Section 4.2) reflects this: these tasks measure whether a model can extract and reason about text embedded in natural images, scanned documents, charts, diagrams, and book covers—exactly the kind of visual data that dominates real-world information work.

Prior Approaches and Where They Fall Short

The paper situates itself within several research threads, each with identifiable limitations.

1. Early Generalist Vision-Language Models (OFA, Unified-IO, CoCa)

Models like OFA (Wang et al., 2022a) transformed specific vision-language tasks into sequence-to-sequence problems, while Unified-IO (Lu et al., 2022a) extended this to tasks like segmentation and depth estimation. CoCa (Yu et al., 2022) used an encoder-decoder structure for both retrieval and generation. These models demonstrated the feasibility of a single architecture handling multiple tasks, but they suffered from poor instruction-following robustness, limited generalization to unseen tasks, and weak in-context learning abilities. The paper explicitly identifies these limitations (Section 5):

"previous vision-language models still have several limitations such as poor robustness in instruction following, limited generalization capabilities in unseen tasks, and a lack of in-context abilities."

These models were designed to output specific task formats (e.g., answer strings for VQA, captions for captioning) rather than to engage in freeform dialogue. Their training pipelines were task-engineered rather than conversation-oriented, limiting their flexibility.

2. LLM-Based LVLMs Without Grounding (BLIP-2, LLaVA, MiniGPT-4)

With the emergence of powerful LLMs, a new wave of LVLMs (BLIP-2, LLaVA, MiniGPT-4, mPLUG-Owl, InstructBLIP) connected frozen or fine-tuned vision encoders to LLMs, achieving strong image description and question-answering capabilities. BLIP-2 (Li et al., 2023c) introduced Q-Former to bridge vision encoders and LLMs. LLaVA (Liu et al., 2023) and MiniGPT-4 (Zhu et al., 2023) pioneered visual instruction tuning to improve dialogue abilities.

However, these models share a critical blind spot: they understand images holistically but cannot pinpoint where things are. They lack grounding—the ability to associate language with specific image regions. When asked "Where is the red cup?", they might respond "The red cup is on the table" but cannot provide bounding box coordinates. This limits their utility for tasks requiring spatial precision or interaction with physical objects based on visual input.

Furthermore, the paper notes that even for instruction-following, the gap between open-source and proprietary models remained substantial. GPT-4V (OpenAI, 2023) demonstrated strong capabilities across description, reasoning, and grounding, while open-source alternatives lagged behind.

3. Grounding-Enhanced Models (Kosmos-2, Shikra, BuboGPT)

A few recent models—Kosmos-2 (Peng et al., 2023), Shikra (Chen et al., 2023a), and BuboGPT (Zhao et al., 2023)—specifically set out to add visual grounding to LVLMs. Kosmos-2 used grounded image-text pairs from the GRIT dataset to teach models to generate bounding boxes alongside captions. Shikra introduced referential dialogue, allowing models to accept bounding box inputs and produce bounding box outputs within conversation. BuboGPT integrated grounding with audio understanding.

The paper acknowledges these contributions but positions Qwen-VL as going further by unifying grounding with strong performance on conventional vision-language benchmarks, text-oriented understanding, and multilingual dialogue within a single model. Prior grounding models often traded off general VQA or captioning performance for grounding ability, or vice versa. The Qwen-VL series aims to eliminate this tradeoff.

4. Document-Specialized Models (mPLUG-DocOwl)

mPLUG-DocOwl (Ye et al., 2023a) incorporated document understanding into LVLMs by training on digital document data. While this addressed text-oriented understanding, it focused narrowly on document formats rather than broadly on text-in-the-wild (natural images containing text, charts, diagrams). The paper's approach is more comprehensive, covering both structured documents and unstructured text-in-the-wild scenarios.

5. The Training Data Bottleneck

Underlying many of these limitations is a data problem. Fine-grained visual understanding—grounding and OCR—requires training data with bounding box annotations. Such data is expensive to curate at scale compared to the billions of web-crawled image-text pairs used for coarse-grained pretraining. The paper tackles this directly by constructing a multi-task training dataset (Table 3) that includes 3.5M grounding samples from GRIT, 8.7M referential grounding samples, 8.7M grounded captioning samples, and 24.8M OCR samples (both synthetic and from real PDF/HTML sources). The data construction strategy—particularly the synthetic OCR data generated via SynthDOG with natural scene backgrounds—is a key enabler that prior work did not fully exploit.

How Qwen-VL Positions Itself

The paper positions the Qwen-VL series as a unified model that does not choose between competing capabilities—it simultaneously achieves:

• State-of-the-art performance on conventional benchmarks (image captioning, general VQA) that matches or exceeds much larger models like Flamingo-80B.
• Fine-grained visual understanding (grounding, text reading) that approaches specialist models.
• Instruction following and dialogue that outperforms other open-source LVLM chatbots.
• Multilingual capability (English and Chinese) inherited from the Qwen-7B language model backbone.
• Multi-image reasoning through interleaved image-text training data.
• Few-shot in-context learning that the paper explicitly demonstrates (Section 4.4, Figure 4) and claims outperforms equivalently-sized models like Flamingo-9B and IDEFICS-9B.

This positioning is captured in Figure 1, which shows Qwen-VL achieving state-of-the-art performance across a broad range of tasks compared to other generalist models—not just excelling in one narrow domain. The qualitative examples in Figure 2 reinforce this by showing a single model handling multi-image dialogue, multilingual conversation, text reading, localization, and fine-grained recognition.

The paper's methodological contribution is the three-stage training pipeline (Section 3) that progressively builds capabilities: Stage 1 (pretraining on 1.4B image-text pairs) establishes coarse vision-language alignment, Stage 2 (multi-task pretraining on 7 tasks with higher resolution) adds fine-grained capabilities including grounding and OCR, and Stage 3 (supervised fine-tuning on 350k instruction samples) converts the model into an interactive chatbot. This staged approach is deliberate: each stage builds on the previous one, and the paper takes care to verify that capabilities from early stages are preserved (e.g., pure-text ability does not degrade, as shown in Appendix E.4).

A key design philosophy is simplicity in architecture but sophistication in data and training. The model architecture (Section 2.1) is straightforward—a ViT encoder, a single-layer cross-attention adapter, and the Qwen-7B LLM. There's no complex routing, no mixture-of-experts, no modality-specific processing branches. The innovation is in what data the model sees and when it sees it during training, not in architectural novelty. The position-aware VL adapter with 2D positional encodings is a targeted fix for the loss of spatial information during compression, but even this is a single cross-attention layer—a minimal intervention.

This positions Qwen-VL as an argument that open-source models can close the gap with proprietary systems without requiring architectural breakthroughs or massive scale increases. The model has 9.6B total parameters (1.9B ViT + 0.08B adapter + 7.7B LLM)—modest by contemporary standards—yet competes with or surpasses models many times its size on several benchmarks. The implicit message is that careful data curation and a well-designed training curriculum matter as much as, or more than, raw parameter count.

3. Technical Approach

3.1 Reader Orientation

Qwen-VL is a vision-language model system that takes images and text as input and produces text (with optional bounding box coordinates) as output—it's a single neural network that can look at pictures, read text within them, pinpoint objects by their coordinates, answer questions, and hold conversations, all in multiple languages. The core problem it solves is that prior open-source vision-language models were either good at holistic image understanding OR good at fine-grained localization, but not both simultaneously in a single model that also maintains strong dialogue capabilities—Qwen-VL's solution is a three-stage training curriculum that progressively builds capabilities: first learning coarse image-text alignment from 1.4 billion web-crawled pairs, then adding fine-grained skills (grounding, OCR, text reading) through multi-task training on seven diverse tasks at higher resolution, and finally converting the model into an interactive chatbot through supervised instruction fine-tuning, all while using a simple architectural design where complexity lives in the data and training procedure rather than in novel model components.

3.2 Big-Picture Architecture (Diagram in Words)

The Qwen-VL system has three main components connected in a feedforward pipeline:

1. Visual Encoder (Vision Transformer, ViT-bigG from OpenCLIP): Takes an input image resized to a specific resolution (224×224 in Stage 1, 448×448 in Stages 2–3), splits it into patches with a stride of 14, and produces a sequence of visual feature vectors. At 448×448 resolution, this yields 1024 feature vectors. This component is initialized from OpenCLIP's pretrained weights and is frozen during the final supervised fine-tuning stage.

2. Position-aware Vision-Language Adapter: A single cross-attention layer that compresses the variable-length visual feature sequence (256 vectors at 224×224, 1024 vectors at 448×448) into a fixed-length sequence of 256 vectors. It does this using 256 learnable query vectors that attend to the visual encoder's output. Critically, 2D absolute positional encodings are added to the query-key pairs in the cross-attention to preserve spatial information during compression. This component is randomly initialized in Stage 1 and trained throughout all stages.

3. Large Language Model (Qwen-7B): A 7.7B parameter autoregressive language model that receives the compressed 256 visual feature vectors (bracketed by special <img> and </img> tokens) interleaved with text tokens, and generates text output. The LLM also handles bounding box inputs and outputs through a string-based coordinate format normalized to [0, 1000) and wrapped in <box>/</box> tokens, with referential text marked by <ref>/</ref> tokens. The LLM is frozen in Stage 1, fully trained in Stage 2, and trained alongside the adapter (but not the visual encoder) in Stage 3.

Information flows sequentially: Image → Visual Encoder → feature sequence → VL Adapter (cross-attention compression) → 256 fixed-length vectors → LLM (alongside text tokens) → generated text output. For tasks involving bounding boxes, the LLM generates coordinate strings in the format (Xtopleft, Ytopleft),(Xbottomright, Ybottomright) which are parsed as spatial locations.

3.3 Roadmap for the Deep Dive

• First, the training objective and loss function, because it's shared across all three stages and defines what the model is learning to do at a fundamental level—understanding this upfront makes the stage-by-stage breakdown more coherent.
• Second, the visual encoder and adapter architecture in detail, including the cross-attention compression mechanism, the 2D positional encoding scheme, and the ablation that motivated the choice of 256 learnable queries—since this is the only architectural innovation and the bridge between vision and language.
• Third, the input-output interface, focusing on the special token system for images, bounding boxes, and references, because this token-based approach to grounding is what allows the LLM to handle spatial information without architectural modifications.
• Fourth, the three-stage training pipeline in chronological order (pretraining → multi-task pretraining → supervised fine-tuning), explaining what gets trained, what data is used, and what capability is acquired at each stage, since the staged curriculum is the paper's core methodological contribution.
• Fifth, the data construction strategy for grounding and OCR tasks, including the synthetic OCR generation pipeline and the bounding box normalization scheme, because data quality and format design are what enable the fine-grained understanding capabilities.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and training methodology paper whose core idea is that a straightforward architecture—when trained with a carefully staged curriculum on diverse, well-formatted multimodal data—can unify holistic image understanding with fine-grained localization and text reading in a single model that also maintains strong dialogue capabilities.

Training Objective: Next-Token Prediction with Cross-Entropy Loss

All three training stages share the same fundamental objective: minimize the cross-entropy loss on text tokens. There is no contrastive loss, no detection loss, no specialized grounding objective—everything is cast as text generation.

The training objective is:

$\mathcal{L} = -\sum_{t=1}^{T} \log P_\theta(y_t | y_{<t}, x)$

where $\theta$ represents all trainable model parameters, $x$ is the input sequence (which may include image features interleaved with text tokens), $y_t$ is the target text token at position $t$, $y_{<t}$ are all preceding tokens, and $T$ is the total number of target tokens in the sequence.

What it computes: For each position in the target sequence, the model predicts a probability distribution over its entire vocabulary given all previous context (including visual features). The cross-entropy loss penalises the model when it assigns low probability to the correct next token. The total loss is the sum (or equivalently, average) of these per-token losses across the entire target sequence. The model is trained to maximize the likelihood of generating the correct text given the input.

Why this form: Casting everything as next-token prediction means there is no architectural distinction between "vision-language tasks" and "pure language tasks." Image captioning, visual question answering, referring expression comprehension, grounded captioning, OCR, and pure text generation all use the same cross-entropy objective—the only difference is what tokens appear in the input and target sequences. This is a deliberate design choice that avoids the complexity of multi-task loss balancing (where different objectives might have different scales or convergence rates) and allows the model to share representations across all tasks. The tradeoff is that tasks with natural non-textual outputs (like bounding box coordinates) must be converted to text format, but the paper shows this works effectively with proper tokenization and formatting.

The paper also specifies that during training, loss is only computed on target tokens (the "answer" or "completion" portion), not on the input prompt or special formatting tokens that structure the task. This is stated implicitly in the description of the supervised fine-tuning data format (Appendix B.2):

"we ensure the consistency between prediction and training distributions by only supervising answers and special tokens (blue in the example), and not supervising role names or question prompts."

The "blue" tokens refer to the model's expected output—the actual answers, bounding box strings, or generated text—while role markers like <im_start>user or the question text are not included in the loss computation. This prevents the model from learning to simply copy the input prompt and forces it to learn the mapping from input to output.

Visual Encoder: Vision Transformer with Frozen OpenCLIP Initialization

The visual encoder is a Vision Transformer (ViT) using the ViT-bigG architecture from OpenCLIP (Ilharco et al., 2021), containing 1.9 billion parameters. The paper provides minimal architectural details about the ViT itself—the key information is about how images are processed through it.

Image preprocessing and patchification: During training and inference, input images are resized to a specific square resolution. In Stage 1 (pretraining), images are resized to 224×224 pixels. In Stages 2 and 3 (multi-task pretraining and supervised fine-tuning), images are resized to 448×448 pixels. The ViT splits these images into non-overlapping patches with a stride of 14 pixels. This means:

• At 224×224: The image is divided into a grid of $224/14 = 16$ patches per side, producing $16 \times 16 = 256$ patch embeddings.
• At 448×448: The image is divided into a grid of $448/14 = 32$ patches per side, producing $32 \times 32 = 1024$ patch embeddings.

Each patch is linearly projected into a vector (embedding dimension not specified in the paper, but consistent with ViT-bigG), and positional encodings are added. The ViT then processes these through multiple transformer layers (self-attention + feedforward) to produce a sequence of visual feature vectors of the same length as the number of patches: 256 vectors at low resolution, 1024 vectors at high resolution.

Why ViT-bigG from OpenCLIP: The ViT-bigG checkpoint is pretrained via contrastive language-image alignment, meaning its features are already organized in a way that correlates with natural language semantics. This provides a strong initialization for the vision-language adapter to build upon. The paper doesn't ablate alternative visual encoders, but the choice is motivated by OpenCLIP's demonstrated strong transfer performance. The visual encoder is frozen during Stage 1 (only the adapter is trained to align with it) and during Stage 3 (to preserve visual features during dialogue fine-tuning). In Stage 2, the visual encoder is unfrozen and trained alongside the LLM and adapter, allowing it to adapt to the higher resolution (448×448) and the specific demands of fine-grained tasks like OCR and grounding.

Window vs. global attention ablation (Appendix E.3): The paper considered using window attention in the ViT to reduce computational cost at higher resolutions. Window attention restricts self-attention to local patches (e.g., within a 224×224 window) in most layers, with only occasional global attention layers. The ablation results in Figure 8 and Table 10 show that window attention causes significantly higher training loss compared to vanilla (global) attention at 448×448 resolution, while providing minimal speed improvement (10 seconds per iteration for global vs. 9 seconds for window). At 896×896 resolution, window attention is substantially faster (25 vs. 60 seconds per iteration) but still reaches only comparable loss to the 448×448 global attention model while taking 2.5× longer to train. The paper therefore uses vanilla global attention at 448×448 as the practical sweet spot.

Position-Aware Vision-Language Adapter: Cross-Attention Compression

This component is the only architectural innovation in Qwen-VL and addresses a critical efficiency problem: feeding long sequences of visual features directly into the LLM would be computationally expensive because the LLM's self-attention cost scales quadratically with sequence length. A 1024-length visual feature sequence would dominate the LLM's context window and drastically slow down both training and inference.

Architecture: The VL adapter is a single cross-attention layer with randomly initialized weights. It uses a set of $K$ learnable query vectors (embeddings) of the same dimensionality as the LLM's token embeddings, where $K = 256$ was chosen after ablation (described below). The image features from the visual encoder serve as keys and values in the cross-attention operation. The adapter compresses the variable-length visual feature sequence (256 or 1024 vectors) into a fixed-length sequence of 256 vectors:

$\text{CompressedFeatures} = \text{CrossAttn}(Q = \text{LearnableQueries}, K = \text{ImageFeatures}, V = \text{ImageFeatures})$

where LearnableQueries is a set of 256 trainable embedding vectors and ImageFeatures is the output of the visual encoder.

What it computes: Each of the 256 learnable query vectors attends to all visual feature vectors from the ViT, computing attention weights that determine how much information to extract from each spatial location. The output is a weighted combination of the visual features, producing 256 compressed feature vectors. These 256 vectors form a summary of the image that the LLM can process efficiently. The compression ratio is 1:1 at 224×224 resolution (256 → 256, no compression) and 4:1 at 448×448 resolution (1024 → 256). The adapter essentially learns which visual information is worth preserving and how to aggregate spatial information into a compact representation.

Why this form: Cross-attention with learnable queries allows the adapter to learn task-specific compression strategies—it can learn to preserve fine-grained spatial details for grounding tasks while aggregating coarser semantic information for captioning tasks. A simpler pooling operation (e.g., average pooling or taking the CLS token only) would lose spatial information entirely. A resampling approach like Perceiver-style latent arrays is conceptually similar but the single-layer cross-attention here is maximally simple. The paper explicitly ablated the number of queries (Appendix E.2), finding that 256 queries provides the best convergence tradeoff at 448×448 resolution where the input has 1024 patches—too few queries (64 or 144) lose visual information, while too many (400) slow convergence without benefit.

2D Absolute Positional Encodings for Spatial Awareness: The paper identifies a specific risk with compression: positional information about where each visual feature came from in the original image might be lost during the cross-attention pooling. To mitigate this, 2D absolute positional encodings are incorporated into the query-key pairs in the cross-attention mechanism. This means that when a learnable query attends to a particular visual feature vector, it "knows" the (x, y) position in the original image grid that the feature came from. The paper states:

"considering the significance of positional information for fine-grained image comprehension, 2D absolute positional encodings are incorporated into the cross-attention mechanism's query-key pairs to mitigate the potential loss of positional details during compression."

This is critical for tasks like grounding and OCR, where the model needs to know where in the image a detected object or text is located. Without positional encodings, the compressed features would be a "bag of visual features" with no spatial reference, making it impossible to generate accurate bounding box coordinates.

Adapter parameter count and efficiency: The adapter contains only 0.08 billion (80 million) parameters—less than 1% of the total 9.6B model parameters. This means the vision-language interface is extremely lightweight, with most of the model's capacity devoted to visual encoding (1.9B) and language understanding/generation (7.7B).

Where the adapter is trained: In Stage 1, only the adapter and visual encoder are trained (LLM frozen). In Stage 2, the adapter is trained alongside the LLM and visual encoder. In Stage 3, only the adapter and LLM are trained (visual encoder frozen). This means the adapter is trained throughout all three stages, progressively adapting first to coarse alignment, then to fine-grained tasks at higher resolution, and finally to dialogue formats.

Input-Output Interface: Token-Based Multimodal Representation

The Qwen-VL model communicates all modalities—images, text, and bounding boxes—through a unified token-based interface. This is a deliberate design choice that avoids modality-specific processing branches or architectural modifications to the LLM. The LLM sees everything as "text" (a sequence of token IDs), where some of those "text" tokens actually represent visual features or spatial coordinates.

Image input tokens: After the visual encoder and adapter produce a compressed sequence of 256 feature vectors, these vectors are not directly fed into the LLM as separate embeddings. Instead, two special tokens are added to the model's vocabulary: <img> and </img>. The image feature sequence is bracketed by these tokens in the input sequence: <img> [256 visual feature vectors] </img>. The <img> and </img> tokens signal to the model (and to the data processing pipeline) where image content begins and ends. The visual feature vectors themselves are treated as if they were token embeddings—they have the same dimensionality as the LLM's token embeddings and are directly concatenated with text token embeddings in the input sequence.

For multi-image inputs, the paper uses a "Picture id:" prefix to distinguish different images:

"Picture 1: <img>...image features...</img> Picture 2: <img>...image features...</img> What is the difference between these two images?"

This allows the model to reference specific images in multi-image reasoning, which is evaluated qualitatively in the multi-image examples (Figure 2).

Bounding box output format: For tasks requiring spatial localization—such as referring expression comprehension ("where is the red cup?") or grounded captioning ("describe what's at this location")—the model needs to generate coordinate information. The paper converts bounding boxes to a string format normalized to the range [0, 1000):

$(X_{\text{topleft}}, Y_{\text{topleft}}), (X_{\text{bottomright}}, Y_{\text{bottomright}})$

where $X_{\text{topleft}}$, $Y_{\text{topleft}}$, $X_{\text{bottomright}}$, and $Y_{\text{bottomright}}$ are integers in $[0, 1000)$ representing the normalized coordinates of the top-left and bottom-right corners of the bounding box.

What it computes: Given an image of arbitrary dimensions, the bounding box is first expressed in absolute pixel coordinates, then normalized by dividing by the image width (for X coordinates) and image height (for Y coordinates), and finally scaled to the range [0, 1000). For example, if an image is 800 pixels wide and an object's bounding box has $x_{\text{min}} = 200$ and $x_{\text{max}} = 400$, the normalized coordinates would be $(200/800) \times 1000 = 250$ and $(400/800) \times 1000 = 500$. The coordinate string is then tokenized using the LLM's standard text tokenizer—digits, commas, and parentheses are all existing tokens in the vocabulary.

Why this form: The paper explicitly states why no special coordinate vocabulary is needed:

"The string is tokenized as text and does not require an additional positional vocabulary."

This is important because it means the model can generate coordinates using the same token prediction mechanism it uses for text—no additional detection head, coordinate regression layer, or specialized output module is needed. The normalization to [0, 1000) provides sufficient precision (three significant digits) for most grounding applications while keeping the coordinate strings compact. The choice of 1000 rather than a smaller range (like 100) provides finer granularity, while avoiding the very long strings that would result from using actual pixel coordinates (which could be 4+ digits). This is a pragmatic engineering choice that balances precision against sequence length.

Bounding box special tokens <box> and </box>: The coordinate string is wrapped in <box> and </box> tokens to clearly distinguish coordinate expressions from regular text. For example:

"The cat is sitting <box>(250,300),(500,700)</box> on the couch."

These tokens are added to the model's vocabulary as special tokens during training. They serve two purposes: (1) they make parsing the model's output unambiguous, and (2) they allow the model to learn that the content between <box> and </box> should be interpreted as spatial coordinates rather than narrative text.

Referential grounding tokens <ref> and </ref>: To associate descriptive text with corresponding bounding boxes, another pair of special tokens is introduced: <ref> and </ref>. These mark the text phrase that a bounding box refers to. For example, in grounded captioning data:

"<ref>bees</ref><box>(661,612),(833,812)</box><box>(120,555),(265,770)</box> gathering nectars from <ref>an apricot flower</ref><box>(224,13),(399,313)</box>"

Here, the word "bees" is associated with two bounding boxes (indicating multiple bees in the image), and "an apricot flower" is associated with one bounding box. The <ref>/</ref> tokens allow the model to learn the correspondence between noun phrases and image regions—a capability tested in the "Caption with Grounding," "Referring Grounding," and "Grounded Captioning" tasks (Table 3).

Why this token-based approach: By representing all modalities—images, text, coordinates, references—as sequences of tokens, Qwen-VL achieves two key properties. First, architectural simplicity: the LLM does not need to be modified at all. It receives a sequence of embedding vectors (some from text tokens, some from visual features) and predicts the next token—the LLM's self-attention mechanism handles cross-modal interactions automatically. Second, unified training: all tasks, regardless of their input-output modalities, are trained with the same next-token prediction objective. There is no need for multi-task loss balancing, auxiliary detection losses, or modality-specific objective functions.

This approach builds on the "sequence-to-sequence" paradigm established by OFA and Unified-IO but with the critical addition of the <ref>/</ref> tokens for referential association, which prior models handled differently (e.g., through separate grounding heads or position tokens).

Stage 1: Pretraining on 1.4 Billion Image-Text Pairs

The first training stage establishes coarse-grained vision-language alignment: the model learns to associate images with their textual descriptions at a holistic level. This stage uses the largest data volume but the simplest task format and the lowest input resolution.

Data: 1.4 billion image-text pairs, filtered from an initial 5 billion web-crawled pairs. As detailed in Table 2, the data sources include LAION-en (280M after cleaning), LAION-COCO (300M), DataComp (300M), Coyo (200M), academic caption datasets (CC12M, CC3M, SBU, COCO Caption), LAION-zh (105M Chinese pairs), and in-house Chinese data (220M). The cleaned dataset is 77.3% English and 22.7% Chinese, establishing the model's multilingual foundation from the very first stage.

Cleaning procedure: Appendix A.1 details an 8-step cleaning pipeline: removing pairs with extreme aspect ratios or too-small images, filtering by CLIP score (dataset-specific thresholds), removing text with non-English/non-Chinese characters or emoji, filtering by text length, cleaning HTML tags, and removing text with "unregular patterns." For academic datasets, pairs containing special tags are removed. When multiple captions exist for the same image, the longest one is selected. The cleaning is aggressive: the overall retention rate is 28% (1.4B out of 5B), with individual datasets ranging from 14% (LAION-en) to 100% (CC3M, COCO Caption, in-house Chinese data).

Training configuration in Stage 1:

• Image resolution: 224×224
• LLM sequence length (total tokens): 512
• ViT sequence length: 256 (since 224/14 = 16, 16×16 = 256 patches)
• Batch size: 30,720 image-text pairs (achieved via gradient accumulation over 6 steps)
• Optimizer: AdamW with $\beta_1 = 0.9$, $\beta_2 = 0.98$, $\epsilon = 10^{-6}$
• Peak learning rate: $2 \times 10^{-4}$, minimum: $1 \times 10^{-6}$, schedule: cosine decay with 500 warmup steps
• Weight decay: 0.05
• Gradient clipping: 1.0
• ViT layer-wise learning rate decay: 0.95 (each layer's learning rate is multiplied by 0.95 compared to the layer above)
• ViT drop path rate: 0 (no stochastic depth)
• Total steps: 50,000, consuming approximately 1.5 billion image-text samples

Note that 50,000 steps × 30,720 batch size = 1.536 billion samples, which is slightly more than the 1.4 billion unique samples—this means some samples are seen more than once, or the dataset is slightly oversampled. Each image is trained on only once (one epoch) as stated in Appendix E.1.

What is trained: Only the visual encoder (ViT) and the VL adapter are optimized. The LLM (Qwen-7B) is completely frozen. This is a crucial design choice: by freezing the LLM, the pretraining stage cannot degrade the LLM's language capabilities. The visual encoder and adapter must learn to produce representations that the frozen LLM can interpret—an "alignment" objective where the vision components adapt to the language model, not vice versa.

Training objective: Standard next-token prediction with cross-entropy loss, where the input is the image features and a prompt (likely "Generate the caption in English:" or similar, though the paper doesn't specify the exact prompt format for Stage 1), and the target is the image caption text.

Convergence behavior (Appendix E.1, Figure 6): The training loss decreases steadily from approximately 3.0 to approximately 1.8 over 1.5 billion images. Interestingly, the paper reports that zero-shot VQA performance on VQAv2 increases from roughly 48% to 56% accuracy during pretraining, even though the training data consists only of image-caption pairs with no VQA-specific data. This suggests that the learned visual representations transfer to question-answering without explicit training, a property inherited from the strong OpenCLIP initialization and reinforced by the alignment process.

Why this stage comes first: Starting with coarse alignment on a massive scale establishes a foundation of vision-language correspondence. The model learns that images and text relate to each other, even if imprecisely. The low resolution (224×224) and frozen LLM make this stage computationally efficient—the batch size of 30,720 is enormous, enabling rapid processing of 1.5 billion samples in 50,000 steps. This stage is about quantity over quality: the data is noisy, the resolution is low, but the sheer volume of diverse image-text pairs gives the model broad coverage of visual concepts.

Stage 2: Multi-Task Pretraining

The second training stage transforms the model from a coarse image captioner into a versatile vision-language system by training simultaneously on seven diverse tasks at higher resolution with the LLM unfrozen. This stage is where fine-grained capabilities—grounding, OCR, text reading—are acquired.

Data: Seven task categories with 76.9 million total samples (as summarized in Table 3):

1. Captioning (19.7M): Similar data sources to Stage 1 but far fewer samples and excluding LAION-COCO. This maintains the model's image description ability.
2. Visual Question Answering (3.6M): GQA (scene understanding), VGQA, VQAv2 (general VQA), DVQA (data visualization), OCR-VQA (text in book covers), DocVQA (document understanding), TextVQA (text in natural images), ChartQA (chart reasoning), AI2D (scientific diagrams). The inclusion of OCR-VQA, DocVQA, TextVQA, and ChartQA means text-oriented VQA is present from the start of multi-task training.
3. Grounding (3.5M): From the GRIT dataset, which provides captions with grounded noun phrases. The paper follows Kosmos-2's usage with "minor modifications." This task teaches the model to generate captions where noun phrases are associated with bounding boxes.
4. Referring Grounding (8.7M): From GRIT, Visual Genome, RefCOCO, RefCOCO+, and RefCOCOg. The task is: given an image and a referring expression (e.g., "the red cup on the left"), output the bounding box. This teaches the model to localize objects from natural language descriptions.
5. Grounded Captioning (8.7M): The same datasets as referring grounding, but the task is reversed: given an image and a bounding box, generate a description of the region. This teaches the model to describe specific image regions.
6. OCR (24.8M): The largest category. Includes synthetic OCR data generated via SynthDOG in both English and Chinese (with natural scene backgrounds from COCO train2017 and unlabeled2017), plus real OCR data from Common Crawl PDF and HTML pages with their text annotations and bounding boxes. This teaches the model to read text in images and associate text with spatial locations.
7. Pure-text Autoregression (7.8M): In-house text-only data. This prevents catastrophic forgetting of language capabilities during multimodal training.

Notice the symmetry: referring grounding and grounded captioning use the same datasets but reversed task formats, creating a duality that the paper explicitly mentions. This bidirectional training (image+text→box and image+box→text) likely helps the model learn a rich joint representation of visual regions and their descriptions.

Data format examples (Box B.1 in Appendix B): The paper provides explicit examples showing how each task is formatted as a text sequence. For instance:

• Caption with Grounding: <img>...image...</img>Generate the caption in English with grounding: Beautiful shot of <ref>bees</ref><box>(661,612),(833,812)</box><box>(120,555),(265,770)</box> gathering nectars from <ref>an apricot flower</ref><box>(224,13),(399,313)</box><eos>
• Referring Grounding: <img>...image...</img><ref>the ear on a giraffe</ref><box>(176,106),(232,160)</box><eos>
• OCR: <img>...image...</img>OCR with grounding: <ref>It is managed</ref><quad>(568,121),(625,131),(624,182),(567,172)</quad>...<eos>

The OCR task uses <quad> and </quad> tokens (not <box>) for quadrilateral coordinates, since text in natural scenes is often not axis-aligned and requires four-point polygons rather than axis-aligned rectangles. This is a subtle but important detail: the model learns two different spatial formats (<box> for rectangles, <quad> for quadrilaterals) within the same training stage.

Interleaved image-text data construction: The paper states that interleaved image-text data is constructed "by packing the same task data into sequences of length 2048." This means multiple task examples (potentially from different tasks) are concatenated into a single training sequence up to 2048 tokens, separated by appropriate task prompts. This teaches the model to handle multi-image and multi-turn scenarios where images and text are arbitrarily interleaved.

Bounding box data construction for grounding (Appendix A.3): For the GRIT dataset, the paper found "many recursive grounding box labels in one caption." A greedy cleaning algorithm is applied to "make sure each image contains the most box labels with no recursive box labels." This suggests that GRIT's raw annotations sometimes have nested or overlapping bounding box annotations, and the paper simplifies them to a flat, non-overlapping set to avoid confusing the model during training. For other grounding datasets (Visual Genome, RefCOCO variants), the paper "simply concatenate[s] the noun/phrase with respective bounding box coordinates"—a straightforward formatting step.

OCR data generation pipeline (Appendix A.4): The synthetic OCR data deserves particular attention because it's the single largest task category (24.8M samples) and is critical for text-reading ability. The process uses SynthDOG (Kim et al., 2022) with the following configuration:

• Backgrounds: COCO train2017 and unlabeled2017 dataset splits (natural scenery images)
• Fonts: 41 English fonts and 11 Chinese fonts
• Text: Generated using SynthDOG's default hyperparameters
• Labels: Generated text locations are tracked and converted to quadrilateral coordinates

This synthetic data provides perfectly labeled text-in-the-wild examples where every character's location is known exactly—something impossible to achieve at scale with human annotation. The use of natural scene backgrounds (rather than blank backgrounds) ensures the model learns to read text in realistic cluttered environments.

For real OCR data, PDF pages are processed using PyMuPDF (a PDF rendering library):

1. Extract all text and bounding boxes per page
2. Render each page as an image
3. Filter: remove too-small images, images with too many/few characters, images containing Latin Extended-A/B or Private Use Area Unicode blocks

HTML pages are processed similarly using Puppeteer (a headless browser) for rendering and text extraction. The filtering for specific Unicode blocks is interesting: Latin Extended-A/B characters are removed likely because they are rare and might cause tokenization issues, and Private Use Area characters are removed because they are font-specific and inconsistent across platforms.

Training configuration in Stage 2:

• Image resolution: 448×448 (doubled from Stage 1)
• ViT sequence length: 1024 (since 448/14 = 32, 32×32 = 1024 patches)
• LLM sequence length: 2048 (quadrupled from Stage 1's 512)
• Learnable query numbers: 256 (same as Stage 1, but now compressing 1024→256, a 4:1 ratio)
• Batch size: 4,096 (dramatically smaller than Stage 1's 30,720, reflecting the higher resolution and more complex data)
• Peak learning rate: $5 \times 10^{-5}$ (4× lower than Stage 1)
• Minimum learning rate: $1 \times 10^{-5}$
• Warmup steps: 400
• Total steps: 19,000
• Model parallelism: used for both ViT and LLM (2-way parallelism, as indicated by "✗" for pipeline parallelism and "2" for model parallelism in Table 8)
• All other optimizer settings: same as Stage 1 (AdamW, $\beta_1=0.9$, $\beta_2=0.98$, $\epsilon=10^{-6}$, weight decay 0.05, gradient clipping 1.0, cosine decay schedule)

What is trained: The entire model—visual encoder, VL adapter, and LLM—is now unfrozen and trained jointly. This is the critical difference from Stage 1: the LLM's weights are updated, allowing it to adapt its language representations to the multimodal tasks. The visual encoder also adapts to the higher resolution (448×448) and the specific demands of fine-grained tasks.

Why freeze in Stage 1 but unfreeze now: Stage 1 established coarse alignment without risking damage to the LLM's language capabilities. By Stage 2, the visual representations are already somewhat aligned with the LLM's embedding space, so unfreezing the LLM is less risky—the visual features it receives are already somewhat interpretable. The higher resolution and fine-grained tasks require the LLM to learn new capabilities (generating coordinates, associating phrases with boxes, reading text) that require updating its weights. The pure-text autoregression data (7.8M samples) acts as a regularizer, ensuring the LLM does not forget its language abilities while learning these new visual skills.

Design choice: resolution increase to 448×448: The paper states this is to "reduce the information loss caused by image down-sampling." At 224×224, fine text and small objects would be compressed below the resolution needed for OCR or precise grounding. At 448×448, the model has 4× more visual tokens (1024 vs 256) before compression, preserving more spatial detail. The VL adapter compresses this to 256 tokens in both cases, meaning the LLM sees the same number of visual tokens but with higher-quality information content.

Stage 3: Supervised Fine-Tuning for Dialogue

The third stage converts the multi-task pretrained model (Qwen-VL) into an interactive chatbot (Qwen-VL-Chat) by fine-tuning on instruction-formatted dialogue data.

Data: 350,000 instruction-tuning samples. The paper provides limited detail about the data composition but describes three sources:

1. Caption data or dialogue data generated through LLM self-instruction: This is existing data that "often only addresses single-image dialogue and reasoning and is limited to image content comprehension."
2. Manually annotated, model-generated, and strategy-concatenated dialogue data: Constructed specifically to "incorporate localization and multi-image comprehension abilities." This is critical because the self-instruction data would not naturally include bounding box outputs or multi-image reasoning.
3. Pure text dialogue data: Mixed in to "ensure the model's universality in dialogue capabilities."

The total size of 350k samples is markedly smaller than the pretraining datasets, following the standard instruction-tuning paradigm where a relatively small amount of high-quality conversational data can significantly improve dialogue behavior.

Dialogue format (Appendix B.2): The paper uses the ChatML format (OpenAI), where each interaction is marked with special tokens:

<im_start>user
Picture 1: <img>...image features...</img>What is the sign in the picture?<im_end>
<im_start>assistant
The sign is a road closure with an orange rhombus.<im_end>
<im_start>user
How is the weather in the picture?<im_end>
<im_start>assistant
The shape of the road closure sign is an orange rhombus.<im_end>

Several details are notable:

• <im_start> and <im_end> mark the beginning and end of each message.
• The role (user or assistant) follows immediately after <im_start>.
• Images are referenced by "Picture id:" where the id corresponds to input order (for multi-image dialogues).
• The image features are inserted inline, bracketed by <img> and </img>.

Training objective in Stage 3: The loss is computed only on the assistant's responses (the "answer" tokens) and the special tokens (<im_start>, <im_end>, the role label, and the image/box tokens if they appear in the response), but not on the user's questions or the role names. As specified in Appendix B.2:

"we ensure the consistency between prediction and training distributions by only supervising answers and special tokens (blue in the example), and not supervising role names or question prompts."

The example clarifies: in the sequence <im_start>assistant\nThe sign is...<im_end>, tokens from "The" through <im_end> receive loss, but <im_start>assistant\n does not. This prevents the model from learning to mimic user inputs (which would be useless for dialogue) and focuses learning on generating appropriate responses.

Training configuration in Stage 3:

• Image resolution: 448×448 (same as Stage 2)
• ViT sequence length: 1024
• LLM sequence length: 2048
• Batch size: 128 (dramatically smaller than Stage 2's 4,096, reflecting the smaller dataset)
• Peak learning rate: $1 \times 10^{-5}$ (5× lower than Stage 2)
• Minimum learning rate: $1 \times 10^{-6}$
• Warmup steps: 3,000 (much longer warmup relative to total steps, likely to stabilize training on the small dataset)
• Total steps: 8,000
• Gradient accumulation: 8 steps (128 × 8 = 1024 effective batch size per accumulation)
• ViT learning rate decay: 0 (the visual encoder is frozen, so no layer-wise decay applies)
• Model parallelism: 2-way (same as Stage 2)

What is trained: The visual encoder (ViT) is frozen. Only the VL adapter and the LLM are optimized. Freezing the ViT at this stage prevents the small dialogue dataset (350k samples) from distorting the visual representations learned from the much larger multi-task dataset (76.9M samples). The adapter and LLM are fine-tuned to adapt to the conversational format while preserving the visual understanding capabilities acquired in Stage 2.

Why this stage is necessary: After multi-task pretraining, Qwen-VL can perform diverse vision-language tasks but does so in a rigid, task-prompted format. It expects specific prompts like "Generate the caption in English with grounding:" or {question} Answer:. SFT teaches the model to respond to natural language instructions in a conversational format, to maintain context across multiple turns, to handle multi-image comparisons, and to know when to output bounding boxes versus plain text based on the conversational context rather than a task-specific prompt.

The paper confirms that the model successfully transfers its capabilities to dialogue:

"We confirm that the model effectively transfers these capabilities to a wider range of languages and question types."

This transfer is evaluated in Section 4.5 on TouchStone, SEED-Bench, and MME, where Qwen-VL-Chat outperforms other open-source LVLM chatbots.

Synthetic OCR Data Generation Pipeline

The OCR data category (24.8M samples, the largest in Stage 2) deserves its own detailed breakdown because it's a key enabler of the model's text-reading capabilities and the paper provides specific implementation details in Appendix A.4.

SynthDOG configuration:

• Background images: COCO train2017 and unlabeled2017 splits. These contain diverse natural scenes (indoor, outdoor, urban, rural) with varying lighting, clutter, and textures. Using these as backgrounds rather than blank canvases ensures the synthetic OCR examples mimic the visual complexity of real-world text-in-the-wild.
• Fonts: 41 English fonts and 11 Chinese fonts. The diversity of English fonts (41) versus Chinese fonts (11) reflects the greater typographic variation in Latin scripts compared to the more standardized Chinese character shapes. The fonts presumably cover serif, sans-serif, handwriting, display, and other styles to teach the model robustness to font variation.
• Text generation: SynthDOG's default hyperparameters control text content, size, color, orientation, and placement. The process generates text strings (in English or Chinese) and renders them onto the background images at varying positions, sizes, and rotations.
• Label generation: SynthDOG tracks the exact pixel locations of rendered text and outputs quadrilateral coordinates (four corner points). These are converted to the <quad> format: (x1,y1),(x2,y2),(x3,y3),(x4,y4). The use of quadrilaterals rather than axis-aligned rectangles is important for text in natural scenes, which can appear at arbitrary orientations and perspectives.

Real OCR data from PDF and HTML:

• PDF processing (PyMuPDF): For each PDF page: extract all text annotations with their bounding boxes, render the page as an image, filter based on image size and character count, and remove pages with rare Unicode blocks.
• HTML processing (Puppeteer): Similar pipeline but using a headless browser for rendering. HTML pages present additional challenges (dynamic layout, varied fonts, CSS styling) that provide complementary diversity to the PDF data.
• Unicode filtering rationale: Removing Latin Extended-A/B blocks eliminates accented characters and special Latin variants that are rare and might cause tokenization or evaluation issues. Removing Private Use Area characters eliminates font-specific glyphs that don't have consistent Unicode semantics. These filters improve data quality by focusing on commonly occurring characters.

Visualization example (Figure 5): The second row of Figure 5 shows examples of the synthetic OCR data, though the paper doesn't describe them in detail. The grounding data (first row) shows images with overlaid bounding boxes and associated text labels.

This OCR data construction strategy is cost-effective: 24.8M labeled examples are generated automatically with perfect ground-truth labels, without any human annotation cost. The combination of synthetic data (controlled, perfectly labeled, diverse backgrounds) and real data (authentic document layouts, realistic text distributions from Common Crawl) provides both coverage and realism.

Bounding Box Normalization and Tokenization

The coordinate normalization scheme is a critical detail that enables the LLM to generate spatial outputs without modification. The paper specifies:

"For any given bounding box, a normalization process is applied (within the range [0, 1000)) and transformed into a specified string format: (Xtopleft, Ytopleft), (Xbottomright, Ybottomright)"

Normalization process: Given an image of width $W$ pixels and height $H$ pixels, and a bounding box defined by absolute pixel coordinates $(x_1, y_1, x_2, y_2)$ where $(x_1, y_1)$ is the top-left corner and $(x_2, y_2)$ is the bottom-right corner:

$X_{\text{topleft}} = \left\lfloor \frac{x_1}{W} \times 1000 \right\rfloor$

$Y_{\text{topleft}} = \left\lfloor \frac{y_1}{H} \times 1000 \right\rfloor$

$X_{\text{bottomright}} = \left\lfloor \frac{x_2}{W} \times 1000 \right\rfloor$

$Y_{\text{bottomright}} = \left\lfloor \frac{y_2}{H} \times 1000 \right\rfloor$

where $\lfloor \cdot \rfloor$ denotes floor (rounding down to the nearest integer), $W$ is the image width in pixels, $H$ is the image height in pixels, and the multiplication by 1000 scales the normalized [0, 1] coordinates to [0, 1000).

What it computes: The absolute pixel coordinates of an object in the image are converted to resolution-independent normalized coordinates. For a 200-pixel-wide object centered in an 800-pixel-wide image, the X coordinates would be approximately $(300/800) \times 1000 = 375$ and $(500/800) \times 1000 = 625$. These normalized coordinates are resolution-independent—the same object in a higher-resolution version of the same image would produce the same normalized coordinates (assuming the same cropping).

Tokenization: The normalized coordinates are integers, so the string representation uses only digits, commas, spaces, and parentheses. For example, (250,300),(500,700) is tokenized by the LLM's standard BPE tokenizer into tokens like "(", "250", ",", "300", ")", ",", "(", "500", ",", "700", ")". Each of these is an existing token in the vocabulary—"250" might be a single token or split into subword tokens depending on the tokenizer's vocabulary. The paper notes that "the string is tokenized as text and does not require an additional positional vocabulary," meaning no special coordinate tokens are added to the vocabulary beyond <box>, </box>, <ref>, </ref>, <quad>, and </quad>.

Why this form: The normalization to [0, 1000) provides three significant digits of precision. For a 1000-pixel-wide image, this corresponds to 1-pixel precision; for a 4000-pixel-wide image, it corresponds to 4-pixel precision. This is a reasonable tradeoff for most object localization and OCR tasks—it's precise enough to locate objects but coarse enough that the coordinate strings are short (typically 15–25 characters per bounding box). The alternative of using absolute pixel coordinates would produce strings of varying length (e.g., "(1,1),(64,64)" vs. "(1024,768),(2048,1536)") and would not generalize across different image resolutions. The range [0, 1000) rather than [0, 1) with decimals avoids the period character and makes tokenization cleaner (no decimal points to tokenize).

Why <ref> and <box> tokens are paired: The referential grounding task requires the model to associate specific text phrases with spatial locations. Without <ref> and </ref> markers, the model would have to learn this association implicitly from word order and punctuation—a harder learning problem. The explicit markup makes the training signal unambiguous: "this bounding box corresponds to this specific phrase." This is analogous to how HTML uses tags to structure content, making parsing unambiguous for both humans and models.

Design Choice Summary: Why This Architecture and Training Strategy?

The paper makes several deliberate design choices that collectively define Qwen-VL's approach. Understanding the rationale behind these choices clarifies what the paper considers important:

1. Simplicity in architecture, complexity in data: The model has no novel architectural components—standard ViT, standard cross-attention, standard decoder-only LLM. The innovation is entirely in what data is used and how training is staged. This philosophy makes the model easier to implement and reproduce, and it demonstrates that careful data engineering can substitute for architectural novelty.

2. Unified token-based interface for all modalities: By representing images as feature vectors, bounding boxes as coordinate strings, and references as marked-up text—all within the same token stream—the model requires no modality-specific processing branches. This is a bet that the transformer's self-attention mechanism can learn to route information appropriately across modalities without explicit architectural guidance.

3. Staged training with progressive unfreezing: The three stages progressively unlock model components: Stage 1 trains only vision components (alignment without risking LLM capabilities), Stage 2 trains everything (full multimodal integration), Stage 3 freezes the visual encoder (preserving visual capabilities during dialogue adaptation). This staged approach manages the risk of catastrophic forgetting at each transition.

4. Resolution increase at Stage 2: Rather than training at high resolution from the start (which would be computationally expensive on 1.4B images), the paper starts at low resolution for the large-scale alignment stage and increases resolution only when fine-grained tasks are introduced. This is an efficiency optimization: 1.4B images at 224×224 is much cheaper than at 448×448, but the fine-grained tasks (grounding, OCR) genuinely benefit from the higher resolution and their smaller dataset size (76.9M total) makes it affordable.

5. Dual-task structure for grounding: The symmetric training on referring grounding (text→box) and grounded captioning (box→text) using the same datasets teaches the model a bidirectional mapping between language and spatial locations. This is analogous to how bilingual translation models are trained on source→target and target→source pairs to learn a shared representation.

6. Synthetic OCR data at scale: The largest task category (24.8M OCR samples) relies heavily on synthetic data, avoiding the prohibitive cost of human-annotated text-in-the-wild data. The combination with real PDF/HTML data ensures the model also sees authentic document layouts and text distributions.

7. Pure-text data in every stage: Both Stage 2 (7.8M pure-text autoregression samples) and Stage 3 (pure text dialogue data) include text-only data. This is a deliberate hedge against catastrophic forgetting of language capabilities, and Appendix E.4 (Table 11) confirms it works: Qwen-VL maintains MMLU (50.7), CMMLU (49.5), and C-Eval (51.1) scores comparable to its LLM initialization (49.9, not reported for CMMLU, and 48.5 respectively), and even improves slightly.

4. Key Insights and Innovations

Innovation 1: Unifying Fine-Grained and Coarse-Grained Visual Understanding in a Single Model Without Tradeoffs

The dominant assumption in prior open-source LVLM research was that there is an inherent tension between holistic image understanding (captioning, general VQA) and fine-grained visual capabilities (grounding, OCR, text reading). Models that excelled at one typically underperformed at the other. Kosmos-2 and Shikra added grounding but did not simultaneously demonstrate state-of-the-art performance on conventional VQA and captioning benchmarks at the level of the best generalist models. Specialist models like PALI-X achieved strong results on individual tasks through task-specific fine-tuning, sacrificing the unified generalist capability.

Qwen-VL's central intellectual move is to demonstrate that this tradeoff is not fundamental—it is an artifact of training data composition and curriculum design, not of architecture or capacity constraints. By training simultaneously on seven diverse tasks in Stage 2 (Table 3), with carefully balanced data ratios and symmetric task formats (referring grounding and grounded captioning as dual tasks on the same datasets), the model achieves:

• 85.8 CIDEr on zero-shot Flickr30K captioning, outperforming Flamingo-80B (a model with 8× more parameters) and approaching the 84.5 of the specialist InstructBLIP-FlanT5-XL (Table 4, Section 4.1)
• 89.36 accuracy on RefCOCO grounding, competitive with specialist models like G-DINO-L (90.56) and exceeding the generalist Shikra-13B (87.83) by a clear margin (Table 6, Section 4.3)
• 63.8 on TextVQA, where prior generalist models like BLIP-2 (42.4) and InstructBLIP (50.7) lag far behind (Table 5, Section 4.2)

The significance here is not just the raw numbers—it is the falsification of the presumed tradeoff. The paper provides evidence that a single model with modest parameter count (9.6B total) can simultaneously compete with both generalist models on holistic tasks and specialist models on fine-grained tasks, given the right training data and curriculum. This finding shifts the research question from "how do we balance these competing capabilities?" to "what training data composition enables them to coexist and potentially reinforce each other?"

The bidirectional grounding task design (text→box and box→text on the same data) is particularly important conceptually: it implies that the model learns a shared representation space where language phrases and spatial locations are interconvertible, analogous to how bilingual models learn a shared semantic space between languages. The paper does not explicitly make this analogy, but the symmetry of the training setup suggests it.

This is a fundamental reframing rather than an incremental improvement, because it changes what the field should expect from a single model. Prior work implicitly accepted that different architectures or training regimes were needed for different capability classes; Qwen-VL argues that diverse capabilities can coexist if trained appropriately, with evidence spanning a broad enough benchmark suite (Tables 4–7) to make the claim credible.

Innovation 2: The Three-Stage Training Curriculum as a Capability-Stacking Strategy

Before Qwen-VL, the dominant training paradigm for LVLMs was either single-stage pretraining followed by instruction tuning (LLaVA, MiniGPT-4) or two-stage alignment (BLIP-2 with Q-Former pretraining then LLM fine-tuning). These approaches typically introduced all capabilities simultaneously or in a single curriculum shift, without explicit consideration of when different model components should be frozen or unfrozen relative to the capabilities being acquired.

Qwen-VL introduces a deliberately sequenced three-stage curriculum where the freezing/unfreezing schedule is the key mechanism for managing catastrophic forgetting while stacking capabilities:

• Stage 1: Train only vision components (ViT + adapter) on massive noisy data at low resolution, with LLM frozen—establishing coarse alignment without risking language capability degradation.
• Stage 2: Unfreeze everything (ViT + adapter + LLM) on higher-resolution, higher-quality multi-task data—the LLM adapts its representations to incorporate spatial and textual understanding simultaneously, while pure-text data (7.8M samples) acts as a regularizer against language forgetting.
• Stage 3: Refreeze the visual encoder, fine-tune only the adapter and LLM on dialogue data—preserving the hard-won visual capabilities from Stage 2 while adapting interaction style.

The conceptual innovation is the recognition that the sequence of freezing/unfreezing controls which capabilities are at risk of being overwritten at each stage. Stage 1 protects language. Stage 3 protects vision. Stage 2, where everything is trainable and data is most diverse, is the integration phase where capabilities can mutually reinforce. The evidence that this works comes from Appendix E.4 (Table 11): Qwen-VL's pure-text performance (MMLU 50.7, CMMLU 49.5, C-Eval 51.1) does not degrade from its LLM initialization (MMLU 49.9, C-Eval 48.5) and even shows slight improvement after Stage 2's multi-task training.

This is an incremental but practically significant contribution. Prior work used freezing strategies (BLIP-2 froze both vision encoder and LLM during Q-Former training; LLaVA froze the visual encoder during instruction tuning), but these were ad hoc choices rather than a principled curriculum design. Qwen-VL makes the freezing schedule itself a deliberate variable in the training recipe, tied to specific capability-acquisition goals at each stage. The contribution is more engineering than theoretical, but it provides a replicable template for future LVLM training.

Innovation 3: Token-Based Spatial Representation as a Sufficient Interface for Grounding

The field's default assumption—inherited from object detection and vision-language grounding research—was that spatial localization requires specialized architectural components: detection heads, coordinate regression layers, region proposal networks, or dedicated position tokens with separate vocabularies. Prior grounding models (Kosmos-2, Shikra) used location tokens or specialized positional encodings that modified the model architecture. The implicit belief was that spatial reasoning is fundamentally different from linguistic reasoning and requires architectural accommodation.

Qwen-VL's counterargument—executed through its design—is that spatial coordinates treated as plain text tokens are sufficient for accurate grounding, with no architectural modifications to the LLM beyond adding four special markup tags (<box>, </box>, <ref>, </ref>). The normalization to [0, 1000) means the model generates coordinates using the same next-token prediction mechanism it uses for words. The LLM's self-attention learns to route visual features to coordinate generation and vice versa without explicit spatial reasoning modules.

The evidence that this works is compelling: 89.36 on RefCOCO val (Table 6), approaching specialist models like G-DINO-L (90.56) and ONE-PEACE (92.58) while using no detection-specific architecture. The model generates coordinate strings like "(250,300),(500,700)" exactly as it would generate the word "cat"—through autoregressive sampling from the same vocabulary distribution.

This is a fundamental conceptual shift, not an incremental improvement. It challenges the premise that spatial reasoning requires specialized neural mechanisms, suggesting instead that transformer attention over a sufficiently rich multimodal token stream can learn spatial reasoning as an emergent capability. If correct, this has implications beyond Qwen-VL: it implies that adding new "modalities" to LLMs may often be achievable through tokenization design rather than architectural modification—a democratizing insight for future multimodal research.

The <ref>/</ref> token pair is the key enabling design for this: by explicitly marking which text phrase corresponds to which bounding box in the training data, the model receives unambiguous supervision for the text-region association without needing a separate grounding module. This markup-based approach is analogous to how structured prediction tasks in NLP (named entity recognition, relation extraction) can be formatted as text-to-text problems with appropriate markup—Qwen-VL extends this insight to the spatial domain.

A caveat: the paper does not ablate against alternative spatial representation schemes (e.g., dedicated position tokens, continuous coordinate regression). The claim that text tokens are sufficient is supported by the strong performance, but the claim that they are optimal or better than alternatives is not tested. The innovation is the demonstration of sufficiency at high performance, not proof of superiority over all alternatives.

Innovation 4: Synthetic OCR Data at Scale as an Enabler of Text-Reading Generalization

Most prior LVLMs with text-reading capabilities either relied on external OCR pipelines (feeding extracted text as additional input) or trained on relatively small collections of human-annotated text-in-the-wild data. The field lacked a clear recipe for giving LLM-based vision models native OCR capability—the ability to read text directly from pixels without a separate OCR engine.

Qwen-VL's contribution is demonstrating that 24.8M synthetic OCR examples, generated automatically via SynthDOG with diverse natural backgrounds and 52 fonts, combined with real PDF/HTML data, is sufficient to give a 9.6B model strong text-reading capabilities across diverse benchmarks. The results in Table 5 show 63.8 on TextVQA (where BLIP-2 gets 42.4 and InstructBLIP gets 50.7), 65.1 on DocVQA (approaching Pix2Struct-Large's 76.6), and 75.7 on OCR-VQA.

The intellectual move is recognizing that for text reading, diversity of backgrounds, fonts, and layouts matters more than photorealism of individual examples. Synthetic data—perfectly labeled, arbitrarily scalable, controllable for diversity—can substitute for expensive human annotation if the rendering pipeline covers sufficient visual variation. The combination with real PDF and HTML data provides authentic layout diversity that pure synthetic data might miss.

This is an incremental but practically enabling contribution. SynthDOG existed before Qwen-VL, and the idea of synthetic data for OCR is not new. What Qwen-VL adds is the demonstration that synthetic OCR data can be integrated into a generalist LVLM training pipeline at scale (24.8M samples, the single largest task category in Stage 2) and that the resulting capability transfers to diverse downstream benchmarks (TextVQA for natural images, DocVQA for documents, ChartQA for charts, AI2D for diagrams, OCR-VQA for book covers). The breadth of transfer is the key finding: the model does not just learn to read synthetic text on COCO backgrounds but generalizes to reading text in completely different visual contexts.

For practitioners, this provides a cost-effective template: rather than collecting and annotating text-in-the-wild data for each new domain, invest in a diverse synthetic rendering pipeline and augment with a modest amount of real data from the target domains. The paper's specific configuration (41 English fonts, 11 Chinese fonts, COCO backgrounds, SynthDOG defaults) serves as a concrete starting point.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on a broad set of vision-language benchmarks summarized in Table 9 (Appendix D). These cover image captioning (Nocaps val, Flickr30K karpathy-test), general VQA (VQAv2 test-dev, OKVQA val, GQA test-balanced, ScienceQA-Img test, VizWiz VQA test-dev), text-oriented VQA (TextVQA val, DocVQA test, ChartQA test, OCR-VQA test, AI2Diagram test), referring expression comprehension (RefCOCO val/testA/testB, RefCOCO+ val/testA/testB, RefCOCOg val/test, GRIT refexp test), instruction following (TouchStone English and Chinese, SEED-Bench Image and Video, MME Perception and Cognition), and few-shot learning (OKVQA, Vizwiz, TextVQA, Flickr30K). The datasets span conventional vision-language tasks, fine-grained localization, text reading, and real-world dialogue—chosen to test the full breadth of capabilities the paper claims.

• Base model(s). The Qwen-VL models are evaluated in two variants: Qwen-VL (the checkpoint after Stage 2 multi-task pretraining, used for all conventional vision-language benchmarks, grounding, and few-shot evaluations) and Qwen-VL-Chat (the checkpoint after Stage 3 supervised fine-tuning, used for dialogue and instruction-following benchmarks). Both share the same 9.6B parameter architecture: ViT-bigG visual encoder (1.9B), VL adapter (0.08B), and Qwen-7B LLM (7.7B). The paper also references Qwen-VL's LLM initialization (an intermediate Qwen-7B checkpoint) for pure-text capability comparison in Appendix E.4.

• Metrics. The paper uses task-specific metrics as listed in Table 9. For image captioning: CIDEr score (↑), which measures consensus between generated and reference captions using n-gram overlap weighted by TF-IDF. For general VQA: VQA Score (↑) on VQAv2, OKVQA, and VizWiz (calculated as min(#humans_agreeing_on_answer/3, 1) averaged over questions), Exact Match accuracy (↑) on GQA, and Top-1 accuracy (↑) on ScienceQA-Img (constrained to multiple-choice options). For text-oriented VQA: VQA Score (↑) on TextVQA, ANLS (↑) on DocVQA (Average Normalized Levenshtein Similarity, measuring edit distance between predicted and ground-truth answers), Relaxed EM (↑) on ChartQA, and Exact Match (↑) on OCR-VQA and AI2Diagram. For referring expression comprehension: Accuracy (↑) at Intersection-over-Union (IoU) threshold of 0.5—a predicted bounding box is correct if its IoU with the ground-truth box exceeds 0.5. For instruction following: GPT-4 Score (↑) on TouchStone (GPT-4 rates model responses), and Accuracy (↑) on SEED-Bench and MME.

• Baselines. The paper compares against three categories of models. Generalist models include Flamingo-9B and Flamingo-80B (Alayrac et al., 2022), Unified-IO-XL (Lu et al., 2022a), Kosmos-1 and Kosmos-2 (Peng et al., 2023), BLIP-2 with Vicuna-13B (Li et al., 2023c), InstructBLIP with Vicuna-13B (Dai et al., 2023), Shikra-7B and Shikra-13B (Chen et al., 2023a), GPV-2, OFA-L* (Wang et al., 2022a), VisionLLM-H, mPLUG-DocOwl with LLaMA-7B (Ye et al., 2023a), and Pix2Struct-Large (1.3B). Specialist SOTAs (models fine-tuned or designed specifically for individual tasks) include PALI-17B, PALI-X-55B (Chen et al., 2023b), G-DINO-L, UNINEXT-H, ONE-PEACE (Wang et al., 2023), LLaVa+GPT-4, and CFR. Instruction-following baselines include VisualGLM, PandaGPT, MiniGPT4 (Zhu et al., 2023), InstructBLIP, LLaMA-AdapterV2 (Gao et al., 2023), LLaVA (Liu et al., 2023), and mPLUG-Owl (Ye et al., 2023b). For few-shot learning (Section 4.4, Figure 4), comparisons are against Flamingo-9B, OpenFlamingo-9B, IDEFICS-9B, Flamingo-80B, and IDEFICS-80B.

• Generation budget / compute accounting. The paper does not use a unified compute budget metric like FLOPs or generation count for comparing Qwen-VL to baselines. Comparisons are made on a per-benchmark basis using standard evaluation protocols (e.g., greedy decoding for captioning and VQA, single forward pass for referring expression comprehension). The model size (9.6B parameters) serves as the implicit efficiency metric—the paper emphasizes that Qwen-VL outperforms models with substantially more parameters (Flamingo-80B at ~80B, PALI-X at 55B) while using fewer. For few-shot learning, the paper specifies that "naïve random sample" exemplar selection is used rather than sophisticated methods like RICES.

• Cross-validation / statistical protocol. No explicit cross-validation or statistical significance testing is reported. For the instruction-following benchmarks (TouchStone, SEED-Bench, MME), standard evaluation protocols from each benchmark are followed. For few-shot learning experiments, the paper notes that results could be improved with sophisticated exemplar selection (RICES) but uses random sampling for consistency. The paper does not report confidence intervals, standard deviations, or multiple evaluation runs for any benchmark result.

Main Quantitative Results

Image Caption and General Visual Question Answering (Section 4.1, Table 4)

Headline numbers: Qwen-VL achieves 85.8 CIDEr on zero-shot Flickr30K, surpassing Flamingo-80B (67.2) by 18.6 points and approaching the specialist InstructBLIP-FlanT5-XL (84.5). On general VQA, Qwen-VL reaches 79.5 on VQAv2, 58.6 on OKVQA, and 59.3 on GQA—the highest among all generalist models compared.

Side-by-side comparisons on image captioning: On Nocaps (0-shot), Qwen-VL scores 121.4 CIDEr, essentially tying with InstructBLIP (121.9) and substantially exceeding BLIP-2 (103.9), Unified-IO-XL (100.0), and Kosmos-2 (not reported for Nocaps). On Flickr30K (0-shot), Qwen-VL's 85.8 exceeds Flamingo-80B (67.2) by a wide margin and outperforms Kosmos-2 (80.5) and BLIP-2 (71.6). Qwen-VL-Chat scores slightly lower at 120.2 and 81.0 on these benchmarks, reflecting a modest degradation after dialogue fine-tuning. The specialist SOTA (PALI-17B) achieves 127.0 on Nocaps, still higher than Qwen-VL.

Side-by-side comparisons on general VQA: On VQAv2, Qwen-VL (79.5) outperforms Unified-IO-XL (77.9), Shikra-13B (77.36), BLIP-2 (65.0), Kosmos-2 (51.1), Kosmos-1 (51.0), and Flamingo-80B (56.3). The specialist SOTA PALI-X-55B reaches 86.1—7 points higher. On OKVQA, Qwen-VL (58.6) exceeds BLIP-2 (45.9), Flamingo-9B (44.7), and Shikra-13B (47.16), with specialist SOTA PALI-X-55B at 66.1. On GQA, Qwen-VL (59.3) leads all generalist models, with BLIP-2 at 32.3 and InstructBLIP at 49.5, though the specialist SOTA (CFR at 72.1) is substantially higher. On zero-shot ScienceQA-Img, Qwen-VL (67.1) and Qwen-VL-Chat (68.2) both exceed BLIP-2 (61.0) and InstructBLIP (63.1), though the LLaVa+GPT-4 combination reaches 92.53. On zero-shot VizWiz, Qwen-VL (35.2) outperforms Flamingo-80B (31.6), Flamingo-9B (28.8), Kosmos-1 (29.2), and BLIP-2 (19.6), with Qwen-VL-Chat further improving to 38.9—the highest among generalist models, though PALI-X-55B (specialist) reaches 70.9.

Key observation: Qwen-VL's advantage over other generalist models is particularly pronounced on knowledge-intensive VQA (OKVQA: 58.6 vs. BLIP-2 45.9) and zero-shot generalization tasks (VizWiz: 35.2 vs. Flamingo-80B 31.6). The gap between Qwen-VL and specialist SOTAs remains substantial on some benchmarks (VizWiz: 35.2 vs. 70.9; ScienceQA-Img: 67.1 vs. 92.53), indicating that task-specific training still provides significant headroom.

Text-Oriented Visual Question Answering (Section 4.2, Table 5)

Headline numbers: Qwen-VL achieves 63.8 on TextVQA, 65.1 on DocVQA, 65.7 on ChartQA, 62.3 on AI2D, and 75.7 on OCR-VQA—leading or strongly competitive with all generalist models on every benchmark.

Side-by-side comparisons: On TextVQA, Qwen-VL (63.8) substantially exceeds InstructBLIP (50.7), BLIP-2 (42.4), and mPLUG-DocOwl (52.6), though PALI-X-55B (specialist with single-task fine-tuning) reaches 71.44. This 21-point gap over BLIP-2 is the largest relative improvement in this category and demonstrates the effectiveness of the 24.8M OCR training samples. On DocVQA, Qwen-VL (65.1) exceeds mPLUG-DocOwl (62.2) but trails Pix2Struct-Large (76.6), a model specifically designed for structured document understanding. On ChartQA, Qwen-VL (65.7) leads all compared models including mPLUG-DocOwl (57.4) and Pix2Struct-Large (58.6). On AI2D (scientific diagrams), Qwen-VL (62.3) substantially exceeds Pix2Struct-Large (42.1)—a 20-point gap suggesting that the model's general visual reasoning transfers well to diagram understanding. On OCR-VQA, Qwen-VL (75.7) leads Pix2Struct-Large (71.3) and reaches parity with PALI-X-55B (75.0, with OCR pipeline).

Key observation: Qwen-VL-Chat shows modest degradation on most text-oriented benchmarks (61.5 vs. 63.8 on TextVQA, 62.6 vs. 65.1 on DocVQA, 57.7 vs. 62.3 on AI2D, 70.5 vs. 75.7 on OCR-VQA) but improves on ChartQA (66.3 vs. 65.7). This pattern mirrors the captioning results—dialogue fine-tuning slightly reduces performance on task-formatted benchmarks but the losses are generally small (2–5 points), suggesting the Stage 2 capabilities are well-preserved.

Specialist gap analysis: PALI-X-55B with single-task fine-tuning and external OCR pipeline achieves 71.44 on TextVQA, 80.0 on DocVQA, 70.0 on ChartQA, 81.2 on AI2D, and 75.0 on OCR-VQA. Qwen-VL matches or approaches this on OCR-VQA (75.7) and ChartQA (65.7 vs. 70.0) but shows larger gaps on DocVQA (65.1 vs. 80.0) and AI2D (62.3 vs. 81.2). The DocVQA gap may reflect Pix2Struct's specialized document pretraining; the AI2D gap suggests room for improvement on diagram understanding specifically.

Referring Expression Comprehension (Section 4.3, Table 6)

Headline numbers: Qwen-VL achieves 89.36 on RefCOCO val, 92.26 on RefCOCO test-A, 85.34 on RefCOCO test-B, 83.12 on RefCOCO+ val, 88.25 on RefCOCO+ test-A, 77.21 on RefCOCO+ test-B, 85.58 on RefCOCOg val, 85.48 on RefCOCOg test, and 78.22 on GRIT refexp—setting new state-of-the-art among generalist models on all nine benchmarks.

Side-by-side comparisons: On RefCOCO val, Qwen-VL (89.36) exceeds Shikra-13B (87.83), Shikra-7B (87.01), VisionLLM-H (86.70), and OFA-L* (79.96). The specialist SOTA G-DINO-L (90.56) and ONE-PEACE (92.58) are 1–3 points higher. On RefCOCO test-A (focused on people), Qwen-VL (92.26) slightly exceeds G-DINO-L (91.19 per the table, though G-DINO-L reports 93.19—the paper shows 93.19) and trails ONE-PEACE (94.18) and UNINEXT-H (94.33) by 2 points. On RefCOCO test-B (focused on non-people objects), Qwen-VL (85.34) substantially leads Shikra-7B (80.24), Shikra-13B (81.81), and OFA-L* (76.39), while trailing G-DINO-L (88.24) and ONE-PEACE (89.26). On RefCOCO+ (harder, appearance-based descriptions), Qwen-VL achieves 83.12/88.25/77.21 on val/test-A/test-B, consistently leading Shikra-7B (81.60/87.36/72.12) and Shikra-13B (82.89/87.79/74.41), with ONE-PEACE leading at 88.77/92.21/83.23. On RefCOCOg (longer, more natural descriptions), Qwen-VL (85.58/85.48) leads Shikra-13B (82.64/83.16) and matches ONE-PEACE (89.22/89.27) within a few points. On GRIT refexp, Qwen-VL (78.22) leads Shikra-13B (69.03) by 9 points.

Key observation: Qwen-VL-Chat achieves comparable performance to Qwen-VL on referring expression benchmarks (RefCOCO val: 88.55 vs. 89.36; RefCOCOg test: 86.32 vs. 85.48), with similar variance. This is notable because referring expression comprehension was explicitly incorporated into the SFT data through manual annotation and model generation (Section 3.3), and the results confirm this capability transfers to dialogue without degradation—unlike the slight regression seen on some VQA benchmarks.

Generalist vs. specialist analysis: The gap between Qwen-VL and the best specialist model (typically ONE-PEACE or UNINEXT-H) is consistently 2–5 points across RefCOCO variants. Given that these specialist models are designed specifically for object detection and grounding—with architectures optimized for localization—Qwen-VL's performance using only a text-based coordinate output without any detection-specific components is remarkable. The 9-point lead over Shikra-13B on GRIT refexp (78.22 vs. 69.03) suggests that Qwen-VL's text-to-coordinate mapping benefits particularly from the dual-task grounding training (referring grounding + grounded captioning) compared to Shikra's approach.

Few-Shot Learning on Vision-Language Tasks (Section 4.4, Figure 4)

Headline numbers: Qwen-VL achieves better few-shot performance than Flamingo-9B, OpenFlamingo-9B, and IDEFICS-9B on OKVQA, Vizwiz, TextVQA, and Flickr30K across shot counts from 0 to 8, and is "even comparable with much larger models (Flamingo-80B and IDEFICS-80B)."

Side-by-side comparisons: The paper presents results in Figure 4 only—no numerical table is provided, so exact numbers must be read from the plot. On OKVQA, Qwen-VL (0-shot ~59%, 4-shot ~62%, 8-shot ~63%) consistently tracks above Flamingo-9B (0-shot ~45%, 4-shot ~50%, 8-shot ~52%) and approaches Flamingo-80B (8-shot ~57%). On Vizwiz, Qwen-VL (0-shot ~35%, 4-shot ~42%, 8-shot ~43%) exceeds Flamingo-9B (0-shot ~29%, 4-shot ~35%) and matches Flamingo-80B (8-shot ~41%). On TextVQA, Qwen-VL (0-shot ~64%, 4-shot ~65%) leads Flamingo-9B (0-shot ~30%, 4-shot ~32%) by a massive 33-point margin, and even Flamingo-80B only reaches ~38% at 4-shot. On Flickr30K, Qwen-VL (0-shot ~86, 4-shot ~93, 8-shot ~95 CIDEr) leads Flamingo-9B (0-shot ~61, 4-shot ~72) and approaches Flamingo-80B (8-shot ~83).

Key observation: Qwen-VL's few-shot gains are modest on most benchmarks (2–5 points from 0-shot to 8-shot) compared to Flamingo's steeper scaling curves, suggesting Qwen-VL already achieves near-saturation in its zero-shot performance. The largest gap between Qwen-VL and similarly-sized models appears on TextVQA, where the 30+ point advantage reflects the OCR training data—Flamingo and IDEFICS lack native text-reading capabilities. The paper explicitly notes that "sophisticated few-shot exemplar construction methods such as RICES are not used despite better results would be achieved," implying these results may understate Qwen-VL's few-shot potential.

Instruction Following in Real-World User Behavior (Section 4.5, Table 7)

Headline numbers: Qwen-VL-Chat achieves 645.2 on TouchStone (English) and 401.2 on TouchStone (Chinese), 58.2/65.4/37.8 on SEED-Bench (All/Image/Video), and 1487.58/360.71 on MME (Perception/Cognition)—ranking first among all compared LVLMs on all three benchmarks.

Side-by-side comparisons on TouchStone: On English TouchStone, Qwen-VL-Chat (645.2) exceeds mPLUG-Owl (605.4), LLaVA (602.7), LLaMA-AdapterV2 (590.1), InstructBLIP (552.4), MiniGPT4 (531.7), and PandaGPT (488.5). On Chinese TouchStone, Qwen-VL-Chat (401.2) is compared only to VisualGLM (247.1)—a 154-point advantage, reflecting Qwen-VL's multilingual training data. The paper notes these are GPT-4 evaluated scores, where GPT-4 rates the quality of model responses.

Side-by-side comparisons on SEED-Bench: On the All split, Qwen-VL-Chat (58.2) exceeds all compared models: InstructBLIP (53.4), Qwen-VL (56.3 without SFT), LLaVA (33.5), mPLUG-Owl (34.0), LLaMA-AdapterV2 (32.7), MiniGPT4 (42.8). On the Image split, Qwen-VL-Chat (65.4) leads InstructBLIP (58.8), Qwen-VL (62.3), LLaVA (37.0), and mPLUG-Owl (37.9). On the Video split, Qwen-VL-Chat (37.8) lags behind Qwen-VL (39.1) and InstructBLIP (38.1), which is notable because the paper states the model's visual capabilities "can be effectively transferred to video tasks by simply sampling four frames"—the SFT stage slightly reduces video understanding. The small gap (1.3 points) suggests the transfer is real but not perfect.

Side-by-side comparisons on MME: Qwen-VL-Chat achieves 1487.58 on Perception—the highest among all compared models, exceeding InstructBLIP (1212.82) by 275 points, LLaMA-AdapterV2 (972.67), mPLUG-Owl (967.34), VisualGLM (705.31), PandaGPT (642.59), MiniGPT4 (581.67), and LLaVA (502.82). On Cognition, Qwen-VL-Chat (360.71) also leads, ahead of InstructBLIP (291.79), mPLUG-Owl (276.07), LLaMA-AdapterV2 (248.93), and others. The Perception score is particularly striking—Qwen-VL-Chat is in a different performance tier from other open-source LVLMs, with InstructBLIP being the closest at 275 points behind.

Key observation: Qwen-VL (without SFT) achieves 56.3/62.3/39.1 on SEED-Bench, and Qwen-VL-Chat achieves 58.2/65.4/37.8. The SFT stage improves Image (+3.1) and All (+1.9) scores but reduces Video (-1.3). This slight video degradation after SFT may indicate that the dialogue fine-tuning data (which the paper notes "often only addresses single-image dialogue") does not include sufficient multi-frame or video content, causing mild forgetting of temporal reasoning.

Ablation Studies and Robustness Checks

Number of learnable queries in the vision-language adapter (Appendix E.2, Figure 7): The paper tests 64, 144, 256, and 400 learnable queries. At the beginning of training, fewer queries produce lower initial loss (less capacity → easier optimization). At convergence (1k–5k steps), 256 queries achieves the best training loss, with 64, 144, and 400 all converging to slightly higher loss values. The paper chooses 256 because it balances information preservation at high resolution (where 1024 ViT features are compressed 4:1) with convergence efficiency. The experiment is conducted with ViT-L/14 at 224×224 resolution, not the full ViT-bigG at 448×448, making it a approximate rather than exact ablation.

Window attention vs. global attention for Vision Transformer (Appendix E.3, Figure 8, Table 10): This ablation compares standard (global) self-attention in all ViT layers against window attention (local 224×224 windows in most layers, global in 1/4 layers). At 448×448 resolution, window attention causes "significantly higher" training loss than global attention, while providing minimal speed improvement (9 vs. 10 seconds per iteration). At 896×896 resolution, window attention is 2.4× faster (25 vs. 60 seconds per iteration) and reaches comparable loss to the 448×448 global attention model by 5000 steps, but takes 2.5× longer to train overall. The paper chooses 448×448 with global attention as the practical optimum. This is a revealing negative result: window attention—a standard efficiency technique in vision transformers—does not work well for this multimodal training setup at moderate resolutions.

Performance on pure-text benchmarks after multimodal training (Appendix E.4, Table 11): Qwen-VL achieves 50.7 on MMLU, 49.5 on CMMLU, and 51.1 on C-Eval, compared to its LLM initialization intermediate checkpoint at 49.9 on MMLU and 48.5 on C-Eval. This demonstrates no degradation—and slight improvement—in pure-text capabilities after the full three-stage multimodal training. For context, Qwen-VL's performance is comparable to other 7B-scale LLMs (LLaMA2-7B: 46.8 MMLU, Baichuan2-7B: 54.2 MMLU, ChatGLM2-6B: 47.9 MMLU) but lags behind the final released Qwen-7B (58.2 MMLU, 62.2 CMMLU, 63.5 C-Eval). The paper explains this gap by noting that Qwen-VL used an intermediate (not final) Qwen-7B checkpoint, developed in parallel.

Convergence of the pre-training stage (Appendix E.1, Figure 6): The training loss decreases from approximately 3.0 to 1.8 over 1.5 billion images. Zero-shot captioning performance on Flickr30K (CIDEr) improves from roughly 64 to 76, and zero-shot VQA on VQAv2 (accuracy) improves from roughly 48% to 56%—despite no VQA data being present in Stage 1 training. This emergent VQA capability from pure captioning data validates the quality of the learned vision-language alignment.

SFT stage impact on task-formatted benchmarks: While not presented as a formal ablation, comparing Qwen-VL and Qwen-VL-Chat across Tables 4, 5, and 6 reveals the effect of supervised fine-tuning. On captioning, Qwen-VL-Chat shows modest decline (121.4→120.2 Nocaps, 85.8→81.0 Flickr30K). On general VQA, similarly small drops (79.5→78.2 VQAv2, 58.6→56.6 OKVQA, 59.3→57.5 GQA) with improvement on VizWiz (35.2→38.9). On text-oriented VQA, mixed: declines on TextVQA (63.8→61.5), DocVQA (65.1→62.6), AI2D (62.3→57.7), OCR-VQA (75.7→70.5), but improvement on ChartQA (65.7→66.3). On referring expression, essentially flat (89.36→88.55 RefCOCO val, 85.58→85.96 RefCOCOg val). The overall pattern is that SFT causes small (2–5 point) regressions on heavily task-formatted benchmarks in exchange for strong gains on dialogue benchmarks (TouchStone, SEED-Bench, MME in Table 7). This is an expected tradeoff, not a failure mode.

Critical Assessment

Does Qwen-VL Actually Achieve State-of-the-Art Across All Claimed Dimensions, or Only Selected Ones?

The paper's central claim is that Qwen-VL "sets new records for generalist models under similar model scales on a broad range of visual-centric benchmarks" (Abstract) and Figure 1 presents a radar chart showing Qwen-VL outperforming other generalist models across all axes. The evidence in Tables 4–7 largely supports the breadth claim—Qwen-VL leads on the majority of benchmarks—but requires careful qualification.

The claim holds most strongly for: (1) zero-shot image captioning on Flickr30K (85.8 vs. next best generalist Kosmos-2 at 80.5—Table 4); (2) general VQA across multiple benchmarks (VQAv2, OKVQA, GQA, VizWiz—Table 4); (3) referring expression comprehension on all nine RefCOCO/RefCOCO+/RefCOCOg/GRIT splits (Table 6), where the lead over the next best generalist (Shikra-13B) ranges from 1.5 points on RefCOCO val to 9.2 points on GRIT refexp; (4) instruction following on MME Perception (1487.58 vs. 1212.82 for InstructBLIP—a 275-point gap that puts Qwen-VL-Chat in a different tier).

The claim requires qualification for: (1) text-oriented VQA, where Qwen-VL leads generalist models but trails Pix2Struct-Large on DocVQA (65.1 vs. 76.6 in Table 5)—Pix2Struct is categorized as a generalist model in Table 5 despite having only 1.3B parameters and being specifically designed for structured documents, making it unclear whether this counts as a "generalist model under similar model scales"; (2) Nocaps captioning, where Qwen-VL (121.4) is virtually tied with InstructBLIP (121.9); (3) ScienceQA-Img, where Qwen-VL (67.1) leads generalist models but the LLaVa+GPT-4 combination (92.53) is far ahead—though this uses GPT-4 as a judge and is arguably not a "similar model scale" comparison.

The claim overstates for: Specialist SOTA comparisons. Figure 1 and the abstract emphasize state-of-the-art performance, but Tables 4–6 consistently show specialist models (PALI-X-55B, ONE-PEACE, UNINEXT-H, G-DINO-L) outperforming Qwen-VL by 2–20 points depending on the benchmark. The paper is transparent about these gaps in the tables but the radar chart in Figure 1 only compares against generalist models, not specialists. A reader seeing only the abstract and Figure 1 might incorrectly infer Qwen-VL matches or exceeds all models including specialists. The paper should be read as demonstrating SOTA among generalist models, not SOTA in an absolute sense.

Weakness: No Ablation of the Dual-Task Grounding Setup

The paper trains on referring grounding (text→box) and grounded captioning (box→text) using the same datasets—a symmetric dual-task design that is highlighted as important in Section 3.2. However, there is no ablation comparing this dual-task training against single-task training (e.g., referring grounding only or grounded captioning only). The contribution of the dual-task structure to the strong grounding results (Table 6) is therefore assumed but not demonstrated. A reader cannot determine whether the 9-point lead over Shikra-13B on GRIT refexp (78.22 vs. 69.03) comes from the dual-task format, from the larger grounding dataset, from the higher resolution, from the VL adapter's positional encodings, or from some interaction of all factors. This is a significant missing experiment for a paper that claims innovation in the training methodology.

Weakness: Single-Model Evaluation with No Replication Across Seeds or Initializations

All results are reported as single numbers without confidence intervals, standard deviations, or multiple training runs. For a paper releasing a model series (with checkpoints and code available), the absence of any variance estimates makes it impossible to determine whether the reported advantages (e.g., 85.8 vs. 80.5 on Flickr30K) are statistically reliable or within the noise of training stochasticity. The 500-question Flickr30K karpathy-test split and the varying sizes of VQA test sets mean some comparisons involve relatively small evaluation sets, where variance could be meaningful. At minimum, the paper should report the standard evaluation protocol for each benchmark (e.g., whether test sets have fixed answers or require server submission) and any run-to-run variation observed.

Weakness: Missing OCR Ablation—Synthetic vs. Real Data Contribution

The OCR data category is the largest in Stage 2 (24.8M samples) and comprises both synthetic SynthDOG data and real PDF/HTML data. The paper does not ablate the relative contribution of these two data sources. It is unknown whether the synthetic data alone would suffice for the strong text-reading results in Table 5, or whether the real data is essential. Given that synthetic data generation is one of the paper's methodological contributions, understanding its standalone effectiveness is important for practitioners who might want to replicate the approach without access to Common Crawl PDF/HTML data.

Weakness: Pure-Text Capability Comparison Uses Different Qwen-7B Checkpoint

The paper acknowledges that Qwen-VL uses an "intermediate" Qwen-7B checkpoint rather than the final released version. The final Qwen-7B achieves 58.2 MMLU, 62.2 CMMLU, 63.5 C-Eval (Table 11), while Qwen-VL's LLM initialization achieves 49.9 MMLU and 48.5 C-Eval—a substantial gap of 8.3 MMLU points. The claim that "Qwen-VL do[es] not compromise any pure-text ability" is supported relative to its initialization (50.7 vs. 49.9 on MMLU) but not relative to what the model could have been if built on the final Qwen-7B. A version of Qwen-VL using the final Qwen-7B checkpoint—which presumably existed or could have been created—is not evaluated. The paper's explanation that the models were "developed at a very similar period" is a temporal constraint, not a scientific justification. This is a genuine limitation: the reader cannot know whether Qwen-VL's text capabilities are inherently preserved through multimodal training or whether they simply started from a weaker text baseline.

Weakness: Instruction-Following Evaluation Relies on GPT-4 as Judge

TouchStone scores (Table 7) are generated by GPT-4 evaluating model responses. While this is an established methodology (the TouchStone benchmark itself is defined this way), GPT-4's judgments introduce an uncontrolled variable—GPT-4's own biases, its potential familiarity with Qwen-VL's outputs (if Qwen-VL or similar models were in GPT-4's training data), and its evaluation consistency across different model outputs. The paper does not report human correlation studies for the TouchStone scores or discuss potential evaluation biases. The SEED-Bench and MME results (which use fixed multiple-choice or yes/no formats with ground-truth answers) provide a more objective complement, and Qwen-VL-Chat leads on these as well—but the TouchStone advantage (645.2 vs. 605.4 for the next best model) is a substantial component of the instruction-following claim and rests on a less objective foundation.

Weakness: No Analysis of Failure Modes or Qualitative Error Patterns

Despite evaluating on 15+ benchmarks and reporting strong aggregate performance, the paper provides zero systematic analysis of when or why Qwen-VL fails. There are no error categories, no per-task breakdown of failure types (e.g., does grounding fail more on small objects? Does OCR fail on rotated text? Does VQA fail on compositional questions?), and no qualitative error examples beyond the curated success cases in Figure 2. For a model that claims fine-grained visual understanding, understanding its limitations is as important as understanding its strengths—especially for practitioners deciding whether to deploy it for specific applications. The GRIT data cleaning note (Appendix A.3) about "recursive grounding box labels" hints at known failure modes in the training data, but this is not connected to any evaluation of model behavior.

Missing Experiment: Resolution Scaling Beyond 448×448

The paper ablates window vs. global attention (Appendix E.3) and shows that 896×896 with window attention reaches comparable loss to 448×448 with global attention, but takes 2.5× longer to train. This ablation is framed as justifying the choice of 448×448, but it raises an unanswered question: does higher resolution improve downstream task performance if training cost were not a constraint? For fine-grained tasks like OCR and small-object grounding, 448×448 may still be a bottleneck—the paper does not test whether pushing to 896×896 (or using a multi-scale training approach) would close the remaining gap to specialist models on DocVQA (65.1 vs. 80.0) or AI2D (62.3 vs. 81.2). The window attention results in Figure 8 show the loss is higher for window attention at 448×448, but this doesn't answer whether global attention at 896×896 (if computationally feasible) would improve task performance.

Missing Experiment: Contribution of 2D Positional Encodings in VL Adapter

The paper introduces 2D absolute positional encodings in the cross-attention mechanism of the VL adapter to "mitigate the potential loss of positional details during compression" (Section 2.1). This is presented as important for fine-grained understanding, but there is no ablation comparing adapter performance with and without these positional encodings. The contribution of this architectural choice to grounding accuracy (Table 6) is therefore unquantified. Given that the adapter is the only architectural novelty in Qwen-VL, understanding whether its position-aware design actually matters is essential for evaluating the paper's technical contribution.

Summary: What the Experiments Do and Do Not Demonstrate

What they demonstrate convincingly: (1) A single 9.6B model trained with a three-stage curriculum on diverse multimodal data can simultaneously perform well on image captioning, general VQA, text-oriented VQA, referring expression comprehension, and instruction following—with leading results among open-source generalist models on most benchmarks. (2) The model's grounding capabilities, using only text-based coordinate output, approach specialist detection models within a few points. (3) Multimodal training does not degrade the LLM's pure-text performance relative to its initialization.

What they do not demonstrate: (1) Which specific components of the training recipe (dual-task grounding, positional encodings in the adapter, synthetic OCR data, Stage 2 resolution increase, the freezing/unfreezing schedule) are necessary versus merely sufficient for the results. The paper describes a recipe that works but provides almost no ablation evidence for why it works. (2) Whether the model's performance advantage over baselines is statistically reliable or within training variance. (3) How the model fails—what types of errors it makes, where the remaining gaps to specialist models come from, and what would be needed to close them. (4) Whether the claimed capabilities transfer to deployment scenarios beyond the specific benchmarks evaluated.

6. Limitations and Trade-offs

Limitation 1: No Ablation of the Core Training Recipe—The Components Driving Performance Are Unidentified

The paper presents a three-stage training pipeline with specific design choices—dual-task grounding (referring grounding + grounded captioning on the same datasets), 2D positional encodings in the VL adapter, synthetic OCR data mixed with real PDF/HTML data, a specific freezing/unfreezing schedule across stages, and a resolution increase from 224×224 to 448×448 at Stage 2. The paper treats these as a package deal but provides almost no ablation evidence isolating the contribution of any individual component.

The experiments in Appendix E do ablate two narrow technical choices—the number of learnable queries (Appendix E.2, Figure 7) and window vs. global attention (Appendix E.3, Figure 8)—but these are optimization details, not the core methodological claims. There is no ablation of the dual-task grounding format (the paper never tests referring grounding without grounded captioning, or vice versa), no ablation of the 2D positional encodings in the VL adapter (arguably the only architectural innovation), no ablation comparing synthetic-only vs. real-only OCR data, and no ablation of the Stage 2 resolution increase (no variant trained entirely at 224×224 for comparison).

The consequence is that a practitioner reading this paper cannot determine which parts of the recipe are necessary to reproduce the results and which are incidental. The 9-point lead over Shikra-13B on GRIT refexp (78.22 vs. 69.03, Table 6) could be attributed to the dual-task format, the higher resolution, the larger grounding dataset, the positional encodings, or some interaction—and the paper provides no evidence to distinguish among these explanations. This limits both reproducibility (a re-implementer doesn't know where to invest effort) and scientific understanding (the paper demonstrates that the recipe works but not why it works).

The paper makes no attempt to mitigate this limitation. The ablation appendix (Appendix E) is entirely focused on convergence efficiency and architectural hyperparameters, not on the data and training design choices that constitute the paper's claimed innovations. No future work is suggested regarding ablation studies, and the publicly released models and code (linked in the paper header) do not include ablated variants that would allow the community to perform these comparisons post-hoc.

Limitation 2: The Model's Pure-Text Capability Uses a Weaker LLM Initialization, Inflating the "No Degradation" Claim

The paper claims that multimodal training does not degrade language capabilities, citing Appendix E.4 (Table 11): Qwen-VL achieves 50.7 on MMLU, 49.5 on CMMLU, and 51.1 on C-Eval, compared to its LLM initialization at 49.9 on MMLU and 48.5 on C-Eval—a small improvement. The paper concludes:

"Qwen-VL do not compromise any pure-text ability."

However, the Qwen-7B checkpoint used to initialize Qwen-VL is explicitly described as an intermediate checkpoint, not the final released version. The final released Qwen-7B achieves 58.2 on MMLU, 62.2 on CMMLU, and 63.5 on C-Eval (Table 11)—meaning Qwen-VL's LLM backbone started 8.3 MMLU points behind what was ultimately achievable with the same model family. The paper acknowledges this temporal coincidence:

"The reason why we did not use the final released checkpoint of Qwen-7B is that Qwen-VL and Qwen-7B were developed at a very similar period."

The consequence is that the "no degradation" claim is true relative to a weaker baseline but unverified relative to the stronger final checkpoint. It is unknown whether building Qwen-VL on the final Qwen-7B (58.2 MMLU) would preserve that higher text capability after multimodal training, or whether the three-stage pipeline would cause regression toward the ~50 MMLU level. The claim that multimodal training is safe for language capabilities—a practically important guarantee for practitioners who want to add vision to an existing strong LLM—is therefore supported only for the specific intermediate checkpoint, not for the best available Qwen-7B.

The paper does not mitigate this limitation by training a version of Qwen-VL on the final Qwen-7B checkpoint, nor does it analyze whether the final checkpoint's additional language capability would survive the three-stage pipeline. This is a genuine missing experiment: a practitioner deciding whether to build a multimodal model on their strongest available LLM cannot infer from this paper whether language performance will be preserved or partially lost.

Limitation 3: The Model's Performance Drops After Dialogue Fine-Tuning on Multiple Benchmarks, and the Tradeoff Is Not Analyzed

Comparing Qwen-VL (after Stage 2) with Qwen-VL-Chat (after Stage 3 SFT) across Tables 4–6 reveals a consistent pattern of degradation on task-formatted benchmarks after dialogue fine-tuning, but the paper never acknowledges or analyzes this as a tradeoff:

The drops are individually modest (1–5 points) but systematic—nearly every benchmark shows a decline after SFT, with the largest drops on fine-grained tasks (OCR-VQA: −5.2, AI2D: −4.6, Flickr30K: −4.8). The only exceptions where Qwen-VL-Chat improves are VizWiz (35.2 → 38.9, +3.7) and ChartQA (65.7 → 66.3, +0.6). On referring expression comprehension (Table 6), performance is essentially flat (changes within ±1 point).

The consequence is that practitioners face an unacknowledged tradeoff: the SFT stage that enables dialogue capability (demonstrated by TouchStone, SEED-Bench, MME in Table 7) comes at the cost of modestly reduced performance on task-formatted evaluations. For applications that need both dialogue interaction and high accuracy on specific tasks (e.g., a document QA chatbot that must maintain competitive DocVQA scores), this tradeoff matters but is not characterized.

The paper does not mitigate this limitation. The degradation is visible in the tables but is never discussed in the text. There is no analysis of whether the drops are uniform across difficulty levels (do easy questions degrade less?), whether they can be reduced by mixing more task-formatted data into the SFT stage, or whether a different SFT recipe (e.g., a smaller learning rate, or a subset of the SFT data) could recover the lost performance. The paper treats the SFT stage as purely additive, but the data shows it is substitutive—gaining dialogue at a modest cost to task accuracy—and this substitution goes unexamined.

Limitation 4: No Analysis of Failure Modes, Error Patterns, or Robustness to Distribution Shift

The paper evaluates Qwen-VL on 15+ benchmarks (Tables 4–7) and reports aggregate accuracy metrics on each. However, it provides zero systematic analysis of when, how, or why the model fails—no per-task error taxonomy, no breakdown by question type or difficulty within benchmarks, no qualitative error examples beyond the curated success cases in Figure 2, and no evaluation of robustness to distribution shifts (e.g., does OCR accuracy degrade on rotated text, unusual fonts, low-contrast images?).

The consequence is that a practitioner cannot assess whether the model's performance is uniformly strong or concentrated on easy examples within each benchmark. For instance, Qwen-VL achieves 63.8 on TextVQA (Table 5)—but does it succeed only on large, high-contrast text while failing on small or occluded text? Does the model's grounding accuracy (89.36 on RefCOCO val, Table 6) degrade on small objects, occluded objects, or atypical viewpoints? Without error analysis, these questions are unanswerable, and the aggregate numbers may overstate practical reliability if failures cluster in specific regimes that matter for deployment.

The GRIT data cleaning note (Appendix A.3) hints at known issues in the training data—"many recursive grounding box labels in one caption"—but this is not connected to any evaluation of whether the model reproduces similar errors (e.g., generating overlapping or nested bounding boxes). The paper's qualitative examples (Figure 2) are exclusively successful cases, providing no insight into the model's limitations.

The paper makes no attempt to mitigate this limitation. There is no failure analysis section, no per-category accuracy breakdown, and no adversarial or out-of-distribution evaluation. Future work is suggested only in broad terms (Section 6: "further enhancing Qwen-VL's capabilities"), not in terms of understanding current failure modes.

Limitation 5: The OCR Pipeline Depends on Synthetic Data, but Real-World Text Diversity Is Not Benchmarked

The single largest task category in Stage 2 is OCR data (24.8M samples, Table 3), comprising synthetic SynthDOG data (generated with 41 English fonts and 11 Chinese fonts on COCO backgrounds) and real PDF/HTML data from Common Crawl. The paper demonstrates strong text-reading results on five benchmarks (Table 5), but the benchmarks themselves (TextVQA, DocVQA, ChartQA, AI2D, OCR-VQA) represent specific text-in-the-wild domains and may not cover the full diversity of real-world text that a deployed model would encounter.

The key unexamined question is: what text-reading capabilities does the model fail to acquire from this data mixture? The 41 English fonts and 11 Chinese fonts represent a specific—and limited—sample of typographic diversity. Real-world text appears in handwriting, decorative fonts, heavily stylized logos, low-resolution signage, curved or perspective-warped surfaces, and dozens of non-Latin/non-Chinese scripts that are not represented in the training data. The PDF and HTML data add authentic document layouts but likely skew toward standard office fonts (Arial, Times New Roman) and web-safe fonts. The paper does not evaluate whether the model's OCR capabilities generalize to text styles, scripts, or degradation levels not covered by the training distribution.

The consequence is that a practitioner deploying Qwen-VL for a text-reading application cannot predict whether performance will transfer from the evaluated benchmarks to their specific domain. If their application involves handwritten text, Arabic script, heavily stylized logos, or low-resolution surveillance imagery, the paper provides no evidence about expected performance—and the training data composition (52 fonts total, no handwriting fonts mentioned) suggests performance may degrade substantially.

The paper partially mitigates this by evaluating on diverse text-reading benchmarks (natural images, documents, charts, diagrams, book covers) that collectively cover multiple domains—but this is a test of breadth within the evaluated benchmarks, not a test of generalization to unseen text distributions. No per-font or per-script breakdown is provided that would allow practitioners to assess coverage of their target domain.

Limitation 6: Single Model Family, Single LLM Backbone, Single Scale—Evidence for Generalizability Is Absent

All experiments use one specific configuration: ViT-bigG visual encoder initialized from OpenCLIP, Qwen-7B LLM backbone (intermediate checkpoint), and 9.6B total parameters. The paper demonstrates strong performance on this specific setup, but provides no evidence about how the approach transfers to other model families, other LLM backbones, other visual encoders, or other parameter scales.

The paper's methodological claims—that the three-stage curriculum, dual-task grounding, and token-based spatial representation are effective—are validated only within this single instantiation. A practitioner using a different LLM backbone (e.g., LLaMA, Mistral, or a non-Qwen model) or a different visual encoder (e.g., EVA-CLIP, SigLIP, or a different ViT variant) cannot infer from this paper whether the same training recipe would transfer. The paper's findings might depend on specific properties of Qwen-7B's tokenizer (which handles the <box>/</box> coordinate format), its multilingual pretraining (which provides the Chinese capability), or its specific embedding dimensionality (which constrains the VL adapter design).

The paper acknowledges this implicitly by using only one model but does not discuss the generalizability question. The "Qwen-VL series" branding suggests multiple model sizes, but only the 9.6B configuration is described and evaluated—there is no Qwen-VL-1B or Qwen-VL-13B variant. The scaling behavior of the approach (does performance improve with larger LLMs? does the three-stage curriculum benefit larger models differently?) is entirely unexamined.

The consequence for practitioners is that adopting the Qwen-VL training recipe for a different model family requires substantial risk and re-engineering without guidance from the paper about which design choices are model-specific versus model-agnostic. The public release of models and code partially mitigates this by enabling reproduction on the same architecture, but does not address transferability.

The paper does not discuss this as a limitation. Future work in Section 6 mentions "scaling up the model size" as a direction but does not frame the current single-scale evaluation as a limitation to be addressed.

7. Implications and Future Directions

How This Work Changes the Landscape

Qwen-VL shifts the conversation around open-source LVLMs from architecture design toward training curriculum design as the primary lever for capability improvement. The paper's central empirical claim—that a straightforward architecture (standard ViT, single cross-attention layer, standard decoder-only LLM) can unify holistic image understanding with fine-grained localization and text reading at performance levels competitive with both much larger generalist models and specialized detection architectures—implies that the bottleneck in prior open-source LVLMs was not insufficient architectural novelty but insufficient training data diversity and poorly sequenced training stages.

This is a reframing, not a paradigm shift. The individual components of Qwen-VL are all established—ViT encoders for vision, cross-attention for modality alignment, decoder-only LLMs for text generation, synthetic data for OCR—and the paper does not introduce a fundamentally new type of model. Rather, it demonstrates that a carefully orchestrated assembly of existing components, sequenced in a specific training order with specific data mixtures, can substantially outperform prior assemblies. The contribution is in the recipe, not in new ingredients. For practitioners, this reframing is valuable because it lowers the barrier to building capable LVLMs: invest in data curation and training curriculum design rather than architectural innovation.

The paper resolves a latent contradiction in prior work: whether grounding and holistic understanding are in tension. Kosmos-2 and Shikra added grounding capabilities but did not simultaneously demonstrate state-of-the-art general VQA or captioning. Specialist models like PALI-X achieved strong individual task results through task-specific fine-tuning. Qwen-VL falsifies the assumption that these capabilities trade off against each other—the model simultaneously leads generalist models on both general VQA (Table 4: 58.6 OKVQA vs. Shikra-13B 47.16) and grounding (Table 6: 89.36 RefCOCO vs. Shikra-13B 87.83). The implication is that prior tradeoffs were artifacts of training data composition (not enough grounding data, or grounding data introduced at the wrong training stage) rather than fundamental capacity limits.

The paper also provides evidence that token-based spatial representation is sufficient for grounding at high accuracy—a finding that redirects research attention away from specialized detection architectures toward input-output format design. If spatial coordinates expressed as text tokens can achieve 89.36 on RefCOCO val (within 3 points of specialist G-DINO-L at 90.56, Table 6), then the research question becomes: what is the minimal tokenization scheme needed to represent spatial information for transformer-based models, and can this approach extend to other "structured output" modalities beyond bounding boxes (segmentation masks, keypoints, depth maps)?

Several research directions become more attractive as a result of this work:

• Training curriculum optimization for multimodal models: The paper provides a concrete template (Stage 1: coarse alignment at low resolution, Stage 2: multi-task fine-grained integration at high resolution with LLM unfrozen, Stage 3: dialogue adaptation with visual encoder frozen) but does not ablate the ordering, resolution schedule, or freezing decisions. Understanding which aspects of this curriculum are necessary vs. incidental is now a high-value research question.

• Token-based representations for broader output modalities: If bounding boxes work as text tokens, what about polygons, segmentation masks (as polygon strings), depth maps (as quantized depth strings), or keypoint coordinates? Qwen-VL's success suggests a research program of converting diverse structured outputs to text formats and testing whether LLMs can learn to generate them accurately without modality-specific heads.

• Multilingual and cross-lingual vision-language models: Qwen-VL's strong Chinese performance (Chinese TouchStone: 401.2 vs. VisualGLM 247.1, Table 7; Chinese pretraining data: 22.7% in Stage 1) demonstrates that bilingual vision-language training is feasible without sacrificing English performance. This opens the door to truly multilingual LVLMs covering dozens of languages.

Directions that become relatively less attractive:

• Complex modality-specific adapters: Qwen-VL achieves strong performance with a single cross-attention layer (0.08B parameters, less than 1% of total model size). The success of this minimal adapter suggests that heavy adapter architectures (Q-Former in BLIP-2, perceiver resamplers with multiple layers, learned pooling with multiple attention heads) may be unnecessarily complex for basic vision-language alignment—their additional parameters and architectural complexity may not translate to better downstream performance compared to simpler designs with better training data.

• Detection-specific grounding architectures: The 2–3 point gap between Qwen-VL and specialist detection models (ONE-PEACE at 92.58 vs. Qwen-VL at 89.36 on RefCOCO val, Table 6) suggests that architectural specialization for grounding provides diminishing returns relative to generalist approaches with diverse training data. Future effort may be better spent on improving the training data and curriculum for generalist models rather than designing better detection heads.

Follow-Up Research This Work Enables

Ablation of the dual-task grounding structure against single-task training with identical data volume. The paper trains on referring grounding (text→box) and grounded captioning (box→text) using the same 8.7M samples each (Table 3), creating a symmetric bidirectional mapping. There is no ablation comparing this dual-task setup against: (a) 17.4M samples of referring grounding only, or (b) 17.4M samples of grounded captioning only. A follow-up study would train three variants of Qwen-VL starting from the Stage 1 checkpoint, varying only the Stage 2 grounding task composition while keeping total grounding sample count constant, and evaluate on all nine RefCOCO/RefCOCO+/RefCOCOg/GRIT splits. The key question: does the 9-point lead over Shikra-13B on GRIT refexp (78.22 vs. 69.03, Table 6) come from the dual-task format, or simply from having more grounding data (17.4M total vs. Shikra's reported dataset)? If single-task training with double the data matches dual-task performance, the bidirectional design is incidental. If dual-task substantially outperforms, it suggests that learning the text↔box mapping in both directions creates a richer representational space for spatial reasoning.

Isolating the contribution of 2D positional encodings in the VL adapter to grounding accuracy. The position-aware VL adapter (Section 2.1) is the paper's only architectural innovation—a single cross-attention layer with 2D absolute positional encodings added to query-key pairs. The paper never ablates this design choice. A minimal follow-up would train two Stage 2 variants, identical except for the presence/absence of 2D positional encodings in the adapter, and evaluate on referring expression comprehension benchmarks (Table 6). The hypothesis: removing positional encodings should disproportionately hurt grounding accuracy (especially on RefCOCO test-A/B, which distinguish people vs. objects by location) while having minimal impact on holistic tasks (captioning, general VQA). If the performance gap is small across all tasks, the positional encoding claim is overemphasized. If grounding drops substantially while captioning holds steady, it provides evidence that the adapter's positional awareness is specifically important for spatial reasoning tasks.

Stress-testing OCR generalization to scripts, fonts, and degradation levels not in the training data. The paper's OCR training uses 41 English fonts and 11 Chinese fonts rendered on COCO backgrounds via SynthDOG (Appendix A.4), plus real PDF/HTML data. The five text-oriented VQA benchmarks (Table 5) evaluate specific domains but do not systematically test generalization. A diagnostic follow-up would construct a held-out OCR test set stratified by: (a) font family (serif, sans-serif, handwriting, decorative), (b) script (Latin, Chinese, Arabic, Cyrillic, Devanagari—scripts not in training), (c) text size (large/medium/small relative to image dimensions), (d) orientation (horizontal, rotated, curved, perspective-warped), and (e) image degradation (Gaussian blur, JPEG compression, low contrast). Evaluating Qwen-VL's per-category accuracy on this diagnostic set would reveal whether the model has learned robust text recognition or has overfit to the specific font-rendering distribution in SynthDOG. If performance drops sharply on handwriting scripts (not in the 41 English fonts) or non-Latin/non-Chinese scripts, it identifies a concrete limitation of synthetic data reliance and suggests that font diversity in the OCR generation pipeline is a critical hyperparameter for generalization.

Testing whether the three-stage curriculum's freezing schedule generalizes to other LLM backbones and model scales. The paper uses one LLM (Qwen-7B intermediate checkpoint, 7.7B parameters) and one visual encoder (ViT-bigG from OpenCLIP, 1.9B). It is unknown whether the specific freezing schedule—Stage 1: freeze LLM, train ViT+adapter; Stage 2: unfreeze all; Stage 3: freeze ViT, train adapter+LLM—is optimal for this specific model combination or generalizes as a principle. A systematic replication would apply the same three-stage recipe to: (a) LLaMA-7B or Mistral-7B as the LLM backbone, (b) EVA-CLIP or SigLIP as the visual encoder, (c) a larger LLM (13B or 70B parameters). The key measurement: does catastrophic forgetting of language capabilities (measured by MMLU/C-Eval before vs. after multimodal training, following Appendix E.4) vary across LLM families under the same curriculum, and does the optimal freezing schedule shift with model scale? If the schedule transfers robustly, it becomes a reusable recipe; if different LLMs require different Stage 1/2/3 boundaries, it suggests the freezing decisions are architecture-specific and need per-model tuning.

Quantifying the pretraining-inference tradeoff for fine-grained visual understanding. The paper compares Qwen-VL to larger models (Flamingo-80B, PALI-X-55B) on accuracy but never accounts for the total computational cost of achieving that accuracy, including pretraining. Qwen-VL ingests 1.5 billion image-text samples in Stage 1 alone (50,000 steps × 30,720 batch size), plus 76.9M multi-task samples in Stage 2. A FLOPs-matched comparison—analogous to the scaling law analyses in Hoffmann et al. (2022) for text—would construct a plot of RefCOCO accuracy vs. total training FLOPs for Qwen-VL at multiple scales (by training smaller variants: e.g., ViT-B + Qwen-1.8B, ViT-L + Qwen-7B, ViT-bigG + Qwen-7B) and compare against the scaling curves of specialist models (G-DINO, UNINEXT, ONE-PEACE) and other generalist models (Shikra, Kosmos-2). This would answer the question: for a given FLOPs budget, is it more efficient to train a generalist model with diverse tasks or a specialist model focused on grounding? The paper's current results suggest generalist models can approach specialist performance (89.36 vs. 92.58 on RefCOCO val), but the total training cost may be substantially higher for the generalist—a FLOPs-matched analysis would quantify this tradeoff.

Practical Applications and Downstream Use Cases

Document digitization and question answering for scanned forms and reports. Qwen-VL's strong DocVQA performance (65.1 ANLS, Table 5) and OCR-VQA performance (75.7 EM) make it directly applicable to automated document processing: extracting structured information from scanned invoices, contracts, academic papers, and government forms. A deployment scenario would feed scanned document pages to Qwen-VL with queries like "What is the total amount on this invoice?" or "Extract the contract date and parties involved," and the model would both read the relevant text (using its OCR capability) and answer the question (using its VQA capability) in a single forward pass. The advantage over a pipeline approach (separate OCR engine + separate QA model) is reduced engineering complexity and the ability to handle text that requires visual context for disambiguation (e.g., text in tables, text near logos). The 65.1 DocVQA score—while trailing Pix2Struct-Large's 76.6—is sufficient for many semi-automated workflows where a human reviews the model's extracted information. The model's grounding capability (89.36 RefCOCO, Table 6) adds the ability to highlight exactly where in the document each extracted piece of information came from, providing interpretability for the human reviewer.

Multilingual visual assistance for tourism, navigation, and accessibility. Qwen-VL's bilingual training (English and Chinese, with 22.7% Chinese pretraining data in Stage 1) and text-reading capabilities (63.8 TextVQA, Table 5) support a real-world application: a mobile app that photographs signage, menus, or product labels in one language and answers questions about them in another. A Chinese-speaking tourist in an English-speaking country could photograph a restaurant menu and ask (in Chinese) "What vegetarian options are available, and how much do they cost?" Qwen-VL would read the English text, understand the question in Chinese, and respond in Chinese—all within a single model without separate translation and OCR components. The zero-shot VizWiz performance (38.9 for Qwen-VL-Chat, Table 4) is relevant here: VizWiz contains photos taken by visually impaired users, often with poor framing, blur, and challenging lighting. Qwen-VL's relative strength on this benchmark (leading all generalist models) suggests robustness to non-ideal image conditions common in mobile photography scenarios. The TouchStone Chinese score (401.2 vs. VisualGLM's 247.1, Table 7) provides additional evidence for strong Chinese-language instruction following in this use case.

Fine-grained visual inspection and inventory management. Qwen-VL's grounding ability (89.36 RefCOCO val, 88.55 for the Chat variant, Table 6) combined with its referring expression comprehension enables applications where a user needs to identify and count specific objects in images of shelves, warehouses, or retail displays. A quality control inspector could upload an image of a product shelf and query "Show me all products with damaged packaging" or "Count how many items are missing from the second shelf." The model would output bounding boxes around relevant regions (using the token-based coordinate format) and provide a textual answer. The GRIT refexp score of 78.22 (Table 6)—9 points above Shikra-13B—is particularly relevant for this scenario: GRIT contains diverse, naturalistic referring expressions similar to what a human inspector might use conversationally rather than the more template-like expressions in RefCOCO. The 9.6B parameter count and public model release mean this application can run on commodity GPU hardware without requiring the datacenter-scale infrastructure needed for larger proprietary models.

When to Prefer This Method

The paper does not explicitly articulate a decision rule for preferring Qwen-VL over named alternative approaches (e.g., "use Qwen-VL instead of Shikra when..." or "prefer the three-stage curriculum over single-stage training when..."). The comparison is made implicitly through benchmark results rather than through explicit tradeoff analysis. The paper's positioning—that Qwen-VL unifies capabilities that prior models handled separately—suggests a preference when application requirements span multiple capability types (general VQA + grounding + OCR + dialogue) that would otherwise require multiple specialized models, but this is not stated as an explicit decision rule with conditions.