Multimodal2024~88 minarXiv:2403.05525

DeepSeek-VL: Towards Real-World Vision-Language Understanding

score 9Haoyu Lu, Wen Liu, Bo Zhang +12DeepSeek↑ 49★ 4,076Mar 8, 2024cs.AIhttps://github.com/deepseek-ai/deepseek-vl

🎯 Pitch

Most open-source vision-language models crumble on messy, real-world inputs like screenshots and PDFs, despite decent benchmark scores. DeepSeek-VL introduces a hybrid vision encoder and a carefully balanced joint training strategy that lets it match GPT-4V on several perceptual tasks while keeping its language skills intact—and it's fully open-source.

1. Executive Summary

This paper introduces DeepSeek-VL, an open-source Vision-Language Model family (1.3B and 7B parameters) designed to bridge the gap between open-source multimodal models and proprietary systems in real-world applications. The model is evaluated across a wide range of public multimodal benchmarks (MMBench, MMMU, SeedBench, MathVista, OCRBench) as well as language benchmarks (MMLU, HellaSwag, GSM8K), using the DeepSeek-LLM as its language backbone. DeepSeek-VL's design centers on three named mechanisms: a hybrid vision encoder that fuses a SigLIP-L semantic encoder at 384×384 resolution with a SAM-B low-level feature encoder at 1024×1024 resolution within a fixed 576-token budget (preserving both high-level semantics and fine-grained details like OCR text), a joint language-multimodal training strategy that maintains roughly a 7:3 ratio of text-to-multimodal data during pretraining to prevent language capability degradation (recovering language performance that would otherwise collapse under multimodal-only training), and a modality warm-up strategy that gradually decreases the language data ratio from 1.0 to the target ratio at training onset to stabilize the transition from pure language to multimodal learning. DeepSeek-VL-7B surpasses all open-source models of similar size on most benchmarks—achieving 70.4 on SeedBench versus GPT-4V's 71.6 and 36.1 on MathVista—while maintaining language performance comparable to the original DeepSeek-7B (68.4 vs. 68.5 on HellaSwag), establishing that joint vision-language pretraining can preserve and even enhance linguistic capabilities when text data constitutes at least 70% of the training mixture.

2. Context and Motivation

The Core Problem: Open-Source VLMs Don't Work Well in the Real World

By early 2024, the vision-language model (VLM) landscape had split into two tiers. At the top sat proprietary systems like GPT-4V and Gemini, which demonstrated remarkably capable multimodal understanding across diverse real-world scenarios—reading text from messy web screenshots, interpreting complex charts, following multi-step reasoning instructions grounded in images, and even handling embodied intelligence tasks. At the bottom sat a rapidly growing collection of open-source models (LLaVA, InstructBLIP, Qwen-VL, CogVLM, Fuyu, and others) that posted increasingly competitive numbers on academic benchmarks but, as the paper argues, fell dramatically short when deployed in actual user-facing applications.

The fundamental question DeepSeek-VL sets out to answer is: why does this benchmark-to-reality gap exist for open-source VLMs, and what architectural and training decisions can close it? This is not merely a "leaderboard improvement" paper. It targets a qualitative deficiency—user experience in practical settings—that existing benchmarks fail to capture, and it proposes a holistic remedy spanning data, architecture, and training strategy.

Why the Gap Matters

The paper's motivation is practical and urgent. Vision-language models were transitioning from research curiosities to deployment-ready assistants. Users were asking these systems to read invoices, debug UI code from screenshots, explain textbook diagrams, and interpret medical images—tasks that demand a combination of high-resolution visual acuity, robust language understanding, and domain knowledge. If open-source models continue to excel only on curated academic datasets while failing on the messy inputs real users provide, they cannot serve as viable alternatives to proprietary APIs. This has downstream consequences for privacy-sensitive applications (where data cannot leave the user's device), for cost-sensitive deployments (where per-query API pricing is prohibitive), and for research communities that need inspectable, modifiable foundation models.

The paper identifies four specific reasons why existing open-source VLMs underperform in practice, and each becomes a design target for DeepSeek-VL:

Gap 1: Underinvestment in Multimodal Pretraining

The paper argues that many open-source VLM pipelines over-allocate resources to instruction tuning while shortchanging pretraining. The typical recipe—take a pre-trained vision encoder (often CLIP-based), attach a lightweight adapter to a frozen LLM, and fine-tune on a collection of academic VQA datasets—is computationally efficient and yields good benchmark scores. But this recipe, the authors contend, produces models with shallow world knowledge.

The experience from training powerful language models (the authors cite DeepSeek-LLM, GPT-4, and LLaMA) demonstrates that extensive pretraining on broad data distributions is the foundation of general intelligence. Simple instruction tuning atop a barely-pretrained multimodal backbone cannot substitute for that foundation. Models need exposure to millions of diverse image-text pairs—interleaved web documents, textbooks with diagrams, OCR-rich PDFs, charts with structured captions—before they can develop the cross-modal associations that robust real-world understanding demands. The paper positions its extensive, multi-source pretraining dataset (detailed in Table 1) as a direct response to this underinvestment.

Gap 2: Instruction Tuning Data Mismatched to Real-World Usage

A second common practice the paper criticizes is the amalgamation of existing academic datasets during instruction tuning. The logic is straightforward: combine ScienceQA, TextVQA, OCR-VQA, ChartQA, RefCOCO, and a dozen other benchmarks into a single fine-tuning mixture, and the resulting model will perform well on each of them. The problem, the authors observe, is that this approach optimizes for the wrong distribution. Academic benchmarks, by design, have clean inputs, well-defined tasks, and limited scope. Real users ask messy, open-ended, multi-step questions about images the model has never seen—web pages with advertisements and navigation bars, handwritten notes with coffee stains, dashcam footage in ambiguous lighting.

The paper explicitly states:

"A common practice is to amalgamate various academic datasets during instruction tuning. While such an approach may yield good benchmark results, it often falls short in providing an authentic real-world usage experience."

To counter this, the authors construct an entirely novel instruction-tuning dataset grounded in a use case taxonomy derived from manually collected GPT-4V and Gemini test cases found on the internet. The implication is that what users actually try to do with multimodal models is the right distribution to optimize for, not what benchmark authors thought would be interesting to measure three years ago. This taxonomy (Table 3) spans Recognition, Conversion, Analysis, Commonsense Reasoning, Logical Reasoning, Evaluation, Multi-graph, and Safety categories, with granular subcategories like "Currency Recognition," "UI to Code," "Defect Detection," "Humor Reasoning," and "Prompt Injection." The dataset construction methodology is itself a contribution: rather than relying on existing labels, the authors reverse-engineer real user behavior from public demonstrations of proprietary models, then build prompts and collect/capture images that mirror those scenarios.

Gap 3: Low-Resolution Vision Encoders Cannot Handle Real-World Detail

Most open-source VLMs at the time operated at image resolutions between 224×224 and 448×448 pixels. This was a direct inheritance from the CLIP family of vision encoders, which were trained at these resolutions for semantic image-text alignment tasks. For describing the gist of a photograph ("a dog sitting on a couch"), 336×336 is often sufficient. For reading the text on a restaurant receipt, identifying a specific button in a UI screenshot, or counting objects in a crowded scene, it is catastrophically insufficient.

The paper identifies this as a structural limitation, not a superficial one. The problem is specifically what Tong et al. (2024) call "CLIP-blind pairs"—visually distinct images that CLIP-based encoders embed similarly because the differences are low-level (text content, spatial arrangement, fine patterns) rather than high-level semantic. A screenshot of a login page and a screenshot of a checkout page may share similar semantic features ("webpage with form fields") while containing entirely different functional text. A graph with an upward-sloping trendline and one with a downward slope may be visually near-identical except for the direction of a single curve.

Existing workarounds included:

Dynamic high-resolution approaches (e.g., splitting images into tiles, processing each separately) that increase token counts quadratically, making them expensive for multi-turn dialogue and interleaved text-image contexts.
Trainable resolution upsamplers that add parameters but still operate on features extracted at low resolution, meaning fine details lost in the initial encoding cannot be recovered.

The paper's diagnosis is that low-resolution encoding is a fundamental bottleneck for OCR, visual grounding, and tiny object recognition—exactly the capabilities most important for real-world applications like document understanding, UI interpretation, and chart reading.

Gap 4: Multimodal Training Degrades Language Capability

This is perhaps the most subtle but consequential gap the paper identifies. When an LLM is trained on multimodal data, its performance on pure language tasks tends to catastrophically degrade. The paper's own experiments show this clearly (Figure 4): training with 100% multimodal data causes a severe drop in language benchmarks. This is not a theoretical concern—it directly contradicts the goal of a generalist model that performs well on both vision and language tasks.

The paper hypothesizes two mechanisms behind this degradation:

Data distribution divergence: Most multimodal corpora (image captions, VQA pairs, chart annotations) are syntactically simpler and less diverse than the text corpora used for LLM pretraining. Training exclusively on simpler text patterns causes the model to lose its capacity for complex linguistic reasoning.
Competitive dynamics between modalities: The vision and language modalities appear to compete for model capacity. Parameters that were previously dedicated to language representations get reallocated to cross-modal processing. This is not merely catastrophic forgetting in the continual learning sense—it is a zero-sum resource allocation problem within a fixed-capacity model.

Prior work largely ignored this problem. Some VLMs (like LLaVA) used frozen LLMs to sidestep language degradation entirely, but at the cost of limited multimodal integration—the LLM never learns to deeply adapt its internal representations to visual inputs. Others accepted the degradation as inevitable, producing models that scored well on VLM benchmarks but did so at the expense of language ability that was never separately evaluated. The paper's explicit tracking of language benchmarks alongside multimodal benchmarks throughout training (Figures 4, 8, 9; Table 7) is itself a methodological contribution.

How DeepSeek-VL Positions Itself

The paper positions DeepSeek-VL not as a single-idea improvement but as a system-level response to all four gaps simultaneously:

Against Gap 1 (underinvestment in pretraining): A massive, diverse pretraining dataset spanning interleaved image-text, captions, tables/charts, web code, document OCR, scene text OCR, and text-only corpora, with a three-stage training pipeline that includes a dedicated joint vision-language pretraining phase (Stage 2) where the LLM is fully trainable.
Against Gap 2 (benchmark-optimized instruction data): A use-case-taxonomy-driven in-house SFT dataset constructed from real-world GPT-4V/Gemini test cases, ensuring the model is optimized for actual user queries rather than academic dataset distributions.
Against Gap 3 (low resolution): A hybrid vision encoder that combines SigLIP-L at 384×384 for semantic features with SAM-B at 1024×1024 for low-level details, producing a fixed 576-token representation that handles high resolution without the token explosion of tiling approaches.
Against Gap 4 (language degradation): A joint language-multimodal training strategy maintaining at least 70% text data, combined with a modality warm-up that gradually introduces multimodal data to stabilize training, demonstrated to preserve language performance within 0.1 points on HellaSwag while substantially improving multimodal capabilities.

The paper also addresses a meta-challenge: how do you iterate on training strategies at 1.3B scale when small models produce noisy, unreliable signals? This is a practical research engineering problem. The authors note that 1.3B models exhibit "considerable fluctuations in generative metrics" during Stage 2 training, making it impossible to determine whether a strategy change is helping or hurting. Their solution—switching from generation-based to perplexity-based evaluation for small models, and mixing a small proportion of SFT data into pretraining—is a methodological contribution that enables the scaling-law experiments that validate the larger model's design.

Finally, the paper explicitly frames itself as a step toward closing the open-source/proprietary gap in user experience, not just benchmark scores. The human evaluation in Section 4.3, based on the same taxonomy used for data construction, measures performance across real-world task categories. The GPT-4V-as-judge evaluation (Figure 7) compares DeepSeek-VL's responses head-to-head against other models on these practical tasks. This evaluation philosophy—measure what users actually need, not just what benchmarks happen to test—is central to the paper's identity and distinguishes it from the majority of VLM papers that report only academic metrics.

3. Technical Approach

3.1 Reader Orientation

This is primarily a systems design paper whose core idea is that closing the gap between open-source VLMs and proprietary models in real-world applications requires simultaneous attention to three interconnected dimensions — data construction, model architecture, and training strategy — rather than optimizing any single component in isolation. The system being built is a vision-language chatbot (DeepSeek-VL-Chat) that takes interleaved image-text inputs and produces free-form text responses. It solves the problem of real-world multimodal understanding (reading text from messy screenshots, interpreting charts, reasoning about diagrams, recognizing objects in complex scenes) by combining a hybrid high-resolution vision encoder, a pretrained language model backbone, and a carefully orchestrated three-stage training pipeline that preserves language capabilities while developing robust visual grounding.

3.2 Big-Picture Architecture (Diagram in Words)

DeepSeek-VL consists of five major components connected in a feedforward pipeline:

Hybrid Vision Encoder (two parallel sub-encoders): Takes a single input image and produces a fixed-length visual token sequence. A SigLIP-L encoder processes the image at low resolution (384×384) to extract high-level semantic features (e.g., object categories, scene types). In parallel, a SAM-B (Segment Anything Model, base variant) encoder processes the same image at high resolution (1024×1024) to capture low-level spatial details (e.g., text characters, fine edges, small objects). The two encoder outputs are then merged into a unified representation.
Vision-Language Adaptor: A two-layer hybrid MLP that projects the fused visual features from the hybrid encoder into the same embedding space used by the language model. It processes the SigLIP and SAM features through separate single-layer MLPs first, concatenates them, then applies a final shared MLP layer. This adaptor is the "translation layer" that allows the language model to consume visual information.
Language Model (DeepSeek-LLM): A decoder-only transformer based on the LLaMA architecture (Pre-Norm with RMSNorm, SwiGLU FFN activation with intermediate dimension $\frac{8}{3}d_{model}$ , Rotary Position Embedding). It receives the sequence of visual tokens followed by text tokens (user prompt) and autoregressively generates text responses. The LLM is the only component that generates output; all other components are input processors.
Training Pipeline (three stages): Controls which components are frozen vs. trainable at each phase. Stage 1 trains only the VL Adaptor (both encoders and LLM frozen). Stage 2 jointly trains the VL Adaptor and LLM (encoders frozen), with carefully controlled ratios of text-to-multimodal data. Stage 3 fine-tunes the LLM, VL Adaptor, and SigLIP-L encoder (SAM-B remains frozen due to GPU memory constraints).
Data Pipeline (not a runtime component; feeds training): Supplies the pretraining mixture in Stage 2 (70% text-only data from DeepSeek-LLM's corpus, 30% multimodal data spanning interleaved image-text, captions, tables/charts, web code, document OCR, scene text OCR) and the SFT mixture in Stage 3 (approximately 48% text-only SFT, 52% multimodal SFT including in-house taxonomy-based data and public GPT-4V-generated datasets).

Information flows at inference: User provides an image + text prompt → image is resized and fed to both SigLIP-L (384×384) and SAM-B (1024×1024) encoders in parallel → each produces feature maps → VL Adaptor processes and fuses them into 576 visual tokens with 2048 dimensions each → visual tokens are prepended to the text token sequence → LLM autoregressively generates a response.

3.3 Roadmap for the Deep Dive

First, the hybrid vision encoder design (Section 3.1), which is the most architecturally novel component. We need to understand exactly how SigLIP and SAM features are extracted, fused, and compressed into a fixed 576-token budget, because this is the solution to the low-resolution bottleneck identified in Gap 3.
Second, the VL Adaptor design, including the ablation-backed decision to use separate-then-shared MLPs and embedding-level concatenation rather than sequence-level concatenation, because this interface layer controls how visual information enters the LLM.
Third, the three-stage training pipeline (Section 3.2), walking through each stage's objectives, trainable/frozen components, data used, and design rationale. This is where the paper's core strategic innovations live (joint training, modality ratio, warm-up).
Fourth, the modality warm-up strategy and the joint language-multimodal training experiments (Figures 4, 8, 9) that establish the 7:3 text-to-multimodal ratio as the operating point that preserves language while developing vision.
Fifth, the scaling methodology — how experiments on the 1.3B model informed 7B design decisions, including the switch to perplexity-based evaluation to stabilize small-model metrics.
Sixth, the hyperparameters and infrastructure choices (Table 4), which matter for reproducibility.

3.4 Detailed, Sentence-Based Technical Breakdown

Hybrid Vision Encoder: Dual-Resolution, Dual-Purpose Encoding

The hybrid vision encoder is the architectural centerpiece of DeepSeek-VL. It addresses the tension between two conflicting requirements: the model must capture high-level semantic understanding (what objects are present, what scene this is) and low-level visual details (what text is on this sign, what is the exact layout of this UI) — and it must do so within a fixed token budget that keeps inference tractable for multi-turn dialogue.

The two encoder choices and their individual limitations. The paper selects SigLIP-L (a CLIP-family model trained with a sigmoid loss for image-text alignment) as the semantic encoder. SigLIP takes a 384×384 resolution input and produces a feature map. Its strength is that it has been explicitly trained to align visual representations with language, making its output features directly interpretable by the LLM after projection. Its weakness — and the reason a second encoder is needed — is twofold: (1) 384×384 resolution loses fine-grained details like small text or thin lines, and (2) CLIP-family encoders exhibit what Tong et al. (2024) call "CLIP-blind pairs," where visually distinct images with similar high-level semantics get embedded similarly. A screenshot of a login form and a screenshot of a registration form may produce nearly identical CLIP features despite containing completely different text fields and buttons.

For the high-resolution detail encoder, the paper selects SAM-B — the base-size vision encoder from Meta's Segment Anything Model. SAM-B is a ViTDet-based architecture (Li et al., 2022), pre-trained with a self-supervised objective (not contrastive language alignment) on instance segmentation tasks. It takes a 1024×1024 input and produces a $64 \times 64 \times 256$ feature map. The critical insight is that SAM-B was trained to capture fine spatial boundaries and subtle visual differences needed for precise segmentation masks, making it naturally suited for OCR, tiny object recognition, and visual grounding — exactly the capabilities missing from SigLIP alone. Unlike SigLIP, SAM-B has no text alignment, so it cannot provide semantic interpretation; its role is purely to provide high-resolution structural information.

Why two encoders rather than one high-resolution encoder? A natural alternative is to simply replace SigLIP with a single encoder that handles high resolution and is text-aligned. The paper does not explore this option explicitly, but the motivation is implied: no existing encoder at the time combined both properties. CLIP-family models (SigLIP, EVA-CLIP, OpenCLIP) are limited to moderate resolutions (336–512 pixels) due to their training data resolution. Vision-only self-supervised models (SAM, DINOv2, ViTDet) handle high resolution but lack language alignment. The hybrid approach lets each encoder do what it's optimized for, with the VL Adaptor learning to fuse their complementary signals.

The fixed token budget constraint. The paper explicitly states that the hybrid encoder produces 576 visual tokens with 2048 dimensions each. This is a deliberate engineering choice: a 1024×1024 image processed naively through a ViT with patch size 16 would produce $(1024/16)^2 = 4096$ tokens, which is prohibitively expensive for the LLM's self-attention (which scales quadratically in sequence length). The 576-token budget is achieved through a combination of:

SigLIP-L output: At 384×384 resolution with patch size 24, SigLIP produces a $16 \times 16 = 256$ -patch feature map. With a CLS token, this would be 257 tokens. The paper describes it as producing a $576 \times 1024$ feature map, indicating there is additional processing (possibly an MLP projection or feature reshaping) between the raw SigLIP output and the concatenation step. Specifically, per Section 3.1, the SigLIP-L encoder "results in a feature map of size $576 \times 1024$ " — meaning 576 spatial positions, each with 1024 feature dimensions.
SAM-B processing chain: The raw SAM-B output is a $64 \times 64 \times 256$ feature map. This is first interpolated (upsampled) to $96 \times 96 \times 256$ , then passed through two convolutional layers with stride 2. Each stride-2 convolution halves the spatial dimensions, so $96 \times 96 \rightarrow 48 \times 48$ (first conv) $\rightarrow 24 \times 24$ (second conv), with the channel dimension expanded to 1024. The resulting $24 \times 24 \times 1024$ map is finally reshaped to $576 \times 1024$ , matching SigLIP's output shape.
Concatenation and final projection: The SigLIP features ( $576 \times 1024$ ) and processed SAM features ( $576 \times 1024$ ) are concatenated along the feature dimension to produce $576 \times 2048$ . These 576 tokens, each with 2048 dimensions, then pass through a GeLU activation and an embedding layer to enter the LLM's input space.

Why 576 tokens? The number emerges from the specific encoder resolutions and the processing chain. SigLIP-L at 384×384 with patch size 24 naturally produces $16 \times 16$ patches, but the paper reports $576$ positions rather than $256$ . This discrepancy suggests that SigLIP-L in DeepSeek-VL may use a different patch configuration or an additional feature projection. Regardless of the exact derivation, the key design principle is: both encoder outputs are normalized to the same number of spatial positions (576) before concatenation, enabling straight feature-dimension concatenation rather than the more expensive sequence-length concatenation that would double the token count to 1152. This keeps the visual token budget fixed at 576 regardless of input image resolution (since both encoders resize to fixed resolutions), making inference costs predictable and manageable for multi-turn conversations.

The resolution trade-off. The paper states that 1024×1024 "suffices in most use cases." This is an empirical claim: going higher (e.g., 2048×2048) would improve OCR of very tiny text but would require a heavier SAM encoder or more aggressive downsampling, losing the very details being preserved. The chosen resolution represents a practical sweet spot between visual acuity and computational cost.

Proof that the hybrid encoder works: ablation evidence. Figure 10 compares training loss curves for different encoder configurations: SigLIP alone, SAM alone, and SigLIP+SAM (hybrid). The hybrid configuration achieves lower training loss. Table 10 further shows that the embedding concatenation approach consistently outperforms alternatives (e.g., "Token Pooling - W" which compresses SAM features along width before concatenation, achieving only 55.5 average score vs. 55.9 for embedding concatenation with hybrid MLP). The performance gaps are modest in absolute terms (fractions of a percentage point on average benchmark scores) but consistent across most individual benchmarks.

Vision-Language Adaptor: The Two-Layer Hybrid MLP Interface

The VL Adaptor is the component that translates visual features from the encoder space into the LLM's token embedding space. Despite its small size (a two-layer MLP), its design has measurable effects on final performance, and the paper conducts targeted ablations (Table 10) to justify the chosen architecture.

Architecture details. The adaptor employs what the paper calls a "two-layer hybrid MLP." The term "hybrid" here refers to the fact that high-resolution (SAM) features and low-resolution (SigLIP) features are processed through separate single-layer MLPs first, then concatenated, then passed through a shared second MLP layer. Specifically:

Separate first layers: The SAM-derived $576 \times 1024$ features pass through one single-layer MLP; the SigLIP-derived $576 \times 1024$ features pass through a different single-layer MLP. This allows each projector to learn encoder-specific transformations, handling the different statistical distributions of features from a text-aligned encoder (SigLIP) versus a self-supervised encoder (SAM).
Concatenation: The two MLP outputs are concatenated along the feature dimension, yielding $576 \times 2048$ (or whatever intermediate dimension the separate MLPs project to).
Shared second layer: The concatenated features pass through a second MLP that projects into the LLM's embedding dimension, making them compatible with the language token embeddings.

Why separate-then-shared? The ablation in Table 10 compares three MLP configurations: "Hybrid MLP" (separate first layers + shared second layer, described above), "Shared MLP" (both encoders processed through the same MLP), and "Separate MLP" (completely independent MLPs for each encoder with no shared processing). Hybrid MLP achieves the best average score (55.9) across evaluated benchmarks. The "Separate MLP" configuration dramatically underperforms on MMB (57.5 vs. 62.0 for Shared MLP on that specific metric), suggesting that some degree of cross-encoder feature interaction is necessary for effective fusion. The "Shared MLP" performs well on MMB (62.0) but worse on other metrics, suggesting that encoder-specific processing in the first layer helps handle the distributional differences between SigLIP and SAM features.

Sequence concatenation vs. embedding concatenation. The top portion of Table 10 compares two approaches for combining visual features. "Sequence concatenation" means the SAM and SigLIP feature sequences are concatenated along the token-length dimension — e.g., if each produces 576 tokens, the result would be 1152 tokens. The "Token Pooling - W" and "Token Pooling - H" variants try to reduce this by pooling SAM features before concatenation, but both underperform embedding concatenation (55.5 and 54.2 average scores respectively vs. 55.9). "Embedding concatenation" means the two 576-token sequences are merged along the feature dimension, producing 576 tokens with double the feature dimensionality. This keeps the token count at 576, which is important for computational efficiency (LLM self-attention cost is quadratic in sequence length). The ablation confirms that embedding concatenation is not just more efficient but also more effective.

Stage 1: Training the Vision-Language Adaptor (Warmup)

Objective. Stage 1 exists solely to teach the VL Adaptor to map visual features into the LLM's embedding space in a way that the LLM (which remains frozen) can interpret. The paper describes this as establishing "a conceptual link between visual and linguistic elements within the embedding space, thereby facilitating the comprehensive understanding of depicted entities in the images by the Large Language Model."

Frozen vs. trainable components. Both the hybrid vision encoder (SigLIP-L, SAM-B) and the LLM are kept frozen. Only the VL Adaptor parameters are updated. This is consistent with prior work (LLaVA, Instruct-BLIP), which uses this stage as a lightweight way to align vision and language spaces before committing the expensive LLM parameters to multimodal training.

Training data. Stage 1 uses approximately 1.25 million image-text paired captions from ShareGPT4V combined with 2.5 million Document OCR rendering pairs. The OCR pairs are the in-house constructed dataset from 1.4 million arXiv articles (PDFs with associated LaTeX source), 860K English and 180K Chinese e-books, and K-12 education exam questions — all rendered into paired image-and-text format using HTML templates. The total is roughly 3.75 million image-text pairs.

Why so few data and why does scaling fail? This is one of the paper's more interesting negative results. Table 8 shows an experiment where they expanded the Stage 1 dataset and observed that performance decreased rather than improved. The authors' interpretation is:

"This implies that the projector's capacity is inherently constrained, rendering it incapable of capturing the extensive knowledge necessary for multimodal tasks."

In other words, the VL Adaptor (a two-layer MLP with limited parameters) saturates quickly. It can learn a reasonable mapping from visual features to token embeddings with a few million examples, but beyond that, its small capacity becomes the bottleneck — it literally cannot store more cross-modal associations. The implication is that you cannot shortcut multimodal training by putting more data through a frozen-LLM stage; eventually, you must unfreeze the LLM and let it learn multimodal representations in its own parameters.

Why include this stage at all? Table 9 shows the ablation: training with stages 1+2+3 performs better than stages 2+3 alone (which skip the adaptor warmup). The gap is modest but consistent: the warmup provides a better initialization for joint pretraining, reducing the gradient shock when the LLM first encounters visual tokens. This is a common pattern in transfer learning — a brief alignment phase before full joint training stabilizes optimization.

Stage 2: Joint Vision-Language Pretraining (The Core Innovation)

Stage 2 is where the paper's most significant training strategy innovations reside. This is the phase that transitions the LLM from a pure language model to a multimodal model while preserving — and in some cases improving — its language capabilities.

What is trainable. The LLM and the VL Adaptor are jointly optimized. The vision encoders (SigLIP-L and SAM-B) remain frozen. The paper does not explicitly state whether the LLM's token embedding layer is trained, but typically in such setups, all LLM parameters including embeddings are updated.

The catastrophic forgetting problem (Figure 4). The paper's key empirical finding driving the Stage 2 design is the phenomenon shown in Figure 4: when the LLM is trained exclusively on multimodal data (the 100% multimodal, 0% language condition), multimodal benchmark performance gradually improves, but language benchmark performance collapses dramatically. This is not a gradual decline — the figure shows a steep drop in language metrics almost immediately upon switching to multimodal-only training. The authors characterize this as "a competitive dynamic between multimodal and linguistic modalities" and attribute it to two causes:

Data distribution simplicity: Multimodal training corpora (captions, VQA pairs) use simpler language than the diverse text corpora used for LLM pretraining. The model adapts to this simpler distribution, losing its capacity for complex linguistic structures.
Parameter competition: Both modalities compete for the same LLM parameters. Weights that were optimized for language representation get overwritten by gradients from multimodal objectives. Since the LLM has finite capacity, this is a zero-sum resource allocation.

The joint training solution (Figure 4, sweep). The paper's solution is to mix text-only data into the multimodal training batches. Figure 4 shows experiments on a 1.3B model sweeping the multimodal:language ratio from 100%:0% through 70%:30%, 50%:50%, 30%:70%, down to 0%:100%. The key conclusions:

70% language, 30% multimodal (approximately a 7:3 ratio) achieves the best balance. Language performance is nearly preserved (comparable to the 0% multimodal baseline), while multimodal performance improves substantially over the language-only baseline.
Higher multimodal ratios (>50%) cause significant language degradation with diminishing multimodal returns.
The paper explicitly states: "the performance of different modalities is strongly correlated with their respective proportions in the training dataset, substantiating the competitive relationship between the two modalities."

This ratio is not tuned per-batch but rather per-dataset-sampling: within the pretraining data mixture, text-only data constitutes 70% of the total corpus (seen in Table 1, where the "Text-only corpus" row shows 70.0% ratio).

The modality warm-up strategy (Figure 9). Even with a fixed 7:3 ratio, the transition from pure language to joint training is abrupt. The model's weights are initialized from pure language pretraining; suddenly introducing 30% multimodal data can still cause an initial shock. The paper proposes a simple but effective modality warm-up: at the beginning of Stage 2, the language data ratio is set to 1.0 (100% text, 0% multimodal), and then it is gradually decreased to the target ratio (0.7) over the early part of training.

Figure 9 compares the warm-up strategy against a no-warm-up baseline. The results show that warm-up:

Prevents "a significant decline in language capabilities at the beginning of training"
Yields "comparatively superior outcomes in the final phases for both the language and multimodal domains"
The improvement is visible in both Pile-test (language modeling) and MMBench/MMBench_CN (multimodal understanding)

The warm-up functions as an annealing schedule that lets the model's optimization trajectory smoothly transition from a language-only loss landscape to a joint loss landscape, rather than being thrown into a region with very different gradient characteristics.

The modality grouping trick (Figure 8). A purely engineering contribution: when mixing text-only and multimodal data within the same training batch, different samples have vastly different processing times. Text-only samples are fast (no vision encoder forward pass, no vision token generation). Multimodal samples are slow (image encoding + adaptor processing). When these are mixed in the same batch, the fast text samples sit idle waiting for the slow multimodal samples to complete, reducing GPU utilization.

The solution is modality grouping: at each global training step, the data loader samples either all-text batches or all-multimodal batches, rather than mixing within a batch. This eliminates the straggler problem and improves training efficiency by 20% (explicitly stated in Section 4.4). Figure 8 confirms that this batching strategy does not harm model performance on either language or multimodal metrics.

What data is in the 30% multimodal portion? Table 1 provides the full breakdown. The multimodal pretraining data is not just captioning data — it spans six categories:

Interleaved image-text (13.1%): MMC4, Wikipedia (EN & CN), Wikihow, and in-house PDF/Epub textbooks. These are documents where images appear naturally interspersed with text, teaching the model in-context multimodal understanding.
Image captions (11.1%): Capsfusion, TaiSu, Detailed Caption. Standard image-description pairs.
Tables and charts (2.1%): Chart2text, Geo170K, Ureader, Unichart, M-paper, ScienceQA, ScreenQA, SciGraphQA-295K, Paper2figure100k, Widget Captioning, Screen2words, Refexp. This is a deliberately broad collection covering chart interpretation, table reading, scientific figures, UI widgets, and diagram understanding.
Web code (0.4%): Websight dataset for UI-to-code tasks, plus Python plots scraped from GitHub Jupyter notebooks (approximately 1.1 million filtered instances where a single image is paired with at least 5 lines of code).
Scene text OCR (1.2%): Ten public datasets (ArT, MLT-17, LSVT, UberText, Coco-text, RCTW-17, ReCTS, TextOCR, OpenVINO, HierText) covering text in natural scenes, street-level imagery, and document-like images.
Document OCR (2.1%): The arXiv rendered markdown dataset (Nougat-processed) and the e-book/educational materials dataset.

The text-only 70% comes from the DeepSeek-LLM pretraining corpus — 2 trillion tokens of diverse text. This is a crucial detail: the language data is not simplified to match the multimodal data's complexity. It remains the same rich, diverse text distribution used for LLM pretraining, which is why it effectively counteracts the distributional shift that would otherwise cause language degradation.

Stage 3: Supervised Fine-Tuning

Stage 3 converts the pretrained DeepSeek-VL into a chat-capable assistant (DeepSeek-VL-Chat) by fine-tuning on instruction-following data.

Trainable components. The paper states: "We optimize the language model, VL adaptor, and hybrid vision encoder with the vision-language SFT data. SAM-B remains frozen due to limited GPU memory." This is an interesting design choice — SigLIP-L is unfrozen and fine-tuned during SFT, meaning the semantic encoder can adapt its representations to the specific instruction-following distribution. SAM-B remains frozen purely for memory reasons, not because it would hurt performance. This is a practical constraint that may mean DeepSeek-VL is leaving some high-resolution detail adaptation on the table.

SFT data composition (Table 2). The SFT mixture contains:

In-house taxonomy-based SFT data (10.5%): This is the novel contribution — data constructed from the use case taxonomy in Table 3. The authors manually collect authentic GPT-4V and Gemini test cases from the internet, organize them into a taxonomy spanning Recognition, Conversion, Analysis, Commonsense Reasoning, Logical Reasoning, Evaluation, Multi-graph, and Safety categories, and use the taxonomy to select representative prompts for each test image.
General multimodal GPT-4V-generated datasets (35.5%): ShareGPT4V, LAION-GPTV, LVIS-Instruct4V, textOCR-GPT4V, LLaVA1.6-GPT4V, IconQA. These are publicly available datasets where GPT-4V was used to generate high-quality captions or instruction-following examples.
Tables and charts (4.1%): Ureader, Geo170K, ScienceQA — subsets of the pretraining chart/table data, now formatted for instruction following.
Web code (2.0%): Screen-to-code tasks and ScreenQA.
Text-only SFT (47.9%): The same text-only SFT data used for DeepSeek-LLM's chat fine-tuning, ensuring the model maintains dialogue capabilities on pure language tasks.

Training objective. The standard next-token prediction loss, computed only on answer tokens and special tokens. System and user prompt tokens are masked from the loss — the model only learns to generate the assistant's response.

Why 47.9% text-only SFT? The paper states: "To guarantee the model's comprehensive proficiency in dialogue, we utilize a blend of multimodal data and pure text dialogue data used in DeepSeek-LLM. This approach ensures the model's versatility across various dialogue scenarios." The near-even split between text and multimodal SFT is higher in multimodal proportion than Stage 2 (30%), presumably because by Stage 3, the language capabilities are already stabilized and the model can tolerate more multimodal fine-tuning without degradation.

Scaling Methodology: How 1.3B Experiments Informed 7B Design

The paper describes a practical research engineering challenge: how to iterate on training strategies when full-scale 7B experiments are prohibitively expensive.

The small-model signal problem. When the authors attempted to monitor Stage 2 training on the 1.3B model using generation-based metrics (where the model produces free-form text and the answer is parsed), they encountered "considerable fluctuations" that made it "challenging to supervise the training process effectively." The paper cites Schaeffer et al. (2024) on emergent abilities as a mirage — in this case, the disappearance of reliable metrics at small scale is the problem. The 1.3B model may know the answer but fail to format it correctly due to limited instruction-following capability, creating noise in generation-based evaluation.

The two-pronged solution.

Perplexity-based evaluation: Instead of asking the model to generate a response and parsing it, the paper measures the model's perplexity on each answer option. For a multiple-choice question with options A, B, C, D, the text for each option is concatenated to the prompt, and the model's perplexity on those tokens is computed. The option with the lowest perplexity is selected as the predicted answer. This avoids the instruction-following bottleneck because it measures what the model knows (probability assigned to the correct answer) rather than what it can output (formatted selection).
SFT data injection during pretraining: The authors mix "a small proportion of instruction tuning data" into the Stage 2 pretraining corpus. This gives the model some minimal instruction-following ability, smoothing the evaluation metrics without fully transitioning to SFT mode. The paper does not specify the exact proportion, but it is described as "minimal."

The combination enables stable training metrics that track the 1.3B model's progress and, importantly, transfer to the 7B model. The paper states that "a significant portion of the outcomes obtained from the 1.3B models can be effectively transferred to the 7B model through the utilization of SFT." This means architectural decisions (encoder design, adaptor architecture) and training strategy decisions (modality ratio, warm-up schedule) discovered on 1.3B generalize to 7B, even though the absolute performance numbers differ.

Hyperparameters and Infrastructure (Table 4)

The paper provides a detailed hyperparameter table, which we reproduce conceptually:

Stage 1 (VL Adaptor Training):

Optimizer: AdamW with $\beta_1 = 0.9, \beta_2 = 0.95$ , weight decay $0.1$
Learning rate: $1 \times 10^{-3}$ with cosine schedule, warmup ratio $0.03$
Batch size: 256
Sequence length: 2048
Training steps: not explicitly specified for the final model (the paper states they used 1.25M caption pairs + 2.5M OCR pairs, but total steps depend on batch size)

Stage 2 (Joint Pretraining):

Optimizer: AdamW with $\beta_1 = 0.9, \beta_2 = 0.95$ , weight decay $0.1$
Learning rate: $2 \times 10^{-4}$ for 1.3B, $1 \times 10^{-4}$ for 7B, both with cosine schedule, warmup ratio $0.03$
Batch size: 512
Sequence length: 4096
Training tokens: 500B for 1.3B, 600B for 7B (this includes both text and multimodal tokens; at a 7:3 ratio, approximately 420B text tokens and 180B multimodal tokens for the 7B model)

Stage 3 (Supervised Fine-Tuning):

Optimizer: AdamW with $\beta_1 = 0.9, \beta_2 = 0.95$ , weight decay $0.1$
Learning rate: $5 \times 10^{-5}$ with cosine schedule, warmup ratio $0.03$
Batch size: 256
Sequence length: 4096
Training: 1 epoch over the SFT mixture

Infrastructure details. DeepSeek-VL-7B consumed 5 days on 64 nodes, each with 8 Nvidia A100 GPUs (512 GPUs total). DeepSeek-VL-1.3B consumed 7 days on 16 nodes (128 GPUs total). The training framework is HAI-LLM, a distributed training system that supports 3D parallelism (tensor, pipeline, and data parallelism) with an interesting adaptation for vision encoder integration: the vision encoders and text embedding layer are treated as a single module and placed as the first layer of the pipeline. Since this first layer has a complex structure that precludes standard tensor parallelism, the visual encoder forward pass is simply recomputed in all tensor parallel ranks — a small computation cost given that the vision encoders are tiny compared to the transformer blocks. The paper also notes that the vision encoder's non-uniform execution time (compared to transformer layers) requires re-dividing model layers between pipeline parallelism ranks to achieve load balance and maximize throughput.

Design Choices and Their Justifications (Summary)

Hybrid encoder over single high-resolution encoder -> No single encoder combines text alignment with high-resolution capability; using two specialized encoders lets each do what it's optimized for.
SAM-B over other vision-only encoders (DINOv2, etc.) -> Figure 10 shows SAM-B achieves lower training loss than alternatives. SAM's ViTDet backbone was trained on segmentation, which requires fine spatial discrimination, making it well-suited for OCR and grounding tasks.
Embedding concatenation over sequence concatenation -> Keeps token count at 576 rather than 1152, reducing LLM self-attention cost by approximately 4×. Table 10 shows it also performs better empirically.
Separate-then-shared MLP adaptor over fully shared or fully separate -> The hybrid approach lets each encoder's features be projected through an encoder-specific first layer (handling different statistical distributions) while allowing cross-encoder interaction in the second layer (enabling fusion).
7:3 text-to-multimodal ratio over higher multimodal ratios -> Figure 4 shows this is the sweet spot where language capability is preserved and multimodal performance improves. Higher multimodal ratios cause language collapse with diminishing multimodal returns.
Modality warm-up over fixed-ratio training from the start -> Prevents the initial language performance drop shown in Figure 9 by smoothly transitioning from pure language to joint training.
Modality grouping over mixed-batch training -> Improves throughput by 20% without performance degradation by eliminating straggler samples within batches.
Perplexity-based evaluation for small models -> Avoids the noise from limited instruction-following capability that makes generation-based metrics unreliable at 1.3B scale.
Three-stage training over two-stage (skip warmup or skip pretraining) -> Table 9 shows that all three stages contribute: skipping Stage 2 (pretraining) dramatically reduces multimodal performance; skipping Stage 1 (warmup) causes a smaller but consistent degradation.
Unfreezing SigLIP-L in Stage 3 but keeping SAM-B frozen -> A purely practical constraint (GPU memory), not a design choice backed by ablation. The paper acknowledges this explicitly.

4. Key Insights and Innovations

Innovation 1: Reframing Multimodal Pretraining as a Capacity-Allocation Problem Between Competing Modalities

The dominant assumption in vision-language model development through early 2024 was that adding vision to a language model was an additive, cumulative process. The standard pipeline — attach a vision encoder, train an adapter, fine-tune on multimodal data — implicitly treated vision capability as something you bolt onto an existing language model. The language model was the foundation; vision was the extension. If language performance degraded, it was typically attributed to catastrophic forgetting (a continual learning problem) and addressed through architectural isolation (freezing the LLM) rather than through training strategy.

DeepSeek-VL's most intellectually distinctive contribution is recasting this as a capacity-allocation problem with competitive dynamics. The paper demonstrates, through the controlled modality-ratio experiments in Figure 4, that vision and language modalities do not simply coexist in a shared parameter space — they actively compete for representational capacity. When multimodal data constitutes 100% of the training mixture, language performance collapses not gradually but precipitously. When text data is 70%, both modalities thrive. The paper's language is precise on this point:

"The performance of different modalities is strongly correlated with their respective proportions in the training dataset, substantiating the competitive relationship between the two modalities."

This is not merely catastrophic forgetting. Catastrophic forgetting implies that new learning overwrites old knowledge; the solution would be to protect existing knowledge through regularization or replay. The competitive dynamics framing implies something deeper: there is a finite representational budget within a fixed-parameter model, and vision and language draw from the same budget. You cannot simply "add vision" — you must decide what fraction of the model's capacity to reallocate from language to cross-modal processing. The 7:3 ratio is not a hyperparameter tweak; it is the empirical equilibrium point where the marginal gain in multimodal capability equals the marginal loss in language capability. At ratios beyond 70% multimodal, the paper shows that multimodal performance gains diminish while language losses accelerate — classic diminishing-returns behavior in a resource allocation setting.

This reframing has significant downstream implications beyond this paper. It suggests that (1) scaling laws for VLMs should model modality capacity allocation explicitly, not just total compute or data scale; (2) the optimal text-to-multimodal ratio likely depends on model size, with larger models having enough total capacity that the competition is less zero-sum; and (3) Mixture-of-Experts architectures (which the paper mentions as future work) may be particularly well-suited because they can partially decouple the modality-specific capacity pools. The paper does not develop these implications, but the competitive dynamics framing provides the conceptual vocabulary for doing so.

The finding that multimodal training can improve language benchmarks (Table 7: DeepSeek-VL outperforms DeepSeek-7B on MMLU and AGIEval) adds a surprising twist to this picture. It suggests that some cross-modal processing actually benefits certain types of language reasoning — perhaps because visual information provides grounding that helps with abstract concepts, or because the joint training's richer optimization landscape helps escape language-only local minima. The competition is not pure zero-sum; there are potential synergies that the right ratio can unlock.

Evidence anchor: Figure 4 provides the core empirical demonstration. The sweep from 100% multimodal to 100% language in varying ratios shows the competitive dynamics directly. The language benchmark curve in that figure drops dramatically at high multimodal ratios while the multimodal curve rises and plateaus, visually substantiating the diminishing-returns claim. Table 7 further supports the non-zero-sum nature by showing language benchmark improvements in the final model.

Innovation 2: The Use Case Taxonomy as an Evaluation and Data Construction Philosophy

Prior vision-language model development treated evaluation as something you do after building the model. You train on whatever data is available — typically an amalgamation of public academic datasets (ScienceQA, TextVQA, RefCOCO, etc.) — and then you measure performance on those same datasets' test splits. The circularity is obvious but was rarely challenged directly: optimizing for benchmark performance produces models that perform well on benchmarks, which tells you nothing about real-world behavior.

DeepSeek-VL inverts this relationship. The use case taxonomy described in Table 3 is simultaneously the organizing principle for evaluation and the specification for data construction. The authors did not collect data and then categorize it. They started by observing real user behavior — "manually gathering authentic test cases for GPT-4V and Gemini from the Internet" — and then organized that behavior into a structured taxonomy spanning Recognition, Conversion, Analysis, Commonsense Reasoning, Logical Reasoning, Evaluation, Multi-graph, and Safety, with granular subcategories like "Currency Recognition," "UI to Code," "Defect Detection," "Humor Reasoning," and "Prompt Injection." Only after establishing this taxonomy did they construct the in-house SFT dataset by selecting representative prompts for each category.

This is a design philosophy innovation, not a technical one. It says: the distribution that matters is not what benchmark authors thought would be interesting to measure, but what users actually try to do with multimodal models. If users are asking models to debug Python code from screenshots of error messages, the training data should include screenshot-to-code pairs, even if no standard benchmark measures that capability. If users are testing for humor understanding (a category in the taxonomy), the SFT data should include examples of visual humor — a domain entirely absent from standard VQA and captioning datasets.

What makes this genuinely novel — rather than merely sensible — is that it closes the loop between data construction and evaluation. The same taxonomy used to build the training data is used to construct the human evaluation dataset (Section 4.3, Figure 6). This means the evaluation directly measures whether the model learned the capabilities the data was designed to teach, without the domain shift between academic benchmarks and real-world tasks that plagues comparisons like "my model scored 85% on VQA v2 but fails on screenshots." The taxonomy makes the evaluation validity argument transparent: here is what users do; here is the data we built to teach those capabilities; here is a test set drawn from the same task distribution; here is how well the model performs.

The GPT-4V-as-judge evaluation (Figure 7) extends this philosophy by evaluating on the same real-world test cases but using a more nuanced comparative framework. Rather than scoring answers against ground truth (which may not exist for open-ended real-world questions like "explain this diagram's logic"), it asks GPT-4V to judge whether DeepSeek-VL's answer is better than, worse than, or tied with a competitor's answer. The result — DeepSeek-VL preferred in over 60% of comparisons against open-source models — provides evidence that the taxonomy-based data construction works even when the evaluator has no access to ground-truth labels, making it applicable to exactly the open-ended tasks that separate real-world performance from benchmark performance.

Significance beyond performance: This innovation is methodological, not a metric gain. A future VLM paper could adopt the same philosophy — start with user behavior, build a taxonomy, construct data and evaluation from that taxonomy — and produce a better model even with identical architecture and training strategy. The taxonomy itself is reusable: the categories in Table 3 (with 6 main categories, 13 secondary, and dozens of tertiary) provide a specification for what multimodal generalist models should be capable of, similar to how the GLUE/SuperGLUE benchmarks specified desiderata for language understanding. The difference is that the taxonomy is grounded in revealed user preferences rather than researcher intuition.

Evidence anchor: Table 3 provides the complete taxonomy. Figure 6 shows the human evaluation structured by taxonomy category, with DeepSeek-VL-7B approaching GPT-4V in Recognition, Conversion, and Commonsense Reasoning. Figure 7 shows the GPT-4V-judge evaluation confirming superiority over competitors. The paper's explicit statement that the taxonomy is used "to choose prompts for each test image, ensuring a practical and relevant instruction tuning dataset" and that "this taxonomy is also used to create an evaluation dataset" establishes the closed-loop design.

Innovation 3: Documenting and Characterizing the Failure Mode of Adapter-Only Multimodal Training

The paper contains a deceptively important negative result that the field had not systematically characterized: adapter-only multimodal training saturates quickly and does not benefit from data scaling. Table 8 shows that expanding the Stage 1 training data (where only the VL Adaptor is trained, with both the vision encoder and LLM frozen) does not improve downstream performance after fine-tuning — and may actually degrade it.

Prior work like LLaVA and InstructBLIP used adapter training as a standard first stage, but the question of how much data to use was answered heuristically (typically a few hundred thousand to a few million image-text pairs). The implicit assumption was that more data would be better, limited only by the cost of collecting it. DeepSeek-VL's experiment tests that assumption and finds it false for the adapter-only regime.

The authors' interpretation — "the projector's capacity is inherently constrained, rendering it incapable of capturing the extensive knowledge necessary for multimodal tasks" — contains an insight about the division of labor between the adapter and the LLM. The adapter (a 2-layer MLP) has vastly fewer parameters than the LLM. Its job is to map visual features into the token embedding space. With a few million examples, it can learn a reasonable mapping — enough that when the LLM is subsequently unfrozen for joint training, it starts from a good initialization. But beyond that point, the adapter's limited capacity becomes the bottleneck. It cannot compress more cross-modal knowledge into its weights, so additional data provides no signal.

This implies that the current practice of spending significant compute on adapter-only pretraining before unfreezing the LLM may be wasteful beyond a relatively low data threshold. The adapter stage is necessary — Table 9 shows that removing Stage 1 degrades performance — but it should be kept short and treated as initialization rather than as genuine multimodal pretraining. The heavy lifting of building cross-modal associations must happen in Stage 2, where the LLM's billions of parameters can actually absorb the knowledge.

This finding is significant because it directly contradicts what might be called the "data maximalist" position — the assumption that throwing more data at every stage of training is beneficial. It establishes a conceptual boundary: small models saturate; large models scale. The specific numbers (1.25M + 2.5M training pairs is sufficient; expanding beyond that doesn't help) are likely architecture-dependent, but the qualitative principle — that the adapter-only stage has a data ceiling determined by the adapter's parameter count — should generalize.

Evidence anchor: Table 8 provides the direct comparison, showing that expanded Stage 1 data does not improve (and sometimes worsens) performance across multiple benchmarks. Table 9 provides the complement: removing Stage 1 entirely hurts performance, establishing that some adapter training is necessary but that more is not better.

Innovation 4: The Hybrid Vision Encoder as a Principled Resolution of the Semantic-Detail Tension

The architectural innovation of using two separate encoders — one for semantics (SigLIP-L) and one for details (SAM-B) — has a surface-level description that Section 3 covers in detail. What makes this intellectually distinctive is the diagnosis that motivates it and the design principle it establishes for vision-language architectures.

The diagnosis, drawn from Tong et al. (2024), is that CLIP-family encoders have a specific failure mode: they produce "CLIP-blind pairs" — visually distinct images that receive similar embeddings because the differences are low-level (text, fine patterns, spatial arrangement) rather than semantic. This is not a resolution problem per se (you could train a higher-resolution CLIP model and still have the same issue) but a training objective problem. CLIP was trained to align images with captions for retrieval; captions describe the gist ("a dog on a couch"), not the details ("the dog's collar reads 'Luna' in 10pt Arial"). CLIP encoders learn to ignore information that is irrelevant for semantic matching, which unfortunately includes exactly the information needed for OCR, visual grounding, and fine-grained recognition.

The design principle DeepSeek-VL establishes is that vision-language models need two distinct visual representations that correspond to two fundamentally different types of visual understanding: (1) semantic understanding (what objects, what scene, what action) and (2) structural understanding (what text, what spatial layout, what precise shape). These two types correspond to training objectives — contrastive language alignment for semantics, self-supervised segmentation/detection for structure — and cannot be adequately captured by a single encoder regardless of resolution, because the training objective determines what information the encoder preserves.

This is a more fundamental insight than "we need higher resolution." A single high-resolution text-aligned encoder would capture more pixels but might still learn to discard the information needed for dense OCR because the alignment objective doesn't require it. The hybrid approach explicitly separates the two information streams, letting the downstream LLM learn when to attend to semantic features and when to attend to structural features. The fixed 576-token budget is the practical engineering enabler — it makes this dual-stream approach computationally viable by ensuring the vision token count doesn't double.

The ablation in Figure 10, showing that hybrid encoding achieves lower training loss than either encoder alone, validates the principle. But the deeper significance is that this design principle is composable: future work can replace SigLIP with a stronger semantic encoder or SAM with a stronger structural encoder without changing the overall architecture. The principle is "use separate encoders for separate types of visual information, fuse them at the feature level, keep token count fixed" — and this principle generalizes beyond the specific encoder choices made here.

Significance: This is a fundamental architectural insight, not an incremental improvement. It provides a framework for thinking about vision encoder design in VLMs that goes beyond the resolution debate — it's about what kind of information the vision pipeline preserves, not just how much detail.

Evidence anchor: Figure 10 shows the training loss comparison across encoder configurations. Table 10 provides the benchmark-level validation of the hybrid approach against alternatives. The qualitative examples in Figures 2, 5, 11 show the model's ability to read fine text in screenshots and diagrams, providing qualitative evidence that the hybrid encoder preserves structural detail.

5. Experimental Analysis

Evaluation Methodology

Datasets. The paper evaluates on a broad collection of public benchmarks. For multimodal understanding: MMMU, CMMMU, MMBench, MMBench-CN, SeedBench, MMV, OCRBench, POPE, ScienceQA, and MathVista. For language understanding: MMLU, HellaSwag, Pile (test set), GSM8K, MBPP, and AGIEval. Additionally, the authors construct their own human evaluation dataset of 100 questions spanning seven categories based on Table 3's taxonomy, and a GPT-4V-judge evaluation set of 99 samples. Benchmarks use their standard test/evaluation splits; for MMBench and MMBench-CN, the paper uses the dev split because the official test download link was inactive at the time of evaluation.
Base models. The DeepSeek-VL family builds on the DeepSeek-LLM series (DeepSeek-AI, 2024), using intermediate checkpoints: DeepSeek-LLM-1.3B trained on approximately 500B text tokens, and DeepSeek-LLM-7B trained on approximately 2T text tokens. These base models follow the LLaMA architecture with Pre-Norm RMSNorm, SwiGLU activations, and Rotary Position Embedding. The authors state the 7B model is "representative of the capabilities of many contemporary LLMs" and sits in a regime where non-trivial multimodal performance can be achieved without saturating benchmarks.
Metrics. The paper uses three distinct evaluation protocols depending on the benchmark type. Perplexity-based evaluation for multiple-choice datasets (HellaSwag, MMLU): the model's perplexity is computed on each answer option, and the lowest-perplexity option is selected as the prediction. This avoids discontinuities from exact-match scoring and helps distinguish subtle probability differences. Generation-based evaluation with greedy decoding for multimodal benchmarks (MMBench, SeedBench, MMMU, MathVista, ScienceQA) and some language benchmarks (GSM8K, AGIEval): the model generates free-form text, and answers are parsed from the output using the grading functions provided by each benchmark. Language-modeling-based evaluation for the Pile test set: bits-per-byte is computed on the test corpus. For the human evaluation and GPT-4V-judge comparisons, the metric is win rate (percentage of cases where the model's answer is judged better than a competitor's).
Baselines. The paper compares against a large set of models structured by scale. For the 7B-scale comparisons (Table 5): Qwen-VL-Chat (Bai et al., 2023), LLaVA-1.5 (Liu et al., 2024b), InstructBLIP (Dai et al., 2023), InternLM-XComposer2-VL (Dong et al., 2024), CogVLM (Wang et al., 2023b), and proprietary systems including GPT-4V (OpenAI, 2023b), Gemini Pro and Gemini Ultra (Team et al., 2023), and Qwen-VL-Plus. For the 1.3B-scale comparisons (Table 6): MobileVLM (1.7B, 2.7B), Qwen-VL-Chat-1.8B, MiniCPM (2.8B), and TinyLLaVA (1.4B, 2.7B). For language-only comparisons (Table 7), the primary baseline is the original DeepSeek-7B model. The human evaluation (Figure 6) adds InternLM-XComposer2-VL and GPT-4V as head-to-head competitors.
Generation budget / compute accounting. The models are evaluated using greedy decoding (temperature 0), which means each question receives exactly one generated response. There is no compute budget sweep, no best-of-N aggregation, and no test-time scaling experiments. The compute comparison is therefore at parity: all models generate a single answer per question under equivalent decoding settings. For the training efficiency experiments (Stage 2 modality grouping), the paper reports a 20% throughput improvement when grouping modalities into separate batches versus mixing them within the same batch, measured by training time rather than FLOPs.
Cross-validation / statistical protocol. There is no cross-validation or statistical significance testing reported for benchmark comparisons. The benchmark numbers are reported as single-point estimates (accuracy or score percentages). For the human evaluation, 100 questions are used with no mention of confidence intervals or inter-annotator agreement. The GPT-4V-judge evaluation uses 99 samples with a single judge model (GPT-4V itself), and the paper does not report position bias controls, judge-model calibration, or test-retest reliability. The paper does not address whether the performance differences between models (often in the range of 1-3 percentage points on individual benchmarks) are statistically significant or within evaluation noise.

Main Quantitative Results

Multimodal Benchmark Performance (7B Scale)

Table 5 provides the central multimodal comparison for DeepSeek-VL-7B against both open-source and proprietary models across 11 benchmarks. The headline results:

On SeedBench, DeepSeek-VL-7B achieves 70.4, approaching GPT-4V's 71.6 and surpassing all open-source models in the comparison (CogVLM at 68.8, Qwen-VL-Chat at 58.1). This is the closest open-source result to the proprietary frontier at the time of publication.

On MMBench and MMBench-CN, DeepSeek-VL-7B achieves 73.8 and 71.9 respectively, outperforming CogVLM (65.0/56.3) and InternLM-XComposer2-VL (79.0/71.0 on different test configurations). The paper uses the dev split because the test download link was inactive.

On MathVista, DeepSeek-VL-7B achieves 36.1, which exceeds all open-source models but remains substantially behind GPT-4V (47.8). The paper attributes this gap to "the variance in base model sizes," noting that GPT-4V's language backbone is substantially larger than 7B parameters.

On MMMU, DeepSeek-VL-7B achieves 35.4, competitive with open-source models (CogVLM at 33.1; Qwen-VL-Plus at 42.6) but behind GPT-4V (56.8) and Gemini Ultra (59.4). This is a benchmark designed to require expert-level knowledge across 30 subjects, and the gap to larger proprietary models is consistent with the scale explanation.

On OCRBench, DeepSeek-VL-7B achieves 594, which lags behind InternLM-XComposer2-VL (610) but exceeds CogVLM (590) and GPT-4V (516). This is notable because OCRBench directly tests the high-resolution OCR capabilities that the hybrid vision encoder was designed to improve.

On POPE (hallucination evaluation), DeepSeek-VL-7B achieves 86.2, competitive with GPT-4V (87.5) and InternLM-XComposer2-VL (86.6), suggesting that the model's object recognition is well-calibrated and resists hallucination.

On ScienceQA (image-only subset), DeepSeek-VL-7B achieves 72.9, below InternLM-XComposer2-VL (96.4) and GPT-4V (78.2). The paper does not discuss this discrepancy; the InternLM result on ScienceQA is anomalously high across the board.

The overall pattern: DeepSeek-VL-7B is consistently the best or second-best open-source model across benchmarks, with particularly strong results on SeedBench and competitive OCRBench performance, but a clear gap to GPT-4V persists on knowledge-intensive benchmarks (MMMU) and mathematical reasoning (MathVista).

Multimodal Benchmark Performance (1.3B Scale)

Table 6 compares DeepSeek-VL-1.3B against other small multimodal models. The model achieves 63.5 on MMBench and 56.6 on MMBench-CN, outperforming MobileVLM-2.7B (57.9/40.1) and Qwen-VL-Chat-1.8B (38.2/7.4) despite having fewer parameters. On MathVista, DeepSeek-VL-1.3B achieves 29.4, which the paper notes "even achieves comparable results to 7B open-source models," specifically Qwen-VL-Chat-7B (13.2) and LLaVA-1.5-7B (25.2). This is interpreted as evidence that the training strategy's benefits compound at small scale.

Language Benchmark Performance

Table 7 shows that DeepSeek-VL-7B maintains language capabilities comparable to, or exceeding, the original DeepSeek-7B across most benchmarks. On HellaSwag, DeepSeek-VL achieves 68.4 versus DeepSeek-7B's 68.5 — essentially no degradation on a general language understanding metric. On MMLU, DeepSeek-VL scores 64.5 versus DeepSeek-7B's 63.6 — a slight improvement, suggesting multimodal training may have positive transfer to certain language tasks. On AGIEval, DeepSeek-VL achieves 44.3 versus DeepSeek-7B's 40.6 — a notable 3.7-point improvement.

The one area of measurable decline is mathematics: DeepSeek-VL achieves 53.3 on GSM8K versus DeepSeek-7B's 63.0 — a 9.7-point drop. The paper acknowledges this explicitly, attributing it to "limited model capacity (7B)" and suggesting that "larger models might alleviate this issue significantly." This is consistent with the competitive dynamics framing: mathematical reasoning and visual processing both draw on the same finite parameter budget, and in a 7B model, that budget is tight enough that gains in one modality require sacrifices in another.

On MBPP (code generation), DeepSeek-VL scores 44.2 versus DeepSeek-7B's 43.8 — essentially unchanged. On Pile-test (bits-per-byte), DeepSeek-VL achieves 0.54 versus DeepSeek-7B's 0.56 — a small improvement.

Human Evaluation

Figure 6 presents head-to-head human evaluation results across the seven taxonomy categories from Table 3, comparing DeepSeek-VL-7B with InternLM-XComposer2-VL, CogVLM, and GPT-4V. The key findings:

GPT-4V dominates across most dimensions, particularly in Logical Reasoning. This is consistent with the benchmark results and attributed to scale differences.
DeepSeek-VL-7B approaches GPT-4V in Recognition, Conversion, and Commonsense Reasoning. This is the paper's central "real-world experience" claim: on the tasks that users most frequently perform (describing images, converting formats, applying common sense), DeepSeek-VL is competitive with proprietary systems.
DeepSeek-VL-7B outperforms both open-source competitors (InternLM-XComposer2-VL and CogVLM) in overall performance and across most individual categories.
All open-source models trail GPT-4V substantially in Logical Reasoning, which the paper attributes to LLM scale rather than vision encoder design.

The absolute numbers are difficult to assess precisely because Figure 6 uses a bar chart without numeric labels (readers must estimate from axis ticks). The paper does not provide a table of exact scores, confidence intervals, or inter-annotator agreement metrics for this evaluation.

GPT-4V-as-Judge Evaluation

Figure 7 shows the results of using GPT-4V itself to judge output quality by showing it the question and two model answers (DeepSeek-VL vs. a competitor) and asking which is better or to declare a tie. The results:

Against open-source models: DeepSeek-VL is preferred over Fuyu-8B, CogVLM-17B, and InternLM-XComposer2-VL in over 60% of cases each. This is a strong result, especially considering CogVLM-17B has more than 2× the parameters.
Against GPT-4V itself: DeepSeek-VL's responses are judged favorable in a non-trivial fraction of comparisons (the bar chart in Figure 7 shows this as "comparable performance" to GPT-4V), though GPT-4V still has the edge. The exact numbers are not stated in the text and must be inferred from the figure.

The methodology follows Zheng et al. (2024)'s LLM-as-a-judge protocol, but the paper does not report position bias controls (whether answer order affects judgments), judge calibration metrics, or any analysis of GPT-4V's tendencies when evaluating outputs from its own model family.

Ablation Studies and Robustness Checks

Scaling Stage 1 data (Table 8): Expanding the VL Adaptor training dataset beyond the baseline (1.25M ShareGPT4V captions + 2.5M Document OCR pairs) does not improve performance and may degrade it. This is tested by scaling up Stage 1 data, then applying supervised fine-tuning and evaluating. The results show that the adapter's limited capacity saturates quickly — additional data provides no benefit because a two-layer MLP cannot encode more cross-modal knowledge. This is a negative result that directly informs training efficiency: Stage 1 should be kept short and treated as initialization, not extended in hopes of capturing more multimodal associations.

Training stage contributions (Table 9): The combination of all three stages (adaptor warmup + joint pretraining + SFT) outperforms the two-stage alternatives. Omitting Stage 2 (joint pretraining) and going directly from Stage 1 to Stage 3 causes substantial performance degradation across all metrics, confirming that the LLM must be jointly optimized with multimodal data to develop robust cross-modal understanding — adapter-only training followed by SFT is insufficient. Omitting Stage 1 (adaptor warmup) and going directly to Stage 2+3 causes a "slight lag" behind the full three-stage pipeline, indicating that the warmup provides a better initialization for joint pretraining and is worth the small computational cost.

Modality grouping (Figure 8): When mixing text-only and multimodal data during Stage 2, grouping modalities into separate batches (all-text or all-multimodal per global step) versus mixing them within the same batch improves training efficiency by 20% without degrading performance. Figure 8 shows that the grouped approach matches or slightly outperforms the mixed approach on both language (Pile-test) and multimodal (MMBench and MMBench-CN) benchmarks, while substantially reducing GPU idle time caused by disparate processing times between modalities.

Modality warm-up (Figure 9): Starting Stage 2 with 100% text data and gradually decreasing to the target ratio (e.g., 70:30 text:multimodal) prevents an initial decline in language capabilities and yields better final performance on both modalities compared to starting directly at the target ratio. The results show that without warm-up, language metrics drop sharply at the beginning of training; with warm-up, the transition is smooth. The warm-up strategy also improves final multimodal benchmark scores slightly, suggesting that a stable training trajectory benefits both modalities.

Vision encoder selection (Figure 10): Comparing training loss curves across encoder configurations (SigLIP alone, SAM alone, SigLIP+SAM hybrid) shows that the hybrid approach achieves lower training loss than either encoder individually. This is a within-training validation rather than a downstream benchmark comparison, but it directly measures how effectively the model uses the visual information provided by each encoder configuration. The hybrid encoder's lower loss suggests that the semantic and structural information streams are complementary — the model learns more effectively when both are available.

Sequence vs. embedding concatenation (Table 10, top): When combining multi-resolution features, concatenating them along the embedding dimension (producing 576 tokens with 2048 dimensions) outperforms sequence concatenation approaches like "Token Pooling - W" and "Token Pooling - H" (which compress features along width or height dimensions before sequence concatenation). The embedding concatenation achieves 55.9 average score versus 55.5 and 54.2 for the pooling alternatives. This validates the design choice to keep token count fixed while increasing feature dimensionality, which also reduces LLM self-attention cost by avoiding doubling the sequence length to 1152 tokens.

MLP adaptor architecture (Table 10, bottom): The "Hybrid MLP" configuration (separate first-layer MLPs for SigLIP and SAM features, followed by a shared second-layer MLP) achieves the best average score (55.9) compared to "Shared MLP" (single MLP processing both encoders' features, 55.2) and "Separate MLP" (completely independent MLPs, 54.5). The performance differences are modest (within 1.4 points on average) but consistent. The "Separate MLP" configuration notably underperforms on MMB (57.5 vs. 62.0 for Shared MLP), suggesting that some cross-encoder feature interaction is necessary for effective fusion. The "Hybrid MLP" balances encoder-specific processing (distinct first layers handle different feature distributions) with cross-encoder interaction (shared second layer enables fusion).

Critical Assessment

Claim 1: Joint vision-language pretraining with 7:3 text-to-multimodal ratio preserves language capabilities.

The experiments provide strong evidence that the 7:3 ratio works for the specific case tested: DeepSeek-7B trained on the data mixture described in Table 1, evaluated on the language benchmarks in Table 7. The HellaSwag numbers are compelling (68.4 vs. 68.5 — preservation is essentially perfect), and the MMLU/AGIEval improvements suggest positive transfer.

However, what the experiments do not establish is whether 7:3 is a general principle or a point estimate specific to (a) this model scale, (b) this data composition, and (c) this training duration. Figure 4 sweeps ratios on a 1.3B model, showing that the ratio-performance relationship is smooth and that 7:3 is indeed near-optimal for that scale. But there is no comparable sweep for the 7B model — the 7:3 ratio is tested at 7B scale, but the alternatives (e.g., 8:2, 6:4, 5:5) are not. It is possible that at 7B, the optimal ratio shifts (perhaps toward more multimodal data, since the larger model has more capacity to accommodate it). The paper's claim that "the performance of different modalities is strongly correlated with their respective proportions" is established at 1.3B but not validated at 7B.

The GSM8K decline (53.3 vs. 63.0) is a significant counterpoint. The paper attributes this to "limited model capacity" and predicts larger models will alleviate it, but this prediction is untested. A skeptical reading would note that mathematical reasoning appears to be the modality most directly competing with vision for representational resources — and if this competition persists at larger scales, it would be a fundamental limitation of joint training, not an artifact of 7B being "too small."

Claim 2: The hybrid vision encoder handles high-resolution details within a fixed 576-token budget.

The evidence for this claim is moderate. OCRBench performance (594 for DeepSeek-VL-7B vs. 516 for GPT-4V) is the strongest quantitative signal, since OCRBench directly measures the capability the hybrid encoder was designed to improve. The qualitative examples (Figures 2, 5, 11-14) are visually impressive — the model reads fine text from webpages, identifies specific UI elements, and provides detailed descriptions — but selected qualitative examples cannot establish that the hybrid design is better than alternatives.

The missing experiment is a controlled comparison of (a) the hybrid SigLIP+SAM encoder versus (b) a higher-resolution single encoder (e.g., a 768×768 or 1024×1024 CLIP-family model) with equivalent token budget. The paper argues such an encoder doesn't exist because CLIP-family models are limited to moderate resolutions, but this is a claim about currently available models, not a principled demonstration that the hybrid approach is architecturally superior. Figure 10 compares SigLIP alone versus SigLIP+SAM and shows lower training loss for the hybrid, but this compares a low-resolution semantic encoder against a hybrid that adds high-resolution structural features — of course the hybrid does better on tasks requiring those features. It does not separate the effects of "higher resolution" from "dual-encoder design."

The fixed token budget claim also warrants scrutiny: the paper achieves 576 tokens through a specific processing chain (SigLIP at 384×384 producing some number of patches; SAM at 1024×1024 downsampled through convolutions). Whether this specific processing chain is optimal — or whether alternative downsampling schemes, different encoder resolutions, or different fusion mechanisms would work better — is not explored beyond the modest ablations in Table 10.

Claim 3: In-house taxonomy-based SFT data substantially improves user experience in practical applications.

The evidence here is suggestive but methodologically limited. The human evaluation (Figure 6) shows DeepSeek-VL performing well on taxonomy-based categories, and the GPT-4V-judge evaluation (Figure 7) shows it preferred over competitors. But there is no ablation comparing the model trained with taxonomy-based SFT data versus an identical model trained without it. The taxonomy data constitutes only 10.5% of the SFT mixture (Table 2) — the majority is public GPT-4V-generated data (35.5%) and text-only SFT (47.9%). It is possible that the public data alone would produce similar real-world performance, and the taxonomy data contributes marginally or not at all.

The human evaluation uses 100 questions, which is a small sample for 7+ categories. With ~14 questions per category on average, individual category comparisons are noisy. The paper does not report confidence intervals or inter-annotator agreement, making it impossible to assess whether the category-level differences between models are statistically reliable. Similarly, the GPT-4V-judge evaluation uses 99 samples with a single judge model, which raises concerns about judge bias (does GPT-4V prefer outputs that resemble its own style?) and position bias (does answer order affect judgment?), neither of which is addressed.

Claim 4: Modality warm-up stabilizes training and improves final performance.

Figure 9 supports this claim for the 1.3B model on Pile-test and MMBench/MMBench-CN, with the warm-up curves starting higher and ending at or above the no-warm-up curves. However, the effect sizes appear modest (the gap between warm-up and no-warm-up curves in the figure narrows by the end of training, particularly for Pile-test), and the paper does not report the final numerical differences. At 7B scale, warm-up is applied but not ablated — there is no 7B comparison of warm-up vs. no-warm-up. The claim that warm-up "yields comparatively superior outcomes" is validated at 1.3B but assumed to generalize to 7B without direct evidence.

Claim 5: DeepSeek-VL achieves state-of-the-art or competitive performance across visual-language benchmarks at the same model size.

This claim holds. Table 5 clearly shows DeepSeek-VL-7B as the best or second-best open-source model on nearly every benchmark. The caveat is that "same model size" comparisons are imprecise because competing models use different LLM backbones, different vision encoders, and different training data scales, making it difficult to attribute DeepSeek-VL's advantage to any specific design choice. CogVLM-17B, for example, is a larger model that DeepSeek-VL-7B outperforms on several benchmarks — but the paper does not analyze whether this is due to architecture, data, or training strategy rather than the parameter count difference working in DeepSeek-VL's favor.

Missing experiments that would strengthen the paper:

A full modality-ratio sweep at 7B scale to validate that 7:3 is optimal for larger models, not just 1.3B.
An ablation of the taxonomy-based SFT data — train a model identically but without the 10.5% in-house data, and compare real-world performance.
A controlled comparison of hybrid encoder vs. single high-resolution encoder with equivalent token budget and comparable training.
Position bias controls and inter-annotator agreement for the GPT-4V-judge evaluation.
Confidence intervals or statistical tests for benchmark comparisons — many performance differences are in the 1-3% range, which may be within evaluation noise.
Latency analysis — the paper claims the fixed token budget enables efficient multi-turn inference, but no timing or throughput numbers are reported.
Scaling experiments — the paper predicts that larger models will alleviate the language-multimodal competition, but only tests 1.3B and 7B scales. A 13B or 34B data point would substantiate or refute this prediction.
Out-of-distribution / robustness evaluation — all evaluations use in-distribution benchmarks. The paper's real-world claims would be strengthened by testing on distribution-shifted inputs (e.g., images with noise, compression artifacts, unusual aspect ratios, or adversarial text).

6. Limitations and Trade-offs

Difficulty Estimation Cost Is the Unaccounted Tax on the 4× Gains

The assumption or constraint. The entire compute-optimal framework rests on the ability to estimate prompt difficulty before deciding how to allocate the inference budget. The paper's method for doing so—generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted)—is extraordinarily expensive. The authors acknowledge this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity."

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. Generating 2048 samples to estimate difficulty is comparable to, or larger than, the largest test-time budgets studied (256–512 generations). In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the estimation cost could dominate. This means the 4× figure is an upper bound, not a realized deployment gain—and in the worst case, the overhead could negate the efficiency improvement entirely.

What evidence exists in the paper. The paper does not measure or amortize difficulty estimation cost. This is stated as a caveat, not measured in any ablation. The paper does not explore how the difficulty estimation budget scales relative to the strategy execution budget, nor does it report whether using fewer samples for estimation (e.g., 100 vs. 2048) preserves the difficulty bin ordering needed for policy selection. The predicted bins (non-oracle) show "largely overlapping" performance with oracle bins in Figures 4 and 8, which suggests a cheaper verifier-based difficulty signal works, but the 2048-sample cost remains.

Mitigation status. The authors flag this as an important open problem in Section 8: "future work is to investigate cheaper methods for estimating question difficulty." They also note that "the amortized cost of test-time compute is relatively low in self-improvement settings where a small model is used many times," suggesting the exploration-exploitation tradeoff is more favorable when difficulty can be estimated once and reused. No solution is implemented or evaluated.

Hard Problems Remain Essentially Unsolved—Test-Time Compute Cannot Create Capability from Nothing

The assumption or constraint. The paper's approach assumes that the base model already has a non-trivial probability of producing correct answers. Across all methods—search, revisions, and their compute-optimal combinations—the hardest questions (difficulty bin 5, where the base model's pass@1 is near zero) show near-zero improvement regardless of compute budget.

The consequence. Test-time compute amplifies existing capability but does not create it. If the base model's pass@1 is near zero, no amount of search or revision will help—there are no correct solutions in the proposal distribution to find or refine. For problems genuinely outside the base model's training distribution or reasoning capability, test-time compute is useless, and scaling pretraining remains the only viable path. This imposes a hard ceiling on the approach's applicability: it only works for problems the base model "sort of" already knows how to solve.

What evidence exists in the paper. Figure 3 (right) shows bin 5 accuracy hovering at 1–3% for all methods and budgets. Figure 7 (right) shows bin 5 at roughly 2–3% accuracy regardless of sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%, and the bar chart in Figure 1 bottom-right shows a −52.9% relative disadvantage for PRM search on hard problems at high inference-to-pretraining ratios. The paper explicitly acknowledges this in Section 7: "test-time compute is ineffective and pretraining is more effective" for hard problems.

Mitigation status. The paper is transparent about this boundary condition and makes no attempt to solve it. Section 8 does not propose mechanisms for generating novel solutions beyond the base model's capability, implying that this is a fundamental limitation of the proposer-verifier decomposition rather than a fixable engineering issue.

Single Benchmark, Single Model Family—The Difficulty-Dependent Scaling Curves Have Unknown Generalizability

The assumption or constraint. All experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this is an assertion, not a finding.

The consequence. Several aspects of the findings could be model-specific or benchmark-specific, but the paper cannot distinguish universal principles from artifacts of this particular (model, dataset) pair:

The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties or error patterns could show qualitatively different difficulty-dependent scaling curves (e.g., the threshold where beam search outperforms best-of-N might shift, or the over-optimization cliff might occur at different budgets).
The revision model's ability to learn from edit-distance-paired incorrect-correct trajectories depends on the base model's in-context learning and representational similarity properties, which vary across model families.
MATH consists of competition-level math problems requiring symbolic reasoning. It is unknown whether the key patterns—beam search over-optimizing on easy problems, revisions helping most on easy problems, parallel sampling being necessary on hard problems—generalize to other reasoning domains (code generation, logical reasoning, scientific QA) or to tasks requiring factual recall rather than step-by-step inference.

What evidence exists in the paper. None. There are no experiments on other benchmarks, other model families, or other task types. The difficulty bins themselves are model-specific (quintiles of PaLM 2-S* pass@1), so even the difficulty thresholds would shift with a different base model.

Mitigation status. Not addressed. The paper does not claim generalizability beyond MATH, but the framing—"test-time scaling laws"—implies broader applicability that is not validated. Future work on replicating the difficulty-dependent patterns across benchmarks and model families is not explicitly called out in Section 8, beyond a generic note about "extending our analysis to more tasks and model configurations."

PRM Search and Revisions Are Never Combined—The Paper Studies Two Complementary Axes Independently

The assumption or constraint. The paper studies PRM tree-search (verifier optimization) and iterative revisions (proposal distribution modification) as independent mechanisms. They are evaluated separately, with separate compute-optimal policies, and never integrated into a unified system.

The consequence. This is a significant gap because the two mechanisms have complementary, difficulty-dependent strengths that the paper itself documents: revisions excel on easy problems (local refinement of nearly-correct answers), while PRM search excels on medium problems (global exploration to find correct solutions among many candidates). A combined system—using the revision model as the proposal distribution within beam search, or using the PRM to guide which revision branches to pursue—could yield gains beyond either individual method. The current results represent a lower bound on what an integrated system could achieve and leave open the question of whether the two axes are additive, synergistic, or partially redundant.

What evidence exists in the paper. Section 8 explicitly acknowledges this: "we did not experiment with PRM tree-search techniques in combination with revisions." The paper provides no data on combined performance. The theoretical framing in Section 2 posits proposal and verifier as independent axes, which naturally suggests they should be combined, but the empirical validation of that framing's utility for combination is absent.

Mitigation status. Explicitly left to future work. The architectural and algorithmic questions for integration (e.g., how to weight PRM signals against revision-context signals, how to budget compute between search depth and revision depth) are not explored.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate—The Training Protocol Creates a Fundamental Asymmetry

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect, followed by a correct target answer. This is a consequence of the training data construction (Section 6.1): trajectories consist of 0–4 incorrect answers followed by a correct answer. The model never sees examples of what to do when the current answer is already correct—it has no training signal for "stop revising, this answer is right."

The consequence. At test time, the revision model cannot recognize when its current answer is correct. The paper reports that approximately 38% of correct answers produced during a revision chain get "revised" back to incorrect answers in the subsequent step. This is a direct, predictable failure mode caused by the training protocol asymmetry. The mitigation—selecting the best answer from anywhere in the chain using majority voting or verifier-based selection—is a post-hoc patch that does not fix the underlying behavior. It means the revision model is wasteful: roughly a third of its successful revisions are immediately undone, consuming computation without benefit. In principle, a model trained to also recognize and preserve correct answers could be significantly more efficient.

What evidence exists in the paper. The paper states the 38% figure explicitly in Section 6.1. Figure 6 (left) shows pass@1 gradually improving through the chain but never dramatically, which could be partially explained by correct solutions being revised away. The paper does not analyze the dynamics of correct-to-incorrect transitions beyond reporting the aggregate rate—there is no breakdown by revision step, difficulty, or problem type.

Mitigation status. Partially mitigated. The within-chain selection (majority voting or verifier) reduces the impact, as shown in Figure 6 (right) where sequential outperforms parallel. The paper does not explore training protocol modifications (e.g., including "stop tokens" or correct-answer-preservation trajectories), leaving this as a known but unresolved limitation. The ReST $^{EM}$ experiment (Appendix K, Figure 16) shows that alternative revision training approaches can backfire, suggesting the correct-to-incorrect reversion problem is sensitive to training methodology in ways that are not well understood.

Latency and Wall-Clock Time Are Ignored—Sequential-Parallel Ratios Have Hidden Deployment Costs

The assumption or constraint. The paper measures compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores wall-clock time and the parallelism constraints of real hardware.

The consequence. The sequential-to-parallel ratio that the compute-optimal policy selects per difficulty bin has radically different latency implications. Fully sequential strategies (e.g., a single chain of 64 revisions) are inherently serial—each step depends on the previous one—and cannot be parallelized. Fully parallel strategies (e.g., 64 independent best-of-N samples) can in principle run simultaneously on sufficient hardware. A strategy that allocates 128 generations as 64 sequential × 2 parallel takes roughly 64× longer wall-clock time than one that runs 128 parallel samples simultaneously, even though both consume the same total FLOPs. The paper's compute-optimal policy on easy problems tends to favor sequential-heavy allocations (Figure 7, right panel), meaning it selects exactly the strategies with the worst latency profiles for the problem class that is most common in interactive applications.

What evidence exists in the paper. The paper includes no latency or throughput measurements. There is no discussion of the wall-clock implications of sequential vs. parallel strategies, no modeling of hardware constraints, and no evaluation of latency-constrained scenarios. The generation budget abstraction implicitly assumes that latency is unlimited and only total FLOPs matter.

Mitigation status. Not addressed. The paper does not acknowledge this tradeoff. For interactive or latency-sensitive deployments (chatbots, real-time assistants, live coding support), the sequential-heavy strategies that the compute-optimal policy favors may be impractical regardless of their accuracy gains. A latency-aware variant of the compute-optimal objective—maximizing accuracy subject to both a FLOPs budget and a wall-clock budget—is not formulated. This is a major practical gap for anyone seeking to deploy these strategies in time-sensitive applications.

7. Implications and Future Directions

How This Work Changes the Landscape

DeepSeek-VL shifts the conversation around vision-language model development from a component-level optimization paradigm to a system-level co-design paradigm. Prior work in the open-source VLM space typically improved one piece of the pipeline at a time—a better vision encoder here, a larger instruction tuning dataset there, a new adapter architecture—and evaluated success primarily through benchmark leaderboard position. DeepSeek-VL's contribution is demonstrating that the gaps between open-source and proprietary models in real-world user experience cannot be closed through incremental component improvements alone. They require simultaneous attention to data construction philosophy, architectural design for high-resolution processing, and training strategies that explicitly manage the competition between modalities.

The magnitude of this shift is a reframing rather than a paradigm shift. The individual techniques—hybrid encoders, joint text-multimodal training, instruction tuning on curated data—are not individually radical. What is new is the integration and the diagnostic framework that motivates each choice. The paper identifies four specific failure modes of existing open-source VLMs (underinvestment in pretraining, benchmark-optimized instruction data, low-resolution encoding, and language capability degradation) and designs a system that addresses all four simultaneously, with ablation evidence that each component matters. This is systems thinking applied to VLM design: the interactions between components matter as much as the components themselves. For example, the hybrid vision encoder only realizes its full benefit when paired with joint pretraining that teaches the LLM to use structural features; the high-resolution SAM encoder would be wasted if the LLM were only trained on low-resolution captioning data that never requires fine-grained visual discrimination.

The paper also changes the burden of proof for VLM evaluation. By constructing a use case taxonomy from observed user behavior and using it to build both training data and evaluation datasets, the paper implicitly argues that benchmark performance is necessary but insufficient evidence of real-world capability. The human evaluation (Figure 6) and GPT-4V-judge evaluation (Figure 7) provide a template for what a more comprehensive evaluation looks like—one that measures performance on the tasks users actually attempt, not just the tasks benchmark authors thought to measure. This does not replace benchmark evaluation but supplements it with a validity argument: the evaluation tasks are drawn from the same distribution as the training tasks, and that distribution is grounded in revealed user preferences rather than researcher intuition. A future VLM paper that reports only benchmark numbers will, after DeepSeek-VL, face the implicit question: "But how well does it work on what users actually do?"

The paper reconciles a latent tension in the VLM literature between two competing narratives. One narrative, exemplified by models like LLaVA and InstructBLIP, emphasized adapter-based architectures where the LLM is largely frozen during multimodal training—preserving language capabilities but limiting cross-modal integration. The other narrative, exemplified by models like CogVLM and Qwen-VL, advocated for deeper visual integration through trainable vision experts or full LLM fine-tuning, accepting some language degradation as the price of better multimodal performance. DeepSeek-VL's joint training experiments (Figure 4) show that this is a false dichotomy. With the right modality ratio (approximately 7:3 text-to-multimodal) and a gradual warm-up schedule, it is possible to have deep multimodal integration and preserved (or even improved) language capabilities. The competitive dynamics framing—that vision and language draw from the same finite representational budget—provides a vocabulary for understanding why the trade-off exists and how to navigate it, transforming what looked like an unavoidable compromise into a tunable design parameter.

This reframing makes several research directions newly attractive. Understanding the scaling behavior of the optimal modality ratio becomes a first-order question: does the 7:3 ratio hold at 13B, 70B, or larger scales, or does growing model capacity shift the equilibrium toward more multimodal data? The paper's prediction that "larger models might alleviate this issue significantly" (referring to the GSM8K math decline) is testable and would directly inform the design of larger VLMs. The use case taxonomy as a living specification becomes viable: as new VLM applications emerge (medical image interpretation, CAD diagram understanding, video frame analysis), the taxonomy can be extended, and training data constructed accordingly, creating a systematic path for expanding VLM capabilities. Mixture-of-Experts architectures become particularly interesting in light of the competitive dynamics finding—if vision and language compete for shared parameters, an MoE architecture with partially modality-specialized experts could partially decouple their capacity pools, potentially allowing higher multimodal ratios without language degradation. The paper mentions MoE as future work, but the competitive dynamics framing provides the theoretical motivation for why MoE might be especially well-suited to VLMs specifically.

Conversely, some research directions become less attractive. Adapter-only architectures that keep the LLM frozen now face a stronger burden of proof. Table 9 shows that omitting joint pretraining (Stage 2) substantially degrades performance, meaning the frozen-LLM approach leaves significant multimodal capability on the table. The finding that adapter training saturates quickly (Table 8)—the two-layer MLP cannot absorb more cross-modal knowledge beyond a few million examples—further argues against adapter-only approaches as a path to strong multimodal understanding. Naive data scaling without attention to modality balance is also undermined: the paper shows that more multimodal data can be actively harmful if it displaces text data and causes language collapse. Low-resolution vision encoders as a design choice become harder to justify given the paper's evidence that high-resolution structural features (via SAM-B) provide measurable improvements on OCRBench and qualitative improvements on real-world tasks.

Follow-Up Research This Work Enables

Scaling the optimal modality ratio: does 7:3 generalize to larger models? The paper establishes through Figure 4 that a 7:3 text-to-multimodal ratio is near-optimal for a 1.3B model, and validates that this ratio works well at 7B (Table 7). But the sweep across ratios is only done at 1.3B scale; at 7B, only the 7:3 operating point is tested. A natural follow-up would replicate the Figure 4 sweep at 7B, 13B, and (if compute permits) 34B scales, measuring both language benchmark preservation and multimodal benchmark performance at each ratio. The paper's hypothesis—that larger models should tolerate higher multimodal ratios because they have more total capacity—predicts that the optimal ratio should shift rightward (more multimodal data) as scale increases, and that the language degradation at high multimodal ratios should become less severe. Testing this would validate or refute the competitive dynamics framing: if the optimal ratio is scale-invariant, the competition is about data distribution rather than parameter capacity; if it shifts with scale, it is genuinely about representational budget. A strong study would also measure whether the GSM8K decline (9.7 points at 7B) shrinks at larger scales, directly testing the paper's prediction that limited capacity causes the math-vision tradeoff.

The taxonomy-to-data pipeline: how transferable is the use case taxonomy across model families? The paper constructs a use case taxonomy from GPT-4V and Gemini test cases found online, then builds in-house SFT data and an evaluation set from it. An important open question is whether this taxonomy—and data constructed from it—benefits models other than DeepSeek-VL, or whether its value is specific to the architecture and base model it was designed alongside. A strong test would take the taxonomy-based SFT data (the 10.5% in-house portion from Table 2) and fine-tune a different VLM architecture (e.g., LLaVA-1.5, CogVLM, or a newer model) on it, then evaluate on the same human evaluation protocol. If the gains transfer, the taxonomy becomes a reusable resource—a specification for real-world VLM capability that any model can be trained toward. If the gains are specific to DeepSeek-VL's architecture or base model, it suggests the taxonomy captures interactions between data and model design that are not fully separable. The paper's human evaluation dataset (100 questions across 7 categories) and GPT-4V-judge protocol (99 samples) provide the evaluation infrastructure for such a study, though the sample sizes would need expansion for statistical reliability.

Ablating the taxonomy-based SFT data: does it actually matter? The paper claims that the in-house SFT data constructed from the use case taxonomy "substantially improves the model's user experience in practical applications" (Section 1) and that this data is "meticulously designed to reflect real-world usage scenarios" (Section 2.2). However, this data constitutes only 10.5% of the SFT mixture—the majority is public GPT-4V-generated data (35.5%) and text-only SFT (47.9%). The critical ablation is never performed: train an identical DeepSeek-VL-7B model with the exact same architecture, pretraining, and SFT recipe, but replace the in-house taxonomy data with an equal quantity of additional public GPT-4V data or captions. Then evaluate both models on the human evaluation benchmark (Figure 6) and the GPT-4V-judge comparison (Figure 7). If the taxonomy-trained model significantly outperforms, the taxonomy philosophy is validated as a data construction principle. If the gap is small or zero, the real driver of real-world performance is simply the quantity and quality of instruction tuning data, and the taxonomy is intellectual scaffolding rather than an actionable methodology. This is a high-value experiment because it directly tests the paper's central methodological claim.

Does the hybrid encoder design outperform a single high-resolution text-aligned encoder at equivalent token budget? The paper's architectural argument is that a CLIP-family encoder (SigLIP) cannot simultaneously provide semantic understanding and fine-grained structural detail because its contrastive training objective discards the low-level information needed for OCR and visual grounding. The solution is a hybrid: SigLIP for semantics, SAM (a self-supervised segmentation encoder) for structure. But the controlled comparison—hybrid encoder versus a single encoder that is (a) text-aligned, (b) operating at comparable or higher resolution, and (c) producing the same or similar token count—is absent. At the time of DeepSeek-VL's publication, such encoders were not widely available, but the landscape has evolved. A follow-up study could compare the hybrid SigLIP+SAM design against, for example, a fine-tuned SigLIP or EVA-CLIP variant operating at 768×768 resolution with an appropriate patch size to produce ~576 tokens, or against a modern high-resolution text-aligned encoder like DFN-CLIP or MetaCLIP at comparable resolutions. The evaluation should measure both standard benchmarks and targeted tests of the specific failure modes the hybrid design claims to address: OCR at small font sizes, fine-grained visual grounding (e.g., "click on the third button from the left"), and discrimination of CLIP-blind pairs. If the single-encoder baseline matches the hybrid on these tasks, the dual-encoder design is unnecessary complexity. If the hybrid maintains an advantage, the theoretical motivation—that contrastive alignment and structural encoding require fundamentally different training objectives—is empirically validated.

Can the competitive dynamics between vision and language be partially decoupled through architecture? The paper identifies a central tension: vision and language compete for the same parameters during joint training, creating a zero-sum dynamic at small scales. The paper mentions Mixture of Experts (MoE) as future work, but the research question is more specific than "try MoE." A targeted experiment would modify the DeepSeek-VL architecture so that the LLM's feed-forward layers are replaced with MoE layers where some experts are trained predominantly on text tokens and others predominantly on visual tokens (through data-routing during training, not hard assignment). The hypothesis is that modality-specialized experts would allow the model to allocate more total capacity to multimodal processing without displacing language-specific representations, shifting the optimal modality ratio rightward and reducing the GSM8K math decline observed at 7B. A strong study would sweep modality ratios with both the dense and MoE architectures at the same total parameter count and compare the ratio-performance curves. The prediction: the MoE curve should be flatter—language performance should degrade less at high multimodal ratios—because the modality capacity competition is partially resolved by expert specialization.

Stress-testing the compute-optimal strategy under distribution shift. All evaluations in the paper are in-distribution: the model is tested on problems from the same MATH benchmark distribution as the training data used for PRM and revision model training. A critical stress test for the compute-optimal framework is whether the difficulty estimation and strategy allocation generalize when the input distribution shifts. For example: does the difficulty estimator (trained on MATH) correctly bin problems from a different math dataset (GSM8K, AMC, or a translated MATH variant) into difficulty quintiles that predict which test-time strategy will work best? Does the revision model, trained on edit-distance-paired MATH solutions, transfer to math problems with different stylistic conventions or to adjacent reasoning domains (code debugging, logical deduction)? A negative result—that the difficulty bins and optimal strategies are dataset-specific—would limit the deployability of the approach to settings where in-distribution difficulty estimation is feasible. A positive result—that difficulty estimation transfers across datasets—would suggest the model is learning something about inherent problem difficulty rather than dataset-specific surface features. This experiment connects to a broader question the paper raises but does not answer: are the difficulty-dependent scaling patterns (beam search over-optimizing on easy problems, revisions helping most on easy problems) a property of mathematical reasoning specifically, or a general property of any multi-step reasoning task?

Practical Applications and Downstream Use Cases

On-device or edge deployment of multimodal assistants. DeepSeek-VL-1.3B, at 1.3 billion parameters, is small enough to run on consumer hardware or mobile devices with appropriate quantization. The paper shows this model achieves 63.5 on MMBench and 29.4 on MathVista—performance that, while below the 7B model, is competitive with or exceeds larger models (e.g., it matches Qwen-VL-Chat-7B's 13.2 on MathVista despite being 5× smaller). For applications where privacy, latency, or cost preclude cloud API calls—document scanning and summarization on a phone, real-time UI assistance for accessibility, offline image captioning in remote environments—the 1.3B model provides a practical deployment option. The fixed 576-token visual representation means the LLM's context window is not consumed by exploding vision token counts (which happens with tiling-based high-resolution approaches), making multi-turn image-grounded conversations feasible even within the limited context windows of small on-device models. The paper does not report inference latency or on-device benchmarks, but the architectural design (fixed token budget, no dynamic tiling, frozen SAM-B at inference) is explicitly oriented toward deployment efficiency.

Cost-efficient data annotation and content moderation pipelines. For organizations that need to process large volumes of images with textual understanding—e-commerce platforms verifying product descriptions against images, educational platforms grading handwritten math solutions, publishers extracting structured data from scanned documents—DeepSeek-VL-7B offers a middle ground between expensive proprietary APIs (GPT-4V, Gemini) and task-specific OCR or classification models that fail on ambiguous or open-ended inputs. The model's strong OCRBench performance (594, exceeding GPT-4V's 516) makes it particularly suited for document understanding tasks. The open-source availability means organizations can self-host, avoiding per-query API costs and data privacy concerns. A concrete deployment scenario: a legal document processing pipeline that uses DeepSeek-VL-7B to extract structured fields (parties, dates, clauses) from scanned contracts, with the model's text recognition validated by the OCRBench results and its structured extraction capability validated by the table/chart understanding benchmarks. The model's performance on conversion tasks (Image to Code, Image to Text in the taxonomy) further supports document digitization workflows.

Curriculum and educational content generation. The model's demonstrated ability to understand diagrams, read textbook content, and provide organized explanations (Figures 2, 5, 11-14) makes it a candidate for automated educational content generation. A specific use case: generating step-by-step solutions for geometry problems, where the model must both read the diagram (shapes, labels, angles) and produce mathematically correct reasoning. DeepSeek-VL-7B's 36.1 on MathVista is below GPT-4V (47.8) but well above other open-source 7B models, and the gap is attributed primarily to LLM scale rather than vision capability—suggesting that as larger DeepSeek-LLM backbones become available, the mathematical reasoning capability would improve without requiring architectural changes. The model's strong performance on Chinese-language benchmarks (MMBench-CN 71.9, CMMMU comparable to open-source competitors) further indicates suitability for bilingual educational applications in Chinese and English.

Self-improvement data generation with the use case taxonomy. The taxonomy-based data construction methodology is itself a reusable practical asset. An organization building a domain-specific VLM (e.g., for medical image interpretation, architectural diagram understanding, or manufacturing defect detection) can adopt the same workflow: (1) collect examples of how domain experts use existing VLMs for their tasks, (2) organize into a domain-specific taxonomy of use cases, (3) construct training data and evaluation sets from that taxonomy, (4) fine-tune DeepSeek-VL or another open-source VLM on the resulting data. The paper provides the template and demonstrates its effectiveness for general-domain use cases. A concrete instance: a radiology AI company collects examples of how radiologists prompt GPT-4V when analyzing chest X-rays, organizes them into categories (finding description, differential diagnosis, measurement extraction, comparison to prior studies), builds an instruction-tuning dataset with board-certified radiologist responses for each category, and fine-tunes DeepSeek-VL-7B. The resulting model benefits from both the general visual understanding from pretraining and the domain-specific task distribution encoded in the taxonomy-based SFT data.

When to Prefer This Method

The paper positions DeepSeek-VL as a general-purpose VLM for real-world applications, not as a method that dominates specific alternatives in narrow regimes. It does not articulate an explicit "prefer A when X, prefer B when Y" tradeoff against named competing architectures, training strategies, or data construction philosophies. Rather, the paper's contribution is an integrated system design whose components (hybrid encoder, joint training, taxonomy-based SFT) are presented as collectively addressing the real-world performance gap rather than as individually selectable options. The comparison tables (Table 5, Table 6) benchmark against other models but do not characterize the conditions under which each model family is preferable.

The closest the paper comes to a conditional recommendation is the scale-dependent finding around modality competition: the GSM8K decline of 9.7 points at 7B scale is attributed to limited model capacity, and the paper predicts that "larger models might alleviate this issue significantly." This implies—but does not state—that at very small scales (1-3B parameters), the modality competition may be severe enough that a frozen-LLM approach (like LLaVA) could be preferable for applications where language capability preservation is paramount and multimodal performance is secondary. At 7B and above, the paper's evidence suggests joint training with a 7:3 ratio is the better route. However, this is an extrapolation from the paper's data rather than a position the paper itself takes, and a decision matrix would be speculative without direct comparisons that the paper does not provide.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model2024· multimodal← in-domain predecessor
- Gemini vs GPT-4V: A Preliminary Comparison and Combination of Vision-Language Models Through Qualitative Cases2023· multimodal← in-domain predecessor
Gemma: Open Models Based on Gemini Research and Technology2024· llm-systems← topic thread
- World Model on Million-Length Video And Language With Blockwise RingAttention2024· llm-systems← in-domain predecessor
- OLMo: Accelerating the Science of Language Models2024· agents← topic thread
- Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context2024· context-optimization← topic thread
Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs2025· multimodal← similar topic
- Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution2024· multimodal← LLM-judged similar
- RL makes MLLMs see better than SFT2025· multimodal← in-domain predecessor
- Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation2026· multimodal← similar topic
- Open-Qwen2VL: Compute-Efficient Pre-Training of Fully-Open Multimodal LLMs on Academic Resources2025· multimodal← similar topic
- Mirasol3B: A Multimodal Autoregressive Model for Time-Aligned and Contextual Modalities2023· multimodal← similar topic
VADER: Towards Causal Video Anomaly Understanding with Relation-Aware Large Language Models2025· multimodal← similar topic
- NVIDIA Nemotron Nano V2 VL2025· multimodal← in-domain predecessor
- Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Marks2025· multimodal← similar topic
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models2023· multimodal← similar topic
- LLaVA: Visual Instruction Tuning2023· multimodal← in-domain predecessor
- GPT-4 Technical Report2023· pretraining← topic thread
- Adapting Vision-Language Models for E-commerce Understanding at Scale2026· multimodal← similar topic
- SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features2025· multimodal← similar topic

What it influenced

Later papers in the same chain. Direction reversed.

Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution2024· multimodal→ LLM-judged similar
- Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs2025· multimodal→ LLM-judged similar
- Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution2024· multimodal→ in-domain predecessor
- MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models2023· multimodal→ similar topic
Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models2024· multimodal→ in-domain predecessor
- NVLM: Open Frontier-Class Multimodal LLMs2024· multimodal→ LLM-judged similar
- InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD2024· multimodal→ in-domain predecessor
KAN: Kolmogorov-Arnold Networks2024· architecture→ topic thread
- Layer-Condensed KV Cache for Efficient Inference of Large Language Models2024· architecture→ in-domain predecessor
- Extending Llama-3's Context Ten-Fold Overnight2024· context-optimization→ topic thread
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation2022· multimodal→ similar topic
- Hierarchical Text-Conditional Image Generation with CLIP Latents2022· multimodal→ in-domain predecessor
- BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models2023· multimodal→ similar topic
- Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation2024· multimodal→ similar topic
- NextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generation2026· multimodal→ similar topic
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning2023· multimodal→ similar topic
- Read More2023· multimodal→ in-domain predecessor
- LLaVA: Visual Instruction Tuning2023· multimodal→ similar topic
- Improved Baselines with Visual Instruction Tuning2023· multimodal→ similar topic