Improved Baselines with Visual Instruction Tuning

ArXiv: 2310.03744

🎯 Pitch

A simple MLP connector, not complex visual resamplers, unlocks state-of-the-art multimodal performance using only 1.2M publicly available examples—trained in ~1 day on a single 8-A100 node. This overturns the assumption that massive pretraining data is needed, as the model beats InstructBLIP and Qwen-VL across 11 benchmarks while using up to 1,000× fewer pretraining pairs.

1. Executive Summary

This paper presents the first systematic study to investigate the design choices of large multimodal models (LMMs) under the LLaVA framework, analyzing how simple modifications—an MLP vision-language connector (replacing a linear projection layer) and incorporating academic-task-oriented VQA data with response formatting prompts (appending "Answer the question using a single word or phrase")—establish stronger baselines that achieve state-of-the-art across 11 benchmarks. The resulting model, LLaVA-1.5, uses merely 1.2M publicly available data and finishes training in ~1 day on a single 8-A100 node, yet outperforms competitors trained on orders of magnitude more data (e.g., InstructBLIP's 129M pretraining pairs and Qwen-VL's 1.4B pairs). Scaling the input resolution further via a grid-splitting strategy yields LLaVA-1.5-HD, which improves detailed perception and reduces hallucination, establishing that test-time resolution scaling can substitute for extensive vision-language pretraining only when the base model's architecture preserves data efficiency through simple design choices rather than complex visual resamplers.

2. Context and Motivation

The Core Problem: We Don't Know How to Build a Good LMM with Limited Resources

The fundamental question this paper tackles is deceptively simple: what is the minimal set of design choices needed to build a large multimodal model (LMM) that excels at both short-form academic benchmarks AND natural visual conversations? This matters because the field is in a peculiar state—we have many LMMs achieving impressive results, but we don't understand why they work differently, or which design decisions are actually responsible for their capabilities.

The landscape that motivated this work looks like this: LLaVA [36] excels at conversational-style visual reasoning, producing detailed, helpful responses when users ask open-ended questions about images. InstructBLIP [14] excels at VQA benchmarks that demand single-word or short answers. But neither does both well—LLaVA falls short on academic benchmarks requiring terse answers, and InstructBLIP overfits to short-answer formats, producing "yes" or "no" even when users request detailed explanations (Section 1, Table 1a). This is a capability split that makes neither model suitable as a general-purpose assistant.

The nature of this split is particularly frustrating because it's unclear which differences between these models cause the divergence. Is it architecture? Training data? The specific fine-tuning recipe? Each model differs along multiple axes simultaneously:

• Architecture: LLaVA uses a single linear layer to connect vision to language; InstructBLIP uses a Qformer—a visual resampler with hundreds of millions of trainable parameters pretrained on 129M image-text pairs.
• Training data: LLaVA trains on ~600K visual instruction examples (conversations, descriptions, reasoning); InstructBLIP adds academic VQA datasets like VQA-v2 but also uses its 129M pretraining pairs.
• What gets fine-tuned: LLaVA fine-tunes the entire LLM during instruction tuning; InstructBLIP only fine-tunes the Qformer, keeping the LLM frozen.
• Output format handling: LLaVA's training data contains natural conversational responses; InstructBLIP's includes short VQA answers alongside conversations, creating ambiguity about when to produce each format.

Any of these could explain the capability split. The prior literature offered conjectures—perhaps it's the amount of training data, perhaps the usage of Qformer—but no one had disentangled these factors in a controlled setting. This paper's central contribution is performing exactly that disentanglement: starting from LLaVA, systematically adding one modification at a time and measuring the effect across both academic and conversational benchmarks.

Why This Gap Matters

The problem has both practical and scientific dimensions.

Practical: The accessibility problem. The dominant approach to improving LMMs in 2023 was to scale up—more pretraining data, larger models, more expensive training pipelines. InstructBLIP pretrains its Qformer on 129M image-text pairs. Qwen-VL [3] pretrains on 1.4B pairs and includes in-house (non-public) data. IDEFICS [22] trains on 353M pairs. These approaches produce strong results but make it prohibitively expensive for most academic labs to participate in LMM research. If LMM development requires web-scale pretraining budgets, the field consolidates around a handful of industrial labs.

Section 1 states this concern explicitly: "We hope this makes state-of-the-art LMM research more accessible." The paper's explicit goal is not just to beat benchmarks but to democratize the ability to build strong LMMs. If the core finding holds—that a simple MLP connector with the right instruction-tuning data mixture can outperform complex resamplers trained on orders of magnitude more data—then the barrier to entry drops dramatically. Training LLaVA-1.5 takes ~1 day on a single 8-A100 node using only publicly available data. This is a qualitatively different regime from the multi-node, million-dollar pretraining runs that competitors require.

Scientific: The confounding problem. Without controlled studies, the field cannot make progress. If Model A beats Model B on Benchmark X, but A and B differ in architecture, pretraining data volume, instruction-tuning data mixture, fine-tuning procedure, and base LLM, the result is scientifically uninformative—it tells us nothing about why A outperformed B or what future models should prioritize. This makes every new LMM paper an isolated data point rather than a contribution to cumulative knowledge.

The paper explicitly frames itself as "the first systematic study to investigate the design choices of LMMs in a controlled setting" (Section 1). This is not marketing language—it's a precise description of the methodological gap the paper fills. Prior work compared entire systems; this paper holds the LLaVA framework constant and varies individual components, measuring each component's marginal contribution.

Theoretical: Understanding the role of visual instruction tuning vs. vision-language pretraining. An implicit debate runs through the LMM literature: how much vision-language alignment pretraining is actually necessary? BLIP-2, InstructBLIP, and Qwen-VL all invest heavily in this stage, training visual resamplers on hundreds of millions or billions of image-text pairs before any instruction tuning. LLaVA uses a radically simpler approach—a single linear projection trained on just 600K pairs—yet achieves comparable or better conversational performance. This raises a provocative question that Section 4.2 makes explicit: "raises questions upon the common belief that LMMs require significant amount of vision-language alignment pretraining... despite that the vision encoders (e.g. CLIP, OpenCLIP, EVA-CLIP) are already pretrained on web-scale image-text paired data."

If the vision encoder already maps images to a semantically meaningful space (courtesy of CLIP's web-scale pretraining), maybe the cross-modal connector just needs to translate that space into the LLM's embedding space—a simple geometric transformation that a linear layer can learn from modest data. Complex resamplers like Qformer might be solving a harder problem than necessary, or worse, might introduce optimization challenges that require massive pretraining data to overcome.

The paper's empirical results support this interpretation: an MLP projection trained on 558K pairs outperforms Qformer trained on 129M pairs. This is not just a practical win—it's conceptual evidence that the vision-language alignment problem is simpler than the field assumed, at least when building on top of strong pretrained vision encoders and when the downstream task is instruction following rather than retrieval or generation.

Where Prior Approaches Fall Short

The paper identifies specific limitations in four categories:

1. Inability to balance short-form and long-form answering.

The most visible failure mode is InstructBLIP's tendency to produce short answers even when users request detailed explanations (Table 1a). When asked "Is this unusual? Please explain in detail," InstructBLIP responds with "yes." The paper identifies two root causes (Section 3.2):

• Ambiguous prompts on the response format. VQA datasets typically use formats like Q: {Question} A: {Answer} with no explicit instruction about answer length. When such data is mixed with conversational data during training, the model cannot learn when to be verbose versus concise—the prompt format provides no signal.

• Not fine-tuning the LLM. InstructBLIP only fine-tunes the Qformer during instruction tuning; the LLM (Vicuna) is frozen. The Qformer's visual output tokens must therefore control the LLM's output length (long vs. short), a form of prefix tuning. But the Qformer has limited capacity compared to the LLM and may simply lack the ability to modulate output length reliably across diverse instructions. The frozen LLM has strong priors from its pretraining that may conflict with the Qformer's attempted control.

This is a concrete, testable diagnosis: if the problem is ambiguous prompts + frozen LLM, then (a) explicit format prompts + (b) fine-tuning the LLM should fix it. The paper demonstrates exactly this in Table 2, where adding format prompts boosts MME performance from 1197.0 to 1323.8 while maintaining conversational capability.

2. Over-engineering the vision-language connector.

InstructBLIP's Qformer and Qwen-VL's visual resampler are complex modules with hundreds of millions of parameters. They require massive pretraining to converge. Yet the paper shows that a simple 2-layer MLP—with far fewer parameters and far less pretraining data—outperforms them. This suggests these complex connectors are solving a self-imposed problem: they attempt to compress visual features into a small number of "visual tokens" to reduce sequence length (and thus LLM inference cost), but the compression itself introduces an optimization challenge that requires massive data to overcome.

More fundamentally, these resamplers may be throwing away information. The paper's approach keeps all visual patches, letting the LLM attend to the full visual feature map. On tasks requiring fine-grained visual perception (OCR, detailed description, object localization), this full-attention approach has an inherent advantage. The resampler's compression is a tradeoff (efficiency vs. information preservation) that may not be worth making, especially when training efficiency is the bottleneck rather than inference latency.

3. Data scaling without corresponding capability scaling.

Section 5.2 presents a nuanced finding about hallucination. The LLaVA-Instruct dataset contains some hallucinated content in its detailed descriptions. The natural assumption is that training on hallucinated data causes models to hallucinate. But the paper finds that scaling input resolution to 448² (via LLaVA-1.5-HD) significantly reduces hallucination, even though the training data contains the same hallucinated examples. Their interpretation: "when the input resolution is not sufficient for the model to discern all details in the training data, and the amount of data that is at that granularity beyond the model's capability becomes large enough, the model learns to hallucinate."

This inverts the conventional wisdom. It's not just about cleaning the training data—it's about matching data granularity to model capability. If you train on detailed descriptions but your model can't actually see the details (because resolution is too low), the model learns to fabricate details to match the expected output distribution. This is a fundamental insight about LMM training that prior work, focused on data quality alone, missed.

4. Lack of controlled comparisons.

This is the meta-problem. Every prior LMM paper introduces a complete system—architecture, pretraining data, instruction data, training recipe—and compares against other complete systems. Differences in any component could explain performance differences. The field lacked a study that:

• Starts from a single, well-understood baseline (LLaVA)
• Modifies one component at a time
• Evaluates on both academic and conversational benchmarks
• Reports the marginal contribution of each change

Without this, the literature is a collection of anecdotes. The paper's "road map" (Section 1) and the systematic scaling in Table 2 directly address this gap.

How This Paper Positions Itself

The paper positions itself not as proposing a fundamentally new architecture or training paradigm, but as performing the controlled ablation study the field needed. Its contributions are methodological and empirical rather than architectural:

As a systematic study. The paper explicitly frames itself as "the first systematic study to investigate the design choices of LMMs in a controlled setting under the LLaVA framework" (Section 1). This is the primary contribution claim. The paper does for LMMs what controlled ablation studies did for self-supervised visual representation learning (e.g., Chen et al.'s SimCLR systematically studying projection heads, batch size, and augmentations [9]): it isolates variables and measures their effects.

As a rebuttal to the "scale everything" trend. Figure 1 directly contrasts LLaVA-1.5's data efficiency against competitors: 1.2M total training samples vs. 129M (InstructBLIP), 50M (Qwen-VL-Chat), and 1400M (BLIP-2). The paper's position is that data quality and smart design choices matter more than data quantity, at least for the current generation of LMMs. This aligns with findings in NLP (e.g., LIMA [61] showing that 1,000 carefully curated examples can outperform large-scale instruction tuning) but extends it to the multimodal domain with systematic evidence.

As a bridge between academic and conversational capabilities. The paper identifies the multitask balancing problem (short-form VQA vs. long-form conversation) as solvable with simple prompt engineering + LLM fine-tuning, contradicting prior work [7] that found naive VQA data merging causes models to lose conversational ability. The key insight—that explicit format prompts combined with LLM fine-tuning enable the model to switch between output modes based on the instruction—is simple in retrospect but was not demonstrated in prior work, which either avoided VQA data entirely (LLaVA) or accepted degraded conversation ability (InstructBLIP).

As a foundation for studying open problems. The paper positions LLaVA-1.5 not as a final system but as a reproducible baseline that makes it feasible to study harder problems: higher resolution (Section 3.4), hallucination (Section 5.2), compositional generalization (Section 5.3), and data efficiency (Section 5.1). By establishing that strong performance is achievable with simple architecture and modest compute, the paper enables future work that would be prohibitively expensive if complex resamplers and massive pretraining were required.

What the paper explicitly does NOT claim. The paper does not claim to solve hallucination, achieve multimodal AGI, or replace larger-scale approaches in all regimes. The limitations section acknowledges prolonged training for high-resolution images, lack of multiple-image understanding, limited problem-solving in certain domains, and continued susceptibility to hallucination. This is a paper about establishing strong, affordable, and scientifically informative baselines—not about pushing the frontier of maximum possible performance at any cost.

3. Technical Approach

3.1 Reader Orientation

The LLaVA-1.5 system is a large multimodal model that takes an image and a text instruction as input and produces a text response, trained through a two-stage process of vision-language alignment pretraining followed by visual instruction tuning. The core problem it solves is how to build a model that can simultaneously excel at both short-form academic benchmarks (where the model must produce single-word or short-phrase answers to visual questions) and open-ended visual conversations (where the model must produce detailed, helpful descriptions and reasoning), using only publicly available data and modest computational resources, by making careful, incremental design choices rather than scaling data volume or model complexity.

3.2 Big-Picture Architecture (Diagram in Words)

The LLaVA-1.5 system has four major components arranged in a feedforward pipeline:

1. Vision Encoder (CLIP-ViT-L-336px): A frozen pretrained vision transformer that takes an input image of resolution 336×336 pixels and produces a grid of visual feature vectors—one per 14×14 pixel patch—capturing the semantic content of the image.

2. Vision-Language Connector (2-layer MLP): A trainable multi-layer perceptron that takes the flattened grid of visual feature vectors from the vision encoder and projects each one into the embedding space of the language model. This is the only component that bridges the vision and language modalities, translating CLIP's visual representations into tokens the language model can process as if they were words.

3. Large Language Model (Vicuna-13B): A pretrained instruction-tuned language model that receives the projected visual tokens interleaved with the text instruction tokens and autoregressively generates a text response. During visual instruction tuning, the LLM's parameters are fully updated, allowing it to learn to attend to and reason over visual information.

4. Training Data Mixture (665K examples): A collection of datasets spanning VQA, OCR, region-level perception, visual conversations, and text-only conversations, each with explicit response formatting prompts that teach the model when to produce short answers versus detailed responses.

Information flows as follows: an image enters the system → the frozen CLIP vision encoder produces a grid of patch features → the MLP connector projects each patch feature into the LLM's embedding space → these projected visual tokens are concatenated with the text instruction tokens → the Vicuna LLM processes the combined sequence and autoregressively generates a response token by token.

For the high-resolution variant (LLaVA-1.5-HD), the image is first split into 224×224 patches, each encoded independently by the CLIP encoder, then the resulting feature maps are merged back into a single large feature map. A downsampled global view of the entire image is also encoded and concatenated to provide context about the overall image layout.

3.3 Roadmap for the Deep Dive

• First, the LLaVA baseline and the multitask balancing problem, because understanding what LLaVA could and could not do—and why InstructBLIP failed at conversations—establishes the design constraints that every subsequent modification addresses.
• Second, the response formatting prompt mechanism, because it is the simplest intervention (zero architectural changes) that solves the short-form vs. long-form output format ambiguity, and it is a prerequisite for incorporating academic VQA data without degrading conversational ability.
• Third, the vision-language connector upgrade from linear projection to MLP, along with the scaling of the vision encoder resolution from 224² to 336², because these architectural changes improve the model's raw visual perception capacity and are orthogonal to the data-side improvements.
• Fourth, the full training data mixture of 665K examples (academic VQA, OCR, region-level data, visual conversations, text-only conversations), because the data mixture determines what capabilities the model can acquire during instruction tuning, and the paper's composition of this mixture is a deliberate design choice.
• Fifth, the LLaVA-1.5-HD high-resolution extension, because it introduces a novel inference-time scaling strategy (grid-splitting with global context) that avoids the need for positional embedding interpolation or additional pretraining, while improving fine-grained perception and reducing hallucination.
• Sixth, the training procedure and hyperparameters, which tie together the full pipeline and explain how the model achieves its data efficiency and training speed.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and empirical analysis paper whose core idea is that a simple architecture (CLIP encoder + MLP connector + full LLM fine-tuning) with carefully designed training data (explicit format prompts + diverse task mixture) can outperform much more complex approaches (visual resamplers, massive pretraining, frozen LLMs) by avoiding self-imposed optimization bottlenecks.

The LLaVA Baseline and the Multitask Balancing Problem

The starting point for LLaVA-1.5 is the original LLaVA [36], which established a minimal architecture for visual instruction tuning: a CLIP ViT-L/14 vision encoder at 224×224 resolution, a single linear projection layer to map visual features into the LLM's word embedding space, and the Vicuna language model fine-tuned end-to-end on 158K visual instruction examples (conversations, detailed descriptions, and complex reasoning questions generated by text-only GPT-4 from COCO images).

The original LLaVA demonstrates strong performance on conversational benchmarks that evaluate open-ended visual reasoning—it produces coherent, detailed, helpful responses when users ask questions like "What is unusual about this image?" or "Describe this scene in detail." On LLaVA-Bench-in-the-Wild [36], which uses GPT-4 to judge response quality on diverse real-world images, LLaVA achieves 62.8% (7B model, Table 4) and on MM-Vet [55], which evaluates integrated capabilities across recognition, knowledge, spatial awareness, language generation, and math, LLaVA achieves 25.5% (7B model, Table 4).

However, the original LLaVA has two critical limitations that prevent it from being evaluated on—or performing well on—academic VQA benchmarks:

Limitation 1: Inability to produce short-form answers. LLaVA's training data consists entirely of conversational-style responses generated by GPT-4. These are naturally verbose because GPT-4, when asked to describe an image or answer a question about it, produces complete sentences with explanation. For example, if the question is "What color is the shirt?", LLaVA would generate "The man is wearing a yellow shirt" rather than simply "Yellow." This makes it impossible to fairly evaluate LLaVA on datasets like VQA-v2 [19] or GQA [21] that use exact string matching against short ground-truth answers (e.g., "yellow"). The paper notes that "it is challenging to evaluate the original LLaVA on academic datasets like VQA-v2 that demand open-ended short answers" (Section 4.2).

Limitation 2: Yes/no question bias. Because LLaVA's training data lacks yes/no questions (the GPT-4 generated conversations rarely contain binary-answer questions), the model has no exposure to the format where a brief "yes" or "no" is the appropriate response. When faced with such questions at test time, LLaVA tends to default to "yes" regardless of the image content, as noted in Section 3.1: "tends to answer yes for yes/no questions due to the lack of such data in the training distribution."

The InstructBLIP approach attempts to solve these limitations by incorporating academic VQA datasets (VQA-v2, GQA, etc.) directly into the instruction tuning data. However, this creates a new problem: the model now sees two contradictory types of responses during training—short answers from VQA datasets (e.g., "yellow") and long conversational responses from LLaVA-Instruct (e.g., "The man is wearing a yellow shirt, and he appears to be standing on..."). The model must infer from context which format to produce, but the training data provides ambiguous signals.

The paper identifies two specific mechanisms that cause InstructBLIP's failure (Section 3.2):

First, ambiguous prompt formatting. VQA datasets in InstructBLIP are formatted simply as Q: {Question} A: {Answer} with no explicit instruction about the desired response format. When the same model also sees conversational prompts like "Is this unusual? Please explain in detail" followed by a paragraph-long response, the prompt alone does not disambiguate whether "Q: [Question] A:" expects a single word or a detailed explanation. The model learns a statistical shortcut: because the Q: ... A: format appears most frequently with short answers, the model defaults to short answers even when the conversational prompt explicitly requests detail. Table 1a shows the concrete failure: when InstructBLIP is asked "Is this unusual? Please explain in detail," it responds with just "yes."

Second, freezing the LLM during instruction tuning. InstructBLIP only fine-tunes the Qformer (the visual resampler) during instruction tuning, keeping the LLM (Vicuna) frozen. This means the Qformer's output tokens—typically 32 learned query tokens that attend to visual features—must encode not only the visual content but also a control signal for the LLM that determines output length and format. This is a form of prefix tuning [33], where a small set of learned vectors prepended to the input conditions the frozen LLM's generation behavior. But the Qformer has limited capacity (it is approximately the size of a BERT-base model, roughly 110M parameters) compared to the 7B or 13B Vicuna models, and may simply lack the representational capacity to reliably control output format across the diverse range of instructions users might provide. The Qformer's 32 visual tokens must simultaneously represent fine-grained visual information AND modulate generation length—a compression that likely fails on one or both dimensions.

The paper's diagnosis suggests a simple fix: if the problem is ambiguous prompts + lack of LLM fine-tuning, then (a) make prompts explicit about desired format and (b) fine-tune the LLM so it can learn to modulate its own output based on the instruction rather than relying on a bottlenecked prefix. This is exactly what the response format prompting mechanism addresses.

Response Format Prompting

The key insight of response format prompting (Section 3.2) is that the ambiguity between short-form and long-form answering can be resolved by making the desired output format an explicit part of the instruction text, and then fine-tuning the LLM on data that contains these format instructions, so the model learns to attend to the format specification and modulate its generation accordingly.

The mechanism works through a simple data preprocessing step: for every VQA training example, the authors append a short natural language instruction to the end of the question that explicitly specifies the expected answer format. The specific formulation used is:

"Answer the question using a single word or phrase."

This is appended directly to the VQA question. So instead of the ambiguous format:

Q: What color is the shirt? A:

The training example becomes:

What color is the shirt? Answer the question using a single word or phrase.

with the expected response being simply "Yellow."

Similarly, for multiple-choice datasets (A-OKVQA, ScienceQA), a different format prompt is used:

"Answer with the option's letter from the given choices directly."

This tells the model to produce something like "A" or "B" rather than a full sentence justifying the choice.

For OCR-based captioning datasets (TextCaps), the instruction is:

"Provide a one-sentence caption for the provided image."

Table 7 in the appendix provides the complete mapping of datasets to their response format prompts.

Why this works. The paper demonstrates through a zero-shot experiment (Table 1b) that the model can already partially generalize to format instructions even before visual instruction tuning—i.e., after only the first-stage vision-language alignment pretraining. When prompted with "What is the color of the shirt that the man is wearing? Answer the question using a single word or phrase," the pretrained model already responds with "Yellow" (the correct short answer) rather than the verbose "The man is wearing a yellow shirt" that it produces without the format prompt. This indicates that:

1. The base LLM (Vicuna) already possesses the capability to follow explicit format instructions from its text-only instruction tuning—it can produce short or long responses based on the prompt.
2. The vision-language alignment pretraining successfully transfers this capability to the multimodal setting—the model can follow format instructions even when the question involves visual content.
3. What was missing was simply the training signal to reinforce this behavior during visual instruction tuning, so the model reliably produces format-appropriate responses across all evaluation benchmarks.

Why this is sufficient where InstructBLIP failed. The paper does not just assert that format prompts work—it explains why they solve both failure modes that plagued InstructBLIP:

• Against the ambiguous prompts problem: the format instruction is now part of the question text itself, visible to both the vision-language connector (which sees it as part of the input sequence) and the LLM (which processes it through its full self-attention mechanism). The model does not need to infer format from an opaque prefix; it sees an explicit natural language command.

• Against the frozen LLM problem: because LLaVA-1.5 fine-tunes the entire LLM during instruction tuning, the model's attention mechanisms and generation policy can be updated to reliably attend to and obey the format instruction. The LLM learns to adjust its output distribution—producing short answers when the format prompt says "single word," producing detailed explanations when no such constraint is present—through standard autoregressive language modeling fine-tuning.

Validation in Table 2. The paper provides quantitative evidence for the format prompt's contribution by adding VQA-v2 data without and with the format prompt, measuring three representative benchmarks: GQA (short-form VQA), MME (yes/no perception questions), and MM-Vet (natural visual conversations). The results tell a clear story:

• Row 1 (LLaVA baseline + VQA-v2, no format prompt): MME score = 1197.0
• Row 2 (LLaVA baseline + VQA-v2 + format prompt): MME score = 1323.8

The 126.8-point improvement on MME comes solely from adding explicit format instructions to the training data—the dataset content is identical. This is strong evidence that the format ambiguity, not the data content, was the bottleneck. Moreover, MM-Vet (conversational benchmark) remains stable at 26.3 (vs. 25.5 baseline), confirming that adding VQA data with format prompts does not degrade conversational capability.

Generalization to unseen format instructions. Section 4.3 demonstrates an emerging property: the model generalizes to format instructions it was never explicitly trained on. The VizWiz dataset [20] includes questions that are unanswerable from the provided image (e.g., the image is too dark, blurry, or doesn't contain the queried object), and the expected response is "Unanswerable." The authors use a format prompt not seen during training: "When the provided information is insufficient, respond with 'Unanswerable'. Answer the question using a single word or phrase." The model correctly produces "Unanswerable" for 67.8% of unanswerable questions (up from 11.1% without the format prompt), demonstrating that it has learned the general skill of reading and following a format instruction rather than merely memorizing specific format prompt templates. Similarly, Table 6 shows the model following a complex constrained JSON output format—extracting structured information from a driver's license image—despite never being trained on JSON-format generation tasks.

MLP Vision-Language Connector and Resolution Scaling

The original LLaVA uses a single linear projection layer to connect vision to language: given a visual feature vector $v \in \mathbb{R}^{d_{\text{vision}}}$ from the CLIP encoder, the projection computes $v' = W v$, where $W \in \mathbb{R}^{d_{\text{vision}} \times d_{\text{language}}}$ is a learned weight matrix. This is a simple affine transformation with no non-linearity, making it equivalent to a single fully-connected layer with no activation function.

The upgrade to MLP (Section 3.3). LLaVA-1.5 replaces the linear projection with a two-layer multi-layer perceptron (MLP) with a non-linear activation function between the layers:

$v_{\text{hidden}} = \text{GELU}(W_1 v + b_1)$

$v' = W_2 v_{\text{hidden}} + b_2$

where $W_1 \in \mathbb{R}^{d_{\text{vision}} \times d_{\text{hidden}}}$, $W_2 \in \mathbb{R}^{d_{\text{hidden}} \times d_{\text{language}}}$, and $b_1, b_2$ are learned bias vectors. The GELU (Gaussian Error Linear Unit) activation function is the same non-linearity used in the transformer layers of both the vision encoder and the language model, providing a smooth, differentiable approximation to the ReLU.

What this computes. The MLP transforms each visual patch feature through two successive learned linear projections separated by a non-linearity. The first projection expands (or preserves) the dimensionality and applies GELU, creating an intermediate representation that can model non-linear interactions between visual feature dimensions. The second projection maps this intermediate representation into the exact dimensionality of the LLM's token embedding space. This is the same architectural pattern used in the feed-forward networks within transformer blocks, where a two-layer MLP with an expansion factor is applied to each token representation independently.

Why MLP instead of linear. The paper's motivation draws on findings from self-supervised visual representation learning [9, 10], where replacing a linear projection head with an MLP projection head consistently improves the quality of learned representations. The intuition is that a linear projection can only learn a rotation and scaling of the visual feature space—it cannot recombine features in a non-linear way. An MLP with a non-linearity can learn more complex transformations, potentially disentangling visual concepts that are linearly entangled in the CLIP representation. For example, a linear projection might struggle to simultaneously map "red car" and "red apple" to regions in the LLM's embedding space that preserve both the color concept ("red") and the object concept ("car" vs. "apple") in a way the language model can easily access. An MLP could learn feature interactions that separate these dimensions.

Empirical validation. Table 2, Row 3 (+MLP VL connector), shows that adding the MLP connector improves MME from 1323.8 to 1355.2 (+31.4 points), GQA from 46.8 to 47.3 (+0.5), and MM-Vet from 26.3 to 27.8 (+1.5). These are consistent but modest improvements, suggesting the MLP provides additional representational capacity without introducing optimization difficulties.

Scaling the vision encoder resolution (Section 3.3, Row 6). The original LLaVA uses CLIP-ViT-L at 224×224 pixel input resolution, producing 16×16 = 256 visual patches (each 14×14 pixels, since ViT-L/14 uses a patch size of 14). LLaVA-1.5 upgrades to CLIP-ViT-L-336px, the highest resolution publicly available CLIP checkpoint, which accepts 336×336 pixel inputs and produces 24×24 = 576 visual patches. This is a simple model swap—the vision encoder is still frozen and pretrained, just at a higher resolution.

The increase from 256 to 576 visual tokens means the LLM receives 2.25× more visual tokens per image. Since the LLM's self-attention cost scales quadratically with sequence length, this significantly increases both training and inference computation (the paper notes training is "∼2× as long as LLaVA"). However, the resolution increase is necessary for tasks requiring fine-grained visual perception.

Empirically (Table 2, Row 6), scaling to 336² resolution improves GQA from 50.3 to 51.4 and MME from 1426.5 to 1450. MM-Vet slightly decreases from 30.8 to 30.3, suggesting that for conversation-focused benchmarks, the additional visual tokens may not always help—the model may need to learn to effectively attend to the larger visual context, a capability that improves when the LLM is scaled to 13B.

Academic-Task-Oriented Data Mixture

The paper's data strategy is to incorporate diverse academic VQA datasets into the visual instruction tuning stage, each with a specific response format prompt that teaches the model the expected output style for that task type. This is described in Section 3.3 and detailed in Appendix A.2 (Tables 7 and 8). The complete instruction-tuning data mixture totals 665K examples.

Dataset composition. The mixture is constructed from eight data sources, each serving a distinct purpose:

1. LLaVA-Instruct [36] (158K examples): The original visual instruction data generated by GPT-4 from COCO images, containing three types of examples—conversations (multi-turn QA about an image), detailed descriptions (comprehensive paragraph-length image descriptions), and complex reasoning (questions requiring step-by-step inference). This data teaches the model to engage in natural, helpful visual conversations. No format prompt is used because the expected response is conversational.

2. ShareGPT [46] (40K examples): Text-only multi-turn conversations collected from users interacting with ChatGPT, covering diverse topics and including multilingual exchanges. Despite containing no images, this data serves two critical functions: (a) it teaches the model to produce longer, more detailed responses in general, which transfers to visual conversations (Section 5.3), and (b) it provides multilingual training signal—the model learns to respond in whichever language the user writes in, a capability that transfers to visual conversations even though all visual instruction data is English-only. The paper filters out invalid conversations (following Vicuna's preprocessing [12]) and truncates, rather than splits, conversations longer than 2048 tokens.

3. VQA-v2 [19] (83K examples): The canonical visual question answering dataset with open-ended questions about COCO images. Each question has multiple human-annotated short answers (typically 1-3 words). Format prompt: "Answer the question using a single word or phrase."

4. GQA [21] (72K examples): A compositional VQA dataset that tests relational reasoning (e.g., "What color is the object to the left of the red car?"). Format prompt: "Answer the question using a single word or phrase."

5. OKVQA [41] (9K examples): Open-knowledge VQA that requires external knowledge beyond the image content (e.g., "What brand is this phone?"). Format prompt: "Answer the question using a single word or phrase."

6. A-OKVQA [45] (66K examples): An augmented version of OKVQA that provides multiple-choice options and requires commonsense reasoning about the image. This dataset is converted to multiple-choice format, and each QA pair is augmented $k$ times (where $k$ is the number of answer choices per question) to balance the representation of multiple-choice data in the mixture. Format prompt: "Answer with the option's letter from the given choices directly."

7. OCR datasets (OCRVQA [42]: 80K examples, TextCaps [47]: 22K examples): OCRVQA tests the model's ability to read text in images (e.g., book covers, signs, product labels) and answer questions about it. The paper samples 80K conversations from the full OCRVQA dataset. TextCaps requires generating captions that incorporate text present in the image (e.g., "A sign that says 'Fresh Produce' above a basket of apples"). Format prompt for both: "Answer the question using a single word or phrase" for OCRVQA and "Provide a one-sentence caption for the provided image" for TextCaps.

8. Region-level VQA (Visual Genome [25]: 86K examples, RefCOCO [24, 40]: 48K examples): These datasets teach the model to understand spatial references and localize objects within images. Visual Genome provides dense annotations of objects, attributes, and relationships in images, and the paper samples 10 annotations per image. RefCOCO provides referring expressions—natural language descriptions that uniquely identify a specific object in an image (e.g., "the red cup on the left side of the table"). The format prompts alternate randomly between "Provide a short description for this region" (for region descriptions) and "Provide the bounding box coordinate of the region this sentence describes" (for localization). The RefCOCO conversations are split into segments of fewer than 10 conversations each to manage sequence length.

Data preprocessing strategies. The appendix (A.2) describes several efficiency optimizations:

• Merging QA pairs: For all VQA datasets, multiple questions about the same training image are merged into a single multi-turn conversation. This reduces the number of separate training examples while providing the model with a more natural interaction pattern—it sees multiple questions about the same image in sequence, simulating a user asking follow-up questions.

• Filtering and truncation: ShareGPT conversations are filtered to remove invalid ones (e.g., empty, nonsensical) using Vicuna's preprocessing pipeline [12]. Long conversations exceeding 2048 tokens are truncated rather than split into multiple examples (the authors note this differs from Vicuna's approach, which splits long conversations).

• Counterbalancing: A-OKVQA examples are replicated to ensure each multiple-choice option appears equally often as the correct answer across the training set, preventing the model from learning a positional bias (e.g., always selecting option "A").

• Modality-alternating batches: During training, the authors observe that text-only conversations (ShareGPT) are often longer than visual conversations. To speed up training, each batch is sampled from only a single modality (all visual or all text-only), which increases training throughput by 25% compared to mixed-modality batches without affecting final performance. This is a purely computational optimization—the model still sees both modalities equally over the course of an epoch.

• Uniform sampling: All data sources are concatenated and sampled with equal probability during training, meaning each of the 665K examples has the same chance of being selected in any given training step. This is a deliberate choice to avoid any dataset dominating the training signal.

Why this composition works. The data mixture is designed to provide complementary capabilities that transfer compositionally (Section 5.3). The VQA datasets teach the model to extract specific information from images and produce concise answers, the OCR datasets teach text reading, the region-level datasets teach spatial grounding, the LLaVA-Instruct data teaches open-ended visual reasoning, and the ShareGPT data teaches general instruction-following and multilingual capabilities. The paper argues these capabilities are not isolated—for example, the detailed visual knowledge from VQA datasets improves the factual accuracy of conversational responses, and the language fluency from ShareGPT improves the quality of visual descriptions.

Scaling to Higher Resolution: LLaVA-1.5-HD

Section 3.4 and Appendix A.1 introduce LLaVA-1.5-HD, an extension that scales LLaVA-1.5 to higher input resolutions without requiring additional vision-language pretraining or positional embedding interpolation.

The motivation. The 336² resolution of the highest available CLIP checkpoint limits the model's ability to perceive fine details. This is particularly problematic for tasks involving text reading (OCR), small object recognition, or detailed scene understanding where information is encoded at a granularity finer than 336×336 pixels can capture. Increasing resolution further requires a strategy beyond simply swapping in a higher-resolution CLIP encoder, because no such encoder is publicly available.

Why standard approaches are insufficient. The conventional approach to higher-resolution vision transformers is positional embedding interpolation [3, 32]: take a ViT trained at 224², interpolate its learned positional embeddings to a higher resolution (e.g., 448²), and then fine-tune the ViT backbone at the new resolution on large-scale image-text data so it adapts to the interpolated positions. This has two drawbacks that the paper seeks to avoid:

1. It requires large-scale fine-tuning data. The ViT backbone must be adapted to the new resolution through training on image-text pairs. Qwen-VL [3] uses 1.4B image-text pairs for this purpose. This is exactly the kind of expensive pretraining the paper argues is unnecessary.

2. It locks the model to a fixed resolution. After interpolation and fine-tuning, the model expects inputs of exactly the new resolution. Inference at any other resolution requires re-interpolation and likely degrades performance. There is no flexibility to handle images of varying aspect ratios naturally.

The LLaVA-1.5-HD solution: split-encode-merge. Instead of adapting the vision encoder to a new resolution, LLaVA-1.5-HD keeps the encoder fixed at its native 224² resolution and handles higher-resolution inputs by splitting the image into patches, encoding each patch independently, and merging the resulting features. This is illustrated in Figure 2.

The complete pipeline proceeds as follows:

Step 1: Target resolution selection. The input image is first padded (if necessary) to fit one of a predefined set of resolution configurations that support up to 6 grids. The supported configurations are: 1×1, 1×2, 1×3, 1×4, 1×5, 1×6, 2×2, 2×3, and their transposes (e.g., 2×1, 3×1). This constrains the maximum input resolution to 672×448 or 448×672 (since each grid cell is 224²). The selection algorithm enforces two criteria:

• Detail preservation: the chosen resolution should capture as much detail from the original image as possible—i.e., it should not downsample an image that natively has more resolution than the selected grid.
• Resource efficiency: the resolution should not be excessively large relative to the input—e.g., it should not select a 6-grid resolution for a low-resolution image that would be adequately represented by 2 grids.

This adaptive selection means different images can be processed at different resolutions based on their content, unlike the fixed-resolution approach of interpolated ViTs.

Step 2: Splitting. The padded image is divided into a grid of 224×224 patches. If the target configuration is 2×3, the image is split into 6 patches of 224×224 pixels each. Each patch is then independently encoded by the frozen CLIP-ViT-L-224px vision encoder, producing a feature map of 16×16 = 256 visual tokens per patch. Crucially, this means each patch's visual features are computed with the exact same positional embeddings the encoder was trained with—there is no interpolation, no distribution shift, and no need for encoder fine-tuning.

Step 3: Merging. The 16×16 feature maps from each patch are arranged into a single large feature map corresponding to the original image layout. For a 2×3 grid, this produces a 32×48 feature map (2×16 = 32 height, 3×16 = 48 width). This merged feature map is then processed to prepare it for the LLM.

Step 4: Postprocessing. Three postprocessing operations are applied to the merged feature map to make it efficient and informative for the LLM:

• Padding removal: features corresponding to padded regions of the image (from Step 1) are discarded. If the original image is 500×300 pixels and was padded to 672×448 (a 3×2 grid), the features for the padded area are removed, reducing the number of visual tokens the LLM must process.

• Row-end tokens: a special separator token is appended to the end of each row of features. This provides the LLM with explicit information about the 2D layout of the feature map—it can distinguish where one row ends and the next begins, which is important for spatial reasoning. Without this, the flattened 1D sequence of visual tokens loses all spatial structure.

• Flattening: the 2D feature map (with row-end tokens) is flattened into a 1D sequence of visual tokens suitable for input to the transformer-based LLM.

Step 5: Global context addition. A critical design choice is the inclusion of a global context image. In addition to the high-resolution patches, the entire original image is resized to a single 224×224 view, encoded by the same CLIP encoder, and its 256 visual tokens are concatenated with the high-resolution patch features. This serves two purposes:

• Context provision: the global view gives the LLM a holistic understanding of the entire scene, compensating for the fragmentation introduced by splitting the image into independent patches. Without this, the LLM sees 6 separate 224×224 patches but has no representation of how they fit together spatially beyond the row-end tokens.

• Artifact reduction: the split-encode-merge process can introduce boundary artifacts—objects that straddle patch boundaries are split across two feature maps, and the encoder processes each half independently, potentially losing the relationship between them. The global view provides a complementary representation where these boundary objects are intact, helping the LLM recover the correct spatial relationships.

The paper reports an ablation (Section 4.2, "Global context") showing that adding the global context image to the high-resolution patches improves GQA by +0.9 (62.9→63.8), MME by +71.7 (1425.8→1497.5), and MM-Vet by +3.2 (31.9→35.1) on a 7B model. The large improvement on MM-Vet (a conversational benchmark) suggests the global context is particularly important for tasks requiring coherent scene-level reasoning.

Training for LLaVA-1.5-HD. A key advantage of the split-encode-merge approach is that it requires no additional pretraining. The vision encoder processes only native 224² patches—exactly the images it was trained on—so the visual features are high-quality from the start. The MLP vision-language connector was pretrained on 224² features during the first-stage alignment pretraining, but since each 224² patch is encoded identically to how training images were encoded, the connector can process high-resolution patch features without modification. The paper states: "since we compute the visual features on the original 224² resolution that the vision encoder is trained on, we do not perform additional pretraining. We also do not perform additional high resolution pretraining for the visual projectors, and perform visual instruction tuning directly on the higher-resolution images" (Appendix A.1.2).

This is significant because it means the data efficiency of LLaVA-1.5 is preserved even at higher resolutions—the model can be trained on 665K instruction-tuning examples at 448² effective resolution without any intermediate high-resolution pretraining stage. The computational cost increases because the LLM processes more visual tokens (for a 2×3 grid with a global view, 2×3×256 + 256 = 1792 visual tokens vs. 576 for LLaVA-1.5 at 336²), but the training pipeline remains simple and fast.

Resolution and hallucination (Section 5.2). The paper reports an intriguing finding related to hallucination: training on detailed descriptions in LLaVA-Instruct at 336² resolution can cause the model to hallucinate visual details, but scaling to 448² via the split-encode-merge approach significantly reduces this hallucination. The interpretation is that when the model's input resolution is insufficient to discern the fine details described in the training data, it learns to fabricate plausible-sounding details rather than reporting what it actually sees. At higher resolution, the model can actually perceive the details, so it learns to ground its descriptions in visual evidence. This suggests a coupling between data granularity (how detailed the training descriptions are) and model capability (how fine-grained the model's perception is) that must be balanced to avoid hallucination.

Training Procedure and Hyperparameters

LLaVA-1.5 follows the same two-stage training protocol as the original LLaVA [36], with hyperparameters detailed in Appendix A.3 (Table 9).

Stage 1: Vision-Language Alignment Pretraining.

The goal of this stage is to train the vision-language connector (the MLP) to map CLIP visual features into the LLM's word embedding space, so that the LLM can interpret visual tokens as if they were language tokens. Only the MLP connector parameters are updated during this stage; both the vision encoder and the LLM are kept frozen.

Data: The pretraining dataset is the same 558K image-text pairs used in the original LLaVA—a subset of LAION-CC-SBU filtered to contain only images with brief captions. Each training example pairs an image with its caption text, and the model is trained to autoregressively predict the caption tokens given the image tokens.

Hyperparameters:

• Batch size: 256
• Learning rate: 1e-3 (this is halved compared to the original LLaVA's 2e-3, due to the MLP connector being used instead of a linear layer—the MLP has more parameters and may require a more conservative learning rate to converge stably)
• Learning rate schedule: cosine decay
• Learning rate warmup ratio: 0.03 (the learning rate linearly increases from 0 to the base learning rate over the first 3% of training steps)
• Weight decay: 0 (no L2 regularization)
• Epochs: 1
• Optimizer: AdamW
• DeepSpeed stage: 2 (ZeRO-2 optimization for memory-efficient distributed training, which shards optimizer states and gradients across GPUs)

Duration: ~6 hours on 8× A100 GPUs.

What the model learns: After this stage, the MLP connector has learned a mapping that translates visual features into the LLM's token embedding space such that the visual tokens function approximately as "visual words." The model can produce plausible captions for images, as demonstrated in Table 1b, where the pretrained model (before any instruction tuning) answers visual questions reasonably but in a verbose conversational style inherited from the base LLM's pretraining.

Stage 2: Visual Instruction Tuning.

This stage fine-tunes the model on the 665K instruction-following data mixture to teach it to follow diverse user instructions involving visual content. Both the MLP connector and the LLM are updated during this stage; the vision encoder remains frozen.

Data: The full 665K instruction-tuning mixture described in Section 3.4 (LLaVA-Instruct + ShareGPT + VQA datasets + OCR datasets + region-level datasets), each with appropriate response format prompts.

Hyperparameters:

• Batch size: 128
• Learning rate: 2e-5
• Learning rate schedule: cosine decay
• Learning rate warmup ratio: 0.03
• Weight decay: 0
• Epochs: 1
• Optimizer: AdamW
• DeepSpeed stage: 3 (ZeRO-3 optimization, which additionally shards model parameters across GPUs, enabling training of the full 13B LLM with visual tokens on 8 GPUs)

Duration: ~20 hours on 8× A100 GPUs for LLaVA-1.5 at 336² resolution (the training is approximately 2× as long as the original LLaVA due to the increased number of visual tokens from higher resolution—576 tokens vs. 256 tokens, which increases both the memory footprint and the computation per training step).

Total training time: ~26 hours (6 hours pretraining + 20 hours instruction tuning) on a single 8-A100 node, or approximately 1 day. This makes the entire LLaVA-1.5 training pipeline reproducible by academic labs with access to standard GPU clusters.

Inference: At inference time, the model uses greedy decoding (temperature = 0) to select the most likely next token at each generation step, ensuring deterministic and reproducible outputs for evaluation. The vision encoder processes the input image(s) once to produce visual features, the MLP connector projects them, and the LLM autoregressively generates the response.

Why fine-tune the entire LLM? This is a deliberate design choice that distinguishes LLaVA-1.5 from approaches like InstructBLIP (which freezes the LLM) and has both empirical and conceptual justification. Empirically, Table 2 shows that fine-tuning the LLM allows the model to learn format instructions and balance multiple task types. Conceptually, the LLM is the component responsible for generating text, and if it is frozen, all adaptation to new tasks must be mediated by the visual prefix—a bottleneck that limits what the model can learn. Full fine-tuning allows the LLM's internal representations to adapt to visual inputs, potentially learning to attend to visual tokens in task-appropriate ways, to integrate visual information with linguistic knowledge, and to modulate output style based on format instructions.

4. Key Insights and Innovations

Innovation 1: The Capability Split in LMMs Is Not an Architectural Problem—It's a Prompt Format Ambiguity Problem

The paper's most important diagnostic contribution is identifying why InstructBLIP and LLaVA have complementary failure modes—and more critically, why this split is not fundamental. Before this work, the field operated under the implicit assumption that balancing short-form academic VQA with open-ended conversation required either architectural complexity (visual resamplers like Qformer) or large-scale data curation (converting VQA answers to conversational format via GPT-4, as in [7]). InstructBLIP's Qformer was deliberately designed to compress visual features into a small number of "control tokens" that could modulate the frozen LLM's output, and its failure on conversational benchmarks was attributed to the difficulty of this control task—or to the quality of its training data.

LLaVA-1.5 reveals that the entire multitask balancing problem reduces to a prompt engineering issue: when VQA questions are appended with an explicit format instruction ("Answer the question using a single word or phrase"), and when the LLM is fine-tuned (not frozen), the model learns to switch between output formats based on the natural language instruction alone. No architectural mechanism for length control is needed. No data conversion pipeline is needed. No tradeoff between VQA accuracy and conversational quality emerges—Table 2 shows MM-Vet remaining stable at ~26–27 as VQA data is added with format prompts (Rows 1–4), while MME jumps from 809.6 to 1355.2.

This is a reframing rather than an incremental improvement. The dominant assumption in the field—that balancing task formats requires architectural solutions—was wrong. The problem was never about compressing visual information differently or controlling the LLM through a bottleneck; it was about giving the LLM the linguistic signal to know which output mode is appropriate. The evidence for this claim is particularly strong because the paper isolates the mechanism: Row 1 in Table 2 adds VQA data without format prompts and achieves MME 1197.0; Row 2 adds the format prompt and jumps to 1323.8. Same VQA data, same model architecture—the 126.8-point gap is purely attributable to explicit format instructions.

This finding generalizes beyond the specific format prompts used in training. Section 4.3 demonstrates the model following format instructions it never saw during training: responding "Unanswerable" to insufficient VizWiz questions (67.8% accuracy vs. 11.1% without the prompt), generating constrained JSON output (Table 6), and adapting response detail based on conversational context (Table 10). This indicates the model has learned a general skill of reading and obeying format specifications, not just pattern-matching to specific prompt templates. The significance extends beyond the paper's benchmark results—it suggests that many capability gaps attributed to architecture or data scale may actually be annotation-format artifacts that can be resolved at the prompt level.

Innovation 2: The Vision-Language Alignment Pretraining Orthodoxy Is Unnecessary When Building on CLIP

A central premise in the LMM literature prior to this work was that effective vision-language alignment requires a dedicated pretraining stage with substantial data: BLIP-2 uses 129M image-text pairs to train Qformer, Qwen-VL uses 1.4B pairs, and IDEFICS uses 353M pairs. These are not incidental design choices—they reflect a genuine architectural constraint. Visual resamplers like Qformer introduce hundreds of millions of randomly initialized parameters that must learn to extract task-relevant visual features and compress them into a small number of output tokens. Training these parameters from scratch on the downstream instruction-tuning data alone (typically 1–2M examples) would be insufficient—hence the massive alignment pretraining.

LLaVA-1.5 challenges this orthodoxy by demonstrating that a simple MLP connector trained on just 558K image-text pairs achieves stronger multimodal performance than Qformer trained on 129M pairs. This is not a small efficiency gain—it is a ~230× reduction in pretraining data while outperforming on both academic benchmarks (Table 3: 80.0 vs. 49.5 on VQA-v2, 63.3 vs. 33.4 on GQA for 13B models) and conversational benchmarks (Table 4: 1531.3 vs. 1212.8 on MME, 36.1 vs. 25.6 on MM-Vet for equivalently sized models).

The conceptual insight is deeper than "MLPs are more data-efficient than Qformer." The paper suggests that the vision-language alignment problem is fundamentally simpler than the field assumed, at least when the vision encoder is already pretrained on web-scale image-text data (as CLIP is). CLIP's contrastive pretraining objective explicitly aligns visual and textual representations in a shared semantic space. A simple learned projection—whether linear or MLP—may therefore only need to learn a geometric transformation (rotation, scaling, non-linear warping) that translates CLIP's semantic space into the specific embedding geometry of the target LLM. This is a low-complexity mapping that does not require discovering visual concepts from scratch, because CLIP already organized the visual feature space semantically.

The architectural choice of Qformer represents a different hypothesis: that vision-language alignment requires learned attention over visual features to extract task-relevant information, and that the 32 query tokens must dynamically select different visual information depending on the instruction. This is a more expressive but harder-to-train model that demands large-scale pretraining. The paper's results suggest this expressivity may be unnecessary—or even harmful, if the Qformer discards information that the LLM could have used directly—when the downstream task is instruction following rather than, say, image-text retrieval where learned compression might be beneficial.

This finding is a fundamental challenge to the dominant architectural paradigm, not a minor efficiency improvement. It implies that future LMM development should invest compute in improving the pretrained vision encoder (which transfers across models) and in curating high-quality instruction-tuning data, rather than in training complex cross-modal connectors. The paper makes this point explicitly (Section 4.2): "raises questions upon the common belief that LMMs require significant amount of vision-language alignment pretraining... despite that the vision encoders (e.g. CLIP, OpenCLIP, EVA-CLIP) are already pretrained on web-scale image-text paired data." It also explains why LLaVA-1.5's 7B model outperforms IDEFICS-80B (Table 4, MME 1510.7 vs. 54.5, MM-Vet 31.1 vs. IDEFICS-9B's results)—the simpler architecture with direct visual access may actually be a better design than a heavily-parameterized cross-modal bridge that compresses visual information through a bottleneck.

Innovation 3: Hallucination Arises from a Mismatch Between Data Granularity and Model Perceptual Capacity—Not Just from Noisy Training Labels

The standard narrative around hallucination in LMMs attributes it to training data quality: if the model's training descriptions contain fabricated details, the model will learn to fabricate. The corollary is that fixing hallucination requires cleaning the training data. This paper, through its high-resolution scaling experiments, identifies a subtler and more actionable mechanism (Section 5.2).

The paper observes that LLaVA-Instruct's detailed descriptions sometimes contain hallucinated content (a known artifact of GPT-4's generation process). When LLaVA-1.5 is trained on this data at 336² resolution, it inherits some tendency to hallucinate. But when the same training data is used with LLaVA-1.5-HD at an effective 448² resolution (via the split-encode-merge approach), hallucination is significantly reduced—despite the training data containing the same fabricated examples. The paper's interpretation is a diagnostic insight:

"when the input resolution is not sufficient for the model to discern all details in the training data, and the amount of data that is at that granularity beyond the model's capability becomes large enough, the model learns to hallucinate"

This inverts the causal arrow. The model is not simply memorizing and reproducing training data errors. It is learning a generative strategy: if the training data expects detailed descriptions, but the model's visual input does not contain enough information to ground those details, the model learns to fabricate plausible content to match the expected output distribution. The hallucination is an adaptation to the model's perceptual bottleneck, not a direct reflection of training data quality.

This is a new diagnostic concept with direct practical implications: improving data quality (removing hallucinated descriptions) may help, but it treats the symptom rather than the cause. The more fundamental intervention is to match data granularity to model perceptual capacity. If you want the model to produce detailed, accurate descriptions, you must first ensure it can see the details. The paper demonstrates this concretely through the high-resolution results: LLaVA-1.5-HD at 448² improves on benchmarks requiring fine-grained perception (OCR, detailed description) while reducing hallucination, all without any data cleaning.

This finding also explains why the split-encode-merge approach in LLaVA-1.5-HD is not merely an inference-time trick but addresses a fundamental training dynamic. By preserving the native 224² patch encoding (where CLIP features are high-quality) while providing the LLM with full-resolution visual access, the model can learn to ground its outputs in perceptible visual evidence rather than compensating for missing information with generation.

Innovation 4: The "Grid-Split" High-Resolution Strategy Eliminates the Need for Positional Embedding Interpolation and Resolution-Specific Pretraining

The conventional approach to higher-resolution vision transformers—positional embedding interpolation—creates a dependency chain: interpolate the positional embeddings → the ViT's internal representations are now operating out-of-distribution → fine-tune the ViT on large-scale image-text data to adapt → the model is now locked to the new resolution. This approach, used by Qwen-VL and BLIP-2, makes high-resolution support expensive (requiring the large-scale fine-tuning data) and inflexible (the model can only process images at the trained resolution).

LLaVA-1.5-HD's split-encode-merge strategy (Section 3.4, Figure 2) is a fundamentally different solution to the resolution scaling problem: instead of adapting the encoder to a new resolution, keep the encoder at its native resolution and adapt the input representation. Each 224×224 image patch is processed by the CLIP encoder using the exact positional embeddings it was trained with—there is zero distribution shift for the encoder. The adaptation burden falls entirely on the LLM, which must learn to integrate multiple patch features and reconstruct the global image layout.

This is a conceptual reframing of the resolution problem from a vision-encoder challenge to a language-model challenge. It works because:

1. CLIP features are already high-quality at native resolution—the encoder was trained on 224² images, so its features are maximally reliable at this resolution. Any interpolation-based approach degrades feature quality to some degree, even after fine-tuning.

2. The LLM has the capacity to integrate spatial information—transformer self-attention can, in principle, learn to relate features from different patches based on their content and positional encoding (augmented by the row-end tokens). This is a learnable skill that the visual instruction tuning stage provides training signal for.

3. Data efficiency is preserved—because the encoder and connector require no additional training for high resolution, the model can be trained directly on the 665K instruction-tuning examples at high resolution, maintaining the same modest data requirements.

The evidence that this approach works comes from Table 4, where LLaVA-1.5-HD (13B) achieves best or near-best performance across all 12 benchmarks, including significant improvements on detail-sensitive tasks (MM-Vet 39.4 vs. 36.1 for LLaVA-1.5, TextVQA 62.5 vs. 61.3). The global context ablation (Section 4.2) confirms that the LLM benefits from having both the high-resolution patches (for detail) and the downsampled global view (for spatial coherence), with the combination providing +3.2 points on MM-Vet over the high-resolution patches alone.

The significance extends beyond this paper's results. The split-encode-merge strategy is a general technique that can be applied to any resolution beyond what the vision encoder natively supports, with any vision encoder, without modifying the encoder's architecture or training. It decouples resolution scaling from encoder fine-tuning, making high-resolution LMMs accessible to researchers who cannot afford massive vision-language pretraining runs. This aligns with the paper's broader goal of democratizing LMM research.

Innovation 5: Compositional Capabilities in LMMs Can Emerge from Independently Trained Task Combinations—Without Explicit Joint Training

Section 5.3 documents phenomena where LLaVA-1.5, trained on a set of distinct tasks (VQA, OCR, visual conversation, text-only conversation), generalizes to tasks requiring combinations of these capabilities that were never explicitly represented in the training data. The two clearest examples:

Multilingual visual conversation (Figure 5). The model is trained on English-only visual instruction data and multilingual text-only ShareGPT data. These are separate data sources—there is no example in the training data of a user asking a visual question in Spanish and receiving a Spanish answer. Yet at test time, the model responds to Spanish, Japanese, and Korean visual questions in the appropriate language, translating both the visual understanding capability (from English visual data) and the multilingual generation capability (from text-only ShareGPT) into the compositional task.

Improved visual writing (Table 11). Adding ShareGPT text-only data improves the quality and detail of the model's visual descriptions, even though ShareGPT contains no images. The model composes the language fluency and detailed-response style learned from text-only conversations with the visual grounding learned from visual instruction data, producing more elaborate and well-structured visual descriptions than LLaVA (which had only visual instruction data).

This is significant because it suggests data-efficient capability acquisition through composition: rather than exhaustively collecting training data for every possible task combination (e.g., multilingual visual instruction data for every language, or visual writing data in various styles), LLaVA-1.5 can acquire these capabilities by training on the component skills independently and relying on the LLM's ability to recombine them. This is a potential scaling law of a different kind—not scaling data volume, but scaling the diversity of capability "building blocks" that can be composed combinatorially.

The paper is appropriately cautious about this finding, noting that composition is imperfect—Korean visual responses contain errors (Figure 5, errors marked in red), and being able to answer VQA questions about object attributes does not guarantee those attributes are accurately described in a full-scene description. But the existence of compositional generalization even at this preliminary level is a proof-of-concept that challenges the "collect data for every task variant" approach and suggests a more efficient path: train on a carefully chosen set of base capabilities, and let the LLM handle the combinations.

This insight connects to broader questions in the field about whether LMMs can serve as general-purpose assistants without requiring exhaustive coverage of every possible input-output format in the training data. The paper's evidence is preliminary but points toward a positive answer, particularly for combinations of linguistic and visual capabilities that the base LLM already handles compositionally in the text domain.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on 12 benchmarks, split into two categories. Academic-task-oriented benchmarks include VQA-v2 [19] (open-ended short answers), GQA [21] (compositional visual reasoning), VizWiz [20] (zero-shot generalization on questions from visually impaired users), ScienceQA-IMG [38] (the image subset with multiple choice, used zero-shot), and TextVQA [48] (text-rich visual question answering). Instruction-following LMM benchmarks include POPE [34] (hallucination evaluation on three COCO subsets: random, popular, adversarial; F1 score reported), MME-Perception [17] (yes/no visual perception questions), MMBench [37] and MMBench-CN [37] (Chinese-translated version; multiple choice with all-round shuffling for robustness), SEED-Bench [27] (multiple choice on images and videos; video evaluated using the middle frame), LLaVA-Bench-in-the-Wild [36] (GPT-4-evaluated visual conversations on diverse real-world images), and MM-Vet [55] (GPT-4-evaluated integrated capabilities across recognition, knowledge, spatial awareness, language generation, and math).

• Base model(s). The primary models are LLaVA-1.5 variants built on Vicuna-7B and Vicuna-13B [60], where Vicuna is an instruction-tuned version of LLaMA fine-tuned on ShareGPT user conversations. The vision encoder is CLIP-ViT-L-336px for LLaVA-1.5 and CLIP-ViT-L-224px for the high-resolution LLaVA-1.5-HD (which uses patch-splitting to achieve an effective resolution of up to 448²). An ablation in Section 4.4 additionally compares Vicuna-v1.1, Vicuna-v1.3, Vicuna-v1.5, and LLaMA-2-Chat as base LLMs. The authors argue Vicuna is "representative of the capabilities of many contemporary LLMs" and sits in a regime where the model has non-trivial multimodal capabilities but substantial room for improvement through instruction tuning.

• Metrics. For VQA-v2, GQA, VizWiz, and TextVQA, the metric is accuracy against ground-truth short answers, where VQA-v2 and GQA use the standard VQA evaluation protocol (exact string matching after normalization). For multiple-choice benchmarks (ScienceQA-IMG, MMBench, MMBench-CN, SEED-Bench), accuracy is measured as the fraction of questions where the model selects the correct option letter. For POPE, the F1 score is reported across three splits (random, popular, adversarial) to evaluate hallucination robustness. For MME-Perception, the score is the sum of accuracy across 10 perception subtasks, each scored yes/no with a maximum of 2000 total points (1531.3 for LLaVA-1.5-13B). For LLaVA-Bench-in-the-Wild and MM-Vet, GPT-4 evaluates response quality on dimensions of correctness and helpfulness, producing a percentage score (72.5% for LLaVA-1.5-13B on LLaVA-Bench and 36.1 on MM-Vet). All inference uses greedy decoding (temperature = 0) for reproducibility.

• Baselines. The paper compares against BLIP-2 [32] (Vicuna-13B, 129M pretraining pairs, Qformer-based), InstructBLIP [14] (Vicuna-7B and Vicuna-13B variants, 129M pretraining pairs, Qformer fine-tuned while LLM frozen, 1.2M instruction-tuning samples), Shikra [8] (Vicuna-13B, 600K pretraining pairs, 5.5M fine-tuning samples, referential dialogue), IDEFICS-9B and IDEFICS-80B [22] (LLaMA-based, 353M pretraining pairs, 1M instruction-tuning samples, Flamingo-like architecture), Qwen-VL and Qwen-VL-Chat [3] (Qwen-7B, 1.4B pretraining pairs with in-house data, 50M instruction-tuning samples, visual resampler), and the original LLaVA-7B [36] (Vicuna-7B, 558K pretraining pairs, 158K instruction-tuning samples, linear projection connector, 224² resolution). Specialist state-of-the-art PaLI-X-55B [11] is also cited for reference on VQA benchmarks. All baseline results are taken from the respective papers or from standard leaderboard submissions as reported in the paper's Tables 3 and 4.

• Generation budget / compute accounting. The paper does not use a per-example generation budget as a compute metric (unlike search-based papers that count number of generations). Instead, compute is reported in terms of training resource requirements: total dataset size (1.2M publicly available samples across pretraining and instruction tuning), total training time (~6 hours pretraining + ~20 hours instruction tuning on 8× A100 GPUs), and the fact that the model finishes full training in ~1 day on a single 8-A100 node. Comparisons emphasize data efficiency: LLaVA-1.5 uses 558K pretraining pairs vs. InstructBLIP's 129M, Qwen-VL's 1.4B, and IDEFICS' 353M; and 665K instruction-tuning samples vs. Qwen-VL-Chat's 50M. At inference time, the computational cost differs between LLaVA-1.5 (576 visual tokens from 336² input) and LLaVA-1.5-HD (variable, up to 1792 visual tokens for 6-patch grids with global context), but no FLOPs-matched comparison between the two is performed—the paper reports accuracy improvements from resolution scaling without normalizing for the additional inference compute.

• Cross-validation / statistical protocol. The paper does not employ cross-validation or report confidence intervals. The scaling experiments in Table 2 are conducted as single-run ablations starting from a base configuration and adding components sequentially. The ablation on LLM choices (Section 4.4, Figure 3) and data efficiency (Section 5.1, Figure 4) similarly report single-run results. All evaluation is on standard test sets with greedy decoding, which provides deterministic outputs. The lack of error bars or multiple-random-seed results means the statistical reliability of the reported gains—particularly for small absolute improvements (e.g., the +0.5 GQA improvement from the MLP connector in Table 2 Row 3 vs. Row 2)—cannot be assessed from the reported data. This is a methodological limitation given the paper's goal of establishing "reproducible baselines."

Main Quantitative Results

Academic-Task-Oriented Benchmarks

The headline results for academic benchmarks appear in Table 3. LLaVA-1.5-13B achieves the best performance among LMMs (excluding the specialist PaLI-X-55B) on 4 out of 5 benchmarks and ranks second on VizWiz behind Qwen-VL-Chat. The specific numbers are:

• VQA-v2: LLaVA-1.5-13B achieves 80.0% (vs. InstructBLIP-13B at 49.5%, Qwen-VL-Chat at 78.2%, BLIP-2 at 65.0%). The 7B variant achieves 78.5%. LLaVA-1.5-HD further improves to 81.8%.

• GQA: LLaVA-1.5-13B achieves 63.3% (vs. InstructBLIP-13B at 33.4%, Qwen-VL-Chat at 57.5%, BLIP-2 at 41%). The 7B variant achieves 62.0%. LLaVA-1.5-HD reaches 64.7%.

• VizWiz: LLaVA-1.5-13B achieves 53.6% (vs. Qwen-VL-Chat at 38.9%, InstructBLIP-13B at 33.4%, IDEFICS-80B at 36.0%). LLaVA-1.5-HD improves to 57.5%, which the paper attributes to the model's ability to process fine-grained details needed to determine when questions are unanswerable.

• ScienceQA-IMG: LLaVA-1.5-13B achieves 71.6% (vs. Qwen-VL-Chat at 68.2%, InstructBLIP-13B at 63.1%, BLIP-2 at 61%). LLaVA-1.5-HD reaches 71.0%, essentially flat.

• TextVQA: LLaVA-1.5-13B achieves 61.3% (vs. Qwen-VL-Chat at 61.5%, InstructBLIP-13B at 50.7%, BLIP-2 at 42.5%). LLaVA-1.5-HD improves to 62.5%. On this benchmark, Qwen-VL-Chat (61.5%) and the specialist PaLI-X-55B (71.4%) outperform LLaVA-1.5-13B.

The key comparison is against InstructBLIP-13B, which uses the same base LLM (Vicuna-13B) but with Qformer and 129M pretraining pairs. LLaVA-1.5-13B outperforms InstructBLIP-13B by margins of +30.5 on VQA-v2, +29.9 on GQA, +20.2 on VizWiz, +8.5 on ScienceQA-IMG, and +10.6 on TextVQA. Given that both models share the same LLM and that LLaVA-1.5 uses 230× less pretraining data, these gaps represent strong evidence that the simple MLP connector + LLM fine-tuning + format prompt approach is more effective than the Qformer + frozen LLM approach for these benchmarks.

Instruction-Following LMM Benchmarks

Table 4 reports results on benchmarks designed specifically for evaluating instruction-following LMMs. LLaVA-1.5-13B achieves the best overall performance across all 7 benchmarks evaluated (with some baselines missing scores on certain benchmarks because they were not evaluated in the original papers):

• POPE (hallucination, F1 scores): LLaVA-1.5-13B achieves 87.1/86.2/84.5 on random/popular/adversarial splits (vs. InstructBLIP-14B at 87.7/77/72, BLIP-2 at 89.6/85.5/80.9). The original LLaVA-7B achieved 76.3/72.2/70.1. The improvement on the adversarial split (+14.4 points over original LLaVA, +12.5 over InstructBLIP) is particularly notable because adversarial questions are designed to probe hallucination in challenging cases.

• MME-Perception (yes/no visual perception): LLaVA-1.5-13B achieves 1531.3 out of 2000 (vs. Qwen-VL-Chat at 1487.5, InstructBLIP-14B at 1212.8, original LLaVA-7B at 809.6). The 7B variant achieves 1510.7, also substantially outperforming all prior models. MME is highlighted as a key indicator because it evaluates the model's ability to answer yes/no perception questions—exactly the format that the response format prompts target during training.

• MMBench: LLaVA-1.5-13B achieves 67.7 (vs. Qwen-VL-Chat at 60.6, InstructBLIP-8B at 36, original LLaVA-7B at 38.7). MMBench-CN (Chinese): LLaVA-1.5-13B achieves 63.6 (vs. Qwen-VL-Chat at 56.7, InstructBLIP-8B at 23.7, original LLaVA-7B at 36.4). The +7.3 point advantage over Qwen-VL-Chat on Chinese benchmarks is notable because Qwen-VL is explicitly trained on Chinese multimodal instructions while LLaVA-1.5 is not—the paper attributes this to multilingual transfer from the text-only ShareGPT data (Section 4.3).

• SEED-Bench (images and videos): LLaVA-1.5-13B achieves 68.2 on the image split and 42.7 on the video split (vs. Qwen-VL-Chat at 65.4/37.8, InstructBLIP-14B at 58.8/38.1, BLIP-2 at 46.4/36.7). The video evaluation uses only the middle frame, so this measures static image understanding rather than temporal reasoning.

• LLaVA-Bench-in-the-Wild (GPT-4-evaluated visual conversations): LLaVA-1.5-13B achieves 72.5 (vs. InstructBLIP-14B at 58.2, BLIP-2 at 38.1, original LLaVA-7B at 62.8). LLaVA-1.5-HD achieves 72.0, a slight regression.

• MM-Vet (GPT-4-evaluated integrated capabilities): LLaVA-1.5-13B achieves 36.1 (vs. InstructBLIP-14B at 25.6, BLIP-2 at 22.4, original LLaVA-7B at 25.5). LLaVA-1.5-HD improves to 39.4, the largest gain from resolution scaling on any benchmark (+3.3 points over LLaVA-1.5-13B).

The pattern across these benchmarks reveals that LLaVA-1.5's improvements over the original LLaVA are most dramatic on MME (+721.7 points from 809.6 to 1531.3 for the 13B model) and least dramatic on Pope (roughly +10-15 points across splits). The MME improvement is largely attributable to the inclusion of yes/no VQA data during training (Table 2 Row 1: +387.4 from adding VQA-v2 without format prompts) and the format prompt itself (Row 2: +126.8 additional), which directly address LLaVA's original yes/no answer bias and inability to produce short-form answers.

LLaVA-1.5 Scaling Progression (Table 2)

Table 2 provides the core ablation showing how each modification incrementally improves performance on three representative benchmarks: GQA (short-form VQA with compositional questions), MME (yes/no perception), and MM-Vet (natural visual conversations). These three benchmarks are chosen to represent complementary capabilities—short-form accuracy, output format control, and open-ended visual reasoning, respectively.

The scaling progression for the 7B model starts from the original LLaVA (which has no VQA data and uses a linear connector at 224²) and adds modifications sequentially:

• Row 0 (LLaVA baseline, 7B, 224²): MME = 809.6, MM-Vet = 25.5. GQA is not reported because the original LLaVA cannot produce short-format answers needed for GQA evaluation.

• Row 1 (+VQA-v2, no format prompt): MME jumps to 1197.0 (+387.4), GQA becomes measurable at 47.0, MM-Vet rises to 27.7 (+2.2). The large MME gain comes from exposing the model to yes/no questions during training; the MM-Vet gain suggests that VQA data provides visual knowledge that transfers to conversations.

• Row 2 (+Format prompt): MME further increases to 1323.8 (+126.8), while GQA remains essentially flat at 46.8 and MM-Vet slightly decreases to 26.3 (-1.4). The MME improvement confirms that explicit format instructions help the model learn when to produce yes/no answers vs. longer responses; the stable GQA and MM-Vet indicate no degradation on other task formats.

• Row 3 (+MLP VL connector): MME rises to 1355.2 (+31.4), GQA to 47.3 (+0.5), MM-Vet to 27.8 (+1.5). These are consistent but modest improvements across all three benchmarks, suggesting the MLP provides additional representational capacity that benefits all task types approximately equally.

• Row 4 (+OKVQA/OCR): MME reaches 1377.6 (+22.4), GQA rises to 50.0 (+2.7), MM-Vet to 29.6 (+1.8). Adding open-knowledge and OCR data improves GQA more substantially, likely because GQA requires both visual perception and external knowledge in some questions.

• Row 5 (+Region-level VQA): MME reaches 1426.5 (+48.9), GQA to 50.3 (+0.3), MM-Vet to 30.8 (+1.2). The large MME gain is notable—region-level data improves yes/no perception questions, possibly because object localization training sharpens the model's attention to specific image regions.

• Row 6 (+Scale up resolution to 336²): MME reaches 1450 (+23.5), GQA to 51.4 (+1.1), MM-Vet slightly decreases to 30.3 (-0.5). The resolution scaling improves detail-oriented benchmarks (GQA, MME) but does not immediately benefit conversational quality, where the additional visual tokens may require more training or larger LLM capacity to utilize effectively.

• Row 7 (+GQA): GQA increases to 62.0** (the asterisk indicates training images of GQA were observed, making this partially in-distribution evaluation), MME reaches 1469.2 (+19.2), MM-Vet to 30.7 (+0.4). The GQA score jump reflects training on the GQA dataset itself, which is noted as a caveat for fair comparison with models that did not train on GQA.

• Row 8 (+ShareGPT): MME reaches 1510.7 (+41.5), GQA stable at 62.0*, MM-Vet to 31.1 (+0.4). The MME increase from adding text-only data is notable—it suggests that language-only conversational data improves the model's ability to answer yes/no perception questions, possibly by improving general instruction-following robustness.

• Row 9 (+Scale up LLM to 13B): MME reaches 1531.3 (+20.6), GQA to 63.3*, MM-Vet to 36.1 (+5.0). The MM-Vet jump is the most significant effect of scaling the LLM, confirming that conversational quality depends more on LLM capacity than do structured-output benchmarks like MME.

Several patterns emerge from this progression. First, the MME score grows from 809.6 to 1531.3 (an 89% increase), with the largest single contribution from adding VQA-v2 (R1: +387.4) and the second largest from region-level VQA (R5: +48.9). Second, MM-Vet grows from 25.5 to 36.1 (a 42% increase), with nearly half of that gain coming from the LLM scale-up alone (R9: +5.0). Third, no single modification degrades performance on any benchmark by a large margin—the approach appears to be consistently additive, with the worst regression being MM-Vet from R5 to R6 (-0.5 with resolution scaling).

High-Resolution Scaling (LLaVA-1.5-HD)

Table 3 and Table 4 include LLaVA-1.5-HD results across all benchmarks. The high-resolution variant uses split-encode-merge with an effective resolution up to 448² (compared to 336² for LLaVA-1.5) and adds a global context image. The improvements over LLaVA-1.5-13B are:

• Academic benchmarks (Table 3): VQA-v2 +1.8 (80.0→81.8), GQA +1.4 (63.3→64.7), VizWiz +3.9 (53.6→57.5), ScienceQA-IMG -0.6 (71.6→71.0), TextVQA +1.2 (61.3→62.5).

• Instruction-following benchmarks (Table 4): Pope random +0.4 (87.1→87.5), Pope popular +0.2 (86.2→86.4), Pope adversarial +0.5 (84.5→85.0), MME -31.2 (1531.3→1500.1), MMBench +1.1 (67.7→68.8), MMBench-CN -1.7 (63.6→61.9), SEED-Bench all +1.0 (61.6→62.6), LLaVA-Bench-in-the-Wild -0.5 (72.5→72.0), MM-Vet +3.3 (36.1→39.4).

The pattern is mixed: LLaVA-1.5-HD improves on detail-oriented benchmarks (VizWiz +3.9, MM-Vet +3.3, VQA-v2 +1.8) but slightly degrades on MME (-31.2) and MMBench-CN (-1.7). The global context ablation (Section 4.2) on a 7B model shows that the global context image contributes meaningfully (GQA: 62.9→63.8, +0.9; MME: 1425.8→1497.5, +71.7; MM-Vet: 31.9→35.1, +3.2). The MME degradation in the 13B model (-31.2) despite the 7B global context improving MME (+71.7) suggests that the 13B model may overfit to the high-resolution features or that the additional visual tokens introduce noise for yes/no perception questions that the 7B model handles better.

Data Efficiency (Section 5.1, Figure 4)

Figure 4 reports the relative performance of LLaVA-1.5 when the 665K instruction-tuning data mixture is randomly downsampled to 10%, 20%, 30%, 40%, 50%, and 100%. The results are normalized so that the full-data performance is 1.0 for each benchmark. The key findings:

• 50% data: The model maintains more than 98% of full-dataset performance. Performance on MMBench, ScienceQA-IMG, and POPE does not decrease at all (remains at 1.0 or slightly above), and MMBench performance even slightly improves above the full-data baseline.

• 30% data: Performance remains steady compared to 50%, with no substantial degradation observed. The paper states that "the model's performance remains steady when further downscaling the data from 50% to 30%," suggesting a plateau in the data-performance curve.

• 10% data: Performance degrades notably, but the paper only states that "the full data mixture provides the best knowledge coverage" without reporting the exact degradation magnitude at 10%.

This is a suggestive finding for data efficiency but has important caveats: the downsampling is performed randomly across the full data mixture, meaning each dataset is represented proportionally in the subsample. The "less-is-more" effect (slight improvements at 50% on MMBench) is consistent with findings in NLP instruction tuning [61], suggesting that some data redundancy exists in the 665K mixture. The paper frames this as evidence that "there is room for further improvements in data efficiency" and that more sophisticated dataset compression strategies could be developed.

LLM Choice Ablation (Section 4.4, Figure 3)

Figure 3 compares four LLM variants: Vicuna-v1.1 (LLaMA-1-based, 70K ShareGPT instruction data), Vicuna-v1.3 (LLaMA-1-based, 150K ShareGPT, 2× the instruction data of v1.1), Vicuna-v1.5 (LLaMA-2-based, 150K ShareGPT), and LLaMA-2-Chat (LLaMA-2-based with additional RLHF). Performance is reported relative to the best-performing variant for each benchmark.

• Vicuna-v1.5 achieves the best overall performance, outperforming Vicuna-v1.3 on most benchmarks, which the paper attributes to LLaMA-2 being a stronger base model than LLaMA-1 ("suggesting the importance of the base language model").

• MMBench-CN reveals a base-model dependence: Vicuna-v1.3 performs significantly worse than Vicuna-v1.5 on Chinese questions, despite both being trained on the same 150K ShareGPT data. The paper attributes this to inherent multilingual capabilities of the base LLaMA-2 model.

• LLaMA-2-Chat underperforms on MMBench-CN and TextVQA compared to Vicuna-v1.5. The paper hypothesizes that LLaMA-2-Chat's RLHF training data is predominantly English and lacks the multilingual content present in ShareGPT, limiting transfer to Chinese questions. For TextVQA, ShareGPT data collected from in-the-wild ChatGPT usage may contain noisy OCR-like artifacts from users copying text, providing relevant training signal that LLaMA-2-Chat's cleaner RLHF data lacks.

• Vicuna-v1.1 vs. v1.3: The 2× increase in ShareGPT instruction data (70K→150K) shows improvements across all benchmarks, confirming that instruction-tuning data quantity matters for the base LLM before visual training.

Ablation Studies and Robustness Checks

Response format prompt impact (Table 2, Rows 1 vs. 2): Adding the explicit format prompt ("Answer the question using a single word or phrase") to VQA-v2 data while keeping all other factors constant improves MME from 1197.0 to 1323.8 (+126.8), while GQA remains stable (47.0→46.8) and MM-Vet slightly decreases (27.7→26.3). This is the cleanest demonstration that format ambiguity, not VQA data content, was the bottleneck for MME performance. The format prompt effect on VizWiz unanswerable questions is reported separately in Section 4.3: accuracy on unanswerable questions increases from 11.1% to 67.8%, demonstrating generalization to an unseen format instruction.

MLP vs. linear connector (Table 2, Rows 2 vs. 3): Replacing the single linear layer with a 2-layer MLP (with GELU activation) improves MME by +31.4 (1323.8→1355.2), GQA by +0.5 (46.8→47.3), and MM-Vet by +1.5 (26.3→27.8). The improvements are modest but consistent across all three representative benchmarks, suggesting the MLP provides additional representational capacity without overfitting or optimization difficulties.

Vision encoder resolution scaling at 7B (Table 2, Rows 5 vs. 6): Upgrading from CLIP-ViT-L-224px (256 visual tokens) to CLIP-ViT-L-336px (576 visual tokens, 2.25× more tokens) improves GQA by +1.1 (50.3→51.4) and MME by +23.5 (1426.5→1450), but decreases MM-Vet by -0.5 (30.8→30.3). This suggests that while more visual tokens help structured perception tasks, they may introduce attention-dilution effects for open-ended conversation tasks at smaller LLM scales.

OCR and open-knowledge VQA data (Table 2, Rows 3 vs. 4): Adding OKVQA, A-OKVQA, OCRVQA, and TextCaps (while keeping resolution at 224² and connector as MLP) improves GQA by +2.7 (47.3→50.0, the largest single-dataset gain for GQA), MME by +22.4 (1355.2→1377.6), and MM-Vet by +1.8 (27.8→29.6). The GQA improvement suggests that open-knowledge VQA data transfers to compositional reasoning questions even when the knowledge domains differ.

Region-level VQA data (Table 2, Rows 4 vs. 5): Adding Visual Genome and RefCOCO yields a large MME improvement (+48.9, from 1377.6 to 1426.5), the third-largest single-modification gain after adding VQA-v2 itself. GQA improves only +0.3 (50.0→50.3) and MM-Vet improves +1.2 (29.6→30.8). The paper does not ablate Visual Genome and RefCOCO separately, so their individual contributions to the MME gain cannot be isolated.

ShareGPT text-only data (Table 2, Rows 7 vs. 8): Adding 40K text-only conversations from ShareGPT to the instruction-tuning mixture while keeping all visual data constant improves MME by +41.5 (1469.2→1510.7), the fourth-largest single-modification gain. GQA remains flat at 62.0*, and MM-Vet improves +0.4 (30.7→31.1). The MME improvement from text-only data is a non-obvious finding—it suggests that general instruction-following capability learned from text conversations transfers to structured multimodal tasks. The paper documents additional qualitative benefits in Section 5.3, including multilingual visual conversation (Figure 5) and improved visual writing (Table 11), which are attributed to compositional transfer from ShareGPT.

LLM scale (Table 2, Rows 8 vs. 9): Scaling the base LLM from Vicuna-7B to Vicuna-13B (while keeping all data and architecture constant) improves MM-Vet by +5.0 (31.1→36.1), the largest absolute improvement for any single modification on the conversational benchmark. MME improves by +20.6 (1510.7→1531.3), and GQA improves by +1.3* (62.0→63.3). This confirms that conversational visual reasoning benefits disproportionately from larger LLM capacity, while structured-output tasks approach saturation faster.

GQA as training data (Table 2, Rows 6 vs. 7): Adding GQA to the training data (italicized with asterisk in the table) causes a large jump in GQA test performance from 51.4 to 62.0* because the training images overlap with the test images. This is flagged as a caveat—the 62.0* and 63.3* scores are not directly comparable with models that did not train on GQA, making GQA a less informative benchmark for cross-model comparison. The paper still reports it for completeness but notes the asterisk.

Global context for high resolution (Section 4.2, 7B ablation): On the 7B model, adding the global context image to high-resolution patches improves GQA by +0.9 (62.9→63.8), MME by +71.7 (1425.8→1497.5), and MM-Vet by +3.2 (31.9→35.1). This ablation demonstrates that the LLM benefits from having both fine-grained patch features and a holistic scene view, and that the split-encode-merge approach without global context loses important spatial information. The MME improvement (+71.7) is particularly striking and suggests yes/no perception questions require understanding how objects relate spatially across the entire scene.

Verification that format prompting does not degrade conversation (Table 2, Row 2 MM-Vet): MM-Vet scores across Rows 0-4 remain in a narrow range of 25.5-27.8, with the Row 2 value of 26.3 being statistically indistinguishable from the baseline of 25.5. This is an important negative result: adding VQA data with format prompts does NOT cause the conversational degradation observed in InstructBLIP, supporting the paper's claim that format prompts + LLM fine-tuning solve the multitask balancing problem.

Tricky question handling (Table 5, qualitative): LLaVA-1.5 can detect factual errors in questions (e.g., asking about a "desert" when the image shows a beach) when prompted to verify, responding "There are no deserts in the image. The image features a beach..." while the original LLaVA fabricates context ("The scene is set in a desert, which is unusual for a beach setting"). This qualitative comparison demonstrates that the improved visual grounding from academic VQA training reduces hallucination in adversarial questioning scenarios.

Constrained format generation (Table 6, qualitative): LLaVA-1.5 extracts structured information from a driver's license image into a JSON format with specified keys, despite never being trained on JSON generation tasks. Compared to GPT-4V (evaluated in a separate study [52]), LLaVA-1.5 makes several errors (incorrect DLN, missing address details, "DONOR" field as "N/A" when it should be "VETERAN"), but correctly identifies the output format and extracts most fields. This demonstrates generalization to unseen output structures—a capability the paper attributes to compositional transfer from the model's instruction-following training.

Critical Assessment

The experimental evaluation is comprehensive in coverage (12 benchmarks spanning academic VQA, hallucination, perception, conversation, and multilingual evaluation) and effective at demonstrating the paper's central practical claim: that a simple architecture (MLP connector + full LLM fine-tuning) with carefully formatted training data can achieve state-of-the-art results using modest public data and compute. The side-by-side comparisons against InstructBLIP in Table 3 are particularly informative because both models share the same base LLM (Vicuna-13B) and the same pretraining objective (vision-language alignment), yet LLaVA-1.5 outperforms by 10-30 points across academic benchmarks and 4-10 points on conversational benchmarks—gaps that are large enough to be meaningful despite the lack of confidence intervals.

However, several aspects of the experimental design limit the strength of the conclusions that can be drawn:

The core claim about data efficiency rests on a comparison against baselines that were not designed to be data-efficient. The paper emphasizes that LLaVA-1.5 uses 558K pretraining pairs vs. InstructBLIP's 129M and Qwen-VL's 1.4B. But InstructBLIP and Qwen-VL did not optimize for pretraining data efficiency—they used whatever data was available. The fair question is not "can LLaVA-1.5 beat InstructBLIP with 230× less data?" but rather "would InstructBLIP still need 129M pairs if it also used an MLP connector and full LLM fine-tuning?" The paper cannot answer this because it does not ablate the connector architecture while holding data constant in a way that isolates data volume. A controlled experiment that trains Qformer with varying pretraining data quantities (558K, 10M, 129M) would be needed to determine whether the data efficiency gain comes from the MLP architecture or from other differences (LLM fine-tuning, format prompts, data mixture). The absence of such an experiment means the paper demonstrates a more data-efficient system but not necessarily that the architecture is inherently more data-efficient.

The ablated progression in Table 2 conflates correlation with causal contribution. The sequential addition of components (VQA-v2, then format prompt, then MLP, then OCR, etc.) means the marginal contribution of each component is computed against a changing baseline. For example, the +23.5 MME gain from scaling resolution (Row 6) is computed against a baseline that already includes region-level VQA data (Row 5). The marginal contribution of region-level data might be different if resolution scaling had been added first. This ordering dependence is acknowledged implicitly (the paper describes building a "road map") but the magnitude of interaction effects is not quantified. The claim that each component provides orthogonal benefits is therefore assumed rather than demonstrated.

The format prompt contribution in isolation is not cleanly isolated. Row 2 in Table 2 adds the format prompt to training data that already contains VQA-v2 examples (from Row 1). The MME improvement (+126.8) could partially reflect a synergy effect: the format prompt helps only because VQA data is already present. The paper would be strengthened by a two-factor ablation showing MME with format prompts but WITHOUT VQA data—which would presumably show no improvement (since there would be no short-answer training examples to apply the format prompt to). This ablation is not reported, so the standalone contribution of format prompts (independent of VQA data volume) remains unclear.

The claim that hallucination is a granularity-capability mismatch (Section 5.2) is supported by qualitative observation but not systematic measurement. The paper states that LLaVA-1.5-HD "significantly reduces hallucination" compared to LLaVA-1.5, but the only quantitative hallucination evaluation is POPE (Table 4), where LLaVA-1.5-HD improves over LLaVA-1.5-13B by only +0.4/+0.2/+0.5 on the three splits. These are marginal improvements, not "significant reductions." The paper's argument about resolution reducing hallucination appears to rely on qualitative examples (not systematically analyzed) and on improvements in benchmarks that require detailed perception (MM-Vet, VizWiz), which could be due to better visual feature quality rather than reduced fabrication. A dedicated hallucination benchmark that distinguishes between perception errors and fabrication would be needed to validate the granularity-capacity mismatch hypothesis.

The compositional capabilities claim (Section 5.3) is primarily qualitative. The paper demonstrates multilingual visual conversation (Figure 5) and visual writing improvement (Table 11) through single examples. The quantitative evidence—MMBench-CN improvement over Qwen-VL-Chat and LLaVA-Bench-in-the-Wild improvement over original LLaVA—is consistent with compositional generalization but could also be explained by other factors (stronger base LLM, better instruction tuning, the ShareGPT data providing general fluency benefits rather than specifically compositional transfer). A controlled experiment that ablates ShareGPT while measuring multilingual visual benchmarks would provide stronger evidence, but multilingual visual benchmarks with ground-truth answers are not evaluated beyond MMBench-CN (which is multiple choice).

Missing baselines and ablations:

• No comparison to a LLaVA-1.5-sized InstructBLIP. The paper compares LLaVA-1.5-13B against InstructBLIP-13B, but the two models differ in connector architecture, pretraining data volume, LLM fine-tuning, format prompts, and instruction-tuning data mixture simultaneously. A more informative baseline would be InstructBLIP retrained with LLaVA-1.5's instruction-tuning data (with format prompts) while keeping the Qformer frozen—this would isolate the effect of the connector architecture and LLM fine-tuning separately from data differences. Such a baseline is not reported.

• No ablation of the number of MLP layers. The paper upgrades from a single linear layer to a 2-layer MLP, citing self-supervised learning findings [9, 10]. But no experiment compares 1-layer, 2-layer, 3-layer, or wider/narrower MLPs. The optimal depth and width of the connector remain unexplored, so the claim that "a 2-layer MLP" is the right design choice is based on precedent rather than empirical optimization within the LLaVA framework.

• No resolution scaling curve. The paper tests two resolutions (224² and 336²) and one high-resolution variant (effective 448² via patch-splitting), but does not scan resolutions continuously to show how performance scales with input resolution. The claim that "scaling the input image resolution improves the model's capabilities" (Section 3.4) is supported directionally but without a scaling law characterization.

• No evaluation of computational cost vs. benefit for the MLP vs. linear connector. The MLP adds parameters and computation to the connector, but the paper does not report the parameter count increase or the FLOPs overhead. Without this, the +31.4 MME improvement cannot be evaluated against its cost.

• No evaluation of inference latency tradeoffs. LLaVA-1.5-HD processes up to 1792 visual tokens vs. 576 for LLaVA-1.5 (3.1× increase). On a 13B LLM, this significantly increases per-token generation cost and latency. The improvements of +0.4-3.9 points across benchmarks (Table 4) are never weighed against this inference cost, which matters for practical deployment.

• The "~1 day training" claim is not compared to baselines under equivalent constraints. The paper emphasizes that LLaVA-1.5 trains in ~1 day on 8× A100s, but does not report how long baselines would take to train under equivalent hardware or when constrained to the same data (i.e., training InstructBLIP on only 558K pretraining pairs). The training efficiency claim is therefore about absolute wall-clock time for the specific LLaVA-1.5 recipe, not about whether LLaVA-1.5 is more training-efficient than competitors at equivalent data scales.

Single random seed and no error bars. All experiments are reported as point estimates from single training runs. This is standard practice for large model training papers (where multiple runs are prohibitively expensive), but it limits the interpretability of small improvements. The differences between some rows in Table 2 are in the range of 0.3-0.5 points on GQA and 0.4-1.5 points on MM-Vet—these could plausibly be within run-to-run variance. Without error bars or multiple seeds, these small improvements cannot be distinguished from noise. The paper's strongest claims (e.g., 30-point gaps vs. InstructBLIP, 700-point MME improvements) are large enough to be robust to variance, but the finer-grained ablation claims would benefit from statistical characterization.

The data efficiency downsampling experiment (Figure 4) uses only 7B models on a subset of benchmarks. The finding that 50% data maintains >98% performance is reported for the 7B model. Whether this scales to the 13B model (which might benefit more from data volume) is not tested. The benchmarks shown in Figure 4 are also a subset (GQA, ScienceQA-IMG, TextVQA, POPE, MME, MMBench, MMBench-CN, SEED-Bench, MM-Vet)—notably, VizWiz and LLaVA-Bench-in-the-Wild are absent, and these might show different data scaling behavior.

Overall, the experiments strongly support the paper's practical contribution—LLaVA-1.5 is a state-of-the-art, data-efficient, and reproducible baseline—but they provide only partial support for the more ambitious scientific claims about why the architecture is more data-efficient than alternatives. The controlled ablation in Table 2 is systematic but sequential rather than factorial, and the comparison against baselines conflates multiple design differences. The paper would be strengthened by head-to-head experiments that isolate specific factors (connector architecture, LLM fine-tuning, format prompts) while holding data and compute budgets constant across both LLaVA-style and Qformer-style architectures.

6. Limitations and Trade-offs

The Difficulty Estimation and Strategy Selection Cost Is Never Accounted For

The paper's entire adaptive allocation framework rests on the ability to classify each prompt into one of five difficulty quintiles before allocating the test-time compute budget. The method for doing this—generating 2048 complete solutions per question and computing either the pass@1 rate (oracle difficulty) or the average PRM final-answer score (predicted difficulty)—is extraordinarily expensive. Generating 2048 solutions per question costs more FLOPs than the largest test-time budgets studied in the paper (256–512 generations). Section 3.2 acknowledges this explicitly:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

This means the reported 4× efficiency gains over best-of-N (Figures 4 and 8) are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be cost(difficulty estimation) + cost(strategy execution), and the former could dominate the latter for all but the largest test-time budgets. A model deployed with this exact method would see its effective inference cost inflated by the 2048-sample overhead on every query, completely negating the efficiency gains the paper reports.

The paper also frames this as an exploration-exploitation tradeoff in Section 3.2 but provides no guidance on what this tradeoff looks like in practice. How many samples are actually needed for a reliable difficulty estimate? Could 128 samples suffice? 64? 32? The paper never sweeps the number of difficulty estimation samples or measures how strategy selection degrades as estimation quality declines. Without this characterization, a practitioner cannot determine whether the approach is practical at any budget level—they would need to independently establish how much difficulty estimation overhead their specific use case can tolerate.

Mitigation status: The paper explicitly flags this as "a key avenue for future work" (Section 3.2) and proposes training models to predict difficulty directly from the question text, but no such model is developed or evaluated. The predicted difficulty bins (using PRM scores instead of ground-truth correctness) remove the need for access to answers but do NOT reduce the computational overhead—they still require 2048 samples and scoring. The cost problem is entirely unaddressed in the current work.

Hard Problems Remain Fundamentally Unsolved—Test-Time Compute Cannot Substitute for Pretraining When Capability Is Absent

The paper's most sobering result appears consistently across experiments: on the hardest questions (difficulty bin 5, approximately the bottom 20% by base model pass@1), no test-time strategy provides meaningful improvement regardless of budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all search methods and all generation budgets from 4 to 256. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of sequential-to-parallel ratio with the revision model. In the FLOPs-matched comparison (Figure 9, bin 5, blue line), the scaling curve is essentially flat near 0–5% across all budgets, even at R ≪ 1 where the smaller model gets a large inference budget advantage.

This is not a failure of the specific methods—it reflects a fundamental bound on what test-time compute can achieve. If the base model's pass@1 rate is near zero on a class of problems, there are essentially no correct solutions in the proposal distribution for search to find or for revisions to refine toward. Search amplifies what already exists; it does not create new capability. The paper is transparent about this (Section 7, in the takeaway box), stating that test-time compute and pretraining compute "are not 1-to-1 exchangeable" and that on hard problems, "pretraining is almost always more effective."

The practical consequence is significant: for any deployment where the problem distribution includes a substantial fraction of genuinely difficult prompts (problems the base model cannot solve even with 2048 independent attempts), LLaVA-1.5's compute-optimal strategies offer no path to improvement. The model would need to be scaled up via pretraining or supplemented with an entirely different approach (retrieval, tool use, human-in-the-loop). The paper's framework provides no guidance on how to detect this regime at test time—a question that consistently fails under all strategies might be bin 5, or it might be bin 4 where the wrong strategy was selected, and the system has no way to distinguish these cases without ground-truth labels.

Mitigation status: The paper acknowledges this limitation transparently but does not attempt to address it. Section 8 mentions that "we did not experiment with PRM tree-search techniques in combination with revisions," which could potentially expand the set of problems where test-time compute helps, but the fundamental constraint—no correct samples in the proposal distribution—would still apply.

The Revision Model and PRM Are Trained on Outputs from a Different Model Distribution, Creating a Pervasive Distribution Shift Problem

The paper's revision model and process reward model are both trained on data generated by the base PaLM 2-S* model (few-shot prompted), but at inference time they operate on outputs from the fine-tuned revision model—which has been trained to produce different answers than the base model. This creates a distribution mismatch that the paper documents in multiple places but never fully resolves.

The most direct evidence appears in Appendix J (Figure 15a): the PRM trained on base model outputs underperforms the revision-specific ORM when scoring revision model outputs. At 64 generations, sequential + base-LM PRM achieves approximately 40% accuracy versus sequential + revision ORM at approximately 42%. A 2-point gap may seem small, but it indicates that the PRM's scores become less reliable when the proposal distribution shifts—exactly the scenario where over-optimization becomes more dangerous (Section 5.3, Figure 3 right). The paper acknowledges this implicitly by training a separate ORM for the revision experiments rather than reusing the PRM, but does not characterize how this distribution shift would affect the compute-optimal policy selection, which was optimized using base-model PRM scores.

The problem is more subtle for the revision model itself. The revision training data is constructed by pairing independently sampled correct and incorrect base-model outputs, using character-level edit distance to select "close" incorrect answers (Section 6.1). At inference time, however, the revision model sees its OWN previous outputs in context—not base-model outputs. The paper documents one consequence of this mismatch: the 38% correct-to-incorrect reversion rate, where the model "revises" a correct answer into an incorrect one because it was never trained on trajectories where the in-context answer is already correct (Section 6.1). The majority voting / verifier-based selection across the chain mitigates this at selection time but does not prevent it from happening—compute is wasted on generating incorrect revisions from correct starting points.

The ReST^EM experiment (Appendix K, Figure 16) provides more evidence that the distribution shift is severe: attempting to further optimize the revision model with on-policy RL-style training caused performance to degrade substantially, with fully sequential revisions dropping to approximately 33.5% at 256 generations compared to ~38.5% at the optimal ratio. The authors hypothesize that "on-policy data collection exacerbates spurious correlations in revision data," which is another way of saying the training procedure is fragile to the specific choice of data generation distribution.

Mitigation status: The paper pragmatically works around the distribution shift (separate ORM for revisions, majority voting to handle reversion) but does not solve it. The acknowledgment in Section 8 that combining PRM search with revisions was not attempted suggests the authors view the distribution shift between these components as a nontrivial barrier.

The FLOPs-Matched Comparison Uses a Weak Pretraining Baseline That the Paper Acknowledges Is Not Compute-Optimally Trained

The paper's headline finding that "a smaller model with compute-optimal test-time strategies can outperform a ~14× larger model" (Section 7, Figure 9) is a central claim that would influence how organizations allocate compute budgets. The baseline for this comparison is a model with approximately 14× more parameters than PaLM 2-S*, but critically, this larger model scales only parameter count while holding training data fixed—following the LLaMA paradigm (Touvron et al., 2023) rather than compute-optimal pretraining where both parameters and data are scaled proportionally (Hoffmann et al., 2022). The paper explicitly acknowledges this in Section 7:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

This matters because a Chinchilla-optimal model trained with 14× more total FLOPs (scaling both parameters and data by ~sqrt(14) ≈ 3.7×) would likely outperform a parameters-only-scaled model, making the pretraining baseline weaker than it needs to be. The reported advantages of test-time compute over pretraining—e.g., +27.8% on easy questions at R ≪ 1 (Figure 1 bar chart)—may shrink or even reverse against a properly compute-optimal larger model.

Furthermore, the 14× larger model is evaluated with greedy decoding only—no majority voting, no best-of-N, no test-time compute augmentation of any kind. Given that the paper's core argument is that test-time compute is valuable, a fairer comparison would allocate some of the total FLOPs budget to inference-time strategies for the larger model as well. For instance, at R ≪ 1 where inference FLOPs are small relative to pretraining, even a modest best-of-8 for the larger model would be a stronger baseline that is never tested. The paper's comparison essentially asks "small model + smart inference vs. large model + no inference" rather than "small model + smart inference vs. large model + smart inference, under total FLOPs constraint."

Mitigation status: The paper acknowledges the non-optimal pretraining baseline and defers the Chinchilla-optimal comparison to future work. The greedy decoding baseline is an explicit design choice justified by the desire for a "canonical" comparison, but it stacks the deck in favor of test-time compute. The absolute numbers from the FLOPs-matched comparison should be interpreted as an upper bound on test-time compute's advantage, not as a realized deployment gain.

All Findings Are on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*), with Questions About Representativeness

The paper's entire analysis—difficulty-dependent behavior, compute-optimal strategy selection, FLOPs-matched comparisons, verifier over-optimization patterns—is conducted on exactly one benchmark (MATH, 500 test questions) with one model family (PaLM 2-S*). Section 4 states that the authors "believe this model is representative of the capabilities of many contemporary LLMs," but this claim is not tested. Several aspects of the findings could plausibly be model-specific or benchmark-specific:

• PRM over-optimization dynamics depend on the verifier's calibration properties. A model with different output distributions (more/less calibrated confidence, different error patterns) might exhibit qualitatively different difficulty-dependent beam search behavior. The finding that beam search hurts on easy problems (Figure 3 right) could reverse if the verifier is better calibrated on easy questions.

• Revision model effectiveness depends on the base model's in-context learning capability—can it effectively use its own previous (incorrect) answers as informative context? This capability varies widely across model families, and the edit-distance-based data construction procedure (pairing incorrect and correct outputs) might work differently depending on how "structured" the base model's errors are.

• The difficulty bin thresholds are computed relative to PaLM 2-S*'s pass@1 distribution. A stronger or weaker base model would have different bin boundaries, and the optimal strategy per bin might shift. The paper provides no guidance on how difficulty-dependent policies transfer across model scales or families.

• MATH consists exclusively of competition-level math problems requiring multi-step symbolic reasoning. Whether the difficulty-dependent patterns—beam search hurting easy problems, revisions helping easy problems, neither helping hard problems—generalize to other reasoning domains (code generation, logical deduction, scientific question answering) or to tasks requiring factual recall rather than inference is unknown.

The test set of 500 questions, split into five difficulty quintiles of ~100 each, then further split by two-fold cross-validation, means the compute-optimal policy is selected based on ~50 questions per fold per bin. This is a small sample for strategy selection—a few outlier questions in any bin could meaningfully shift which strategy appears optimal. The paper does not report confidence intervals or bootstrap estimates of strategy selection stability.

Mitigation status: The paper acknowledges the single-model limitation only implicitly by stating their belief in representativeness. No experiments on other benchmarks, model families, or domains are conducted. The sample size concern is structural to the MATH benchmark (500 test questions) and cannot be addressed within the paper's framework without access to a larger test set or a different benchmark.

The Sequential Revision Strategy Introduces Serial Dependency That the Paper's "Generation Budget" Metric Ignores, Making Latency-Critical Deployments Infeasible

The paper measures test-time compute in "generations" (number of complete solutions sampled), which serves as a reasonable proxy for total FLOPs but completely ignores wall-clock latency. Sequential revisions are inherently serial: each revision depends on the previous one, and the model must autoregressively generate each revision token-by-token before the next revision can begin. In contrast, parallel best-of-N sampling can be executed simultaneously on sufficient hardware—128 independent samples can be generated in the time it takes to generate 1, assuming 128-way parallelism.

The paper's compute-optimal policy on easy questions (Figure 7 right, bin 2) favors purely sequential revision strategies, and even on medium questions (bin 3) favors a substantial sequential component. A strategy that allocates 128 generations as 8 sequential × 16 parallel takes approximately 8× longer wall-clock time than one that runs 128 parallel samples simultaneously, even though both cost "128 generations" in FLOPs terms. For latency-sensitive applications—interactive assistants, real-time tutoring systems, any deployment where users wait for responses—the sequential-heavy strategies favored by the compute-optimal policy may be impractical regardless of their accuracy advantages.

This tradeoff between total FLOPs and latency is not discussed anywhere in the paper. The "generation budget" abstraction, while useful for establishing equivalence classes of total computation, collapses an important design dimension that practitioners must contend with. A deployment under latency constraints might need to choose between: (a) lower accuracy with fast parallel sampling, (b) higher accuracy with slow sequential revisions, or (c) a larger model with greedy decoding (zero latency overhead, different accuracy profile). The paper's framework provides no guidance on this three-way tradeoff.

Mitigation status: Not addressed. The paper uses "generations" as the universal cost metric throughout and does not discuss latency, throughput, or deployment constraints. The sequential-to-parallel ratio analysis (Section 6, Figure 7) optimizes only for accuracy given a generation budget, not for accuracy given a latency budget. This is reasonable for a research paper establishing scaling principles, but it means the practical deployment recommendations are incomplete for any latency-sensitive setting.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper enacts a methodological shift with practical democratization consequences rather than a paradigm shift. It does not introduce a new architecture, training objective, or capability—LLaVA-1.5's components (MLP connector, format prompts, academic VQA data, full LLM fine-tuning) are individually known techniques. What changes is the standard for what counts as a valid LMM design claim and the accessibility of state-of-the-art multimodal research.

Prior to this work, the literature on large multimodal models operated under an implicit assumption: building a strong visual instruction-following model requires either architectural complexity (visual resamplers like Qformer, cross-attention modules, learned query tokens) or massive vision-language pretraining (100M+ image-text pairs), or both. InstructBLIP trains Qformer on 129M pairs; Qwen-VL trains a visual resampler on 1.4B pairs; IDEFICS uses 353M pairs. These design choices were presented as necessary ingredients, but no controlled study isolated their contributions. The field was accumulating systems without understanding which components mattered.

LLaVA-1.5 systematically dismantles this assumption. By starting from the simplest possible architecture—a frozen CLIP encoder, a two-layer MLP, and full LLM fine-tuning—and adding one modification at a time while measuring the marginal contribution on both academic and conversational benchmarks (Table 2), the paper establishes that:

1. Visual resamplers are unnecessary for strong multimodal performance. A 2-layer MLP trained on 558K pairs outperforms Qformer trained on 129M pairs (Table 3: +30.5 VQA-v2, +29.9 GQA for equivalently-sized models). The compression these resamplers perform—reducing visual features to a small fixed number of tokens—may actively harm performance by discarding information the LLM could otherwise use.

2. Massive vision-language pretraining is unnecessary when building on CLIP. Because CLIP's contrastive objective already aligns visual and textual semantics, the cross-modal connector only needs to learn a geometric translation between embedding spaces—a low-complexity mapping that modest data suffices for. The paper makes this point explicitly (Section 4.2): "raises questions upon the common belief that LMMs require significant amount of vision-language alignment pretraining."

3. The multitask balancing problem—short-form VQA vs. long-form conversation—is a prompt formatting issue, not an architectural one. InstructBLIP's failure on conversational benchmarks (producing "yes" when asked to explain in detail, Table 1a) was attributed to the difficulty of controlling a frozen LLM through Qformer tokens. LLaVA-1.5 shows that explicit natural language format instructions, combined with full LLM fine-tuning, solve the problem with no architectural mechanism for output length control. The 126.8-point MME improvement from adding format prompts alone (Table 2, Row 1→Row 2) isolates this effect.

The practical consequence is a dramatic reduction in the barrier to entry for LMM research. Training LLaVA-1.5 requires ~1 day on a single 8-A100 node with only publicly available data. This is qualitatively different from the multi-node, million-dollar pretraining runs competitors require. Academic labs that could not previously participate in state-of-the-art LMM development now have a reproducible, affordable baseline that achieves best-in-class performance across 11 benchmarks.

This also reorients research priorities. If visual resamplers and massive pretraining are unnecessary for current benchmarks, future work should invest in: (a) improving pretrained vision encoders (since the connector's simplicity means vision encoder quality directly determines LMM capability), (b) curating higher-quality instruction-tuning data mixtures (since data composition drives the remaining performance gaps), and (c) developing better evaluation benchmarks that stress capabilities current simple architectures cannot achieve (to determine whether resamplers become necessary at higher capability levels). The paper makes Qformer-style architectures look like over-engineered solutions to a problem that didn't exist—at least at the current performance frontier.

The paper also resolves a specific contradiction in prior work. Chen et al. [7] found that adding VQA data to visual instruction tuning causes models to overfit to short-form answers and lose conversational ability, and proposed converting VQA answers to conversational format via GPT-4 as a solution. LLaVA-1.5 demonstrates that this conversion is unnecessary—explicit format prompts achieve the same balancing effect with zero data transformation cost. The contradiction arose because [7]'s baseline used ambiguous prompts and did not fine-tune the LLM; format prompts + LLM fine-tuning resolve both issues.

What the paper does NOT change. The work does not advance the theoretical understanding of how multimodal models compose visual and linguistic information. It provides no new insight into attention patterns, representation geometry, or failure modes of cross-modal reasoning. It does not solve hallucination (despite the granularity-capacity mismatch hypothesis in Section 5.2, the quantitative hallucination improvements in POPE are marginal). It does not establish LMM scaling laws comparable to the text-domain scaling literature. And it does not demonstrate new capabilities beyond what was already achievable with more expensive approaches—it matches or modestly exceeds existing performance with dramatically less resource investment. The contribution is methodological (how to do LMM research properly) and practical (how to build strong LMMs cheaply), not conceptual.

Follow-Up Research This Work Enables

1. Systematic study of vision encoder quality vs. connector complexity tradeoffs. The paper's central claim—that a simple connector suffices when the vision encoder is strong—is tested with exactly one vision encoder (CLIP-ViT-L). A natural follow-up would train LLaVA-1.5-style models with varying vision encoders (CLIP vs. OpenCLIP vs. EVA-CLIP vs. DINOv2 vs. SigLIP, spanning different pretraining objectives, resolutions, and parameter counts) while varying connector complexity (linear, 1-layer MLP, 3-layer MLP, lightweight Qformer) and measuring whether the "simple connector" advantage holds universally or depends on encoder quality. The key metric would be the interaction effect: does connector complexity matter more with weaker vision encoders? Does a Qformer ever outperform an MLP when the vision encoder is randomly initialized or weakly pretrained? This would establish the boundary conditions for the paper's central architectural claim.

2. Data mixture optimization with formal experimental design. The paper's data scaling in Table 2 adds datasets sequentially in a fixed order, and the data efficiency experiment (Figure 4) performs random downsampling of the final mixture. Neither answers the question: what is the optimal composition of the 665K training mixture under a fixed data budget? A follow-up could use design-of-experiments methods (fractional factorial design, Bayesian optimization over dataset inclusion weights) to measure interaction effects—e.g., does the benefit of region-level VQA data depend on whether OCR data is already present? Does ShareGPT data help only when VQA data is abundant? The paper's finding that 50% data yields >98% performance suggests substantial redundancy; identifying which datasets are redundant vs. essential would produce a minimal high-quality training recipe, advancing the "less-is-more" agenda the paper gestures toward.

3. Scaling the number of visual tokens—does information density or sequence length dominate? LLaVA-1.5 processes 576 visual tokens (336² images with ViT-L/14). LLaVA-1.5-HD processes up to 1792 tokens. The pattern in Table 2 (resolution scaling improves MME +23.5 but slightly degrades MM-Vet -0.5 at 7B) suggests a tension: more visual tokens provide finer-grained information but may dilute the LLM's attention or push important visual details beyond the effective context window. A follow-up could systematically vary the number of visual tokens (by varying ViT patch size at fixed resolution, or by varying resolution with fixed patch size) while measuring performance on benchmarks that require global scene understanding vs. fine-grained detail perception. The result would characterize whether current LMMs are bottlenecked by visual information quantity or by the LLM's ability to process long visual sequences—a distinction with direct implications for whether future work should focus on better vision encoders or better long-context LLMs.

4. Multilingual multimodal instruction tuning with controlled language coverage. LLaVA-1.5 demonstrates emergent multilingual visual conversation (Figure 5) from English-only visual data + multilingual text-only ShareGPT data, but errors persist (Korean responses contain mistakes). A controlled follow-up would vary the multilingual coverage of the text-only data (e.g., ShareGPT subsets filtered by language: English-only, English+Spanish, English+Spanish+Japanese) while measuring visual conversation quality in each language using translated MMBench and LLaVA-Bench prompts. This would characterize the compositional transfer curve: how much target-language text data is needed for a given level of visual conversation quality in that language? The result would inform whether the "compositional capabilities" finding (Section 5.3) is a scalable path to multilingual LMMs or a limited phenomenon that requires explicit multilingual visual instruction data for high-quality results.

5. High-resolution scaling with compute-matched comparisons. LLaVA-1.5-HD improves performance on detail-oriented benchmarks (VizWiz +3.9, MM-Vet +3.3) at the cost of processing 3.1× more visual tokens (up to 1792 vs. 576). But the paper never matches the compute budget between resolution variants—LLaVA-1.5-HD simply uses more inference FLOPs. A strong follow-up would compare resolution scaling against alternative uses of the same additional FLOPs: e.g., LLaVA-1.5 at 336² with best-of-N majority voting (3 samples to match inference cost) vs. LLaVA-1.5-HD at 448² with greedy decoding. The question is whether spending extra FLOPs on higher resolution is more efficient than spending them on additional sampling at lower resolution. This would establish whether the split-encode-merge strategy is genuinely a compute-efficient approach to improved perception or merely an accuracy-for-FLOPs trade that any method could replicate by spending more compute.

6. Stress-testing the format prompt approach on ambiguous or conflicting instructions. The paper shows that explicit format prompts work when the desired format is clearly specified ("Answer using a single word") and the test instruction matches the training format prompts. What happens when instructions are ambiguous, conflicting, or adversarial? For example: "Answer using a single word, but also explain your reasoning." Or format instructions that contradict the expected evaluation format for a benchmark (training with "Answer briefly" but evaluating on a benchmark that scores long-form responses). A systematic stress-test would characterize whether the model has learned a generalizable skill of following format instructions or has merely memorized a set of prompt templates—the distinction matters for deployment robustness when users provide unpredictable formatting requests.

Practical Applications and Downstream Use Cases

1. Academic research on large multimodal models with limited compute budgets. The most immediate application of LLaVA-1.5 is as a reproducible baseline for LMM research in academic labs. Prior to this work, an academic group wanting to study visual instruction tuning had two choices: use a weak model that doesn't represent state-of-the-art behavior, or invest in prohibitively expensive pretraining (129M+ image-text pairs, multi-node training runs) just to reach a competitive starting point. LLaVA-1.5 eliminates this dilemma. A research group with access to a single 8-A100 node can train the full 13B model in ~1 day, achieve state-of-the-art performance on 11 benchmarks, and then use it as a testbed for studying open problems—hallucination intervention, compositional generalization, adversarial robustness, fairness in multimodal outputs—without the confounding factor of a subpar base model. The paper's release of code and model weights makes this immediately actionable.

2. Cost-efficient deployment of visual assistants for routine visual question answering. For organizations deploying visual QA systems at scale (e.g., customer support for product images, accessibility tools for visually impaired users generating image descriptions, document understanding pipelines), LLaVA-1.5 offers a clear cost advantage. The 7B model achieves 78.5% on VQA-v2, 50.0% on VizWiz, and 1510.7 on MME—performance levels that, prior to this work, were associated with models requiring 100× more pretraining data. A small model that can run on a single GPU with strong visual understanding capability changes the economics of deployment: lower hardware requirements, faster inference (fewer parameters), and no dependency on proprietary models or cloud APIs. The fact that LLaVA-1.5 uses only public data is additionally important for commercial deployments concerned about data provenance and licensing.

3. Data annotation and curation pipelines for multimodal datasets. LLaVA-1.5's ability to follow diverse format instructions (single-word answers, multiple choice, JSON output, descriptive paragraphs) and its strong performance on region-level tasks (Visual Genome, RefCOCO) make it a candidate for automated dataset annotation. For example, generating dense region descriptions for object detection datasets, answering compositional questions about images for VQA dataset expansion, or extracting structured information (Table 6: driver's license fields) from document images. The model's demonstrated ability to generalize to unseen format instructions (VizWiz "Unanswerable" response, JSON formatting) without task-specific fine-tuning means a single deployed model could serve multiple annotation tasks with different output schemas—reducing the engineering overhead of maintaining task-specific annotation models.

4. Multilingual visual assistance from English-only visual training. LLaVA-1.5's emergent multilingual capability (Figure 5, MMBench-CN 63.6 vs. Qwen-VL-Chat 56.7, despite Qwen being trained on Chinese multimodal data) is a practically significant finding for organizations serving multilingual user bases. The model can respond to visual questions in Spanish, Japanese, Chinese, and other languages present in ShareGPT, even though all visual instruction data is English. This means a single LLaVA-1.5 deployment can serve users in multiple languages without requiring language-specific visual instruction data, translation pipelines, or separate models per language. The capability is imperfect (Korean errors documented in Figure 5) but functional, and the paper's analysis suggests that expanding the multilingual text-only data (ShareGPT coverage) would improve the relevant languages at near-zero marginal cost for visual data collection.

When to Prefer This Method

The paper explicitly positions LLaVA-1.5 against approaches that use complex visual resamplers (Qformer in InstructBLIP, visual resampler in Qwen-VL) and massive vision-language pretraining (129M–1.4B pairs). The decision rule that emerges from the empirical results is:

Prefer LLaVA-1.5's simple connector + full LLM fine-tuning + format prompt approach when:

• Compute budget for vision-language pretraining is limited (you cannot afford 100M+ image-text pair pretraining and/or multi-node training runs). The paper demonstrates that 558K public pretraining pairs on 8× A100s for ~6 hours achieves better downstream performance than 129M+ pairs with complex connectors.
• The base vision encoder is strong and well-aligned with language (CLIP, OpenCLIP, EVA-CLIP). The paper's architecture relies on the vision encoder already mapping images to a semantically meaningful space; with a weaker encoder, the simple connector may not suffice.
• The deployment requires balancing multiple output formats (short-form VQA, long-form conversation, structured JSON, multiple-choice). Format prompts with full LLM fine-tuning provide flexible output control without architectural modifications.
• Reproducibility and public data are requirements—LLaVA-1.5 uses only publicly available data, making it suitable for academic research, benchmarking, and commercial deployments with data provenance constraints.
• Training speed matters—the full pipeline completes in ~1 day on a single 8-A100 node.

The paper does NOT present evidence that the simple connector approach is preferable when:

• Inference latency is the primary constraint and visual token compression is necessary (e.g., deploying on edge devices with strict per-token latency budgets). The paper processes all visual patches, which means more tokens, more attention computation, and higher per-token generation cost. Resamplers like Qformer that compress visual features into fewer tokens may be preferable in latency-critical scenarios despite their higher training cost.
• The task requires cross-modal retrieval or generation beyond instruction following (e.g., image-text matching, text-to-image retrieval). Qformer was designed for these tasks and the paper does not evaluate LLaVA-1.5 on them.
• The vision encoder is not pretrained on web-scale image-text data (e.g., a supervised ImageNet-pretrained ViT, or a randomly initialized encoder). The simple connector's data efficiency depends on CLIP's existing semantic alignment; without it, more complex alignment mechanisms and more pretraining data may be necessary.
• The goal is to push the absolute performance frontier regardless of training cost—the paper matches or modestly exceeds prior work, but does not demonstrate that the simple architecture has a higher performance ceiling than more complex alternatives given unlimited resources.