An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models

ArXiv: 2309.09958

🎯 Pitch

Scaling LLaVA from 13B to 65B delivers consistent multimodal gains, but the real surprise is that visual instruction tuning actually boosts pure language capability—the model gets better at text-only tasks through multimodal training. Meanwhile, LoRA and QLoRA tuning achieve near-identical performance to full fine-tuning at a fraction of the cost, with simple data mixing of multimodal and language-only instructions giving the strongest results overall.

---

1. Executive Summary

This paper presents an empirical study of scaling LLaVA (Large Language and Vision Assistant) up to 33B and 65B/70B parameters, systematically evaluating how model size, image resolution, data mixing, and parameter-efficient training methods affect multimodal and language capabilities on LLaVA-Bench and MM-VET. The work investigates three scaling factors—model size (expanding from 7B to 65B), image resolution (224×224 versus 336×336), and data mixing (interleaving language-only ShareGPT instructions with multimodal LLaVA-80K data)—alongside LoRA/QLoRA tuning (low-rank adaptation with ranks 8 to 64, versus full-model fine-tuning). Scaling from 13B to 65B lifts LLaVA-Bench overall scores from 70.1 to 74.2, and MM-VET total scores from 32.5 to 36.4, with higher image resolution contributing an additional 2–3 points consistently across all model sizes. LoRA/QLoRA tuning reaches comparable performance to full-model fine-tuning while reducing training cost substantially (e.g., 33B with LoRA rank 64 trains in 5.8 GPU-hours per node versus 9.17 for full fine-tuning), though performance gains from increasing rank diminish—rank 64 achieves 71.8 versus 72.0 for full tuning on the 33B model, establishing that parameter-efficient methods are a viable substitute for full fine-tuning only when paired with sufficiently large learning rates and LoRA alpha values.

2. Context and Motivation

The Core Problem: Scaling Multimodal Models Beyond the 13B Frontier

In the wake of GPT-4's release in early 2023, researchers observed something remarkable: proprietary multimodal models that could analyze images, answer questions about them, and engage in visual dialogue were achieving strikingly capable results. The open-source community responded rapidly with projects like LLaVA and MiniGPT-4, which showed that visual instruction tuning—fine-tuning a pretrained language model on image-text instruction pairs—could reproduce some of the most compelling multimodal behaviors at a fraction of the cost. LLaVA's original paper demonstrated that training a 7B Vicuna model for just 3 hours on 8 A100 GPUs produced a system with strong visual reasoning in the wild.

However, by mid-2023, the field had hit a scaling wall—not because scaling was known to be ineffective, but because the cost of experimenting at larger scales was prohibitive for most academic and open-source research groups. The LLaVA recipe had been validated at 7B and 13B parameter scales, but the behavior of this approach when applied to 33B, 65B, or 70B models remained entirely unexplored. This gap matters because the leap from 13B to 65B represents a fundamentally different computational regime: full-model fine-tuning a 65B LLM requires approximately 4 nodes of 8 A100-80G GPUs, compared to half that for the 13B model, placing such experiments financially out of reach for many labs even if the scientific question is straightforward.

This paper explicitly frames this as an access and knowledge barrier in Section 1:

"the high compute cost has led most existing studies to utilize 7B and 13B LLMs. Thus, the impact of significantly scaling up the model size to e.g., 33B and 65B remains unexplored."

The "gap" is therefore both empirical—we don't know how LLaVA-style visual instruction tuning behaves at scale—and pragmatic—we lack reference implementations and baselines that would let smaller groups build on large-model results without independently reproducing the expensive training runs. The paper positions itself as a bridge: by conducting the scaling experiments and releasing checkpoints, it aims to "make state-of-the-art LMM research at a larger scale more accessible, thus helping establish stronger baselines for future research."

Why This Problem Matters: Real-World Impact and Theoretical Significance

The scaling question carries weight for several distinct but interconnected reasons.

Practical deployment economics. As the paper demonstrates in Table 1, smaller open-source LMMs trained with visual instruction tuning can already approach or exceed the performance of commercial systems like Bing Chat and Google Bard on certain benchmarks. If scaling from 13B to 65B yields consistent improvements—particularly on capabilities like complex reasoning and detailed description generation that depend on strong language competencies—then open-source models become viable substitutes for proprietary APIs in an expanding range of applications. This matters for organizations that cannot or will not send visual data to third-party services due to privacy, latency, or cost constraints. The paper makes this point indirectly through its benchmark comparisons: LLaVA-65B scores 74.4 overall on LLaVA-Bench versus Bard's 77.8 and Bing Chat's 71.5, demonstrating that an open-source model can be competitive with the most advanced commercial offerings at the time.

The composition of multimodal capabilities. A deeper scientific question lurks beneath the scaling experiments: which capabilities improve with model size, and which do not? If scaling only improves language fluency while leaving visual recognition untouched, the return on investment is fundamentally different than if it unlocks new forms of integrated reasoning. The MM-VET benchmark used in this paper (Table 2) decomposes performance into six core vision-language capabilities—recognition, OCR, knowledge, language generation, spatial awareness, and math—allowing the authors to trace precisely where scaling helps. The finding that knowledge and generation improve most dramatically (LLaVA-65B + Data Mixing scores 30.4 on knowledge and 32.3 on generation versus LLaVA-13B's 23.5 and 26.4) while math and spatial awareness remain roughly flat suggests that scaling the language model disproportionately enhances capabilities that depend on stored knowledge and fluent output generation, not those requiring visual precision or symbolic reasoning. This is a non-obvious result that has implications for how resources should be allocated in multimodal model development—it implies that further scaling the LLM backbone may yield diminishing returns on perceptual tasks unless the vision encoder is also scaled.

The cost-performance frontier. Perhaps the most practically actionable question the paper addresses is whether parameter-efficient methods like LoRA and QLoRA can substitute for full-model fine-tuning at larger scales. Full fine-tuning a 65B model costs on the order of 13.5 GPU-hours per node per epoch (Table 4), making iterative experimentation—hyperparameter sweeps, data ablations, architecture variations—financially painful. If LoRA with sufficiently high rank can match full fine-tuning performance at a fraction of the cost, the barrier to entry for large-scale multimodal research drops dramatically. The paper's demonstration that LoRA rank 64 on a 33B model achieves 71.8 on LLaVA-Bench versus 72.0 for full fine-tuning, at roughly 63% of the training cost, has direct implications for how research groups of different sizes should allocate their compute budgets.

Prior Approaches and Where They Fall Short

LLaVA and MiniGPT-4: the small-model baseline. The original LLaVA paper established the two-stage visual instruction tuning paradigm: Stage 1 aligns a frozen vision encoder's features to the LLM's embedding space via a trainable projection layer, and Stage 2 fine-tunes on multimodal instruction data. This was validated on 7B and 13B Vicuna models, with LLaVA-13B achieving 70.1 on LLaVA-Bench and 26.4 on MM-VET (Tables 1 and 2). MiniGPT-4 explored a similar idea concurrently, using a single projection layer to connect a frozen vision encoder to a frozen Vicuna LLM, achieving 22.1 on MM-VET with an 8B model and 24.4 with a 14B variant. These works established the feasibility of visual instruction tuning but left the scaling axis unexplored—both the data scaling (LLaVA-80K is relatively small) and the model scaling questions were unanswered.

BLIP-2 and InstructBLIP: alternative architectures at comparable scales. BLIP-2 introduced a Q-Former module that sits between the vision encoder and LLM, learning to extract visual features relevant to the language model. InstructBLIP extended this with instruction-aware visual feature extraction by feeding the instruction text into the Q-Former. These models operated at the 8B–14B scale, achieving 22.4 (BLIP-2-12B) and 26.2 (InstructBLIP-8B) on MM-VET. While architecturally different from LLaVA, they shared the same fundamental limitation: no results above the ~14B parameter threshold, and no systematic investigation of how their design choices would interact with much larger language models.

Flamingo and OpenFlamingo: the few-shot paradigm. Flamingo pioneered a different approach, interleaving pretrained vision and language models with cross-attention layers and training on interleaved image-text sequences rather than instruction-following data. OpenFlamingo reproduced this architecture at 9B parameters, achieving 21.8 on MM-VET. The Flamingo family demonstrated strong few-shot visual understanding but was not designed for the conversational, instruction-following modality that LLaVA targets. Its scaling behavior also remained undocumented in the open literature at the 33B+ scale.

Proprietary systems: the opaque ceiling. At the time of this paper's writing, GPT-4 and its multimodal capabilities represented an upper bound that the open-source community could not reproduce, understand, or systematically improve upon. MM-ReAct, which chains GPT-4 with visual tools, achieved 44.6 on MM-VET—substantially above any end-to-end open-source model. Bard and Bing Chat, while accessible via API, provided no transparency into their architecture, training data, or scaling methodology. This opacity meant that open-source researchers lacked a clear picture of what architectural choices, data strategies, and scaling investments were necessary to approach proprietary performance levels.

Where all prior approaches fell short. The common thread across these prior works is that they answered one type of question—"does visual instruction tuning work?" or "is this architecture viable?"—but left open the systematic scaling question: how do the gains distribute across different capabilities as you scale? What is the return on investment for each additional 10 billion parameters? Do parameter-efficient methods remain competitive as models grow, or does the gap between LoRA and full fine-tuning widen? The paper notes this explicitly in Section 3.2, where it organizes its investigation around three directed research questions: which scaling factors matter, when should parameter-efficient training be considered, and can we build an LMM with strong capabilities in both language and multimodal domains. These questions had no answers in the prior literature because no one had run the experiments.

How This Paper Positions Itself Relative to Existing Work

The paper positions itself not as proposing a new architecture or training paradigm, but as providing the missing empirical foundation that enables the research community to reason about scaling decisions. This is a deliberate and important distinction: the paper's contribution is the set of reference experiments and derived recommendations, not the methods themselves. The LLaVA architecture, two-stage training procedure, and evaluation benchmarks are all inherited from prior work. What is new is the combination of:

Scale: conducting experiments at 33B and 65B/70B. No prior open-source visual instruction tuning work had operated at these parameter counts. The paper fills this gap with systematic comparisons across four model sizes (7B, 13B, 33B, 65B) on multiple benchmarks. The scaling curves themselves—how performance changes as a function of model size—were previously unknown.

Ablation across three scaling factors simultaneously. Prior work on scaling typically varied one factor at a time (usually model size). This paper examines model size, image resolution, and data mixing jointly (Table 3a), allowing it to estimate the relative contribution of each factor. The finding that image resolution consistently contributes 2–3 points across all model sizes, and that data mixing contributes an additional ~2 points, provides concrete guidance on where to invest compute resources when scaling up.

Cost-effectiveness analysis of parameter-efficient methods. The paper doesn't just report that LoRA works—it provides a cost-performance matrix (Table 4) that lets practitioners make economic decisions about which tuning method to use at which model scale. The finding that increasing LoRA rank from 8 to 64 provides meaningful gains (70.3 → 71.8 on LLaVA-Bench for the 33B model) but that further increases to 128 or 512 yield diminishing returns (65.5 → 66.1 → 68.1 under suboptimal hyperparameters) establishes a practical saturation point. The explicit reporting of GPU-hours per node enables direct cost comparison across methods, which is rare in research papers but essential for deployment decisions.

The surprising language capability finding. The paper's most novel empirical result is arguably in Table 5: visual instruction tuning not only preserves but can actually improve pure language capability. The LLaVA-70B model based on LLaMA-2-70B-Chat achieves 65.1 on MMLU, which is 2.4 points higher than the base LLaMA-2-70B-Chat (62.5 → 65.1, transposed from Vicuna-65B's 62.5 in Table 5). The authors explicitly flag this as a first-of-its-kind finding:

"To the best of our knowledge, this is the first reported result which shows visual instruction tuning improve language ability of large-scale LMM."

This result challenges the assumption that adding a visual modality necessarily comes at the cost of language performance—instead, the complex reasoning and long-form answer generation present in multimodal instruction data may provide a form of cross-modal transfer that strengthens the underlying language model. The paper doesn't claim to fully explain this phenomenon, but establishing its existence at scale is a significant contribution that opens new research directions.

Establishing reference baselines. By releasing checkpoints and code, the paper lowers the barrier for follow-up work. Prior to this study, a researcher wanting to experiment with, say, a new visual instruction tuning dataset at the 65B scale would first need to reproduce the expensive baseline training run before even beginning their own experiments. By providing trained checkpoints, the paper makes it possible for other groups to build directly on large-scale LLaVA models, accelerating the pace of research.

In essence, the paper occupies a crucial but often undervalued role in the research ecosystem: it is an infrastructure contribution that converts the hypothetical possibility of scaling into concrete, measured outcomes, enabling the rest of the community to make informed decisions about where to invest their own limited compute resources.

3. Technical Approach

3.1 Reader Orientation

What is being built: The paper constructs a family of large multimodal models by taking a pretrained vision encoder and a pretrained large language model, connecting them with a trainable linear projection layer, and then fine-tuning the combined system on multimodal instruction-following data.

What problem this solves: It provides a systematic empirical answer to the question "what happens when you scale visual instruction tuning from 13B to 65B parameters, and which training recipes give the best performance-per-dollar at each scale?" The solution takes the form of a set of reference experiments that vary three factors—model size, image resolution, and data mixing—while comparing full-model fine-tuning against parameter-efficient alternatives (LoRA and QLoRA), producing concrete recommendations about when to use each method and what performance gains to expect.

3.2 Big-Picture Architecture (Diagram in Words)

The LLaVA system has four major components connected in a pipeline:

1. Vision Encoder (frozen): A pretrained CLIP ViT-L/14 model that converts an input image into a sequence of visual feature vectors. This encoder is never updated during training—its weights remain exactly as they were after CLIP pretraining. It outputs features from the layer before the final projection, giving a grid of patch-level representations.

2. Linear Projection Layer (trainable): A single weight matrix that maps the vision encoder's output dimension (1024 for ViT-L/14) to the language model's embedding dimension (e.g., 6656 for the 33B Vicuna model, 8192 for the 65B Vicuna model). This is the only component trained in Stage 1 and serves as a learned "translation" between the visual and linguistic representation spaces.

3. Large Language Model (trainable or frozen depending on method): A pretrained instruction-tuned LLM—Vicuna for most experiments, LLaMA-2-Chat for the 70B variant—that receives the projected visual features concatenated with text tokens and generates text responses autoregressively. In Stage 2, this model is either fully fine-tuned or adapted via LoRA/QLoRA.

4. Training Data Pipeline (two-stage):

   • Stage 1 uses a concept-balanced subset of the LAION-CC-SBU dataset with 558K image-caption pairs to train only the linear projection layer.
   • Stage 2 uses the LLaVA-80K multimodal instruction dataset (with optional mixing of language-only ShareGPT data) to fine-tune the LLM and projection layer jointly.

Information flows as follows during inference: an input image is processed by the frozen CLIP ViT → the resulting visual features pass through the trained linear projection → the projected features are treated as "visual tokens" and prepended to the text instruction tokens → the combined sequence is fed into the LLM, which generates a text response autoregressively.

3.3 Roadmap for the Deep Dive

• First, the two-stage training protocol—what happens in Stage 1 (feature alignment) versus Stage 2 (visual instruction tuning), why the stages are separated, and what data is used in each—since this is the foundation that all model variants share.
• Second, the scaling factors and their configurations—how model size, image resolution, and data mixing are varied independently, with specific numbers for each experiment, so we can understand what "scaling LLaVA" concretely means.
• Third, parameter-efficient training methods (LoRA and QLoRA)—the mechanics of low-rank adaptation, the hyperparameter search that revealed learning rate and alpha as critical knobs, and the cost-performance tradeoffs quantified in GPU-hours, since this is where the paper's most practically actionable recommendations emerge.
• Fourth, the evaluation framework—the benchmarks (LLaVA-Bench, MM-VET, MM-Bench, Vicuna-80, MMLU), the scoring methodology (GPT-4 as evaluator for open-ended outputs), and beam search as a test-time enhancement—because the choice of evaluation determines what "improvement" means and how it should be interpreted.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical scaling study whose core idea is that LLaVA's visual instruction tuning recipe can be extended to much larger language models (33B–70B), and that the choice of training method (full fine-tuning versus LoRA/QLoRA), image resolution, and data mixing strategy significantly impacts the performance-per-cost tradeoff at each scale.

---

The Two-Stage Training Protocol

The training procedure follows exactly the two-stage recipe established in the original LLaVA paper, applied now to larger models with careful hyperparameter tuning at each stage.

Stage 1: Pre-training for Feature Alignment

What happens. The linear projection layer—a single matrix $W_{\text{proj}} \in \mathbb{R}^{d_{\text{vision}} \times d_{\text{LLM}}}$ where $d_{\text{vision}} = 1024$ (the output dimension of the CLIP ViT-L/14 feature extraction layer) and $d_{\text{LLM}}$ depends on the language model (6656 for Vicuna-33B, 8192 for Vicuna-65B)—is trained to map visual features into the LLM's word embedding space. During this stage, both the vision encoder and the LLM are completely frozen. Only the projection matrix receives gradient updates.

Training data. The Stage 1 training uses "the concept-balanced subset of LAION-CC-SBU data with 558K samples" (Section 2). Each sample consists of an image paired with a text caption. The model is trained to generate the caption text conditioned on the image, which teaches the projection layer to produce visual token representations that the LLM can interpret as meaningful input—essentially learning a direct mapping from CLIP's visual feature space to the embedding space the language model already understands.

Why this stage is needed. The LLM was pretrained on text tokens exclusively. Its embedding space encodes semantic relationships between words and concepts expressed in language. The CLIP vision encoder was trained to align images and text in a shared contrastive embedding space, but its output features live in a different vector space with different dimensionality and different semantic organization. Without Stage 1 training, feeding raw CLIP features directly into the LLM would be like feeding the language model "words" from a vocabulary it has never seen—the representations would be essentially random noise from the LLM's perspective. The projection layer learns a translation that makes visual features look approximately like "visual words" in the LLM's native embedding language, initializing the system so that Stage 2 can focus on instruction-following rather than basic image-to-text alignment.

Training configuration. Stage 1 uses "a learning rate of $1 \times 10^{-4}$ with no weight decay, and a learning rate with linear decay and linear warmup for 3% of training steps in total" (Section 2, Hyper-parameters). The maximum sequence length is 2048 tokens. The DeepSpeed library with ZeRO3 optimizer is used for all full-model training runs, except QLoRA runs which use ZeRO2.

Stage 2: Visual Instruction Tuning

What happens. Both the linear projection layer and the LLM are trained jointly on multimodal instruction-following data. In full-model fine-tuning, all LLM parameters are updated. In LoRA/QLoRA, only low-rank adapter matrices are updated (the base LLM weights remain frozen). The vision encoder stays frozen throughout Stage 2—a deliberate design choice that preserves the visual representations learned during large-scale CLIP pretraining and keeps training costs manageable.

Training data. The core dataset is LLaVA-80K, which consists of approximately 80,000 multimodal instruction-following examples. Each example includes an image, a user instruction or question about the image, and a target assistant response. When data mixing is applied, language-only instruction data from ShareGPT (conversations collected from the ShareGPT.com platform, containing purely textual user-assistant exchanges) is interleaved with the multimodal data. For the 33B Vicuna model, the ShareGPT data contains approximately 370 million tokens (as noted in Section 2 when discussing the Vicuna-33B checkpoint). For the 65B model, the authors processed their own ShareGPT data containing 159 million tokens.

Training configuration for full fine-tuning. Stage 2 uses "a learning rate of $2 \times 10^{-5}$ in full fine-tuning to train 1 epoch for all the models" with a total batch size of 512 across 4 A100 nodes (each with 8 A100-80G GPUs). All other hyperparameters—sequence length (2048), DeepSpeed ZeRO3, linear decay with warmup—are inherited from Stage 1.

Training configuration for LoRA/QLoRA. For parameter-efficient training, the learning rate is increased to $1 \times 10^{-4}$, and the LoRA alpha (a scaling factor applied to the low-rank update) is set to 2 times the LoRA rank. The total batch size is reduced to 64, running on 1 A100 node for the 33B model and 2 nodes for the 65B model. The paper explicitly notes that a hyperparameter search was conducted and that "larger LoRA alpha or equivalently larger learning rate was crucial to get the best performance" (Section 2).

---

Scaling Factors and Their Configurations

The paper manipulates three independent variables to study their effects on multimodal performance.

Model Size

Four language model sizes are compared: 7B, 13B, 33B, and 65B (with an additional 70B variant using LLaMA-2-70B-Chat instead of Vicuna). The 7B and 13B models use existing public Vicuna checkpoints. The 33B model uses the public Vicuna-33B-v1.3 checkpoint. The 65B model required the authors to train their own Vicuna-65B from scratch, since no public 65B Vicuna checkpoint existed at the time.

The linear projection dimensions scale with the LLM's embedding size:

• For 7B and 13B Vicuna (both using LLaMA-1 architecture with embedding dimension 5120): projection is $1024 \rightarrow 5120$
• For 33B Vicuna (embedding dimension 6656): projection is $1024 \rightarrow 6656$
• For 65B Vicuna (embedding dimension 8192): projection is $1024 \rightarrow 8192$

The key observation here is that the projection layer grows linearly with the LLM's embedding dimension, but this represents a negligible fraction of total parameters—0.02% to 0.05% of the full model size—so the scaling behavior is dominated by the LLM's capacity, not the connector.

Image Resolution

The pretrained CLIP ViT-L/14 vision encoder operates on fixed-size square inputs. The paper compares two resolution variants:

• $224 \times 224$ pixels: The standard CLIP input resolution. Images are resized and center-cropped to this square before encoding.
• $336 \times 336$ pixels: A higher-resolution variant where the ViT is fine-tuned (during CLIP pretraining, not during LLaVA training) to accept larger inputs. This produces a $24 \times 24$ grid of patch features (each patch covering $14 \times 14$ pixels, with $336/14 = 24$ patches per side) compared to the $16 \times 16$ grid at $224 \times 224$ resolution.

The resolution change affects LLaVA through the number of visual tokens fed into the LLM. At $224 \times 224$, the ViT produces 256 patch representations (plus a CLS token, which is typically discarded or used separately). At $336 \times 336$, it produces 576 patch representations. Since these are all treated as tokens in the LLM's input sequence, higher resolution consumes more of the 2048-token context window, leaving less room for text instructions and generated responses. The paper implicitly evaluates the tradeoff: the 2–3 point improvement from $336 \times 336$ (Table 3a) comes at the cost of roughly doubling the number of image tokens.

Data Mixing

Data mixing refers to whether the Stage 2 training uses only the LLaVA-80K multimodal instruction data, or whether it also includes language-only ShareGPT conversation data. The mixed configuration interleaves both types of examples during training, with the goal of preserving the LLM's original language capabilities while adding multimodal skills.

The mechanism is straightforward: during training, some batches contain image-text pairs (requiring the model to process visual input and generate appropriate responses), and other batches contain text-only conversations (requiring the model to behave as a standard language model). The model sees both types of examples within each epoch, with the mixing ratio implicitly determined by the relative sizes of the two datasets. The paper does not specify an explicit mixing ratio—presumably the two datasets are concatenated and sampled uniformly, meaning the ratio is approximately the ratio of their sizes (80K multimodal examples versus the available ShareGPT conversation turns).

The motivation is that training exclusively on multimodal data might cause the model to "forget" some of its language-only instruction-following capabilities—a form of catastrophic forgetting where the model overfits to the visual domain. By including language-only data, the model is encouraged to maintain its general conversational abilities. The surprising result in Table 5—that data mixing sometimes improves multimodal performance while not significantly affecting language performance—suggests that the relationship is more nuanced than simple forgetting prevention.

---

Parameter-Efficient Training Methods: LoRA and QLoRA

LoRA Mechanics

LoRA (Low-Rank Adaptation) decomposes weight updates into a low-rank factorization. For a weight matrix $W_0 \in \mathbb{R}^{d \times k}$ in the LLM, LoRA represents the update as:

$W = W_0 + BA$

where $B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$ are trainable matrices, and $r$ is the rank (a hyperparameter much smaller than $d$ and $k$). During training, $W_0$ is frozen—it receives no gradient updates—while $B$ and $A$ are updated via standard backpropagation.

What this computes: For each forward pass, the model multiplies its input activation $x$ by $W_0$ as usual, then adds the correction term $BAx$. Since $r \ll \min(d, k)$, the number of trainable parameters is $r(d + k)$, which is orders of magnitude smaller than the $d \times k$ parameters in $W_0$. At inference time, the product $BA$ can be computed once and merged into $W_0$: $W_{\text{merged}} = W_0 + BA$, making inference exactly as efficient as the original model.

Why this form over alternatives: Full fine-tuning updates all $d \times k$ parameters per weight matrix, which requires storing gradient information and optimizer states (typically 2–3× the model size in memory) for every parameter. LoRA reduces memory consumption by a factor of roughly $(d \times k) / (r(d + k))$, which for typical transformer dimensions can be 100–1000×. The low-rank assumption captures the empirical observation that weight updates during fine-tuning tend to have low intrinsic dimension—the model doesn't need to change all its parameters to adapt to a new task; it only needs to adjust along a small number of directions in weight space. The factorized form also provides a natural regularization effect, preventing overfitting on small datasets by constraining the expressivity of the update.

LoRA Configuration in This Paper

The paper applies LoRA to the LLM's weight matrices (the specific matrices are not enumerated, but standard practice applies LoRA to the query and value projection matrices in self-attention layers). The linear projection layer between vision encoder and LLM is always trained fully, even in LoRA runs, since it is already small (roughly 7M parameters for the 33B model).

Key hyperparameters and their effects:

• Rank $r$: Tested at $r = 8$ and $r = 64$. Higher rank means more trainable parameters and more expressive adaptation. For the 33B model, increasing rank from 8 to 64 improves LLaVA-Bench from 70.3 to 71.8 (Table 4), approaching the full fine-tuning score of 72.0. For the 13B model, rank 64 already matches full fine-tuning (70.1 for both).

• LoRA alpha: A scaling factor $\alpha$ that multiplies the LoRA update before addition: $W = W_0 + \frac{\alpha}{r} BA$. The paper sets $\alpha = 2r$ (so the effective scaling factor is 2), and finds this is significantly better than smaller values. When $\alpha = 16$ with learning rate $2 \times 10^{-5}$, performance drops from 71.8 to 65.5—a dramatic 6.3-point decline that demonstrates the sensitivity of LoRA training to these hyperparameters. The authors note that "larger learning rate and alpha value of LoRA improves the results significantly" and that under the same setting with reduced alpha and learning rate, even increasing rank to 128 or 512 yields only modest improvements (65.5 → 66.1 → 68.1).

• Learning rate: $1 \times 10^{-4}$ for LoRA runs, which is 5× higher than the $2 \times 10^{-5}$ used for full fine-tuning. The higher learning rate compensates for the smaller number of trainable parameters and the implicit constraint of the low-rank update.

The cost-performance tradeoff (Table 4). For the 33B model, LoRA rank 64 requires 5.80 GPU-hours per node per epoch versus 9.17 for full fine-tuning—a 37% reduction in training time. For the 65B model, LoRA rank 64 requires 9.17 GPU-hours versus 13.50 for full fine-tuning—a 32% reduction. The number of trainable parameters drops from 33B (full) to 0.49B (LoRA rank 64) for the 33B model, and from 65B to 0.81B for the 65B model.

The rank-performance ceiling. Increasing rank beyond the range where it saturates to full fine-tuning performance yields diminishing returns. The paper reports that for the 13B model, rank 64 matches full fine-tuning, so higher ranks provide no benefit. For the 33B model, rank 64 (71.8) nearly matches full fine-tuning (72.0), and the gap is small enough that further rank increases would likely provide minimal practical gain. The paper explicitly notes that merely increasing rank without properly tuning learning rate and alpha yields poor results—the interaction between rank, alpha, and learning rate is complex, and naively scaling rank alone does not recover full performance.

QLoRA Mechanics

QLoRA extends LoRA by quantizing the frozen base model weights to 4-bit precision (from the standard 16-bit brain floating point), further reducing memory consumption. The trainable LoRA adapter matrices remain in 16-bit precision, and during the forward pass, the quantized weights are dequantized on-the-fly to compute the base model's contribution, while the LoRA adapters add their correction in full precision.

The paper uses QLoRA only for the 13B model at rank 64 (Table 4), reporting 4.68 GPU-hours per node per epoch versus 4.79 for standard LoRA—a modest 2% reduction in compute time, but with significantly lower GPU memory requirements. The paper notes that "QLoRA requires lower GPU memory cost and running-time cost than LoRA" and that "when large models (e.g., 65B) are trained with DeepSpeed ZeRO2 mode, they can fit into GPU with QLoRA, while yield the OOM issue with LoRA." This is the critical practical advantage: QLoRA enables training on hardware that cannot accommodate full-precision LoRA, making it the only viable parameter-efficient option for the largest models under constrained GPU memory budgets.

When Full Fine-Tuning Is Still Preferred

The paper implicitly establishes a decision boundary: for models up to 13B, LoRA with sufficient rank and properly tuned hyperparameters matches full fine-tuning, making full fine-tuning unnecessary. For the 33B model, LoRA rank 64 nearly matches full fine-tuning (71.8 vs. 72.0), and the decision depends on whether that 0.2-point gap matters for the application and whether the 37% training cost savings are worth it. For the 65B model, full fine-tuning still holds a measurable advantage (72.3 vs. likely sub-72.0 for LoRA, though the paper only reports LoRA-64 performance for 33B in Table 4), suggesting that at the largest scales studied, the gap between parameter-efficient and full fine-tuning may persist—possibly because the low-rank constraint becomes too restrictive for the adaptation needed by very large models.

---

Evaluation Framework

The paper evaluates models on five benchmarks spanning multimodal and language-only capabilities.

Multimodal Benchmarks

LLaVA-Bench (In-the-Wild). A set of 24 images with 60 manually curated questions covering indoor and outdoor scenes, memes, paintings, and sketches. Each question is categorized into one of three task types:

• Conversation: Simple visual recognition and QA questions, testing the model's ability to identify objects, describe scenes, and answer straightforward queries about image content.
• Detailed description: Characterizing the image with a long paragraph, testing the model's ability to generate fluent, comprehensive visual descriptions.
• Complex reasoning: Deducing implications from an image, testing deeper understanding that goes beyond surface-level recognition.

Evaluation uses GPT-4 (gpt4-0314) as a judge: the model's generated response and a reference "gold" response are presented to GPT-4, which scores the relative quality of the model's output. Scores are reported for each task type and as an overall average. Beam search with sizes 1 (greedy) and 5 are both evaluated—beam search 5 consistently outperforms greedy by 3–4 points (e.g., LLaVA-13B improves from 70.1 to 73.5; LLaVA-33B from 73.9 to 74.8).

MM-VET. A benchmark consisting of 200 images and 218 question-answer pairs designed to evaluate six core vision-language capabilities and their combinations:

• Recognition (Rec): Identifying objects, scenes, and entities in images
• OCR: Reading and understanding text within images
• Knowledge: Applying world knowledge to visual inputs
• Generation (Gen): Producing fluent, contextually appropriate text responses
• Spatial awareness (Spatial): Understanding spatial relationships and layouts
• Math: Performing mathematical reasoning on visual inputs

Evaluation again uses GPT-4 (gpt4-0613) as a judge, scoring open-ended model outputs against reference answers. The paper reports per-capability scores and a total aggregate. Importantly, the benchmark distinguishes between models that possess individual capabilities and those that can integrate multiple capabilities to solve complex tasks requiring, for example, simultaneous recognition, OCR, and spatial reasoning.

MM-Bench. A set of 2,974 multiple-choice questions evaluating models' reasoning skills across six categories:

• Logic Reasoning (LR): Deductive and inductive logical inference
• Attribute Reasoning (AR): Reasoning about object properties and attributes
• Relation Reasoning (RR): Understanding relationships between entities
• Fine-grained Single-instance Perception (FP-S): Detailed perception of individual objects
• Fine-grained Cross-instance Perception (FP-C): Detailed perception of relationships between multiple objects
• Coarse Perception (CP): Broad scene understanding

Language-Only Benchmarks

Vicuna-80. A set of 80 diverse text-only questions used in the Vicuna project to evaluate instruction-following ability in real-world language tasks. The evaluation is performed by GPT-4, which compares the model's responses to reference answers and assigns scores. This benchmark tests whether visual instruction tuning degrades the model's original language capabilities—a concern because Stage 2 training on multimodal data might cause catastrophic forgetting of purely linguistic skills.

MMLU (Massive Multitask Language Understanding). A comprehensive benchmark spanning 57 subjects across STEM, humanities, social sciences, and other domains, evaluating the model's factual knowledge and reasoning ability across diverse topics. Each question is multiple-choice with four options. The paper reports the overall MMLU accuracy score. This benchmark is particularly important because it tests whether adding a visual modality to the LLM affects its capacity to store and retrieve factual knowledge—a core capability that would be concerning to lose.

Beam Search at Inference Time

The paper reports results with beam search sizes of 1 (equivalent to greedy decoding) and 5. Beam search is a test-time technique that maintains multiple candidate sequences and selects the one with the highest cumulative probability at the end, rather than selecting the single most probable token at each step (greedy). The consistent improvement from beam search 1 to beam search 5—visible across all model sizes in Table 1 and yielding approximately 3–4 additional points on LLaVA-Bench—demonstrates that even with fixed model weights, additional inference-time computation improves output quality. The paper notes that this improvement comes with "negligible increase of inference latency," though it does not quantify the latency increase explicitly.

Cost Accounting

The paper's cost reporting uses an unconventional but practically useful unit: "GPU Hours per node." This is computed as (\text{running time} \times \text{#GPUs}) / 8, where 8 is the number of GPUs per node. The result is the equivalent number of hours that would be consumed if the training ran on a single node of 8 GPUs, even if the actual training uses multiple nodes in parallel. The paper provides a concrete pricing reference: "$13.63/hour (ND A100 v4 series) on Azure," enabling practitioners to estimate total training cost by multiplying the GPU-hours per node by the number of epochs and the hourly rate. For example, full fine-tuning the 65B model for 1 epoch costs approximately `$13.50 \times 13.63 \approx $184$` in compute, while LoRA rank 64 on the 33B model costs approximately `$5.80 \times 13.63 \approx $79$`.

---

Summary of Key Design Choices and Their Justifications

• Frozen vision encoder: Preserves representations learned during large-scale CLIP pretraining (400M image-text pairs) and eliminates the need to backpropagate through a large vision model during LLaVA training, substantially reducing compute requirements. The tradeoff is that the vision features cannot adapt to the specific requirements of the LLM or the downstream tasks—the system relies entirely on the projection layer to bridge the modality gap.

• Two-stage training separated by which components are frozen: Stage 1 trains only the projection layer because training the LLM at this stage would cause it to overfit to the simplistic caption-generation task before seeing the more complex instruction-following data. Isolating the projection training ensures that the LLM's instruction-following capabilities are preserved and only "activated" for multimodal inputs in Stage 2.

• Data mixing as a performance lever rather than a forgetting prevention tool: The original motivation for mixing language-only data with multimodal data was presumably to prevent catastrophic forgetting of language capabilities. The empirical finding that it predominantly improves multimodal performance rather than language performance suggests a different mechanism: the language-only data may provide a form of regularization that prevents the model from overfitting to the specific distribution of multimodal instructions, or it may provide auxiliary training signals (e.g., complex reasoning patterns in text-only conversations) that transfer to multimodal settings.

• LoRA alpha = 2× rank with elevated learning rate: The large alpha and learning rate combination effectively amplifies the magnitude of the low-rank update, compensating for the restricted expressivity of the low-rank decomposition. Without this amplification, the LoRA update is too constrained to match full fine-tuning performance, even with high rank. The sensitivity to these hyperparameters—a 6.3-point drop from suboptimal settings—highlights that LoRA's practical effectiveness depends critically on hyperparameter tuning, not just on choosing an appropriate rank.

• Beam search as a near-zero-cost performance boost: The consistent 3–4 point improvement from beam search across all model sizes and task types suggests that the models produce multiple plausible outputs and that the highest-probability sequence (greedy) is often not the best one. This is a property of the model's uncertainty calibration—beam search effectively performs a limited form of search over the output space at minimal additional cost, leveraging the model's own probability estimates to select better completions.

4. Key Insights and Innovations

Innovation 1: Visual Instruction Tuning as a Language Capability Enhancer, Not a Zero-Sum Tradeoff

The dominant assumption in multimodal model development—inherited from the broader transfer learning and multi-task learning literature—is that adding a new modality or task to a pretrained model creates a capability tradeoff: gains in the new domain come at the cost of degraded performance in the original domain. This is the classic catastrophic forgetting concern. When LLaVA's two-stage training was introduced, it was designed to preserve the LLM's language abilities while adding vision—the implicit framing was defensive, treating language capability as something to be protected from degradation during multimodal training.

This paper overturns that framing with a genuinely surprising empirical result in Table 5: LLaVA-70B, built on LLaMA-2-70B-Chat and trained with data mixing, achieves 65.1 on MMLU—a 2.4-point improvement over the base Vicuna-65B's 62.5 (and 2.0 points over LLaMA-2-70B-Chat's 63.1, which appears as a separate baseline in Table 5). The authors explicitly flag this as unprecedented:

"To the best of our knowledge, this is the first reported result which shows visual instruction tuning improve language ability of large-scale LMM."

This is not an incremental refinement of an existing finding—it is a fundamental reframing of what multimodal training can accomplish. The mechanism is not fully explained in the paper, but the authors hypothesize that "the inclusion of complex reasoning questions, and long-form answers in LLaVA-Instruct-158K, which helps maintain the language capabilities of LLaVA" (referring to the expanded 158K dataset used for the 70B model). The implication is that high-quality multimodal instruction data doesn't just add visual skills—it provides auxiliary training signal that strengthens the underlying language model, possibly through exposure to more diverse reasoning patterns, longer-form structured outputs, or cross-modal grounding that reinforces abstract concepts.

The significance extends beyond this single finding. It challenges the conceptual model that treats vision and language as competing for a fixed-capacity representation budget. Instead, it suggests that certain forms of multimodal training may induce positive transfer, where learning to ground language in visual contexts improves the model's pure language capabilities. This is analogous to how bilingual humans sometimes show enhanced metalinguistic awareness—the second modality doesn't subtract from the first; it enriches it.

Critically, the paper also shows that this effect is not universal or automatic. The 13B and 33B models show slight drops or flat performance on Vicuna-80 and MMLU after visual instruction tuning without data mixing (Vicuna-13B: 79.9 → LLaVA-13B: 79.6 on Vicuna-80; Vicuna-33B: 59.0 → LLaVA-33B: 56.1 on MMLU). The 65B model without data mixing actually gains on MMLU (62.5 → 62.6), but the gain is marginal. The substantial improvement emerges only at the 70B scale with data mixing. This suggests that the positive transfer effect is scale-dependent—larger models may have sufficient capacity to absorb multimodal training as a beneficial regularizer rather than a competing objective, while smaller models experience the expected interference. This scale-dependent crossover from negative to positive transfer is a novel empirical phenomenon that was invisible at the 7B–13B scales where prior work operated.

The data mixing result is particularly informative for interpreting this effect. Table 5 shows that data mixing improves multimodal benchmarks (LLaVA-Bench: 72.0 → 73.9 for 33B; 72.3 → 74.2 for 65B) while having inconsistent effects on language benchmarks—sometimes improving them (33B MMLU goes from 56.1 to 58.6) and sometimes slightly degrading them (65B Vicuna-80 goes from 84.5 to 82.6). This decoupling—where data mixing helps multimodal performance without consistently helping or hurting language performance—suggests that the language enhancement from visual instruction tuning and the language preservation from data mixing operate through different mechanisms. One enhances language through cross-modal reasoning transfer; the other prevents forgetting through continued exposure to the original data distribution.

Prior work (the original LLaVA paper, MiniGPT-4, InstructBLIP) either did not report language-only benchmark performance or reported it only to show that capabilities were roughly preserved. None had observed or claimed that visual instruction tuning could improve pure language metrics. This finding therefore establishes a new research direction: understanding when and why multimodal training produces positive transfer to unimodal capabilities, and whether this can be deliberately engineered through curriculum design, data selection, or architectural choices.

Innovation 2: The Cost-Performance Frontier as an Explicit, Quantifiable Interface Between Training Method and Model Scale

Most deep learning papers report performance improvements and mention training cost as a practical note. This paper makes a fundamentally different move: it treats the cost-performance relationship as a first-class scientific object of study, not an implementation detail. Table 4 is the centerpiece of this contribution, presenting a matrix that simultaneously reports LLaVA-Bench performance, GPU-hours per node per epoch, and the number of trainable parameters across four model sizes (7B, 13B, 33B, 65B) and multiple training methods (full fine-tuning, LoRA rank 8, LoRA rank 64, QLoRA rank 64).

This is a reframing of the problem from "which method works best?" to "what is the performance-per-dollar at each (method, scale) combination?" The distinction matters because practitioners don't choose methods in the abstract—they choose methods under a budget constraint. A method that achieves 72.0 on LLaVA-Bench but costs 9.17 GPU-hours is not simply "better" than a method that achieves 71.8 but costs 5.80 GPU-hours. The paper doesn't declare a winner; it provides the data that lets readers make that determination based on their own cost tolerance.

The specific findings that emerge from this framing are practically significant:

• LoRA's effectiveness is not monotonic with scale. At 13B, LoRA rank 64 matches full fine-tuning exactly (70.1 vs. 70.1). At 33B, it nearly matches (71.8 vs. 72.0). At 65B, the paper doesn't provide a direct LoRA-64 comparison, but the trend suggests the gap may persist or widen. This is not a trivial saturation curve—it suggests that the low-rank assumption becomes less adequate as model capacity grows, and that the maximum effective rank for a given performance target scales with model size in a way that is not yet characterized.

• The rank-performance curve has a hyperparameter-dependent ceiling. The paper's finding that reducing alpha and learning rate causes a 6.3-point drop (71.8 → 65.5) even at the same LoRA rank 64 reveals that LoRA's practical ceiling is set by hyperparameter tuning quality, not by the fundamental expressivity of low-rank adaptation. This is a diagnostic insight: the common complaint that "LoRA doesn't match full fine-tuning" may often reflect inadequate hyperparameter optimization rather than an intrinsic limitation of the method. The paper's explicit comparison of alpha/learning rate combinations provides concrete guidance for practitioners who might otherwise conclude that LoRA is inherently inferior.

• QLoRA's memory advantage enables a training regime otherwise impossible. The note that 65B training with LoRA encounters out-of-memory errors under DeepSpeed ZeRO2 while QLoRA succeeds is not presented as the main finding but has major practical implications. It establishes that for the largest models on constrained hardware, QLoRA is not just cheaper—it is the only viable option among parameter-efficient methods. This defines an absolute boundary in the cost-performance space where QLoRA is the singleton element.

• Cost scales more with total parameters than with trainable parameters. The GPU-hour cost for LoRA rank 64 on the 65B model (9.17) is nearly identical to full fine-tuning of the 33B model (9.17), even though LoRA-64 trains only 0.81B parameters while full-33B trains 33B parameters. This reveals that the dominant cost is the forward pass and gradient computation through the frozen base model—the trainable parameter count is a poor proxy for training cost in LoRA settings. This non-obvious finding matters because practitioners often assume that reducing trainable parameters proportionally reduces cost, when in fact the frozen forward pass dominates.

Prior work on parameter-efficient fine-tuning (Hu et al., 2021 for LoRA; Dettmers et al., 2023 for QLoRA) established that these methods work for language-only tasks at modest scales. This paper extends the analysis to the multimodal domain at scales up to 65B, but more importantly, it provides the cost-performance matrix that enables economic decision-making. This is an incremental contribution in terms of novelty (LoRA and QLoRA were already known), but a significant practical contribution because it converts abstract claims about "efficiency" into concrete numbers that can be multiplied by cloud pricing to estimate total cost.

Innovation 3: Disaggregating the Multimodal Scaling Signal Across Capability Dimensions

The field's default approach to evaluating multimodal models is to report aggregate benchmark scores—a single number that collapses complex performance profiles into a scalar metric. This paper, through its systematic reporting of MM-VET's six sub-scores across model scales (Table 2), performs a disaggregated scaling analysis that reveals which capabilities scale with model size and which do not.

The pattern is striking and non-obvious. Comparing LLaVA-13B (LLaMA-2) to LLaVA-65B (Data Mixing):

• Knowledge jumps dramatically: 26.3 → 30.4 (a 15.6% relative improvement)
• Generation improves substantially: 28.8 → 32.3 (12.2% relative improvement)
• Recognition and OCR improve modestly: 38.4 → 41.8 (8.9%) and 21.0 → 27.9 (32.9%)
• Spatial awareness shows mixed results: 28.0 → 30.5 (8.9%)
• Math remains essentially flat: 7.7 → 7.3 (a slight decline)

This differential scaling pattern is the key insight. Scaling the language model disproportionately improves capabilities that depend on stored knowledge and fluent generation, while leaving perceptual capabilities relatively unchanged. The math result is particularly informative: mathematical reasoning on visual inputs apparently requires capabilities that are neither knowledge retrieval nor language generation, and simply making the LLM larger does not address the bottleneck. This result is robust across model sizes (the math score hovers between 7.3 and 7.7 from 7B to 65B) and across data mixing variations.

This is a diagnostic contribution rather than a performance contribution. The value is not in the absolute numbers but in the decomposition of scaling effects that reveals where investment matters and where it does not. If an organization is deciding whether to scale from 33B to 65B for their multimodal application, the decision depends entirely on which capabilities matter for their use case. If they need knowledge-intensive visual QA, the scaling investment pays off. If they need precise spatial reasoning or math from visual inputs, it does not—and they should invest instead in better vision encoders, higher-resolution inputs, or specialized reasoning modules.

This finding also implicitly challenges the "scaling hypothesis"—the idea that simply making models larger will eventually solve all capability deficits. The math score's flatness across a 9× increase in model parameters (7B → 65B) is a clear counterexample: some capabilities appear to be gated by factors other than LLM capacity. The paper doesn't fully explain this gating, but it points toward the role of the vision encoder (which remains frozen and unchanged across all experiments) as a potential bottleneck. The implication, which the paper explicitly flags in Section 4, is that "future work [should investigate] how to scale the vision encoder to enhance the visual capabilities and improve model performance on vision recognition and understanding tasks."

Prior work on scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) established that language model performance improves smoothly with scale, but those analyses operated on aggregate metrics like perplexity or downstream task averages. The multimodal scaling literature had no comparable analysis because no one had systematically varied model size across a sufficiently wide range while holding architecture constant. This paper, by spanning 7B to 65B with the same LLaVA architecture, provides the first evidence that multimodal scaling is not uniform across capability dimensions—a finding that complicates the simple "bigger is better" narrative and provides concrete guidance for resource allocation in multimodal model development.

Innovation 4: Data Mixing as a Multimodal Performance Lever, Not a Language Forgetting Countermeasure

The standard motivation for mixing language-only data with task-specific data during fine-tuning is catastrophic forgetting prevention: by continuing to expose the model to its original training distribution, you prevent the model from overwriting useful representations with task-specific ones. This is the defensive framing that dominates the instruction tuning and continual learning literature.

This paper's results in Tables 3, 5, and indirectly in the MM-Bench results (Table 3b) suggest a fundamentally different interpretation. Data mixing predominantly improves multimodal performance, not language performance—exactly the opposite of what the forgetting-prevention hypothesis would predict.

The evidence is clear across multiple model scales and benchmarks:

• LLaVA-Bench (Table 3a): Data mixing improves performance from 72.0 → 73.9 for the 33B model and 72.3 → 74.2 for the 65B model. These are substantial gains (1.9 and 1.9 points respectively) on what is purely a multimodal benchmark.

• MM-VET (Table 2): Data mixing improves LLaVA-33B from 32.9 → 34.1 and LLaVA-65B from 35.5 → 36.4. The gains are concentrated in OCR (25.0 → 27.1 for 33B; 28.2 → 27.9 for 65B), knowledge (26.2 → 26.2 [flat for 33B]; 26.2 → 30.4 for 65B), and generation (28.2 → 28.6; 28.3 → 32.3).

• Language benchmarks (Table 5): The effect is inconsistent and often negative. For the 33B model, data mixing improves MMLU (56.1 → 58.6) but degrades Vicuna-80 (85.3 → 80.3). For the 65B model, data mixing degrades Vicuna-80 (84.5 → 82.6) and has a slightly negative effect on MMLU (62.6 → 62.2).

This pattern—multimodal improvement without consistent language improvement—is incompatible with the simple forgetting-prevention story. If mixing prevented forgetting, we would expect language benchmarks to improve or at least remain stable, with multimodal benchmarks potentially unchanged. Instead, we see multimodal benchmarks improving and language benchmarks showing mixed effects. This suggests a different mechanism: the language-only data provides auxiliary training signal that transfers to multimodal tasks, possibly because it contains complex reasoning patterns, multi-turn conversation structures, or diverse instruction formats that help the model learn to follow instructions in general, with multimodal tasks benefiting from this general instruction-following improvement.

The MM-Bench results in Table 3b provide further evidence. The LLaVA-65B with data mixing and 336×336 resolution achieves 68.1 overall—a dramatic improvement over the LLaVA-7B baseline's 36.2. While some of this gain comes from model scaling and resolution, the comparison is suggestive that data mixing contributes meaningfully. The per-skill breakdown shows the largest absolute gains in relation reasoning (28.6 → 72.3) and coarse perception (40.4 → 68.1)—both of which plausibly benefit from the kind of relational and descriptive language found in text-only conversation data.

This is a conceptual reframing rather than a new method. Data mixing was already a known technique; what is new is the evidence that its primary benefit in the multimodal setting is not defensive (preventing loss) but constructive (enhancing multimodal capabilities through cross-task transfer). This reframing matters because it changes how practitioners should think about data mixing: rather than asking "how much language data do I need to prevent forgetting?" they should ask "what kind of language data provides the most transferable signal for my multimodal tasks?" The answer may involve curating language data specifically for its similarity to multimodal reasoning patterns, not just for its coverage of the original language distribution.

The paper doesn't fully develop this reframing—it presents the data mixing results as one factor among several—but the empirical pattern is clear and robust enough to constitute a distinct insight. Prior work on instruction tuning (Wang et al., 2023; the Vicuna and LLaMA-2-Chat training recipes) used data mixing extensively but analyzed its effects primarily through the lens of language benchmark preservation. This paper's multimodal context reveals a different function for the mixed data that challenges the standard interpretation and opens new questions about optimal data composition strategies.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary multimodal benchmarks are LLaVA-Bench (24 images, 60 questions covering conversation, detailed description, and complex reasoning tasks), MM-VET (200 images, 218 questions evaluating recognition, OCR, knowledge, generation, spatial awareness, and math), and MM-Bench (2,974 multiple-choice questions across six reasoning categories). Language-only evaluation uses Vicuna-80 (80 diverse text questions for instruction-following) and MMLU (57 subjects across STEM, humanities, and social sciences). All benchmarks are described in Section 3, with LLaVA-Bench and MM-VET detailed in Sections 3.1, MM-Bench introduced in Table 3b, and language benchmarks in Section 3.2.

• Base model(s). The paper builds on Vicuna models at 7B, 13B, 33B, and 65B scales (instruction-tuned from LLaMA-1), plus one variant using LLaMA-2-70B-Chat. The 7B and 13B Vicuna checkpoints are public; the 33B uses the public Vicuna-33B-v1.3 checkpoint; the 65B was trained by the authors since no public 65B Vicuna existed. The vision encoder is CLIP ViT-L/14, frozen during all LLaVA training. The choice of Vicuna is inherited from the original LLaVA paper and reflects the dominant open-source instruction-tuned LLM family at the time.

• Metrics. LLaVA-Bench uses GPT-4 (gpt4-0314) to score model outputs against reference answers, reporting relative scores per task type (conversation, detail, reasoning) and an overall average. MM-VET uses GPT-4 (gpt4-0613) to score open-ended outputs, reporting per-capability scores and a total aggregate. MM-Bench reports accuracy on multiple-choice questions across six reasoning categories. Vicuna-80 uses GPT-4 evaluation against reference answers. MMLU reports overall accuracy across 57 subjects. GPU-hours per node per epoch (computed as (running time × #GPUs) / 8) is used for cost accounting, with Azure pricing provided ($13.63/hour for ND A100 v4).

• Baselines. Commercial systems include Bard-0718, Bing-Chat-0629, and MM-ReAct-GPT-4. Open-source multimodal models include LLaMA-Adapter v2-7B, OpenFlamingo-9B, MiniGPT-4 (8B and 14B variants), BLIP-2-12B, InstructBLIP (8B and 14B variants), Otter-9B, and earlier LLaVA versions (7B, 13B at 224×224 and 336×336 resolutions). Language-only baselines are the base Vicuna and LLaMA-2-Chat models before visual instruction tuning.

• Generation budget / compute accounting. For benchmark comparisons, the primary axis is model scale (7B → 13B → 33B → 65B/70B) with fixed training recipes. Beam search sizes 1 and 5 are compared for LLaVA-Bench (Table 1). For training cost comparisons (Table 4), the metric is GPU-hours per node per epoch, with per-node pricing provided to enable total cost estimation. All training budgets are 1 epoch on the LLaVA-80K dataset (or LLaVA-80K + ShareGPT for data mixing runs).

• Cross-validation / statistical protocol. The paper does not employ cross-validation or statistical significance testing. LLaVA-Bench results are reported as "averaging 3 repeated evaluation runs with same set up" (Table 4 note) to account for sampling variance in GPT-4 evaluation, but no confidence intervals or standard deviations are reported for the main benchmark tables. MM-VET reports standard deviations (e.g., LLaVA-33B: 32.9±0.3) from evaluator variance, but these are not used for statistical comparisons between models. The paper is primarily a phenomenological scaling study rather than a hypothesis-testing experiment, so the absence of formal statistical protocols is consistent with its empirical survey character, though it limits the strength of comparative claims, particularly on small benchmarks like LLaVA-Bench (24 images).

Main Quantitative Results

Scaling Model Size: Consistent Gains with Diminishing Character

Headline numbers. Table 1 shows LLaVA-13B achieving 70.1 overall on LLaVA-Bench (beam=1), rising to 73.9 for LLaVA-33B and 74.2 for LLaVA-65B. With beam search 5, the progression is 73.5 (13B) → 74.8 (33B) → 74.4 (65B). On MM-VET (Table 2), LLaVA-13B scores 32.5 total, LLaVA-33B scores 32.9, and LLaVA-65B reaches 35.5—a 3.0-point improvement from 13B to 65B.

The scaling curve is concave. The jump from 13B to 33B (+3.8 points on LLaVA-Bench with data mixing) is substantially larger than from 33B to 65B (+0.3 points without data mixing, though the 65B model uses different ShareGPT data). This pattern of diminishing returns is most visible in the beam=1 comparisons: 70.1 (13B) → 73.9 (33B) → 74.2 (65B), where the marginal gain from 33B to 65B is only 0.3 points. Beam search 5 partly masks this by showing similar top-line numbers (74.8 for 33B vs. 74.4 for 65B), but the sub-scores reveal that 65B's beam=5 performance drops on conversation (59.4 vs. 72.6 for 33B) while improving on reasoning (88.7 vs. 83.5). This suggests that beam search interacts differently with model scale—larger models may have different calibration properties that affect beam search's benefit.

Capability-specific scaling (MM-VET, Table 2). Comparing LLaVA-13B to LLaVA-65B (Data Mixing): recognition improves from 38.4 to 41.8 (+3.4), OCR from 21.0 to 27.9 (+6.9), knowledge from 26.3 to 30.4 (+4.1), generation from 28.8 to 32.3 (+3.5), spatial from 28.0 to 30.5 (+2.5), and math from 7.7 to 7.3 (-0.4). Math is the only capability that does not improve—and actually slightly declines—from 13B to 65B, indicating that LLM capacity is not the bottleneck for visual mathematical reasoning. The gains in OCR (+6.9) and knowledge (+4.1) are the largest absolute improvements, consistent with the hypothesis that scaling the LLM primarily enhances capabilities that depend on stored knowledge and text processing rather than visual precision.

Comparison to commercial systems. LLaVA-65B (beam=5) scores 74.4 overall on LLaVA-Bench, placing it between Bing Chat's 71.5 and Bard's 77.8 (Table 1). This is remarkable for an open-source model and validates the claim that visual instruction tuning at scale produces competitive multimodal performance. However, the small size of LLaVA-Bench (24 images) means these comparisons should be treated as indicative rather than definitive—the paper acknowledges this in Section 3.1: "LLaVA-Bench is small (thus the comparison might not be statistically significant)."

Image Resolution: Consistent 2–3 Point Improvement

Headline numbers. Table 3a shows the effect of increasing image resolution from 224×224 to 336×336 across all four model sizes on LLaVA-Bench: 7B improves from 63.6 to 65.9 (+2.3), 13B from 67.1 to 70.1 (+3.0), 33B from 69.3 to 72.0 (+2.7), and 65B from 70.3 to 72.3 (+2.0). The improvement is remarkably consistent across a 9× range in model parameters, suggesting that resolution benefits are largely independent of LLM capacity—they arise from providing finer-grained visual information that any sufficiently capable language model can exploit.

Stability across scales. The fact that the resolution gain neither increases nor decreases systematically with model size is informative. If larger LLMs could extract proportionally more value from higher-resolution inputs (because they have more capacity to process the additional visual tokens), we would expect the gap to widen with scale. If larger LLMs already saturated the visual information at 224×224, we would expect the gap to narrow. The observed stability (~2.5 points) suggests that resolution benefits operate through a mechanism that is approximately additive with respect to LLM scale—likely because higher resolution simply provides visual details that are absent at lower resolution, and any LLM can benefit from having that information available.

MM-Bench results (Table 3b). The combination of 336×336 resolution with data mixing and 65B scale produces dramatic gains over the 224×224, no-data-mixing 7B baseline: overall score rises from 36.2 to 68.1. The per-category breakdown shows the largest absolute gains in relation reasoning (28.6 → 72.3) and coarse perception (40.4 → 67.2), though these reflect the combined effect of all three scaling factors, not resolution alone.

Data Mixing: Multimodal Gains Without Consistent Language Preservation

Headline numbers. Table 3a shows data mixing improves LLaVA-Bench scores: 33B goes from 72.0 to 73.9 (+1.9) and 65B from 72.3 to 74.2 (+1.9). On MM-VET (Table 2), data mixing improves 33B from 32.9 to 34.1 (+1.2) and 65B from 35.5 to 36.4 (+0.9). On MM-Bench (Table 3b), the 65B with data mixing and 336×336 resolution achieves 68.1 overall, though the isolated contribution of data mixing cannot be extracted from this single number.

Language benchmark effects are inconsistent (Table 5). For the 33B model, data mixing improves MMLU from 56.1 to 58.6 (+2.5) but degrades Vicuna-80 from 85.3 to 80.3 (-5.0). For the 65B model, data mixing degrades Vicuna-80 from 84.5 to 82.6 (-1.9) and has minimal effect on MMLU (62.6 → 62.2). This asymmetry—multimodal benchmarks consistently improve while language benchmarks show mixed effects—is the key pattern that motivates the reinterpretation of data mixing as a multimodal performance lever rather than a forgetting countermeasure.

The LLaMA-2-70B-Chat variant. LLaVA-70B (data mixing) achieves 69.8 on LLaVA-Bench and 35.4 on MM-VET (Table 5), which is moderately lower than the Vicuna-65B variant (74.2 and 36.4). However, on language benchmarks, LLaVA-70B dramatically outperforms: MMLU rises from 62.5 (Vicuna-65B) to 65.1 (LLaVA-70B)—a 2.6-point gain that the paper flags as the first demonstration of visual instruction tuning improving pure language capability. This result is not strictly comparable to the Vicuna-65B because the base model (LLaMA-2-70B-Chat vs. Vicuna-65B) and the instruction tuning data differ, but it establishes existence: under the right conditions, multimodal training can enhance unimodal language performance.

Parameter-Efficient Training: LoRA/QLoRA Approaches Full Fine-Tuning with Reduced Cost

Headline numbers. Table 4 organizes the cost-performance tradeoff across model sizes and training methods on LLaVA-Bench (all models trained on LLaVA-80K only, no data mixing, 336×336 resolution):

* The 65B LoRA-64 score of 72.2 is implied by the table structure (the 72.2 appears in the Performance row between 71.8 and 72.3, associated with the 65B column), though the exact mapping requires careful reading of Table 4's column alignment.

LoRA rank scaling. For the 33B model, increasing LoRA rank from 8 to 64 improves performance by 1.5 points (70.3 → 71.8) while adding only 1.18 GPU-hours (4.62 → 5.80). The gap to full fine-tuning narrows from 1.7 points at rank 8 to 0.2 points at rank 64. For the 13B model, LoRA rank 64 already matches full fine-tuning exactly (70.1 for both), suggesting that the effective rank needed to saturate performance scales with model size—13B models are adequately served by rank 64, while 33B+ models may benefit from higher ranks, though the paper does not test ranks beyond 64 for the 33B with optimal hyperparameters.

QLoRA vs. LoRA. QLoRA rank 64 on the 33B model achieves 71.6 versus 71.8 for standard LoRA—a 0.2-point gap at 0.94 GPU-hours less cost (4.68 vs. 5.80). The primary advantage of QLoRA is memory reduction: "when large models (e.g., 65B) are trained with DeepSpeed ZeRO2 mode, they can fit into GPU with QLoRA, while yield the OOM issue with LoRA" (Section 3.2). This makes QLoRA the only viable parameter-efficient option for the largest models under constrained GPU memory, not because it is more performant, but because it is feasible where LoRA is not.

Hyperparameter sensitivity. The paper reports a striking ablation on LoRA hyperparameters (Section 3.2): with the same rank 64 on the 33B model, reducing the learning rate to 2×10⁻⁵ and alpha to 16 causes performance to drop from 71.8 to 65.5—a 6.3-point decline. Further increasing rank under these suboptimal hyperparameters yields only modest recovery: rank 128 achieves 66.1, rank 512 achieves 68.1. This demonstrates that LoRA's effectiveness is dominated by learning rate and alpha, not by rank per se, and that inadequate hyperparameter tuning can leave substantial performance on the table even with high-rank configurations.

Beam Search: Consistent Near-Zero-Cost Improvement

Headline numbers. Table 1 shows beam search 5 consistently outperforms beam search 1 across all model sizes on LLaVA-Bench: LLaVA-13B improves from 70.1 to 73.5 (+3.4), LLaVA-33B from 73.9 to 74.8 (+0.9), LLaVA-65B from 74.2 to 74.4 (+0.2). The gain from beam search shrinks as model size increases—the largest models show near-identical performance between greedy and beam search—which may indicate that larger models are better calibrated (their greedy output is already near-optimal) or that the benchmark's discriminative power saturates at the high end.

Task-type interaction. The beam search benefit varies by task type. For LLaVA-65B, beam search improves reasoning from 87.3 to 88.7 (+1.4) and detail from 62.3 to 65.7 (+3.4), but degrades conversation from 63.8 to 59.4 (-4.4). This negative effect on conversation tasks is not observed for the 33B model (70.2 → 72.6), suggesting a complex interaction between model scale, task type, and beam search that the paper does not investigate further.

Ablation Studies and Robustness Checks

• LoRA learning rate and alpha: When learning rate is reduced to 2×10⁻⁵ and alpha to 16 (keeping rank=64 on 33B), LLaVA-Bench drops from 71.8 to 65.5—a 6.3-point decline. This demonstrates that hyperparameter quality, not rank, is the primary determinant of LoRA performance. The paper explicitly notes: "Under the same setting, large ranks leads to little improvement. e.g., we increase the rank from 64 to 128 and 512, it improves from 65.5 to 66.1 and 68.1, respectively."

• LoRA rank scaling under suboptimal vs. optimal hyperparameters: Under the suboptimal hyperparameter regime (alpha=16, lr=2×10⁻⁵), increasing rank from 64 to 128 to 512 yields 65.5 → 66.1 → 68.1—a total improvement of 2.6 points that still falls 3.7 points short of the optimal-hyperparameter rank-64 score of 71.8. This reveals that the rank-performance curve under suboptimal hyperparameters has a much lower ceiling, and that simply "turning up the rank" cannot compensate for poor hyperparameter choices.

• Training data scale (LLaVA-80K vs. competing models): The paper contrasts LLaVA-33B and LLaVA-65B scores with those of earlier models. LLaVA-13B (LLaMA-2, 336px) achieves 32.9 on MM-VET; LLaVA-33B achieves 32.9; LLaVA-65B achieves 35.5. The fact that 33B does not outperform 13B without data mixing (32.9 vs. 32.9) suggests that model scaling alone, on the relatively small LLaVA-80K dataset, may yield diminishing returns—the model may be data-limited rather than capacity-limited at 33B. Data mixing partly alleviates this (33B + data mixing reaches 34.1), consistent with the hypothesis that larger models require proportionally more training data to realize their capacity advantage.

• Resolution × model size interaction: Table 3a provides an implicit ablation by showing resolution effects at each model size. The gain from 336×336 is 2.3 (7B), 3.0 (13B), 2.7 (33B), and 2.0 (65B)—no systematic trend. This stability across a 9× parameter range suggests resolution benefits are approximately independent of LLM capacity, operating through an additive mechanism rather than a multiplicative one that would scale with model size.

• Vicuna vs. LLaMA-2-Chat base model: Table 5 compares LLaVA-70B (LLaMA-2-70B-Chat base) with LLaVA-65B (Vicuna base), both with data mixing. LLaVA-70B scores lower on multimodal benchmarks (69.8 vs. 74.2 on LLaVA-Bench; 35.4 vs. 36.4 on MM-VET) but higher on MMLU (65.1 vs. 62.2). This tradeoff—better language, worse multimodal—may reflect differences in the base models' instruction tuning (LLaMA-2-Chat vs. Vicuna's ShareGPT-based tuning) or differences in the multimodal training procedure, but the paper does not isolate these factors.

• MM-VET evaluator variance: The paper reports standard deviations for MM-VET scores (e.g., LLaVA-33B: 32.9±0.3, LLaVA-65B: 35.5±0.3). These are small relative to the between-model differences (35.5 − 32.9 = 2.6, versus ±0.3 standard deviation), suggesting that evaluator noise is not a confound for the main scaling comparisons. However, for finer-grained comparisons (e.g., LLaVA-33B with vs. without data mixing: 32.9±0.3 vs. 34.1±0.3), the overlapping error bars temper the strength of the conclusion, though the consistent pattern across model sizes supports the directional finding.

• Beam search size × model size interaction: Beam search 5 improves 13B by 3.4 points, 33B by 0.9 points, and 65B by 0.2 points on LLaVA-Bench overall (Table 1). The diminishing benefit at larger scales may indicate improved greedy decoding quality as models grow (larger models are better calibrated, so their most probable output is more often correct) or may reflect ceiling effects on the small LLaVA-Bench dataset. The fact that beam search actually degrades conversation performance for the 65B model (63.8 → 59.4) while improving it for 33B (70.2 → 72.6) suggests a previously undocumented interaction between model scale and search strategy that warrants further investigation.

Critical Assessment

Do the experiments support the claim that scaling LMM consistently enhances model performance?

The experiments demonstrate that scaling from 13B to 65B improves both LLaVA-Bench and MM-VET scores. However, the consistency of improvement is qualified by diminishing returns and capability-specific effects. The jump from 13B to 33B on LLaVA-Bench is 3.8 points; from 33B to 65B it is 0.3 points (Table 3a, no data mixing). On MM-VET without data mixing, 13B achieves 32.5, 33B achieves 32.9—essentially flat—while 65B reaches 35.5. The claim of "consistent" enhancement is therefore only true in the aggregate direction, not in the per-step marginal gain. Additionally, math capability on MM-VET is flat or slightly declining across the full 7B to 65B range (Table 2: 7B at 11.5, 13B at 7.7, 33B at 7.7, 65B at 15.0—but the 65B number without data mixing is not separately reported), demonstrating that scaling does not uniformly enhance all capabilities.

The experiments do not isolate the effect of model size from confounding factors. The 65B model uses a different Vicuna checkpoint (trained by the authors on different ShareGPT data with 159M tokens versus 370M for Vicuna-33B) and the 70B variant uses LLaMA-2-Chat rather than Vicuna. These base model differences mean that the "scaling" curve from 7B to 70B is not a clean interpolation along a single architecture/data axis. The paper acknowledges some of these differences (Section 2 notes the different token counts) but does not control for them, making it difficult to attribute performance differences purely to parameter count.

Do the experiments support the claim that LoRA/QLoRA performance is comparable to full-model fine-tuning?

The claim is supported at 13B (both achieve 70.1) and nearly supported at 33B (LoRA-64: 71.8 vs. Full: 72.0, a 0.2-point gap). However, the claim's generality is limited by several factors. First, the performance at 65B with LoRA is not directly compared to full fine-tuning in a clean head-to-head—the paper reports 72.2 for LoRA-64 on 65B but the full fine-tuning score of 72.3 appears in the same table without explicit statement that they are directly comparable under identical data conditions. Second, the hyperparameter sensitivity result (6.3-point drop from suboptimal alpha/lr) demonstrates that LoRA's "comparability" is contingent on careful tuning—a caveat that matters for practitioners who may not have the compute budget for extensive hyperparameter sweeps. Third, QLoRA's 71.6 versus full fine-tuning's 72.0 on the 33B model represents a 0.4-point gap that, while small, may be consequential for applications where every point matters, and the paper does not test whether this gap persists at 65B.

A significant missing experiment is QLoRA at 65B with full performance reporting. The paper notes that QLoRA is necessary to fit 65B models in GPU memory under ZeRO2, and reports training costs for 65B LoRA-64, but does not provide a QLoRA-64 score for the 65B model. Given the practical importance of QLoRA as the only memory-feasible option at the largest scale, the absence of this number is a notable gap.

Do the experiments support the claim that higher image resolution and data mixing improve LMM performance?

The resolution claim is robustly supported across all four model sizes (Table 3a), with a consistent 2–3 point improvement that shows no systematic interaction with model scale. The weakness is that only two resolution values (224×224 and 336×336) are tested—intermediate resolutions or higher resolutions (e.g., 448×448) are not explored, so the shape of the resolution-performance curve (linear? logarithmic? saturating at some threshold?) is unknown. The paper also does not report whether the resolution benefit varies across task types—does higher resolution help more for OCR and spatial tasks (as one would expect) than for knowledge and reasoning tasks? The MM-VET sub-scores could answer this question but are not presented with and without the resolution change in isolation.

The data mixing claim is supported but mechanistically underexplored. Tables 3a and 5 show consistent multimodal gains from data mixing, but the effect on language benchmarks is inconsistent (33B MMLU improves, 65B MMLU is flat, 33B Vicuna-80 degrades by 5 points). This mixed language effect actually strengthens the paper's argument that data mixing serves multimodal performance rather than language preservation, but it also raises questions the paper does not answer: what is the mechanism? Is the benefit from increased data quantity (80K → 80K + ShareGPT), from task diversity (multimodal + text-only instructions), or from specific properties of the ShareGPT data (multi-turn structure, response length, reasoning complexity)? Without ablating these factors—e.g., comparing ShareGPT mixing to mixing with a different text-only dataset, or comparing different mixing ratios—the claim remains empirical but not explanatory.

Do the experiments support the claim that visual instruction tuning can improve pure language capability?

This is the paper's most novel claim, and it is supported by a single data point: LLaVA-70B achieves 65.1 on MMLU versus 63.1 for LLaMA-2-70B-Chat (Table 5). However, the strength of this evidence is limited by several factors. First, the improvement is measured against the base LLaMA-2-Chat model, not against a control that trained on the same data without images—it is possible that the improvement comes from the additional instruction-tuning data (LLaVA-80K contains complex reasoning and long-form answers, as the paper notes) rather than from the visual modality per se. A text-only control that trained on the textual portions of LLaVA-80K would distinguish between "visual instruction tuning improves language" and "additional instruction data improves language."

Second, the effect is not consistently observed: the 13B and 33B Vicuna-based models show flat or declining MMLU after visual instruction tuning (Table 5), and even the 65B Vicuna-based model without data mixing shows only a marginal gain (62.5 → 62.6). The positive transfer only appears at the 70B scale with the LLaMA-2-Chat base model, making it a scale-dependent and base-model-dependent phenomenon. The paper does not explain why this effect emerges at 70B but not at 65B, nor why it appears with LLaMA-2-Chat but not with comparably-sized Vicuna.

Third, the LLaVA-70B is trained on LLaVA-Instruct-158K (mentioned in the paper's discussion of the result) rather than the standard LLaVA-80K used for other models, introducing a confound between dataset size and model architecture.

A rigorous test of this claim would require: (1) training the same base model at multiple scales with and without images, (2) measuring language benchmarks before and after multimodal training, and (3) including a text-only control that receives equivalent additional training data. The paper's single positive data point is suggestive but does not constitute a controlled experiment.

What experiments would have strengthened the paper?

Data scaling alongside model scaling. The paper notes that LLaVA-80K is small and that "larger data size is essential to train a larger model" (Section 3.2). Yet no experiments vary the training data quantity to test this conjecture. A data scaling curve (performance vs. number of multimodal instruction examples at each model size) would reveal whether the diminishing returns from 33B to 65B reflect a data bottleneck—a model that is underfit to a small dataset will show flat scaling regardless of capacity.

Isolated resolution scaling at each task type on MM-VET. The paper reports resolution effects only on LLaVA-Bench overall scores (Table 3a). Breaking out MM-VET sub-scores by resolution would test the intuitive hypothesis that OCR and spatial tasks benefit more from higher resolution than knowledge and generation tasks. The absence of this analysis is particularly notable because MM-VET was designed to decompose multimodal capabilities and is well-suited to answer this question.

Direct 65B LoRA vs. 65B Full comparison under identical conditions. Table 4 shows both numbers but does not discuss them head-to-head. If LoRA-64 at 65B scores 72.2 and full fine-tuning scores 72.3, the gap is negligible, but this conclusion requires the reader to infer that the training data and evaluation conditions are identical. An explicit statement and ideally multiple runs to estimate variance would strengthen this comparison.

QLoRA at 65B with performance reporting. The paper establishes QLoRA as the memory-feasible method for 65B training but only reports its performance at 33B. Practitioners considering 65B training need to know whether QLoRA's performance at that scale matches the trends observed at 33B.

MMLU sub-scores for LLaVA-70B. The 65.1 MMLU score is an aggregate across 57 subjects. Sub-scores would reveal whether the language improvement is concentrated in subjects that overlap with visual reasoning (e.g., physics, chemistry, biology where diagrams are common in training data) or is broadly distributed, which would distinguish between cross-modal transfer and simply "more data" explanations.

Controlled data mixing ratio experiments. The paper mixes LLaVA-80K with ShareGPT at the natural ratio of their sizes, but does not test whether this ratio is optimal. Experiments varying the mixing ratio (e.g., 90% multimodal / 10% text vs. 50/50 vs. 10/90) would characterize the tradeoff surface and provide practical guidance for dataset composition.

Statistical significance on LLaVA-Bench. With only 24 images and 60 questions, LLaVA-Bench comparisons (especially between models differing by 0.2–0.3 points, as in the 33B LoRA vs. Full comparison) are inherently noisy. The paper averages 3 evaluation runs for Table 4 but does not report variance for LLaVA-Bench specifically, making it difficult to assess whether small differences are signal or noise. Bootstrap confidence intervals or paired comparison tests would clarify which scaling decisions produce statistically reliable improvements.

6. Limitations and Trade-offs

Limitation 1: Single Benchmark Family and Single Vision Encoder Architecture

The assumption or constraint. All multimodal experiments use the CLIP ViT-L/14 vision encoder, frozen throughout training, and evaluate on three benchmarks—LLaVA-Bench, MM-VET, and MM-Bench—all of which use GPT-4 as the evaluator for open-ended outputs. The language model backbone spans two families (Vicuna based on LLaMA-1, and LLaMA-2-Chat for the 70B variant), but the vision encoder architecture, pretraining procedure, and resolution are held constant. The paper acknowledges this scope constraint only in passing (Section 4):

"We leave it to future work how to scale the vision encoder to enhance the visual capabilities and improve model performance on vision recognition and understanding tasks."

The consequence. The paper's findings about which capabilities scale with LLM size and which do not cannot be separated from the specific vision encoder used. The observation that math and spatial reasoning scores remain flat from 7B to 65B (Table 2: math hovers between 7.3 and 15.0 across model sizes, spatial between 28.0 and 33.0) could reflect either a genuine ceiling on what language model scaling can achieve for visual reasoning, or a bottleneck imposed by the frozen CLIP encoder's representational limits. If the CLIP ViT-L/14 discards fine-grained spatial information or cannot adequately encode mathematical notation in images, then scaling the LLM alone would predictably fail to improve those capabilities regardless of how much capacity is added. This confound means that the paper's central finding—differential scaling across capability dimensions—cannot be interpreted as a finding about multimodal scaling in general. It is a finding about multimodal scaling given this specific vision encoder. A practitioner considering whether to invest in LLM scaling versus vision encoder scaling for their application cannot determine from this paper whether upgrading the vision encoder would unlock those flat capabilities.

Furthermore, all three multimodal benchmarks use GPT-4 as an automated evaluator. While the MM-VET paper established reasonable correlation between GPT-4 evaluation and human judgment, the evaluator itself is a moving target (the paper uses gpt4-0314 for LLaVA-Bench and gpt4-0613 for MM-VET—different model versions). Any systematic bias in GPT-4's evaluation of multimodal outputs—for instance, favoring verbose or stylistically fluent responses over factually accurate ones—would propagate into all the reported scores. The small standard deviations on MM-VET (±0.1 to ±0.3) measure evaluator consistency, not evaluator accuracy, and cannot detect systematic evaluator bias.

What evidence exists in the paper. Table 2 shows math scores that are essentially flat across a ~9× parameter increase (7B: 11.5, 13B: 7.7, 33B: 7.7, 65B: 15.0). The 65B jump to 15.0 appears only with data mixing and may reflect the increased training data rather than model scale. Spatial awareness similarly shows modest gains (28.0 → 33.0 from 13B to 65B) that are smaller than the gains in knowledge (26.3 → 30.4) and generation (28.8 → 32.3). The paper presents these as findings about capability-specific scaling, but provides no evidence that the vision encoder is not the bottleneck. No experiment varies the vision encoder architecture, pretraining dataset, or resolution beyond the 224×224 vs. 336×336 comparison (which uses the same encoder, just fine-tuned at higher resolution during CLIP pretraining).

Mitigation status. The paper does not mitigate this limitation. The acknowledgment in Section 4 is a direction-setting statement, not a controlled investigation. The authors explicitly defer vision encoder scaling to future work. No experiments test whether a different vision encoder (e.g., ViT-G, EVA-CLIP, or a detection-based feature extractor) would change the scaling patterns, particularly for spatial and math capabilities.

---

Limitation 2: Training Data Scale Is Held Constant While Model Scale Increases

The assumption or constraint. All models are trained on the same LLaVA-80K multimodal instruction dataset for one epoch, regardless of parameter count. The data mixing experiments add ShareGPT text-only data, but the multimodal component remains fixed at ~80K examples across all model sizes from 7B to 65B. The paper acknowledges this tension explicitly (Section 3.2):

"We conjecture that larger data size is essential to train a larger model. For example, if we only train on LLaVA-80K data, we see smaller gain when model size becomes larger."

The consequence. The scaling curves reported in this paper confound two effects: the benefit of additional model capacity, and the extent to which that capacity is utilized given a fixed data budget. If larger models are increasingly data-limited on the small LLaVA-80K dataset, then the observed diminishing returns from 33B to 65B (Table 3a: 72.0 → 72.3 without data mixing, a gain of only 0.3 points) reflect data starvation, not a fundamental ceiling on what larger models can achieve. The true scaling behavior of LLaVA with model size is unknown—it could be that 65B models trained on proportionally larger multimodal datasets (e.g., 320K or 640K examples) would show substantially larger gains over their 33B counterparts.

This data limitation also affects the interpretation of difficulty-dependent capability scaling. The observation that knowledge and generation improve with model scale while math and spatial awareness do not (Table 2) could be an artifact of data, not architecture: the LLaVA-80K dataset may contain abundant examples requiring knowledge retrieval and fluent generation, but few examples that exercise mathematical reasoning or precise spatial understanding from visual inputs. If the training data does not provide learning signal for certain capabilities, then scaling the model cannot improve those capabilities regardless of capacity. The paper cannot distinguish between "this capability does not benefit from LLM scaling" and "this capability is not adequately represented in the training data."

The data mixing experiments partially support this concern. When ShareGPT text-only data is added, performance improves across multiple capabilities (Table 2: OCR goes from 25.0 to 27.1 for 33B, knowledge from 26.2 to 30.4 for 65B, generation from 28.3 to 32.3 for 65B). This suggests that the models were data-limited on LLaVA-80K alone and that additional training signal—even text-only—unlocks gains that pure model scaling could not. If text-only data can produce these improvements, it is plausible that additional multimodal data would produce even larger ones, but this experiment is not run.

What evidence exists in the paper. The evidence for data limitation is primarily the flattening of the scaling curve without data mixing. On LLaVA-Bench (Table 3a): 7B→13B improves by 3.5 points (63.6→67.1 at 224×224), 13B→33B improves by 2.2 points (67.1→69.3), and 33B→65B improves by only 1.0 point (69.3→70.3). The marginal return per 10B parameters shrinks dramatically. On MM-VET (Table 2): LLaVA-13B scores 32.5, LLaVA-33B scores 32.9 (+0.4), LLaVA-65B scores 35.5 (+2.6). The jump from 33B to 65B is larger than from 13B to 33B, which is inconsistent with simple capacity saturation and may reflect the different base model training (the 65B Vicuna was trained by the authors on different data). The flat performance from 13B to 33B on MM-VET without data mixing is the clearest signal of data limitation at intermediate scales.

The paper's LLaVA-70B variant, trained on LLaVA-Instruct-158K (roughly double the standard dataset), achieves 65.1 on MMLU—substantially above the Vicuna-based models trained on LLaVA-80K (55.0–62.6). While not a controlled comparison (different base model, different dataset size, different data composition), this result is consistent with the hypothesis that data quantity matters significantly at larger scales.

Mitigation status. The paper does not attempt to mitigate this limitation through experimentation. The acknowledgment is a conjecture in the discussion, not a tested hypothesis. No experiments vary the multimodal training data quantity to construct a data scaling curve. The paper suggests future work using "much larger datasets to investigate in detail whether and how different methods of training data selection and mixing can improve the quality of much larger LMM" (Section 4), but provides no empirical evidence about what scaling behavior to expect. A practitioner cannot determine from this paper whether to invest in a larger model or a larger dataset when scaling up their multimodal system.

---

Limitation 3: The Language-Improvement Claim Rests on a Single Data Point with Multiple Confounds

The assumption or constraint. The paper's most striking claim—that "visual instruction tuning can sometimes improve LMM's pure language capability" (Abstract) and specifically that "visual instructing tuning improve language ability of large-scale LMM" (Section 3.2)—rests on one result: LLaVA-70B (based on LLaMA-2-70B-Chat) achieves 65.1 on MMLU compared to 63.1 for the base LLaMA-2-70B-Chat (Table 5). The authors explicitly claim:

"To the best of our knowledge, this is the first reported result which shows visual instruction tuning improve language ability of large-scale LMM."

The consequence. This single data point has at least three confounds that prevent attribution of the MMLU improvement to visual instruction tuning specifically:

1. Dataset size confound: The LLaVA-70B model was trained on LLaVA-Instruct-158K, not the standard LLaVA-80K. This means the model received roughly double the instruction-tuning data of the other LLaVA variants. The MMLU improvement from 63.1 to 65.1 could be entirely attributable to additional language instruction data (the LLaVA-Instruct conversations contain complex reasoning and long-form answers, as the paper notes), with the visual modality being irrelevant. No text-only control—training the same base model on the textual portions of LLaVA-Instruct-158K without images—is provided.

2. Base model confound: LLaMA-2-70B-Chat is a different base model than the Vicuna models used for all other experiments. It uses a different pretraining corpus, different instruction-tuning recipe, and different alignment procedure. The MMLU improvement from 63.1 to 65.1 cannot be directly compared to the Vicuna-65B's trajectory (62.5 → 62.6 without data mixing, 62.5 → 62.2 with data mixing) because the starting points and architectures differ. The claim that visual instruction tuning improves language capability relies on a cross-base-model comparison that is inherently uncontrolled.

3. Scale confound: The positive transfer appears only at 70B. The 13B model shows a decline on MMLU after visual instruction tuning (55.8 → 55.0, Table 5), the 33B model without data mixing shows a decline (59.0 → 56.1), and even the 33B with data mixing only recovers to 58.6—still below the base Vicuna-33B's 59.0. The 65B with data mixing shows a minimal decline (62.5 → 62.2). Only at 70B with LLaMA-2-Chat does the effect reverse. This scale-dependence means the claim cannot be generalized—it applies specifically to 70B-class models with LLaMA-2-style training, and possibly only with the LLaVA-Instruct-158K dataset. A practitioner considering visual instruction tuning for a 13B or 33B model would observe the opposite effect (language degradation) based on this paper's own data.

What evidence exists in the paper. Table 5 contains all the relevant numbers. The MMLU progression for Vicuna-based models is: Vicuna-13B 55.8 → LLaVA-13B 55.0 (−0.8); Vicuna-33B 59.0 → LLaVA-33B no-mix 56.1 (−2.9) → LLaVA-33B mix 58.6 (−0.4); Vicuna-65B 62.5 → LLaVA-65B no-mix 62.6 (+0.1) → LLaVA-65B mix 62.2 (−0.3). The LLaMA-2-70B-Chat progression is: base 63.1 → LLaVA-70B mix 65.1 (+2.0). The Vicuna-80 scores show a similar pattern: Vicuna-13B 79.9 → LLaVA-13B 79.6 (−0.3); Vicuna-33B 85.6 → LLaVA-33B no-mix 85.3 (−0.3) → LLaVA-33B mix 80.3 (−5.3); Vicuna-65B 83.2 → LLaVA-65B no-mix 84.5 (+1.3) → LLaVA-65B mix 82.6 (−0.6). Across 8 Vicuna-based comparisons, 6 show language degradation, 1 shows marginal improvement, and 1 shows moderate improvement. The single LLaMA-2-based comparison shows improvement.

Mitigation status. The paper does not mitigate these confounds. It presents the LLaVA-70B MMLU result as evidence for the claim without discussing the dataset size difference, the base model difference, or the scale-dependence of the effect. No controlled experiment isolates the contribution of visual data from additional instruction data. The claim is therefore best interpreted as an existence proof—under some conditions (70B scale, LLaMA-2-Chat base, LLaVA-Instruct-158K data), visual instruction tuning does not degrade language performance and may improve it—rather than a general finding about the relationship between multimodal and unimodal training.

---

Limitation 4: LoRA/QLoRA Findings Are Hyperparameter-Sensitive in Ways the Paper Only Partially Characterizes

The assumption or constraint. The paper's recommendation that LoRA/QLoRA are "comparable to the performance of full-model fine-tuning" (Abstract) and provide a "good performance-cost trade-off" (Section 3.2) assumes that practitioners can reproduce the hyperparameter configuration that makes this true. The paper's own ablation reveals that this configuration is narrow: reducing the learning rate from 1×10⁻⁴ to 2×10⁻⁵ and LoRA alpha from 2×rank to 16 causes a 6.3-point performance collapse on LLaVA-Bench for the 33B model with rank 64 (71.8 → 65.5). The paper notes (Section 3.2):

"In the experiments, we find that the hyperparameters of LoRA have a large impact of performance: (i) Large learning rate and alpha value of LoRA improves the results significantly."

The consequence. The practical utility of LoRA/QLoRA depends critically on whether the effective hyperparameter regime transfers across model scales, architectures, and datasets. The paper establishes that for Vicuna-based LLaVA models on LLaVA-80K, learning rate 1×10⁻⁴ and alpha=2×rank work well at 13B and 33B. But these values were found through hyperparameter search (Section 2: "We conducted a set of hyperparameter search and for LoRA runs, and found larger LoRA alpha or equivalently larger learning rate was crucial to get the best performance"), and the paper does not report the search range, the number of configurations tested, or whether the optimal values differ across scales. A practitioner applying LoRA to a different model family (e.g., LLaMA-3, Mistral, or Qwen), a different multimodal dataset, or a different task would need to repeat this hyperparameter search—potentially at substantial cost, since the search itself requires training runs at the target scale.

The sensitivity is not symmetric: using suboptimal hyperparameters is punished far more severely than using optimal hyperparameters is rewarded. The 6.3-point drop from "good" to "bad" hyperparameters at rank 64 is larger than the total gain from scaling from 7B to 13B (~4 points) or from adding data mixing (~2 points). This means that a practitioner who fails to tune LoRA hyperparameters adequately would be better off using a smaller model with well-tuned full fine-tuning than a larger model with poorly-tuned LoRA. The paper provides no diagnostic for determining whether a given LoRA configuration is near-optimal without running a full hyperparameter sweep.

Furthermore, the finding that increasing rank from 64 to 512 under suboptimal hyperparameters only recovers from 65.5 to 68.1—still well below the 71.8 achieved with rank 64 and optimal hyperparameters—reveals that LoRA rank cannot compensate for poor learning rate and alpha choices. The interaction is sharp: the effective capacity of LoRA is determined jointly by rank, alpha, and learning rate, and the paper only explores a small region of this 3D space. The reported optimal configuration (rank=64, alpha=128, lr=1×10⁻⁴) is a point estimate, not a characterized basin.

What evidence exists in the paper. Section 3.2 and Table 4 provide the key evidence. The 6.3-point drop is explicitly reported. The paper also reports that under suboptimal hyperparameters, rank 128 achieves 66.1 and rank 512 achieves 68.1—a diminishing-returns curve with a low ceiling. However, the paper does not report whether the optimal hyperparameters change with rank (e.g., does rank 512 need a different alpha or learning rate than rank 64?), whether they change with model scale (does 65B LoRA need different settings than 33B LoRA?), or whether they transfer across dataset compositions (does data mixing change the optimal LoRA configuration?). The hyperparameter search methodology is not described in sufficient detail for reproduction—the number of configurations tested, the search strategy (grid, random, Bayesian), and the validation metric used for selection are not specified.

Mitigation status. The paper partially mitigates this limitation by explicitly reporting the effective hyperparameter values (learning rate 1×10⁻⁴, alpha=2×rank) and by demonstrating the sensitivity through the ablation. This is more transparency than most papers provide. However, the paper does not provide a characterization of the stable hyperparameter region—e.g., "alpha between 1.5×rank and 3×rank with lr between 8×10⁻⁵ and 2×10⁻⁴ all produce performance within 0.5 points of optimal." Without this, the reported configuration is a recipe, not a principle, and its transferability to new settings is unknown. The paper also does not discuss the cost of the hyperparameter search itself, which should be included in any total-cost comparison between LoRA and full fine-tuning—if finding the right LoRA hyperparameters requires training 5–10 model variants, the cost advantage over full fine-tuning narrows considerably.

---

Limitation 5: Difficulty Estimation and Per-Task Capability Boundaries Are Not Characterized

The assumption or constraint. The paper reports aggregate benchmark scores and capability-specific sub-scores (MM-VET's six dimensions), but does not analyze how performance varies with problem difficulty within each capability. All questions within a capability dimension are weighted equally, and there is no difficulty stratification. A model that improves on "recognition" by 3 points could be improving exclusively on easy recognition tasks (identifying common objects in canonical poses) while remaining unchanged on hard ones (identifying rare objects in cluttered scenes). The aggregate score masks this distribution.

The consequence. A practitioner cannot determine from this paper whether scaling from 13B to 65B will help on the specific kinds of problems their application faces. If an application involves predominantly easy multimodal tasks (simple visual QA, standard object recognition), the scaling gains reported in this paper may overstate the practical benefit—the 13B model may already perform adequately, and the gains from scaling accrue primarily on tasks that are irrelevant to the deployment. Conversely, if an application involves hard multimodal tasks (detailed spatial reasoning, mathematical problem-solving from diagrams, OCR in challenging fonts or layouts), the aggregate improvements may overstate the benefit because those hard tasks show flat scaling curves. The MM-VET math sub-score (flat from 7B to 65B) demonstrates that this is a real concern, but the paper does not extend this difficulty analysis to other capabilities.

Without difficulty stratification, the paper also cannot identify ceiling effects on benchmarks. LLaVA-Bench overall scores cluster in the 70–75 range for 13B–65B models (Table 1), with beam search 5 scores of 73.5, 74.8, and 74.4. These numbers are close enough that the benchmark may be saturating—there may be a set of "easy" questions that all models get right and a set of "hard" questions that no model gets right, with only a narrow slice of questions actually discriminating between model scales. If this is the case, the true scaling behavior on that discriminative slice may be steeper (or flatter) than the aggregate suggests. The paper acknowledges this possibility for LLaVA-Bench specifically (Section 3.1: "LLaVA-Bench is small (thus the comparison might not be statistically significant)"), but does not analyze difficulty distributions on any benchmark.

The absence of difficulty analysis also means the paper provides no guidance on error modes. Do larger models make different kinds of mistakes than smaller ones, or the same kinds of mistakes less frequently? Do they fail on the same hard examples (suggesting a fundamental capability ceiling) or do they succeed on examples where smaller models fail (suggesting scaling unlocks new capabilities)? The aggregate scores cannot answer these questions, and the MM-VET sub-scores only decompose by capability, not by difficulty within capability.

What evidence exists in the paper. The MM-VET sub-scores (Table 2) provide the closest thing to a difficulty decomposition by separating capabilities that plausibly differ in difficulty (math is harder than recognition for current models). The flat math score (7.7 for 13B, 7.7 for 33B, 15.0 for 65B with data mixing) demonstrates that a hard capability does not improve with scale, while easier capabilities like recognition (38.4 → 41.8) show modest gains. However, this is capability decomposition, not difficulty decomposition—within recognition, there are presumably easy and hard examples, and the paper does not analyze how the 3.4-point gain distributes across them.

The small size of LLaVA-Bench (24 images, 60 questions) is noted in Section 3.1 as a limitation for statistical significance, but the paper does not examine whether the benchmark's composition (e.g., ratio of easy to hard questions) affects the interpretability of the scores. Similarly, MM-VET (200 images, 218 questions) and MM-Bench (2,974 questions) are used without difficulty analysis.

Mitigation status. The paper does not attempt to mitigate this limitation. No difficulty annotations, item response theory analysis, or per-question performance breakdowns are provided for any benchmark. The capability decomposition in MM-VET is a partial mitigation—it reveals that some capabilities do not scale—but it does not replace within-capability difficulty analysis. The paper's recommendation to use larger models for knowledge-intensive tasks and to invest in vision encoder scaling for perceptual tasks (implicit in Section 4) is a capability-level recommendation that would be sharpened by difficulty-level analysis: a practitioner might find that for knowledge tasks below a certain difficulty threshold, even a 7B model suffices, while for knowledge tasks above that threshold, scaling to 65B is necessary. The current results cannot support this kind of deployment decision.

---

Limitation 6: Beam Search and Inference-Time Compute Scaling Are Unexplored Despite Being a Free Parameter

The assumption or constraint. The paper reports beam search 1 vs. beam search 5 as a secondary result in Table 1, showing that beam search consistently improves LLaVA-Bench scores for smaller models (13B: +3.4 points) but provides diminishing or even negative returns for larger models (33B: +0.9; 65B: +0.2 overall, with a -4.4 point degradation on conversation tasks). The paper treats this as a minor observation and does not systematically explore inference-time compute scaling—no sweep of beam sizes beyond 1 and 5, no comparison to other decoding strategies (nucleus sampling, best-of-N, temperature sweep), and no analysis of how beam search interacts with model scale, task type, or difficulty.

The consequence. The paper's headline numbers for larger models may understate their achievable performance. If beam search 5 degrades conversation performance for the 65B model (63.8 → 59.4, Table 1), but beam search with size 3 or with a different search strategy improves it, then the reported numbers do not represent the model's true capability. More broadly, the paper treats model scaling and inference-time computation as independent factors, but they are likely interdependent—larger models may benefit from different decoding strategies than smaller ones, and the optimal beam size may depend on task type. By not exploring this space, the paper leaves a free parameter on the table that could significantly shift the reported performance comparisons.

This limitation is particularly relevant given the paper's cost-effectiveness framing. Beam search with size 5 approximately triples inference cost relative to greedy decoding (5× as many tokens generated in the final step, plus the cost of maintaining the beam), yet the paper claims this comes with "negligible increase of inference latency" (Section 3.1). This claim is not quantified and is inconsistent with standard beam search cost analysis—for autoregressive models, beam search 5 increases computation by roughly a factor of 5 during the search process (though with potential optimizations like shared key-value caches, the practical overhead may be lower). If a practitioner is choosing between a 13B model with beam search 5 (scoring 73.5) and a 65B model with greedy decoding (scoring 74.2), the inference cost of the 13B+beam5 configuration may approach or exceed that of the 65B+greedy configuration, fundamentally changing the cost-performance tradeoff from what Table 4 implies.

The negative beam search result for 65B conversation tasks is particularly concerning and unexplored. The paper reports that beam search 5 causes a 4.4-point drop on conversation (63.8 → 59.4) while improving reasoning (87.3 → 88.7) and detail (62.3 → 65.7). This task-dependent sign reversal suggests that beam search is not simply a "more compute → better outputs" mechanism—it can actively harm performance on certain task types for certain model scales. Without understanding why, practitioners cannot safely deploy beam search as a default inference-time enhancement. The paper provides no analysis of this phenomenon—no example outputs, no investigation of whether the degradation reflects beam search's known tendency toward shorter, less diverse outputs, and no test of whether smaller beam sizes avoid the degradation while preserving the gains on reasoning and detail.

The beam search observation also connects to a broader missed opportunity: the paper does not position inference-time compute as a scaling axis in its own right, analogous to how it treats model size, image resolution, and data mixing. A systematic sweep of inference-time compute budgets (varying beam size, sampling temperature, or best-of-N) at each model scale would reveal whether inference-time compute can substitute for model scale—i.e., whether a 13B model with aggressive beam search can match a 33B model with greedy decoding, and at what cost. This is the central question of test-time compute scaling, and the paper's data contains suggestive evidence (13B+beam5: 73.5 vs. 33B+greedy: 73.9) without systematically exploring it.

What evidence exists in the paper. Table 1 provides the only beam search comparison, limited to sizes 1 and 5 on LLaVA-Bench only. The per-task sub-scores reveal the task-dependent effects described above. The paper does not report beam search results for MM-VET, MM-Bench, or the language benchmarks. No analysis of beam search cost, latency, or interaction with model scale is provided beyond the "negligible increase" claim. The MM-VET and MM-Bench results use unspecified decoding strategies—presumably greedy (beam=1), but this is not stated explicitly.

Mitigation status. The paper does not mitigate this limitation. Beam search is presented as a minor performance boost, not as a scaling axis worthy of systematic study. The negative conversation result for 65B is reported without comment or investigation. The paper does not suggest future work on inference-time compute scaling. This is a significant gap because inference-time compute is a practical degree of freedom that every deployment must choose, and the paper's own data shows it can change the relative ordering of models.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new architecture, a new training objective, or a new benchmark. Its contribution is infrastructural: it establishes what happens when you scale visual instruction tuning to model sizes that were previously inaccessible to open-source research, and it provides the checkpoints, cost data, and capability decompositions that enable the rest of the field to reason about scaling decisions without independently reproducing expensive training runs. This is best characterized as a foundational empirical contribution—analogous to the role that scaling law papers (Kaplan et al., 2020; Hoffmann et al., 2022) played for language model pretraining, but applied to the multimodal instruction-tuning regime.

The specific shifts this work introduces are threefold.

First, it reframes the relationship between language and vision in multimodal training from a zero-sum tradeoff to a potentially synergistic interaction. The dominant assumption in the field—inherited from continual learning and multi-task training research—was that adding a visual modality to a language model would come at the cost of degraded language performance. The paper's finding that LLaVA-70B achieves 65.1 on MMLU, a 2.0-point improvement over its base LLaMA-2-70B-Chat's 63.1 (Table 5), challenges this assumption with concrete evidence. The authors explicitly flag this as unprecedented—"the first reported result which shows visual instruction tuning improve language ability of large-scale LMM"—and while the finding is limited by confounds (different dataset size for the 70B model, different base model architecture, no text-only control), its existence reframes the question from "how do we prevent language degradation during multimodal training?" to "under what conditions does multimodal training improve language capabilities?" This is a conceptual reframing that changes what researchers should measure and expect.

Second, it provides the first disaggregated picture of multimodal scaling across capability dimensions. The MM-VET results in Table 2 decompose the aggregate scaling signal into six sub-capabilities, revealing a sharply uneven pattern: knowledge and generation improve substantially with model scale (26.3 → 30.4 and 28.8 → 32.3 from 13B to 65B with data mixing), recognition and OCR improve modestly (38.4 → 41.8 and 21.0 → 27.9), and math remains essentially flat (7.7 → 7.3). This differential scaling pattern provides a diagnostic that was previously unavailable. It tells the field that scaling the LLM backbone is an effective strategy for capabilities that depend on stored knowledge and fluent output generation, but a poor strategy for capabilities that require precise visual reasoning or symbolic manipulation from visual inputs. This finding challenges the simple "bigger is better" narrative and redirects attention toward the vision encoder as the likely bottleneck for perceptual and mathematical capabilities—a hypothesis the paper explicitly flags for future work.

Third, it converts the cost-performance relationship for parameter-efficient training from abstract claims into a concrete decision matrix. Table 4's simultaneous reporting of LLaVA-Bench scores, GPU-hours per node per epoch, and trainable parameter counts across four model scales and five training configurations (full fine-tuning, LoRA-8, LoRA-64, QLoRA-64 at multiple scales) provides a directly actionable resource. The finding that LoRA rank 64 on the 33B model achieves 71.8 versus 72.0 for full fine-tuning at 63% of the training cost, combined with the critical caveat that LoRA performance collapses by 6.3 points under suboptimal hyperparameters, establishes both the promise and the risk of parameter-efficient methods at scale. This is not a novel method—LoRA and QLoRA were both known—but the systematic characterization of their scaling behavior and hyperparameter sensitivity in the multimodal domain is new and practically valuable.

The paper also resolves a tension in the prior literature. Earlier work on the original LLaVA (7B and 13B) and MiniGPT-4 demonstrated that visual instruction tuning works at small scales. Concurrent commercial systems (GPT-4, Bard, Bing Chat) demonstrated strong multimodal capabilities at unknown scales. The open question was whether the small-scale results would transfer to larger models, or whether visual instruction tuning would hit diminishing returns, overfitting, or other scale-dependent failure modes. This paper's answer is that scaling transfers, but unevenly: the aggregate numbers improve, but the gains concentrate in language-heavy capabilities while perceptual capabilities stall. This provides a more nuanced picture than either "scaling works" or "scaling doesn't work"—it tells researchers where scaling pays off and where it does not.

The research directions this paper makes more attractive include: scaling the vision encoder in tandem with the LLM (since the current bottleneck for perceptual capabilities is likely visual, not linguistic), investigating data scaling laws for multimodal instruction tuning (since the fixed LLaVA-80K dataset appears to limit larger models), and exploring the conditions under which multimodal training produces positive transfer to language capabilities. Research directions this paper makes less attractive include: scaling the LLM alone while keeping the vision encoder and training data fixed (the diminishing returns from 33B to 65B suggest this strategy is reaching exhaustion on current benchmarks), and deploying parameter-efficient methods without thorough hyperparameter tuning (the 6.3-point drop from suboptimal settings demonstrates that LoRA's cost advantage evaporates if tuning is inadequate).

Follow-Up Research This Work Enables

Data scaling laws for multimodal instruction tuning, holding architecture constant. The paper's most revealing negative result is the flattening of performance from 33B to 65B on the fixed LLaVA-80K dataset. On LLaVA-Bench without data mixing (Table 3a), 33B scores 72.0 and 65B scores 72.3—a gain of only 0.3 points. On MM-VET without data mixing (Table 2), 33B scores 32.9 and 65B scores 35.5—a more substantial 2.6-point gain, but one that is confounded by the different Vicuna-65B base model training. The authors explicitly conjecture that "larger data size is essential to train a larger model," but provide no evidence. A direct follow-up would train LLaVA at 7B, 13B, 33B, and 65B on multimodal instruction datasets of increasing size—e.g., 20K, 80K, 320K, 1.28M examples—using a consistent data generation pipeline (e.g., GPT-4V-generated instruction data from a fixed set of images). The key measurement is whether the scaling exponent (performance vs. data size) changes with model scale: if larger models benefit proportionally more from additional data, then the current paper's diminishing returns are a data artifact, not a capacity ceiling. This experiment would cost roughly 16× the compute of the current paper (4 data sizes × 4 model scales) plus the cost of generating the larger datasets, making it feasible for a well-resourced academic lab or industry group. The result would directly inform whether the community should invest in scaling models, scaling data, or both.

Vision encoder scaling with controlled LLM scale to isolate perceptual bottlenecks. The paper's finding that math and spatial reasoning scores remain flat from 7B to 65B (Table 2: math hovers between 7.3 and 15.0 depending on data mixing and model scale, spatial between 28.0 and 33.0) raises the question of whether the frozen CLIP ViT-L/14 vision encoder is the limiting factor. A controlled experiment would fix the LLM at 13B or 33B and vary the vision encoder—comparing CLIP ViT-L/14 (current), ViT-G/14 (roughly 4× the parameters), and possibly a detection-based encoder (e.g., features from a frozen DINOv2 or EVA-02 model). The hypothesis is that perceptual capabilities (recognition, OCR, spatial, math) would improve with vision encoder scale while knowledge and generation would remain relatively flat (since those depend primarily on the LLM). If confirmed, this would establish a design principle: scale the vision encoder for perceptual tasks, scale the LLM for knowledge-intensive tasks. If disconfirmed—if even a much larger vision encoder does not improve math and spatial scores—then the bottleneck is architectural (the CLIP pretraining objective may not encode the right kind of visual features for mathematical reasoning) rather than capacity-limited. This experiment would require training the projection layer for each vision encoder variant (Stage 1), then fine-tuning on LLaVA-80K (Stage 2), at a cost roughly 3–4× the current paper's 13B experiments—manageable for a single model scale comparison.

Controlled experiment to distinguish cross-modal transfer from additional data in the MMLU improvement. The paper's most provocative finding—that LLaVA-70B improves MMLU by 2.0 points over its base model—has three confounds: the LLaVA-70B was trained on LLaVA-Instruct-158K (2× the standard dataset), it uses a different base model (LLaMA-2-70B-Chat vs. Vicuna), and the improvement only appears at 70B scale. A clean follow-up would train three variants of LLaMA-2-70B-Chat: (A) the base model with no additional training, (B) a text-only control trained on the textual portion of LLaVA-Instruct-158K (stripping out images and treating each example as a language-only instruction), and (C) the full LLaVA-70B with images. If variant B matches variant C on MMLU, the improvement is purely from additional instruction-tuning data, and the "visual instruction tuning improves language" claim is false—it's just "more data improves language." If variant B shows no improvement over A but variant C shows the +2.0 point gain, then the visual modality specifically drives the improvement, supporting the cross-modal transfer hypothesis. If both B and C show improvement but C shows more, the effect is partially data-driven and partially modality-driven. This experiment would cost roughly 2 additional training runs (text-only control and replication of the full LLaVA-70B for direct comparison) and would provide the field with a definitive answer to a question the current paper only raises.

Difficulty-stratified evaluation of multimodal scaling to identify where gains accrue. The paper reports aggregate and capability-specific scores but provides no difficulty decomposition—we do not know whether scaling from 13B to 65B improves performance on easy, medium, or hard questions within each capability. An item-level analysis of MM-VET, using the per-question accuracy of a baseline model (e.g., LLaVA-7B) as a difficulty proxy, would bin questions into difficulty quintiles and plot scaling curves separately for each bin. The key question: does scaling help more on easy questions (suggesting that larger models are better at exploiting existing patterns), on medium questions (suggesting genuine capability expansion), or on hard questions (suggesting new reasoning strategies)? The paper's own data hints at an answer—the flat math score suggests hard questions drive the aggregate, since easy math questions might already be solvable at 7B—but a systematic analysis would reveal whether this pattern generalizes. This analysis requires no new training, only per-question scoring logs from the existing model runs, making it essentially zero-cost to execute on the already-collected data.

Systematic characterization of the LoRA hyperparameter stability region across model scales. The paper demonstrates that LoRA performance is acutely sensitive to learning rate and alpha (6.3-point drop from suboptimal settings on 33B), but provides only a single optimal configuration. A follow-up study would perform a grid search over learning rate {5×10⁻⁶, 1×10⁻⁵, 2×10⁻⁵, 5×10⁻⁵, 1×10⁻⁴, 2×10⁻⁴} and alpha {0.5×rank, 1×rank, 2×rank, 4×rank} at model scales 7B, 13B, 33B, and 65B, all on LLaVA-80K, measuring LLaVA-Bench and MM-VET. The output would be a stability map: for each model scale, what region of (lr, alpha) space produces performance within 0.5 points of the optimum? If the stable region shrinks with model scale, then LoRA becomes progressively harder to tune at larger scales—a finding that would temper the paper's recommendation of LoRA as a cost-effective alternative. If the stable region is roughly constant, then the hyperparameter transfer is reliable. The cost of this study is a 4×6×4 = 96-training-run grid (scales × lr values × alpha values), though intelligent search strategies (Bayesian optimization, successive halving) could reduce this substantially. This is a practical contribution that would directly inform deployment decisions for organizations adopting LoRA at scale.

Inference-time compute scaling as a function of model size and task type. The paper's beam search results (Table 1) reveal a striking pattern: beam search 5 improves LLaVA-Bench overall by 3.4 points at 13B, 0.9 points at 33B, and 0.2 points at 65B, while degrading conversation performance at 65B by 4.4 points. This interaction between model scale and decoding strategy is not explained or explored. A systematic study would sweep beam sizes {1, 2, 3, 5, 10}, sampling temperatures {0.0, 0.2, 0.5, 0.8, 1.0}, and best-of-N {1, 4, 16, 64} for LLaVA at 7B, 13B, 33B, and 65B on LLaVA-Bench and MM-VET, reporting per-task-type performance and total inference FLOPs. The key question: can inference-time compute substitute for model scale? If LLaVA-13B with best-of-64 achieves the same MM-VET score as LLaVA-65B with greedy decoding, then practitioners with latency tolerance can achieve large-model performance at small-model cost. If the negative beam search effect on 65B conversation is replicated and explained (e.g., beam search systematically prefers shorter, less informative responses on conversational tasks), it would establish an important boundary condition on when test-time compute is beneficial versus harmful. This study requires no new training, only inference-time sweeps over existing checkpoints, making it the lowest-cost follow-up with the highest practical impact.

Practical Applications and Downstream Use Cases

Cost-effective multimodal deployment for knowledge-intensive applications. Organizations deploying multimodal assistants for tasks that depend on stored knowledge and fluent explanation—visual question answering about historical artifacts, scientific diagram explanation, product recommendation with visual context—should prefer larger LLaVA models (33B–65B) based on this paper's MM-VET results. The knowledge sub-score improves from 26.3 (LLaVA-13B) to 30.4 (LLaVA-65B with data mixing), and generation improves from 28.8 to 32.3 (Table 2). These 4–5 point improvements represent a meaningful gain in the quality of knowledge-grounded visual responses. The LoRA/QLoRA results further enable this: a team with a single A100-80G node can fine-tune LLaVA-33B with LoRA rank 64 in 5.8 GPU-hours per epoch (Table 4) at approximately $79 (using the paper's Azure pricing of$13.63/hour), compared to roughly $184 for full fine-tuning of the 65B model. If the application's domain requires custom fine-tuning on proprietary visual data, LoRA on 33B offers 99% of the performance (71.8 vs. 72.0 on LLaVA-Bench) at 42% of the full-65B training cost. The beam search results in Table 1 further suggest that inference-time compute (beam=5) can provide an additional ~1 point improvement at negligible deployment cost, making the 33B+LoRA+beam5 configuration a particularly attractive cost-performance sweet spot.

On-device or edge deployment with resolution-aware model selection. The paper's finding that image resolution contributes a consistent 2–3 points across all model sizes (Table 3a: 7B improves from 63.6 to 65.9 at 336×336; 65B improves from 70.3 to 72.3) provides a practical lever for deployment scenarios where inference latency or memory constraints limit model size. If an application runs on a device that can only support a 7B or 13B model (e.g., a mobile phone, an edge server with limited GPU memory), upgrading the input resolution from 224×224 to 336×336 recovers performance equivalent to roughly half a model-scale jump (the 7B at 336×336 scores 65.9, close to the 13B at 224×224 which scores 67.1). This is a configuration change, not a model change—it requires no additional parameters, no fine-tuning, and only a modest increase in visual token count (256 → 576 tokens). For latency-sensitive applications where larger models are infeasible, higher resolution is a free performance improvement. The paper's numbers provide the quantitative basis for this decision: trading resolution for model scale yields approximately 2.5 LLaVA-Bench points of improvement at zero added model parameters.

Training pipeline design for organizations with constrained GPU memory. The paper's finding that QLoRA is the only method that avoids out-of-memory errors for the 65B model under DeepSpeed ZeRO2 establishes a hard boundary for practitioners with limited hardware. If an organization has access to a single node of 8 A100-80G GPUs and wants to fine-tune a multimodel model at the 65B scale, QLoRA is the only viable option—full fine-tuning with ZeRO3 may fit but the paper's 65B full fine-tuning used 4 nodes (32 GPUs), and standard LoRA hits OOM under ZeRO2. The QLoRA-33B result (71.6 on LLaVA-Bench, 4.68 GPU-hours) provides a baseline for what to expect: performance is 0.4 points below full fine-tuning (72.0) at roughly half the training cost (4.68 vs. 9.17 GPU-hours). While the paper does not report QLoRA performance at 65B, the 33B results suggest the gap to full fine-tuning is small, and the memory feasibility advantage is absolute. The practical decision rule is: if you have 4+ nodes, full fine-tune; if you have 1–2 nodes and need 65B scale, use QLoRA; if you have 1 node and 33B scale is acceptable, use LoRA rank 64 for the best performance-cost ratio. The paper's GPU-hour numbers and explicit Azure pricing enable direct cost estimation for each option.

Data composition for domain-specific multimodal fine-tuning. The paper's data mixing results (Tables 3a and 5) show that interleaving language-only instruction data with multimodal data improves multimodal performance by ~2 points on LLaVA-Bench (33B: 72.0 → 73.9; 65B: 72.3 → 74.2) while having inconsistent effects on language benchmarks. This finding has practical implications for organizations fine-tuning LLaVA on domain-specific visual data. If the goal is purely to maximize multimodal task performance—say, a medical image QA system—then mixing the multimodal data with general-domain text instruction data (e.g., ShareGPT-style conversations) is likely beneficial, even if the language data is not domain-specific. The mechanism appears to be improved instruction-following generalizability, not domain adaptation. If the goal is to preserve general language capabilities alongside multimodal skills, the picture is more complex: data mixing helped MMLU for 33B (56.1 → 58.6) but not for 65B (62.6 → 62.2), and degraded Vicuna-80 for both (85.3 → 80.3 for 33B; 84.5 → 82.6 for 65B). The practical recommendation based on this paper is to include text-only instruction data during multimodal fine-tuning, monitor both multimodal and language benchmark performance, and be prepared for some language capability tradeoff at the cost of multimodal gain—the ratio is approximately +2 multimodal points for −1 to −3 language points depending on model scale and benchmark.