InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model

🎯 Pitch

Specializing differently, a 7B vision-language model can match or beat GPT-4V and Gemini Pro on major benchmarks like MMBench and MME-Perception, becoming the first at this scale to break 1700. The trick is Partial LoRA (PLoRA): it updates only the image tokens, so the base LLM’s reasoning stays sharp while still achieving top-tier multimodal understanding.

1. Executive Summary

This paper introduces InternLM-XComposer2, a vision-language model based on InternLM2-7B that excels at both free-form text-image composition and multimodal comprehension. The core architectural innovation is Partial LoRA (PLoRA), a low-rank adaptation approach that applies additional LoRA parameters exclusively to image tokens while leaving language tokens untouched, thereby aligning visual features with the LLM without degrading its pre-trained linguistic capabilities. The model sets new state-of-the-art results on six of twelve benchmarks among open-source models with comparable scale—notably becoming the first 7B-parameter model to exceed 1700 on MME-Perception and achieving nearly 80% accuracy on MMBench—while matching or surpassing GPT-4V and Gemini Pro on MathVista, AI2D, MMBench, MM-Bench Chinese, SEED-Bench, and MME, establishing that a properly aligned 7B model can rival or exceed commercial APIs on specific vision-language tasks when the alignment strategy preserves the base LLM’s original reasoning strengths.

2. Context and Motivation

The Specific Gap: MLLMs Can't Both Understand and Create

The paper identifies a clear but underexplored problem in the multimodal large language model (MLLM) landscape: existing models are designed for either comprehension or composition, but not both simultaneously at a high level. The authors observe that while traditional MLLMs like LLaVA, InstructBLIP, and Qwen-VL are adept at interpreting images and engaging in text-image dialogues, they are fundamentally reactive — they respond to questions about visual inputs but lack the ability to generate rich, structured, interleaved text-image content that synthesizes visual and textual information in a flexible, user-directed manner.

This is not merely a limitation of capability breadth; it represents a fundamental architectural and training design choice that partitions the MLLM space into "understanders" and "composers." The composition side had been explored by InternLM-XComposer (the predecessor), but that work was limited to generating text-image articles based on titles alone — essentially a single-input, single-output pipeline. It lacked the sophistication to handle free-form composition instructions such as detailed textual specifications, structured outlines, reference images, or combinations thereof. This meant users could not specify nuanced requirements like writing style, structure, length, or visual material constraints — precisely the capabilities needed for practical content creation workflows.

The gap manifests as a dual capability deficit:

On the composition side: No existing model could take diverse, multi-modal inputs (outlines, specifications, reference images, style directives) and produce high-quality, interleaved text-image articles with literary quality and stylistic control. The predecessor InternLM-XComposer could generate articles from titles, but this is a narrow slice of what content creation demands.
On the comprehension side: While InternLM-XComposer achieved leading performance at its inception, the authors explicitly note it still possessed "significant potential for enhancement in detailed perception and complex reasoning capabilities" — the fine-grained visual understanding needed for tasks requiring precise perception, multi-step logical reasoning, and extensive knowledge integration.

This dual gap motivates the paper's title: "Mastering Free-form Text-Image Composition and Comprehension." The paper positions itself as solving both problems simultaneously within a single 7B-parameter model, rather than treating them as separate capabilities requiring separate systems.

Why This Matters: Practical Content Creation + Benchmark Competitiveness

The importance of this dual capability spans both practical deployment and research significance.

Practical impact: AI-assisted content creation. The ability to generate interleaved text-image content from flexible, multi-modal instructions has direct commercial applications. Consider a marketing team that needs a blog post about French pastries with specific sections, images, and tonal requirements; or an educator who wants a structured article about pandas with reference images, factual accuracy, and cultural analysis; or a travel blogger who needs a diary-style post with specific landmarks, food descriptions, and emoji-laden social media formatting. InternLM-XComposer2's demonstration figures (Figures 3–10) show exactly these use cases, generating long-form articles with embedded images, code explanations, poetry, travel diaries, and even classical Chinese essays — all from diverse input formats.

What makes this practically significant is the flexibility of input specification. Users can provide:

Titles only (as in the predecessor): e.g., "French Pastries: A Sweet Indulgence"
Titles with detailed section requirements and word counts: e.g., the ballet article requiring specific sections on history, technique, classic works, and cultural value, each with minimum word counts
Reference text materials: e.g., the Chinese ink painting article incorporates provided textual background about ink techniques, color variations, and artistic philosophy
Reference images: the model can use user-provided images rather than relying on retrieval
Stylistic and formatting directives: emoji usage, social media tone, SEO keywords, classical Chinese format (文言文), poetry structure, code documentation style

This flexibility makes the model a genuine content creation tool rather than a constrained demo. The predecessor InternLM-XComposer could only handle the first input type (title-based), making it more of a proof-of-concept than a practical tool.

Research significance: Alignment without capability loss. From a research perspective, the paper addresses a fundamental challenge in multimodal model design: how do you teach an LLM to process visual information without destroying its hard-won language capabilities? This is the modality alignment problem stated explicitly in Section 3.1:

"A desired alignment should potentially enrich the LLM with new modality-specific knowledge, while simultaneously preserving its inherent capabilities. Current methods predominantly adopt one of two approaches: they either treat the visual token and language token equally or as entirely distinct entities. We contend that the first approach overlooks the inherent property distinctions between modalities, while the second approach results in a substantial alignment cost."

The tension is real. If you fine-tune an LLM's full parameters on multimodal data, you risk catastrophic forgetting of language abilities — the model might learn to see but lose its eloquence. If you freeze the LLM entirely and only train an adapter, the alignment may be weak, producing a model that can caption images but can't deeply reason about them using the LLM's knowledge. The paper's Partial LoRA approach represents a principled middle ground that preserves linguistic integrity while enabling deep visual alignment, and the empirical results (Tables 3–4) validate this tradeoff by showing the model excels at both pure-text creative writing benchmarks (CreationBench, Table 6) and vision-language benchmarks simultaneously.

Benchmark competitiveness as validation. The paper's claim that InternLM-XComposer2 "matches or even surpasses GPT-4V and Gemini Pro in certain assessments" (Abstract, Figure 1) matters because it challenges the assumption that model scale is the primary driver of multimodal performance. A 7B model outperforming commercial APIs with presumably much larger models on MathVista (57.6 vs. GPT-4V's 49.9), AI2D (78.7 vs. 78.2), MMBench (79.6 vs. 77.0), and MME (2242.7 vs. GPT-4V's 1926.5) suggests that alignment strategy and data quality can compensate for — and in specific dimensions surpass — raw parameter count. This is practically relevant for deployment scenarios where API costs, latency, or privacy concerns make local 7B models attractive if they can match commercial performance on target tasks.

Prior Approaches and Where They Fall Short

The paper implicitly critiques several categories of prior work:

1. Pure comprehension MLLMs (the dominant paradigm). Models like LLaVA, InstructBLIP, Qwen-VL-Chat, and CogVLM represent the mainstream approach: take a pre-trained vision encoder (typically CLIP ViT), connect it to an LLM via a lightweight projector or Q-former, and fine-tune on vision-language instruction data. These models are evaluated on their ability to answer questions about images, follow visual instructions, and engage in multimodal dialogue. They have no composition capability — they cannot generate interleaved text-image articles. Their training objectives and data are entirely comprehension-oriented.

2. The first-generation composition model (InternLM-XComposer). The predecessor introduced the concept of using MLLMs for text-image composition, but with critical limitations:

Fixed input format: Only titles could be provided as input, not outlines, reference materials, style specifications, or reference images.
Retrieval-dependent image selection: Images were retrieved from a database based on text content, meaning the model had no mechanism for users to specify visual preferences or provide custom images.
Comprehension ceiling: While state-of-the-art at its time, the model's perception and reasoning capabilities left room for improvement.

The paper positions InternLM-XComposer2 as addressing all three limitations: free-form multi-modal inputs, user-provided image materials, and significantly improved comprehension benchmarks.

3. Standard modality alignment strategies. The paper identifies two common alignment approaches (Section 3.1) and explains why both are suboptimal:

Treating visual and language tokens equally (full fine-tuning or applying the same adapter to all tokens): This ignores that visual tokens and language tokens have fundamentally different properties. Visual tokens from a CLIP encoder represent continuous, high-dimensional perceptual features; language tokens represent discrete, symbolic linguistic units. Applying the same transformation to both assumes a homogeneity that doesn't exist, potentially degrading the LLM's carefully learned language representations.
Treating them as entirely distinct entities (separate encoders, separate adapters, or frozen LLM with a complex bridging module like Q-former): This increases alignment cost because the model must learn to translate between completely separate representational spaces. It can also create a bottleneck where the LLM's pre-trained knowledge isn't fully accessible to the visual processing pipeline because the modalities never share representational space.

The paper's Partial LoRA is positioned as a third way: shared base weights for both modalities (leveraging the LLM's universal processing capabilities), with modality-specific low-rank adaptations only for visual tokens. This preserves the LLM's language representations while allowing visual features to be projected into a form the LLM can effectively process.

4. Data quality and diversity limitations in prior work. The paper emphasizes that "quality and diversity of the training data are pivotal" (Section 1) and describes a meticulous data curation strategy spanning general semantic alignment, world knowledge alignment, and vision capability enhancement during pre-training (Table 1), plus multi-task training covering eight distinct task categories during SFT (Table 2). This is positioned against prior approaches that might rely on noisier, less curated datasets — the paper is making an implicit claim that data engineering is as important as architecture for achieving strong multimodal performance.

5. The hallucination problem. Section 4.1 specifically evaluates on POPE and HallusionBench (Table 5), achieving SOTA among open-source models (87.7 F1 on POPE, 60.3 on HallusionBench). This positions the model as addressing the well-documented hallucination issues in MLLMs — where models confidently describe visual content that doesn't exist — that plague systems like LLaVA and InstructBLIP. The implicit claim is that the Partial LoRA approach, by better preserving the LLM's original knowledge and reasoning, produces more faithful visual descriptions.

Positioning Relative to Existing Work

The paper positions itself at the intersection of two research trajectories:

Trajectory 1: Scaling comprehension performance. The MLLM field has been on a steady march of benchmark improvements, from BLIP-2 to InstructBLIP to LLaVA-1.5 to CogVLM, with each generation pushing numbers higher on MME, MMBench, and other standard evaluations. InternLM-XComposer2 participates in this trajectory by significantly raising the ceiling for 7B-scale models — its 1712.0 on MME-Perception is the first above 1700 for this scale, and its 79.6 on MMBench approaches levels previously associated only with much larger models or commercial APIs.

Trajectory 2: Enabling composition as a first-class capability. This is where the paper truly differentiates itself. While most MLLMs are evaluated solely on comprehension, InternLM-XComposer2 inherits and extends the composition vision from its predecessor. The key advancement is moving from "title → article" to "arbitrary multi-modal specification → customized article," which is a qualitative leap in practical utility. The paper demonstrates this through an extensive set of qualitatively diverse outputs (Figures 3–10) covering multiple languages (English and Chinese), multiple genres (blog posts, classical essays, poetry, social media posts, technical documentation, travel diaries), and multiple input formats.

The unification thesis. The paper's core positioning claim is that composition and comprehension are not competing capabilities that require tradeoffs, but can be mutually reinforcing when the alignment strategy is designed correctly. The argument is: by using Partial LoRA to preserve the LLM's linguistic capabilities, the model retains its creative writing, logical reasoning, and knowledge retrieval strengths — which directly benefit composition tasks. Simultaneously, by carefully training on diverse visual data, the model develops strong perception and understanding — which benefits comprehension tasks. The model doesn't sacrifice one for the other; it achieves state-of-the-art on both fronts within the same parameter budget.

This unification contrasts with models like CogVLM, which adds a separate visual expert module (increasing effective parameters beyond the base LLM), or models that focus exclusively on comprehension benchmarks without demonstrating creative generation capabilities. InternLM-XComposer2 claims to achieve comprehensive multimodal mastery through a principled architectural choice (PLoRA) rather than through parameter scaling or task-specific modules.

3. Technical Approach

3.1 Reader Orientation

InternLM-XComposer2 is a vision-language model system that takes arbitrary combinations of text instructions, outlines, reference materials, and images as input, and produces either interleaved text-image content (articles, blog posts, poetry, essays with embedded images) or answers to visual questions — depending on what the user requests. The system solves the problem of building a single 7B-parameter model that excels at both creative composition (generating long-form, stylistically controlled, image-embedded text from flexible multi-modal instructions) and visual comprehension (answering questions about images, reading charts, solving math problems with visual components, detecting hallucinations) without sacrificing either capability. The "shape" of the solution is: a frozen-ish large language model (InternLM2-7B) connected to a vision encoder (OpenAI ViT-Large) through a novel Partial LoRA adapter that applies learnable low-rank transformations exclusively to visual tokens, trained in two phases — pre-training for modality alignment followed by supervised fine-tuning on a carefully curated mixture of comprehension tasks and free-form composition data.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components arranged in a feedforward pipeline:

Vision Encoder (OpenAI ViT-Large-14-336): Takes a raw image as input and outputs a sequence of visual feature vectors (visual tokens). Think of it as a frozen "visual word embedder" that converts pixels into a format the language model can potentially process. During pre-training its resolution is increased to 490×490; during composition SFT it is downsampled to 224×224.
Large Language Model (InternLM2-7B-ChatSFT): The core reasoning engine. It receives a concatenated sequence of visual tokens (from the vision encoder, after Partial LoRA processing) and language tokens (from text tokenization), and autoregressively predicts the next token. It's based on InternLM2-7B with strong multilingual capabilities.
Partial LoRA (PLoRA) Module: A set of low-rank matrices ( $W_A$ and $W_B$ ) inserted into every linear layer of the LLM, but activated only for visual tokens. When a visual token passes through a linear layer, the output is $W_0 x_v + W_B W_A x_v + B_0$ (original weights + LoRA adaptation + bias). When a language token passes through, the output is simply $W_0 x_t + B_0$ (original weights + bias only). This is the key architectural innovation. During pre-training, the vision encoder and PLoRA are trained while the LLM is frozen. During SFT, everything is jointly trained but with different learning rates.
Training Data Pipeline (pre-training + SFT): A multi-source data curation system that feeds the model three types of pre-training data (general semantic alignment, world knowledge alignment, vision capability enhancement) and nine types of SFT data (captioning, general QA, science QA, chart QA, math QA, world knowledge QA, conversation, free-form composition, pure-text language data). The composition data is in-house constructed across four dimensions: varied writing styles, flexible text editing, complex instruction adherence, and customization with materials.

Information flows as follows: a user provides a prompt containing text and optionally images → images go through the vision encoder to produce visual tokens → all tokens (visual + language) are concatenated into a single sequence → the sequence passes through the LLM's transformer layers → at each linear layer, PLoRA computes a different transformation for visual tokens vs. language tokens → the LLM autoregressively generates output tokens → if the task is composition, the model identifies positions for image insertion after text generation; if the task is comprehension, the model produces a text answer.

3.3 Roadmap for the Deep Dive

First, Partial LoRA (PLoRA): because it's the architectural centerpiece that enables everything else — understanding why it works and how it differs from standard LoRA is prerequisite to understanding the training strategy.
Second, the pre-training phase: because this is where the vision encoder and PLoRA learn to align visual features with the frozen LLM, establishing the fundamental visual understanding capabilities before any task-specific training.
Third, the supervised fine-tuning phase: because this builds on pre-trained alignment to teach the model specific skills — both comprehension tasks and the critical free-form composition capability that distinguishes this work.
Fourth, the free-form text-image composition data and methodology: because this is the novel capability that separates InternLM-XComposer2 from prior MLLMs and from its predecessor, and understanding the data construction explains how the model achieves such diverse and controllable outputs.
Fifth, the image insertion mechanism and resolution strategy: because the practical details of how images get placed into generated text and why different resolutions are used for different phases reveal important design tradeoffs.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and methods paper whose core idea is that applying low-rank adaptations exclusively to visual tokens, while keeping the LLM's original weights fully active for language tokens, enables a 7B model to simultaneously achieve state-of-the-art visual comprehension and free-form text-image composition without the catastrophic forgetting or weak alignment that plague alternative modality fusion strategies.

Partial LoRA (PLoRA): Modality-Specific Low-Rank Adaptation

What PLoRA is and why it's needed. The fundamental challenge in building multimodal LLMs is that visual tokens (continuous feature vectors from a CLIP encoder) and language tokens (discrete token embeddings) occupy fundamentally different representational spaces, yet they must be processed by the same transformer layers. If you fully fine-tune the LLM on multimodal data, the model's language representations shift to accommodate visual features — this can cause catastrophic forgetting of linguistic capabilities. If you freeze the LLM entirely and only train a lightweight connector (e.g., a single linear projection or a Q-former), the alignment is shallow — the LLM can superficially describe images but can't deeply integrate visual information with its pre-trained knowledge, because the representational mismatch persists at every transformer layer.

PLoRA addresses this by applying adaptation at every linear layer of the LLM, but only to visual tokens. The intuition, stated by the authors in Section 3.1, is:

"We contend that the first approach [treating visual and language tokens equally] overlooks the inherent property distinctions between modalities, while the second approach [treating them as entirely distinct entities] results in a substantial alignment cost."

PLoRA is the middle ground: the LLM's base weights ( $W_0$ ) are shared across modalities, providing a common computational substrate. The low-rank adaptations ( $W_B W_A$ ) are applied only to visual tokens, providing modality-specific transformations that bridge the representational gap without disturbing the language processing pathway.

Formal definition. For each linear layer in the LLM decoder blocks, let $W_0 \in \mathbb{R}^{C_{out} \times C_{in}}$ be the pre-trained weight matrix and $B_0 \in \mathbb{R}^{C_{out}}$ be the bias, where $C_{in}$ and $C_{out}$ are the input and output dimensions of that layer. PLoRA introduces two low-rank matrices: $W_A \in \mathbb{R}^{C_r \times C_{in}}$ and $W_B \in \mathbb{R}^{C_{out} \times C_r}$ , where $C_r$ is the rank (set to 256 for all linear layers). Given an input sequence $x = [x_v, x_t]$ where $x_v$ are visual tokens and $x_t$ are language tokens, the output $\hat{x}$ is:

$\hat{x}_t = W_0 x_t + B_0$

$\hat{x}_v = W_0 x_v + W_B W_A x_v + B_0$

$\hat{x} = [\hat{x}_v, \hat{x}_t]$

where $\hat{x}_t$ is the output for language tokens, $\hat{x}_v$ is the output for visual tokens, and $\hat{x}$ is the concatenated full output. The crucial detail: the LoRA term $W_B W_A x_v$ appears only in the visual token pathway. Language tokens are processed with pure pre-trained weights.

What it computes: For each linear transformation in the transformer, visual tokens receive a modified projection that is the original pre-trained projection ( $W_0 x_v + B_0$ ) plus a learned low-rank correction ( $W_B W_A x_v$ ). The correction is the product of two matrices: $W_A$ first projects the input from $\mathbb{R}^{C_{in}}$ down to $\mathbb{R}^{C_r}$ (a compressed representation), then $W_B$ projects from $\mathbb{R}^{C_r}$ back up to $\mathbb{R}^{C_{out}}$ . Since $C_r \ll C_{in}, C_{out}$ , this correction adds only $C_r \times (C_{in} + C_{out})$ additional parameters per layer, which is a small fraction of the original $C_{in} \times C_{out}$ .

Why this form: The low-rank structure ( $W_B W_A$ ) constrains the adaptation to lie in a low-dimensional subspace of the full parameter space. This acts as a regularizer — it prevents the visual adaptation from overfitting to the pre-training visual data or from dramatically altering the LLM's computational properties. The key property is separability with shared computation: both modalities share the expensive $W_0$ matrix multiply (the LLM's pre-trained knowledge), but visual tokens get an additional cheap correction. This means:

No catastrophic forgetting: The language pathway ( $W_0 x_t + B_0$ ) is mathematically identical to the original LLM — the pre-trained weights and biases are preserved exactly during pre-training (when the LLM is frozen) and receive only gentle updates during SFT (with a learning rate scale factor of 0.2).
Deep alignment: Because the correction is applied at every layer (not just at the input), visual features are progressively transformed to align with the LLM's internal representations at multiple levels of abstraction — from low-level features in early layers to high-level semantic features in later layers.
Parameter efficiency: The rank-256 PLoRA adds only a small fraction of the LLM's total parameters (the paper doesn't specify exact count, but for a 7B model with typical hidden dimensions around 4096, rank-256 LoRA on all linear layers adds roughly 2–3% additional parameters per LoRA module, shared across modalities only for the $W_A, W_B$ matrices).
Gradient isolation: During pre-training, the LLM is frozen (no gradients flow through $W_0$ ), so the PLoRA matrices learn exclusively from the visual alignment signal without interference from language-objective gradients.

Configuration specifics. The paper states: "For the Partial LoRA, we set a rank of 256 for all the linear layers in the LLM decoder block." This means every linear projection in every transformer decoder layer — query, key, value, output projections in self-attention; the two linear layers in each feed-forward network; and any layer-norm-adjacent projections — gets a pair of $W_A, W_B$ matrices of rank 256. The choice of rank 256 is relatively high for LoRA (common ranks are 8–64), suggesting the authors believe that aligning visual features requires a richer adaptation subspace than typical language-task fine-tuning.

Pre-Training Phase: Modality Alignment with Frozen LLM

Objective. The pre-training phase teaches the vision encoder and PLoRA to produce visual token representations that the frozen LLM can effectively process. The LLM parameters are not updated during this phase — only the vision encoder and the PLoRA matrices are trained. This is explicitly stated: "During the pre-training phase, the LLM remains constant while both the vision encoder and Partial LoRA are fine-tuned to align the visual tokens with the LLM."

Why freeze the LLM? The authors provide a specific, measurable justification: "The fixed LLM also enables us to directly use in-context learning performance as a measure of pre-training quality." Since the LLM's language capabilities are unchanged, any improvement in the model's ability to answer questions about images (using in-context examples) can be attributed to better visual alignment, not to the LLM memorizing answers. Additionally, freezing the LLM during pre-training prevents the visual alignment signal from distorting the pre-trained language representations — a form of catastrophic forgetting prevention at the earliest stage.

Three data objectives. The pre-training data is organized around three progressively more sophisticated goals, with data sources listed in Table 1:

Objective 1: General Semantic Alignment. The goal is basic object recognition and scene understanding — the MLLM should learn that a particular pattern of visual features corresponds to "a cat" or "a beach." The training data consists of image-caption pairs from ShareGPT4V-PT (high-quality detailed captions), COCO, NoCaps, TextCaps, LAION-400M, SBU, and CC-3M. The authors note that ShareGPT4V-PT provides "high-quality, detailed captions" while COCO provides "concise and precise captions" — the mixture ensures the model learns both broad scene understanding and fine-grained detail. The instruction template is simple: "Describe this image briefly/in detail."

Objective 2: World Knowledge Alignment. The goal is to connect visual recognition with the LLM's pre-trained factual knowledge. For instance, recognizing an image of Albert Einstein should trigger the LLM's stored knowledge about relativity, not just surface-level features like "a man with white hair." The authors construct a "concept dataset" filtered from InternLM-XComposer's concept data. Critically, they note that "the text in the concept data only partially describes the content in the image and their relationship is complex to model," so they use a deliberately broad instruction: "Tell me something about this image." This loose coupling between image and text means the model learns to retrieve relevant knowledge even when the image-text association is indirect, rather than memorizing strict image→fact mappings.

Objective 3: Vision Capability Enhancement. The goal is to teach vision-specific skills that go beyond object recognition: Optical Character Recognition (reading text in images), object localization (identifying where objects are), and understanding structured images (charts, tables, diagrams). Datasets include WanJuan, Flickr, and MMC-Instruction. The paper states that "corresponding instructions" are implemented for each dataset, implying task-specific prompting (e.g., for OCR: "Read the text in this image").

Training hyperparameters. The pre-training configuration is specified in detail:

Vision encoder: OpenAI CLIP ViT-L-14-336, with resolution increased to 490×490. The standard CLIP ViT-L uses 336×336 input; the higher resolution provides finer visual detail, particularly beneficial for OCR and small object recognition.
Batch size: 4906. This is a relatively large batch size, consistent with the need to process many image-text pairs for stable contrastive-style alignment.
Epochs: 2. The limited number of epochs prevents overfitting to the pre-training distribution, which is important since the model will later be fine-tuned on different data.
Learning rate schedule: Increases to a maximum of $2 \times 10^{-4}$ over the first 1% of training steps (warmup), then decays to 0 following a cosine schedule. This is a standard transformer training recipe; the warmup prevents early training instability from large gradients on randomly initialized PLoRA weights.
Layer-wise learning rate decay (LLDR) for vision encoder: 0.90. This means the learning rate for the $k$ -th layer from the top is multiplied by $0.90^k$ . The effect: early layers (which encode low-level features like edges and textures) change very little, preserving the pre-trained CLIP representations; later layers (closer to the output) change more, adapting high-level visual features to the LLM's expected input distribution. The authors explicitly state this is "to preserve the pre-existing knowledge of the vision encoder."

Design choice: why 490×490 resolution? The paper doesn't explicitly justify this specific number, but the logic follows from the need for OCR and detailed perception. Standard CLIP ViT-L at 336×336 may lose fine-grained text and structural details in complex images (charts, diagrams, crowded scenes). Increasing to 490×490 provides roughly $(490/336)^2 \approx 2.1\times$ more pixels, significantly improving resolution for small visual elements. The choice of a non-standard resolution also requires the vision encoder to handle positional embeddings for a larger grid than it was originally trained on; the paper's use of CLIP ViT-L (which uses learned positional embeddings that can be interpolated) makes this feasible.

Supervised Fine-Tuning Phase: Teaching Task Capabilities

Overview. After pre-training establishes basic visual alignment, the SFT phase teaches the model to perform specific tasks by training on curated instruction-response pairs. Unlike pre-training, all components are trainable: the vision encoder, the LLM, and the PLoRA matrices. However, different components receive different learning rate treatments to balance capability acquisition against knowledge preservation.

Two sequential steps. The SFT phase consists of two sub-stages:

Step 1: Multi-task Training. The model is trained on a diverse mixture of vision-language tasks designed to build broad competence. Table 2 lists the task categories and datasets:

Caption: ShareGPT4V, COCO, NoCaps — teaches the model to describe images.
General QA: VQAv2, GQA, OK-VQA — teaches answer generation for visual questions.
Science QA: AI2D, SQA — teaches scientific diagram understanding and reasoning.
Chart QA: DVQA, ChartQA — teaches chart and data visualization reading.
Math QA: MathQA, Geometry3K — teaches mathematical problem solving with visual components.
World Knowledge QA: A-OKVQA, KVQA — teaches knowledge-grounded visual reasoning.
Conversation: LLaVA-150k, LVIS-Instruct4V — teaches multi-turn dialogue about images.

Each task is formatted as a conversational interaction, and the instructions are augmented with GPT-4 "to enhance diversity." This means the same underlying question might be phrased multiple ways (e.g., "What is in this image?" and "Can you describe what you see?" and "Tell me about this picture"), preventing the model from overfitting to specific instruction templates.

Step 2: Free-form Text-Image Composition. This step uses the in-house composition data (detailed in Section 3.4 of the paper, and below in this analysis) along with conversation data from LLaVA-150k, LVIS-Instruct4V, ShareGPT (English and Chinese), and InternLM-Chat. The composition data teaches the model to generate interleaved text-image articles from diverse specifications; the conversation data maintains dialogue capabilities.

Training configuration:

Batch size: 2048. Smaller than pre-training (4906), which is typical for fine-tuning where per-sample diversity matters more than throughput.
Training steps: 3000.
Maximum learning rate: $5 \times 10^{-5}$ — 4× lower than the pre-training maximum ( $2 \times 10^{-4}$ ), reflecting the finer-grained nature of task learning compared to initial alignment.
Data sampling: Weighted sampling based on the number of data points from each source. This prevents large datasets from dominating training while ensuring small but important datasets (e.g., specialized math data) get sufficient exposure.

Component-specific learning strategies (the critical design):

Vision encoder: LLDR of 0.9, matching the pre-training strategy. This continues to protect low-level visual features while allowing high-level adaptation.
LLM: Learning rate scale factor of 0.2. This means the effective learning rate for the LLM parameters is $0.2 \times 5 \times 10^{-5} = 1 \times 10^{-5}$ . The authors explain: "This slows down the update of the LLM, achieving a balance between preserving its original capabilities and aligning it with vision knowledge." This is crucial — updating the LLM too aggressively would overwrite pre-trained language knowledge with visual-task-specific patterns. The 0.2 factor ensures the LLM drifts slowly, integrating visual understanding without losing linguistic fluency.
PLoRA: The paper doesn't specify a separate learning rate for PLoRA during SFT, implying it uses the base maximum learning rate. This makes sense: PLoRA matrices are the primary conduit for visual adaptation, so they should be the most plastic component.

Language preservation mechanism. The paper explicitly states that pure-text SFT data from InternLM2 constitutes "a fixed 10% of the total Supervised Fine-Tuning data." This is a deliberate design choice: by mixing in text-only examples, the model is forced to maintain its language capabilities even as it learns visual tasks. Without this, the model might gradually lose its ability to handle pure-text instructions — a form of modality bias where the model assumes every input should have an image component.

Free-Form Text-Image Composition: Data Construction and Generation Methodology

What "free-form" means. The paper defines free-form text-image composition as "the combination of textual content and visual elements in a flexible and unrestrictive manner." Unlike the predecessor InternLM-XComposer, which generated articles from titles alone, InternLM-XComposer2 accepts diverse inputs: "a title, outline, and writing material, and optionally, any visual requirements like image resources." This means users can specify:

Structural constraints: section headings, word counts, required topics
Stylistic constraints: formal academic prose, social media style with emojis, classical Chinese (文言文), poetic formats
Content materials: reference text to incorporate, factual background to include
Visual specifications: reference images provided by the user (instead of system-retrieved images)

Four-dimensional data construction. The in-house composition dataset is built along four axes, with approximately equal distribution (ratio "approximately 1:1:1:1"):

Dimension 1: Varied Writing Styles. The data spans "a multitude of writing styles, from academic papers to social media posts and poems." This means training examples include: formal academic articles with structured sections and citations; casual blog posts with conversational tone; marketing content with persuasive language; social media posts with emojis, hashtags, and informal grammar; poetry with rhythmic constraints; classical Chinese essays with literary conventions; technical documentation with code blocks and structured explanations. The diversity ensures the model doesn't default to a single "neutral" writing style but can adapt its output register to match the user's implicit or explicit stylistic requirements.

Dimension 2: Flexible Text Editing. The dataset includes "extensive examples of text editing, encompassing a wide spectrum of modifications such as shortening, expanding, and rewriting." This likely means training examples where the input is: an existing piece of text plus an editing instruction (e.g., "Make this more concise," "Expand the section about habitat," "Rewrite this in a more formal tone"), and the output is the edited version. This capability is visible in the qualitative results — Figure 8 shows the model generating content that incorporates specific structural requirements (specific sections about equipment, warm-up, techniques), suggesting it can follow detailed editing and structuring instructions.

Dimension 3: Complex Instruction Adherence. The dataset captures "instances of adhering to complex instructions to create content that caters to diverse demands like titles and outlines, encompassing both text and image-based compositions." The key word is "complex" — these are not simple "write about X" prompts but multi-part specifications. For example, the ballet article instruction (Figure 7) specifies: a title, five required content sections (history, development process, technique, classic works, cultural value), a minimum word count (800+), and specific topics to cover within each section. The model's output follows this structure precisely, suggesting the training data includes many examples of multi-constraint instruction following.

Dimension 4: Customization with Materials. The dataset includes "materials used for personalized content creation, covering both text and images, enabling customizable and unique content creation experiences." This means training examples where the user provides: (a) reference images to incorporate into the article, (b) textual background material to integrate (as in the Chinese ink painting example where reference text about ink techniques is provided), or (c) both. This is a key differentiator from the predecessor, which relied entirely on image retrieval — the model learns to incorporate user-provided visual materials rather than selecting from a database.

Image insertion after text generation. The paper states: "Our method follows previous work [95] to identify suitable positions for image insertion after generating the text content." This means the model first generates the full text article, then — presumably through a post-processing step or learned insertion markers — identifies where images should appear. This is a two-stage process: text generation (handled by the LLM's autoregressive decoding) followed by image placement (which may use heuristics about semantic breaks, section boundaries, or learned insertion tokens).

User-provided vs. retrieved images. A notable distinction from the predecessor: "when users provide their own image materials, these image materials are used for insertion instead of relying on retrieved images [95]." This means the model has two modes: with user-provided images, it directly embeds them at appropriate positions; without user-provided images, it can either retrieve relevant images from a database or (as shown in many qualitative examples) generate text without embedded images. The paper's figures show both cases — Figures 3, 5, 9, and 10 include embedded images; Figures 4, 6, 7, and 8 are primarily text with image placeholders or user-provided visual elements.

Resolution tradeoff for composition. During the composition SFT stage, image input resolution is down-sampled to 224×224 (compared to 490×490 during pre-training). The paper explains: "We also observe that having a high-resolution image input is not essential for text-image composition." This makes intuitive sense: for composition tasks, the model needs to understand the general content and style of reference images (to describe them in text and place them appropriately) but doesn't need to read fine text within images or perform detailed visual reasoning. The lower resolution reduces computational cost and memory usage, which matters when generating long sequences (some qualitative outputs exceed 800 Chinese characters or multiple paragraphs of English).

Image Insertion Mechanism and Resolution Strategy

The insertion pipeline. While the paper doesn't provide exhaustive details about the image insertion mechanism, the described approach ("identify suitable positions for image insertion after generating the text content") combined with the qualitative outputs suggests the following:

The user optionally provides reference images as part of the input.
The model generates text autoregressively, potentially including special tokens or markers indicating "image should go here."
After generation, a post-processing step places the appropriate images at marked positions.
If no user images are provided, the system either retrieves images (using the predecessor's retrieval approach) or generates text-only output.

This is visible in the figures: Figure 3 (French Pastries) includes images of croissants, macarons, and other pastries interspersed with text; Figure 5 (Nature Poetry) includes a landscape image; Figure 9 (Swimming Tutorial) includes a swimming illustration. In each case, images appear at natural breaks in the text — after section headings, at the start of articles, or alongside relevant descriptions.

Why different resolutions for different phases. The paper's resolution strategy is:

Phase	Vision Encoder Input Resolution
Pre-training	`490×490`
SFT (multi-task + composition)	`224×224`

The high resolution during pre-training supports learning fine-grained visual features (OCR, small object recognition, chart reading) that establish strong visual grounding. The lower resolution during SFT is sufficient for composition tasks (understanding image content at a semantic level, placing images, describing visual elements) while reducing computational overhead. This two-tier approach means the vision encoder must handle variable input resolutions — the CLIP ViT-L architecture supports this because its patch embedding layer can process different grid sizes, though positional embeddings may need interpolation for non-standard resolutions like 490×490.

Why the LLM is InternLM2-7B-ChatSFT specifically. The paper uses the "ChatSFT" variant of InternLM2-7B, meaning the model has already undergone supervised fine-tuning for chat/dialogue on pure text data. This is important because: (1) the model already understands conversational formats and instruction following, providing a strong starting point for the multi-task SFT phase; (2) the model has strong multilingual capabilities (explicitly noted: "This model boasts exceptional multi-lingual capabilities"), supporting the bilingual outputs shown in Figures 4, 6, 7, 8 (Chinese) and Figures 3, 5, 9, 10 (English).

Summary of Design Choices and Their Justifications

PLoRA applied to all linear layers with rank 256 rather than just attention layers or input projections: ensures deep alignment at every transformer layer, allowing visual features to be transformed at multiple levels of abstraction. High rank (256) provides sufficient capacity for the complex visual→language mapping.
Frozen LLM during pre-training rather than joint training: prevents catastrophic forgetting and enables using in-context learning as a clean evaluation metric for alignment quality.
Three-tier pre-training data (semantic → knowledge → vision capabilities) rather than uniform training on all data: builds capabilities incrementally, establishing basic recognition before adding knowledge grounding and specialized skills.
LLDR of 0.90 on vision encoder rather than uniform learning rates: preserves low-level CLIP features that generalize well while adapting high-level features that need to match the LLM's expectations.
LLM learning rate scale factor of 0.2 during SFT rather than equal-rate joint training: allows gradual integration of visual capabilities without disrupting pre-trained language knowledge.
10% pure-text data in SFT mixture rather than 100% multimodal data: maintains language capabilities and prevents modality bias.
1:1:1:1 ratio across composition data dimensions rather than emphasizing any single capability: ensures balanced development of writing style, editing, instruction following, and material customization.
224×224 resolution for composition rather than keeping 490×490: reduces computational cost for the long-sequence generation typical of composition tasks, justified by the observation that fine visual detail is less critical for composition than for comprehension.
Image insertion after text generation rather than interleaved generation: simplifies the autoregressive decoding (avoiding the complexity of generating images and text jointly) while still producing coherent interleaved output through post-processing.

4. Key Insights and Innovations

Innovation 1: Reframing Modality Alignment as "Preserve and Adapt" Rather Than "Replace or Bridge"

The dominant framing in MLLM architecture design prior to this work oscillated between two poles: either treat visual tokens as just another language the LLM should learn (full fine-tuning, or applying identical adapters to all tokens), or treat vision as an alien modality requiring a separate translation layer (frozen LLMs with Q-former-style bridges, as in BLIP-2, or visual expert modules grafted onto the side, as in CogVLM). Both framings implicitly accept a zero-sum tradeoff: stronger visual alignment comes at the cost of linguistic degradation, or linguistic preservation comes at the cost of shallow visual understanding.

InternLM-XComposer2's Partial LoRA represents a conceptual reframing: modality alignment is about selective enhancement, not wholesale transformation or separation. The insight is that an LLM's pre-trained weights already constitute a powerful general-purpose reasoning substrate — the problem is not that vision needs a fundamentally different computational architecture, but rather that visual tokens arrive in a representational format the LLM was never trained to process. The solution is to apply learned corrections only where the modality mismatch exists (visual tokens), while leaving the language processing pipeline mathematically identical to the pre-trained model.

This is more than an architectural trick; it is a diagnosis of what actually goes wrong when you fully fine-tune an LLM on multimodal data. The degradation of language capabilities isn't because the LLM "forgets" — it's because the gradient updates from visual data perturb weight matrices that were jointly optimized for language. PLoRA solves this by isolating the perturbation to a low-rank subspace and restricting its application to the modality that needs it. The pre-trained forward pass for language tokens is preserved exactly (not approximately) during pre-training, because the LLM is frozen and the LoRA branch is never activated for x_t.

This framing is significant because it suggests a general principle for multimodal model design: don't retrain what already works; adapt only what's new. The implication extends beyond vision-language models to any scenario where a pre-trained model needs to handle a new input modality (audio, video, sensor data, code). The PLoRA approach provides a template: identify the pre-trained components that should remain invariant, and apply low-rank, modality-gated adaptations everywhere else.

Evidence for the effectiveness of this reframing is not just in the benchmark numbers (Tables 3–4), but in the simultaneous strength on CreationBench (Table 6, where the model scores 6.24 without GPT-4 reference, outperforming Yi-34B-Chat and Qwen-72B-Chat) and on vision-language benchmarks. A model that sacrificed language quality for visual alignment would not outperform much larger pure-text LLMs on a creative writing benchmark. The dual excellence — SOTA on 10 of 11 open-source MLLM benchmarks while beating 34B and 72B pure-text models at creative writing — validates the "preserve and adapt" thesis.

Innovation 2: Establishing "Composition" as a First-Class MLLM Capability Alongside Comprehension

Prior to this work, the MLLM field operated with an implicit hierarchy: comprehension was the primary capability, evaluated through an expanding suite of benchmarks (MME, MMBench, SEED-Bench, etc.), while generation — when addressed at all — meant short-form visual question answering or captioning. The predecessor InternLM-XComposer introduced text-image composition as a concept, but it was a narrow capability: title-in, article-out, with retrieved images.

InternLM-XComposer2's contribution is upgrading composition from a demo capability to a general-purpose, instruction-following skill on par with comprehension. The paper doesn't just add composition as a feature; it articulates a specific taxonomy of what free-form composition requires — varied writing styles, flexible text editing, complex instruction adherence, and customization with materials — and constructs training data that covers these dimensions systematically. This taxonomy is itself a conceptual contribution: it defines what it means for a model to be a general-purpose content creator rather than a template-filling generator.

The significance goes beyond the specific model. By demonstrating that a single 7B model can simultaneously achieve state-of-the-art comprehension and produce diverse, controllable, interleaved text-image content across multiple languages and genres, the paper argues — implicitly but forcefully — that composition and comprehension are not competing objectives that require architectural tradeoffs. They are complementary capabilities that draw on the same underlying strengths (language fluency, visual understanding, instruction following) and can be jointly optimized when the alignment strategy preserves the base model's full capabilities.

This has implications for how the field evaluates MLLMs. If composition is a first-class capability, then benchmarks that only measure comprehension (which is nearly all of them) provide an incomplete picture. A model might ace MMBench while being useless for content creation, or vice versa. The paper's inclusion of CreationBench (Table 6) alongside standard vision-language benchmarks points toward a more holistic evaluation framework where generative quality and instruction-following in creative tasks are measured alongside perceptual accuracy.

The qualitative results (Figures 3–10) are not decorative — they are the primary evidence for this innovation. The model generates classical Chinese essays (Figure 4's ink painting article), emoji-laden social media posts (Figure 8), poetry in Tagore's style (Figure 5), technical tutorials with code (Figure 11), and structured multi-section articles with word-count constraints (Figures 6, 7). The diversity of formats, languages, and input specifications demonstrates that composition is not a narrow, brittle capability but a robust, generalizable skill.

Innovation 3: Diagnosing and Addressing the LLM-Vision Alignment Problem Through Gradient Isolation

The paper's PLoRA design embodies a deeper insight about why multimodal fine-tuning degrades language models: the problem is gradient interference, not capacity limitation. When visual and language objectives backpropagate through the same weight matrices, the gradients from one task can point in directions that are orthogonal or even opposed to the gradients from the other. This isn't "forgetting" in the sense of overwriting memories — it's optimization conflict where the joint loss landscape has no minimum that satisfies both objectives well.

PLoRA solves this through a specific mechanism of gradient isolation during pre-training and gradient damping during fine-tuning. During pre-training, the LLM is frozen — zero gradient flows through W_0, so the language pathway is mathematically untouched. The PLoRA matrices W_A and W_B receive gradients only from visual tokens, learning to correct the visual representations without any language-objective interference. During SFT, the LLM is unfrozen but with a 0.2× learning rate scale factor, meaning the effective step size for LLM parameters is small enough that language capabilities drift gradually rather than being overwritten. Meanwhile, the 10% pure-text data in the SFT mixture provides a direct language-preservation signal.

This is a more nuanced solution than simply freezing the LLM entirely (which limits visual alignment depth) or applying the same LoRA to all tokens (which allows language drift through the shared low-rank pathway). The innovation is the recognition that alignment requires different degrees of plasticity for different modalities, and that the degree of plasticity should be controlled through both architectural gating (PLoRA) and optimization hyperparameters (learning rate scaling, data mixing).

The connection to the hallucination results (Table 5) is telling. MLLMs are notorious for hallucinating — describing visual content that doesn't exist, or making factual claims unsupported by the image. The paper achieves 87.7 F1 on POPE and 60.3 on HallusionBench, SOTA among open-source models. One interpretation: hallucinations arise partly from the LLM's language model prior overwhelming the visual signal when the alignment is weak. If the LLM's language representations have been distorted by aggressive fine-tuning, the model may default to plausible-sounding but visually unfaithful completions. PLoRA's preservation of language integrity may actually improve visual faithfulness, because the model's language prior is well-calibrated and the visual signal is cleanly integrated through the dedicated adaptation pathway.

Innovation 4: Demonstrating That Data Engineering Is an Architectural Design Choice, Not an Afterthought

Most MLLM papers treat training data as a necessary but unglamorous implementation detail — list the datasets, state the mixture, move on. InternLM-XComposer2 treats data construction as a first-order architectural decision with the same importance as model architecture. The paper's data strategy has a clear conceptual structure:

Pre-training data is organized around a progression of cognitive capabilities (semantic → knowledge → specialized vision skills), each with its own objective and instruction template.
SFT data is organized into eight task categories plus composition, with explicit weighting to prevent large datasets from dominating.
Composition data is systematically varied across four orthogonal dimensions (style, editing, instruction complexity, material customization), with balanced distribution, ensuring the model's generation capabilities are broad rather than narrow.
Language preservation is engineered through a fixed 10% pure-text data allocation, not left to chance.

This is a conceptual contribution because it argues — through its meticulous design — that data diversity and structure are as important as model architecture for achieving general-purpose multimodal capabilities. The field's focus on novel architectures (Q-former, perceiver resamplers, visual experts) may be misallocated if the key bottleneck is actually data quality and coverage. InternLM-XComposer2 achieves SOTA results with a relatively simple architectural innovation (PLoRA) combined with a sophisticated data strategy, suggesting that the returns to data engineering may be higher than the returns to architectural complexity at current scales.

The evidence for this is the breadth of the model's capabilities. It's not just that the model scores well on benchmarks — it produces classical Chinese essays, Tagore-style poetry, code explanations, and travel diaries with equal fluency. This breadth cannot come from architecture alone; it requires training data that covers the target distribution of tasks. The fact that the composition data is in-house and constructed specifically for this purpose (rather than scraped from the web) means the authors had precise control over the capability distribution, which likely explains the model's consistent performance across such diverse generation tasks.

A subtle but important point: the paper's use of GPT-4 to augment instructions "to enhance diversity" during multi-task training is a form of data engineering that leverages strong models to improve weaker ones — a technique that is increasingly common but rarely discussed as a methodological choice. The paper is implicitly arguing that using commercial APIs to diversify training data is a legitimate and effective MLLM development strategy, not a form of "cheating" or knowledge distillation that invalidates the comparison.

5. Experimental Analysis

Evaluation Methodology

Dataset. Thirteen benchmarks are used: MathVista, MMMU, AI2D, MME, MMBench, MMBench-Chinese, SEED-Bench (Image), LLaVA-Bench (In-the-Wild), QBench, MM-Vet, HallusionBench, ChartQA, and POPE. Each serves a distinct evaluation dimension — math reasoning with visual contexts (MathVista), college-level multi-disciplinary understanding (MMMU), diagram understanding (AI2D), perception and cognition (MME), general multimodal ability (MMBench and its Chinese variant), generative comprehension (SEED-Bench), real-world instruction following (LLaVA-Bench), low-level vision evaluation (QBench), integrated capability assessment (MM-Vet), hallucination detection (HallusionBench, POPE), and chart reading (ChartQA). The benchmarks include both open-source evaluation suites and closed-source API comparisons.
Base model(s). The model is InternLM-XComposer2, built on InternLM2-7B-ChatSFT with OpenAI CLIP ViT-L-14-336 as the vision encoder. The 7B parameter scale is explicitly chosen to demonstrate that a moderately sized model can compete with much larger systems when the alignment strategy is sound. The paper also reports the predecessor InternLM-XComposer (based on InternLM-7B) as a direct lineage comparison.
Metrics. Each benchmark uses its own standard metric: MathVista, MMMU, AI2D, MMBench, MMBench-Chinese, SEED-Bench, MM-Vet, HallusionBench, ChartQA, and POPE report accuracy (%), with POPE specifically using F1-score averaged across three tracks. MME reports separate Perception and Cognition scores (sum of sub-scores, with maximum 2000 for Perception). LLaVA-Bench and QBench report scores on their respective scales. CreationBench reports four sub-metrics — Creativity (C), Richness (R), User Demand Fulfillment (UDF), and Logical Coherence (LC) — each on a 10-point scale evaluated by GPT-4, plus an average score. The paper reports CreationBench results both with and without a GPT-4 generated reference answer.
Baselines. The paper compares against three categories: (1) Closed-source APIs: GPT-4V, Gemini Pro, and QwenVL-Plus. These are presumably much larger models with undisclosed parameter counts, serving as an upper bound on commercial system performance. (2) Open-source MLLMs at comparable scale: BLIP-2 (FLAN-T5), InstructBLIP (Vicuna-7B), IDEFICS-80B (LLaMA-65B), Qwen-VL-Chat (Qwen-7B), LLaVA (Vicuna-7B), LLaVA-1.5 (Vicuna-13B), ShareGPT4V (Vicuna-7B), CogVLM-17B (Vicuna-7B), LLaVA-XTuner (InternLM2-20B), Monkey-10B (Qwen-7B), and InternLM-XComposer (InternLM-7B). This set spans model sizes from 7B to 80B, enabling both scale-matched and scale-defying comparisons. (3) Pure-text LLMs on CreationBench: GPT-4, Qwen-72B-Chat, and Yi-34B-Chat — these are substantially larger text-only models tested on creative writing, providing a reference for how well the 7B multimodal model preserves language capabilities.
Generation budget / compute accounting. The paper does not use test-time compute scaling (no multiple samples, no search, no revision). All results in Table 3 and Table 4 are from a single generation per prompt. This makes the comparison to methods like best-of-N or beam search (common in other MLLM papers that report self-consistency gains) a conservative one — the reported numbers represent the model's raw single-pass capability. For CreationBench, the paper does not specify generation parameters, but the qualitative outputs show long-form generation without explicit mention of sampling strategies, suggesting greedy or low-temperature decoding.
Cross-validation / statistical protocol. The paper does not report any cross-validation, error bars, confidence intervals, or statistical significance tests. All results are single-point estimates. The test sets for each benchmark have fixed standard splits, and the paper uses these as-is. For CreationBench, the evaluation is performed by GPT-4 as judge, and the paper reports results both with and without a GPT-4 reference answer, which provides a limited form of robustness check against evaluation prompt design. The lack of statistical rigor is a notable limitation — with benchmarks like MME where the score gaps between top models can be less than 100 points (on a ~2000-point scale), it's unclear whether the observed differences are statistically reliable.

Main Quantitative Results

Comparison with Closed-Source APIs

The headline result from Table 3: InternLM-XComposer2 (7B parameters) outperforms GPT-4V and Gemini Pro on six of twelve benchmarks while being competitive on the remainder.

The specific numbers tell a more nuanced story:

Where InternLM-XComposer2 wins decisively:

MathVista: 57.6 vs. GPT-4V 49.9 and Gemini Pro 45.2. This is a +7.7 point absolute margin over GPT-4V — a 15.4% relative improvement. MathVista tests mathematical reasoning with visual contexts, and this result suggests the base InternLM2-7B LLM has unusually strong mathematical capabilities that survive the multimodal alignment process. This is the single largest gap in favor of InternLM-XComposer2 against commercial APIs.
MME: 2242.7 vs. GPT-4V 1926.5 and Gemini Pro 1933.3. The MME score combines Perception and Cognition sub-scores. The paper explicitly highlights becoming "the first model to achieve a score exceeding 1700 on the MME-Perception benchmark" among open-source models (the separate Perception and Cognition scores in Table 4 show 1712.0 and 530.7 respectively). The 2242.7 total is +316.2 above GPT-4V — a 16.4% relative improvement.
MMBench: 79.6 vs. GPT-4V 77.0 and Gemini Pro 73.6.
MMBench-Chinese: 77.6 vs. GPT-4V 74.4 and Gemini Pro 74.3. The strong performance on the Chinese variant reflects InternLM2's "exceptional multi-lingual capabilities" noted in Section 3.1. GPT-4V and Gemini Pro, despite being multilingual, show a noticeable drop on the Chinese benchmark relative to English (GPT-4V: 77.0 → 74.4, a -2.6 point drop), while InternLM-XComposer2 shows a smaller gap (79.6 → 77.6, a -2.0 point drop), suggesting more balanced bilingual performance.
SEED-Bench (Image): 75.9 vs. GPT-4V 69.1 and Gemini Pro 70.7. This +6.8 point gap over GPT-4V is substantial and notable because SEED-Bench tests generative comprehension — the model must produce answers, not just select from options.
AI2D: 78.7 vs. GPT-4V 78.2 and Gemini Pro 73.9. A narrow +0.5 point win over GPT-4V, effectively tied, but clearly ahead of Gemini Pro.

Where InternLM-XComposer2 is competitive but behind:

QBench: 72.5 vs. GPT-4V 74.1 (behind by 1.6 points) but ahead of Gemini Pro 70.6. QBench focuses on low-level vision (image quality assessment), a domain where CLIP-based vision encoders may have inherent limitations since they're trained for semantic, not perceptual, alignment.
LLaVA-Bench: 81.8 vs. GPT-4V 93.1 (behind by 11.3 points, the largest gap). LLaVA-Bench tests real-world instruction following, and GPT-4V's likely much larger model capacity and broader training data presumably give it an advantage in open-ended conversational flexibility.
MM-Vet: 51.2 vs. GPT-4V 67.7 (behind by 16.5 points). MM-Vet tests integrated capabilities, and this substantial gap suggests the 7B model struggles with tasks requiring simultaneous reasoning across multiple modalities and knowledge sources — a capacity-limited behavior.
HallusionBench: 60.3 vs. GPT-4V 65.8 (behind by 5.5 points) but ahead of Gemini Pro 63.9. Hallucination resistance, while SOTA among open-source models (Table 5), still trails the best commercial system.
ChartQA: 72.6 vs. GPT-4V 78.5 (behind by 5.9 points) and behind QwenVL-Plus 78.1. Chart reading requires precise OCR and structured data extraction — the gap may reflect the vision encoder's resolution limitations (490×490 during pre-training may be insufficient for fine chart text).

Where InternLM-XComposer2 is notably behind:

MMMU: 42.0 vs. GPT-4V 56.8 (behind by 14.8 points). MMMU is a college-level multi-disciplinary benchmark requiring expert-level reasoning. This is the most cognitively demanding benchmark, and the 14.8-point gap is the second-largest deficit. The paper notes this is "slightly worse" than the APIs — a diplomatic phrasing for a gap that represents GPT-4V being 35% better. This is the clearest evidence that scale still matters for the hardest reasoning tasks.

Cross-benchmark patterns: The model wins on benchmarks emphasizing perception (MME, SEED-Bench), math reasoning (MathVista), and structured understanding in bilingual contexts (MMBench). It loses most on benchmarks requiring open-ended conversational flexibility (LLaVA-Bench), integrated multi-skill reasoning (MM-Vet, MMMU), and fine-grained chart reading (ChartQA). This pattern is consistent with a 7B model that has excellent perceptual alignment and strong domain-specific reasoning (math) but hits capacity limits on tasks requiring simultaneous deployment of multiple complex capabilities.

Comparison with Open-Source Models

Table 4 compares against eleven open-source MLLMs. The headline: InternLM-XComposer2 achieves state-of-the-art on 10 of 11 benchmarks, with the single exception being MM-Vet where CogVLM-17B scores 54.5 vs. InternLM-XComposer2's 51.2.

The margin of improvement is substantial and consistent:

MathVista: 57.6. The nearest open-source competitor is Monkey-10B at 34.8 — a gap of +22.8 points. This is not incremental improvement; it's a qualitative leap. The next best are CogVLM-17B (34.7) and Qwen-VL-Chat (33.8). All other open-source models score in the 23–29 range. This suggests InternLM2-7B's mathematical training data and architecture are fundamentally better suited to math reasoning than Vicuna, Qwen, or LLaMA at comparable scales.
MMBench: 79.6. Nearest competitor is LLaVA-XTuner (InternLM2-20B) at 75.1 — a +4.5 point gap over a model with nearly 3× more parameters. InternLM-XComposer2 is the only sub-10B model above 79%, and it beats models at 20B (LLaVA-XTuner) and 17B (CogVLM-17B at 65.8) by wide margins.
MME-Perception: 1712.0. This is the first 7B model above 1700. The nearest competitor at similar scale is ShareGPT4V (Vicuna-7B) at 1567.4 — a +144.6 point gap (9.2% relative improvement). Even the 10B+ models (Monkey-10B at 1522.4, InternLM-XComposer at 1528.4) don't approach this number. The MME-Cognition score of 530.7 similarly dominates — the nearest is Monkey-10B at 401.4, a +129.3 point gap.
MMBench-Chinese: 77.6. As with the closed-source comparison, the model's bilingual strength appears here. The nearest competitor is LLaVA-XTuner (InternLM2-20B) at 73.7. InternLM-XComposer (the predecessor) scored 72.4 — the +5.2 point improvement over the predecessor is likely attributable to InternLM2's stronger multilingual pretraining.
LLaVA-Bench: 81.8. Nearest open-source competitor is CogVLM-17B at 73.9 — an +7.9 point gap. This is notable because LLaVA-Bench tests open-ended generation quality, and beating a 17B model by 7.9 points with a 7B model suggests PLoRA's language preservation strategy is working as intended.
SEED-Bench: 75.9. Nearest competitor is LLaVA-XTuner (20B) at 70.2 — a +5.7 point gap. Again, the 7B model outperforms a 20B model from the same LLM family (InternLM), suggesting the PLoRA alignment strategy yields better results than whatever alignment approach LLaVA-XTuner uses.

The MM-Vet exception: CogVLM-17B achieves 54.5 vs. InternLM-XComposer2's 51.2. CogVLM uses a different architectural strategy — a dedicated visual expert module that adds parameters beyond the base LLM — which may provide advantages for the integrated reasoning tasks that MM-Vet emphasizes. The 3.3-point gap is modest, and InternLM-XComposer2 still ranks second among open-source models.

Scale-defying behavior: The most striking pattern in Table 4 is the consistent outperformance of much larger models: 7B beats 17B (CogVLM), 20B (LLaVA-XTuner on InternLM2), and 80B (IDEFICS-80B). This isn't one lucky benchmark — it's systematic across 10 of 11 evaluations. The only model at comparable scale that approaches InternLM-XComposer2's performance is ShareGPT4V (Vicuna-7B), and even there the gaps are large (e.g., 57.6 vs. 25.8 on MathVista). This suggests that InternLM2-7B's base capabilities combined with PLoRA alignment produce a fundamentally more capable multimodal model than prior approaches at the same parameter count.

Hallucination Evaluation

Table 5 reports results on POPE and HallusionBench:

POPE: 87.7 F1 (averaged across three tracks). This is SOTA among open-source models. The nearest competitor is LLaVA-1.5 (Vicuna-13B) at 85.9. The predecessor InternLM-XComposer does not report POPE results.
HallusionBench: 60.3. Again SOTA among open-source models. The nearest are Monkey-10B at 58.4 and InternLM-XComposer at 57.0. The paper explicitly notes outperforming "all open-source models" and "the closed-source API, QwenVL-Plus" (56.4). Against GPT-4V (65.8) and Gemini Pro (63.9), the model trails but is competitive.

The hallucination results are important because they validate the claimed benefits of PLoRA for faithful visual understanding. The argument (implicitly) is: preserving the LLM's language capabilities through PLoRA produces more honest visual descriptions because the model doesn't confabulate to reconcile distorted language priors with visual input. While the paper doesn't provide ablation experiments directly linking PLoRA to hallucination reduction, the SOTA hallucination scores are consistent with this hypothesis and distinguish InternLM-XComposer2 from models like LLaVA (44.1 on HallusionBench) and InstructBLIP (53.6 on HallusionBench) that use simpler alignment approaches.

Composition Quality: CreationBench

Table 6 evaluates creative writing capability using CreationBench from OpenCompass. Baselines are pure-text LLMs: GPT-4, Qwen-72B-Chat, and Yi-34B-Chat — all substantially larger than 7B. The evaluation uses GPT-4 as judge across four dimensions (Creativity, Richness, User Demand Fulfillment, Logical Coherence), each on a 10-point scale.

Without GPT-4 reference answer:

InternLM-XComposer2: Average 6.24, with sub-scores: Creativity 5.11, Richness 6.12, UDF 7.03, LC 7.45.
GPT-4: Average 6.32, with sub-scores: Creativity 5.22, Richness 5.98, UDF 7.17, LC 7.47.
Qwen-72B-Chat: Average 5.70.
Yi-34B-Chat: Average 6.03.

The 7B model scores within 0.08 points of GPT-4 on average (6.24 vs. 6.32), and actually exceeds GPT-4 on Richness (6.12 vs. 5.98) and effectively ties on Logical Coherence (7.45 vs. 7.47). It clearly outperforms Qwen-72B-Chat (a 10× larger model) by 0.54 points and Yi-34B-Chat (a ~5× larger model) by 0.21 points.

With GPT-4 reference answer:

InternLM-XComposer2: Average 5.90, with sub-scores: Creativity 5.21, Richness 5.76, UDF 6.27, LC 6.93.
GPT-4: Average 5.98.
Qwen-72B-Chat: Average 5.31.
Yi-34B-Chat: Average 5.71.

The gap to GPT-4 narrows to 0.08 points again. The model continues to outperform much larger pure-text LLMs. The "with reference" setting is a harder evaluation because the judge compares against a GPT-4 generated answer, and smaller models typically suffer more from this direct comparison. That InternLM-XComposer2 maintains competitive scores suggests its writing quality is genuinely high, not just an artifact of lenient evaluation prompts.

Interpreting CreationBench: This is the key evidence for the paper's claim that PLoRA preserves language capabilities. A 7B model that lost significant language fluency during multimodal alignment would not compete with GPT-4 on creative writing. The fact that it does — and beats 34B and 72B pure-text models — is strong evidence that PLoRA's gradient isolation strategy works as intended. However, it's worth noting that CreationBench scores are inherently subjective (GPT-4 as judge is not a perfect evaluator) and that the model was trained on InternLM2-7B-ChatSFT, which already underwent chat fine-tuning that may have improved its writing relative to base models. The CreationBench result likely reflects a combination of InternLM2's strong base writing ability and PLoRA's effective preservation of that ability, rather than PLoRA itself teaching the model to write better.

Ablation Studies and Robustness Checks

The paper does not contain explicit ablation experiments in the traditional sense. There is no table showing "InternLM-XComposer2 with PLoRA vs. with standard LoRA vs. with full fine-tuning," no ablation of the rank parameter (256 vs. 128 vs. 64), no ablation of the three pre-training objectives, no ablation of the LLDR decay factor, and no ablation of the 10% pure-text SFT data allocation. This is a significant gap — the paper's central architectural claim is that PLoRA is superior to alternative alignment strategies, but this claim is never tested through controlled comparison.

What the paper does provide as implicit ablation evidence:

InternLM-XComposer vs. InternLM-XComposer2 comparison (Table 4, row "InternLM-XC" vs. "Ours"): The predecessor uses InternLM-7B with a different alignment approach (not PLoRA) and achieves substantially lower scores: MathVista 29.5 vs. 57.6 (+28.1), MMMU 35.6 vs. 42.0 (+6.4), MME-Perception 1528.4 vs. 1712.0 (+183.6), MMBench 74.4 vs. 79.6 (+5.2), SEED-Bench 66.1 vs. 75.9 (+9.8), HallusionBench 57.0 vs. 60.3 (+3.3). This is strong evidence that something about the new system is better, but it conflates multiple changes: the LLM (InternLM → InternLM2), the alignment strategy (previous approach → PLoRA), the training data (likely expanded and improved), and the training recipe. It's impossible to attribute the gains specifically to PLoRA.

Resolution choice for composition: The paper states that "having a high-resolution image input is not essential for text-image composition" and down-samples to 224×224 during composition SFT. The qualitative outputs (Figures 3–10) serve as evidence that this resolution is sufficient, but there's no controlled comparison showing that 490×490 produces worse or equivalent composition quality.

Layer-wise learning rate decay (LLDR) of 0.90: Applied to the vision encoder in both pre-training and SFT. No ablation testing 0.95, 0.85, or no decay is reported. The choice is justified by the conceptual argument that early CLIP features should be preserved — a reasonable hypothesis, but untested.

Learning rate scale factor 0.2 for LLM during SFT: No ablation testing 0.1, 0.5, or 1.0. The choice is crucial to the paper's thesis that slow LLM updates preserve language capabilities, but there's no empirical evidence for this specific value.

The 10% pure-text data allocation: No ablation testing 0%, 5%, 20%, or 50%. The paper claims this is necessary to maintain language capabilities, but provides no evidence that removing it would cause degradation, nor evidence that 10% is optimal.

Rank 256 for PLoRA: No ablation testing other ranks. The choice of 256 is unusually high for LoRA (common ranks are 8–64). The paper doesn't justify this specific number beyond stating it's what was used.

This absence of ablations is the single biggest weakness of the experimental section. The paper is fundamentally a systems paper claiming that PLoRA is a superior alignment strategy, but it never isolates PLoRA's contribution from the other improvements (newer LLM, better data, refined training recipe). A minimal ablation suite would include:

PLoRA vs. standard LoRA (same rank, applied to all tokens) vs. full fine-tuning, all with the same LLM and data.
PLoRA with different ranks (64, 128, 256, 512) to characterize the scaling of alignment quality with adaptation capacity.
Ablation of the three pre-training objectives (removing one at a time) to measure their individual contributions.
Ablation of the 10% pure-text SFT data to quantify the language preservation benefit.
Ablation of LLDR and LLM learning rate scale to characterize sensitivity to these optimization hyperparameters.

Component-wise learning rate as a de facto ablation: The paper specifies different learning strategies for different components, which implies the authors experimented with different values and found these to work well. However, the process of arriving at these values is not documented, and no negative results from alternative configurations are reported. The choice of LLDR 0.90 for the vision encoder and LR scale 0.2 for the LLM are presented as final configurations, not as the outcome of systematic hyperparameter search.

Data mixture ratios as implicit design choices: The 1:1:1:1 ratio across the four composition data dimensions and the weighted sampling based on dataset size for multi-task SFT are stated but not ablated. It's unclear whether balanced sampling is optimal, or whether certain dimensions (e.g., complex instruction adherence) matter more than others.

Critical Assessment

Claim 1: "InternLM-XComposer2 significantly outperforms existing multimodal models."

This claim is strongly supported by the reported numbers, but the "significantly" qualifier is unverifiable due to the absence of statistical testing. The model achieves SOTA on 10 of 11 open-source benchmarks (Table 4) and beats GPT-4V/Gemini Pro on 6 of 12 benchmarks (Table 3). The margins are large enough that they probably would survive statistical testing — +22.8 points on MathVista over the nearest open-source model, +316.2 points on MME over GPT-4V — but without confidence intervals, "probably" is the operative word. The claim also depends on what "existing multimodal models" means: the comparison set is comprehensive for open-source MLLMs at comparable scales, but it doesn't include every possible model (e.g., some concurrent works might achieve higher scores on specific benchmarks).

Claim 2: "PLoRA enables balancing vision understanding with text composition."

This claim is indirectly supported but never directly tested. The evidence is: (1) the model does well on both vision benchmarks (Tables 3–4) and writing benchmarks (Table 6), proving it can do both; (2) the model outperforms its predecessor (which didn't use PLoRA) on both fronts. However, the paper never runs the crucial experiment: same LLM, same data, same training recipe, but with a different alignment strategy (standard LoRA, full fine-tuning, or a simple linear projector). Without this comparison, we cannot attribute the dual excellence to PLoRA specifically. It could be that InternLM2-7B is simply a much stronger base model than InternLM-7B, and any reasonable alignment strategy would produce good results. The PLoRA design is elegant and theoretically motivated, but the experimental section doesn't prove it's necessary or even better than alternatives.

Claim 3: "Matches or even surpasses GPT-4V and Gemini Pro in certain assessments."

This claim is substantiated but requires careful contextualization. The model does surpass GPT-4V on MathVista (+7.7), MME (+316.2), MMBench (+2.6), MMBench-Chinese (+3.2), SEED-Bench (+6.8), and AI2D (+0.5). However:

On MathVista, the gap likely reflects InternLM2's mathematical training rather than multimodal capability per se. A model with strong math but weak vision might also score well here.
On MME, the perception sub-score dominance may reflect the specific fine-tuning data mixture — MME tests relatively straightforward visual recognition that good alignment should handle well.
The model is substantially worse on MMMU (-14.8), MM-Vet (-16.5), LLaVA-Bench (-11.3), and ChartQA (-5.9). On the hardest reasoning benchmarks, GPT-4V maintains a clear lead.
GPT-4V's parameter count is undisclosed but likely 1–2 orders of magnitude larger than 7B. Beating it on any benchmarks is impressive, but the pattern suggests that scale still dominates for complex reasoning, while careful alignment can close the gap on perception-heavy tasks.

The "certain assessments" qualifier is doing real work here — the model wins on 6 of 12, loses on 4, and essentially ties on 2 (AI2D, QBench). This is a genuine achievement for a 7B model but not a blanket claim of GPT-4V-level performance.

Claim 4: Preserves pre-trained language capabilities through PLoRA.

Supported by CreationBench results but confounded by the base LLM change. The model achieves 6.24 on CreationBench, essentially tied with GPT-4 (6.32). However, InternLM2-7B-ChatSFT (the base model before any multimodal training) is not evaluated on CreationBench, so we don't know if the multimodal training degraded language capabilities from a higher starting point. If InternLM2-7B-ChatSFT would have scored 6.5 on CreationBench, then PLoRA actually lost 0.26 points of writing quality — still impressive relative to other multimodal models, but not evidence of perfect preservation. The paper's logic is: "InternLM2 is a strong LM → our model writes well → therefore we preserved LM capabilities." The missing evidence is the base LM's CreationBench score.

Genuine Weaknesses in the Experimental Design

No ablation of the core architectural innovation. This is the most serious deficit. The paper's title and abstract emphasize PLoRA as the key contribution, but the experimental section provides no controlled evidence that PLoRA outperforms alternatives. The comparison to the predecessor confounds PLoRA with the LLM upgrade and likely data improvements. The field needs to see PLoRA vs. standard LoRA vs. full fine-tuning on identical data with the same base LLM.

Single LLM family, single vision encoder. All experiments use InternLM2-7B with CLIP ViT-L. Whether PLoRA's benefits generalize to other LLM architectures (LLaMA, Mistral, Qwen) or other vision encoders (SigLIP, EVA-CLIP, DINOv2) is unknown. The strong MathVista results may be specific to InternLM2's training data rather than a general property of PLoRA-aligned models.

No statistical reporting. With 500 test questions on many benchmarks, a 1–2 point difference may be within sampling noise. The paper reports single-point estimates without error bars, confidence intervals, or significance tests. Given the increasing compression of scores at the top of MLLM leaderboards, statistical rigor is essential for interpreting small margins.

CreationBench evaluation by GPT-4 introduces circularity. GPT-4 is both a baseline and the evaluator. If GPT-4's evaluation model has systematic biases (e.g., preferring certain writing styles, penalizing non-native English patterns), these biases affect all scores. The paper mitigates this somewhat by reporting both with and without GPT-4 reference answers, but a human evaluation or multi-judge setup would strengthen confidence.

No compositional generalization testing. The paper shows impressive qualitative outputs (Figures 3–10), but these are cherry-picked examples demonstrating specific capabilities. There's no systematic evaluation of composition quality — no benchmark measuring factuality of generated articles, relevance of embedded images, adherence to multi-constraint instructions, or stylistic consistency. The CreationBench evaluation measures pure-text writing quality, not multimodal composition quality. A benchmark that evaluates interleaved text-image content (e.g., "does the image appear in a relevant section?", "does the text accurately describe the image?", "are all user constraints satisfied?") would provide stronger evidence for the composition claims.

Composition data is in-house and unreleased. The four-dimensional composition dataset is described but not made available (at least at time of publication). This makes the composition capability irreproducible and prevents the community from evaluating whether the data construction methodology or the model architecture is the primary driver of composition quality.

No analysis of failure modes. The qualitative results show only successes. There's no error analysis: What kinds of instructions does the model fail to follow? When does it insert images inappropriately? Are there systematic composition failures (e.g., certain writing styles it can't produce, languages where quality degrades)? Error analysis is essential for understanding the boundaries of the composition capability but is entirely absent.

No test-time compute scaling experiments. The model is evaluated with single-pass generation. The paper doesn't explore whether best-of-N, majority voting, or verifier-guided selection would further improve results — standard techniques in the MLLM evaluation literature. Self-consistency on MathVista, for example, could push the already-strong 57.6 even higher. The absence of these experiments makes the comparison to models that are evaluated with self-consistency potentially unfair.

Conditional Boundaries on the Claims

The paper's claims hold most strongly when:

The base LLM is inherently strong in the target capability. The MathVista dominance likely reflects InternLM2's math training, not PLoRA's magic. On MMMU (which requires broader knowledge), the model trails GPT-4V substantially. The implication: PLoRA preserves what's there but doesn't add new capabilities.
The task is perception-heavy rather than reasoning-heavy. Wins on MME, SEED-Bench, and MMBench (which emphasize visual recognition and straightforward QA) vs. losses on MMMU and MM-Vet (which require complex multi-step reasoning). PLoRA's alignment strategy may be particularly effective for perceptual tasks while capacity limits dominate on reasoning.
The language is well-represented in training. The strong MMBench-Chinese results reflect InternLM2's bilingual pretraining. The model's performance on languages not well-represented in the training data is unknown.
Composition is evaluated qualitatively and subjectively. The CreationBench results use GPT-4 as judge, and the qualitative outputs are curated. There's no objective metric for composition quality, making it hard to compare across models or track progress.

Missing Experiments That Would Strengthen the Paper

PLoRA vs. standard LoRA vs. full fine-tuning on identical data with InternLM2-7B.
Ablation of PLoRA rank to characterize the alignment quality vs. parameter efficiency tradeoff.
Base LLM's CreationBench score to quantify language capability preservation.
Systematic composition evaluation with human judgment or automated metrics for constraint satisfaction, relevance, and coherence of interleaved text-image output.
Error analysis of composition failures and hallucination patterns.
Self-consistency or best-of-N results to test whether test-time compute can further boost the already-strong numbers.
Cross-model-family replication to test whether PLoRA's benefits generalize beyond InternLM2 and CLIP ViT-L.

6. Limitations and Trade-offs

Absence of Ablation Experiments for the Core Architectural Claim

The assumption or constraint. The paper's central architectural contribution is Partial LoRA (PLoRA) — the claim that applying low-rank adaptations exclusively to visual tokens, while leaving language tokens to be processed by the pre-trained weights alone, produces superior multimodal alignment compared to alternative strategies (standard LoRA applied to all tokens, full fine-tuning, simple linear projection). Section 3.1 articulates this as a principled design: treating visual and language tokens equally "overlooks the inherent property distinctions between modalities," while treating them as entirely distinct "results in a substantial alignment cost." However, the experimental section provides no controlled comparison between PLoRA and any alternative alignment strategy. The only comparative evidence is against InternLM-XComposer, the predecessor model (Table 4), which confounds the alignment strategy change with a different base LLM (InternLM-7B vs. InternLM2-7B), different training data, and a different training recipe.

The consequence. The paper's headline-level claim — that PLoRA is the mechanism enabling simultaneous excellence in composition and comprehension — is supported by correlation, not causation. An equally plausible interpretation of the results is that InternLM2-7B is a dramatically stronger base language model than InternLM-7B (the predecessor's LLM), and that any reasonable alignment strategy would have produced similar benchmark gains. The large jumps on MathVista (29.5 to 57.6, a +28.1 point improvement) and MME-Perception (1528.4 to 1712.0, a +183.6 point improvement) could be primarily driven by InternLM2's superior mathematical reasoning and language understanding, with PLoRA contributing only marginally beyond what a standard LoRA or full fine-tuning approach would achieve. Without ablations, a practitioner choosing an alignment strategy for their own model has no empirical basis for selecting PLoRA over simpler, better-understood alternatives — the paper offers only architectural intuition, not measured performance differences.

What evidence exists in the paper. None. The paper does not contain a table or figure comparing PLoRA to standard LoRA, full fine-tuning, or a linear projection baseline under controlled conditions (same LLM, same data). The closest thing to an ablation is the InternLM-XComposer comparison in Table 4, but as noted, this conflates multiple variables. The paper also does not report ablations of PLoRA rank (256 vs. alternatives), the three pre-training objectives (removing one at a time), the 10% pure-text SFT data allocation, or the component-specific learning rate strategies. Each of these design choices is justified conceptually but never tested empirically.

Mitigation status. Not addressed. The paper presents PLoRA as a complete, validated solution without acknowledging the absence of comparative alignment experiments as a limitation. No future work is suggested to isolate PLoRA's contribution from confounds.

Unaccounted Cost of In-House Data Curation as a Reproducibility Barrier

The assumption or constraint. The paper's training pipeline relies on a substantial quantity of in-house, unreleased data, most critically the free-form text-image composition dataset described in Section 3.4. This dataset is constructed across four dimensions (varied writing styles, flexible text editing, complex instruction adherence, customization with materials) with each dimension "approximately equal" in proportion. The paper states that this dataset is central to the model's composition capabilities, which are the primary differentiator from prior MLLMs. However, the dataset is not released, its exact size is not specified (the paper says only that the four dimensions have a ratio of "approximately 1:1:1:1"), and the data construction methodology — while described qualitatively — lacks the detail needed for independent reproduction (e.g., how were "complex instructions" generated? What are the specific writing styles covered? How was quality filtering performed?).

The consequence. A practitioner attempting to reproduce InternLM-XComposer2's composition capabilities faces an insurmountable gap: the open-source model weights are available, but the training data that produced the composition behavior is not. This means the composition capability is effectively non-reproducible and non-auditable. A researcher cannot determine whether the model's impressive composition diversity (Figures 3–10, spanning English and Chinese, blog posts, classical essays, poetry, social media posts, code documentation, and travel diaries) arises from generalizable training methodology or from the specific content and quality of the in-house data. Furthermore, the composition capability cannot be transferred to other base LLMs — to build a similar system with LLaMA or Mistral, one would need to reconstruct the composition dataset from scratch based on the paper's high-level description, which is likely to produce different results. The absence of the composition data also means the model's potential biases, factual errors, or stylistic blind spots in creative generation cannot be investigated through data auditing.

What evidence exists in the paper. The paper is transparent that the composition data is in-house: Section 3.4 describes the four dimensions and their 1:1:1:1 ratio, and the qualitative outputs in Figures 3–10 demonstrate the resulting capabilities. However, the paper provides no path to accessing this data, no detailed data card with statistics, examples, or construction protocols, and no discussion of whether synthetic data generation (e.g., using GPT-4 to produce composition training examples) could serve as a substitute. The composition evaluation is entirely qualitative — there is no benchmark or metric for interleaved text-image generation quality, so even if another group attempted reproduction, there would be no standardized way to compare results.

Mitigation status. Not addressed. The paper does not acknowledge the in-house data as a limitation, nor does it discuss plans for release, synthetic alternatives, or community benchmarking for composition quality. The model weights and inference code are released, which enables using the composition capability but not studying or improving it.

Hard Capability Ceiling on Complex Reasoning Tasks

The assumption or constraint. InternLM-XComposer2's strong benchmark performance is concentrated on perception-heavy and domain-specific tasks, while it falls substantially behind GPT-4V on benchmarks requiring multi-step reasoning, knowledge integration, and open-ended problem solving. On MMMU (college-level multi-disciplinary reasoning), the model scores 42.0 vs. GPT-4V's 56.8 — a -14.8 point gap representing a 35% relative deficit. On MM-Vet (integrated capabilities), it scores 51.2 vs. GPT-4V's 67.7 — a -16.5 point gap. On LLaVA-Bench (real-world instruction following), it scores 81.8 vs. GPT-4V's 93.1 — the largest gap at -11.3 points. On ChartQA (chart reading requiring precise extraction and reasoning), it scores 72.6 vs. GPT-4V's 78.5.

The consequence. The paper's claim of GPT-4V-competitive performance (Section 1, Abstract, Figure 1) is true only for a specific subset of tasks — those emphasizing visual recognition, structured understanding, and domain-specific reasoning (math). The 7B parameter budget imposes a fundamental ceiling on the model's ability to handle cognitively demanding tasks that require simultaneous deployment of multiple reasoning capabilities, extensive world knowledge retrieval, and flexible problem-solving strategies. This pattern is consistent with a model whose perceptual alignment is excellent (winning on MME, SEED-Bench, MMBench) but whose reasoning depth is constrained by the base LLM's capacity. The practical implication: InternLM-XComposer2 may be suitable for deployment in applications where the task distribution is skewed toward visual understanding and structured generation (content creation, image description, chart reading, factual QA about images), but it would be unreliable or inadequate for applications requiring expert-level multi-disciplinary reasoning, complex multi-step inference from visual inputs, or open-ended problem solving in novel domains. A user expecting "GPT-4V-level performance" based on the headline claim would be disappointed on these harder tasks.

What evidence exists in the paper. Table 3 provides the direct comparison. The pattern is clear: InternLM-XComposer2 wins on 6 of 12 benchmarks (MathVista, AI2D, MME, MMBench, MMBench-Chinese, SEED-Bench), loses substantially on 4 (MMMU, MM-Vet, LLaVA-Bench, ChartQA), and is competitive on 2 (QBench, HallusionBench). The paper acknowledges that the MMMU score is "slightly worse" than the APIs — a characterization that understates the 35% relative deficit, which is the second-largest gap across all benchmarks. The performance on MMMU is also notably behind the previous open-source SOTA (Yi-VL-34B at 45.9 vs. InternLM-XComposer2 at 42.0 in Table 3), suggesting that on the hardest reasoning tasks, the PLoRA alignment strategy does not compensate for the parameter count difference, and in fact the 7B model underperforms a 34B model from a different family.

Mitigation status. Partially acknowledged but underemphasized. The paper notes the MMMU score in Section 4.1: "despite having only 7B parameters, our model attains a slightly worse score of 43.0% [sic — Table 3 reports 42.0] on the challenging college-level benchmark MMMU." The framing as "despite having only 7B parameters" implies that the lower score is expected and acceptable given the scale, which is a reasonable claim but does not address whether the capability gap is practically significant for deployment. The paper does not discuss the pattern across benchmarks, does not analyze which specific MMMU or MM-Vet sub-tasks the model fails on, and does not provide guidance on what task profiles are safe for deployment. There is no error analysis of hard-case failures.

Absence of Systematic Composition Evaluation

The assumption or constraint. The paper's second major capability claim — free-form text-image composition — is evaluated entirely through qualitative examples (Figures 3–10) and a single benchmark (CreationBench, Table 6) that measures pure-text creative writing quality, not multimodal composition quality. CreationBench's evaluation dimensions (Creativity, Richness, User Demand Fulfillment, Logical Coherence) apply to text-only outputs and do not assess whether embedded images are relevant to surrounding text, whether image placement is appropriate, whether user-provided visual materials are correctly incorporated, or whether multi-constraint composition instructions (e.g., "write an 800-word article with specific sections, incorporate reference text, and use a poetic style") are faithfully followed. The paper's qualitative figures demonstrate these capabilities exist, but there is no quantitative measurement of how reliably they work.

The consequence. The paper's composition capability is demonstrated but not validated. A practitioner considering InternLM-XComposer2 for content creation cannot answer basic deployment questions from the paper's evidence: What fraction of generated articles have correctly placed images? How often does the model follow all user constraints (word count, section structure, stylistic requirements, material incorporation)? Does composition quality degrade for certain writing styles, languages, or input complexities? Are there systematic failure modes (e.g., images placed in semantically inappropriate locations, text that contradicts or ignores provided reference materials, stylistic inconsistency within a single article)? Without these measurements, adopting the model for production content creation is a gamble — the cherry-picked qualitative outputs show what the model can do, not what it typically does. The absence of composition evaluation also makes it impossible to track progress in the field — if another model claims better composition, there is no standardized way to compare.

What evidence exists in the paper. Figures 3–10 provide compelling qualitative evidence that the model can produce diverse composition formats. CreationBench (Table 6) provides quantitative evidence that the model's pure-text writing quality is competitive with GPT-4. However, there is no benchmark, metric, or systematic evaluation of the multimodal composition capability itself — no measurement of image-text coherence, constraint satisfaction rate, or stylistic consistency. The paper does not report how many composition outputs were generated, how many were cherry-picked for the figures, or what fraction of outputs were of acceptable quality. There is no error analysis of composition failures.

Mitigation status. Not addressed. The paper treats the qualitative outputs as sufficient evidence for the composition claim, without acknowledging the absence of systematic evaluation as a limitation. No future work on composition benchmarking is suggested. This is a significant gap given that composition is one of the two pillars of the paper's contribution (the other being comprehension, which is extensively benchmarked).

Latency and Throughput Tradeoffs from Resolution Switching and Long-Sequence Generation

The assumption or constraint. InternLM-XComposer2's design involves a resolution switching strategy: 490×490 during pre-training for detailed visual grounding, then down-sampled to 224×224 during composition SFT because "high-resolution image input is not essential for text-image composition" (Section 3.4). Additionally, the model's composition capability involves generating long sequences — the qualitative outputs in Figures 3–10 show articles ranging from ~200 words (English poetry, Figure 5) to 800+ Chinese characters (Figures 4, 6, 7), with some containing embedded images at multiple positions. The paper provides no latency or throughput measurements for any task, no memory usage analysis, and no discussion of the inference-time cost of processing variable-resolution images through the vision encoder while generating long text sequences.

The consequence. A deployment engineer evaluating InternLM-XComposer2 for a production system needs to understand its computational profile to determine hardware requirements, cost per query, and whether the model can meet latency SLOs (service-level objectives). The paper provides none of this information. Key unanswered questions include: What is the end-to-end latency for a typical composition request (e.g., an 800-word article with 2–3 embedded images)? How does latency scale with output length and number of input images? What is the GPU memory footprint during inference, and does it fit comfortably on consumer hardware (e.g., a single A10 or RTX 4090) or require datacenter GPUs? Does the resolution switching require reloading model weights or reconfiguring the vision encoder, adding deployment complexity? For batch processing scenarios (e.g., generating articles for multiple prompts simultaneously), what is the maximum throughput? Without these measurements, the paper's practical deployability is unclear — a model that takes 30 seconds per long-form article on an A100 may be suitable for offline content creation but not for interactive applications, while a model that runs at 2 seconds per article on a consumer GPU would enable real-time creative assistance.

What evidence exists in the paper. None. The paper does not report generation time, memory usage, throughput, or any other computational efficiency metric. The model is described architecturally (Section 3.1) and by training configuration (Sections 3.2–3.3), but inference latency is never discussed. The qualitative outputs demonstrate sequence lengths and formats but provide no timing information.

Mitigation status. Partially addressed through model release. The paper states that the model series is publicly available on GitHub, which means practitioners can measure inference performance themselves. However, the paper itself provides no guidance on expected performance, no hardware recommendations, and no discussion of whether inference-time optimizations (KV-cache quantization, FlashAttention, speculative decoding) were used or are compatible. The resolution switching strategy — 490×490 for pre-training, 224×224 for composition — raises a practical question not addressed in the paper: does a deployed model need to support both resolutions depending on the task, or is the 224×224 SFT model sufficient for all use cases? If both resolutions are needed, the deployment must either maintain two model configurations or dynamically resize inputs, adding complexity.

Single Model Family Evaluation Without Cross-Architecture Generalization Evidence

The assumption or constraint. All experiments in the paper use a single LLM (InternLM2-7B-ChatSFT) and a single vision encoder (OpenAI CLIP ViT-L-14-336). The paper's claims about PLoRA's effectiveness, the training recipe, and the resulting capabilities are validated exclusively on this specific model combination. The paper does not test whether PLoRA provides similar benefits when applied to other popular LLM families (LLaMA, Mistral, Qwen), other vision encoders (SigLIP, EVA-CLIP, DINOv2), or other model scales (1B, 13B, 70B). Section 2 acknowledges the existence of multiple LLM families (LLaMA, Mistral, Qwen, GLM, Baichuan) and vision encoders (CLIP, EVA-CLIP, Alpha-CLIP), but the experimental section never asks whether the findings generalize.

The consequence. The paper's core methodological contribution — PLoRA as a modality alignment strategy — may be specific to the InternLM2 architecture or training data rather than a general principle. InternLM2-7B possesses specific properties that could interact favorably with PLoRA: exceptional multilingual capabilities (explicitly noted in Section 3.1), strong mathematical reasoning (evidenced by the MathVista dominance), and prior chat fine-tuning (ChatSFT variant). A practitioner using LLaMA-2-7B, which lacks InternLM2's bilingual pretraining and mathematical emphasis, might find PLoRA produces substantially weaker results — perhaps no better than standard LoRA. Similarly, the CLIP ViT-L vision encoder may have representational properties that make it particularly amenable to PLoRA-style adaptation, while other vision encoders with different feature geometries might require different rank configurations or benefit less from the modality-gated approach. The strong MathVista result (57.6, +22.8 over the nearest open-source competitor) is particularly suspect as a potential confound — if InternLM2-7B was trained on mathematical data that happens to overlap with MathVista's problem distribution, the PLoRA alignment strategy gets credit for what is actually a base model property.

What evidence exists in the paper. The only cross-model evidence is the comparison with LLaVA-XTuner (InternLM2-20B) in Table 4. This model uses the same LLM family (InternLM2) but a larger scale (20B) and a different alignment approach (the standard linear projection plus fine-tuning used in LLaVA variants). InternLM-XComposer2 (7B) outperforms LLaVA-XTuner (20B) on 9 of 11 benchmarks, which might suggest PLoRA's superiority. However, this comparison confounds alignment strategy with training data, SFT recipe, and model scale — LLaVA-XTuner was trained by a different team with different data and objectives. The paper does not provide a fair comparison where InternLM2-7B is aligned with PLoRA vs. a standard approach by the same team using the same data. No experiments with LLaMA, Mistral, Qwen, or any non-InternLM2 LLM are reported.

Mitigation status. Not addressed. The paper does not acknowledge the single-model-family limitation, does not discuss whether PLoRA's benefits are expected to transfer to other architectures, and does not suggest cross-family replication as future work. This is a significant gap for a paper that presents PLoRA as a general-purpose alignment strategy. The broader MLLM community cannot adopt PLoRA with confidence until it is shown to work across the diverse ecosystem of open-source LLMs and vision encoders that practitioners actually use.

7. Implications and Future Directions

How This Work Changes the Landscape

InternLM-XComposer2 changes the MLLM landscape by demonstrating that composition can be elevated from a demo capability to a general-purpose, instruction-following skill that coexists with state-of-the-art comprehension in a single 7B model. This is not a paradigm shift in the sense of introducing a fundamentally new learning algorithm or architecture — PLoRA is a thoughtful variant of LoRA, not a departure from the transformer paradigm — but it is a significant reframing of what multimodal models are for. Prior to this work, the MLLM field implicitly treated comprehension as the primary capability, benchmarked extensively through a growing suite of evaluations, while generation meant short-form visual QA or captioning. Composition, when it existed at all (as in the predecessor InternLM-XComposer), was narrow: title-in, article-out, with retrieved images. InternLM-XComposer2 argues, through its systematic data construction and diverse qualitative demonstrations, that generating long-form, stylistically controlled, interleaved text-image content from flexible multi-modal instructions should be a first-class capability alongside visual understanding. The taxonomy of composition requirements — varied writing styles, flexible text editing, complex instruction adherence, and customization with materials — defines what it means for a model to be a general-purpose content creator, providing a vocabulary that future work can build on.

The methodological shift is in treating data engineering as an architectural design choice with equal standing to model architecture. The paper's training data is organized around explicit cognitive progressions during pre-training (semantic alignment → world knowledge alignment → vision capability enhancement) and explicit capability dimensions during SFT (eight task categories plus four-dimensional composition data). The 1:1:1:1 ratio across composition dimensions, the 10% pure-text data allocation for language preservation, and the weighted multi-source sampling with GPT-4-augmented instructions are not afterthoughts — they are presented as first-order design decisions that directly shape the model's capabilities. This is a contrast to much prior MLLM work, where data is listed but not theorized. The implication for the field is that returns to data curation may be higher than returns to architectural complexity at current model scales, and that future MLLM development should invest at least as much effort in systematic data design as in novel alignment modules. This is a reframing of where the bottleneck lies: not in connecting vision to language (a problem PLoRA solves elegantly but not uniquely), but in covering the distribution of tasks and styles that users actually want.

The work also reconciles a latent tension in the MLLM literature between models optimized for benchmark performance and models optimized for generative flexibility. Prior models like CogVLM or LLaVA-1.5 pushed benchmark numbers through architectural innovations (visual experts, improved baselines), while models like the original InternLM-XComposer demonstrated creative generation but lagged on comprehension. InternLM-XComposer2 shows these are not competing objectives — the same PLoRA-aligned model achieves SOTA on 10 of 11 open-source comprehension benchmarks and produces diverse, controllable interleaved content across multiple languages and genres. The reconciliation mechanism is PLoRA's gradient isolation: by preserving the LLM's pre-trained language capabilities through modality-gated adaptation, the model doesn't face a tradeoff between learning to see and retaining the ability to write. This explains why prior models that fully fine-tuned their LLMs may have inadvertently sacrificed generative quality for perceptual accuracy — a hypothesis the paper doesn't prove through ablation but strongly implies through its design choices and CreationBench results.

The work makes certain research directions more attractive: (1) systematic data engineering for capability coverage, including the use of strong models like GPT-4 to diversify training instructions; (2) modality-gated adaptation strategies that preserve pre-trained strengths while adding new modalities; (3) holistic evaluation frameworks that measure both comprehension benchmarks and generative composition quality; (4) investigation of whether smaller models with excellent alignment can substitute for larger models with weaker alignment in deployment. It makes other directions less attractive: (1) pursuing architectural complexity (visual experts, complex bridging modules) without corresponding investment in data diversity, since a simple PLoRA + strong data strategy yields SOTA results at 7B scale; (2) treating composition as an afterthought or a future capability, since InternLM-XComposer2 demonstrates it can be achieved now with systematic data construction; (3) assuming that scale is the primary driver of multimodal performance, since a 7B model outperforms 20B, 34B, and 80B models on most benchmarks.

Follow-Up Research This Work Enables

Controlled ablation of PLoRA against standard alignment strategies. The single most important experiment this paper enables but doesn't conduct is a controlled comparison of PLoRA versus standard LoRA applied to all tokens, full fine-tuning, and a simple linear projection baseline — all using InternLM2-7B as the base LLM and the same training data. The paper's central architectural claim remains untested. A strong follow-up would train four variants: (a) PLoRA as described (rank 256, visual tokens only), (b) standard LoRA (same rank, applied to all tokens), (c) full fine-tuning of the LLM with no LoRA, and (d) a frozen LLM with only the vision encoder and PLoRA trained during pre-training and a single linear projection layer trained during SFT. All variants would use the same pre-training and SFT data mixtures, the same learning rate schedules, and the same component-specific learning rate strategies where applicable. The evaluation would measure comprehension benchmarks (Table 4), CreationBench writing quality (Table 6), and a newly constructed composition quality metric measuring constraint satisfaction and image-text coherence. If PLoRA significantly outperforms standard LoRA and full fine-tuning on both comprehension and composition, the paper's architectural thesis is validated. If the alternatives perform similarly, the gains should be attributed to InternLM2's base capabilities and the data strategy. If full fine-tuning matches PLoRA on comprehension but underperforms on CreationBench, the language preservation mechanism is the key differentiator.

Systematic composition quality benchmarking. The paper's composition capability is demonstrated through cherry-picked qualitative examples but never systematically evaluated. A critical follow-up would construct a benchmark for interleaved text-image composition quality. The benchmark would include 200–500 prompts spanning the four composition dimensions (varied styles, text editing, complex instructions, material customization) in both English and Chinese. Each prompt would specify measurable constraints: word count ranges, required sections, stylistic requirements, reference text to incorporate, reference images to include. Evaluation would use both automated metrics (constraint satisfaction rate measured by a separate LLM judge, image-text relevance measured by CLIP similarity, section structure adherence measured by parsing) and human evaluation (overall quality, creativity, faithfulness to instructions on a 100-example subset). The benchmark would establish a baseline for InternLM-XComposer2's composition reliability and enable comparison with future composition models. An important negative result would be discovering that the model's constraint satisfaction rate is low (e.g., only 60% of outputs meet all specified requirements) despite impressive qualitative examples — which would reframe the composition capability as promising but unreliable for production.

Cross-architecture generalization of PLoRA. The paper validates PLoRA exclusively on InternLM2-7B + CLIP ViT-L. Whether PLoRA's benefits generalize to the broader ecosystem of open-source LLMs and vision encoders determines whether it becomes a standard alignment approach or remains an InternLM-specific technique. A follow-up would replicate the training pipeline with LLaMA-2-7B, Mistral-7B, and Qwen-7B as base LLMs, each paired with CLIP ViT-L and trained on the same pre-training and SFT data (to the extent the composition data is reproducible; if not, the experiments are limited to the comprehension benchmarks using public data). The evaluation would measure benchmark performance relative to each model's standard alignment approach (e.g., LLaVA-style linear projection for LLaMA, Qwen-VL's approach for Qwen). If PLoRA provides consistent improvements across model families, it establishes itself as a general principle. If improvements are specific to InternLM2, the community learns that PLoRA's effectiveness depends on base model properties — perhaps InternLM2's strong multilingual and mathematical training creates representations that benefit more from modality-gated adaptation than other LLMs' representations. The MathVista result (57.6 vs. the nearest open-source competitor at 34.8) is particularly important to isolate: does PLoRA + LLaMA-2-7B also achieve MathVista scores in the 50s, or is InternLM2's math training the dominant factor?

PLoRA rank scaling and the alignment-quality-vs-efficiency frontier. The paper uses a fixed rank of 256 for all PLoRA matrices without justification or ablation. This is a high rank by LoRA standards, suggesting the authors believe visual alignment requires substantial adaptation capacity. A follow-up would systematically vary PLoRA rank (8, 16, 32, 64, 128, 256, 512) and measure both comprehension benchmark performance and total trainable parameter count. The goal is to characterize the alignment-quality-vs-parameter-efficiency frontier: how much adaptation capacity is needed to saturate visual understanding performance? Does CreationBench writing quality degrade at very high ranks (suggesting that excessive adaptation capacity leaks into the language pathway)? Is there a point where increasing rank yields diminishing returns? This experiment would provide practical guidance for practitioners choosing PLoRA rank based on their compute budget and target performance, and would test whether rank 256 is near-optimal or substantially over-provisioned.

Difficulty-stratified error analysis of comprehension failures. The paper reports aggregate benchmark scores but provides no analysis of which questions InternLM-XComposer2 gets wrong, particularly on benchmarks where it underperforms GPT-4V (MMMU, MM-Vet, LLaVA-Bench, ChartQA). A follow-up would categorize errors on MMMU by discipline (physics, medicine, law, etc.) and by cognitive demand (factual recall, multi-step reasoning, visual extraction, knowledge integration) to identify systematic weaknesses. For ChartQA, errors would be categorized as OCR failures, structural misunderstanding, arithmetic errors, or reasoning failures. The goal is to diagnose whether the capability ceiling on complex reasoning is due to the vision encoder's resolution limits (490×490 may be insufficient for fine chart text), the 7B LLM's capacity constraints, or the training data's coverage gaps. If errors cluster in disciplines not well-represented in pre-training data, targeted data augmentation could address the gap. If errors are primarily reasoning failures on multi-step problems, the ceiling is likely capacity-limited and would require either a larger LLM or test-time compute strategies (chain-of-thought, self-consistency). This analysis would transform the paper's "slightly worse" MMMU characterization into actionable diagnostic information.

Self-consistency and test-time compute scaling for comprehension benchmarks. The paper evaluates InternLM-XComposer2 with single-pass greedy or low-temperature decoding. Many MLLM papers report self-consistency gains (generating multiple answers and majority voting), which can add 2–5 percentage points on reasoning benchmarks like MathVista and MMMU. A follow-up would apply self-consistency with 4, 8, 16, and 32 samples on MathVista, MMMU, MMBench, and SEED-Bench to characterize the test-time compute scaling curve for PLoRA-aligned models. The key question: does InternLM-XComposer2 + self-consistency close the gap with GPT-4V on MMMU (42.0 vs. 56.8), or is the gap capacity-limited such that even 32-sample self-consistency provides minimal gains? If self-consistency pushes MMMU above 50, test-time compute becomes a practical strategy for deploying a 7B model on tasks that require larger-model-level reasoning. If self-consistency provides minimal gains, the capacity ceiling is confirmed as the dominant constraint. This experiment would also test whether PLoRA-aligned models exhibit the same self-consistency gains observed in standard MLLMs, or whether the preserved language capabilities affect generation diversity in ways that change the self-consistency tradeoff.

Practical Applications and Downstream Use Cases

AI-assisted content creation for marketing, education, and media. InternLM-XComposer2's demonstrated ability to generate long-form, structured, interleaved text-image articles from flexible multi-modal instructions (Figures 3–10) makes it directly deployable for content creation workflows. A marketing team could provide a product description, target audience, desired tone, and reference images, and receive a complete blog post with embedded product images, section headers, and SEO keywords — as demonstrated in Figure 8's social media post with emoji formatting and auto-generated hashtags (#哈尔滨旅游, #冰雪大世界, etc.). An educator could provide learning objectives, reference materials, and structural requirements (sections on history, technique, cultural value as in Figure 7's ballet article), and receive a comprehensive 800+ character educational article with images. The practical benefit is reducing content creation time from hours to minutes for structured, research-backed articles, with the model handling both the writing and the image selection/placement. The benchmark evidence supporting deployment: the model scores 6.24 on CreationBench (within 0.08 points of GPT-4), demonstrating writing quality competitive with commercial systems, and the hallucination results (87.7 F1 on POPE, 60.3 on HallusionBench) suggest the model is relatively faithful in its visual descriptions — important for content where factual accuracy matters.

Bilingual content generation for Chinese-English markets. InternLM-XComposer2's strong bilingual performance (MMBench 79.6 English vs. MMBench-Chinese 77.6, a gap of only 2.0 points compared to GPT-4V's 2.6-point gap) makes it suitable for organizations operating across Chinese and English language markets. The qualitative outputs demonstrate this bilingualism directly: Figure 4 generates a classical Chinese essay about ink painting incorporating reference text about brush techniques and "墨韵" aesthetics; Figure 6 generates a structured Chinese article about pandas with sections on habitat, conservation, and cultural impact; Figure 5 generates English poetry in Tagore's style; Figure 9 generates an English swimming tutorial with technical terminology. A media company targeting both markets could use a single model for article generation in both languages, rather than maintaining separate models or relying on translation pipelines. The practical benefit is unified content infrastructure for bilingual organizations, with the benchmark evidence showing the model outperforms GPT-4V on MMBench-Chinese (77.6 vs. 74.4) and commercial Chinese-language APIs like QwenVL-Plus (77.6 vs. 70.7), suggesting it may be the best available option for Chinese multimodal content creation at its scale.

On-device or edge deployment of GPT-4V-competitive perception. The 7B parameter scale — combined with benchmark results showing GPT-4V-competitive or superior performance on perception-heavy tasks (MME 2242.7 vs. GPT-4V 1926.5, SEED-Bench 75.9 vs. 69.1) — makes InternLM-XComposer2 a candidate for deployment scenarios where API costs, latency, or data privacy concerns rule out commercial APIs. A healthcare application that needs to analyze medical images and generate structured reports, or a legal application that needs to extract information from document images and produce summaries, could run InternLM-XComposer2 on a single datacenter GPU (A100 or H100) or potentially a consumer GPU with quantization, achieving GPT-4V-level perceptual accuracy on specific tasks without sending sensitive data to external servers. The benchmark evidence supporting this: on MME-Perception specifically (1712.0, SOTA for 7B), the model demonstrates strong visual recognition; on ChartQA (72.6), it shows chart reading capability; and on AI2D (78.7, tied with GPT-4V), it demonstrates scientific diagram understanding. The caveat from the paper's results is that deployment should be limited to perception-heavy tasks — on complex reasoning (MMMU 42.0 vs. GPT-4V 56.8), the model is not a GPT-4V substitute.

Data generation for training improved MLLMs. InternLM-XComposer2's composition capability can be used to generate training data for future multimodal models. The model can produce diverse interleaved text-image articles with varying styles, structures, and constraint profiles, which can serve as supervised fine-tuning data for training composition capabilities in other models. Similarly, the model's strong benchmark performance means it can be used as a judge or data labeler for multimodal tasks — generating reference answers, evaluating candidate responses, or producing synthetic QA pairs from images. This is directly analogous to how GPT-4 is used to augment training instructions in the paper's own methodology (Section 3.3: "the instructions are augmented with GPT-4 to enhance diversity"), but with the advantage of being open-source and deployable at scale without API costs. The practical benefit is enabling the same capability bootstrapping cycle that larger models enable, but with a 7B model that can run on institutional hardware. The benchmark evidence: the model's CreationBench score (6.24, essentially tied with GPT-4 at 6.32) suggests its generated text is of sufficient quality to serve as training targets, and its hallucination scores (POPE 87.7, SOTA among open-source) suggest its visual descriptions are reliable enough for labeling applications.