InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output

ArXiv: 2407.03320

🎯 Pitch

A 7B open-source vision-language model not only matches GPT-4V on 16 key benchmarks but can also generate entire webpages from screenshots. It achieves this by natively training on 24K-token interleaved image–text contexts, enabling ultra-high-resolution understanding and fine-grained video analysis without proprietary scale.

---

1. Executive Summary

This paper introduces InternLM-XComposer-2.5 (IXC-2.5), a versatile large vision-language model that supports long-contextual input and output — natively trained with 24K interleaved image-text contexts and extendable to 96K via RoPE extrapolation — built on a 7B LLM backend. IXC-2.5 advances three comprehension capabilities over its predecessor: Ultra-High Resolution Understanding (a unified dynamic image partition strategy with a native 560×560 ViT encoder), Fine-Grained Video Understanding (treating videos as ultra-high-resolution composite images of densely sampled frames), and Multi-Turn Multi-Image Dialogue (free-form interleaved multi-image conversation), plus two composition applications via extra LoRA parameters — Crafting Webpages (generating HTML/CSS/JavaScript from screenshots, instructions, or resumes) and Composing High-Quality Text-Image Articles (using Chain-of-Thought prompt augmentation and Direct Preference Optimization). Evaluated across 28 benchmarks, IXC-2.5 achieves state-of-the-art among open-source models on 16 benchmarks and matches or surpasses GPT-4V and Gemini Pro on 16 key tasks, establishing that a 7B-parameter LVLM can reach proprietary-API-level versatility when equipped with long-context training and a unified multi-modal input pipeline — though the strongest gains concentrate on tasks where the model's native resolution and context window directly address prior open-source bottlenecks rather than on fundamentally novel reasoning capabilities.

2. Context and Motivation

The Core Problem: Open-Source LVLMs Lack Versatility in Both Input and Output

The fundamental gap this paper addresses is that open-source large vision-language models lag substantially behind proprietary APIs (GPT-4V, Gemini Pro) in versatility — the ability to handle diverse multi-modal tasks within a single unified architecture. While leading closed-source systems support an expansive range of capabilities — ultra-high-resolution image analysis, fine-grained video understanding, multi-turn multi-image dialogue, webpage generation from visual inputs, and long-form article composition — open-source alternatives typically specialize in narrower subsets of these tasks.

This gap is not merely a matter of model scale. The authors explicitly state (Section 1):

"these open-source models still lag behind closed-source leading paradigms in versatility. They lack the ability to perform diverse vision-language comprehension and composition tasks, largely due to limited diversity in training corpus and challenges in managing long-context input and output."

The phrase "limited diversity in training corpus and challenges in managing long-context input and output" identifies two concrete bottlenecks that the paper sets out to address simultaneously. Most existing open-source LVLMs are trained primarily for single-image, single-turn interactions — the classic text-image conversation paradigm exemplified by LLaVA-style models. Extending these systems to handle multiple high-resolution images interleaved with text over long multi-turn dialogues, or to process dense video frame sequences while attending to fine spatial details, requires architectural and training decisions that most prior open-source work did not make.

Why This Versatility Gap Matters

The significance of closing this gap has both practical and research-oriented dimensions:

On the practical side, the absence of long-context multi-modal open-source systems constrains what developers and researchers can build. Real-world applications — analyzing lengthy documents with embedded figures, having extended conversations about collections of photographs, generating illustrated articles from outlines, converting visual designs into functional code — all require models that can (a) ingest and maintain coherence over long interleaved image-text sequences and (b) produce long-form structured outputs containing both text and visual elements. When the only models capable of these tasks are proprietary APIs with usage costs, rate limits, and data privacy concerns, the downstream ecosystem of applications is throttled.

On the research side, the versatility gap obscures which capabilities emerge from scale alone versus which require deliberate architectural design. GPT-4V and Gemini Pro are massive systems whose training recipes are not fully disclosed. Without open-source models achieving comparable versatility at accessible scales, the research community cannot isolate the contributions of model architecture, training data composition, context window design, and alignment strategies. IXC-2.5's choice of a relatively modest 7B LLM backend (InternLM2-7B) is deliberate — the paper aims to demonstrate that versatility at this scale is achievable, thereby providing a testbed for the research community to study which design decisions matter most.

On the deployment side, the 7B parameter count matters enormously. A 7B model with proprietary-API-level versatility can run on consumer hardware, enabling on-device applications, reducing latency, and eliminating per-query costs. If the versatility gap could only be closed at 70B+ scales, the practical impact would be confined to organizations with substantial GPU resources. The explicit framing of IXC-2.5 as achieving GPT-4V level capabilities "with merely 7B LLM backend" (Abstract) signals that efficiency — not just capability — is a central contribution.

The Prior Landscape: Where Existing Open-Source Models Fall Short

The paper situates itself against three categories of prior work, identifying specific limitations in each:

1. LVLMs for Text-Image Conversation

Most existing open-source LVLMs (LLaVA, InstructBLIP, MiniGPT-4, Otter, CogVLM, Qwen-VL, InternLM-XComposer2) are designed for single-image, multi-round conversations. They take one image as input and engage in a dialogue about it. While some models (Flamingo, Qwen-VL, Mantis, VILA, Emu2) support multi-image inputs, the authors argue they do not provide a truly free-form long-contextual multi-turn multi-image interaction experience:

"IXC-2.5 focuses on providing a free-form long-contextual multi-turn multi-image interaction experience, which has not been addressed yet."

The key phrase is "free-form". Multi-image support in prior models is often constrained — images may need to be the same resolution, may not be interleaved with text naturally, or may not support the kind of extended multi-turn dialogue where earlier images are referenced many turns later. The MMDU benchmark (which the paper uses for evaluation and contributed to developing) was specifically designed to test this capability, underscoring that prior evaluation infrastructure also lagged.

2. High-Resolution Image Understanding

The paper identifies two existing strategies for enabling high-resolution understanding, each with limitations:

Strategy 1 — High-resolution visual encoders: Approaches like CogAgent, Mini-Gemini, Vary, Kosmos-2.5, and TextSquare train or use vision encoders that directly accept higher-resolution inputs. The limitation is computational: vision transformer FLOPs scale quadratically with resolution, creating a hard ceiling on how high resolution can go within a fixed compute budget.

Strategy 2 — Patchification: Approaches like Monkey, LLaVA-UHD, Sphinx, OtterHD, mPLUG-DocOwl, and TextMonkey crop high-resolution images into patches, process each patch with a low-resolution vision encoder (typically CLIP), and concatenate the resulting visual embeddings as input to the LLM. This enables arbitrary resolution handling but introduces fragmentation — information crossing patch boundaries can be lost, and the spatial relationships between patches are not inherently encoded.

IXC-2.5's predecessor, IXC2-4KHD, pioneered scaling open-source LVLMs to 4K+ resolutions using a dynamic partition strategy combined with a global-local format. However, it used a 490×490 ViT and employed multiple different cropping strategies depending on the image aspect ratio. IXC-2.5 unifies these into a single scaled identity strategy and upgrades the ViT to 560×560 resolution, providing finer per-patch granularity. The paper explicitly positions this as combining the strengths of both prior strategies:

"IXC-2.5 combines both solutions with a vision encoder trained with a resolution of 560×560 and a dynamic resolution solution proposed in IXC2-4KHD, resulting in further improvements."

3. Video Understanding

Prior video LVLMs have employed a diverse set of engineering solutions to handle the computational challenge of processing many frames: sparse sampling or temporal pooling (Video-ChatGPT, Valley, Video-LLaVA, Self-Chained), compressed video tokens (VideoChat, Chat-UniVi, LLaMA-VID, LongVLM, Video-LLaMA), memory banks (MovieChat, MovieChat+, MA-LMM), and language as a bridging modality (Video Recap, LangRepo, LLoVi). While effective within their design constraints, each approach involves a tradeoff — sparse sampling sacrifices temporal resolution, token compression discards visual detail, and memory banks introduce architectural complexity that complicates end-to-end training.

The paper's key observation is that these video-specific engineering solutions may not be necessary if the base model already handles ultra-high-resolution images and long contexts:

"video analysis can also be formulated to understand a high-resolution composite picture consisting of sampled video frames."

This is a unifying insight: if a model can already ingest and reason over a 96K-token context containing dozens of high-resolution image patches, then videos are just a special case of this capability — dense frame sampling concatenated into a single composite image. This eliminates the need for video-specific temporal modeling, token compression, or memory architectures. The claim is that what previously required specialized design can be handled by a sufficiently general multi-modal long-context architecture.

4. Webpage Generation

The paper identifies a gap between existing work on screenshot-to-code (Pix2Code, Sightseer, DCGen, Design2Code) and practical webpage creation. Prior models are trained primarily on synthetic datasets (WebSight v0.1/v0.2) that the authors critique as limited:

"the synthesized web page datasets have been critiqued for their simplicity and lack of diversity. These studies generally concentrate on the screenshot/sketch-to-code task."

Real-world webpages contain JavaScript for interactivity, diverse CSS frameworks (Tailwind, Bootstrap), and complex responsive layouts that synthetic datasets do not capture. Furthermore, prior work is evaluated almost exclusively on translating screenshots into code — a single modality conversion task. The paper extends this to instruction-aware webpage generation (creating a webpage from a natural language description, not just a screenshot) and resume-to-homepage (structuring personal information from a document into an aesthetically designed personal page), representing a broader vision of web composition as a multi-modal generation capability rather than a narrow translation task.

How This Paper Positions Itself

IXC-2.5 is positioned as a unifying framework rather than a point solution for any single task. The architecture is not novel in its individual components — it follows the established design of IXC2 with a ViT encoder, an LLM decoder, and Partial LoRA for alignment. What is novel is the combination of design decisions that yields versatility across previously disconnected capability dimensions.

The paper identifies long-context capability as the enabling mechanism for this unification. If a model can handle 24K–96K tokens of interleaved image-text input, then:

• Ultra-high resolution understanding becomes tractable because high-resolution images can be partitioned into many patches and processed as a long sequence.
• Fine-grained video understanding becomes a special case of ultra-high-resolution understanding — a video is just many frames concatenated into one composite image.
• Multi-turn multi-image dialogue becomes possible because the model can maintain coherence over long sequences containing many images and many turns of conversation history.
• Webpage/article generation becomes feasible because the model can attend to long input instructions and produce long structured outputs without truncation.

The training recipe — 24K context during pre-training and supervised fine-tuning, extendable to 96K at inference via RoPE extrapolation — is thus the linchpin that makes the unified architecture actually work across all these modalities. Prior models with similar architectures but shorter training contexts could not achieve the same versatility because the architectural capacity was bottlenecked by the effective attention window.

The paper also positions itself as providing a practical alignment pipeline for open-ended generation tasks (article composition) that goes beyond standard supervised fine-tuning. The combination of Chain-of-Thought prompt augmentation, reward model training on GPT-4o-labeled preferences, and DPO alignment addresses the acknowledged challenge that collecting high-quality text-image article data is "rare and expensive" and that "direct fine-tuning on scarce instruction data can lead to unstable responses." This preference alignment component is not claimed as a theoretical innovation — DPO and RLHF are well-established — but rather as a necessary engineering contribution to make the composition capability reliable enough for practical use.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

InternLM-XComposer-2.5 (IXC-2.5) is a multi-modal AI system that can look at images, watch videos, read documents, and have conversations about them, while also being able to write illustrated articles and generate functional webpage code. The core problem it solves is that existing open-source vision-language models are typically good at one or two things (e.g., answering questions about a single photo) but cannot handle the diverse mix of tasks that proprietary systems like GPT-4V manage — especially tasks requiring very long input sequences containing many images interleaved with text, or tasks requiring long structured outputs like HTML code or full articles. The "shape" of the solution is a single unified architecture trained with a 24,000-token context window on diverse data spanning ultra-high-resolution images, dense video frames, multi-image dialogues, webpage source code, and long-form articles, using the same underlying mechanism (a vision encoder feeding into a language model) for all tasks, with task-specific capabilities added via lightweight adapter modules rather than separate specialized models.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components:

1. Vision Encoder (OpenAI ViT-L/14, 560×560 resolution): Takes raw images (or video frames, or rendered webpage screenshots) and converts them into sequences of visual feature vectors — 400 tokens per 560×560 image patch.

2. Unified Dynamic Image Partition Module: Pre-processes any visual input — a single image, multiple images, or a video — into a set of 560×560 patches by resizing and tiling, producing a flat sequence of visual tokens interleaved with text tokens.

3. Large Language Model (InternLM2-7B): The core reasoning engine. It receives the interleaved sequence of visual and text tokens and produces output text (which can be natural language, HTML/CSS/JavaScript code, or article prose). Frozen during pre-training; partially unfrozen during fine-tuning.

4. Partial LoRA (Low-Rank Adaptation): A set of small, trainable weight matrices attached to the linear layers of the frozen LLM decoder blocks (rank 256 during pre-training, rank 512 for webpage generation, rank 256 for article DPO). This adapts the LLM to process visual inputs and produce task-specific outputs without modifying its original language capabilities.

5. Task-Specific LoRA Modules (Webpage, Article): Separately trained LoRA parameters specialized for webpage generation (HTML/CSS/JS code from screenshots, instructions, or resumes) and article composition (long-form illustrated text with DPO alignment for quality and stability).

Information flows as follows: raw visual input → Unified Dynamic Image Partition (resize, pad, split into patches, assign positional indices) → Vision Encoder (each patch becomes 400 tokens) → interleave visual tokens with text tokens (user prompt, conversation history) → LLM + Partial LoRA (produce output tokens autoregressively) → task-specific LoRA (if generating webpages or articles, swap in the specialized adapter weights) → output text. For video, the frames are first concatenated into a single composite image, then processed through the same pipeline as any ultra-high-resolution image. For multi-image dialogue, each image is independently partitioned, indexed, and interleaved with the conversation text.

3.3 Roadmap for the Deep Dive

• First, the Unified Dynamic Image Partition strategy — how arbitrary-resolution images, multi-image inputs, and videos are all converted into a standardized sequence of tokens, since this is the foundation that enables the model to handle diverse visual inputs through a single pipeline.

• Second, the vision encoder upgrade from 490×490 to 560×560 and the token budget implications, since this determines the granularity at which the model perceives visual content and directly impacts high-resolution understanding quality.

• Third, the pre-training phase — datasets, optimization settings, and the Partial LoRA adapter — since this is where visual tokens are first aligned with the LLM's embedding space, establishing the foundation for all downstream capabilities.

• Fourth, the supervised fine-tuning phase — the expanded dataset mixture, the increased maximum partition count (H=24), and the per-component learning rate strategies — since this is where the model acquires its diverse task capabilities, including video understanding, multi-image dialogue, and structured output generation.

• Fifth, the webpage generation training pipeline — the screenshot-to-code data processing, the instruction-aware webpage data synthesis, the resume-to-homepage pipeline, and the LoRA training recipe — since this is the most architecturally distinct capability, requiring specialized data construction and a separate LoRA module.

• Sixth, the article composition pipeline — the Chain-of-Thought prompt augmentation, reward model training, preference data collection, and DPO alignment — since this addresses the qualitatively different challenge of reliable long-form creative generation with multi-modal elements, and introduces the only RL-style optimization in the paper.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and training methodology paper whose core idea is that a single LVLM architecture, when trained with a sufficiently long context window on sufficiently diverse multi-modal data, can match proprietary API performance across a wide range of tasks without task-specific architectural modifications — and that the key enabling factor is a unified input processing strategy that converts all visual inputs (images of any resolution, multiple images, videos) into a single interleaved token sequence format.

---

3.4.1 Unified Dynamic Image Partition Strategy

The central technical mechanism that enables IXC-2.5 to handle diverse visual inputs — single images of arbitrary resolution, multiple images of different sizes, and videos with many frames — is the Unified Dynamic Image Partition strategy. This is an evolution of the approach introduced in IXC2-4KHD, with a key simplification: the paper unifies the different cropping strategies used in the predecessor into a single "scaled identity strategy."

The core idea is that any visual input that is too large to be processed natively by the vision encoder (which has a fixed 560×560 input resolution) is resized and partitioned into a grid of non-overlapping 560×560 patches. Each patch is independently encoded by the ViT into 400 visual tokens. These token sequences are then concatenated and interleaved with text tokens to form the input to the LLM.

The partition algorithm is specified by Equations 1–3:

$pw_1 \times \lceil pw_1 \times h/w \rceil \leq H$

$pw_2 = \lceil w \times s / 560 \rceil$

$pw = \min(pw_1, pw_2) ; ph = \lceil pw \times h/w \rceil$

where:

• $h, w$ are the original image height and width in pixels,
• $H$ is the maximum allowed number of patches (set to 12 during pre-training, 24 during fine-tuning),
• $s$ is a scale factor that controls the minimum effective resolution,
• $pw, ph$ are the computed numbers of patches along the width and height dimensions respectively.

What these equations compute: Given an image of arbitrary size and aspect ratio, the algorithm determines how many 560×560 patches to split it into along each dimension, subject to the constraint that the total number of patches does not exceed $H$. Equation 1 finds the largest width $pw_1$ such that a grid of $pw_1 \times \lceil pw_1 \times h/w \rceil$ patches stays within the budget $H$ — this ensures the partitioning respects the image's original aspect ratio (the height patches scale proportionally with the width patches based on the aspect ratio $h/w$). Equation 2 computes an alternative width $pw_2$ based on a minimum scale factor $s$ — this ensures that even small images are encoded at sufficient effective resolution. Equation 3 takes the minimum of these two widths to stay within both constraints, then computes the corresponding height $ph$ to preserve the aspect ratio.

The image is then resized to $[ph \times 560, pw \times 560]$ pixels (with padding if necessary) and partitioned into $ph \times pw$ patches, each exactly 560×560 pixels. Each patch yields 400 visual tokens from the ViT.

Why this form: The key property is that the algorithm adapts the number of patches to both the image's resolution and its aspect ratio. A very wide but short image (e.g., a panoramic screenshot) will use more patches horizontally than vertically, preserving the spatial layout in the token sequence. A square image will use roughly equal patches in both dimensions. The $\min(pw_1, pw_2)$ in Equation 3 provides a tradeoff: for large images, $pw_1$ (Equation 1) is usually the binding constraint and ensures the patch budget is respected; for small images, $pw_2$ (Equation 2) ensures they are still upscaled to a minimum effective resolution based on the scale factor $s$. Without Equation 2, a small image might use only one patch and lose fine-grained detail; without Equation 1, a large image would exceed the computational budget.

Global-Local Format. After partitioning an image into patches, the system constructs a "global-local" input format. A global view of the entire image is created by resizing the original image to a single 560×560 square (regardless of original aspect ratio). This global view provides coarse contextual information about the full image. The local views are the individual patches from the partition step, which provide high-resolution detail. The visual tokens from the global view are placed first in the sequence, followed by the tokens from each local patch, with special separator tokens (newlines) between them. This ordering allows the LLM to first form a coarse understanding of the whole image, then attend to specific details in individual patches.

Multi-Image Handling. For multiple images in a single conversation turn, each image is independently partitioned using the same algorithm. Each image is assigned an index (e.g., <IMAGE 1>, <IMAGE 2>, etc.) and its visual tokens are placed in the input sequence at the position where that image is referenced in the text, creating an interleaved text-image-text-image format. This is a deliberate design choice over concatenating all images first and all text second — the interleaved format preserves the natural reference relationships between specific text segments and specific images, which matters for multi-turn dialogue where a user might say "What about this one?" while referring to an image they showed earlier.

Video Processing. The paper's approach to video is a direct consequence of the unified partition strategy: instead of designing a separate video processing pipeline, video frames are concatenated along their short side into a single ultra-high-resolution composite image. Specifically, frames are sampled from the video at dense intervals, then tiled side-by-side or stacked vertically to form one very large image. This composite image is then processed through the exact same Unified Dynamic Image Partition pipeline as any other high-resolution image — it is partitioned into 560×560 patches based on its total dimensions, with the patch budget $H=24$ (during fine-tuning) constraining how many total patches can be used.

Additionally, each frame has its frame index written onto the image (numerical overlay) before concatenation, providing an explicit temporal position signal that the ViT can encode. This means the visual tokens for each frame carry both spatial and temporal position information — the spatial position comes from where the patch falls within the composite image, and the temporal index comes from the overlaid number.

For video training, IXC-2.5 uses at most 64 frames per video, and these frames are concatenated and processed as a single large image with at most $H=24$ patches during fine-tuning.

Why unify image and video processing: The explicit rationale is that "video analysis can also be formulated to understand a high-resolution composite picture consisting of sampled video frames." This eliminates the need for video-specific architectural components (3D convolutions, temporal attention, memory banks, token compression) that prior video LVLMs rely on. The tradeoff is that the model must learn to interpret spatial layout as a proxy for temporal order — frames arranged left-to-right or top-to-bottom in the composite image correspond to temporal progression in the video. The frame index overlays make this relationship explicit. The advantage is architectural simplicity and the ability to train on mixed image and video data without modality-specific processing paths.

---

3.4.2 Vision Encoder: Resolution Upgrade and Token Budget

IXC-2.5 reuses the vision encoder from its predecessor IXC2 — an OpenAI ViT-L/14 architecture — but increases its native input resolution from 490×490 to 560×560 pixels. This seemingly modest change has significant implications for the token budget and the model's visual acuity.

Token budget per patch: The ViT-L/14 uses 14×14 pixel patches as its basic processing unit. At 490×490 resolution, each image produces $(490/14) \times (490/14) = 35 \times 35 = 1225$ visual tokens. At 560×560 resolution, each image produces $(560/14) \times (560/14) = 40 \times 40 = 1600$ visual tokens. However, the paper states that each sub-image (patch) yields 400 tokens — this suggests that the ViT's output is further processed (possibly through a pooling or projection layer that reduces the token count) before being fed to the LLM, or that the quoted 400 refers to a different tokenization granularity. The exact reduction mechanism from 1600 raw ViT tokens to 400 effective tokens per patch is not detailed in the paper.

Impact on high-resolution understanding: The resolution increase means each 560×560 patch encodes finer spatial detail than the predecessor's 490×490 patches. When an A4 document or a high-resolution photograph is partitioned into many patches, each patch captures details (small text, fine lines, subtle textures) that would be blurred at the lower resolution. This directly benefits OCR tasks (TextVQA, DocVQA, OCRBench), chart understanding (ChartQA), and form parsing (DeepForm, TableFact) — all tasks where IXC-2.5 shows strong performance in Section 4.

Layer-wise learning rate decay (LLDR) for the vision encoder: During both pre-training and fine-tuning, the vision encoder uses a layer-wise learning rate decay with factor 0.90. This means that lower layers of the ViT (which encode low-level features like edges and textures) are updated more slowly than higher layers (which encode semantic features). The decay factor of 0.90 means each layer's learning rate is 90% of the layer above it. This preserves the pre-trained visual representations learned during the ViT's original CLIP training, preventing catastrophic forgetting of general-purpose visual features while allowing higher-level representations to adapt to the specific needs of the LVLM (e.g., encoding text in images, recognizing document layouts).

Why increase resolution: The authors' predecessor IXC2-4KHD already demonstrated that increasing the effective resolution via patchification dramatically improves performance on document and OCR tasks. The 490→560 upgrade is an incremental refinement: rather than increasing the patch budget $H$ (which would increase the total token count and thus the computational cost), they increase the per-patch resolution so that each patch carries more information for the same token budget. This is a more compute-efficient path to higher visual acuity than simply using more patches at lower resolution.

---

3.4.3 Pre-Training Phase: Aligning Visual Tokens with the LLM

The pre-training phase establishes the fundamental connection between visual features and the language model's embedding space. During this phase, the LLM (InternLM2-7B) is completely frozen while the vision encoder and Partial LoRA parameters are trained.

Dataset composition (Table 1): The pre-training data is organized around three objectives:

• General Semantic Alignment: Large-scale image-caption datasets (ShareGPT4V-PT, COCO, Nocaps, TextCaps, LAION, SBU, CC 3M, ALLaVA) that teach the model to associate visual content with textual descriptions. These are standard web-scale datasets used in most LVLM pre-training recipes.

• World Knowledge Alignment: Concept Data (from the InternLM-XComposer paper) that links visual concepts to factual knowledge, enabling the model to recognize and reason about specific entities, objects, and scenes.

• Vision Capability Enhancement: Specialized datasets targeting visual recognition skills that general captioning may underrepresent — WanJuan (diverse multi-modal), Flicker (detailed captions), MMC-Inst (chart understanding), plus OCR-focused datasets (RCTW-17, CTW, LSVT, ReCTs, ArT) that teach the model to recognize and read text in images.

Key hyperparameters (Section 3.3):

• Batch size: 4096
• Epochs: 2
• Maximum number of patches ($H$): 12
• Vision encoder resolution: 560×560 (upgraded from the 490×490 IXC2 ViT checkpoint)
• Partial LoRA rank: 256 for all linear layers in the LLM decoder block
• Learning rate schedule: linear warmup for the first 1% of training steps to a maximum of 2×10⁻⁴, followed by cosine decay to 0
• Vision encoder LLDR: 0.90 decay factor

Partial LoRA design: Low-Rank Adaptation (LoRA) inserts small, trainable low-rank matrices into the linear layers of the frozen LLM. Instead of updating the full weight matrix $W \in \mathbb{R}^{d \times k}$, LoRA learns two smaller matrices $A \in \mathbb{R}^{d \times r}$ and $B \in \mathbb{R}^{r \times k}$ (where $r \ll \min(d, k)$) such that the effective weight becomes $W + AB$. During training, only $A$ and $B$ receive gradient updates. IXC-2.5 applies LoRA to all linear layers in the LLM decoder block with rank $r=256$. The "Partial" designation means LoRA is applied only to the attention and feedforward layers within each decoder block, not to the embedding layer or output head.

Why freeze the LLM during pre-training: This is standard practice in LVLM training (following LLaVA, InstructBLIP, and IXC2). The pre-training objective is to learn a mapping from visual features to the LLM's existing embedding space — not to modify the LLM's language capabilities. Freezing the LLM preserves its pre-trained knowledge and prevents the relatively small pre-training dataset from corrupting its language modeling abilities. The Partial LoRA serves as a lightweight adapter that learns to "translate" visual information into representations the LLM can process, without requiring updates to the LLM's core parameters.

Why rank 256 for Partial LoRA: The rank determines the capacity of the adaptation. Higher rank means more trainable parameters and greater representational power, at the cost of memory and compute. Rank 256 on all decoder layers provides substantial capacity (millions of parameters) while remaining far smaller than the full LLM (7B parameters). This is sufficient to learn the complex mapping from diverse visual features to the LLM's representational space.

Learning rate warmup and cosine decay: The linear warmup (first 1% of steps) prevents large, destabilizing gradient updates early in training when the randomly initialized LoRA weights have high variance. The cosine decay to zero provides smooth convergence without requiring manual learning rate step-downs.

---

3.4.4 Supervised Fine-Tuning Phase: Acquiring Diverse Task Capabilities

After pre-training aligns visual and language representations, supervised fine-tuning (SFT) teaches the model to perform specific tasks by training on instruction-following examples. The SFT phase uses a substantially expanded dataset and trains all components (vision encoder, Partial LoRA, and the LLM itself, albeit at a reduced learning rate).

Dataset composition (Table 2): The SFT data spans 12 task categories, each with specific datasets:

• Captioning: ShareGPT4V, COCO, Nocaps
• General QA: VQAv2, GQA, OK-VQA, Visual Dialog (VD), RefCOCO-based dialogue (RD), VSR, ALLaVA-QA
• Multi-Turn QA: MMDU (multi-turn multi-image dialogue)
• Science QA: AI2D (diagrams), ScienceQA, TextbookQA, IconQA (abstract diagrams)
• Chart QA: DVQA, ChartQA, ChartQA-AUG
• Math QA: MathQA, Geometry3K, TabMWP (table math), CLEVR-MATH, Super-CLEVR
• World Knowledge QA: A-OKVQA, KVQA, ViQuAE
• OCR QA: TextVQA, OCR-VQA, ST-VQA
• High-Resolution Document QA: InfoVQA, DocVQA, TabFact, WikiTableQuestion (WTQ), DeepForm, Visual MRC
• Video: ShareGPT4Video, ActivityNet
• Conversation: LLaVA-150k, LVIS-Instruct4V, ShareGPT (English and Chinese), InternLM-Chat

Key hyperparameters (Section 3.4):

• Batch size: 2048
• Training steps: 4000
• Maximum patches ($H$): 24 (doubled from pre-training to handle very large images and videos with up to 64 frames)
• Maximum training context window: 24,000 tokens — the MMDU dataset is specifically noted as reaching this limit, implying that the most complex multi-turn multi-image conversations use the full context window
• Maximum learning rate: 5×10⁻⁵
• Vision encoder LLDR: 0.90 (same as pre-training)
• LLM learning rate scale factor: 0.2 (the LLM's effective learning rate is $0.2 \times 5 \times 10^{-5} = 1 \times 10^{-5}$)

Weighted data sampling: Data from different sources are sampled with weights proportional to the number of examples from each source. This prevents small datasets (e.g., specialized chart QA) from being drowned out by large datasets (e.g., general conversation), while still giving more training emphasis to tasks with more available data.

Why separate learning rates per component: The LLM receives a base learning rate of $5 \times 10^{-5}$ multiplied by a scale factor of 0.2, giving an effective rate of $1 \times 10^{-5}$ — one-fifth the rate used for the vision encoder and Partial LoRA. The paper states this "slows down the update of the LLM, achieving a balance between preserving its original capabilities and aligning it with vision knowledge." In practice, this means the vision encoder and LoRA adapt more aggressively to the new task data, while the LLM's language modeling and reasoning capabilities are preserved with gentle fine-tuning. This is a reversal from the pre-training phase where the LLM was entirely frozen — now it is partially unfrozen but constrained.

Why increase $H$ from 12 to 24: The pre-training phase uses at most 12 patches per image. Fine-tuning doubles this to 24, which allows processing of larger images (up to $24 \times 560 \times 560$ pixels of effective area) and video with more frames. This is necessary because the SFT phase includes high-resolution document tasks (DocVQA, InfoVQA, DeepForm) that require fine-grained reading of dense text, and video tasks (using up to 64 frames concatenated and then partitioned into up to 24 patches). The 24-patch budget is a pragmatic choice — doubling again to 48 would provide even higher resolution but would also double the token count and the associated computational cost.

24K context window training: The model is trained with a maximum context window of 24,000 tokens. This is significant because it means the model learns during training to attend over sequences containing dozens of image patches (each 400 tokens) plus thousands of text tokens. The MMDU dataset — which features multi-turn dialogues with multiple images — is the training source that pushes the context window to this limit during SFT. At inference time, the context can be extended to 96K tokens via RoPE (Rotary Position Embedding) extrapolation, a technique that allows position encodings to generalize beyond the trained context length by scaling the rotation frequencies.

Why train with 24K context rather than 96K directly: Training with 96K context would dramatically increase the computational cost (quadratic in sequence length for attention). The 24K→96K gap is bridged by RoPE extrapolation, which adjusts the rotary position embeddings at inference time so that attention patterns that worked for relative positions up to 24K generalize to relative positions up to 96K. This is an established technique (documented in the cited LocalLLaMA reference) that has been validated on pure-text LLMs but is applied here to the more complex setting of interleaved visual-text sequences.

Audio processing (inference only): The web demo supports audio input and output using external tools — Whisper for speech-to-text and MeloTTS for text-to-speech. These are not part of the trained model; they are pre-processing and post-processing steps that convert audio to text before feeding it to IXC-2.5, and convert the text output back to audio. This is explicitly separate from the model training pipeline.

---

3.4.5 Webpage Generation Training Pipeline

Webpage generation is treated as a specialized skill requiring its own training data and a separate LoRA module trained on top of the SFT model. The pipeline has three stages: screenshot-to-code training, instruction-aware webpage generation data synthesis, and resume-to-homepage data synthesis.

Stage 1: Screenshot-to-code training data

The model is trained on three datasets to learn the basic skill of translating visual webpage designs into functional code:

• WebSight v0.1 and v0.2: Existing synthetic datasets containing screenshot-HTML pairs, generated by rendering synthetic webpages and capturing their screenshots. The paper critiques these as limited in "simplicity and lack of diversity" but uses them as a starting point.

• Stack v2 (processed): Real-world web data from The Stack v2 dataset, which contains source code from public repositories. The authors apply a multi-step cleaning pipeline to convert this into screenshot-code training data:

  1. Combine CSS and HTML code from the same repository into single files
  2. Remove all comments, JavaScript code, and external links (keeping only the structural HTML and styling CSS)
  3. Eliminate CSS styles that are not referenced by any HTML element (dead code removal)
  4. Render each cleaned HTML+CSS file in a browser and capture a screenshot
  5. Discard screenshots that failed to render properly (blank pages, broken layouts)
  6. Use the IXC2-4KHD model (the predecessor) to assess the visual quality of the rendered webpages — this is an automated quality filter that rates how well-structured and visually coherent each page appears
  7. Retain approximately 250,000 high-quality webpages after filtering

The result is a dataset that combines synthetic webpages (WebSight) with real-world webpages (Stack v2), addressing the diversity limitation the authors identified in prior work. The real-world data captures the messy, varied nature of actual web development — diverse layouts, non-standard HTML patterns, varying CSS quality — that synthetic data alone cannot represent.

LoRA training for screenshot-to-code: A separate LoRA module with rank 512 is trained. The higher rank (compared to 256 used for the Partial LoRA during pre-training) reflects the complexity of the code generation task — the model must learn HTML syntax, CSS properties, and the mapping from visual layout to code structure, which requires more adaptation capacity than general visual-language alignment. Training settings:

• Batch size: 512
• Epochs: 1
• Learning rate schedule: linear warmup (first 1% of steps) to 1×10⁻⁴, then cosine decay to 0

Stage 2: Instruction-aware webpage generation

Screenshot-to-code is a direct translation task — given a visual input, produce the corresponding code. Instruction-aware webpage generation is more ambitious: given a natural language instruction describing desired webpage characteristics (e.g., "Generate a website for a research institution with a blue navigation bar and sections for research projects"), produce a functional webpage. This requires the model to design the layout, choose appropriate styling, and generate code that matches the semantic intent of the instruction.

Since no existing dataset pairs free-form instructions with webpage code, the authors synthesize the training data using a multi-model pipeline:

1. GPT-4 generates diverse instructions: GPT-4 is prompted to produce varied instructions and concepts for webpage creation, covering different types (portfolios, landing pages, dashboards, blogs), styles (minimalist, corporate, creative), and layouts (single-column, grid, sidebar). This produces instructions that specify high-level design goals rather than pixel-level layouts.

2. Claude-3-sonnet generates the actual code: Each instruction is fed to Claude-3-sonnet, which generates the corresponding HTML, CSS, and JavaScript code. The choice of Claude-3-sonnet over GPT-4 for this step is not explicitly motivated, but the staggered use of different proprietary models likely provides diversity in the generated code styles.

3. Tailwind CSS replaces traditional CSS: To reduce the verbosity of the generated code (CSS files can be very long for complex layouts), the system uses Tailwind CSS — a utility-first CSS framework where styling is applied via short class names directly in HTML elements. This makes the output more compact and easier for the LLM to generate accurately.

This pipeline produces 18,000 high-quality, instruction-aware webpage samples. This is a relatively small dataset compared to the 250K screenshot-to-code samples, so it is used for a second stage of fine-tuning.

Stage 3: Resume-to-homepage

This is a more specialized task: given a resume document (containing a person's education, experience, skills, etc.), generate a personal homepage that structures this information into a visually appealing and interactive webpage.

The data synthesis pipeline follows a similar multi-model approach:

1. GPT-4 generates resume ideas for diverse personas (researchers, students, software engineers, designers, executives) and produces the actual resume content in markdown format. The ideas ensure diversity in profession, career stage, and background.

2. GPT-4 generates homepages from these resumes. The model is tasked with creating well-structured, visually appealing homepages that present the resume information in an engaging format.

3. Claude-3-sonnet adds JavaScript interactivity based on the generated HTML code. This includes interactive elements like navigation menus, scrolling animations, contact forms, portfolio galleries, and section toggles — features that make the homepage a functional website rather than a static document.

The total dataset size is 2,000 samples — significantly smaller than the instruction-aware dataset, reflecting the narrower scope of the task.

Fine-tuning for stages 2 and 3: The LoRA model from the screenshot-to-code stage is further fine-tuned on the combined instruction-aware and resume-to-homepage datasets for 10 epochs. All other training settings (batch size 512, learning rate schedule) remain the same as the screenshot-to-code stage. The higher number of epochs (10 vs. 1) reflects the small dataset size — more passes are needed to learn the new task patterns from limited examples.

Why use separate LoRA modules for webpage generation: The webpage generation capability is kept in separate LoRA weights rather than being merged into the main model. This design choice preserves the base model's general visual-linguistic capabilities while allowing the webpage LoRA to develop specialized syntax knowledge (HTML, CSS, JavaScript grammar) that is very different from natural language. When a user requests webpage generation, the system swaps in the webpage LoRA weights; for general conversation, it uses the base Partial LoRA. This modular approach prevents the specialized code generation training from degrading the model's general conversation abilities — a form of catastrophic forgetting prevention through architectural isolation.

---

3.4.6 Article Composition Pipeline: Supervised Fine-Tuning, Reward Modeling, and DPO Alignment

Generating high-quality text-image articles (illustrated essays, stories, poetry) presents a qualitatively different challenge from the tasks above. The output space is open-ended and creative rather than factual, and quality is subjective. The paper identifies two specific challenges: (1) high-quality text-image article data is "rare and expensive" to collect, and (2) "direct fine-tuning on scarce instruction data can lead to unstable responses" — meaning that simply training on a small number of example articles produces a model that sometimes generates good output and sometimes fails catastrophically.

The solution is a multi-stage alignment pipeline: augmented supervised fine-tuning → reward model training → preference data collection → Direct Preference Optimization (DPO).

Stage 1: Supervised Fine-Tuning with Prompt Augmentation

The starting point is the SFT model $\pi$ from Section 3.4 (the general-purpose model after supervised fine-tuning) and a collection of approximately 5,000 instruction-tuning data samples $\mathcal{D}$ from IXC2, focused specifically on article writing.

The data scarcity problem is addressed by augmenting the prompts using Chain-of-Thought (CoT):

1. The SFT model $\pi$ is used to rewrite the original prompts from $\mathcal{D}$ using CoT, generating step-by-step writing prompts. For example, an original prompt like "Write an article about climate change" might be expanded into a structured prompt specifying sections (introduction, environmental impacts, economic impacts, mitigation strategies, conclusion) and stylistic guidance for each section.

2. These augmented prompts form an expanded dataset $\mathcal{D}^*$. The paper observes that "the SFT model is more effective in generating long-form responses when using these augmented prompts" — the explicit step-by-step structure helps the model produce coherent, well-organized long articles rather than rambling or losing track of the overall structure.

3. The model is fine-tuned on $\mathcal{D}^*$ via LoRA with rank 256 to produce the reference model $\pi_{\text{ref}}$. This establishes a starting point for the alignment pipeline — $\pi_{\text{ref}}$ can generate articles, but with unstable quality.

Why CoT augmentation helps: Long-form generation from short prompts is an underspecified problem — there are many possible valid articles for any given topic. The model lacks guidance on structure, style, and scope. By expanding prompts into step-by-step outlines, the CoT augmentation reduces this underspecification, giving the model a clearer target and reducing the variance in output quality. This is a practical observation rather than a novel technique.

Stage 2: Preference Data Collection

With $\pi_{\text{ref}}$ in hand, the next step is to collect data on which article outputs are preferred — the signal needed for alignment.

1. Response generation: For each prompt in $\mathcal{D}^*$, $\pi_{\text{ref}}$ generates diverse responses using different random seeds. The use of multiple seeds ensures that the model explores the space of possible articles rather than always producing the same output. This yields a collection of 80,000 prompt-response pairs.

2. GPT-4o labeling: From these 80,000 pairs, 2,000 responses are selected and labeled by GPT-4o with "chosen" or "rejected" decisions, along with the reasons for each decision. The reasons provide interpretability — they explain why one response is preferred over another (e.g., better structure, more engaging prose, more relevant illustrations, fewer factual errors). These 2,000 labeled examples serve as reward modeling data.

3. Reward model training: A reward model $\pi_{\text{rm}}$ is trained on these 2,000 labeled pairs. The reward model shares the same architecture as $\pi_{\text{ref}}$ — it is an identically structured LVLM, but trained to output a scalar reward value rather than generated text. Training details for the reward model are not specified in the paper, but the standard approach would be to train a linear head on top of the final hidden state to predict the preference (chosen > rejected), using a Bradley-Terry or similar pairwise ranking loss.

4. Automatic labeling: The trained reward model $\pi_{\text{rm}}$ is then used to automatically label the remaining 78,000 prompt-response pairs (those not labeled by GPT-4o) as chosen or rejected. This leverages the expensive GPT-4o labels to train a cheap automatic labeler, enabling a much larger preference dataset than could be practically collected from the API.

5. Preference dataset construction: The chosen and rejected responses are paired to form the DPO training data $\mathcal{D}_p = \{x, y_w, y_l\}$, where $x$ is the prompt, $y_w$ is the chosen (preferred) response, and $y_l$ is the rejected (dispreferred) response. The final dataset contains 30,000 preference pairs.

Why use GPT-4o for initial labeling: Article quality assessment is subjective and domain-specific. GPT-4o, as a capable proprietary model, provides reasonable quality judgments without requiring human annotators (which would be prohibitively expensive at scale). The "reasons" accompanying the decisions serve two purposes: they provide training signal for the reward model (learning not just what is preferred but why), and they provide a sanity check — the authors can verify that GPT-4o's judgments align with human notions of article quality.

Why train a reward model rather than labeling everything with GPT-4o: Cost and rate limits. Labeling 80,000 responses with GPT-4o would be expensive and slow. Training a reward model on 2,000 GPT-4o labels and using it to label the remaining data is a standard cost-amortization strategy.

Stage 3: Direct Preference Optimization (DPO) Alignment

DPO is used to align $\pi_{\text{ref}}$ with the preference data $\mathcal{D}_p$. The DPO objective is:

$\mathcal{L}_{\text{DPO}}(\pi_\theta, \pi_{\text{ref}}) = \mathbb{E}_{x, y_w, y_l \sim \mathcal{D}_p} \left[ -\log \sigma \left( \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)} \right) \right]$

where:

• $\pi_\theta$ is the policy being optimized (the DPO model),
• $\pi_{\text{ref}}$ is the reference policy (the SFT model from Stage 1),
• $\sigma$ is the logistic sigmoid function $\sigma(z) = 1/(1 + e^{-z})$,
• $\beta$ is a temperature parameter controlling the strength of the preference signal (higher $\beta$ means stronger adherence to preferences),
• $\frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)}$ is the ratio of the current policy's probability of the chosen response to the reference policy's probability — this measures how much the policy has shifted towards preferring $y_w$,
• Similarly for $y_l$, the rejected response.

What this equation computes in operational terms: For each prompt $x$ in the preference dataset, the model computes:

1. The log-ratio of probabilities for the chosen response $y_w$ under the current policy versus the reference policy: $\log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)}$. If the current policy assigns higher probability to $y_w$ than the reference did, this term is positive.
2. The same log-ratio for the rejected response $y_l$: $\log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}$. If the current policy assigns higher probability to $y_l$ than the reference did, this term is also positive.
3. The difference between these two log-ratios, scaled by $\beta$: $\beta[\log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}]$. This score is high when the policy has increased its relative preference for the chosen response over the rejected response, compared to the reference.
4. The sigmoid of this difference, negated and log-transformed: $-\log \sigma(\cdot)$. This is minimized when the difference is large and positive (the policy strongly prefers $y_w$ over $y_l$), and penalized when the difference is small or negative.

The expectation over the dataset averages this loss across all preference pairs.

Why this form: DPO directly optimizes the policy from preference data without training a separate reward model or using reinforcement learning (RL). The key insight of DPO is that the optimal policy under a Bradley-Terry preference model can be expressed in terms of the policy's own probabilities and the reference policy's probabilities, eliminating the need for the explicit reward maximization step that RLHF/PPO requires. The $\beta$ parameter controls how far the policy is allowed to deviate from the reference: larger $\beta$ means the policy can shift more aggressively to favor chosen responses, but also risks overfitting to the preference data. The reference policy $\pi_{\text{ref}}$ in the denominator acts as a regularizer — the policy is penalized for deviating too far from the SFT model, which prevents it from "hacking" the preference signal by assigning extremely high probability to chosen responses in ways that degrade general language quality.

Implementation details: The DPO model $\pi_\theta$ is trained using LoRA with rank 256. The paper makes a notable observation about the training dynamics:

"We observe that our model tends to prioritize minimizing the likelihood of dis-preferred responses $y_l$ over maximizing the likelihood of preferred responses $y_w$ to avoid generating inappropriate or low-quality content."

This means the DPO training is asymmetric — the model learns more from being pushed away from bad outputs than from being pulled toward good ones. This is a property of the DPO objective: when the policy already assigns reasonably high probability to $y_w$ (which is often true since $\pi_{\text{ref}}$ was trained on similar data), the gradient signal for increasing $\pi_\theta(y_w|x)$ is weak, while the signal for decreasing $\pi_\theta(y_l|x)$ can be strong if the reference policy assigned non-negligible probability to the rejected response. This asymmetry is beneficial for article generation because the primary failure mode is not producing mediocre articles (which the SFT model can do) but producing actively bad articles with factual errors, poor structure, or inappropriate content.

Why use DPO rather than PPO (the standard RLHF approach): DPO is simpler to implement (no separate reward model training at inference time, no PPO optimization with its sensitivity to hyperparameters), more stable (the loss is a straightforward classification-style objective rather than a policy gradient), and achieves comparable or better alignment results in many settings. For a system like IXC-2.5 where the alignment is applied to a specific capability (article generation) rather than the entire model, the simplicity of DPO is particularly valuable — it avoids the complexity of running PPO on a large multi-modal model.

---

3.4.7 Summary of Architectural and Training Design Choices

Why a unified architecture across all tasks: The paper's central claim is that long-context training plus diverse data enables a single model to handle tasks that previously required specialized architectures. The design choices consistently reflect this philosophy: video uses the same image partition pipeline rather than a separate video encoder; webpage generation uses the same LLM backbone with swapped LoRA weights rather than a separate code generation model; multi-image dialogue uses the same interleaved format as single-image conversation rather than a specialized multi-image attention mechanism. The tradeoff is that some tasks might be handled more efficiently with specialized architectures (e.g., 3D convolutions for video), but the unified approach provides engineering simplicity, shared representational learning across modalities, and the ability to compositionally combine capabilities (e.g., discussing a webpage screenshot while referencing earlier images in the conversation).

Why Partial LoRA rather than full fine-tuning: Full fine-tuning of a 7B-parameter model on diverse task data risks catastrophic forgetting — the model might lose its pre-trained language capabilities as it adapts to visual tasks. Partial LoRA constrains the adaptation to low-rank updates, preserving the LLM's original weights and allowing the model to fall back on its language capabilities when visual inputs are absent. The separate LoRA modules for webpage generation and article composition extend this principle: by isolating task-specific capabilities in separate low-rank weights, the base model's general abilities are protected from interference.

Why train with 24K context, extrapolate to 96K: This is a compute efficiency decision. The quadratic cost of self-attention means that training with 96K context would be approximately $(96/24)^2 = 16$ times more expensive per token than training with 24K context (ignoring optimizations like flash attention). The RoPE extrapolation approach has been validated on text-only LLMs and is applied here to visual-text sequences, but its effectiveness on interleaved multi-modal data is an empirical finding rather than a guaranteed property — dense image token sequences may have different positional sensitivity than text-only sequences. The paper demonstrates that the extrapolation works in practice (the model handles long multi-image dialogues and dense video frames at inference), but does not ablate the extrapolation quality at different context lengths.

4. Key Insights and Innovations

Innovation 1: Long Context as the Enabling Mechanism for Multi-Modal Unification — Not an Architectural Afterthought

The dominant paradigm in open-source LVLM development has been to treat long-context capability as a capacity to add to an existing model (via extrapolation or sparse attention), not as the fundamental design constraint around which the entire training pipeline should be organized. Prior multi-modal models that support multiple images (Flamingo, Qwen-VL, Mantis, VILA) or high-resolution inputs (Monkey, LLaVA-UHD, mPLUG-DocOwl) typically bolt these capabilities onto architectures originally designed for single-image, short-context interactions. The result is models that can handle long inputs but were not trained to do so natively — the positional encodings may not have seen long interleaved sequences during training, the attention patterns may not have learned to route information across distant image-text boundaries, and the training data mixture may not include sufficiently many long-context examples to make this behavior reliable.

IXC-2.5 inverts this relationship. The paper's core conceptual move is to treat 24K-token native training with interleaved image-text data as the foundation, and then derive the model's diverse capabilities — ultra-high-resolution understanding, fine-grained video analysis, multi-turn multi-image dialogue, long-form article generation — as natural consequences of this foundation rather than as separate features requiring specialized architectures. This is visible in the training design: the MMDU dataset specifically pushes the model to the full 24K context limit during supervised fine-tuning, meaning the model encounters training examples where it must maintain coherent attention across dozens of image patches and thousands of text tokens spanning multiple conversation turns. The 96K inference capability via RoPE extrapolation is then presented as an extension of a capability the model already possesses, not as a zero-shot stretch from a short-context checkpoint.

This reframing matters because it changes what researchers should prioritize when building versatile LVLMs. The standard recipe has been: pick a vision encoder, pick an LLM, pick an alignment strategy, train on instruction data. IXC-2.5's architecture is not novel in its components — ViT-L/14, InternLM2-7B, Partial LoRA with rank 256 are all established choices. What is novel is the organizing principle: design the training context window first, ensure the data mixture saturates it, and verify that all target capabilities are achievable within that context budget. If this principle generalizes, it suggests that the path to closing the versatility gap with proprietary APIs runs through native long-context training, not through larger models or more complex architectures.

The evidence for this claim is distributed rather than concentrated in a single ablation: the model's strong performance on structurally diverse benchmarks — 69.1 on MVBench (video), 90.9 on DocVQA (high-resolution documents), 56.6 on MMDU (multi-image multi-turn dialogue) — is presented as emergent from a single training recipe, with the long-context design as the common enabler. To fully isolate the contribution of native 24K training versus a hypothetical short-context baseline with the same architecture would require an ablation the paper does not perform, but the conceptual claim is that long-context training is necessary (even if not independently sufficient) for the observed versatility.

---

Innovation 2: The Video-as-Composite-Image Insight — Eliminating Video-Specific Architecture Through Input Representation

The video understanding literature for LVLMs has accumulated substantial architectural complexity: sparse sampling (Video-ChatGPT), temporal pooling (Valley), compressed video tokens (LLaMA-VID, Chat-UniVi), memory banks (MovieChat, MA-LMM), and language-mediated bridging (Video Recap, LangRepo). Each of these approaches starts from the premise that video is a fundamentally different modality from images — it has a temporal dimension that requires specialized modeling, and the volume of data (many frames) requires compression or summarization to fit within computational budgets. This premise has produced a subfield with its own specialized architectures, training objectives, and evaluation benchmarks.

IXC-2.5 makes a strikingly simple counterargument: if the image pipeline already handles ultra-high-resolution inputs with a 24K+ token context, then video can be formulated as a composite image — many densely sampled frames tiled into a single picture — and processed through the identical pipeline with no architectural modifications. The frame index overlay (a number written on each frame before concatenation) provides an explicit temporal signal that the ViT can encode alongside spatial features. The 64-frame limit and the H=24 patch budget constrain the effective resolution per frame, but the model learns to balance spatial and temporal detail within this budget.

This is a conceptually important reframing because it challenges the boundary between image and video understanding. If the composite-image approach works — and the empirical results suggest it does, with IXC-2.5 reaching 69.1 on MVBench (+8.7% over the prior SOTA VideoChat2-7B and +25.6% over GPT-4V) and competitive performance on MMBench-Video — then the specialized video architectures of prior work may have been solving a problem that was an artifact of insufficient context window length and resolution, not a fundamental modality gap. The "temporal modeling" that 3D convolutions or temporal attention layers perform may be learnable by a standard ViT operating on spatially arranged frame grids, provided the model has enough context to attend across the full grid and sufficient resolution per frame to capture fine-grained motion cues.

There is a tradeoff, acknowledged implicitly: encoding 64 frames as a composite image and partitioning into 24 patches means each frame receives, on average, less than one full 560×560 patch's worth of resolution — temporal density comes at the cost of per-frame spatial detail. For tasks requiring both high frame rate and high per-frame resolution (e.g., detecting small objects in fast motion), the composite-image approach may underperform specialized architectures that can allocate computational resources differently across time and space. But for the benchmark tasks evaluated, this tradeoff appears favorable.

The significance extends beyond video: this insight is an instance of a broader design philosophy where input representation choices substitute for architectural complexity. By framing video as a special case of ultra-high-resolution images, the model inherits all the optimization, training data, and engineering effort invested in the image pipeline. This is fundamentally a systems insight about how to achieve versatility through representational unification rather than through modular architectures.

---

Innovation 3: Modular Capability Isolation via Task-Specific LoRA — Catastrophic Forgetting Prevention Through Architectural Isolation

Most LVLMs are trained as generalists: a single set of weights handles all tasks. This works well when tasks share underlying competencies (visual grounding, language understanding, reasoning), but it creates tension when tasks require incompatible behaviors. Webpage generation requires the model to produce syntactically strict code (HTML, CSS, JavaScript) where a single missing semicolon breaks functionality. Article composition requires creative, stylistically varied prose where "correctness" is subjective. General visual QA requires factual accuracy and concision. Training on all these tasks jointly risks interference: the syntactic rigidity needed for code generation could bleed into article writing, making prose stilted; the creative freedom needed for articles could bleed into factual QA, increasing hallucination.

IXC-2.5's solution is architectural isolation through separate LoRA modules: the webpage generation capability lives in dedicated LoRA weights (rank 512, trained on screenshot-code, instruction-aware, and resume-to-homepage data), the article composition capability lives in separate LoRA weights (rank 256, trained with DPO), and the general comprehension capabilities live in the base Partial LoRA (rank 256). At inference time, the appropriate LoRA weights are swapped in based on the task. The base model weights (the 7B InternLM2-7B backbone) are shared across all tasks, so fundamental language and visual capabilities transfer, but the task-specific adaptations are isolated in the LoRA parameters.

This is conceptually distinct from the standard "one model, one set of weights" approach and from the "separate models for separate tasks" extreme. It occupies a middle ground: shared representation learning with isolated skill specialization. The benefit is that the webpage generation LoRA can overfit to code syntax patterns without degrading the model's article writing, and the DPO alignment for articles (which pushes the model away from dispreferred outputs) doesn't affect the model's ability to answer factual questions accurately.

The empirical significance of this design choice is difficult to isolate from the paper's results — there is no ablation comparing joint training of all tasks versus modular LoRA training — but the conceptual argument is compelling: as LVLMs are asked to perform an increasingly diverse set of tasks with incompatible output characteristics, architectural isolation of task-specific capabilities becomes a pragmatic necessity, not an optional optimization. The paper demonstrates this principle concretely for two tasks (webpage generation and article composition), but the approach generalizes: future LVLMs could maintain libraries of task-specific LoRA modules (for code, creative writing, multilingual translation, domain-specific reasoning) that are swapped in on demand, with the base model providing the shared multimodal understanding.

The tradeoff is deployment complexity: a system that must detect the task type and swap LoRA weights adds engineering overhead compared to a single unified model. But as task diversity increases, the alternative — training a single model that performs well across all output modalities — may become increasingly difficult, making modular isolation the more scalable approach.

---

Innovation 4: Preference Alignment for Open-Ended Multi-Modal Generation — Extending DPO Beyond Hallucination Reduction

Direct Preference Optimization (DPO) has been applied to LVLMs previously, but primarily for hallucination reduction — training the model to prefer truthful, grounded responses over fabricated ones (as in Silkie, RLHF-V, and prior work cited in Section 2). This is a natural application because hallucination produces clear preference pairs: responses containing fabricated visual details are rejected in favor of responses that accurately describe the image.

IXC-2.5 extends DPO to a qualitatively different alignment target: creative quality and stylistic reliability in long-form article generation. The challenge here is not factual accuracy (articles about climate change can be factually accurate but poorly written) but rather the stability of output quality — the observation that "direct fine-tuning on scarce instruction data can lead to unstable responses." The DPO pipeline addresses this through a chain of data construction steps (GPT-4o labeling with reasons → reward model training → automatic labeling of 78K pairs → DPO training) that collectively produce a model less likely to generate "inappropriate or low-quality content."

What makes this conceptually distinctive is the asymmetric learning dynamic the paper observes: the model "tends to prioritize minimizing the likelihood of dis-preferred responses over maximizing the likelihood of preferred responses." This is not an inevitable property of DPO but rather reflects the specific preference data distribution: when the SFT model already generates reasonable articles most of the time, the chosen responses are not dramatically higher quality than what the model already produces, so the gradient signal for increasing chosen-response probability is weak. The rejected responses, however, represent clear failure modes (poor structure, factual errors, inappropriate tone) that the model can learn to avoid with strong gradient signal.

This observation has implications for how DPO should be applied to open-ended generation tasks more broadly. If the primary benefit of DPO in such settings is suppressing rare but catastrophic failures rather than improving average quality, then the design of the preference dataset should emphasize clear negative examples (rejected responses that are unambiguously bad) over subtle quality distinctions between two acceptable responses. The paper's use of GPT-4o with "reasons" for its decisions supports this: the reasons help the reward model learn to identify specific failure patterns (disorganization, factual errors) rather than just ranking responses on a coarse good-bad axis.

The significance is tempered by the lack of quantitative ablation: the paper does not report DPO-on versus DPO-off metrics for article quality, so the magnitude of the improvement is unclear. The contribution is thus more methodological than empirical — a demonstrated pipeline for applying preference alignment to creative multi-modal generation, with an observed asymmetry in what the model learns — rather than a proof that DPO is essential for article quality. The practical takeaway for practitioners is that the data construction step (using a strong model like GPT-4o to provide detailed reasons, training a reward model, then auto-labeling a large pool of candidates) may be as important as the choice of alignment algorithm.

---

Innovation 5: A Diagnostic Taxonomy of Open-Source VL Capability Gaps — Shifting from "Which Model Wins?" to "What Specifically Is Missing?"

The paper's evaluation across 28 benchmarks is not just a report of state-of-the-art numbers; it functions as a diagnostic decomposition of the versatility gap between open-source and proprietary LVLMs. The five benchmark categories — Video, Structural High-Resolution, General Visual QA, Multi-Turn Multi-Image Dialogue, and Webpage Crafting — are not arbitrary groupings but correspond to specific capability dimensions where prior open-source models systematically underperform proprietary APIs.

This diagnostic framing represents a shift from the common evaluation paradigm in LVLM papers, which typically reports aggregate leaderboard performance and claims superiority if the model wins on a majority of benchmarks. IXC-2.5's evaluation instead maps specific architectural and training decisions to specific benchmark clusters:

• Ultra-high resolution + unified partition → Structural High-Resolution benchmarks (DocVQA 90.9, ChartQA 82.2, InfoVQA 69.9), where the model matches InternVL-1.5-26B (~4× larger) on DocVQA and outperforms DocOwl 1.5-8B by 13% on WikiTableQuestion.
• Native long-context training → Multi-Turn Multi-Image Dialogue (MMDU 56.6, +13.8% over prior open-source SOTA), where the ability to maintain coherence across interleaved image-text sequences is directly tested.
• Video-as-composite-image → Video benchmarks (MVBench 69.1, +8.7% over prior SOTA), where the lack of a specialized video pipeline does not prevent state-of-the-art performance.
• Webpage LoRA with real-world code data → Screenshot-to-code (Design2Code average 85.1, surpassing GPT-4V's 84.8), where the inclusion of cleaned Stack v2 data addresses the synthetic-data diversity limitation the authors identified in prior work.
• DPO for creative generation → Article composition (qualitative examples in Figure 4), where the alignment pipeline addresses output stability.

The conceptual contribution here is not the individual benchmark results but the structure of the evaluation as a diagnostic instrument. By selecting benchmarks that stress-test the specific design decisions (long context, high resolution, multi-image interleaving, code generation, creative alignment), the paper provides a template for how future LVLM papers should evaluate versatility — not as a single aggregated score but as a multi-dimensional capability profile where each dimension corresponds to a specific architectural or training choice. This makes the evaluation explanatory rather than merely competitive: a model that underperforms on MMDU but not on DocVQA likely has a context-length rather than a resolution bottleneck; a model that underperforms on Design2Code but not on general QA likely lacks code-specific training data rather than general visual understanding.

This taxonomy is an incremental contribution — each individual benchmark existed before this paper — but the deliberate grouping and the explicit linking to architectural decisions elevates it beyond a simple leaderboard and makes it a reusable diagnostic framework for the field.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary evaluations span 28 benchmarks organized into five categories: Video Benchmarks (MVBench, MLVU, MME-Video, MMBench-Video, TempCompass), Structural High-Resolution Benchmarks (DocVQA, ChartQA, InfographicVQA, TextVQA, OCRBench, DeepForm, WikiTableQuestion, Visual MRC, TabFact), General Visual QA Benchmarks (MMStar, RealWorldQA, MathVista, AI2D, MMMU, MME, MMBench, MMBench-Chinese, MMBench-v1.1, SEED-Bench Image, MM-Vet, HallusionBench), one Multi-Turn Multi-Image Benchmark (MMDU), and one Webpage Crafting Benchmark (Design2Code). All benchmarks use publicly available test sets; evaluation is conducted primarily through the OpenCompass VLMEvalKit for unified reproduction.

• Base model(s). All experiments use InternLM2-7B as the LLM backend, paired with OpenAI ViT-L/14 as the vision encoder. The 7B scale is deliberately chosen to demonstrate that "GPT-4V level capabilities" can be achieved without requiring massive model sizes. For comparisons against proprietary APIs, the baselines are GPT-4V and Gemini Pro. For open-source comparisons, baselines include InternVL 1.5-26B, Qwen-VL 7B, VideoChat2 7B, DocOwl 1.5-8B, GLM-4v 9B/14B, LLaVa 1.6 8B, WeMM 8B, Design2Code 18B, CogAgent-Chat 18B, WebSight VLM 8B, and others as listed in Tables 3-5.

• Metrics. Accuracy is the primary metric, computed as the fraction of test examples where the model's predicted answer matches the ground truth. Different benchmarks use different grading protocols — the paper relies on VLMEvalKit to standardize this. For Video benchmarks, metrics follow each benchmark's native protocol (MVBench uses accuracy; TempCompass uses accuracy on the determinism subset including MCQA, Y/N, and Caption Match to avoid instability from GPT-3.5-based evaluation). For MMBench variants, the paper scales scores from 0–3 to 0–100 "for easier understanding." For MME, the reported score is the aggregate across perception and cognition subtasks. For Design2Code, the benchmark reports five sub-metrics — Block-Match, Text, Position, Color, CLIP similarity — and an unweighted average, each measuring a specific aspect of visual-to-code translation fidelity.

• Baselines. The paper compares against two tiers:

  • Closed-source APIs: GPT-4V and Gemini Pro, evaluated on the same benchmarks (where API results are available — several benchmarks lack closed-source scores as indicated by dashes in Tables 3-5).
  • Open-source SOTA (with comparable model size): For each benchmark category, the paper identifies the current best-performing open-source model at the time of evaluation and reports its score as "Open-Source Previous SOTA." These include VideoChat2-7B (MVBench), InternVL 1.5-26B (multiple structural and general QA benchmarks), DocOwl 1.5-8B (DeepForm, WikiTableQuestion), GLM-4v (HallusionBench), LLaVa 1.6 8B (MMDU), Design2Code 18B (screenshot-to-code), and others as enumerated in Tables 3-5. For the webpage crafting task, the paper also compares against WebSight VLM-8B and CogAgent-Chat-18B.

• Generation budget / compute accounting. The paper does not use a unified generation budget metric across tasks. For standard benchmark evaluations, each model is evaluated according to each benchmark's standard protocol (typically single-pass greedy or temperature-0 generation, though specific sampling parameters per benchmark are not detailed). For the Design2Code benchmark, the evaluation follows the benchmark's published protocol. There is no systematic comparison controlling for inference FLOPs between IXC-2.5 and baselines — the comparisons are at the model level, not compute-matched. This is a notable departure from the example paper's rigorous per-generation budget accounting.

• Cross-validation / statistical protocol. The paper does not report cross-validation, confidence intervals, statistical significance tests, or multiple-run variance for any benchmark result. The 28-benchmark evaluation is broad but each result is a point estimate. The VLMEvalKit is used for "unified reproduction of the results," which standardizes evaluation protocols but does not provide statistical quantification. For the Design2Code benchmark, results for closed-source APIs and open-source baselines are taken from the benchmark's published leaderboard (or re-evaluated under the same protocol), but the paper does not specify how many runs were averaged or whether different random seeds affect the outputs.

---

Main Quantitative Results

Video Understanding (Table 3)

On video benchmarks, IXC-2.5 substantially outperforms the prior open-source SOTA on 4 of 5 tasks while competing closely with closed-source APIs:

• MVBench: IXC-2.5 achieves 69.1, compared to VideoChat2-7B (prior SOTA) at 60.4 — an 8.7 percentage point absolute improvement. It surpasses GPT-4V's 43.5 by 25.6 points. The Gemini Pro score is not reported for this benchmark.

• MLVU: 58.8 vs. InternVL 1.5-26B at 50.4 (+8.4 points over prior SOTA). GPT-4V scores 49.2; IXC-2.5 leads by 9.6 points. Gemini Pro is not reported.

• MMBench-Video: 55.8 vs. LIVA 34B at 59.0. This is the one video benchmark where IXC-2.5 does not surpass the prior SOTA, falling 3.2 points behind LIVA (a 34B model). It lags Gemini Pro (75.0) by 19.2 points, though the comparison is asymmetric in model scale. GPT-4V (59.9) and IXC-2.5 (55.8) are within 4.1 points of each other.

• TempCompass (determinism subset): 67.1 vs. Qwen-VL 7B at 58.4 (+8.7 points over prior SOTA). It trails Gemini Pro (70.6) by 3.5 points but leads GPT-4V (56.0) by 11.1 points.

• MME-Video: 46.9 vs. InternVL 1.5-26B at 42.0 (+4.9 points over prior SOTA). It falls below Gemini Pro (49.3) by 2.4 points and is not compared to GPT-4V (not reported).

The headline pattern: IXC-2.5's video performance is strongest on MVBench and MLVU (large margins over both open-source and GPT-4V), competitive on TempCompass and MMBench-Video, and moderately ahead of prior open-source on MME-Video. The gap to Gemini Pro is inconsistent — small on TempCompass (3.5 points), moderate on MME-Video (2.4 points), large on MMBench-Video (19.2 points) — likely reflecting differences in video duration, frame density requirements, and reasoning complexity across benchmarks. The paper interprets this as evidence that the composite-image approach works without video-specific architecture, though the MMBench-Video gap to LIVA suggests that specialized temporal modeling may still provide advantages for certain video understanding dimensions.

Structural High-Resolution Understanding (Table 3)

IXC-2.5 achieves strong results on document, chart, and OCR benchmarks, matching or exceeding substantially larger models:

• DocVQA: 90.9, tied with InternVL 1.5-26B (also 90.9) — a model with ~4× more parameters. GPT-4V scores 88.4; Gemini Pro scores 88.1. IXC-2.5 leads both by roughly 2.5–2.8 points.

• ChartQA: 82.2 vs. InternVL 1.5-26B at 83.8 (trailing prior SOTA by 1.6 points). GPT-4V scores 78.5; Gemini Pro scores 74.1. IXC-2.5 leads GPT-4V by 3.7 points.

• InfographicVQA: 69.9 vs. InternVL 1.5-26B at 72.5 (trailing prior SOTA by 2.6 points). GPT-4V scores 75.1; Gemini Pro scores 75.2. IXC-2.5 trails both by roughly 5.2–5.3 points.

• TextVQA: 78.2 vs. InternVL 1.5-26B at 80.6 (trailing by 2.4 points). GPT-4V scores 78.0 and Gemini Pro scores 74.6. IXC-2.5 essentially matches GPT-4V (78.2 vs. 78.0) and leads Gemini Pro by 3.6 points.

• OCRBench: 69.0 vs. InternVL 1.5-26B at 77.6 (trailing by 8.6 points — the largest gap among structural benchmarks). GPT-4V scores 51.6 and Gemini Pro scores 68.0. IXC-2.5 leads GPT-4V by 17.4 points and Gemini Pro by 1.0 point.

• WikiTableQuestion: 53.6 vs. DocOwl 1.5-8B at 40.6 (+13.0 points over prior SOTA). GPT-4V is not reported. This is the strongest relative gain among the structural benchmarks.

• DeepForm: 71.2 vs. DocOwl 1.5-8B at 68.8 (+2.4 points). GPT-4V and Gemini Pro are not reported.

• Visual MRC: 307.5 vs. DocOwl 1.5-8B at 246.4 (+61.1 points). GPT-4V and Gemini Pro are not reported.

• TabFact: 85.2 vs. DocOwl 1.5-8B at 80.2 (+5.0 points). GPT-4V and Gemini Pro are not reported.

The structural benchmark results reveal a clear capability profile: IXC-2.5 excels at form and table understanding tasks (WikiTableQuestion, DeepForm, Visual MRC, TabFact) where it substantially outperforms the prior open-source SOTA (DocOwl 1.5-8B) by margins ranging from +2.4 to +61.1 points. On DocVQA, it matches a 4× larger model. On more complex infographic and chart interpretation tasks (InfographicVQA, ChartQA, TextVQA), it is competitive but slightly behind InternVL 1.5-26B, suggesting that scale still matters for the most demanding visual reasoning. The OCRBench gap (-8.6 points) is the outlier and may indicate that InternVL 1.5-26B's larger capacity provides advantages for the diverse set of OCR sub-tasks that OCRBench aggregates.

General Visual QA (Table 4)

IXC-2.5's performance on general visual QA shows consistent competitiveness with both open-source SOTA and proprietary APIs across a broad range of benchmarks:

• MMStar: 59.9 vs. InternVL 1.5-26B at 57.1 (+2.8 points over prior SOTA). GPT-4V scores 57.1; Gemini Pro scores 42.6. IXC-2.5 leads GPT-4V by 2.8 points and Gemini Pro by 17.3 points. This is a notable result because MMStar is designed to be a challenging benchmark that resists shortcut learning.

• RealWorldQA: 67.8 vs. WeMM 8B at 68.1 (trailing prior SOTA by 0.3 points). GPT-4V scores 68.0; Gemini Pro scores 64.1. The three-way comparison shows GPT-4V (68.0), WeMM (68.1), and IXC-2.5 (67.8) all within 0.3 points of each other — effectively tied. Gemini Pro trails by roughly 3.7 points.

• MathVista: 63.8 vs. WeMM 8B at 54.9 (+8.9 points over prior SOTA). GPT-4V scores 47.8; Gemini Pro scores 45.8. IXC-2.5 leads GPT-4V by 16.0 points and Gemini Pro by 18.0 points. This is one of the largest absolute advantages over proprietary APIs in the paper, suggesting that mathematical reasoning in visual contexts is a particular strength of the IXC-2.5 training recipe.

• AI2D: 81.5 vs. InternVL 1.5-26B at 80.6 (+0.9 points). GPT-4V scores 75.5; Gemini Pro scores 70.2. IXC-2.5 leads GPT-4V by 6.0 points.

• MMMU: 42.9 vs. 360VL 70B at 53.4 (trailing prior SOTA by 10.5 points). GPT-4V scores 56.8; Gemini Pro scores 47.9. This is a substantial gap — MMMU is a multi-discipline benchmark requiring expert-level reasoning across six domains, and IXC-2.5 at 7B falls well below the massive open-source models (360VL 70B) and GPT-4V. It is closer to Gemini Pro (42.9 vs. 47.9, -5.0 points).

• MME: 2,229.0 vs. InternVL 1.5-26B at 2,189.6 (+39.4 points). GPT-4V scores 1,926.5; Gemini Pro scores 1,933.3. IXC-2.5 leads both proprietary APIs by roughly 296–303 points.

• MMBench (MMB): 82.2 vs. InternVL 1.5-26B at 82.3 (trailing by 0.1 points). GPT-4V scores 81.3; Gemini Pro scores 73.9. IXC-2.5 leads GPT-4V by 0.9 points and Gemini Pro by 8.3 points.

• MMBench-Chinese (MMBCN): 80.8 vs. InternVL 1.5-26B at 80.7 (+0.1 points). GPT-4V scores 80.2; Gemini Pro scores 74.3. IXC-2.5 leads GPT-4V by 0.6 points.

• MMBench-v1.1 (MMBv1.1): 79.4 vs. InternVL 1.5-26B at 79.7 (trailing by 0.3 points). GPT-4V scores 79.8; Gemini Pro scores 73.9. IXC-2.5 trails GPT-4V by 0.4 points.

• SEED-Bench Image (SEEDI): 75.4 vs. WeMM 8B at 75.9 (trailing by 0.5 points). GPT-4V scores 69.1; Gemini Pro scores 70.7. IXC-2.5 leads GPT-4V by 6.3 points and Gemini Pro by 4.7 points.

• MM-Vet: 51.7 vs. GLM-4v 14B at 58.0 (trailing by 6.3 points). GPT-4V scores 56.8; Gemini Pro scores 59.2. IXC-2.5 trails both proprietary APIs by 5.1–7.5 points.

• HallusionBench (HallB): 42.4 vs. WeMM 8B at 47.5 (trailing by 5.1 points). GPT-4V scores 46.5; Gemini Pro scores 45.2. IXC-2.5 trails GPT-4V by 4.1 points.

The general QA results show a nuanced picture. On many benchmarks (MMStar, MathVista, AI2D, MME, MMBCN, SEEDI), IXC-2.5 achieves or surpasses GPT-4V performance despite its 7B scale. On others (MMBench variants, RealWorldQA), it is essentially tied with GPT-4V and the open-source SOTA. On benchmarks requiring deep expert reasoning (MMMU) or integrated multi-modal capabilities (MM-Vet), it substantially trails larger models and proprietary APIs. The HallusionBench gap (-5.1 points vs. WeMM, -4.1 vs. GPT-4V) suggests that hallucination reduction remains an area where the model can improve — the DPO alignment for article composition does not appear to have generalized to reducing visual hallucinations in QA settings. The MMMU gap is the most informative negative result: it isolates expert-level multi-disciplinary reasoning as a capability that 7B-scale models have not yet mastered, regardless of training recipe.

Multi-Turn Multi-Image Dialogue (Table 4)

• MMDU: 56.6 vs. LLaVa 1.6 8B at 42.8 (+13.8 points over prior SOTA). GPT-4V scores 66.3. Gemini Pro is not reported.

IXC-2.5 substantially outperforms the prior open-source SOTA by 13.8 points, demonstrating that the native 24K-context training with interleaved image-text data directly addresses a capability gap identified in Section 2. However, GPT-4V retains a 9.7-point lead, suggesting that even with natively long-context multi-image training, the proprietary model maintains an advantage on complex multi-turn conversations over many images — potentially due to scale, training data quantity, or architectural factors not addressed by IXC-2.5's design.

Screenshot-to-Code (Table 5)

• Design2Code average: 85.1. GPT-4V scores 84.8; Gemini Pro scores 79.4; Design2Code-18B (prior open-source SOTA for this task) scores 80.4. IXC-2.5 leads GPT-4V by 0.3 points on average.

Breaking down the sub-metrics:

• Block-Match: 81.9 vs. GPT-4V 85.8 (trailing by 3.9 points). This measures whether the generated code reproduces the correct visual blocks/regions from the screenshot.
• Text: 95.6 vs. GPT-4V 97.4 (trailing by 1.8 points). This measures whether text content from the screenshot is included in the generated code.
• Position: 80.9 vs. GPT-4V 80.5 (+0.4 points). This measures whether elements are positioned correctly.
• Color: 80.8 vs. GPT-4V 73.3 (+7.5 points). This measures whether the correct colors are used. This is IXC-2.5's strongest sub-metric relative advantage.
• CLIP similarity: 86.5 vs. GPT-4V 86.9 (trailing by 0.4 points). This measures overall visual similarity between the rendered generated page and the original screenshot.

The pattern is striking: IXC-2.5 substantially outperforms GPT-4V on color fidelity (+7.5 points) while slightly trailing on spatial layout accuracy (Block-Match -3.9 points, Text -1.8 points) and overall visual similarity (CLIP -0.4 points, Position +0.4 points). The color advantage may reflect the inclusion of real-world web data (Stack v2) in training, which exposes the model to diverse color schemes beyond the synthetic WebSight palette. The Block-Match and Text deficits suggest that precise spatial correspondence between visual input and code output — determining exactly where each text block and visual element should be placed — remains challenging, potentially due to the patch-based image processing losing fine-grained spatial relationships between distant regions of the screenshot.

Importantly, IXC-2.5 also substantially outperforms Design2Code-18B (85.1 vs. 80.4) — a model specifically designed for this task with 2.6× more parameters — across all sub-metrics, with the largest gaps in Block-Match (+3.4) and Color (+13.8). This suggests that the unified training on diverse visual tasks may transfer beneficial capabilities (spatial reasoning, color understanding) to code generation that a task-specialized model trained on narrower data does not acquire.

---

Ablation Studies and Robustness Checks

Vision encoder resolution (490 vs. 560): The paper upgrades the ViT from 490×490 (used in IXC2) to 560×560 but provides no controlled ablation quantifying the performance impact of this change. The inference that higher resolution helps is supported by benchmark comparisons against IXC2 (which used the 490×490 ViT), but these comparisons are confounded by all the other changes between versions (expanded training data, longer context, new tasks). The structural high-resolution benchmark results (DocVQA 90.9, ChartQA 82.2) are consistent with higher resolution being beneficial, but the contribution of resolution alone cannot be isolated without a controlled comparison.

Maximum patch budget (H=12 vs. H=24): During fine-tuning, H is increased from 12 to 24 patches. The paper does not ablate this choice, so the marginal benefit of doubling the patch budget at the cost of increased token count and attention cost is unknown. The H=24 setting is used for video (up to 64 frames) and large documents, but the paper does not report whether performance on standard-resolution images benefits from the higher budget or whether the improvement is confined to very large inputs.

Partial LoRA rank (256 for general, 512 for webpage): The rank choice is stated but not ablated. The rank-512 webpage LoRA likely provides necessary capacity for learning HTML/CSS/JS syntax, but whether rank 256 would suffice with more training data, or whether rank 1024 would further improve performance, is not investigated.

Separate LoRA vs. joint training: The webpage generation and article composition capabilities are isolated in separate LoRA modules rather than being trained jointly with general comprehension. The paper does not compare this modular approach against a single unified LoRA trained on all tasks, so the claimed benefit of architectural isolation (preventing interference between code syntax and creative writing) is not empirically validated. This is a significant missing ablation — the innovation of task-specific LoRA modules is a key architectural claim but is not tested against the simpler alternative.

DPO alignment for article composition: The paper describes a multi-stage pipeline (CoT prompt augmentation → reward model training → DPO) but provides no quantitative results for article generation quality. There is no DPO-on vs. DPO-off comparison, no human evaluation of article quality, no automated metric (perplexity, coherence scores, diversity measures). The qualitative example in Figure 4 shows a generated article, but without a control or comparison, the reader cannot assess whether the DPO pipeline actually improves quality over the SFT baseline. This is the most significant missing evaluation in the paper — the article composition capability is one of two highlighted composition applications, but its effectiveness is demonstrated only anecdotally.

Reward model labeling accuracy: The paper uses GPT-4o to label 2,000 responses, trains a reward model, then uses the reward model to auto-label 78,000 more. The reward model's agreement rate with GPT-4o (or with any ground truth) is not reported. If the reward model's accuracy is low, the 30,000 DPO pairs derived from its labels may contain substantial noise, potentially degrading rather than improving article quality. The paper's observation that the model "prioritizes minimizing the likelihood of dis-preferred responses" could reflect noise in the rejected labels rather than a genuine learning dynamic.

RoPE extrapolation from 24K to 96K: The paper states that the 24K training context can be extended to 96K at inference via RoPE extrapolation, but provides no evaluation of model quality at extended context lengths. Performance degradation curves, attention pattern analyses, or task-specific benchmarks at 48K, 72K, and 96K are absent. The claim that 96K inference works is unvalidated — the paper does not demonstrate any task that requires or benefits from contexts beyond 24K.

Audio input/output: Audio capability is mentioned as a web demo feature using external tools (Whisper, MeloTTS). No evaluation of audio interaction quality or latency is provided, and this is explicitly not part of the trained model. The inclusion in the capabilities overview (Figure 3) may be misleading given it is not an LVLM capability.

Per-benchmark statistical variance: With 28 benchmarks evaluated, some of the narrow margins (e.g., Design2Code average 85.1 vs. GPT-4V 84.8, a 0.3-point difference; MMBench 82.2 vs. InternVL 1.5-26B 82.3, a 0.1-point difference) are unlikely to be statistically significant. Without confidence intervals or multiple-run variance, claims of "surpassing" or "matching" based on these margins should be treated cautiously. The consistent pattern across many benchmarks is more informative than any individual narrow win or loss.

---

Critical Assessment

Central Claim: "IXC-2.5 achieves GPT-4V level capabilities with merely 7B LLM backend." The evidence presents a more qualified picture. On 16 of 28 benchmarks, IXC-2.5 indeed matches or surpasses GPT-4V (and sometimes Gemini Pro), with particularly strong showings on video understanding (MVBench +25.6 over GPT-4V), mathematical reasoning in visual contexts (MathVista +16.0 over GPT-4V), certain document understanding tasks (DocVQA +2.5, WikiTableQuestion +13.0 over prior open-source SOTA), and screenshot-to-code (average +0.3 over GPT-4V). These are genuine achievements for a 7B model.

However, "matches or surpasses" on a benchmark-by-benchmark basis does not establish GPT-4V-level capability in the general sense. GPT-4V's capabilities include robust out-of-distribution generalization, nuanced instruction following, complex multi-step reasoning across modalities, and an extremely low hallucination rate on open-ended tasks — qualities not captured by the specific benchmark suite. The large gaps on MMMU (42.9 vs. 56.8), MM-Vet (51.7 vs. 56.8), HallusionBench (42.4 vs. 46.5), and MMDU (56.6 vs. 66.3) demonstrate that GPT-4V retains substantial advantages on benchmarks requiring deep expert reasoning, integrated multi-modal understanding, and hallucination resistance. The claim would be more accurately stated as "IXC-2.5 matches GPT-4V on 16 of 28 specific benchmarks, with remaining gaps on tasks requiring expert-level reasoning and multi-modal integration."

Central Claim: "Long-context capability is the enabling mechanism for versatility." This claim is supported by the architectural design and the strong multi-image multi-turn dialogue result (MMDU +13.8 over prior open-source SOTA), but the causal link is not isolated. The model differs from its predecessor IXC2 in multiple ways simultaneously: upgraded vision encoder resolution, expanded training data across many new task categories, increased patch budget, and native 24K context training. The MMDU improvement could be attributed to any or all of these factors, not specifically to long-context training. A controlled ablation training the same model with 4K vs. 24K context on identical data would be needed to establish context length as the causal enabler, and this experiment is not performed.

Central Claim: "Unified Dynamic Image Partition enables video understanding without specialized architecture." The video benchmark results strongly support the viability of the composite-image approach — IXC-2.5 achieves 69.1 on MVBench and competitive scores on other video benchmarks without any video-specific architectural components. However, the MMBench-Video result (55.8 vs. LIVA 34B's 59.0 and Gemini Pro's 75.0) tempers this: specialized temporal modeling or larger scale still provides advantages on certain video understanding dimensions. The composite-image approach works well but may not be optimal for all video tasks — it is a demonstrated viable alternative rather than a proven superior one.

Missing experiments that would strengthen the paper:

1. Context length ablation: Train the same model with 4K, 8K, 16K, 24K contexts and measure performance on MMDU, video benchmarks, and high-resolution tasks at each context length. This would directly quantify the contribution of long-context training.

2. DPO ablation for article quality: Compare article outputs from the SFT model (before DPO), the DPO model, and baselines (GPT-4V, Gemini Pro) using both automated metrics and human evaluation. The current anecdotal Figure 4 example is insufficient to validate a major component of the pipeline.

3. Separate vs. joint LoRA training: Train a single LoRA on all tasks (comprehension + webpage + article) and compare against the modular LoRA approach on held-out benchmarks for each task. This would validate the architectural isolation claim.

4. Scaling behavior at 96K context: Evaluate model quality on tasks that require 24K, 48K, 72K, and 96K context — for example, multi-image dialogues with increasing numbers of images, or video understanding with increasing numbers of frames. The 96K extrapolation claim is unsupported without such evaluation.

5. Per-patch resolution vs. patch count tradeoff: Ablate the ViT resolution (490 vs. 560) while holding patch budget constant to isolate the contribution of per-patch resolution to high-resolution understanding tasks.

Conditonal claims and their boundary conditions:

• "GPT-4V-level on 16 benchmarks" holds for the specific benchmarks reported, which overrepresent structural understanding (9 benchmarks) and underrepresent open-ended reasoning. The claim should not be extended to general GPT-4V-level capability.

• "Video understanding without specialized architecture" holds for the specific video benchmarks evaluated, which may not stress-test the composite-image approach's weaknesses (temporal ordering of rapidly changing scenes, long-range temporal dependencies beyond what frame indices can convey). The MMBench-Video gap suggests boundary conditions exist.

• "Long-context as enabling mechanism" holds as a design philosophy (the model was built around native long-context training) but is not causally proven. It is possible that the same model trained with 8K context and the same diverse data mixture would achieve comparable results on most benchmarks, with only MMDU and the most extreme video tasks being affected.

Weaknesses in evaluation design:

• No compute-matched comparisons: All comparisons are at the model level without controlling for inference cost. InternVL 1.5-26B, LIVA 34B, and 360VL 70B have substantially higher inference costs than IXC-2.5-7B, making "competitive or better" results more impressive, but the paper does not quantify this compute advantage.

• Single model family: All results are for IXC-2.5 based on InternLM2-7B. There is no evidence that the design principles (long-context training, unified partition, separate LoRA modules) transfer to other LLM backbones or vision encoder architectures.

• Benchmark coverage gaps: The 28-benchmark suite covers many dimensions but omits several capability areas where proprietary APIs are known to be strong: multilingual multi-modal understanding (beyond Chinese and English), detailed spatial reasoning (object localization, counting), open-ended long-form visual description, and safety/refusal behavior on adversarial multi-modal inputs.

• No error analysis: The paper reports aggregate scores without analyzing types of errors the model makes. On InfographicVQA (-5.2 vs. GPT-4V), does IXC-2.5 fail on numerical reasoning, text reading, or layout understanding? On MMMU (-13.9 vs. GPT-4V), which disciplines and question types account for the gap? Without error analysis, the benchmark results describe how much the model underperforms but not why, making it difficult to prioritize future improvements.

6. Limitations and Trade-offs

6.1 Practical Overhead of the Unified Dynamic Image Partition Is Unaccounted For

The assumption or constraint. The paper's central architectural claim is that a single Unified Dynamic Image Partition strategy can process images of arbitrary resolution, multiple images, and videos without modality-specific engineering. However, the computational cost of this unification is never quantified relative to the baselines it outperforms. When processing a high-resolution document image, IXC-2.5 partitions it into up to 24 patches (during SFT), each encoded independently by a 560×560 ViT producing 400 visual tokens — yielding up to 9,600 visual tokens for a single image, plus a global view patch adding another 400 tokens. For video, 64 frames are concatenated into a composite image and then partitioned into up to 24 patches, meaning each frame receives roughly 0.375 patches on average, trading temporal density for spatial resolution. The paper does not report the token count, FLOPs, or wall-clock latency for any of these configurations, nor does it compare the inference cost against baselines.

This matters because prior models that IXC-2.5 surpasses on benchmarks (e.g., DocOwl 1.5-8B on WikiTableQuestion, VideoChat2-7B on MVBench) may operate at substantially lower token budgets. DocOwl 1.5-8B uses a unified structure learning approach that processes document images more compactly. VideoChat2-7B uses sparse sampling rather than dense frame concatenation. Without compute-matched comparisons, the reader cannot determine whether IXC-2.5's benchmark improvements come from better architectural design or simply from consuming more inference compute per example — a distinction that matters enormously for deployment cost and latency calculations.

The consequence. A practitioner deploying IXC-2.5 for high-resolution document processing or video understanding faces an unknown cost profile. In the worst case, the model may spend more FLOPs per query than a specialized competitor to achieve the same accuracy, making the headline benchmark improvements economically irrelevant. For latency-sensitive applications (real-time video analysis, interactive document QA), the processing of up to 24 image patches through a ViT plus the quadratic attention cost of 9,600+ visual tokens in the LLM may impose unacceptable response times regardless of accuracy gains.

What evidence exists in the paper. None. Section 3 specifies the patch budget (H=12 for pre-training, H=24 for SFT) and the per-patch token count (400), but Section 4 reports only accuracy and benchmark scores — no FLOP counts, token counts per query type, latency measurements, or compute-matched comparisons against baselines. Section 5 notes this explicitly as a missing evaluation dimension.

Mitigation status. The paper does not address this limitation. The token budget is presented as a capability enabler (higher H enables higher resolution and more video frames) without discussing the cost tradeoff. There is no suggestion of dynamic patch budget allocation (using fewer patches for simpler images, more for complex ones) or compute-aware evaluation methodology for future work.

---

6.2 The 96K Context Extrapolation Claim Is Completely Unvalidated

The assumption or constraint. The paper states (Section 3.2, Section 1) that IXC-2.5 is "natively trained with 24K interleaved image-text data, whose context window can be extended to 96K through positional encoding extrapolation." RoPE extrapolation (cited as LocalLLaMA, 2023) adjusts rotary position embeddings at inference time so that attention patterns learned for relative positions up to 24K generalize to positions up to 96K. The paper presents this 96K capability as a headline feature — it appears in the abstract, the introduction, and the model's self-description — but provides zero experiments demonstrating that the model actually works at contexts beyond 24K.

This is a severe gap because RoPE extrapolation is not guaranteed to work out-of-the-box for any model or data distribution. Text-only LLMs require careful tuning of the extrapolation scaling factor (often with calibration data) to maintain coherence at extended lengths. For interleaved visual-text sequences, the extrapolation problem is strictly harder: visual tokens are dense (400 tokens per 560×560 patch) and occupy large contiguous blocks in the sequence, creating attention patterns qualitatively different from the sparse token-token relationships in text. Positional sensitivity may differ for visual vs. text tokens — a patch's absolute position in the sequence carries spatial information about where it falls in the original image, and extrapolating beyond trained positions could scramble this spatial signal.

The consequence. The 96K context claim is currently a theoretical capability unsupported by empirical evidence. A user who deploys IXC-2.5 expecting it to handle 96K-token multi-image dialogues, hour-long videos with dense frame sampling, or novel-length article generation may encounter degraded performance, attention collapse, or outright gibberish beyond 24K. The paper provides no guidance on expected performance drop-off, no calibration protocol for the extrapolation factor, and no indication of which task types are most affected by context extension.

What evidence exists in the paper. None. No evaluation uses contexts beyond 24K. The MMDU benchmark (the training dataset that reaches 24K during SFT) is evaluated at some context length ≤24K (the exact test-time context is not specified). No benchmark in Tables 3-5 requires or benefits from 96K context. The qualitative examples in Figures 2-4 do not involve especially long contexts. The 96K figure appears as an assertion, not a demonstrated property.

Mitigation status. The paper does not acknowledge this as a limitation. The 96K capability is stated as a feature rather than as an untested extrapolation. The LocalLLaMA reference is a community-sourced technique, not a peer-reviewed validation for multi-modal sequences. There is no suggestion of future work to validate or characterize the extrapolation quality, nor any caveat that users should test performance at target context lengths before deployment.

---

6.3 Article Composition Capability Is Asserted Without Quantitative Evaluation

The assumption or constraint. Section 3.6 describes a multi-stage pipeline for article composition: Chain-of-Thought prompt augmentation (5,000 base instructions → expanded dataset D*), supervised fine-tuning (π_ref), reward model training on 2,000 GPT-4o-labeled responses, automatic labeling of 78,000 additional responses, and DPO alignment on 30,000 preference pairs. This pipeline is presented as a core contribution and one of the two "compelling applications" highlighted in the abstract. However, the paper provides no quantitative evaluation of article generation quality — no automated metrics (perplexity, coherence, diversity, factuality), no human evaluation (fluency, informativeness, engagement, illustration relevance), and no comparison against baselines (GPT-4V article generation, IXC2 article generation, or the SFT model π_ref before DPO).

The only evidence provided is a single qualitative example in Figure 4 (an article about climate change). Without a control — e.g., the same prompt given to the pre-DPO model π_ref or to GPT-4V — the reader cannot assess whether the DPO pipeline improves quality, whether the improvement is meaningful, or whether the generated article represents typical or cherry-picked output.

This is particularly problematic given the claimed innovation of the pipeline: the observation that DPO training "prioritizes minimizing the likelihood of dis-preferred responses over maximizing the likelihood of preferred responses" is interesting but may indicate that the reward model's automatic labels are noisy and the DPO objective is primarily suppressing rare catastrophic failures rather than elevating average quality. Without quantitative metrics, the practical benefit of the entire pipeline remains unknown.

The consequence. A practitioner interested in IXC-2.5 for article composition has no basis for estimating output quality. The pipeline requires training a reward model, collecting GPT-4o labels, running DPO — a non-trivial engineering investment — but the paper provides no evidence that this investment yields better articles than simply using the SFT model with a detailed prompt. The risk is that a user adopts the DPO pipeline only to find that article quality is indistinguishable from or worse than the simpler baseline (as occurred with the ReST^EM-trained revision model in the example paper's Appendix K, where additional optimization backfired). Without baseline comparisons, the pipeline's contribution cannot be assessed.

What evidence exists in the paper. A single article generation example in Figure 4, with no comparison to any baseline, no multiple samples demonstrating consistency, and no annotation of quality dimensions. The paper acknowledges that "collecting high-quality text-image articles is a rare and expensive endeavor" (Section 3.6) and that "direct fine-tuning on scarce instruction data can lead to unstable responses," but then does not measure whether the proposed pipeline solves these problems. Section 4 does not include article composition in the benchmark evaluation.

Mitigation status. The paper does not acknowledge the absence of evaluation as a limitation. Article composition is presented as a demonstrated capability alongside benchmark-evaluated tasks. There is no suggestion of future work to develop automated or human evaluation protocols for open-ended multi-modal generation, and no caveat that the DPO pipeline's effectiveness is unvalidated. This is the most significant evaluation gap in the paper, and it undermines one of the two headline composition applications.

---

6.4 Task-Specific LoRA Isolation Is Not Compared Against Joint Training

The assumption or constraint. A key architectural claim (discussed as Innovation 3 in Section 4) is that isolating webpage generation and article composition in separate LoRA modules prevents "catastrophic forgetting" and interference between tasks with incompatible output characteristics — syntactically strict code for webpages, stylistically varied prose for articles, and factual conciseness for general QA. The paper trains three LoRA configurations: the base Partial LoRA (rank 256) for general comprehension, a separate webpage LoRA (rank 512) for code generation, and a separate article LoRA (rank 256) for creative writing with DPO alignment. At inference, the appropriate LoRA is swapped in based on the task.

This architectural choice is presented as an innovation, but the paper provides no comparison against the simpler alternative: training a single LoRA module on all tasks jointly. Without this ablation, the claimed benefits of isolation — preventing code syntax rigidity from degrading article prose, preventing DPO alignment from affecting factual QA accuracy — are hypothetical. It is equally plausible that a single LoRA trained on the union of all tasks would perform comparably or better, benefiting from shared representations across output modalities (e.g., HTML structure knowledge improving document understanding; article writing fluency improving general response quality). The practical deployment cost of maintaining and swapping multiple LoRA modules (detecting task type, loading the correct weights, handling ambiguous or mixed-task queries) is also unquantified versus the simplicity of a single unified model.

The consequence. A developer building on IXC-2.5 must commit to the complexity of multi-LoRA deployment without evidence that it improves over single-LoRA training. If single-LoRA joint training achieves essentially the same benchmark scores, the modular approach adds engineering overhead (task detection, weight management, increased storage for multiple adapter checkpoints) with no performance benefit. Conversely, if joint training does cause interference (degraded webpage generation accuracy, increased QA hallucination after DPO), the modular approach is justified but the paper provides no evidence that such interference occurs.

What evidence exists in the paper. None. The paper does not train a joint-LoRA baseline. The benchmark results in Section 4 evaluate the general comprehension model (with base Partial LoRA) and the webpage model (with webpage LoRA) on their respective tasks, but never compare these against a unified model. The observation about DPO training dynamics (Section 3.6) — that the model prioritizes suppressing rejected responses — hints that DPO modifies the output distribution in ways that could affect non-article tasks, but this is speculation without joint-training measurements.

Mitigation status. The paper does not acknowledge this as a missing experiment. Task-specific LoRA is presented as a design choice, not as a hypothesis requiring validation. There is no discussion of the tradeoff between modularity and simplicity, and no suggestion that joint training should be compared in future work.

---

6.5 Hard Multi-Modal Reasoning Tasks Remain Out of Reach at 7B Scale

The assumption or constraint. The paper positions IXC-2.5 as achieving "GPT-4V level capabilities with merely 7B LLM backend" and as demonstrating that versatility at this modest scale is achievable through long-context training and diverse data. However, the benchmark results reveal a sharp boundary: on tasks requiring deep expert-level multi-disciplinary reasoning (MMMU) and integrated multi-modal understanding with complex instruction following (MM-Vet, MMDU), IXC-2.5 substantially underperforms both proprietary APIs and larger open-source models.

Specifically (Table 4):

• MMMU: 42.9 vs. GPT-4V 56.8 (gap of 13.9 points) and vs. 360VL 70B 53.4 (gap of 10.5 points). MMMU covers six disciplines at college/expert level and tests cross-modal reasoning under time pressure.
• MM-Vet: 51.7 vs. GPT-4V 56.8 (gap of 5.1 points) and vs. GLM-4v 14B 58.0 (gap of 6.3 points). MM-Vet tests integrated capabilities: recognition, OCR, knowledge, spatial awareness, language generation, and math.
• MMDU: 56.6 vs. GPT-4V 66.3 (gap of 9.7 points). MMDU is the benchmark IXC-2.5 was specifically trained for, yet GPT-4V retains a substantial advantage.
• HallusionBench: 42.4 vs. GPT-4V 46.5 (gap of 4.1 points). This tests resistance to visual hallucinations and language priors — IXC-2.5 is more susceptible than GPT-4V to being misled by plausible but incorrect visual-language associations.

This pattern is consistent: IXC-2.5 is competitive or superior on benchmarks testing specific, well-defined capabilities (document reading, chart QA, code generation, video event recognition) but falls behind on benchmarks that require synthesizing multiple capabilities simultaneously under complex instructions. The paper's training recipe — diverse multi-task SFT, long-context exposure, task-specific LoRA — does not close the gap on these integrative reasoning tasks.

The consequence. For applications that demand expert-level reasoning across modalities — medical image analysis with clinical reasoning, legal document review requiring statutory interpretation, scientific figure analysis with domain-specific inference — IXC-2.5 at 7B scale may be insufficient regardless of training recipe. The paper's core premise (that versatility can be achieved through training design rather than scale) has a demonstrated boundary: the breadth of tasks the model can perform is wide (28 benchmarks across five categories), but the depth of reasoning on the most demanding tasks is limited. A practitioner selecting a model for a high-stakes multi-modal reasoning application should expect IXC-2.5 to underperform larger models and proprietary APIs, particularly when instructions are complex and answers require synthesis of multiple visual and textual cues.

What evidence exists in the paper. The MMMU, MM-Vet, HallusionBench, and MMDU results in Table 4, plus the pattern across all benchmarks where IXC-2.5 leads on structured, specific-capability tasks and trails on integrative, reasoning-heavy tasks. Section 4 of this analysis (Innovation 5) notes that the benchmark suite overrepresents structural understanding (9 benchmarks) relative to open-ended reasoning, which may inflate the overall impression of GPT-4V parity.

Mitigation status. The paper acknowledges the MMMU gap implicitly by reporting it (the score appears in Table 4), but does not discuss it as a limitation or analyze why this particular benchmark resists the training recipe's benefits. There is no breakdown by MMMU discipline to identify whether the gap is concentrated in specific domains (e.g., medicine, art) or is uniform. The paper does not suggest that scale is the limiting factor or propose scale-aware training strategies for integrative reasoning. The 7B scale is framed as a feature ("merely 7B"), not as a constraint that may fundamentally limit certain capabilities.

---

6.6 Single Model Family and Single Vision Encoder Architecture

The assumption or constraint. All experiments use a single LLM backbone (InternLM2-7B) and a single vision encoder architecture (OpenAI ViT-L/14, fine-tuned from the IXC2 checkpoint). The paper's claims about design principles — that native 24K training enables versatility, that the Unified Dynamic Image Partition handles diverse visual inputs, that Partial LoRA provides efficient alignment — are demonstrated only within this specific model family. There is no evidence that these principles transfer to other LLM backbones (e.g., LLaMA, Mistral, Qwen) or other vision encoders (e.g., SigLIP, DINOv2, EVA-CLIP).

The choice of starting from the IXC2 checkpoint is particularly important: the vision encoder was already partially aligned with InternLM2 via prior training, and the Partial LoRA design was inherited from IXC2. A researcher attempting to reproduce the approach with a different LLM family or a randomly initialized vision encoder alignment would face additional challenges — achieving stable alignment between the vision encoder and a new LLM embedding space, determining appropriate LoRA ranks and learning rates, and ensuring that long-context training with the new architecture yields comparable attention quality. The paper provides no guidance on these transfer questions because it does not test them.

This matters because the LVLM field is moving rapidly, with new LLM backbones and vision encoders released frequently. If IXC-2.5's success depends on properties specific to InternLM2 (e.g., its pre-training data mixture supporting long-context extension, its attention implementation handling dense visual token sequences efficiently) or the IXC2-pretrained ViT (e.g., its OCR capabilities from prior training on RCTW, CTW, and other text-rich datasets), then the reported results may not replicate with other model families. The paper's claim to provide a "promising research direction" that "can extend to a more contextual multi-modal environment" (Section 5) assumes generalizability that is untested.

The consequence. A practitioner using a non-InternLM2 LLM (for licensing reasons, language support, or integration with existing pipelines) cannot assume that following IXC-2.5's training recipe will yield comparable results. The Partial LoRA design, the 24K context training configuration, the specific data mixture weights, and the per-component learning rate scales were tuned for InternLM2-7B and the IXC2 vision encoder. Transferring these hyperparameters to a LLaMA-3-8B or Mistral-7B backbone may require substantial re-tuning, and there is no guarantee that the target LLM's attention mechanism handles interleaved visual-text sequences at 24K length as effectively. The paper's contribution is thus a point demonstration (what works for one model family) rather than a validated recipe (what works generally).

What evidence exists in the paper. None beyond the single model family. All benchmark results in Tables 3-5 are for IXC-2.5-7B. The paper does not mention experiments with alternative LLM backbones or vision encoders, and does not discuss transferability of the training recipe.

Mitigation status. The paper does not acknowledge this as a limitation. The abstract and conclusion present IXC-2.5 as a general approach ("a versatile large vision language model") rather than as a specific implementation within the InternLM ecosystem. There is no suggestion that future work should validate the design principles on other model families, and no discussion of which design choices might be InternLM2-specific versus generally applicable.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around open-source LVLM development from architectural specialization toward representational unification. The dominant paradigm has been to build separate architectures or specialized processing pipelines for different visual modalities — video models need temporal attention or memory banks, high-resolution document models need custom cropping strategies, multi-image models need dedicated interleaving mechanisms. IXC-2.5 demonstrates that a single unified input processing strategy (the Dynamic Image Partition) combined with native long-context training (24K tokens) can match or exceed specialized architectures across a remarkably broad range of benchmarks — 69.1 on MVBench without a video-specific architecture, 90.9 on DocVQA matching a 4× larger model, 56.6 on MMDU for multi-image dialogue, and 85.1 on Design2Code surpassing GPT-4V.

This is not a paradigm shift in the sense of introducing a fundamentally new architectural component — the ViT, the LLM, and Partial LoRA are all inherited from prior work. It is better characterized as a methodological reframing: the paper makes the case that long-context training should be treated as the organizing design constraint rather than as an after-the-fact capability extension, and that input representation choices (treating video as a composite image, using scaled identity partitioning for arbitrary-resolution images) can substitute for substantial architectural complexity. If this reframing holds up under replication across model families, it suggests that the path to closing the versatility gap with proprietary APIs runs through training infrastructure investments (enabling native long-context training on diverse interleaved data) rather than through novel architectural innovations per se.

The paper also provides a practical resolution to a latent tension in the LVLM literature: the conflict between task breadth and task depth at moderate model scales. Prior work vacillated between two extremes — building specialized models that excel at narrow tasks (e.g., DocOwl for document understanding, VideoChat for video) versus building generalist models that perform adequately across many tasks but trail specialists on each. IXC-2.5 demonstrates a third path: a generalist architecture that matches or exceeds specialists on their own benchmarks through training diversity and unified input processing, without requiring task-specific architectural modifications. The task-specific LoRA modules for webpage generation and article composition extend this principle: they enable deep specialization (matching or exceeding task-specific models like Design2Code-18B) while maintaining architectural unity with the generalist backbone. This blurs the boundary between "generalist" and "specialist" in a productive way — the same model can be both, depending on which LoRA weights are active.

However, the paper also draws a sharper boundary around what remains out of reach: integrative, expert-level multi-modal reasoning (MMMU 42.9 vs. GPT-4V 56.8), nuanced instruction following in complex multi-image contexts (MMDU 56.6 vs. GPT-4V 66.3), and robust hallucination resistance (HallusionBench 42.4 vs. GPT-4V 46.5). These gaps persist despite the long-context training and diverse data mixture, suggesting that scale — of model parameters, of training data volume, or of reasoning-specific training — remains a binding constraint for the hardest tasks. This boundary is useful for the field because it distinguishes what training recipe improvements can plausibly achieve at 7B scale from what likely requires larger models or fundamentally different training objectives (e.g., explicit reasoning supervision, stronger alignment against hallucination).

The paper also makes one research direction substantially less attractive: specialized temporal modeling architectures for video LVLMs. If a model with zero video-specific architectural components can achieve 69.1 on MVBench — outperforming GPT-4V by 25.6 points and the prior open-source SOTA (VideoChat2-7B, which does have video-specific design) by 8.7 points — the marginal benefit of 3D convolutions, temporal attention layers, memory banks, or token compression for standard video benchmarks appears limited. Resources invested in developing ever-more-elaborate video-specific architectures might be better spent on pushing the context length, resolution, and training data diversity of unified architectures. The caveat is MMBench-Video (55.8 vs. LIVA 34B's 59.0 and Gemini Pro's 75.0), which suggests that some video understanding dimensions still benefit from scale or temporal specialization, but the overall pattern makes the composite-image approach the stronger default.

---

Follow-Up Research This Work Enables

Context-length scaling laws for interleaved visual-text sequences. The paper treats 24K as a training context length and 96K as an extrapolation target, but provides no characterization of how model quality degrades — or improves — as context length increases. A natural follow-up would be to train IXC-2.5 variants at 4K, 8K, 16K, 24K, 48K, and 96K contexts (where 48K and 96K use RoPE extrapolation from the 24K checkpoint, but 48K could also be trained natively) and measure performance on a suite of benchmarks that explicitly vary the context demand: MMDU with 2, 5, 10, and 20 images; video understanding with 8, 16, 32, 64, and 128 frames; and document QA with documents of increasing page count. This would produce the first context-length scaling law for multi-modal LVLMs, analogous to how text-only scaling laws (Kaplan et al., 2020) characterize the relationship between context length, perplexity, and downstream task performance. The key question is whether visual-text sequences exhibit the same extrapolation behavior as text-only sequences — do attention patterns for dense visual token blocks degrade differently from text token attention patterns when extrapolating beyond training length? The paper's unsupported 96K claim makes this not just an interesting follow-up but an urgent validation.

Adversarial stress-testing of the video-as-composite-image approach. IXC-2.5's video understanding pipeline — concatenate frames into a composite image, overlay frame indices, partition into patches — makes implicit assumptions about what information can be recovered from spatial layout. A rigorous stress-test would construct a benchmark that deliberately breaks these assumptions: (a) temporal ordering without spatial cues — videos where frames are nearly identical and frame indices are occluded or ambiguous, forcing the model to rely on subtle motion rather than index numbers; (b) cross-frame object tracking — questions that require linking the same object across distant frames where the composite image places those frames in different patches, breaking the spatial locality that the ViT's patch-based attention relies on; (c) rapid scene changes — videos with cuts every 2-3 frames, where the composite image contains dozens of visually discontinuous scenes packed together, testing whether the model can segment the composite image into temporal episodes without explicit boundary markers. A model that excels on MVBench but fails on such stress-tests would reveal that the composite-image approach works by exploiting spatial-layout shortcuts rather than genuine temporal reasoning — a limitation that matters for real-world video understanding (surveillance, sports analysis, instructional video) where frame order cannot be inferred from visual similarity alone.

Single vs. separate LoRA training with interference probing. The paper's architectural isolation strategy — separate LoRA modules for general comprehension, webpage generation, and article composition — is untested against joint training. A clean ablation would train three variants on identical data: (a) a single LoRA trained on all tasks jointly, (b) the paper's separate-LoRA approach, and (c) a mixed approach where code and article LoRAs are merged into one adapter. The evaluation would need to go beyond the standard benchmarks to include interference probes: after training, does the joint-LoRA model produce code with article-like prose (stylistic bleed), does the article DPO model show increased hallucination on factual QA (alignment tax), and does the separate-LoRA model fail on boundary queries that mix modalities (e.g., "write an article about this webpage's design")? The paper's MMMU gap (42.9 vs. 53.4 for 360VL 70B) provides a natural testbed — does joint training degrade MMMU performance, or is the gap attributable to scale alone? This ablation would validate or refute the architectural isolation hypothesis that the paper advances without evidence, and would provide practical guidance on whether the deployment complexity of multiple LoRA modules is justified.

Quantitative evaluation of DPO for open-ended multi-modal generation. The article composition pipeline is the paper's most undersupported claim: a multi-stage alignment process (CoT augmentation → reward model training → DPO) is described in detail but evaluated only through a single qualitative example in Figure 4. A proper evaluation would require: (a) collecting a held-out test set of 100-200 article prompts spanning diverse topics, styles, and illustration requirements; (b) generating articles from the pre-DPO model (π_ref), the post-DPO model (π_θ), and baseline models (GPT-4V, IXC2, the SFT model from Section 3.4 without article-specific training); (c) running a human evaluation with at least 3 annotators per article on dimensions of coherence, informativeness, illustration relevance, stylistic quality, and factual accuracy; (d) measuring automated proxies — perplexity under a strong LM, self-BLEU for diversity, entailment scores for factual consistency between article text and included images. The paper's observation that DPO training "prioritizes minimizing the likelihood of dis-preferred responses" predicts that the DPO model should show lower variance in quality (fewer catastrophic failures) but not necessarily higher mean quality. An evaluation that measures both metrics would test this prediction and determine whether DPO is improving articles or merely suppressing rare garbage outputs — two different mechanisms with different implications for when DPO is worth the engineering cost.

Transfer of the long-context training recipe to other LLM families. The paper's results are demonstrated exclusively with InternLM2-7B. To establish whether native 24K interleaved training is a general principle or an InternLM2-specific success, the recipe should be replicated with at least two other LLM backbones — for example, LLaMA-3-8B (which has strong long-context capabilities from its own pre-training) and Mistral-7B (which uses sliding window attention and may handle dense visual token blocks differently). The replication would need to use the same vision encoder, the same data mixture, the same training hyperparameters, and the same benchmark suite, changing only the LLM backbone. Key measurements: (a) does the benchmark profile (which tasks benefit most from the recipe, which trail proprietary APIs) remain consistent across backbones, or do InternLM2-specific properties (pre-training data distribution, attention implementation) drive the observed pattern? (b) Does the RoPE extrapolation from 24K to 96K work equally well across backbones, or is InternLM2's positional encoding particularly amenable to extrapolation? This would determine whether IXC-2.5 is a point solution or a validated methodology.

Difficulty-aware budget allocation for the dynamic partition strategy. The paper uses a fixed maximum patch budget (H=24 during SFT) regardless of input complexity. A natural extension — inspired by the compute-optimal scaling literature — would be to develop a difficulty estimator that predicts, from a low-resolution global view of the input, how many patches are needed for a given query, and to allocate the patch budget adaptively. For example, a simple document with large text and no tables might need only 4-6 patches to achieve accurate OCR, while a dense infographic with small text and multiple chart types might need the full 24 patches. The estimator could be trained by benchmarking IXC-2.5's accuracy on a range of documents at different patch budgets and learning to predict the budget where accuracy saturates. This would address the paper's unquantified inference cost problem (Section 6.1): instead of always spending the maximum compute budget, the model would spend only what each input requires, making the benchmark improvements economically realizable in deployment. The MMDU benchmark, where GPT-4V retains a 9.7-point lead, would be a useful testbed — does adaptive patch allocation close the gap by spending more compute on complex dialogue turns and less on simple ones, or is the gap driven by factors (scale, reasoning depth) that patch count does not address?

---

Practical Applications and Downstream Use Cases

On-device document intelligence with a 7B model. The paper demonstrates that IXC-2.5 matches or exceeds GPT-4V on structured document understanding tasks — DocVQA 90.9 (vs. GPT-4V 88.4), ChartQA 82.2 (vs. 78.5), WikiTableQuestion 53.6 (no GPT-4V score, but +13.0 points over prior open-source SOTA), TabFact 85.2. For applications that require extracting structured information from documents — invoice processing, contract review, financial report analysis, academic paper QA — IXC-2.5 at 7B parameters is a viable replacement for proprietary API calls. The concrete benefit is cost: at 7B parameters, the model can run on a single consumer GPU (e.g., RTX 4090) with no per-query fees, no data leaving the local environment, and no rate limits. For an organization processing 10,000 documents per month, the savings relative to GPT-4V API costs could be substantial — and the model's strong form understanding (DeepForm 71.2, Visual MRC 307.5) covers exactly the structured extraction use cases that dominate enterprise document workflows. The limitation is that the document must fit within the model's effective resolution budget: an A4 page at 300 DPI contains roughly 8.3 million pixels, which with H=24 patches at 560×560 resolution gives a total effective resolution of about 7.5 million pixels — sufficient for most standard documents, but multi-page contracts or blueprints with dense fine print may exceed the patch budget and require page-level chunking.

Web development prototyping from design mockups or natural language descriptions. The Design2Code benchmark results (average 85.1, surpassing GPT-4V's 84.8 and Design2Code-18B's 80.4) demonstrate that IXC-2.5 with its webpage LoRA can translate both screenshots and free-form instructions into functional HTML/CSS/JavaScript. The color fidelity in particular (80.8 vs. GPT-4V 73.3, a +7.5 point advantage) suggests that the model faithfully reproduces design intent rather than approximating it. For a front-end developer or designer, this enables a workflow where: (a) a design mockup (from Figma, a whiteboard sketch, or a competitor's website screenshot) is fed to IXC-2.5 to generate initial HTML/CSS code; (b) the developer iterates on the generated code rather than writing it from scratch; (c) for simple pages (landing pages, portfolios, dashboards), the generated output may be production-ready with minimal editing. The instruction-aware webpage generation capability (trained on 18,000 GPT-4/Claude-3-sonnet synthesized examples) extends this to non-designers: a product manager who can describe a desired webpage in natural language can obtain functional code without involving a developer for the initial prototype. The resume-to-homepage task (2,000 examples) targets a narrower but high-value use case — personal website creation — where the model handles both content structuring (organizing resume information into sections) and aesthetic design (choosing layouts, color schemes, JavaScript interactivity). The practical limitation is code quality: the Design2Code "Text" sub-metric (95.6 vs. GPT-4V 97.4) and "Block-Match" (81.9 vs. GPT-4V 85.8) indicate that spatial layout fidelity is not perfect, so generated code will require manual adjustment for pixel-perfect designs.

Video content indexing and retrieval for media archives. IXC-2.5 processes videos by concatenating up to 64 frames into a composite image and treating it as an ultra-high-resolution still. The MVBench result (69.1) and the competitive MMBench-Video score (55.8) demonstrate that this approach captures enough temporal and spatial detail to answer questions about video content — event recognition, action sequences, scene understanding. For a media organization with a large video archive, this enables automated content indexing: run IXC-2.5 over the archive to generate dense captions for each video segment, answer structured queries ("find all clips containing a person wearing a red jacket entering a building"), and produce keyword-tagged summaries for search. The 64-frame limit means the model processes roughly 2-3 seconds of video at 24-30 fps in a single forward pass (or longer segments at lower frame rates). For long videos, a sliding window approach with overlapping segments would be needed, and the paper does not test how well the model handles such chunked processing. The absence of video-specific architecture is a deployment advantage: the same model checkpoint handles images, documents, and video without requiring separate video encoding infrastructure. The latency cost (processing 24 patches through a ViT plus attention over 9,600+ visual tokens in the LLM) may be a concern for real-time video applications, but for offline batch processing of archives, throughput rather than latency is the relevant metric.

Multi-image research and analysis assistant. The MMDU benchmark result (56.6, +13.8 over prior open-source SOTA) demonstrates that IXC-2.5 can maintain coherent multi-turn conversations across multiple interleaved images — exactly the capability needed for a research assistant that helps analysts compare charts, cross-reference figures in a paper, or track objects across a collection of photographs. The native 24K interleaved training means the model has learned to route attention across distant image-text boundaries, enabling a natural back-and-forth where a user references "the third image from earlier" and the model correctly retrieves the relevant visual context. The concrete deployment scenario is a local research tool running on a workstation GPU: a scientist analyzing experimental results across dozens of figures, a historian comparing artifacts across multiple museum catalog images, or a medical student reviewing a series of radiology images with associated clinical notes — all within a single session with no data leaving the device. The GPT-4V gap on MMDU (66.3 vs. 56.6) means the assistant will occasionally lose track of context in very long or complex multi-image dialogues, but the +13.8 point improvement over prior open-source models makes it the strongest available option for privacy-sensitive or cost-constrained settings.

---

When to Prefer This Method

The paper does not articulate an explicit decision framework pitting IXC-2.5 against named alternative approaches (e.g., "use composite-image video processing when X, use specialized temporal modeling when Y"), nor does it position its design principles against a clearly defined set of alternatives beyond the generic "prior open-source LVLMs" and "proprietary APIs." The comparisons in Section 4 are model-level benchmark evaluations, not controlled methodological tradeoffs. The paper's contribution is better understood as a recipe demonstration — showing what works for a specific model family and training configuration — rather than as a systematic study of design dimensions that would support a conditional recommendation matrix. A "when to prefer" section would therefore require inventing tradeoffs the paper does not analyze (e.g., "prefer separate LoRA modules when task interference is high," which the paper hypothesizes but never measures). The most honest guidance is: prefer this approach when building within the InternLM2 ecosystem and when the target application profile matches the benchmark categories where IXC-2.5 excels (structured document understanding, video event recognition, screenshot-to-code, multi-image dialogue with up to 24K tokens of context), with the understanding that expert-level integrative reasoning, hallucination resistance, and contexts beyond 24K are unvalidated and may require scale or methods beyond what the paper demonstrates.