InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD

🎯 Pitch

This 7B model scales to native 4K resolution and can outright beat GPT-4V on document and chart benchmarks, hitting 90% on DocVQA. It shatters the ~1500px ceiling by dynamically chopping images into grids, making open-source LVLMs genuinely useful for reading dense text.

1. Executive Summary

This paper introduces InternLM-XComposer2-4KHD, a large vision-language model that expands resolution handling to 4K HD (3840 × 1600) and beyond through dynamic resolution with automatic patch configuration (adaptively partitioning images into 336 × 336 patches while preserving aspect ratios, with layout counts varying per input). Built on InternLM2-7B with an OpenAI ViT-Large/14 vision encoder, the model supports resolutions from 336 pixels to 4K standard — far exceeding prior approaches capped at ~1500 × 1500 — and achieves performance that matches or surpasses GPT-4V and Gemini Pro on 10 of 16 benchmarks, including 90.0% on DocVQA and 81.0% on ChartQA, while outperforming the previous best open-source model on InfographicVQA by nearly 20 percentage points. The paper establishes that scaling training resolution up to 4K HD yields consistent performance improvements without saturation on high-resolution OCR tasks, though these gains are specific to text-rich understanding — perception-oriented benchmarks show negligible resolution dependence.

2. Context and Motivation

The Core Problem: LVLMs Cannot See Fine Details

The fundamental problem this paper tackles is that large vision-language models (LVLMs) have been effectively blind to fine-grained visual content. While these models have demonstrated impressive capabilities in high-level visual understanding — describing scenes, answering questions about prominent objects, recognizing general layouts — they consistently fail on tasks that require reading small text in documents, interpreting dense charts, extracting numbers from tables, or understanding infographics. This is not a minor limitation: it is a fundamental architectural constraint that prevents LVLMs from being useful in enormous categories of real-world applications.

The bottleneck is resolution. Most LVLMs employ vision encoders (typically CLIP-ViT) that process images at fixed, low resolutions — historically 224 × 224 or 336 × 336 pixels. At these resolutions, a typical document page or website screenshot becomes illegible. Text that would be trivially readable by a human becomes a blurred smudge to the model. Charts with axes, labels, and legends collapse into indistinct shapes. The model is forced to answer questions based on the visual equivalent of squinting at a thumbnail.

This matters enormously in practice. The paper argues, implicitly and through its choice of benchmarks, that many of the most valuable real-world applications of vision-language models lie precisely in the high-resolution, text-rich domain:

Document understanding: Invoices, contracts, academic papers, forms, government records — the world runs on documents, and almost all of them contain dense text that must be read precisely.
Chart and figure interpretation: Business dashboards, scientific figures, financial reports — all combine visual structure with textual information that becomes meaningless if the text cannot be read.
User interface comprehension: Software testing, accessibility tools, autonomous GUI agents — these require reading button labels, error messages, menu items, and table contents, often at the native resolution of the screenshot.
Infographics and diagrams: Public health materials, educational content, technical manuals — these blend images, text, and spatial layout in ways that demand high-resolution perception.
Websites and digital content: The modern web is designed for screens far larger than 336 × 336; understanding a webpage requires processing content across resolutions up to and beyond 4K.

The paper's opening paragraph frames this directly as a practical applicability constraint: the limitation "constrains their practical applicability in real-world scenarios." This is a claim about deployment readiness — the existing generation of LVLMs could not be trusted with tasks that humans perform routinely using normal screen resolutions.

The Technical Gap: Resolution Scaling Is Not Trivial

The naive solution to the resolution problem seems obvious: just feed higher-resolution images to the vision encoder. But this is technically infeasible with standard Vision Transformer (ViT) architectures for several interconnected reasons:

Quadratic attention cost. ViT encoders use self-attention, which scales quadratically with the number of input patches. Doubling the input resolution quadruples the number of patches, making the self-attention computation 16× more expensive. For a 4K image (~8 million pixels), a naive ViT would require orders of magnitude more computation than is practical in any deployment scenario.

Fixed resolution pretraining. ViT encoders are pretrained at a specific resolution (e.g., CLIP-ViT at 336 × 336). Their positional embeddings are learned for that specific grid of patches. Simply feeding a larger image requires non-trivial adaptation — the positional encoding scheme must be modified, and the model's internal representations may not transfer cleanly to a different spatial structure.

Aspect ratio diversity. Real-world images come in wildly varying aspect ratios: a document page might be portrait (e.g., 8.5:11), a scientific figure might be wide, a website screenshot might be tall-and-narrow, an infographic might be any shape. A fixed-resolution input forces these images into a square box through padding and resizing, distorting the content and breaking the spatial relationships that matter for understanding structure.

The data problem. High-resolution training data for vision-language tasks is scarce. Most available datasets were collected at standard web resolutions (e.g., 224–512 pixels). Simply collecting new high-resolution data at scale is expensive and time-consuming. This creates a chicken-and-egg problem: models cannot be trained for high-resolution understanding without high-resolution data, and high-resolution data is rarely collected because existing models cannot process it anyway.

These technical barriers explain why resolution has been a persistent bottleneck in the field: it is not that researchers failed to recognize the problem, but rather that solving it requires simultaneously addressing architectural, computational, and data constraints. The paper's core contribution is demonstrating that these barriers can be overcome through a combination of clever image partitioning (keeping the ViT's native 336 × 336 resolution while tiling), dynamic layout adaptation (handling arbitrary aspect ratios), and a training strategy that synthesizes high-resolution experiences from modest-resolution source data.

Prior Approaches and Their Limitations

The paper organizes prior work into two broad strategies, each with specific failure modes that the proposed method addresses.

Strategy 1: High-Resolution or Dual Encoders

This approach modifies the vision encoder itself to handle higher-resolution inputs directly, or adds a second high-resolution encoder alongside the standard low-resolution one.

Representative work cited in the paper:

Vary (Wei et al., 2023): Introduces a new image encoder specifically designed for high-resolution inputs. High-resolution features from this encoder are concatenated with low-resolution embeddings from CLIP-ViT. The approach requires training an entirely separate vision encoder with a different architecture, adding substantial complexity and parameters.
CogAgent (Hong et al., 2023): Separates high-resolution and low-resolution images into distinct vision encoders, then merges their features through a cross-attention module. Like Vary, this doubles the vision backbone and introduces cross-attention mechanisms that increase model complexity.
Mini-Gemini (Li et al., 2024): Similarly uses dual encoders with cross-attention merging, following the same pattern of architectural duplication.

Where this strategy falls short (as argued by the paper):

Architectural complexity: Each approach requires training additional vision encoders or substantial modifications to existing ones. This increases parameter counts, training cost, and engineering complexity.
Inflexibility to varying resolutions: These methods are typically designed for one or a few specific high-resolution settings. They lack a principled mechanism for handling the full spectrum of possible input sizes — from a 336 × 336 icon to a 3840 × 2160 4K screenshot to a 600 × 8000 long-scroll document.
Aspect ratio limitations: Fixed encoder architectures struggle with non-square aspect ratios, which are the norm rather than the exception in document and UI imagery.

The paper's own words on this limitation: "the Vision Transformer (ViT) architecture falls short when dealing with images of varying resolutions and aspect ratios, thereby restricting its ability to handle diverse inputs effectively." This is a critique of both the single-encoder and dual-encoder approaches — they are fundamentally rigid in their spatial handling.

Strategy 2: Cropped Image Patches (The Patch Division Paradigm)

This approach keeps the standard vision encoder at its native resolution but segments high-resolution images into multiple smaller patches (tiles) that each fit within the encoder's input size. The encoder processes each patch independently, and the resulting features are somehow combined for the language model.

Representative work cited in the paper:

Monkey (Li et al., 2023): Employs sliding windows to segment images into patches, then processes them with LoRA fine-tuning. Each patch is treated essentially as an independent image.
TextMonkey (Liu et al., 2024): Builds on Monkey by adding shifted window attention to consider connections between patches and a token resampler to manage the token budget.
LLaVA-NeXT (Liu et al., 2024): Uses a grid-based patching approach with multiple predefined resolution settings.
mPLUG-DocOwl 1.5 (Hu et al., 2024): Partitions document images into patches for OCR-free understanding.
UReader (Ye et al., 2023): Extends patching with specialized modules for document tasks.
Sphinx (Lin et al., 2023) and OtterHD (Li et al., 2023): Also adopt patch-based approaches.

Where this strategy falls short (as articulated by the paper):

Inadequate maximum resolution: The paper explicitly states that these methods "are constrained by an inadequate resolution, typically around 1500 × 1500, which does not satisfy the demands of daily content, e.g., website screenshots, document pages, and blueprints." This is a crucial claim: the existing patch-based methods top out at ~1.5K resolution, which is roughly the equivalent of a low-resolution smartphone screen. A 4K screenshot, a scanned document page, or a detailed blueprint contains approximately 7× more pixels than these methods can ingest.
Rigid resolution configurations: These methods "are confined to either a few predefined high-resolution settings or a limited range of resolutions." In other words, they offer a fixed menu of resolution options (e.g., 224, 336, 672, 1344 in LLaVA-NeXT) rather than a continuous, adaptive mechanism. This means the model cannot fine-tune its resolution allocation to match the specific needs of each input image — a document with tiny text gets the same maximum resolution as one with large text, and a wide-but-short chart might waste patches on empty space while being forced into a square aspect ratio.
No mechanism for dynamic layout: When an image is divided into patches, the spatial relationship between those patches matters enormously. A two-column document layout, for example, requires understanding which text flows to the right and which flows downward. Prior patch-based methods flatten patches into a 1D sequence without clear spatial delimiters, making it difficult for the language model to reconstruct the 2D structure of the original image. The paper addresses this directly with the newline token mechanism (Section 3.2), which is presented as a solution to a previously unaddressed ambiguity.

Unresolved Tensions in Prior Work

Beyond these two strategy-level critiques, the paper identifies or implies several unresolved tensions in prior work that motivate its approach:

The resolution-data chicken-and-egg problem. If high-resolution training data is scarce, how can a model be trained to handle 4K inputs? Prior approaches either required collecting new high-resolution data or accepted training at modest resolutions. The paper's dynamic resolution training strategy (Section 3.3) solves this by taking existing images — most of which are not 4K — and partitioning them into many patches, effectively synthesizing a high-resolution training experience from lower-resolution source data. A 1008 × 1008 image, divided into nine 336 × 336 patches, teaches the model to handle 1008 × 1008 resolution content even though the original training image was not captured at that effective resolution. This is a clever bootstrapping strategy that prior work did not exploit.

The tradeoff between resolution and general capability. Most prior high-resolution methods specialize heavily on document/OCR tasks at the expense of general visual understanding. The paper positions IXC2-4KHD as a general-purpose LVLM that happens to excel at high-resolution tasks, not a document specialist. This is evidenced by competitive performance on non-OCR benchmarks like MMBench, MM-Vet, and MMMU alongside the OCR-focused results. Prior methods rarely achieved strong performance in both categories simultaneously.

The saturation question. A key open question that the paper investigates is whether performance on high-resolution tasks saturates at some resolution ceiling. If models plateau at, say, 1K resolution, then there is no benefit to pushing to 4K — the cost would outweigh the gain. The paper's headline finding on this question is that "scaling training resolution up to 4K HD leads to consistent performance enhancements without hitting the ceiling of potential improvements" (Abstract), and that "saturation not observed even for the 4KHD setting" (Section 4.2). This is a significant empirical finding because it justifies the substantial computational cost of 4K training — the returns have not yet diminished, and even higher resolutions might yield further gains.

How This Paper Positions Itself

The paper positions InternLM-XComposer2-4KHD as both a substantial extension of the patch division paradigm and a qualitative leap in achievable resolution. The positioning is explicit in several ways:

As an extension of the patch division paradigm: "InternLM-XComposer2-4KHD follows patch division paradigm and enhances it by incorporating an innovative extension: dynamic resolution with automatic patch configuration." The paper does not claim to invent the patch division approach — it credits prior work like Monkey (Li et al., 2023) — but argues that prior methods did not push the paradigm far enough. The "enhancement" is making the patch configuration dynamic, aspect-ratio-preserving, and scalable to arbitrary resolutions.

As a resolution pioneer: "a groundbreaking exploration into elevating LVLM resolution capabilities up to 4K HD (3840 × 1600) and beyond." The paper claims to be the first to demonstrate LVLM performance at 4K resolutions, marking a concrete threshold that distinguishes it from the ~1.5K ceiling of prior work.

As a generalist, not a specialist: The paper emphasizes performance on 16 diverse benchmarks spanning OCR, perception, reasoning, and hallucination. It explicitly notes that IXC2-4KHD "achieves comparable results on other general LVLM benchmarks like perception and reasoning," positioning the model as a single system that handles both high-resolution document tasks and standard visual QA.

As a scaling-law investigation: The paper investigates not just whether 4K helps, but the shape of the resolution-performance curve. By testing four resolution settings (HD-9, HD-16, HD-25, 4KHD), it maps out how performance scales with resolution and identifies which task categories benefit. This positions the work as contributing to a broader understanding of resolution scaling in LVLMs, analogous to how scaling laws govern LLM pretraining.

By the specific technical innovations it introduces:

Dynamic resolution with automatic patch configuration (Section 3.2): Unlike fixed-grid partitioning, the method adapts patch count and layout to each image's aspect ratio, constrained only by a maximum patch budget $\mathcal{H}$ . This is framed as solving the scarcity of high-resolution training data by enabling "dynamic training resolution from 336 pixels to 4K standard."
Global-Local Format (Section 3.2): Providing both a full-image thumbnail (global view, resized to 336 × 336 for macro understanding) and the tiled high-resolution patches (local view, for fine details). The paper argues empirically that this dual-view approach is "crucial for the LVLM to correctly understand the image."
Newline token for 2D structure (Section 3.2): Inserting a learnable newline token after each row of image patches to signal the 2D layout to the LLM. This is framed as addressing the "confusion" that arises when "the number of tokens for each row can vary across different images." This is a subtle but important innovation — it acknowledges that LLMs process text as 1D sequences and need explicit structural signals to interpret spatial data.

In relationship to the prior InternLM-XComposer2: The paper assumes the reader is familiar with the XComposer2 architecture (referring to it as the starting point) and focuses almost entirely on the resolution enhancements. This positioning suggests that the resolution mechanism is designed to be modular — it can be attached to the existing XComposer2 framework without requiring fundamental changes to the vision-language alignment or LLM components. The architecture "mainly follows the design of InternLM-XComposer2" (Section 3.1), with the resolution handling as the differentiated contribution.

Summary of the Motivation Chain

Walking through the argument from problem to solution:

Observation: LVLMs cannot understand fine details in images because their vision encoders process fixed, low-resolution inputs (224–336 pixels).
Impact: This blocks LVLMs from document understanding, chart reading, UI comprehension, and other high-value real-world applications.
Why it's hard: Directly increasing ViT resolution is computationally prohibitive (quadratic attention cost), and high-resolution training data is scarce.
Prior attempts: (a) High-resolution encoders are complex, inflexible, and limited to narrow resolution bands. (b) Patch-based methods exist but cap out at ~1.5K resolution with rigid configurations and no explicit 2D structure preservation.
The gap: No existing method supports the full range from 336 pixels to 4K HD with arbitrary aspect ratios, and no one has demonstrated that scaling to 4K yields consistent, unsaturating performance improvements.
The contribution: A dynamic, aspect-ratio-preserving patch partitioning strategy that synthesizes 4K-resolution training experiences from existing data, combined with a global-local view format and explicit 2D structure signals, enabling a 7B-parameter model to match or exceed GPT-4V and Gemini Pro on high-resolution OCR benchmarks while maintaining general-purpose vision-language capabilities.

3. Technical Approach

3.1 Reader Orientation

This paper presents a high-resolution adaptation layer that attaches to an existing large vision-language model (InternLM-XComposer2) to enable it to process images from 336 pixels up to 4K HD (3840 × 1600) and beyond without changing the underlying vision encoder or language model. The system solves the problem that standard vision-language models are effectively blind to fine text and structural details because their vision encoders process fixed, low-resolution inputs (336 × 336 pixels): rather than modifying the ViT to accept larger images directly (which would incur quadratic attention costs and require retraining), the approach keeps the ViT at its native 336 × 336 resolution and dynamically partitions high-resolution images into multiple 336 × 336 patches that are processed independently, then reassembled with explicit spatial signals so the language model can reconstruct the original 2D layout.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components and one meta-strategy:

Vision Encoder (OpenAI ViT-Large/14 at 336 × 336) — The frozen-in-pretraining-resolution image processor. It never processes images larger than 336 × 336; instead, it processes many 336 × 336 tiles extracted from the original image. This is the key architectural constraint that the entire resolution strategy works around.
Dynamic Image Partition Module — A pre-processing algorithm (not a learned component) that takes an input image of arbitrary size and aspect ratio, and given a maximum patch budget $\mathcal{H}$ , determines the optimal grid layout ( $p_w$ patches wide, $p_h$ patches tall) that respects the original aspect ratio while staying within budget. It then resizes the image to exactly $p_w \times 336$ by $p_h \times 336$ pixels and splits it into $p_w \times p_h$ non-overlapping 336 × 336 patches.
Global-Local Format Generator — A formatting layer that produces two complementary views of each image: (a) a global view: the full image resized to 336 × 336, providing macro-level context (scene layout, overall structure, what kind of image this is); and (b) a local view: all the dynamically partitioned 336 × 336 patches, each processed by the ViT, then reassembled into a 2D feature grid. Between the two views, a learned "separate" token is inserted. At the end of each row of patches in the local view, a learned "newline" token (\n) is inserted to explicitly mark the 2D structure before everything is flattened into a 1D sequence for the LLM.
Token Merging Layer (Concatenation-based) — A simple dimensionality reduction step: after the ViT produces 576 tokens per 336 × 336 patch (24 × 24 grid), groups of 2 × 2 adjacent tokens are concatenated along the channel dimension (producing a single token with 4× the feature dimension), then projected by an MLP to the LLM's embedding dimension. This reduces token count to 1/4 of the original (144 tokens per patch) without learned resampling.
Large Language Model (InternLM2-7B) — The frozen base LLM that receives the combined global + local token sequence (with newline and separate tokens interspersed) and generates text responses. During pre-training, the LLM is frozen while the vision encoder and Partial LoRA adapters are trained. During supervised fine-tuning, all components (vision encoder, connector, LoRA, LLM with a small learning rate multiplier) are jointly trained.

What flows through the system for a single inference:

An image enters → Dynamic Image Partition computes the optimal grid layout $p_w \times p_h$ given budget $\mathcal{H}$ and the image's aspect ratio.
The global view is created: the full image is resized to 336 × 336.
The local view is created: the image is resized to $p_w \times 336$ by $p_h \times 336$ , split into $p_w \times p_h$ patches of 336 × 336 each.
Each patch (including the global view patch) goes through the ViT independently, producing 576 tokens per patch.
Token merging reduces each patch's 576 tokens to 144 tokens.
The local view patches are arranged into their $p_h \times p_w$ grid, a newline token is appended after each row, and the grid is flattened into a 1D sequence.
The global view tokens, a separate token, and the flattened local view tokens are concatenated into one sequence.
This sequence is fed to the LLM (via the MLP projector and LoRA adapters), which generates text autoregressively.

3.3 Roadmap for the Deep Dive

First, the dynamic image partition algorithm (Equation 1), because it is the foundational mechanism that determines which patches are created and how the image is represented. Everything downstream depends on this layout decision.
Second, the Global-Local Format and the newline token, because these are the mechanisms that preserve spatial information during the transition from 2D image structure to 1D token sequence. Understanding why the newline token matters requires first understanding how patches are arranged.
Third, the model architecture and training pipeline (pre-training then supervised fine-tuning), because this reveals how the components are trained, which parts are frozen when, what data is used, and how the mixed-resolution strategy works in practice.
Fourth, the inference-time resolution dynamics — the observation that training at one resolution and inferring at a slightly higher one yields benefits — because this is a non-obvious empirical finding that matters for deployment.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and training methodology paper whose core idea is that an LLM-based vision-language model can be made to handle 4K-resolution images by keeping the vision encoder at its fixed 336 × 336 pretrained resolution and dynamically tiling the input image into patches, where the tiling layout adapts to the image's aspect ratio and the total patch count is bounded by a compute budget.

Dynamic Image Partition: The Core Resolution Mechanism

The dynamic image partition algorithm is a pre-processing step that runs before the vision encoder sees any pixels. Its job is to decide, for each input image, how to resize and split that image into 336 × 336 patches such that the original aspect ratio is preserved, the total number of patches stays within a compute budget, and the resulting grid dimensions are integers.

The optimization problem. Given an input image $x$ with original dimensions $[h, w]$ (height and width in pixels), and a maximum patch budget $\mathcal{H}$ (the total number of 336 × 336 patches allowed), the algorithm chooses integers $p_w$ (number of patches per row) and $p_h$ (number of patches per column) subject to two constraints:

$p_w \times p_h \leq \mathcal{H}$

$p_h = \lceil p_w \times h / w \rceil$

where $p_w$ and $p_h$ are both positive integers, $h$ and $w$ are the original image dimensions, and $\lceil \cdot \rceil$ denotes the ceiling function (round up to the nearest integer).

What the constraints mean:

The first constraint ( $p_w \times p_h \leq \mathcal{H}$ ) is the compute budget: the total number of patches cannot exceed $\mathcal{H}$ . Each patch will later be processed by the ViT independently, so the ViT forward-pass cost scales linearly with $\mathcal{H}$ . This constraint is what makes the resolution scaling practical — without it, a large image could require arbitrarily many patches and become computationally infeasible.
The second constraint ( $p_h = \lceil p_w \times h / w \rceil$ ) is the aspect ratio preservation constraint: it ensures that the grid of patches has approximately the same height-to-width ratio as the original image. Specifically, if the original image has aspect ratio $h/w$ , then the grid should have the same ratio $p_h/p_w \approx h/w$ . Rearranging gives $p_h \approx p_w \times h/w$ , and the ceiling function ensures we get an integer while slightly over-covering the image area. The ceiling (round up) rather than floor (round down) is chosen because under-covering could crop out content; over-covering can be handled by padding.

What the algorithm actually does, operationally:

Start from $p_w = 1$ (one patch wide).
Set $p_h = \lceil p_w \times h / w \rceil$ .
Check if $p_w \times p_h \leq \mathcal{H}$ . If yes, increment $p_w$ by 1 and repeat from step 2. If no, take the previous (largest valid) $p_w, p_h$ pair as the solution.
The original image $x$ is then resized to exactly $[p_h \times 336, p_w \times 336]$ pixels. This resizing operation scales the image so that each patch dimension corresponds to 336 pixels in the output space. The image is padded as needed to reach these exact dimensions.
The resized image is split into $p_h \times p_w$ non-overlapping 336 × 336 patches, read row by row (top-left to bottom-right).

Why this form and not simpler alternatives:

Why dynamic and not fixed-grid? A fixed grid (e.g., always 3 × 3, always 4 × 4) would distort non-square images. For example, a 600 × 8000 document scroll forced into a 3 × 3 grid would have each patch cover ~200 × 2667 pixels, making text illegible in the wide dimension. The dynamic approach lets a wide image use, say, $p_w = 8, p_h = 1$ (under $\mathcal{H} = 9$ ), giving each patch an appropriate local resolution.
Why aspect ratio preservation? If you forced a square patch grid to process a non-square image, each patch would contain distorted content — either stretched or squished. The paper's constraint ensures that when a patch is 336 × 336 in pixel space, it corresponds to a square region in the original image content (after resizing maintains the aspect ratio). This means the ViT's positional embeddings — which were pretrained for square images — remain semantically appropriate.
Why the ceiling function? The ceiling ensures complete coverage of the image content. Using the floor would leave some edge strips unprocessed, potentially losing text at document edges, which is particularly problematic for OCR tasks.
Why the greedy $p_w$ maximization? The algorithm finds the largest possible $p_w$ that satisfies the budget, which maximizes the total resolution captured. Since image resizing to $336 \times p_w$ width means that larger $p_w$ preserves more horizontal detail, maximizing $p_w$ subject to the budget extracts the most information. Any smaller $p_w$ would be strictly lower resolution and is therefore suboptimal under the budget.

Examples of the algorithm's behavior under different $\mathcal{H}$ settings:

HD-9 ( $\mathcal{H} = 9$ ): A square image could use $3 \times 3 = 9$ patches, producing an effective resolution of 1008 × 1008. A 1:4 aspect ratio image could use $3 \times 1 = 3$ patches or $1 \times 3 = 3$ patches (depending on orientation). Maximum possible configuration: 9 patches in any layout.
HD-16 ( $\mathcal{H} = 16$ ): Up to $4 \times 4 = 16$ patches, effective square resolution of 1344 × 1344. Maximum token count after merging: 16 × 144 = 2304 tokens for the local view.
HD-25 ( $\mathcal{H} = 25$ ): Up to $5 \times 5 = 25$ patches, effective resolution of 1680 × 1680 if square. Maximum local tokens: 25 × 144 = 3600.
HD-55 ( $\mathcal{H} = 55$ ): This is the "4KHD" setting used during supervised fine-tuning for high-resolution tasks. It allows configurations like $p_w = 11, p_h = 5$ , producing $11 \times 5 = 55$ patches covering 3696 × 1680 pixels. A 4K HD image at 3840 × 1600 would map to approximately $p_w = 11, p_h = 5$ (since $3840/336 \approx 11.4$ , $1600/336 \approx 4.76$ , and $11 \times 5 = 55 \leq 55$ ), covering virtually all the 4K content. Maximum local tokens after merging: 55 × 144 = 7920 tokens.
HD-30 (inference only, not trained): $\mathcal{H} = 30$ . The paper uses HD-30 at inference time for models trained with HD-25, demonstrating that the dynamic training enables generalization to higher patch counts than seen during training.

The naming convention. The paper uses "HD- $\mathcal{H}$ " as shorthand for the dynamic partition strategy with maximum patch budget $\mathcal{H}$ . So "HD-9" means at most 9 patches, "HD-25" means at most 25 patches, and so on. These are not fixed grid sizes — they are maximum budgets, and the actual layout varies by image. Two images processed under HD-9 could have completely different patch layouts if they have different aspect ratios.

Global-Local Format: Dual-View Image Representation

The Global-Local Format is the mechanism by which the patch-based local view (high-resolution details) and the full-image global view (macro context) are combined into a single token sequence for the LLM.

The global view. The full input image $x$ is resized to exactly 336 × 336 pixels (a standard resize, not a center crop and not preserving aspect ratio — it is a forced square resize). This single 336 × 336 image is processed by the ViT, producing 576 tokens, which are then merged to 144 tokens via the concatenation-based merger. The purpose is explicitly described by the paper: "This provides a macro understanding of the image. Empirically, we have found this to be crucial for the LVLM to correctly understand the image." The global view tells the model what kind of scene or document it is looking at, the overall layout, the number of columns, the presence of figures vs. text, etc. — information that can be hard to extract from dozens of disconnected local patches.

The local view. The dynamic partition algorithm produces $p_h \times p_w$ patches of 336 × 336 each. Each patch goes through the ViT independently (the ViT processes them as separate forward passes — there is no cross-patch attention), producing 576 tokens per patch, which are merged to 144 tokens per patch. After processing, the patches are conceptually arranged into their 2D grid: $p_h$ rows, each containing $p_w$ patches.

The newline token mechanism. Before flattening the 2D grid into a 1D sequence, a learned "newline" token (denoted \n) is appended at the end of each row of patches. This token is randomly initialized at the start of pre-training and learned jointly during training. After inserting the newline tokens, the entire grid (all row tokens, plus the newline tokens between rows) is flattened into a single 1D sequence.

What the newline token does, operationally: if the local view has 3 rows of patches (each row containing, say, 4 patches with 144 tokens each = 576 tokens), the flattened sequence for the local view becomes:

[row1_patch1_tokens, ..., row1_patch4_tokens, \n_token, row2_patch1_tokens, ..., row2_patch4_tokens, \n_token, row3_patch1_tokens, ..., row3_patch4_tokens]

The newline token serves as an explicit row delimiter that the LLM can attend to, learning that tokens before \n belong to the same horizontal band of the image.

Why this matters beyond simple flattening: without the newline token, the local view is just a long 1D sequence of image tokens with no indication of where one row ends and the next begins. The LLM would need to infer the 2D structure purely from positional encodings and content patterns — a challenging inductive bias problem, especially since the number of tokens per row varies across images (different $p_w$ for different aspect ratios). The newline token provides an explicit, learned structural signal that the LLM can attend to, making the 2D layout directly accessible. The paper's ablation (Table 8) confirms this: under the challenging 4KHD setting with highly variable layouts, removing the newline token causes a "notable decline in performance on OCR-related tasks."

The separate token. Between the global view tokens (144 tokens) and the local view tokens (variable length), a learned "separate" token is inserted. This token is also randomly initialized and learned during training. Its purpose is to clearly demarcate the boundary between the two views so the LLM knows which tokens provide macro context and which provide fine-grained details.

The full input sequence structure:

[global_view_vit_tokens (144 tokens)] [separate_token] [local_view_patches_row1_tokens] [\n_token] [local_view_patches_row2_tokens] [\n_token] ... [local_view_patches_last_row_tokens]

The total sequence length varies based on the number of patches in the local view. For HD-25 applied to a square image (5 × 5 = 25 patches):

Global view: 144 tokens
Separate token: 1 token
Local view: 25 × 144 = 3600 tokens
Newline tokens: 5 tokens (one per row)
Total: 3750 image tokens (before text prompt and response tokens)

For HD-55 at 4K resolution (11 × 5 = 55 patches):

Global view: 144 tokens
Separate token: 1 token
Local view: 55 × 144 = 7920 tokens
Newline tokens: 5 tokens
Total: 8070 image tokens

This is a substantial token budget — 8070 image tokens is roughly equivalent to 6000+ words of text, making the inference context quite long. The paper acknowledges this computational burden and flags it as a motivation for future work on efficiency: "we have not explored the upper bound due to the computational burden increasing with higher-resolution inputs" (Section 5).

Token Merging Strategy: Reducing Visual Token Overhead

After the ViT processes each 336 × 336 patch, it produces 576 visual tokens (the ViT-L/14-336 uses a 24 × 24 grid of patches internally, plus a CLS token, but the paper uses only the grid tokens after removing the CLS token). If these 576 tokens per patch were fed directly to the LLM, the context length would balloon quickly: 25 patches × 576 = 14,400 tokens for HD-25, and 55 × 576 = 31,680 tokens for HD-55 — impractical for a 7B LLM with a limited context window and prohibitive for autoregressive generation cost.

The concatenation-based merging. The paper applies a simple, non-learned merging operation: adjacent tokens in 2 × 2 neighborhoods are concatenated along the feature dimension. Specifically, for each 2 × 2 block of ViT output tokens (each originally a vector of dimension $d$ , where $d$ is the ViT's hidden dimension), the four vectors are concatenated into a single vector of dimension $4d$ . This produces a grid of 12 × 12 tokens per patch (since 24/2 = 12 in each dimension), i.e., 144 tokens per patch — exactly 1/4 the original count.

The MLP projector. The concatenated 4d-dimensional vectors are then projected to the LLM's embedding dimension $d_{LLM}$ via a learned MLP (Multi-Layer Perceptron). This projection aligns the visual features (now at 4× the ViT's hidden dimension) with the LLM's token embedding space so the LLM can process them alongside text tokens.

Why this form and not a learned resampler:

The paper explicitly ablates this choice (Table 9), comparing the simple concatenation merge against two learned resampling methods:

Re-Sampler (from Qwen-VL, Bai et al., 2023): Uses learnable query vectors that attend to the ViT output tokens, producing a fixed number of output tokens regardless of input size. This requires training the attention mechanism to extract relevant information.
C-Abstractor (from Honeybee, Cha et al., 2023): A convolutional-based resampling module that similarly compresses visual tokens through learned operations.

The results show that concatenation and C-Abstractor perform similarly on most benchmarks, while Re-Sampler performs noticeably worse. The paper's interpretation: "the learnable queries used for gathering information require a great number of data for training, our pre-training data is somewhat lightweight for it to converge fully." In other words, the learned resampling mechanism (Re-Sampler) has additional parameters that need to be trained on the vision-language alignment data, and the available pre-training data volume is insufficient for those parameters to converge. The simpler concatenation approach, with zero learned parameters in the merge step itself (only the downstream MLP projector), avoids this data hunger problem.

This finding — that the connector design matters relatively little between simple concatenation and C-Abstractor — is also consistent with concurrent work (MM-1, McKinzie et al., 2024), which the paper cites as finding that "the influence of the connector is minor."

Implications of the merge factor of 4. The 4:1 compression ratio is a design choice that balances visual token granularity against sequence length. With 144 tokens per 336 × 336 patch, each token covers approximately a 28 × 28 pixel region of the original patch (336/12 = 28). For typical text in documents (e.g., 10–12 point font at 300 DPI), individual characters span approximately 30–40 pixels — meaning each merged token roughly covers the area of a single character or slightly less. This is a reasonable granularity for OCR: fine enough to distinguish individual characters, but not finer than needed.

Pre-Training: Aligning Visual Tokens with the Frozen LLM

The pre-training phase has a specific goal: teach the LLM to interpret the visual token sequences produced by the ViT + merge + projector pipeline, while preserving the LLM's original language capabilities and the ViT's original visual representations as much as possible. The phase is structured as follows:

What is frozen and what is trained:

LLM (InternLM2-7B): FROZEN. The language model's weights are completely untouched during pre-training. This preserves the model's language understanding, reasoning, and generation capabilities exactly as they were after language-only training.
Vision Encoder (OpenAI ViT-L/14-336): TRAINED (with layer-wise learning rate decay). The vision encoder is fine-tuned to adapt its representations to the domain of patched, high-resolution document imagery. However, to prevent catastrophic forgetting of its original visual knowledge (which was trained on diverse natural images), a layer-wise learning rate decay (LLDR) is applied: earlier layers receive smaller learning rates than later layers. The decay factor is set to 0.90, meaning that for each layer $\ell$ counting from the input, the effective learning rate is $\text{base_lr} \times 0.90^{\ell}$ . This ensures the low-level feature detectors (edges, textures, colors) change minimally while higher-level semantic features can adapt to the new data distribution.
Partial LoRA (rank 256): TRAINED. Low-Rank Adaptation (LoRA) adapters are applied to all linear layers in the LLM decoder blocks. These are small, trainable weight matrices that are added to the frozen LLM weights, allowing the model to learn how to process visual tokens without modifying the base LLM parameters. The rank of 256 means each LoRA adapter is a product of two low-rank matrices $A \in \mathbb{R}^{d \times 256}$ and $B \in \mathbb{R}^{256 \times d}$ , where $d$ is the layer's hidden dimension. This yields $2 \times 256 \times d$ trainable parameters per layer instead of $d \times d$ for full fine-tuning — a substantial parameter reduction.
The newline and separate tokens: TRAINED (randomly initialized). These special tokens start from random embeddings and are learned during pre-training alongside the other trainable components.
The MLP projector: TRAINED. The MLP that maps from the concatenated 4d-dimension visual features to the LLM's embedding dimension is learned from scratch.

Training data and objectives (Table 1). The pre-training data is curated around three objectives:

General semantic alignment: Data that teaches the model to connect visual content with language descriptions. Sources include ShareGPT4V (Chen et al., 2023), LAION-400M (Schuhmann et al., 2020), COCO Captions (Chen et al., 2015), and others — standard vision-language alignment datasets.
World knowledge alignment: Data that teaches the model to answer questions requiring factual knowledge about visual content. Sources include VQAv2 (Antol et al., 2015), GQA (Hudson & Manning, 2019), OK-VQA (Marino et al., 2019), A-OKVQA (Schwenk et al., 2022), and others — knowledge-intensive VQA datasets.
Vision capability enhancement (OCR-focused, newly added in this work): Data specifically targeting text reading and document understanding. The paper highlights that it "collected more related data to enhance this specific capability" and lists OCR-specific datasets including OCR-VQA (Mishra et al., 2019), TextVQA (Singh et al., 2019), ChartQA (Masry et al., 2022), DocVQA (Mathew et al., 2021), InfographicVQA (Mathew et al., 2022), and others. These are highlighted in red in Table 1 to mark them as new additions beyond what was used in the base XComposer2 pre-training.

Resolution setting during pre-training. The paper uses HD-25 for pre-training. This means all images during this phase are processed under the dynamic partition constraint of at most 25 patches. The global view and local view format with newline tokens is applied throughout. By training with varying patch layouts (different images produce different $p_w, p_h$ configurations under HD-25), the model learns to handle the variability in token sequence length and layout that comes from dynamic resolution.

How the model learns from variable-layout data. This is the core insight that makes the dynamic resolution training work: by exposing the LLM to many different image resolutions and aspect ratios during pre-training (all produced by the same HD-25 partition mechanism, but each image produces a different layout), the model learns that there is no fixed mapping between absolute position in the sequence and spatial position in the image. Instead, it must learn to use the newline tokens and the structure of the token sequence itself to reconstruct spatial relationships. This is fundamentally different from training with a fixed grid (e.g., always 3 × 3), where the model could memorize positional correspondences.

Training hyperparameters. The pre-training configuration is as follows:

Batch size: 4096
Epochs: 2
Learning rate schedule: Linear warm-up for the first 1% of training steps to a maximum of $2 \times 10^{-4}$ , then cosine decay to 0
Layer-wise learning rate decay for ViT: 0.90 decay factor
Partial LoRA rank: 256, applied to all linear layers in the LLM decoder block
ViT resolution kept at: 336 × 336 (the patches are exactly the ViT's pretrained resolution)
Token merging: 4:1 ratio via channel concatenation of 2 × 2 adjacent tokens
Optimizer: AdamW (implied by the paper's XComposer2 heritage and standard practice)
Vision encoder: OpenAI CLIP ViT-L-14-336

Why pre-train with HD-25 specifically? The paper does not explicitly state why HD-25 was chosen over other values for pre-training, but the reasoning can be inferred: HD-25 provides a substantial resolution increase over prior work (effective square resolution of ~1680 × 1680) while keeping the token sequence length manageable (3600 local tokens max after merging). This is a middle ground — enough resolution to teach the model to handle fine details, but not so many tokens that training becomes computationally prohibitive. The subsequent supervised fine-tuning phase then pushes to HD-55 (4K) for OCR-specific data, while keeping HD-25 for general data.

Supervised Fine-Tuning: Mixed-Resolution Training for Task-Specific Capabilities

After pre-training teaches the model to interpret visual tokens in general, supervised fine-tuning (SFT) teaches it to perform specific tasks — answering questions about documents, reading charts, solving visual math problems, etc. The key innovation in this phase is the mixed-resolution training strategy.

The resolution saturation problem for non-OCR tasks. The paper reports a critical empirical observation (Figure 5 and accompanying text): "we have observed a resolution saturation problem with the aforementioned perception tasks, where the influence of resolution becomes negligible." In other words, tasks like general VQA (MMBench, SEED-Bench), visual reasoning (MMMU, MathVista), and hallucination detection (HallusionBench) do not benefit from resolutions beyond a certain point — the information needed to answer these questions is already available at modest resolutions. Forcing these tasks through HD-55 training would waste computation and potentially cause the model to overfit to high-resolution patterns that do not generalize to these tasks.

The mixed-resolution solution. During SFT, the training data is divided into two categories with different resolution treatments:

HD-OCR QA tasks (DocVQA, ChartQA, InfographicVQA, TextVQA, OCRBench, and related): Trained with the HD-55 setting, allowing up to 55 patches. This enables processing of 4K (3840 × 1600) images without compression, which is critical because these benchmarks contain images where half of the longer side exceeds 2000 pixels, and text at native resolution would be illegible if compressed.
All other tasks (general VQA, reasoning, perception, hallucination detection, etc.): Trained with a dynamic-resolution strategy where images are resized to fall within a range between their original size and the HD-25 specification. The exact phrasing: "Images are resized to fall within a range between their original size and the size specified by the 'HD25' setting." This means that if the original image is large, it may be downsampled somewhat (not all the way to the HD-25 max, but within that range), and if it's small, it stays small. This dynamic treatment "enhances the robustness of the LVLM against differences in input resolution" — the model learns to handle a variety of resolutions for these tasks, which later enables it to generalize to higher resolutions at inference time.

Why mixed resolution is necessary. Training everything at HD-55 would be wasteful and potentially harmful for non-OCR tasks. Training everything at HD-25 would leave performance on the table for OCR tasks. The mixed approach allocates computational resources (longer sequences, more GPU memory, more FLOPs) to the tasks that benefit from them, while keeping non-OCR tasks efficient.

The batch size adjustment for mixed resolution. Because HD-55 produces approximately double the image tokens of HD-25 (55 patches × 144 tokens = 7920 local tokens vs. 25 × 144 = 3600), the two task categories have very different sequence lengths. The paper handles this by adjusting the data loader to use different batch sizes for the two categories: "we adjust the data loader to enable different batch sizes for them and adjust their weight accordingly." This is a practical engineering detail — without this adjustment, HD-55 samples would consume much more GPU memory per sample, limiting the effective batch size and slowing training.

Training data for SFT (Table 2). The SFT data is collected from diverse sources, with the newly added data highlighted in red in Table 2. The data spans:

OCR/Document tasks: DocVQA, ChartQA, InfographicVQA, TextVQA, OCR-VQA, DVQA (data visualization QA), and related.
General VQA: VQAv2, GQA, OK-VQA, A-OKVQA, ScienceQA, IconQA, and others.
Reasoning and math: MathVista, TabMWP, Geometry3K, and others.
Captioning: COCO Captions, TextCaps, NoCaps.
Grounding and referring: RefCOCO, RefCOCO+, RefCOCOg, Visual Genome.
Hallucination and robustness: HallusionBench, MM-Vet.
Instruction following: LLaVA-Instruct, ShareGPT4V, and custom instruction data.

Data from multiple sources are sampled in a weighted manner, with weights based on the number of samples from each source — effectively, importance sampling that prevents any single dataset from dominating the training signal.

Training configuration for SFT:

Batch size: 2048 combined (across both resolution tiers)
Training steps: 3500
Maximum learning rate: $5 \times 10^{-5}$ (lower than pre-training, which used $2 \times 10^{-4}$ , because SFT is more about refinement than alignment)
Vision encoder LLDR: 0.90 (same as pre-training)
LLM learning rate scale factor: 0.2 (relative to the base learning rate of $5 \times 10^{-5}$ , meaning the LLM (via LoRA) learns at $1 \times 10^{-5}$ ). The paper states the rationale: "This slows down the update of the LLM, achieving a balance between preserving its original capabilities and aligning it with vision knowledge."
Training scope: All components are trained jointly — vision encoder, MLP projector, newline/separate token embeddings, and LoRA adapters in the LLM.

The learning rate structure decoded. The vision encoder gets its own layer-wise decay schedule starting from the base rate. The LLM (via LoRA adapters) gets a reduced rate (0.2× base). This asymmetric treatment reflects the paper's philosophy: the vision encoder needs to adapt to the new high-resolution, OCR-heavy data distribution (hence full-rate training with LLDR), while the LLM's language capabilities should be preserved as much as possible (hence the 0.2× multiplier, which acts as a soft constraint against overfitting to vision-specific patterns).

Inference-Time Resolution Dynamics: Generalizing Beyond Training Resolution

One of the paper's most interesting empirical findings is that models trained under one resolution setting can generalize to higher resolutions at inference time without additional training, and this generalization improves performance on text-related tasks.

The phenomenon. Table 6 reports results where models are trained under HD-9, HD-16, or HD-25, and then evaluated at inference using higher resolutions than they were trained on:

IXC2-HD9 (trained with max 9 patches) achieves 50.5% on InfographicVQA when inferred at HD-9. When inferred at HD-16 (using up to 16 patches at test time — more than it ever saw during training), performance jumps to 58.6%, a gain of +8.1%.
IXC2-HD16 achieves 67.6% on DocVQA test when inferred at HD-16, and 69.8% when inferred at HD-25.
Similar patterns hold across multiple OCR benchmarks: higher inference resolution than training resolution yields consistent improvements.

Why this works. The paper's explanation (Section 4.2): "We posit that the dynamic image token length used in training enhances the robustness of the LVLM, leading to better results when the text in the image is more 'clear' in the higher resolution input." In other words, because the model was trained with variable-length image token sequences (each image under the dynamic partition produces a different number of patches depending on its aspect ratio), it learns to process visual tokens in a way that is invariant to the absolute number of tokens. When more patches are provided at inference time, the image is effectively "clearer" (each patch covers a smaller region of the original image, preserving more detail), and the model can benefit from this increased clarity because its training prepared it to handle variable-resolution inputs.

The failure mode on ChartQA. Notably, the paper reports that "the results on ChartQA consistently degrade under this setting." When a model trained at HD-9 is inferred at HD-16, ChartQA performance drops. The paper's speculative explanation: "This could be due to the model becoming confused about the chart structure when the resolution is altered." Charts have strong spatial structure (x-axes, y-axes, bar positions, line trajectories) that may be learned at a particular scale. When the resolution changes, the spatial relationships in token space shift, potentially disrupting the model's learned chart-reading strategies. This is an important case study in how resolution generalization is not uniform — it depends on the type of visual structure being processed.

Practical implication for deployment. This finding means that a model trained with HD-25 can be deployed at inference with HD-30 (or higher) for OCR tasks, gaining a "free" performance boost without any additional training. The paper uses this in its benchmark evaluations: "we have observed that using the 'HD30' setting yields better results on most OCR-related tasks when the LVLM is trained under the 'HD25' setting." This is a form of zero-shot resolution scaling — the model's robustness to variable token lengths, learned during training, translates into an ability to exploit more tokens than it was trained with.

The contrast with perception tasks. The paper repeatedly emphasizes that perception-oriented benchmarks (MMBench, SEED-Bench, AI2D, MME, etc.) show "negligible" dependence on resolution beyond a basic threshold. This reinforces the central thesis: resolution scaling matters specifically for tasks that require reading fine text or discerning detailed structures, not for tasks that can be answered from a macro-level understanding of the image.

Summary of Design Choices and Their Justifications

Dynamic partition over fixed grid: Enables arbitrary aspect ratio handling and resolution up to 4K without architectural changes. A fixed grid would distort non-square images and waste patches on padding.
Aspect ratio preservation constraint ( $p_h = \lceil p_w \times h/w \rceil$ ): Keeps patch content semantically consistent with ViT pretraining (square patches map to square image regions). Without this, stretched patches would fall outside the ViT's training distribution.
Global view alongside local view: Provides macro context that the model cannot easily extract from dozens of disconnected patches. The ablation (Table 7) shows performance drops of up to 4.4% without it.
Newline token between rows: Explicitly marks 2D structure in a 1D token sequence. The ablation (Table 8) shows it becomes more important as resolution variability increases — minimal benefit under fixed HD-9, substantial benefit under dynamic 4KHD.
Simple concatenation over learned resampling for token merging: Avoids data-hungry learned components that underperform given the available pre-training data volume. C-Abstractor performs similarly, but concatenation is simpler.
HD-25 for pre-training: Provides a middle-ground resolution (~1680 × 1680 effective) that balances computational feasibility with resolution benefits, creating variable-layout training experiences that generalize to higher resolutions.
Mixed-resolution SFT (HD-55 for OCR, dynamic for others): Allocates computational resources to tasks that benefit from high resolution while avoiding waste and potential overfitting on tasks where resolution saturates.
Frozen LLM during pre-training, 0.2× LR during SFT: Preserves language capabilities while allowing vision alignment. The aggressive VS conservative learning rate split (1.0× for ViT, 0.2× for LLM) reflects an asymmetric trust in the components' pre-existing knowledge.
Layer-wise LR decay on ViT (0.90): Allows higher-level visual features to adapt to the new data distribution while preserving low-level feature detectors. This is standard practice for fine-tuning pretrained vision models on new domains.

4. Key Insights and Innovations

Innovation 1: Dynamic Resolution as a Training-Data Bootstrap, Not Just an Inference Trick

The paper's most conceptually distinctive move is reframing resolution scaling from an inference-time challenge into a training-data synthesis strategy. Prior patch-based approaches (Monkey, LLaVA-NeXT, TextMonkey) treated image partitioning as a way to feed large images into a fixed-resolution vision encoder at test time — a deployment optimization. This paper inverts that logic: the dynamic partition algorithm is fundamentally a mechanism for synthesizing high-resolution training experiences from modest-resolution source data.

Here's why this is a conceptual shift, not just an engineering detail. The field had long assumed that training models for 4K understanding requires 4K training images — a classic data bottleneck. The paper's insight is that an image of any resolution can be made to simulate a higher-resolution training experience by partitioning it into more patches. A 1008 × 1008 training image, when divided into nine 336 × 336 patches under HD-9, teaches the model to process 1008 × 1008 effective resolution even though the source image was not captured at that resolution. Under HD-25, the same image might be divided into fewer patches, while a larger image gets more. The partition budget (the \mathcal{H} parameter), not the source image's native resolution, determines the effective resolution the model learns to handle.

This reframing has an important consequence that the paper does not explicitly state but which follows from the mechanism: the same training dataset can be used to train models at multiple effective resolutions simply by varying \mathcal{H} during data loading. This means the resolution-performance curve in Figure 5 (HD-9 → HD-16 → HD-25 → 4KHD) is a training design choice, not a data collection achievement. The paper did not need to curate separate 1K, 2K, and 4K training corpora — it reused the same data under different partition budgets. This is a fundamentally different scaling strategy than prior work, which either collected new high-resolution data (expensive) or accepted the limitations of their training data's native resolution (self-limiting).

The failure mode that proves the rule: the paper observes that perception benchmarks like MMBench and SEED-Bench show negligible resolution dependence. This is expected under the bootstrap framing — the information needed to answer "what color is the car?" is available at low resolution, and partitioning the image into more patches doesn't add new information. The resolution scaling gains appear precisely where the bootstrap creates genuinely novel input: text at document-native resolution that would be illegible at lower partition budgets.

Prior work contrast. Monkey and TextMonkey treat partitioning as an inference mechanism for documents; they do not frame it as a training data synthesis strategy. CogAgent and Mini-Gemini add new high-resolution encoders, implicitly accepting that the existing resolution was inadequate. The bootstrap framing is this paper's distinctive conceptual move.

Evidence anchor. The consistent improvement from HD-9 through 4KHD in Figure 5, achieved on the same underlying training data, demonstrates the bootstrap effect directly. The fact that saturation is not observed at 4KHD implies the bootstrap has not been exhausted.

Innovation 2: The Newline Token as an Explicit Spatial Grammar for 1D-Sequence Models

The paper introduces a deceptively simple mechanism — inserting a learnable newline token after each row of image patches before flattening — that addresses a fundamental representational mismatch between 2D images and 1D language model inputs. But the innovation is not the token itself; it is the diagnosis that 1D sequence models fail at 2D understanding because they lack an explicit spatial grammar, and that this grammar can be learned rather than hard-coded.

Prior patch-based methods (Monkey, LLaVA-NeXT, OtterHD, mPLUG-DocOwl) flattened image patches into a 1D sequence in raster-scan order — row by row, left to right — and relied on the LLM's positional encodings and attention mechanism to infer the 2D layout. This is a form of implicit spatial reasoning: the model must figure out from content patterns and relative positions that token 145 belongs to the row below token 1, not the row of token 144. For a fixed grid (e.g., always 3 × 3), this is learnable because the spatial mapping is constant. But under the paper's dynamic partition scheme, the number of tokens per row varies with aspect ratio — one image might have 4 patches per row, another might have 11. The LLM cannot memorize a fixed spatial mapping; it must infer layout from the sequence structure itself, which is ambiguous without row delimiters.

The newline token solves this by making the spatial grammar explicit and learnable. Rather than forcing the LLM to infer "where does this row end?" from content alone, the token provides a dedicated signal. Crucially, it is learned — it starts from a random embedding and acquires its meaning through training — rather than being a hard-coded separator with a fixed embedding. This means the token can develop a rich representation that encodes not just "row boundary" but potentially row-relative information: how many patches this row contains, whether it's the first or last row, where this row sits relative to the global view. The LLM can attend to these tokens to reconstruct the 2D layout without solving a difficult implicit inference problem.

What makes this a conceptual innovation rather than a minor trick: the paper is essentially arguing that multi-modal sequence models need an explicit spatial syntax when their inputs have variable 2D structure. This is a statement about the inductive biases of the architecture — transformers process 1D sequences, and expecting them to infer 2D structure from position IDs alone asks too much, especially when the mapping between position and spatial location changes per-sample. The newline token is a minimal, learnable mechanism for injecting 2D structural information into a 1D processing pipeline. It is analogous to how special tokens (BOS, EOS, SEP) provide structural signals in text processing — but applied to the spatial domain.

Evidence anchor. Table 8 is the key result: under the fixed HD-9 setting (limited aspect ratio diversity), the newline token provides only minor benefit. Under the dynamic 4KHD setting (highly variable layouts), removing it causes a "notable decline in performance on OCR-related tasks." This directly supports the diagnosis — the newline token matters precisely when the spatial structure is variable and must be explicitly communicated.

Innovation 3: Resolution Robustness as an Emergent Property of Dynamic Training

One of the paper's most striking empirical findings is that models trained under a particular resolution budget generalize upward at inference time: a model trained with HD-25 performs better when inferred at HD-30 on OCR tasks, without any additional training (Table 6). The paper treats this as a useful practical observation, but it represents a deeper conceptual finding about resolution robustness as an emergent property of variable-length training.

The standard assumption in the field — implicit in approaches that use fixed-resolution training (most prior LVLMs) — is that a model should be trained at the same resolution it will be deployed at. If you want 4K inference, you train at 4K. If you train at lower resolution, you expect degraded performance at higher resolution (out-of-distribution inputs) or simply no benefit (the model cannot use the extra tokens). The paper's finding overturns this assumption: the model not only tolerates higher inference resolution, it actively benefits from it.

Why does this happen? The paper's explanation — "the dynamic image token length used in training enhances the robustness of the LVLM" — points to a specific mechanism: by training with variable numbers of image tokens (each image, depending on its aspect ratio, produces a different patch count under the same \mathcal{H}), the model learns a resolution-invariant representation of visual content. It cannot rely on absolute position indices to locate information because the same spatial region might appear at different sequence positions in different images. Instead, it must learn to extract information from visual tokens based on their content and relative structure (aided by the newline token), making it robust to changes in the total token count.

This is a form of data augmentation at the sequence level — analogous to how training with varied image crops improves spatial invariance in CNNs. The dynamic partition mechanism means the LLM sees the same visual concepts at many different token sequence lengths during training, and this variability produces a model that generalizes across resolutions rather than overfitting to a specific token layout.

The ChartQA counterexample as a boundary condition. The paper reports that ChartQA performance degrades when inference resolution exceeds training resolution. This negative result is illuminating because it shows the limits of resolution robustness: it works for text reading (where more tokens = clearer characters) but fails for structured visual reasoning (where resolution changes disrupt learned spatial relationships in charts). This is not just a failure mode — it is a diagnostic boundary that distinguishes the types of visual understanding that benefit from resolution scaling (fine-detail extraction) from those that may be harmed by it (spatial reasoning at learned scales).

Significance beyond raw performance. This finding has practical implications for deployment — you can train more cheaply (at lower resolution) and infer at higher resolution, amortizing training cost across many inference queries. But the theoretical implication is more important: it suggests that resolution should be treated as a continuous variable that a model can generalize across, not a fixed architectural parameter. This opens the door to resolution-adaptive inference where the partition budget is dynamically adjusted based on task requirements, available compute, or even mid-inference difficulty assessment.

Evidence anchor. Table 6 provides the direct evidence: IXC2-HD9 improves from 50.5% to 58.6% on InfographicVQA when inferred at HD-16, and similar patterns hold for other OCR benchmarks. The finding is cross-validated across three training resolutions (HD-9, HD-16, HD-25) and multiple OCR datasets.

Innovation 4: The Diagnostic Decomposition of Resolution Sensitivity by Task Category

While not a single method or mechanism, the paper's systematic analysis of which tasks benefit from resolution and which do not constitutes a genuine conceptual contribution. By testing four resolution levels across OCR-heavy and perception-heavy benchmarks (Figures 5, 7; Table 6), the paper establishes a taxonomy of resolution dependence that had not been systematically documented before.

The taxonomy, as established by the paper:

Strong, unsaturating resolution dependence: HD-OCR tasks (DocVQA, InfographicVQA, TextVQA, OCRBench). Performance continues improving from HD-9 through 4KHD with no observed ceiling. This makes intuitive sense — these tasks require reading text at its native resolution, and more patches mean finer text discrimination.
Weak or saturating resolution dependence: Perception and reasoning tasks (MMBench, SEED-Bench, AI2D, MMMU). Performance plateaus at relatively low resolution. These tasks can be answered from macro-level visual understanding — object recognition, scene classification, spatial relationships — which is available at modest resolution.
Potentially harmful resolution dependence: ChartQA under inference-resolution scaling (Table 6), where higher-than-training resolution degrades performance. This is a distinct category: tasks involving structured spatial reasoning at learned scales may be disrupted when the scale changes.
No resolution dependence: Hardest difficulty bins (implicitly, for tasks well outside model capability). Resolution cannot compensate for fundamental capability gaps — a finding that echoes broader themes in the test-time compute scaling literature.

Why this is a contribution beyond the individual results. Prior work on high-resolution LVLMs (Monkey, TextMonkey, DocOwl) evaluated primarily on OCR benchmarks, making it unclear whether resolution improvements were general or task-specific. Prior generalist LVLMs (LLaVA, InstructBLIP) evaluated on perception benchmarks at low resolution, making it unclear whether resolution would help. The paper's systematic sweep across 16 benchmarks under multiple resolution settings provides the first broad empirical characterization of where resolution matters and where it does not. This is less a method innovation than an analytical contribution — a set of boundary conditions that subsequent work can use to decide whether to invest in higher resolution for their specific application.

The practical value of the taxonomy. For practitioners, this decomposition provides a decision framework: if your application is document reading, push resolution as high as computationally feasible. If your application is general visual QA, resolution beyond a modest threshold brings negligible returns. If your application is chart reading at learned scales, be careful about inference-time resolution changes. This is more actionable than a single accuracy number because it decomposes the problem.

Evidence anchor. Figure 5 shows the strong-vs-weak resolution dependence directly: HD-OCR tasks show large, monotonic gains from HD-9 to 4KHD, while AI2D and MMBench are essentially flat. Table 6 shows the ChartQA degradation under inference-resolution scaling. The per-benchmark results in Tables 3-5 contextualize these findings against closed-source and open-source baselines.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation uses 16 diverse benchmarks, spanning 5 HD-OCR datasets (DocVQA, ChartQA, InfographicVQA, TextVQA, OCRBench) and 11 general perception and reasoning benchmarks (MMStar, MathVista, MMMU, AI2D, MME, MMBench, MMBench-Chinese, SEED-Bench Image, QBench-Test, MM-Vet, HallusionBench). All evaluations are conducted using the OpenCompass VLMEvalKit for unified reproduction. DocVQA, ChartQA, InfographicVQA, and TextVQA follow their standard test sets; OCRBench uses its released benchmark suite. The paper does not specify exact sample sizes per benchmark in the main text, but these are standard published benchmarks with fixed test splits.
Base model. The model is InternLM-XComposer2-4KHD (IXC2-4KHD), built on InternLM2-7B as the language backbone with an OpenAI CLIP ViT-Large/14 at 336 × 336 as the vision encoder. The choice of 7B parameters is deliberate: it positions the model in a regime where fair comparison with similarly-scaled open-source models is possible while also demonstrating that performance competitive with much larger proprietary systems (GPT-4V, Gemini Pro) can be achieved through architectural innovations in resolution handling rather than through brute-force parameter scaling. This is a crucial positioning move — the paper is arguing that resolution scaling is an efficiency lever that can substitute for parameter scaling on certain task categories.
Metrics. All benchmarks are evaluated using their standard metric. For DocVQA, ChartQA, InfographicVQA, and TextVQA, this is exact-match accuracy (percentage of questions answered correctly). For MMBench, MMBench-Chinese, and MMStar, it is accuracy. For MME, it is the sum of perception and cognition scores. For MM-Vet, it uses the benchmark's integrated capability score. For HallusionBench, it uses the standard hallucination evaluation metric. For OCRBench, the paper reports a percentage score following the benchmark's scoring protocol. For MathVista and MMMU, standard accuracy is used. All metrics are reported as percentages or raw scores as defined by each benchmark's original evaluation protocol. The diversity of metrics means that numbers across benchmarks are not directly comparable — a 90.0 on DocVQA (percentage correct) vs. a score on MME (sum of sub-scores) measure fundamentally different things — but each number is comparable against other models evaluated under the same protocol.
Baselines. The paper compares against two tiers of baselines. Closed-source APIs: GPT-4V (OpenAI, 2023) and Gemini Pro (Google, 2023) — the dominant proprietary systems at the time of writing. These serve as aspirational upper bounds, and the paper's headline claim is that IXC2-4KHD matches or surpasses them on 10 of 16 benchmarks. Open-source SOTA models: For high-resolution understanding (Table 5), comparisons include Monkey (Li et al., 2023), TextMonkey (Liu et al., 2024), CogAgent (Hong et al., 2023), DocOwl 1.5 (Hu et al., 2024), LLaVA-NeXT (Liu et al., 2024), OtterHD (Li et al., 2023), and Sphinx (Lin et al., 2023). For general LVLM benchmarks (Table 4), comparisons include InternVL-Chat-V1.2 (Chen et al., 2023), Qwen-VL-Chat (Bai et al., 2023), LLaVA-1.5 (Liu et al., 2023), CogVLM-Chat (Wang et al., 2023), and others at comparable parameter scales (approximately 7B–13B parameters). The prior InternLM-XComposer2 (Dong et al., 2024) serves as an implicit baseline: the resolution enhancements are evaluated relative to this predecessor.
Generation budget / compute accounting. The paper does not explicitly define a generation budget in the way language model test-time compute papers do — there is no sampling of multiple candidate answers per question. Instead, the relevant compute metric is total image tokens processed (global view + local view tokens), which scales with the partition budget \mathcal{H}. For HD-9, the maximum is 1561 image tokens; HD-16 is 2653; HD-25 is 4057; and 4KHD (HD-55) is 8737 tokens. Training compute is not systematically compared across resolution settings — the paper acknowledges that "the computational burden increasing with higher-resolution inputs" is a limitation and leaves efficiency to future work. For the main benchmark results (Tables 3–5), IXC2-4KHD uses the maximum resolution available during training: 4KHD for the benchmark submission. No compute-matched comparisons against baselines are provided — the comparisons are purely performance-based, not compute-normalized.
Cross-validation / statistical protocol. The paper does not describe any cross-validation, bootstrapping, or statistical significance testing. Benchmark results are reported as single numbers (e.g., "90.0%" on DocVQA). The ablation studies in Section 4.2 and 4.3 use validation sets and test sets as specified by each benchmark's standard split (e.g., "validation set of InfoVQA, DocVQA, and TextVQA, test set of ChartQA and AI2D, MMBench EN-Test, and a 2k subset of SEEDBench" for the resolution scaling experiments). For the main benchmark tables (Tables 3–5), numbers are produced by a single evaluation run using the VLMEvalKit framework — no ensemble averaging or multi-seed training is reported. This is standard practice for LVLM benchmarking at the time, but it means that small differences between models (e.g., 90.0% vs. 89.5% on a benchmark) should not be interpreted as statistically significant differences without additional context about the variability of the evaluation pipeline.

Main Quantitative Results

Benchmark Performance Against Closed-Source APIs and Open-Source SOTAs

The paper's headline results are presented in Tables 3 and 4, which compare IXC2-4KHD against GPT-4V, Gemini Pro, and a collection of open-source LVLMs at similar parameter scales.

Table 3 (Closed-Source API comparison). The prominent numbers:

DocVQA: IXC2-4KHD achieves 90.0%, surpassing GPT-4V (88.4%) and Gemini Pro (88.1%). This is a non-trivial margin — approximately 1.6–1.9 percentage points above both proprietary systems — on the most widely used document VQA benchmark.
ChartQA: IXC2-4KHD achieves 81.0%, surpassing GPT-4V (78.5%) and Gemini Pro (74.1%). The margin over GPT-4V is 2.5 points; over Gemini Pro it is 6.9 points, which is substantial.
InfographicVQA: IXC2-4KHD achieves 68.6%, competitive with GPT-4V (70.5%) — the only closed-source model that exceeds it — and significantly above Gemini Pro (64.9%). The paper notes this is "the first open-source model that is close to the performance of Closed-Source APIs" on this benchmark, exceeding prior open-source by nearly 20 percentage points.
TextVQA: IXC2-4KHD achieves 78.1%, behind GPT-4V (80.2%) but above Gemini Pro (76.1%).
OCRBench: IXC2-4KHD scores 67.5%, behind GPT-4V (73.6%) and Gemini Pro (72.6%) — this is the one HD-OCR benchmark where both proprietary models maintain a clear lead.
MMStar: IXC2-4KHD achieves 55.4%, above GPT-4V (53.2%) and Gemini Pro (52.8%) — notable because MMStar is specifically designed to be challenging and resistant to data contamination.
MMMU: IXC2-4KHD scores 41.0%, behind GPT-4V (56.8%) and Gemini Pro (47.3%) — MMMU is a multi-discipline expert benchmark where larger models maintain a substantial advantage.
MME: IXC2-4KHD scores 2215.4, competitive with GPT-4V (2211.3) and above Gemini Pro (2109.4).
MMBench: IXC2-4KHD achieves 80.6%, above both GPT-4V (78.0%) and Gemini Pro (78.3%).
MM-Vet: IXC2-4KHD scores 59.6%, behind GPT-4V (67.5%) but above Gemini Pro (56.8%).
HallusionBench: IXC2-4KHD scores 46.9%, behind GPT-4V (49.7%) but above Gemini Pro (45.4%).

The paper claims that IXC2-4KHD "matches or even surpasses GPT-4V and Gemini Pro in 10 of the 16 benchmarks." Counting from Table 3: IXC2-4KHD leads or essentially ties on DocVQA, ChartQA, MMStar, MME, MMBench, MMBench-Chinese, SEED-Bench Image, and QBench-Test; it is genuinely behind on TextVQA, OCRBench, MMMU, MM-Vet, MathVista, and HallusionBench. On InfographicVQA, it trails GPT-4V but leads Gemini Pro. The "10 of 16" claim depends on treating close numbers as "matches" (e.g., MME where 2215.4 vs. 2211.3 is a rounding-level difference) and counting each benchmark equally. Qualitatively, the pattern is clear: IXC2-4KHD is strongest on OCR-heavy benchmarks (where its 4K resolution provides a genuine advantage) and perception benchmarks with strong visual detail components, but falls behind on benchmarks requiring deeper reasoning (MMMU, MathVista) or specialized knowledge beyond visual processing.

Table 4 (Open-source SOTA comparison). The paper compares IXC2-4KHD against InternVL-Chat-V1.2, Qwen-VL-Chat, LLaVA-1.5, CogVLM-Chat, and other 7B–13B models:

MMStar: IXC2-4KHD at 55.4% is the only open-source model above 50% (next best is InternVL-Chat-V1.2 at 49.3%). The paper notes this was "the only method that achieves a higher than 50% score on the challenging MMStar benchmark."
DocVQA: IXC2-4KHD at 90.0% leads the next best open-source model (InternVL-Chat-V1.2 at 83.3%) by 6.7 percentage points.
ChartQA: IXC2-4KHD at 81.0% leads the next best (Qwen-VL-Chat at 71.5%) by 9.5 points.
InfographicVQA: IXC2-4KHD at 68.6% leads the next best (TextMonkey at 50.7%) by 17.9 points — the largest relative margin.
MMBench: IXC2-4KHD at 80.6% ties with Qwen-VL-Chat at 80.6%, with both leading other open-source models.
MM-Vet: IXC2-4KHD at 59.6% leads, with Qwen-VL-Chat at 56.4% second.
MathVista: IXC2-4KHD at 57.5% trails InternVL-Chat-V1.2 at 62.0% — a notable gap on a math reasoning benchmark.
HallusionBench: IXC2-4KHD at 46.9% trails Qwen-VL-Chat at 49.6%.

The pattern is consistent with the closed-source comparison: IXC2-4KHD's advantages are concentrated on benchmarks requiring fine-grained visual understanding (document reading, chart interpretation, infographic comprehension), while it is competitive but not dominant on reasoning-heavy benchmarks.

Table 5 (High-resolution understanding comparison). This table compares IXC2-4KHD specifically against models designed for document/high-resolution understanding — Monkey, TextMonkey, CogAgent, DocOwl 1.5, LLaVA-NeXT, OtterHD, Sphinx:

InfographicVQA: IXC2-4KHD at 68.6% vs. DocOwl 1.5 at 50.7% (+17.9 points) and TextMonkey at 50.6% (+18.0 points). This is the largest gap in the table and demonstrates the benefit of 4K resolution — InfographicVQA images have 50% of longer sides exceeding 2000 pixels, so prior models capped at ~1500 × 1500 were fundamentally resolution-limited.
DocVQA: IXC2-4KHD at 90.0% vs. DocOwl 1.5 at 80.6% (+9.4 points) and TextMonkey at 83.2% (+6.8 points).
ChartQA: IXC2-4KHD at 81.0% vs. TextMonkey at 73.6% (+7.4 points) and DocOwl 1.5 at 71.7% (+9.3 points).
TextVQA: IXC2-4KHD at 78.1% vs. TextMonkey at 75.8% (+2.3 points) and DocOwl 1.5 at 76.3% (+1.8 points). TextVQA has a lower resolution ceiling — the marginal benefit of 4K over ~1.5K is smaller here.
OCRBench: IXC2-4KHD at 67.5% vs. CogAgent at 59.0% (+8.5 points) and TextMonkey at 65.5% (+2.0 points).

The paper notes that IXC2-4KHD "has the largest input resolution" among compared methods and "outperforms open-source LVLMs which are specifically tuned for document understanding" — despite being designed as a general-purpose LVLM rather than a document specialist.

Resolution Scaling Analysis (Section 4.2)

The experiments in Section 4.2 investigate how performance scales with resolution, using four settings: HD-9 (up to 1561 image tokens), HD-16 (2653 tokens), HD-25 (4057 tokens), and 4KHD (8737 tokens). These are evaluated on a standard set: InfoVQA validation, DocVQA validation, TextVQA validation, ChartQA test, AI2D, MMBench EN-Test, and a 2K subset of SEED-Bench (denoted SEED*).

Figure 5: Influence of Training Resolution. The key finding is a divergence between OCR and non-OCR tasks:

InfographicVQA: HD-9 achieves 50.5%. HD-16 achieves 60.7% (+10.2 points). HD-25 achieves 67.9% (+7.2 points over HD-16). 4KHD achieves the highest, with no saturation observed. This is the steepest resolution-performance curve.
DocVQA: HD-9 → HD-16 → HD-25 → 4KHD shows consistent improvement, though with a shallower slope than InfographicVQA. Saturation is not observed.
TextVQA: Shows improvement from HD-9 to HD-16 to HD-25, but with a shallower curve than DocVQA or InfoVQA.
ChartQA: Shows improvement across resolution settings, though the paper notes this is one of the tasks where the inference-resolution effect (Table 6) shows degradation — suggesting the training-resolution and inference-resolution effects are distinct.
AI2D, MMBench EN-Test, SEED*: Performance is essentially flat across all four resolution settings. The paper explicitly states these tasks have "only negligible difference between the four settings" and that "performance is saturated on the resolution."

The paper's interpretation: "High-resolution training is critical for HD-OCR tasks, while its gain on other tasks is minor." This is the empirical basis for the mixed-resolution SFT strategy — OCR tasks get HD-55, other tasks get dynamic resolution within HD-25 bounds.

Table 6: Influence of Inference Resolution. This table reports the generalization effect where models are evaluated at higher resolutions than they were trained on:

IXC2-HD9 → inferred at HD-16: InfographicVQA improves from 50.5% to 58.6% (+8.1%). DocVQA Validation improves from 77.8% to 79.8% (+2.0%). ChartQA decreases from 67.3% to 63.0% (−4.3%) — the degradation case.
IXC2-HD16 → inferred at HD-25: InfographicVQA improves from 60.7% to 67.9% (+7.2%). DocVQA Validation improves from 81.2% to 83.6% (+2.4%). ChartQA again degrades, from 67.1% to 65.3% (−1.8%).
IXC2-HD25 → inferred at HD-30 (not separately tabulated but mentioned in text): The paper states that "using the 'HD30' setting yields better results on most OCR-related tasks when the LVLM is trained under the 'HD25' setting."

The paper summarizes: "The model achieves better performance on text-related tasks when the inference resolution is higher than its training resolution." The exception is ChartQA "consistently degrades under this setting."

Figure 5 combined with Table 6 reading. Taken together, these results establish a nuanced resolution scaling picture: (1) training at higher resolution monotonically improves OCR task performance with no observed saturation; (2) inference at higher-than-training resolution also helps OCR tasks, but only up to a point (and ChartQA is a counterexample); (3) perception tasks are resolution-insensitive across both training and inference. This three-way taxonomy is the paper's key empirical contribution to understanding resolution scaling in LVLMs.

Ablation Studies and Robustness Checks

Global view importance (Table 7): Removing the global view from the Global-Local Format and keeping only the local view (patches) causes performance degradation across all benchmarks. DocVQA drops from 87.9% to 84.3% (−3.6%). InfographicVQA drops from 65.0% to 62.7% (−2.3%). MMBench EN-Test drops from 79.1% to 74.7% (−4.4%, the largest relative drop). The paper's interpretation: "the global view offers a general macro understanding of the image, which the model struggled to derive from the large number of tokens in the local view." This is a conceptually important ablation — it demonstrates that high-resolution patches alone are insufficient without macro context, and that the dual-view design is not redundant. The consistent degradation across all benchmarks (not just OCR tasks) suggests the global view provides universally useful information, likely about scene structure and image type that cannot be reconstructed from patches.

Newline token importance (Table 8): Under the fixed HD-9 setting, removing the newline token causes minor degradation: DocVQA drops from 77.8% to 77.0% (−0.8%), InfographicVQA drops from 50.5% to 50.3% (−0.2%). Under the dynamic 4KHD setting (HD-25 + HD-55), the impact is much larger: DocVQA drops from 87.9% to 84.9% (−3.0%), InfographicVQA drops from 65.0% to 60.2% (−4.8%). The paper's interpretation: "When a fixed high-resolution strategy HD-9 is employed, we observe that the benefit derived from the newline token is minor. This could be attributed to the LVLM's ability to handle limited differences in image ratios after training." Under 4KHD with "significant diversity in both image ratio and token number," the newline token becomes critical. This ablation directly validates the paper's claim that explicit spatial grammar matters when layouts are variable — a non-obvious finding that distinguishes the paper's approach from prior fixed-grid methods.

Token merging strategy (Table 9): Comparing three merging methods at equivalent compression rates (1/4 of original tokens): simple concatenation (the paper's method), C-Abstractor (Cha et al., 2023), and Re-Sampler (Bai et al., 2023). Concatenation and C-Abstractor perform similarly across benchmarks (e.g., DocVQA: 87.9% vs. 87.4%; InfographicVQA: 65.0% vs. 64.8%). Re-Sampler performs systematically worse: DocVQA 83.1% (−4.8% vs. concatenation), InfographicVQA 58.7% (−6.3%), ChartQA 62.4% (−11.5%). The paper argues this is because "the learnable queries used for gathering information require a great number of data for training, our pre-training data is somewhat lightweight for it to converge fully." This is an informative negative result — it suggests that the simplicity of concatenation merging is not just an engineering convenience but a genuine advantage under data-constrained pre-training.

Additional implicit ablations from the resolution scaling experiments: Figure 5 and Table 6 together serve as a de facto ablation of resolution level, demonstrating that resolution is not a binary feature (high vs. low) but a continuous variable with task-dependent effects. The fact that performance continues improving through 4KHD on OCR tasks without saturation is not guaranteed a priori — it is an empirical finding that validates the paper's decision to push to 4K.

Critical Assessment

Do the experiments actually demonstrate that IXC2-4KHD matches or surpasses GPT-4V and Gemini Pro on 10 of 16 benchmarks?

The claim is numerically supported by Tables 3 and 4, but requires nuanced reading. On several of the 10 benchmarks where IXC2-4KHD "matches or surpasses" the proprietary models, the margins are extremely small: MME shows 2215.4 vs. GPT-4V's 2211.3 (a difference of 0.2%), and MMBench shows 80.6% vs. 78.0% (a 2.6-point difference). These differences, computed from single evaluation runs without confidence intervals, could plausibly fall within evaluation noise — slight variations in prompt formatting, answer extraction, or random seed could flip the ordering. The paper does not report any form of uncertainty quantification (standard deviations, bootstrap confidence intervals, multi-seed averages) that would allow the reader to assess whether a 0.2% difference on MME is statistically meaningful. This is a genuine weakness: the headline claim of matching GPT-4V on 10 benchmarks would be stronger if accompanied by evidence that these results are robust to evaluation stochasticity.

More importantly, the claim obscures the benchmarks where IXC2-4KHD is substantially behind GPT-4V. The gap on MMMU (41.0% vs. 56.8%) is 15.8 points — larger than the combined margins on all the benchmarks where IXC2-4KHD "leads." On MM-Vet, the gap is 7.9 points (59.6% vs. 67.5%). On MathVista, the gap is approximately 5 points. These are not small differences, and they concentrate on benchmarks requiring deeper multi-step reasoning and expert knowledge — exactly the types of capabilities where test-time resolution scaling would not be expected to help. This is not a criticism of the model (a 7B model trailing GPT-4V on MMMU is hardly surprising), but the paper's framing as "matches or even surpasses GPT-4V" downplays the significant capability gaps that remain on the hardest reasoning tasks. A more balanced framing would be: IXC2-4KHD achieves competitive performance with GPT-4V on perception and OCR benchmarks, but substantial gaps remain on expert reasoning (MMMU, MathVista) and general visual understanding (MM-Vet).

Do the experiments demonstrate that scaling training resolution to 4K HD yields consistent performance improvements without saturation?

Yes, for OCR tasks specifically. Figure 5 shows monotonic improvement from HD-9 through 4KHD on InfographicVQA, DocVQA, and TextVQA, with no plateau. However, this finding is limited to the specific task category being evaluated. The paper's claim about "no ceiling of potential improvements" is empirically supported only for HD-OCR tasks — the perception benchmarks show saturation at much lower resolutions. This is acknowledged in the paper ("performance is saturated on the resolution that only has negligible difference between the four settings" for perception tasks) but the abstract's claim could be read as applying universally. The distinction is critical: the resolution ceiling is task-dependent, and 4K training is only worthwhile for applications that require reading fine text or discerning detailed structures.

A missing experiment that would strengthen this claim: a direct test of whether 4KHD training damages perception benchmark performance. The paper reports that perception tasks saturate (no benefit from 4K), but does not investigate whether the 4K training actively hurts them compared to training at a lower resolution. If training with long image token sequences (8070 tokens for 4KHD) causes the model to lose some general visual understanding capability due to attention diffusion or distribution shift in token lengths, that would be important to know. The mixed-resolution SFT strategy implicitly acknowledges this risk (by keeping non-OCR tasks at HD-25), but an ablation comparing pure HD-55 training against mixed training on perception benchmarks would directly test whether high-resolution training has asymmetric costs.

Do the experiments support the claim that dynamic resolution with automatic patch configuration is superior to prior patch-based methods?

The claim of superiority is supported by the benchmark comparisons (Tables 3–5), where IXC2-4KHD outperforms Monkey, TextMonkey, DocOwl 1.5, and other patch-based methods on all five HD-OCR benchmarks, often by large margins (e.g., +17.9 points on InfographicVQA over DocOwl 1.5). However, these comparisons conflate multiple factors: the maximum resolution (4K vs. ~1.5K), the dynamic aspect ratio handling, the newline token, the global view, the training data composition, the base LLM quality, and the training recipe. It is impossible to attribute the performance advantage to the dynamic partition mechanism specifically, as opposed to simply having a higher maximum patch budget. A missing ablation that would isolate the value of dynamic partitioning specifically: compare HD-25 dynamic (variable layouts per image) against HD-25 fixed (always 5 × 5 square grid, with padding/resizing to force that aspect ratio) on the same training data. This would reveal whether the adaptive aspect ratio handling contributes beyond the raw patch budget. The paper does not run this experiment, so the value of dynamic (as opposed to high) resolution is not experimentally isolated.

Do the inference-resolution experiments (Table 6) generalize beyond the specific model and training recipe?

The finding that inference at higher-than-training resolution improves text-related task performance is one of the paper's most interesting results, but its robustness is not deeply tested. The experiment covers three training resolutions (HD-9, HD-16, HD-25) and a few inference resolutions, but the relationship appears to have limits — the paper does not test how far this generalization extends. Can an HD-9 model inferred at 4KHD (HD-55) benefit, or does the gap become too large? Is there a point where the distribution shift in token count becomes harmful? The ChartQA degradation at higher inference resolution suggests there is a limit, and mapping out the shape of the generalization curve (does it plateau? peak and then decline?) would make the finding more actionable. Additionally, the mechanism is hypothesized ("dynamic image token length used in training enhances the robustness") but not experimentally verified — an ablation comparing a model trained with fixed patch counts vs. variable patch counts, evaluated on the inference-resolution generalization task, would directly test whether dynamic training is the causal factor enabling this generalization.

Were the right baselines included?

The paper compares against a comprehensive set of open-source LVLMs at similar scales and against the two dominant proprietary APIs. However, one natural baseline is missing: the original InternLM-XComposer2 without the 4KHD modifications, but evaluated at the same training compute budget. Since the 4KHD version processes more image tokens (up to 8070 vs. presumably fewer for the base XComposer2), it incurs higher inference cost. The paper does not provide a compute-normalized comparison showing that 4KHD is better per FLOP rather than simply better because it uses more compute. This is a recurring pattern in the paper — computational cost is acknowledged as a limitation but never accounted for in the comparisons. A compute-matched baseline (e.g., XComposer2 base model given additional test-time compute budget to match the FLOPs of 4KHD inference) would clarify whether the resolution mechanism is a genuine efficiency improvement or simply trades compute for accuracy in the expected way.

Data contamination and benchmark integrity.

The paper does not discuss data contamination between its training data and the evaluation benchmarks. Table 2 lists the SFT data sources, which include DocVQA, ChartQA, InfographicVQA, and TextVQA — the exact benchmarks being evaluated. If the SFT data includes the test sets or near-duplicates of test questions, the benchmark numbers would be inflated. The paper does not address this: there is no decontamination analysis, no description of train/test split adherence, and no mention of whether the evaluation was conducted on truly held-out data. This is a significant omission, especially given the paper's emphasis on benchmark results. The use of standard datasets with public test splits means data leakage is a genuine risk, and the paper's silence on this point makes it difficult for readers to assess the validity of the reported numbers. This is especially concerning for benchmarks like DocVQA, where the training data is widely available and could easily be mixed into the SFT corpus.

Scale of the ablation study test sets.

The ablation experiments in Sections 4.2 and 4.3 use very small evaluation sets: "validation set of InfoVQA, DocVQA, and TextVQA, test set of ChartQA and AI2D, MMBench EN-Test, and a 2k subset of SEEDBench." SEED-Bench has ~19K images in its image split, so 2K is roughly 10%. The validation sets for DocVQA, InfoVQA, and TextVQA are likely in the low thousands of questions. These small sets mean that differences of a few percentage points in the ablation tables (e.g., Table 8: 87.9% vs. 84.9% on DocVQA without newline tokens) represent relatively few questions — potentially tens of questions rather than hundreds. Without confidence intervals, it is unclear whether these ablation differences are robust or within sampling noise. This is not unique to this paper (it is standard practice in LVLM research), but it means the ablation results should be treated as indicative rather than definitive.

Single model architecture and training recipe.

All experiments use one specific configuration: InternLM2-7B + OpenAI ViT-L/14-336 + the paper's training pipeline. There is no evidence that the findings generalize to other base LLMs (e.g., LLaMA, Vicuna, Qwen), other vision encoders (e.g., SigLIP, EVA-CLIP), or other training recipes. The newline token's importance, for example, might depend on the specific LLM's ability to attend to special tokens — a different LLM architecture might handle unmarked row boundaries more gracefully. The resolution scaling curves in Figure 5 might have different shapes for a different vision encoder with different pretraining characteristics. The paper's contributions are demonstrated on a single model stack, and claims about general principles (e.g., "dynamic resolution with automatic patch configuration") should be understood as having been validated only in this specific configuration.

The "pioneering" and "first" claims.

The paper claims to be the first to expand LVLM resolution to 4K HD. This claim is probably correct within the scope of published LVLM research at the time, but the paper does not provide evidence that no prior work achieved anything close (e.g., there were no prior models operating at effective resolutions >2K). A systematic survey of maximum resolutions in prior LVLMs would strengthen this claim, but the paper relies on qualitative statements ("capped at approximately 1500 × 1500") without a comprehensive comparison table showing maximum effective resolution for each prior method.

6. Limitations and Trade-offs

Limitation 1: High-Resolution Training and Inference Is Computationally Expensive, and This Cost Is Never Normalized in Comparisons

The assumption or constraint. The paper's core contribution — scaling resolution to 4K HD — comes with a substantial computational cost that scales linearly with the patch budget $\mathcal{H}$ . Under the HD-55 setting used for 4K, each image produces up to 8,070 visual tokens after merging (144 global + 1 separate + 7,920 local + 5 newline tokens), compared to 1,561 tokens under HD-9 — a 5.2× increase in sequence length. The ViT processes each of the 55 patches independently, so the vision encoder cost scales directly with patch count. The paper explicitly acknowledges this burden but does not account for it in any benchmark comparison:

"we have not explored the upper bound due to the computational burden increasing with higher-resolution inputs. In future work, we plan to explore efficient solutions for accurate LVLM training and inference, enabling our model to handle even higher resolutions while maintaining computational efficiency." (Section 5)

The consequence. All benchmark comparisons in Tables 3–5 are compute-unaware: IXC2-4KHD is evaluated at 4K resolution while competing models are evaluated at their respective (typically lower) resolutions, with no normalization for inference FLOPs, latency, or memory. A practitioner reading the headline "90.0% on DocVQA, surpassing GPT-4V" has no way to assess whether this accuracy gain is worth the additional compute. The model could be achieving better accuracy simply because it processes more visual information, in which case the question becomes: does it use that information efficiently? Without a FLOPs-matched comparison — e.g., giving a competing model a test-time compute budget equivalent to 4KHD's inference cost — it is impossible to distinguish resolution efficiency from brute-force resolution scaling. This is particularly relevant for deployment decisions: a model that achieves 88% accuracy with 1/5 the compute may be preferable to one achieving 90% at full cost, depending on the application's cost-sensitivity.

What evidence exists in the paper. The paper provides token counts per resolution setting in Section 4.2 (1,561 / 2,653 / 4,057 / 8,737 tokens for HD-9 / HD-16 / HD-25 / 4KHD respectively) and acknowledges that "the computational burden increasing with higher-resolution inputs" is a limitation. However, no FLOPs-matched comparison is included. The resolution scaling curves in Figure 5 are plotted against resolution setting, not against compute cost, meaning the x-axis conflates resolution with FLOPs. A reader cannot extract a "accuracy per FLOP" curve from the presented data.

Mitigation status. The paper does not attempt to address this limitation beyond flagging it for future work on "efficient solutions for accurate LVLM training and inference." There is no compute-normalized baseline, no efficiency-prioritized Pareto frontier analysis, and no measurement of wall-clock inference time at different resolutions. The mixed-resolution SFT strategy (HD-55 for OCR, HD-25 for others) is a partial efficiency measure but is not evaluated against a compute-matched uniform-resolution baseline. A practitioner evaluating IXC2-4KHD for deployment against a cheaper model evaluated at lower resolution has no framework for making that tradeoff from the presented data.

Limitation 2: Hard Perception and Reasoning Benchmarks Show Substantial Gaps to GPT-4V That Resolution Scaling Cannot Close

The assumption or constraint. The paper's resolution-scaling approach operates on the input representation — making more visual detail available to the LLM — but does not improve the LLM's fundamental reasoning capabilities. The LLM (InternLM2-7B) is a 7B-parameter model, and no amount of additional image tokens can close the reasoning gap against much larger proprietary models on tasks that require multi-step inference, expert knowledge integration, or abstract reasoning from visual inputs. The paper implicitly assumes that the primary bottleneck for LVLM performance is visual resolution, but this assumption only holds for a subset of tasks.

The consequence. On the benchmarks where resolution does not help — most prominently MMMU and MathVista — IXC2-4KHD trails GPT-4V by large margins. The MMMU gap is 15.8 percentage points (41.0% vs. 56.8%), and the MathVista gap is approximately 5 points. These are the largest absolute differences anywhere in the benchmark tables, and they appear precisely on the benchmarks that require the deepest reasoning — MMMU spans expert-level questions across art, science, engineering, and medicine, demanding reasoning that exceeds what a 7B model can provide regardless of visual fidelity. The headline claim that IXC2-4KHD "matches or even surpasses GPT-4V and Gemini Pro in 10 of the 16 benchmarks" is accurate as a count but elides the fact that the losses are concentrated on the hardest reasoning benchmarks, while the wins are on benchmarks where resolution (rather than reasoning depth) is the primary differentiator. A practitioner whose application requires expert-level visual reasoning would find the 15.8-point MMMU gap disqualifying, regardless of performance on DocVQA.

What evidence exists in the paper. Table 3 provides the direct evidence: the three largest negative gaps to GPT-4V are MMMU (−15.8 points), MM-Vet (−7.9 points), and OCRBench (−6.1 points). Table 4 shows that IXC2-4KHD also trails open-source competitor InternVL-Chat-V1.2 on MathVista (57.5% vs. 62.0%). Figure 5 shows that resolution provides "negligible" benefit on perception benchmarks — meaning the bottlenecks on these tasks are not addressable through the paper's core contribution.

Mitigation status. The paper does not address this limitation directly. The choice of InternLM2-7B as the LLM backbone is a fixed architectural decision, and no experiments investigate whether a larger LLM combined with the same resolution strategy would narrow the reasoning gap. The paper frames itself as demonstrating that resolution scaling can achieve competitive performance, not that it matches GPT-4V in all respects, but the asymmetry of the gains and losses (resolution helps OCR, does not help reasoning; the model wins where resolution matters most and loses where it does not) is an important deployment consideration that is not foregrounded in the abstract or conclusions.

Limitation 3: Dynamic Training Resolution Is a Distributional Challenge That Degrades Chart Understanding When Inference Resolution Exceeds Training Resolution

The assumption or constraint. The paper's dynamic resolution training is designed to teach the model robustness to variable image token lengths, enabling generalization to higher inference resolutions (Section 4.2, Table 6). The implicit assumption is that this generalization is uniformly beneficial: if the model trained with variable patch counts, it should handle more patches at inference without issue. The paper discovers that this assumption does not hold for all task types.

The consequence. ChartQA performance consistently degrades when inference resolution exceeds training resolution. Table 6 shows IXC2-HD9 dropping from 67.3% → 63.0% (−4.3 points) when inferred at HD-16, and IXC2-HD16 dropping from 67.1% → 65.3% (−1.8 points) when inferred at HD-25. This is not a marginal or noisy result — it is consistent across both training resolution conditions and is the only task category where this pattern appears. The paper speculates that "this could be due to the model becoming confused about the chart structure when the resolution is altered" (Section 4.2). This failure mode has practical implications: a practitioner deploying IXC2-4KHD for chart-heavy applications (financial dashboards, scientific figure analysis, business analytics) cannot simply increase inference resolution to "get better results" — they may get worse results. The optimal inference resolution becomes a hyperparameter that must be tuned per task, and the direction of the tuning is not uniform.

What evidence exists in the paper. Table 6 provides the direct measurements. The paper also notes that ChartQA performance does improve with higher training resolution in Figure 5, establishing an asymmetry: training at higher resolution helps ChartQA, but inferring at higher-than-training resolution hurts it. The paper's hypothesis about "chart structure confusion" is speculative — no experiment isolates whether the degradation is due to attention diffusion over longer sequences, disruption of learned spatial-proportional relationships, or some other mechanism.

Mitigation status. The paper acknowledges the ChartQA degradation as an empirical observation but does not investigate its cause or propose a mitigation. The mixed-resolution SFT strategy trains ChartQA at HD-25 for non-OCR tasks (Table 2), but the inference-resolution issue means that even a well-trained model can be degraded by deployment-time resolution choices. A practitioner would need to run their own evaluation to determine the optimal inference resolution for chart tasks, as the paper provides no general guidance beyond the warning that degradation occurs.

Limitation 4: Data Leakage Risk from Training on the Exact Benchmarks Being Evaluated

The assumption or constraint. The paper's supervised fine-tuning data includes the training splits of the exact benchmarks it evaluates on — DocVQA, ChartQA, InfographicVQA, TextVQA, and others are listed in Table 2 as SFT data sources. The paper assumes (implicitly, by not discussing contamination) either that the training and test splits are cleanly separated, or that any overlap does not materially affect the reported numbers.

The consequence. If any test-set questions or near-duplicates appear in the SFT data, the benchmark numbers in Tables 3–5 could be inflated relative to what a truly held-out model would achieve. This is not a hypothetical concern — it is a well-documented problem in LLM benchmarking where test-set leakage through training data can produce artificially high scores that do not reflect genuine generalization. For IXC2-4KHD specifically, the risk is elevated because the model is trained on the benchmark training splits (unlike GPT-4V and Gemini Pro, whose training data composition is proprietary and may include the benchmarks or may not). The reported margins over closed-source models — e.g., 90.0% vs. 88.4% on DocVQA — could partially reflect a memorization advantage rather than superior visual understanding, and without decontamination analysis, the reader cannot assess how much of the gain is genuine.

What evidence exists in the paper. None. The paper does not mention data decontamination, does not describe procedures for ensuring train/test separation, and does not report any analysis of potential overlap between SFT data and test sets. Table 2 lists the data sources but does not specify whether the standard train/validation/test splits were respected. The VLMEvalKit evaluation framework is cited for benchmark evaluation but its data handling is not described. This is a complete absence of evidence, which is itself the concern — a paper making strong benchmark-based claims against proprietary models should ideally demonstrate that those claims are not contaminated by training on the target distributions.

Mitigation status. Not addressed. The paper does not acknowledge data contamination as a concern, propose any mitigation, or provide any analysis of the issue. This is a significant methodological gap, especially given the emphasis on benchmark numbers as the primary evidence of the model's capabilities.

Limitation 5: Single Model Architecture and Training Recipe — No Evidence That Findings Generalize Beyond the InternLM2 + ViT-L/14-336 Stack

The assumption or constraint. All experiments use one specific configuration: InternLM2-7B as the LLM, OpenAI CLIP ViT-Large/14 at 336×336 as the vision encoder, the concatenation-based token merging strategy, and the specific pre-training + SFT pipeline described in Sections 3.3–3.4. The paper implicitly assumes that the dynamic resolution mechanism, the newline token's importance, the global view's contribution, and the resolution scaling curves are properties of the method rather than properties of this specific implementation.

The consequence. A practitioner who wants to apply dynamic resolution with automatic patch configuration to a different model stack — say, LLaMA-3 as the LLM with SigLIP as the vision encoder, or a model with a different vision-language connector — has no guidance on whether the findings transfer. Would the newline token matter as much with a different LLM's positional encoding scheme? Would the resolution scaling curve look different with a vision encoder pretrained at a different resolution? Would the concatenation merging strategy still outperform learned resampling with a larger pre-training dataset? The paper provides no cross-architecture validation, no sensitivity analysis to vision encoder choice, and no experiments with alternative LLM backbones. The contribution is demonstrated on exactly one model stack, and claims about general principles (e.g., "the newline token helps LVLMs understand structural images under dynamic resolution") are supported by evidence from that single stack only.

What evidence exists in the paper. The paper references the original XComposer2 architecture (Section 3.1) and notes that the "model architecture mainly follows the design of InternLM-XComposer2," but provides no architectural ablations beyond the token merging strategy (Table 9) which compares connectors within the same overall stack. There are no experiments with different ViT variants, different LLM families, or different pre-training data scales. The resolution scaling curves (Figure 5) reflect one particular ViT's ability to extract information from 336×336 patches at different patch densities — a different ViT might saturate earlier or later.

Mitigation status. Not addressed. The paper treats the architecture as fixed and focuses all innovation on the resolution mechanism. This is reasonable for a model-release paper (the contribution is a specific model, not a universal recipe), but the framing as a general method ("dynamic resolution with automatic patch configuration") implies broader applicability that is not empirically supported. A practitioner considering adopting the method for a different model family would need to run their own validation to confirm that the findings replicate.

Limitation 6: Difficulty Estimation Cost and Static Resolution Allocation — No Mechanism for Per-Instance Adaptive Budgeting

The assumption or constraint. The dynamic partition mechanism selects the patch layout per image based on the image's aspect ratio and a fixed maximum budget $\mathcal{H}$ , but it does not adapt the budget itself based on the image's content. A document page with large, easily-readable text receives the same budget (under HD-55) as one with tiny, dense text that genuinely requires 4K resolution to read. The paper assumes that allocating maximum resolution uniformly to all images in a task category (OCR vs. non-OCR) is optimal, but provides no mechanism for per-image assessment of whether that resolution is actually needed.

The consequence. In deployment, this is a wasteful allocation of compute. Many document images — forms with large field labels, slides with readable text, receipts with standard font sizes — can be answered correctly at much lower resolutions. Processing every image at 4K (7,920 local tokens) when HD-9 (1,440 local tokens, a 5.5× reduction) would suffice wastes inference FLOPs with zero accuracy gain. This is analogous to the difficulty estimation problem in test-time compute scaling — the paper's resolution allocation is uniform within task categories rather than adaptive to instance difficulty. A deployment handling millions of document queries would incur the full 4K inference cost on every query, even though a large fraction could be answered correctly at a fraction of that cost. The paper's mixed-resolution SFT strategy (HD-55 for OCR, dynamic for others) operates at the dataset level, not the instance level — it is a training design choice, not a deployment-time optimization.

What evidence exists in the paper. Figure 5 and Table 6 show that average performance improves with resolution, but these are aggregate statistics. The paper does not report any breakdown of how performance varies with resolution within a dataset as a function of image content — e.g., do images with small text benefit more from 4K than images with large text? No per-instance resolution sensitivity analysis is provided. The finding that inference at higher-than-training resolution helps on average (Table 6) does not preclude many individual images receiving zero benefit (or even degradation, as with ChartQA).

Mitigation status. Not addressed. The paper does not propose any mechanism for adaptive resolution budgeting at inference time, and the difficulty estimation problem is not discussed. This is a missed opportunity for efficiency: if a lightweight classifier could predict whether an image needs 4K resolution to be answered correctly (analogous to the difficulty estimator in test-time compute scaling papers), the average inference cost could be substantially reduced without sacrificing accuracy. The paper's future work on "efficient solutions" (Section 5) likely refers to architectural efficiency rather than adaptive budgeting, but both directions are relevant.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new paradigm for vision-language modeling — it works squarely within the established patch-division framework that Monkey, TextMonkey, and others pioneered. What it changes is the perceived ceiling on that paradigm. Before this work, the field operated with an implicit assumption that patch-based high-resolution methods topped out around 1,500 × 1,500 pixels. The evidence for this assumption was circumstantial — prior methods reported results at those resolutions, and no one had demonstrated meaningful gains beyond them — but it shaped research priorities. Teams invested in alternative approaches (high-resolution encoders, dual-encoder architectures, frequency-domain processing) partly because the patch-division path seemed to have reached diminishing returns.

The paper's central empirical finding — that scaling resolution from ~1K to 4K yields consistent, unsaturating performance improvements on OCR tasks (Figure 5) — shifts the perceived ceiling upward by roughly an order of magnitude in pixel count. This is not a paradigm shift, but it is a substantial reframing of what the patch-division approach is capable of. The implication is that prior work was not hitting a fundamental limit of the method but rather an artifact of conservative patch budgets. The paper's maximum budget of H=55 patches produces an effective resolution of 3,696 × 1,680 — far beyond the ~1,500 × 1,500 of Monkey or LLaVA-NeXT — and the absence of saturation at that level suggests budgets of H=100 or H=200 could yield further gains.

This reframing matters because it redirects research effort back toward the patch-division paradigm. Approaches that add high-resolution encoders (Vary, CogAgent, Mini-Gemini) or operate in alternative representational domains (DocPedia's frequency-domain processing) now face a higher bar: they must demonstrate advantages over simply pushing the patch budget higher, which requires no new architecture, no additional encoder training, and no domain-specific representations. The conceptual load shifts from "how do we design a vision pipeline that handles high resolution?" to "how do we make the patch-division pipeline efficient enough to scale to extremely high patch counts?" This is fundamentally an efficiency problem, not a representation problem — and it repositions research from architectural innovation toward systems optimization.

The paper also reconciles a latent tension between the self-correction and text-reading literatures. Prior work on high-resolution LVLMs focused almost exclusively on OCR and document tasks, while general-purpose LVLMs operated at low resolution and performed poorly on text-heavy benchmarks. The implicit narrative was that high-resolution capability comes at the cost of general visual understanding — that specializing for documents means sacrificing performance on natural images, scene understanding, and reasoning. The paper's demonstration that a single model can achieve state-of-the-art OCR performance while remaining competitive on perception and reasoning benchmarks (Tables 3-4) breaks this tradeoff narrative. The resolution mechanism, because it operates at the input level without modifying the vision encoder's pretrained representations, is orthogonal to general visual capability — it adds fine-detail information without removing or distorting the macro-level features that drive performance on standard benchmarks. This is an existence proof that the field does not need to choose between document fluency and general visual intelligence.

A more subtle shift concerns how we think about training data for vision-language models. The paper's dynamic partition mechanism, applied during training, effectively synthesizes higher-resolution training experiences from existing data — a 1,008 × 1,008 image, divided into 9 patches, teaches the model to process that effective resolution without requiring native 1K or 2K training images. This is a conceptual inversion of the standard data-scaling narrative: rather than collecting new data to match the desired input resolution, the method stretches the resolution budget of existing data. If this logic extends, a single curated dataset could support training at multiple effective resolutions, amortizing data collection costs across a family of models. This changes how we evaluate the cost of resolution — it is primarily a compute cost (more patches = more ViT forward passes and longer LLM sequences), not a data cost. For organizations with large image-text corpora but limited high-resolution content, this is a significant practical insight.

Follow-Up Research This Work Enables

Measuring the shape of the resolution-accuracy curve beyond 4KHD. The paper demonstrates that performance on InfographicVQA, DocVQA, and TextVQA continues improving from HD-9 through 4KHD with no observed saturation (Figure 5), but stops at H=55 due to computational constraints. The natural next step is to map the full curve: does the benefit continue linearly, decelerate, or eventually plateau? A strong experiment would train a series of models at HD-9, HD-25, HD-55, HD-100, and HD-200 (if computationally feasible) on a controlled data budget, measuring accuracy per unit of inference compute on the 5 HD-OCR benchmarks. The shape of this curve has direct practical implications — if gains persist to HD-200, the field should invest in efficiency techniques to make those resolutions deployable; if the curve plateaus at an effective resolution of, say, ~8K, then we have identified a natural resolution ceiling for the patch-division approach that would justify a return to alternative architectures.

Isolating the causal contribution of dynamic aspect ratio handling from raw patch budget. The paper's benchmark comparisons (Tables 3-5) show IXC2-4KHD outperforming prior patch-based methods, but these comparisons confound multiple factors: higher maximum patch count, dynamic aspect ratio adaptation, the newline token, the global view, and training data composition. A controlled ablation would compare three conditions on identical training data: (a) dynamic partition with H=25 (the paper's method), (b) fixed-grid partition with H=25 where all images are resized and padded to a 5×5 square grid regardless of aspect ratio, and (c) fixed-grid partition with H=25 where images are center-cropped to a square and then partitioned 5×5. The difference between (a) and (b) would isolate the value of aspect ratio preservation; the difference between (b) and (c) would isolate the value of complete image coverage. If dynamic partitioning provides minimal advantage over square-grid-with-padding on benchmark averages, the paper's headline innovation (automatic patch configuration) would be revealed as less important than the raw patch budget — a negative result that would refocus effort on efficiency rather than layout algorithms.

Decontamination analysis and benchmark integrity verification. The paper trains on the training splits of DocVQA, ChartQA, InfographicVQA, and TextVQA (Table 2) and reports test-set performance without any discussion of data leakage. A critical follow-up is a systematic decontamination study: for each benchmark, measure the semantic similarity (using embedding-based or n-gram overlap methods) between SFT training samples and test-set questions, identify any near-duplicates, and report performance both with and without those samples included in training. If removal of near-duplicate training samples causes a significant performance drop (e.g., >2 percentage points on DocVQA), the paper's claims of surpassing GPT-4V would be partially attributable to memorization rather than genuine visual understanding. This type of audit is becoming standard practice in LLM evaluation and its absence here is a gap that needs filling before the reported numbers can be fully trusted.

Understanding and mitigating the ChartQA inference-resolution degradation. The paper reports that ChartQA performance consistently degrades when inference resolution exceeds training resolution (Table 6: IXC2-HD9 drops from 67.3% to 63.0% at HD-16 inference, and IXC2-HD16 drops from 67.1% to 65.3% at HD-25 inference). The paper speculates about "chart structure confusion" but provides no mechanistic investigation. A strong follow-up would design diagnostic experiments: (a) test whether the degradation is due to attention diffusion over longer sequences by comparing ChartQA performance at HD-16 inference for models trained at HD-9 vs. models trained at HD-9 but with sequence-length-matched padding during training (so the model is accustomed to long sequences even if the extra tokens are padding), (b) measure whether the degradation is concentrated in specific chart types (bar charts vs. line charts vs. pie charts) or specific question types (reading specific values vs. understanding trends vs. comparing quantities), and (c) test whether providing the global view at the training resolution while scaling only the local view to higher resolution mitigates the degradation (the global view would maintain the learned spatial scale, while the local view would provide finer textual detail). Understanding the mechanism is important because chart reading is a high-value application, and the current guidance — "don't increase inference resolution for charts" — is an unsatisfying rule of thumb without understanding the underlying cause.

Dynamic, per-instance resolution allocation. The paper's resolution strategy is uniform: all OCR-task images get HD-55 during SFT and inference. This wastes compute on images where lower resolution would suffice — a document with 24-point font does not need 4K. A natural extension is to train a lightweight resolution-predictor module that, given only the global view (336 × 336 thumbnail), predicts the minimum effective resolution needed to answer the question correctly. This could be trained using the PRM's final-answer score or correctness signal as a supervisory target: for each training sample, determine the lowest H-setting at which the model produces the correct answer, and train a classifier to predict that H from the global view features. At inference time, the predictor estimates difficulty and routes easy images to a low-resolution pipeline and hard images to the full 4K pipeline. The paper's resolution robustness finding (Table 6) — that models generalize to higher-than-training resolution — provides evidence that such a routing system would not need retraining for each H level, since a single model can handle multiple resolutions. The expected benefit: substantial inference cost reduction (potentially 2-4×, given that many document images do not require 4K) with minimal accuracy loss, analogous to the compute-optimal scaling gains observed in language model test-time compute allocation.

Cross-architecture validation of the newline token's importance. The paper demonstrates that the newline token is critical under the 4KHD setting (Table 8: removing it drops DocVQA from 87.9% to 84.9% and InfographicVQA from 65.0% to 60.2%) but only for the InternLM2-7B + ViT-L/14-336 stack. Would a different LLM architecture (with different positional encoding schemes — e.g., RoPE vs. ALiBi vs. learned absolute positions) show the same sensitivity? A strong follow-up would replicate the newline token ablation on at least two additional LLM families (e.g., LLaMA-3-8B with RoPE and a Mamba-based model with no explicit position encodings) using the same vision encoder. If the benefit of the newline token is consistent across LLM architectures, it supports the paper's claim that explicit spatial grammar is a general requirement for 1D-sequence models processing 2D variable-layout images. If the benefit disappears for certain positional encoding schemes (e.g., RoPE's relative position encoding might already provide sufficient spatial disambiguation), it would refine our understanding: the newline token is a compensatory mechanism for specific LLM inductive biases, not a universal design principle. This distinction matters for architecture design in future multimodal models.

Practical Applications and Downstream Use Cases

Large-scale document digitization and question-answering pipelines. Organizations with large repositories of scanned documents — legal firms with case files, healthcare systems with patient records, government agencies with archival documents — could deploy IXC2-4KHD (or a model using the same dynamic resolution approach) to enable natural-language querying of document collections. The concrete benefit: the model's 90.0% accuracy on DocVQA and 68.6% on InfographicVQA means that, for the first time, an open-source 7B model can answer questions about document content at a level competitive with GPT-4V, without sending sensitive documents to external APIs. The 4K resolution handling is critical here because scanned documents at 300 DPI produce images where text is legible only at effective resolutions exceeding 2,000 pixels on the longer side — exactly the regime where prior open-source models capped at ~1,500 pixels would fail. A deployment scanning 10 million document pages would process each at 4K (potentially 8,000+ tokens per page), so the computational cost is substantial, but the alternative (manual review or proprietary API costs) may justify it for high-value document workflows.

Automated UI testing and GUI agent systems. A model that can read text at native screen resolutions is a prerequisite for automated GUI agents that interact with software through visual perception — clicking buttons, reading error messages, understanding form layouts. IXC2-4KHD's 4K capability means it can process full-resolution screenshots of modern displays (3840 × 2160) without downsampling, preserving button text, menu labels, and small UI elements that would be illegible at lower resolutions. The model's strong performance on TextVQA (78.1%, competitive with GPT-4V) demonstrates its ability to read text in natural images, which translates directly to reading UI text. A practical deployment would integrate IXC2-4KHD into an agent loop: (1) capture a 4K screenshot, (2) use the model to answer "what is the text of the error message?" or "which button says 'Submit'?", (3) use the model's answer to drive the next UI action. The dynamic resolution handling is particularly valuable here because UI screenshots vary enormously in aspect ratio — tall scrolling pages, wide dashboards, modal dialogs — and prior fixed-grid methods would distort or crop critical UI elements.

Scientific figure and chart accessibility. Making scientific literature accessible to visually impaired readers requires models that can read figure labels, extract data from charts, and describe visual trends in natural language — all tasks that demand high-resolution perception. IXC2-4KHD's 81.0% on ChartQA and the mixed-resolution training strategy (which preserves performance on general visual understanding while adding OCR capability) make it a strong candidate for automated figure description pipelines. A deployment scenario: a preprint server processes thousands of PDFs daily, extracting figures and routing them to IXC2-4KHD for description generation. The model reads axis labels, legend entries, and data point values at 4K resolution, then generates a natural-language summary of the figure's message. The paper's finding that ChartQA degrades under inference-resolution scaling (Table 6) is an important caveat here — the deployment should use the same resolution for chart tasks as was used during training, not arbitrarily increase it expecting better results. This limitation means the pipeline would need task-specific resolution routing: 4K for text-heavy infographics, HD-25 for chart understanding, and dynamic resolution for general images.

Education technology and assessment grading. Automated grading of handwritten or printed student work — math problem sets, lab reports, annotated diagrams — requires reading text at the resolution of the original submission. A 4K-capable model can process high-resolution scans of student work without the information loss that downsampling causes, potentially reading marginal annotations and small mathematical notation that would be lost at lower resolution. The model's MathVista performance (57.5%) is behind the best proprietary models but competitive among open-source 7B models, suggesting it could handle the reading-and-transcription component of grading (extracting what the student wrote) even if a larger reasoning model is needed for the scoring component. The key practical advantage is that IXC2-4KHD is open-source and can run on institutional infrastructure, avoiding the privacy and compliance issues of sending student work to external APIs — a critical consideration for educational deployments.