Revisiting Multimodal Positional Encoding in Vision-Language Models

ArXiv: 2510.23095

🎯 Pitch

Simply giving 3D video coordinates the full frequency spectrum of rotary embeddings—instead of partitioning channels into t/h/w chunks—can boost long-video understanding by 7+ points while fixing a catastrophic 20-point drop in document QA suffered by prior methods. This extensive benchmark sweep reveals that position encoding failures in VLMs are predictable from three principles, and two plug-and-play RoPE variants (MHRoPE and MRoPE-I) that follow them consistently outperform specialized designs across image, video, and grounding tasks.

---

1. Executive Summary

This paper conducts a systematic empirical analysis of multimodal Rotary Positional Embedding (RoPE) for vision-language models, decomposing the design space into three axes—position design, frequency allocation, and compatibility with text-only RoPE—and evaluating seven existing methods plus two novel variants on over 20 image, video, and grounding benchmarks using Qwen2.5-VL-7B as the backbone. From this analysis, the authors distill three core guidelines—positional coherence (ensuring unambiguous layout and appropriate modality intervals), full frequency utilization (giving each positional axis access to the complete frequency spectrum rather than chunking it into non-overlapping bands), and preservation of textual priors (keeping the text RoPE identical to the pre-trained LLM's)—which collectively explain why prior methods fail in predictable ways, such as VideoRoPE's diagonal layout causing modalities-confusion-driven collapse on document benchmarks (dropping DocVQA from ~82% to ~60%) or MRoPE's chunked frequency allocation impairing long-video understanding. Guided by these principles, the paper proposes two plug-and-play variants—Multi-Head RoPE (MHRoPE) (dedicating distinct attention heads to different positional axes, preserving full frequency resolution per axis) and MRoPE-Interleave (MRoPE-I) (distributing channels to t/h/w axes in a fine-grained round-robin pattern so each axis sees the full spectrum)—both of which incorporate spatial-reset, a mechanism that resets spatial coordinates for each visual content to leverage the LLM's attention-sink bias toward small position IDs. Across the full benchmark suite, MRoPE-I achieves the best overall performance, outperforming vanilla RoPE by 2.67 points on MMMU, 5.28 points on ChartQA, and 3.27 points on RefCOCOval, while MHRoPE and MRoPE-I together establish that providing each positional axis with the full frequency spectrum—rather than partitioning it—is the key driver of robust multimodal positional encoding, with the gains generalizing across distinct VLM architectures (Qwen3-VL-4B and -8B) and remaining effective only when text RoPE compatibility is strictly preserved.

2. Context and Motivation

The Core Problem: Multimodal Positional Encoding Is Designed Ad Hoc

The fundamental question this paper tackles sounds deceptively simple: how should a vision-language model assign position IDs to tokens from different modalities? Despite positional encoding being a foundational component of every transformer-based VLM—it determines how the model perceives spatial relationships in images, temporal order in videos, and sequence structure in text—the paper argues that the field has no principled framework for designing it. Instead, the landscape has become "fragmented" with "highly specialized solutions" (Section 1), each targeting a specific modality or application niche without a unified understanding of what makes one design better than another.

This matters for several concrete reasons that the paper identifies:

• Three-dimensional visual structure is lost in naive approaches. The simplest method—flattening everything into a 1D sequence and applying vanilla RoPE—discards the native geometry of visual content. A token at pixel (h=10, w=20) in frame t=3 has spatial relationships that a flat position index cannot express. The paper shows this degrades performance on tasks requiring spatial reasoning, such as visual grounding (RefCOCO) and chart understanding (ChartQA), where vanilla RoPE consistently underperforms multi-dimensional designs (Table 2).

• Existing multi-dimensional designs introduce their own pathologies. Methods that do preserve 3D structure (MRoPE, VideoRoPE, HoPE, CircleRoPE, IL-RoPE, Omni-RoPE) each make different choices about how to assign coordinate values and how to distribute the model's frequency spectrum across positional axes. These choices are not cosmetic—they create qualitatively different failure modes. VideoRoPE's "diagonal layout" causes position ID overlap between visual tokens and generated text tokens, leading to "modalities confusion in generation" where the model produces nonsensical repetitions like "1111..." (Section 2.2.2, Table 4). MRoPE's chunked frequency allocation forces the temporal axis entirely into high-frequency channels, creating a "rapid decay of attention over time" that impairs long-video understanding (Section 2.3.2). CircleRoPE collapses all video frames onto a single spatial ring, eliminating temporal information entirely for video inputs.

• No method has been systematically compared on equal footing. The paper notes (Section 1, Table 1) that prior methods were developed and evaluated in isolation—some optimized for image understanding only, others for video comprehension, still others for image generation. No prior work had implemented all these methods in a controlled setting with identical training data, model architecture, and hyperparameters, making it impossible to determine whether performance differences came from the positional encoding design or from confounding factors like different base models, different training recipes, or different data scales.

Why This Problem Matters Beyond Academic Curiosity

The paper's investigation has direct practical implications for the VLM research community, which is converging toward a standardized architecture: vision encoder → connector → LLM backbone (Section 3.3). Since positional encoding operates entirely within the LLM backbone, and since most LLM backbones share a standard transformer architecture with minimal structural variation, the choice of multimodal RoPE strategy can be ported across different VLM families with minimal engineering effort. The paper demonstrates this generalization explicitly by replicating their experiments on Qwen3-VL-4B and -8B architectures (Table 3, Tables 10–11), which differ from the main Qwen2.5-VL backbone in multiple architectural details (removal of window attention, addition of DeepStack, application of QK-Norm). The consistent superiority of MHRoPE and MRoPE-I across these architectural variants confirms that the guidelines and methods are not backbone-specific.

There is also a cost argument. The paper shows that poorly-chosen position designs can cause catastrophic degradation: VideoRoPE drops DocVQA from ~82% to ~60% compared to vanilla RoPE, despite being optimized for video tasks (Table 2). For practitioners building general-purpose VLMs, selecting the wrong positional encoding strategy can silently destroy performance on an entire task category—damage that might be misattributed to other components of the system during debugging. The paper's systematic ablation framework (Tables 4, 5) provides a diagnostic toolkit for understanding why a particular method fails.

At a theoretical level, the paper reveals that positional encoding design is not a monotonic optimization problem where "more structure" or "more axes" is always better. The failure modes are non-obvious: adding a diagonal layout to improve inter-modal symmetry destroys document understanding (Table 4); enlarging the modality interval to match vanilla RoPE's spacing causes the model to ignore visual input entirely (Section 3.4.1); scaling rotary bases to better match spatial coordinate ranges breaks knowledge transfer from the pre-trained LLM (Table 4). These counterintuitive results suggest that multimodal RoPE design occupies a narrow performance landscape where small deviations from optimal choices produce disproportionately large penalties.

Where Prior Approaches Fall Short

The paper identifies specific limitations in existing methods along three design axes, summarized in Table 1 and analyzed in detail throughout Section 2:

Position Design Failures (Section 2.2)

1D sequential approaches (vanilla RoPE, V2PE): These discard 3D visual structure entirely. V2PE's dynamic position scaling for visual tokens partially mitigates the rapid growth of position IDs, which helps with long-context extrapolation (Figure 5), but cannot recover the lost spatial information. The paper confirms this empirically: simply adding 3D structure to vanilla RoPE provides an immediate boost to grounding performance (Table 4, "vanilla RoPE → + 3D structure" raises grounding from 73.48 to 74.40).

MRoPE's coordinate assignment: While MRoPE preserves 3D structure and successfully avoids positional overlap between modalities using its "max-plus-1" update rule (Equation 2), the paper discovers it exhibits a visual attention sink—attention concentrates on the top-left corner of images and the first frame of videos (Figure 2). This is analogous to the well-known attention sink phenomenon in LLMs where the initial tokens absorb disproportionate attention. The paper argues this happens because MRoPE assigns small position IDs to the (t=0, h=0, w=0) region, which aligns with the LLM's bias toward small indices. While not catastrophic, this biases the model's visual processing in ways that spatial-reset can redirect more productively.

VideoRoPE/HoPE's diagonal layout: The diagonal layout (Figure 1c) shifts spatial coordinates along the vertical and horizontal axes for each video frame, creating what the authors describe as a theoretically elegant "inter-modal symmetry." The paper's experiments reveal this elegance is illusory: for high-resolution images like documents, the shifted spatial coordinates can extend into the index range later assigned to generated text tokens. This causes the model to confuse its own generated output with visual input—a phenomenon the authors term "modalities confusion in generation" (Section 2.2.2). The consequence is catastrophic on document benchmarks, where VideoRoPE achieves only 60.13 on DocVQA versus ~82% for MRoPE (Table 2), and the ablation in Table 4 confirms the diagonal layout specifically is responsible for this collapse.

CircleRoPE's collapsed temporal axis: By arranging all visual tokens on a circle orthogonal to the text axis (Figure 1d), CircleRoPE makes all visual tokens equidistant from text tokens, promoting uniform attention across images. However, this design has no temporal dimension—all video frames are projected onto the same ring, eliminating the model's ability to distinguish time points. The paper finds this produces poor video understanding (Table 2: CircleRoPE achieves only 51.09 average video score, versus 52.95 for HoPE and 52.58 for MHRoPE).

IL-RoPE/Omni-RoPE's text compatibility break: These methods reset spatial coordinates for each image (similar to spatial-reset) but also set the spatial dimensions of text tokens to zero (Figure 1e). This breaks the alignment with the pre-trained LLM's RoPE, which uses a single 1D position index. The ablation in Table 4 ("+ text spatial-reset") shows this causes a significant performance drop across all categories—image scores drop from 66.65 to 58.27, grounding from 75.85 to 68.20—confirming that "preserving RoPE alignment for text is critical for successfully adapting LLMs into VLMs" (Section 3.4.1).

Frequency Allocation Failures (Section 2.3)

MRoPE's chunked allocation: The standard MRoPE partitions the d feature dimensions into three contiguous blocks dedicated to t, h, and w respectively (Section 2.3.2). Because rotational frequencies decay with channel index, this forces the temporal axis to occupy exclusively the highest-frequency channels (those with small i, corresponding to rapidly decaying attention). The consequence is a "rapid decay of attention over time" that impairs long-range temporal modeling—the model struggles to connect information across distant frames. Additionally, the h and w axes receive different, non-overlapping frequency ranges, giving them asymmetric long-range decay properties (Figure 4a). This asymmetry means the model cannot learn consistent spatial relationships—horizontal and vertical distances are encoded with fundamentally different frequency characteristics.

VideoRoPE/HoPE/IL-RoPE's correction creates a new problem: These methods attempt to fix MRoPE's temporal bias by moving the temporal axis to low-frequency channels, which preserves long-range temporal attention. However, the paper identifies an unaddressed trade-off: this forces the spatial dimensions into "a restricted, and often exclusively high-frequency, band" (Section 2.3.2). This severely limits multi-scale spatial reasoning—the model loses the ability to capture both fine-grained details (which require high frequencies) and global spatial layouts (which require low frequencies). The performance impact is visible in Table 2: VideoRoPE and HoPE achieve only 66.20 and 66.60 on OCRBench (which requires fine-grained character-level spatial understanding) versus 72.70 for MRoPE and 74.00 for MRoPE-I.

Frequency resolution coarsening: Any method that partitions feature dimensions across axes inherently reduces the frequency resolution available to each axis. With d/3 channels per axis instead of d, the geometric progression of frequencies becomes coarser—there are fewer distinct frequency values, meaning less granular control over the attention decay pattern. The paper notes this "performance implications of this reduced granularity is under-exploration" (Section 2.3.2), and Table 5 shows that more uniform allocation strategies (interleave, multi-head) consistently outperform chunked approaches.

Compatibility Failures (Section 2.4)

The paper identifies that some methods violate what should be an implicit constraint: text tokens should use exactly the positional encoding that the LLM was pre-trained with. The experiments in Table 4 confirm this is not merely theoretical. When the authors scaled the rotary base for spatial dimensions (from 1,000,000 to 10,000, motivated by the smaller coordinate range), image performance dropped from 66.65 to 60.15. When they applied text spatial-reset (setting text spatial dimensions to zero per IL-RoPE/Omni-RoPE), image performance dropped from 66.65 to 58.27. Both modifications were "well-intentioned" but broke the transfer of pre-trained knowledge from the LLM.

How This Paper Positions Itself

The paper frames itself not as proposing a single new method but as conducting the first systematic investigation into multimodal RoPE design (Section 1: "there has been little systematic investigation into multimodal position encoding"). Its contribution is threefold:

1. An analytical framework. By decomposing multimodal RoPE into three independent design axes (position design, frequency allocation, text compatibility) and evaluating seven methods under controlled conditions, the paper provides the first unified lens for understanding why different methods succeed or fail. This is not a taxonomic exercise—the decomposition yields predictive power. For instance, knowing that VideoRoPE's diagonal layout creates position ID overlap explains its collapse on document benchmarks; knowing that MRoPE's chunked frequencies restrict spatial multi-scale modeling explains its underperformance on grounding.

2. Three design guidelines that emerge from the analysis. The guidelines (positional coherence, full frequency utilization, preservation of textual priors) are not axioms assumed beforehand but empirical distillations from the systematic comparison. Each guideline has a specific failure mode associated with its violation: positional ambiguity → modalities confusion (VideoRoPE); restricted frequency spectrum → impaired multi-scale modeling (MRoPE's spatial. axes); broken text RoPE → failed knowledge transfer (IL-RoPE/Omni-RoPE).

3. Two methods that implement the guidelines and demonstrate their necessity. MHRoPE and MRoPE-I are deliberately simple—"plug-and-play variants that require no architectural changes" (Section 1)—to isolate the effect of the design principles from confounding architectural innovations. Their consistent superiority across benchmarks (Tables 2, 3, 10, 11) serves as evidence that the guidelines are sufficient for robust multimodal RoPE design.

The paper explicitly contrasts its holistic approach with the fragmented prior landscape (Section 1): "Some methods focus exclusively on image understanding, others on video comprehension, and a third group on image generation... the development of a truly robust and versatile VLM requires a more holistic positional encoding strategy." The authors position MHRoPE and MRoPE-I as general-purpose solutions that work across images, videos, and grounding simultaneously—a claim supported by the completeness of their evaluation suite (20+ benchmarks) and the replication across architectures.

A subtle but important aspect of the paper's positioning: it does not claim that either MHRoPE or MRoPE-I is the "final answer." Section 2.5 and Appendix D.1 explicitly discuss trade-offs between the two methods—MRoPE-I is simpler to implement and slightly more performant, while MHRoPE is more scalable to additional positional axes. This suggests the paper views its guidelines, not its specific implementations, as the lasting contribution. The methods serve as existence proofs that following the guidelines produces robust multimodal RoPE.

3. Technical Approach

3.1 Reader Orientation

This paper builds an analytical framework and two concrete positional encoding mechanisms (MHRoPE and MRoPE-I) for vision-language models. The core problem is that VLMs must encode positions for heterogeneous data—1D text sequences, 2D images with (height, width) coordinates, and 3D videos with (time, height, width) coordinates—but existing methods make ad hoc design choices that create predictable failure modes (modalities confusion, impaired spatial reasoning, broken knowledge transfer from pre-trained LLMs). The solution takes the form of three design guidelines distilled from systematic empirical comparison, plus two lightweight implementations that satisfy those guidelines by ensuring every positional axis (temporal, vertical, horizontal) has access to the full frequency spectrum of Rotary Positional Embedding, while preserving exact compatibility with the text-only RoPE that the underlying LLM was pre-trained with.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four conceptual components operating at different levels of abstraction:

1. Base LLM with vanilla RoPE (Qwen2.5-7B) — the pre-trained language model that processes all tokens (text and visual). It expects position IDs for its Rotary Positional Embedding, which applies rotational transformations to query and key vectors before attention computation. This component is frozen in its architecture but extended to accept multi-dimensional position tuples.

2. Vision Encoder + Connector (QwenViT from Qwen2.5VL) — converts images and video frames into visual token embeddings. The ViT is frozen during training; the connector (a learned projection) is unfrozen. These components produce the token embeddings that enter the LLM backbone, but they do not participate in positional encoding—that happens entirely inside the LLM.

3. Position Assignment Module (logical component, not a learned network) — determines what position ID tuple (t, h, w) to assign to each token in the sequence. For text tokens, this uses a 1D index (pos, 0, 0). For visual tokens, this uses the native 3D coordinates of each token within its source image/video, with an optional spatial-reset mechanism that resets (h, w) coordinates for each new visual content. This module also determines the modality interval—the gap in position IDs between different modalities.

4. Frequency Allocation Strategy (design choice, not a learned component) — determines how the LLM's d RoPE feature dimensions (organized into d/2 frequency pairs) are distributed across the three positional axes (t, h, w). This can be done by partitioning channels into contiguous blocks (MRoPE), by dedicating different attention heads to different axes (MHRoPE), or by interleaving channels in a round-robin pattern (MRoPE-I). This choice determines which frequencies each axis can use to encode positional information.

Information flows as follows: an interleaved multimodal sequence (e.g., <system prompt>, <video 1>, <text>, <image 1>, <text>, <generated text>) enters the system → the vision encoder converts visual inputs to token embeddings → the connector projects these to the LLM's input dimension → the position assignment module computes a (t, h, w) tuple for every token → the frequency allocation strategy maps these tuples to rotational frequencies → the LLM applies RoPE transformations to query/key vectors using these frequencies → standard self-attention proceeds with position-aware similarity scores.

3.3 Roadmap for the Deep Dive

• First, vanilla RoPE mechanics (Section 2.1)—the mathematical foundation that all multimodal variants build upon. Understanding how RoPE encodes relative position through rotation is prerequisite to understanding why frequency allocation matters.
• Second, position design (Section 2.2)—how position IDs (t, h, w) are assigned to tokens. This determines whether modalities get confused, whether 3D structure is preserved, and whether the LLM's attention-sink bias toward small IDs helps or hurts visual processing.
• Third, frequency allocation (Section 2.3)—how the d/2 frequency channels are distributed across positional axes. This is where MRoPE's chunking creates temporal bias and spatial asymmetry, where VideoRoPE's correction creates new spatial limitations, and where MHRoPE and MRoPE-I's full-spectrum strategies solve both problems.
• Fourth, the spatial-reset mechanism and attention sink phenomenon (Section 2.2.3)—the empirical discovery that motivates resetting spatial coordinates per visual content, and the motion disentanglement benefit this provides for video understanding.
• Fifth, text compatibility (Section 2.4)—why preserving the exact text-only RoPE formulation is critical and what happens when well-intentioned modifications (scaling rotary bases, resetting text spatial dimensions) break knowledge transfer.
• Sixth, the two proposed methods (Section 2.5)—MHRoPE's head-level partitioning and MRoPE-I's channel-level interleaving, with their trade-offs in implementation complexity, performance, and scalability.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis and method proposal paper whose core idea is that three design guidelines—positional coherence, full frequency utilization, and preservation of textual priors—are individually necessary and collectively sufficient for robust multimodal RoPE, and that two simple implementations (MHRoPE and MRoPE-I) can satisfy all three simultaneously while requiring no architectural changes to the transformer backbone.

---

Vanilla RoPE: The Mathematical Foundation

Before analyzing multimodal extensions, we must understand exactly what vanilla RoPE does and why its frequency structure matters. RoPE is not an additive position embedding; it is a rotational transformation applied to the query and key vectors before the dot-product attention computation. This rotation has a specific mathematical property that makes attention scores depend only on relative position, while the frequency allocation determines how quickly attention decays with distance.

The RoPE Attention Score Formula

Given a query vector q at position m and a key vector k at position n, the attention score S is computed as:

$S = (R_m q)^\top (R_n k) = q^\top R_m^\top R_n k = q^\top R_{n-m} k$

where R_m is a block-diagonal rotation matrix parameterized by absolute position m and a set of fixed frequencies θ_i, and R_{n-m} is the corresponding matrix for the relative position (n - m).

What it computes: the standard dot-product attention score between query and key, but with both vectors rotated by position-dependent orthogonal matrices. The crucial algebraic property is that R_m^\top R_n = R_{n-m}—the product of the transpose of the position-m rotation and the position-n rotation equals the rotation for the relative position (n-m). This means the attention score S depends only on the content of q and k and their relative distance (n-m), not on their absolute positions.

Why this form: the rotation approach has two key advantages over additive position embeddings (like the original transformer's sinusoidal encodings). First, relative position encoding generalizes better to sequence lengths unseen during training—the model learns to attend based on distance rather than memorizing absolute position patterns. Second, the rotation preserves the norm of the vectors (R is orthogonal), which means position encoding does not distort the magnitude of attention scores.

The Frequency Spectrum

The rotation matrix R_m is constructed as a block-diagonal matrix with d/2 rotation blocks, each operating on a pair of dimensions (2i, 2i+1). The rotation angle for block i is m · θ_i, where the frequencies are defined as:

$\theta_i = \text{base}^{-2i/d}, \quad i \in [0, d/2 - 1]$

where d is the embedding dimension (128 for the Qwen2.5-7B models used in this paper, meaning 64 frequency pairs), and base is a hyperparameter (set to 1,000,000 in all experiments, Section 3.1).

What it computes: a geometric progression of 64 frequencies, starting from θ_0 = 1,000,000^0 = 1 (the highest frequency, completing a full rotation at every position increment) and decaying to θ_{63} = 1,000,000^{-126/128} ≈ 1.25 × 10^{-6} (the lowest frequency, requiring ~800,000 position steps to complete a full rotation).

Why this form: the geometric spacing creates a spectrum from high-frequency channels (which encode fine-grained, short-range positional differences—the model can distinguish position m from m+1 with high precision) to low-frequency channels (which encode coarse, long-range positional structure—the model can track whether two tokens are roughly in the same region of a long document). This spectrum is what gives RoPE its long-range decay property: the upper bound on the attention score between two tokens decreases as their relative distance increases, with the decay rate governed by the interaction of all frequency channels. The paper derives this upper bound in Appendix D.2, showing it can be approximated by (1/(d/2)) · Σ_{i=1}^{d/2} |S_i|, where S_i is a partial sum of complex exponentials at frequency θ_i. The geometric spacing ensures that some frequencies maintain non-trivial attention even at very large distances (low frequencies) while others enforce sharp locality (high frequencies).

Implication for multimodal RoPE: the full frequency spectrum (all 64 pairs) is important for modeling distances at all scales. When multimodal methods partition these 64 frequency pairs across positional axes (giving, say, 21 pairs to temporal, 21 to height, 22 to width), each axis receives only a truncated subset of the spectrum—it loses either the high frequencies (impairing fine-grained distinction) or the low frequencies (impairing long-range attention). This is the central insight that motivates MHRoPE and MRoPE-I's full-spectrum strategies.

---

Position Design: Assigning Coordinates to Tokens

Position design governs the absolute position identifiers m assigned to each token. In vanilla RoPE, m is a scalar—the token's index in the 1D sequence. In multimodal RoPE, m becomes a tuple (m_t, m_h, m_w) for temporal, vertical, and horizontal coordinates. The challenge is determining what numeric values these coordinates should take, especially at the boundaries between modalities (text→image, image→text, video frame→video frame) and across interleaved sequences containing multiple images and text segments.

1D Sequential Design (Vanilla RoPE, V2PE)

The simplest approach (Figure 1a): treat the entire multimodal sequence as a flat 1D list. Position indices increment sequentially: m_i = m_{i-1} + s_mod, where s_mod is a step size. Vanilla RoPE uses s = 1 for all tokens. V2PE introduces variable step sizes for visual tokens, setting s_visual ∈ {1, 1/2, ..., 1/256}, which slows the growth of position IDs for visual content.

The paper identifies two problems with this approach (Section 2.2.1). First, it discards the native 3D structure of visual content—a token at pixel (10, 20) in frame 3 has spatial relationships to its neighbors that a flat index cannot express. The empirical consequence is degraded grounding performance (Table 4: vanilla RoPE grounding score 73.48 vs. 74.40 when 3D structure is added). Second, position IDs grow linearly with sequence length and can become very large in multimodal contexts (a single high-resolution image can contribute thousands of tokens, followed by text, followed by more images). Figure 5 shows that vanilla RoPE suffers a sharp performance drop when extrapolating from the 32K training context to 128K/256K, which the paper attributes to "excessively fast-growing position IDs" degrading extrapolation capability.

V2PE's dynamic position scaling partially addresses the growth problem—by using small step sizes for visual tokens, the position IDs advance more slowly through long visual content—but does not recover the lost 3D structure.

MRoPE's Multi-Dimensional Design

MRoPE (Wang et al., 2024a) extends the position identifier to a 3D tuple (m_t, m_h, m_w). The key design question is: how do you assign these coordinates so that tokens from different modalities don't collide (have the same position tuple), tokens within the same image have meaningful spatial relationships, and the numeric values stay bounded?

MRoPE's solution (Figure 1b) treats each visual content (an image or a set of video frames) as a 3D block. Within a block, each visual token receives coordinates based on its native position: the temporal index (frame number), the vertical coordinate (normalized to the image grid, so patches range from 0 to H-1 where H is the number of vertical patches), and the horizontal coordinate (0 to W-1). After a visual block ends, the next text token's temporal position "jumps" past the maximum coordinate used in the block, using the update rule:

$m_{\text{next}}^t = \max(m_{\text{prev}}^t, m_{\text{prev}}^h, m_{\text{prev}}^w) + 1$

where m_prev refers to the maximum coordinates reached in the preceding visual block.

What it computes: a scheme where the temporal axis acts as a global counter that always advances. After processing an image of size H × W patches, the next temporal position is max(t_max, H-1, W-1) + 1. For text tokens, spatial coordinates are typically set to 0, so a text token after an image would have position (max(H,W), 0, 0). For the next image, temporal coordinates would continue from this new baseline.

Why this form: the max-plus-1 rule guarantees no positional overlap between modalities—the temporal coordinate of any subsequent token is strictly larger than any coordinate used in the preceding visual content. This prevents the model from confusing "token at the bottom-right of image 1" with "token at the start of the text response," which would happen if coordinates simply continued sequentially without the jump.

The Visual Attention Sink Problem

The paper discovers a previously unreported phenomenon in MRoPE (Section 2.2.3, Figure 2): attention concentrates heavily on the top-left corner (t, h, w) = (0, 0, 0) of each image or video frame. When the ChartQA dataset is used with duplicated images and the prompt "Describe the two images in detail," the average attention scores (computed across all attention heads and samples) show a strong bias toward small position IDs in the visual content.

This is analogous to the well-known attention sink in LLMs, where the initial tokens of a sequence (typically position 0 or 1) absorb disproportionate attention regardless of their semantic content. In MRoPE, the visual tokens at (0, 0, 0)—the top-left corner of the first frame of a video, or the first image in a sequence—become an unintended attention sink. The paper attributes this to the LLM's pre-trained bias toward small position IDs: the base LLM was trained on 1D text sequences where position 0 (the BOS token or first word) naturally serves as an attention sink, and when visual tokens are placed at coordinates near 0, they inherit this bias.

This insight directly motivates spatial-reset (Section 2.2.3): resetting the spatial coordinates (h, w) to start from 0 for each new visual content. Rather than treating this attention sink as a bug, spatial-reset deliberately positions it at the start of each image/frame to align with the LLM's existing bias, "accelerating visual adaptation" (Section 2.2.3). The empirical validation in Section 3.4.1 (Table 4) confirms this: adding spatial-reset to a 3D-position design improves grounding from 74.40 to 75.85, image benchmarks from 65.87 to 66.65, and video from 51.29 to 52.36.

Spatial-Reset and Motion Disentanglement for Video

The paper identifies a second benefit of spatial-reset specific to video understanding: it disentangles the representation of motion (Section 2.2.3). Consider an object token at spatial coordinates (h_1, w_1) at time t_1 and the same object at (h_2, w_2) at time t_2.

Under standard MRoPE (without spatial-reset): the spatial coordinates are not independently specified—they are implicitly coupled to the temporal coordinate. The absolute positions are m_1 = (t_1, t_1 + h_1, t_1 + w_1) and m_2 = (t_2, t_2 + h_2, t_2 + w_2). The resulting relative position vector is:

$m_{\text{rel}} = (t_2 - t_1, (t_2 - t_1) + (h_2 - h_1), (t_2 - t_1) + (w_2 - w_1))$

where the spatial relative displacements (h_2 - h_1) and (w_2 - w_1) are entangled with the temporal offset (t_2 - t_1).

Under spatial-reset: the positions are defined as m_1 = (t_1, h_1, w_1) and m_2 = (t_2, h_2, w_2), yielding a cleanly separated relative vector:

$m'_{\text{rel}} = (t_2 - t_1, h_2 - h_1, w_2 - w_1)$

Why this matters: the disentangled version gives the model a pure motion signal—the temporal change is isolated in the first component, and the spatial displacement appears independently in the second and third components. The paper argues this "more intuitive" representation "provides a cleaner inductive bias for the model to learn from" (Section 2.2.3). Rather than having to decompose a mixture of temporal and spatial changes, the attention mechanism can directly attend based on pure temporal similarity (is this the same time point?), pure vertical similarity (is this the same row?), or any combination thereof.

Failure Modes of Alternative Position Designs

The ablation study in Table 4 systematically tests what happens when each design principle is violated, using MRoPE-I's interleaved frequency allocation as the fixed backbone and varying only the position assignment strategy.

Diagonal layout (VideoRoPE/HoPE): Implementing the diagonal layout—where visual frames are shifted along both vertical and horizontal axes (Figure 1c)—causes severe degradation on document-centric benchmarks. DocVQA drops from 83.72 to 60.13, InfoVQA from 58.24 to 37.42. The paper identifies a specific failure mode: "repetitive, nonsensical text generation (e.g., '1111...'), which occurs even when the layout is applied only at inference time" (Section 3.4.1). The root cause is position ID overlap: for high-resolution document images, the shifted spatial coordinates extend into the index range later assigned to generated text tokens, causing the model to misinterpret its own output as visual input—the "modalities confusion in generation" described in Section 2.2.2.

Enlarged modality interval (RoPE-Tie-like): Artificially enlarging the gap between modality position ranges to match vanilla RoPE's sequential spacing (so that text tokens always have much larger indices than visual tokens) also degrades document performance: DocVQA drops from 83.72 to 70.43, ChartQA from 62.12 to 51.28 (Table 4). However, the failure mode is different: the model "generated fluent but contextually irrelevant text, effectively ignoring the visual input" (Section 3.4.1). This suggests that while some modality separation is necessary (to prevent confusion), too large a gap makes cross-modal attention difficult—the model treats visual tokens as being "too far away" to attend to.

Text spatial-reset (IL-RoPE/Omni-RoPE): Setting spatial coordinates to zero for text tokens as well as resetting for visual tokens (Figure 1e) causes a uniform performance drop across all categories: image drops from 66.65 to 58.27, grounding from 75.85 to 68.20, video from 52.36 to 50.71. This breaks text RoPE compatibility by changing the position encoding that the LLM was pre-trained with, confirming that "preserving RoPE alignment for text is critical for successfully adapting LLMs into VLMs" (Section 3.4.1).

Scaling rotary base for spatial axes: Motivated by the smaller coordinate range of spatial dimensions (images typically have H, W ≤ 70 patches, while temporal axes can span thousands of frames), the authors tested reducing the rotary base for spatial axes from 1,000,000 to 10,000 to better match the coordinate scale. This produced a clear drop in image performance (66.65 → 60.15). The paper concludes that "even well-intentioned deviations from the base LLM's RoPE formulation can break compatibility and severely impair knowledge transfer" (Section 3.4.1).

---

Frequency Allocation: Distributing Rotational Frequencies Across Axes

Frequency allocation governs how the d/2 rotational frequency pairs (64 pairs for the 128-dimensional embeddings used in Qwen2.5-7B) are assigned to encode the three positional axes (t, h, w). This determines which frequencies—high (good for fine-grained, short-range distinctions), medium, or low (good for coarse, long-range structure)—each axis can use to represent positional information. The paper argues this is the dimension where most prior methods make their critical error.

MRoPE's Chunked Allocation and Its Consequences

The standard MRoPE partitions the d feature dimensions into three contiguous blocks, dedicating one block to each axis (Section 2.3.2, Figure 3). Since rotational frequencies decay monotonically with channel index i (recall θ_i = base^{-2i/d}), this means:

• Temporal axis (t): assigned the lowest channel indices (small i), corresponding to the highest frequencies. These frequencies rotate rapidly with position changes, causing attention scores to decay quickly over temporal distance. The consequence: the model loses the ability to attend across distant time steps—long-range temporal dependencies are suppressed.

• Height axis (h): assigned the middle channel indices, corresponding to medium frequencies. These provide moderate-range spatial attention.

• Width axis (w): assigned the highest channel indices (large i), corresponding to the lowest frequencies. These rotate slowly, allowing long-range horizontal attention.

The asymmetry problem: because h and w receive different, non-overlapping frequency ranges, they exhibit different long-range decay properties (visualized in Figure 4a). A vertical distance of Δh patches and a horizontal distance of Δw patches are encoded with fundamentally different frequency characteristics—the attention decay with vertical distance follows a different curve than the decay with horizontal distance. The paper argues this "asymmetry can impair the model's ability to learn consistent spatial relationships" (Section 2.3.2), because the model cannot treat (Δh, Δw) as a rotation-invariant 2D distance vector.

The video problem: the temporal axis being forced entirely into high frequencies means attention over time decays extremely rapidly. For long videos (hundreds or thousands of frames), the model struggles to connect information from early frames to later ones—the attention scores become vanishingly small. The paper quantifies this through benchmarking: MRoPE underperforms on long-video understanding tasks (Table 2: MLVU 63.26 vs. 66.05 for VideoRoPE, LVBench 39.22 vs. 40.99 for HoPE, though both VideoRoPE and HoPE have their own problems on other tasks).

VideoRoPE and HoPE's Correction (and Its New Problem)

VideoRoPE and HoPE attempt to fix MRoPE's temporal bias by inverting the allocation: they move the temporal axis to the low-frequency channels (large i), which preserve long-range temporal attention by providing slowly-decaying attention scores over time. The spatial axes are pushed into the remaining higher-frequency channels (Section 2.3.2).

The paper identifies the unaddressed trade-off: this correction "force(s) the spatial dimensions into a restricted, and often exclusively high-frequency, band" (Section 2.3.2). The model loses low-frequency spatial channels, which are necessary for:

1. Global spatial layouts: understanding the overall structure of a scene or document—which regions are above/below/left/right of others at a coarse level.
2. Multi-scale spatial reasoning: simultaneously capturing fine details (requiring high frequencies for sharp spatial localization) and broad contextual relationships (requiring low frequencies for smooth spatial attention).

The empirical impact is visible on tasks requiring fine-grained spatial understanding. OCRBench (which tests character-level recognition accuracy) drops from 72.70 (MRoPE) to 66.20 (VideoRoPE) and 66.60 (HoPE) in Table 2. The paper attributes this to the spatial axes losing access to the full frequency spectrum needed for multi-scale processing.

IL-RoPE's Interleaving Attempt

IL-RoPE employs a form of channel interleaving (mixing t, h, w channels rather than keeping them in contiguous blocks), but reserves the lowest-frequency channels for the temporal dimension (Section 2.3.2, Figure 3). This is a partial fix: it gives spatial axes access to some mid and high frequencies, but still denies them the low frequencies needed for global spatial context. The ablation in Table 5 (labeled "IL-RoPE-like") confirms this is suboptimal: overall score 63.07 versus 64.95 for full interleaving.

The Frequency Resolution Coarsening Problem

Any method that partitions the 64 frequency pairs across three axes (rather than giving each axis access to all 64) inherently reduces frequency granularity. With, say, 21 channels per axis, the geometric progression base^{-2i/d} now has only 21 distinct frequency values instead of 64—the steps between consecutive frequencies are larger, meaning the model has less precise control over the attention decay profile for that axis. The paper notes this "performance implications of this reduced granularity is under-exploration" (Section 2.3.2), and Table 5 empirically demonstrates that more uniform allocation (which gives each axis access to more distinct frequencies) consistently improves performance.

---

The Two Proposed Frequency Allocation Strategies

The paper proposes two methods that give each positional axis access to the full frequency spectrum, thereby avoiding all the problems described above. Both build on MRoPE's position design enhanced with spatial-reset, and both maintain strict compatibility with text-only RoPE.

Multi-Head RoPE (MHRoPE)

MHRoPE operates at the attention head level rather than the channel level (Section 2.3.3, Figure 3). The key idea: different attention heads are dedicated to different positional axes, and each head uses the full 64-frequency-pair spectrum to encode its assigned axis.

Specifically, the N attention heads are partitioned into (at least) three groups. Heads in group 1 encode only the temporal axis—their queries and keys are rotated using only the temporal coordinate m_t, with all 64 frequency pairs applied to encode temporal position. Heads in group 2 encode only the vertical axis using m_h with all 64 frequency pairs. Heads in group 3 encode only the horizontal axis using m_w with all 64 frequency pairs. For Group Query Attention (GQA), the partitioning is done on the key-value heads and repeated on the corresponding query heads (Section 2.3.3 footnote).

What happens during attention: in a temporal head, the attention score between two tokens depends on their temporal distance Δt = m_t^query - m_t^key, with the full spectral resolution enabling both fine-grained temporal discrimination (high frequencies) and long-range temporal attention (low frequencies). Spatial distances have zero effect on attention scores in this head—two tokens at the same time but different spatial positions are treated as positionally identical. Conversely, in a spatial head, temporal distance is ignored and only the spatial coordinates matter.

Why this works: by partitioning at the head level rather than the channel level, MHRoPE preserves the full 64-frequency-pair resolution for every axis. There is no loss of granularity. The information from different positional axes is integrated not within individual attention heads but across heads in the multi-head attention output, which is naturally designed to combine information from different representational subspaces. The paper draws inspiration from recent work on "partial RoPE" (Barbero et al., 2025) which demonstrated channel-level redundancy in RoPE—the insight that not all channels need position encoding—and extends this to the head level.

Scalability advantage: MHRoPE's design is "more scalable" (Section 2.3.3) as the number of positional axes grows. If a future VLM needs to encode, say, 5 positional axes (time, height, width, depth, and an additional modality-specific dimension), partitioning 64 channels across 5 axes would leave each axis with only ~13 frequency pairs—severely degraded resolution. MHRoPE can simply dedicate additional attention heads to new axes without sacrificing frequency resolution, since each head always gets the full 64-pair spectrum. This is a forward-looking design choice that anticipates more complex multimodal models.

Limitation (Appendix D.1): MHRoPE achieves "minor performance deficit" compared to MRoPE-I, which the authors attribute to "head-level information partitioning, which prevents the integration of different positional axes within the self-attention mechanism." In other words, because individual attention heads see only one positional axis, no single head can compute attention based on the combined spatio-temporal relationship between two tokens—the integration happens only after multi-head concatenation. Additionally, MHRoPE introduces complexity with distributed training paradigms like tensor parallelism (Appendix D.1), because the head partitioning must be respected when splitting heads across devices.

MRoPE-Interleave (MRoPE-I)

MRoPE-I operates at the channel level but distributes channels in a fine-grained, round-robin pattern rather than in contiguous blocks (Section 2.3.3, Figure 3). The 64 frequency pairs (corresponding to 128 feature dimensions) are assigned to the three axes in a repeating cycle: channel 0 → temporal, channel 1 → height, channel 2 → width, channel 3 → temporal, channel 4 → height, channel 5 → width, and so on.

What this achieves: each positional axis receives channels spanning the entire frequency spectrum. Because channel index i maps to frequency θ_i = base^{-2i/d}, the interleaving ensures that every axis gets some high-frequency channels (small i), some medium-frequency channels, and some low-frequency channels (large i). The temporal axis is no longer restricted to high frequencies—it gets low-frequency channels (e.g., channel 60, 63) that preserve long-range temporal attention. The spatial axes are no longer restricted to partial spectra—they get both high frequencies for fine-grained localization and low frequencies for global spatial context.

The uniform decay property: Figure 4b visualizes the long-range decay as a function of relative distance for all three axes under MRoPE-I. Unlike MRoPE's asymmetric decay curves (Figure 4a), where the temporal decay is rapid, the vertical decay is moderate, and the horizontal decay is slow, MRoPE-I produces a unified decay profile—all three axes exhibit the same attention decay with distance. This means the model can learn consistent spatio-temporal relationships: a temporal offset of Δt and a spatial offset of Δs are encoded with the same frequency characteristics, making the attention computation invariant to whether the distance is in time or space.

Extrapolation compatibility (Appendix D.3): the uniform frequency distribution makes MRoPE-I "compatible with extrapolation algorithms like NTK-aware and YaRN, which function by rescaling the frequency spectrum." These algorithms work by identifying a cutoff frequency and rescaling all frequencies below it to extend the effective context window. With MRoPE's chunked allocation, each axis has a different frequency range, making it unclear where to apply the cutoff—should it be at the boundary between temporal and height channels? At the boundary between height and width? MRoPE-I's interleaved design means all axes share the same full spectrum, so a single cutoff can be applied symmetrically. Table 6 validates this: MRoPE-I + YaRN achieves 43.6 on LVBench and 64.1 on MLVU at 256K context, versus 41.2 and 63.3 for MRoPE + YaRN.

Channel allocation ratio ablation: Table 8 investigates different ratios of channels allocated to t:h:w under the interleaved design. The balanced allocation (t:h:w = 24:20:20) achieves the best overall performance (64.95). Increasing temporal allocation to 32:16:16 reduces grounding ability (74.65 vs. 75.85 for 24:20:20) because fewer spatial channels means reduced high-frequency capacity for fine-grained localization. Eliminating temporal channels entirely (0:32:32, effectively treating everything as a spatial-only problem) slightly improves image metrics (66.42 vs. 66.65) but degrades video (51.01 vs. 52.36), as expected since temporal information is lost. The 24:20:20 ratio provides the best balance between spatial precision and temporal continuity.

Implementation simplicity: Appendix D.1 notes that MRoPE-I is simpler to implement than MHRoPE because it avoids the engineering complexities of head-level partitioning in distributed training. The channel assignment is a simple remapping of frequency indices to axes that can be implemented as a lookup table or a modulo operation, requiring no changes to the attention mechanism itself.

Why the paper currently recommends MRoPE-I over MHRoPE: two reasons (Appendix D.1). First, MRoPE-I has a "consistent (albeit slight) performance advantage"—visible in Table 2 where MRoPE-I achieves 63.79 image average vs. 62.92 for MHRoPE, and 75.85 grounding vs. 74.92. Second, MRoPE-I is simpler to implement, particularly with tensor parallelism where head partitioning adds complexity. However, MHRoPE is recognized as "potentially more scalable" for future models with many positional axes.

---

Compatibility with Text-Only RoPE

The paper identifies text compatibility as a "critical" design axis that is often violated by well-intentioned modifications (Section 2.4). The principle is simple: the positional encoding applied to text tokens in the VLM should be identical to the encoding the LLM was pre-trained with. Any deviation—changing the frequency allocation, modifying position IDs, or scaling rotary bases—breaks the knowledge transfer from pre-training and causes performance degradation.

Why this matters: most VLMs are initialized from pre-trained LLMs (here, Qwen2.5-7B). The LLM has learned to associate specific position-dependent attention patterns with linguistic structure—for example, attending to nearby tokens for syntax, distant tokens for coreference resolution, and the sequence start for global context. Changing the positional encoding for text tokens forces the model to relearn these associations from scratch during VLM fine-tuning, with limited multimodal data (only ~2M samples in this paper's training set, Section 3.1) compared to the trillions of tokens seen during pre-training.

Empirical validation: the paper tests two deviations from text compatibility (Table 4). First, scaling the rotary base for spatial dimensions from 1,000,000 to 10,000—motivated by the fact that spatial coordinate ranges (~0-70 patches) are much smaller than temporal ranges (up to thousands of frames for long videos)—drops image performance from 66.65 to 60.15. Second, applying text spatial-reset (setting text tokens' (h, w) to (0, 0), as in IL-RoPE and Omni-RoPE) drops image performance from 66.65 to 58.27. Both modifications seem reasonable in isolation but catastrophically damage the transfer of pre-trained knowledge.

How MHRoPE and MRoPE-I maintain compatibility: both methods use vanilla RoPE for text tokens—a 1D sequential position index with the full 64-frequency-pair spectrum and the base LLM's rotary base (1,000,000). The multi-dimensional extensions (3D coordinates, spatial-reset, frequency interleaving/head partitioning) apply only to visual tokens. When the sequence contains only text (which can happen in multi-turn dialogues where the user asks a follow-up question without an image), the system degenerates exactly to the original LLM's RoPE—no modifications, no performance loss.

This design choice is reflected in Figure 1f: text tokens sit on the temporal axis with their 1D position IDs (and h=0, w=0), while visual tokens receive full 3D coordinates with spatial-reset. The compatibility is not approximate—it is exact. The only change for text tokens is that their (h, w) coordinates are zero rather than undefined, which does not affect the rotation applied because MRoPE-I's interleaving assigns h and w channels that are simply not used (or are used with zero rotation when m_h = 0, m_w = 0, which is the identity transformation).

---

Summary of Design Choices and Their Justifications

• Spatial-reset (position design): resets (h, w) to start from 0 for each visual content, rather than continuing spatial coordinates across images. Justification: aligns visual tokens with the LLM's attention-sink bias toward small position IDs (Figure 2), accelerating visual adaptation; plus disentangles motion representation for video (Equations 3–4).

• Full frequency spectrum per axis (frequency allocation): both MHRoPE (head-level) and MRoPE-I (channel-level) give each axis access to all 64 frequency pairs, from high to low. Justification: avoids MRoPE's temporal bias (temporal stuck in high frequencies → rapid decay → poor long-video modeling), avoids VideoRoPE/HoPE's spatial restriction (spatial stuck in high frequencies → impaired multi-scale spatial reasoning), and avoids frequency resolution coarsening (partitioning reduces granularity of the geometric frequency progression).

• Uniform decay across axes (frequency allocation): MRoPE-I's interleaving produces identical long-range decay profiles for t, h, and w (Figure 4b). Justification: enables consistent spatial relationship learning—the model can treat (Δt, Δh, Δw) as a rotation-invariant vector rather than having different decay curves for each dimension.

• Text RoPE identity preservation: text tokens use exactly the original LLM's RoPE with no modifications. Justification: preserves pre-trained knowledge transfer; any deviation (scaling rotary base, resetting text coordinates) causes large performance drops (Table 4).

• Balanced 24:20:20 channel ratio (MRoPE-I only): slightly more channels to temporal than to each spatial axis. Justification: Table 8 shows this balance achieves optimal overall performance (64.95), with more temporal channels improving video but hurting spatial grounding if overdone, and fewer temporal channels hurting video without proportionally benefiting grounding.

• Stride δ = 1 for temporal spacing: the temporal position advances by 1 between consecutive frames (Section 3.1, Table 9). Justification: smaller strides (0.5) and larger strides (2) both degrade video performance; dynamic strides (V2PE-style) do not provide significant benefit.

4. Key Insights and Innovations

Innovation 1: Decomposing Multimodal RoPE into Three Independent, Diagnosable Design Axes

Prior to this paper, the landscape of multimodal positional encoding was a collection of point solutions—MRoPE, VideoRoPE, HoPE, CircleRoPE, IL-RoPE, Omni-RoPE—each proposed in isolation, optimized for a specific modality or task, and evaluated under different experimental conditions. There was no shared vocabulary for discussing why one method differed from another, nor any framework for predicting when a particular design would fail. The paper's first major intellectual contribution is the recognition that all these methods can be decomposed into three orthogonal design choices: position design (how coordinate tuples are assigned to tokens and how they advance across modalities), frequency allocation (how the RoPE frequency spectrum is distributed across positional axes), and compatibility with text-only RoPE (whether text tokens retain the LLM's original positional encoding). This decomposition is not merely taxonomic—it is diagnostic. Once the axes are separated, the failure modes of prior methods become predictable rather than mysterious. VideoRoPE's diagonal layout is a position design flaw that creates positional overlap between visual and generated text tokens, independently of its frequency allocation strategy (which happens to be well-motivated for video). MRoPE's chunked frequency allocation creates a temporal bias that impairs long-video understanding, independently of its position design (which is sound in most respects). CircleRoPE's collapsed temporal axis is a position design flaw that destroys video understanding, regardless of its novel circular layout for images. This decomposition transforms the problem from "which of seven bespoke methods works best?" to "which combination of three independent design choices is optimal?"—a reduction that makes the design space tractable. The empirical validation that these axes are genuinely independent comes from the ablation studies (Tables 4 and 5), where position design and frequency allocation are varied separately while holding the other fixed, and the effects are largely additive. This is a conceptual advance that the paper does not claim as its core contribution (it presents itself primarily as a systematic comparison and two new methods), but it is arguably the most lasting one: future work designing multimodal RoPE for new modalities or architectures can reason about these three axes independently rather than starting from scratch.

Innovation 2: Full Frequency Utilization as the Key Driver of Robust Performance

The paper's central empirical finding is that every positional axis—temporal, vertical, horizontal—needs access to the complete RoPE frequency spectrum, from the highest frequencies (for fine-grained, short-range discrimination) to the lowest (for coarse, long-range structure). This is counterintuitive in two ways. First, the default assumption in prior work (MRoPE, VideoRoPE, HoPE, IL-RoPE) was that frequency allocation was a resource to be divided across axes—since there are only d/2 frequency pairs, each axis should get a subset. The paper shows this intuition is wrong: partitioning the spectrum creates worse performance on all tasks than giving each axis the full spectrum, even though the latter means each axis sees the same frequencies (potentially creating ambiguity). The ambiguity turns out not to matter—the axes are distinguished by their position ID values, not their frequency content. Second, the paper demonstrates that the specific pattern of spectrum partitioning creates predictable, asymmetric failure modes. MRoPE's chunking forces the temporal axis into high frequencies only, which explains why MRoPE underperforms on long-video tasks (Table 2: MLVU 63.26, LVBench 39.22) despite being strong on images. VideoRoPE and HoPE's inversion (temporal → low frequencies, spatial → high frequencies) fixes the video problem but breaks fine-grained spatial reasoning (OCRBench drops from 72.70 to 66.20/66.60), because spatial axes lose the high frequencies needed for character-level discrimination. IL-RoPE's partial interleaving reserves low frequencies for temporal, again starving spatial axes of global context. The paper's demonstration that both MHRoPE (head-level full spectrum) and MRoPE-I (channel-level full spectrum via interleaving) outperform all partitioned approaches (Tables 2, 5) is the empirical proof that full-spectrum allocation is not merely beneficial but necessary for a general-purpose VLM that must handle images, videos, and grounding simultaneously. This is a fundamental shift from the prior paradigm of "divide the spectrum to specialize axes" to "give every axis everything and let the multi-head attention or the position ID values handle disambiguation."

Innovation 3: Positional Coherence as a Unified Explanation for Catastrophic Failure Modes

The paper identifies and names a class of failures—what it terms violations of "positional coherence"—that manifest as catastrophic, non-graceful degradation rather than mild performance drops. The most striking example is VideoRoPE/HoPE's diagonal layout, which causes DocVQA to collapse from ~82% to ~60% (Table 2) and InfoVQA from ~58% to ~37%. The paper does more than report these numbers; it provides a mechanistic explanation: the diagonal layout shifts spatial coordinates for each video frame along both axes (Figure 1c), and for high-resolution documents, these shifted coordinates extend into the index range later assigned to generated text tokens. This creates positional overlap—the model cannot distinguish its own generated output (text tokens at positions (t_large, 0, 0)) from visual tokens that happen to land at similar positions. The result is "modalities confusion in generation," where the model produces nonsensical repetitions like "1111..."—a failure mode that the paper demonstrates occurs even when the diagonal layout is applied only at inference time (Section 3.4.1), confirming it is a positional encoding problem rather than a training issue. This is significant because it identifies a hard constraint on position design: positional ambiguity between modalities is not a soft trade-off (slightly worse performance) but a cliff edge (complete collapse). Similarly, the paper shows that enlarging the modality interval to match vanilla RoPE's spacing (a RoPE-Tie-like strategy) causes a different catastrophic failure: the model generates "fluent but contextually irrelevant text, effectively ignoring the visual input" (Section 3.4.1). The mechanism is different—cross-modal attention becomes too difficult because visual tokens are "too far away" in position space—but the consequence is equally severe. These findings reveal that position design for multimodal RoPE has a narrow viable region: too little separation → modalities confusion (diagonal layout), too much separation → visual ignoring (enlarged interval). The "positional coherence" guideline is not a vague aesthetic preference but a precise empirical constraint that prior methods violated without understanding the consequences.

Innovation 4: Spatial-Reset as a Mechanism that Repurposes an LLM Bias for Visual Adaptation

The paper's discovery of the visual attention sink (Figure 2, Section 2.2.3) and the spatial-reset mechanism it motivates represents a different kind of innovation from the other contributions—not a design principle distilled from systematic comparison, but an empirical observation that leads to a counterintuitive optimization. The finding is that MRoPE exhibits a concentrated attention pattern where the top-left corner (t=0, h=0, w=0) of visual content absorbs disproportionate attention, analogous to the attention sink phenomenon in LLMs where initial tokens serve as attention anchors. The paper's insight is not to fight this bias but to repurpose it: by resetting spatial coordinates for each new visual content, the attention sink is repositioned at the start of every image or video frame, aligning with the LLM's pre-existing bias toward small position IDs. This is clever because it turns a potential limitation (the LLM has an ingrained preference for attending to small position values) into a feature (that preference can be targeted at the beginning of each visual input, giving the model a consistent "entry point" for visual processing). The ablation in Table 4 confirms this is not placebo: adding spatial-reset to an already-3D position design yields gains across all categories (grounding +1.45, image +0.78, video +1.07). The paper also identifies a secondary benefit unique to video: spatial-reset disentangles the representation of motion by ensuring that relative position vectors decompose cleanly into (Δt, Δh, Δw) rather than an entangled mixture (Δt, Δt + Δh, Δt + Δw) (Equations 3–4). This is a concrete example of how a seemingly small coordinate bookkeeping change can alter the inductive bias that the model learns from. The significance of spatial-reset extends beyond the specific performance gains—it demonstrates that multimodal position design should account for the pre-existing biases of the language model backbone, not just the geometry of the visual content. Most prior work treated position design as a purely geometric problem (how to assign coordinates that preserve spatial structure), but the attention sink phenomenon shows that the LLM's training history imposes constraints and opportunities that geometric reasoning alone would miss.

Innovation 5: Text RoPE Compatibility as a Hard Constraint, Not a Tunable Hyperparameter

The paper establishes that preserving the exact text-only RoPE formulation from the pre-trained LLM is not a matter of degree—a slight modification might be tolerable—but a binary constraint: any deviation causes substantial performance degradation. When the authors scaled the rotary base for spatial dimensions from 1,000,000 to 10,000 (a "well-intentioned" change motivated by the smaller coordinate range of spatial axes), image performance dropped from 66.65 to 60.15 (Table 4), a 6.5-point absolute decline. When they applied text spatial-reset (setting text spatial coordinates to zero, as IL-RoPE and Omni-RoPE do), image performance dropped even further, from 66.65 to 58.27. These are not marginal degradations—they wipe out most of the gains that any frequency allocation or position design improvement provides. The underlying mechanism is straightforward but underappreciated in the VLM literature: the LLM backbone has been pre-trained on trillions of tokens with a specific positional encoding, and it has learned to associate linguistic function (syntax, coreference, discourse structure) with specific position-dependent attention patterns. Changing the text RoPE breaks these associations, and the ~2M multimodal SFT samples used in this paper's training (Section 3.1) are insufficient to relearn them from scratch. This finding has a practical implication that extends beyond this paper's specific methods: VLMs should be designed so that when the input contains only text (e.g., follow-up questions in multi-turn dialogue), the positional encoding degenerates exactly to the base LLM's RoPE. Both MHRoPE and MRoPE-I achieve this trivially by leaving text tokens untouched, but the paper's contribution is demonstrating that this is not an optional design nicety—it is essential for performance. The IL-RoPE and Omni-RoPE counterexamples (which modify text position encoding to support image editing functionality) show that even application-motivated modifications to text RoPE come at a steep cost that is rarely acknowledged in those papers' evaluations. This establishes a clear principle for future work: multimodal RoPE extensions should be additive (adding new capabilities for visual tokens) rather than modifying (changing how text tokens are encoded), and any proposed method that does modify text encoding must demonstrate that its benefits on the target task outweigh the certain degradation on general multimodal understanding.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use approximately 2M high-quality supervised fine-tuning (SFT) samples covering image captioning, OCR, visual reasoning, visual grounding, document comprehension, long video understanding, and multi-turn dialogue (Section 3.1). No specific dataset name is given for the SFT mixture; the paper states "all experiments share identical training data." For evaluation, the paper adopts more than 20 benchmarks spanning images, videos, and grounding. Image benchmarks include MMMU (Yue et al., 2024), MMBench (Liu et al., 2024a), MMStar (Chen et al., 2024), OCRBench (Liu et al., 2024b), AI2D (Kembhavi et al., 2016), RealWorldQA (X.AI., 2024), DocVQA (Mathew et al., 2021), TextVQA (Singh et al., 2019), InfoVQA (Mathew et al., 2022), ChartQA (Masry et al., 2022), and BLINK (Fu et al., 2024) for multi-image reasoning. Video benchmarks include MVBench (Li et al., 2024), STAR (Wu et al., 2021), VideoMME (Fu et al., 2025), LVBench (Wang et al., 2024b), MLVU (Zhou et al., 2024b), and Charades-STA (Zhou et al., 2024a). Grounding is evaluated on the RefCOCO series (Kazemzadeh et al., 2014): RefCOCO val/testA/testB, RefCOCO+ val/testA/testB, and RefCOCOg val/test. No explicit test-set sizes are provided for individual benchmarks; the paper follows "the official evaluation protocol of Qwen2VL" (Section 3.1).

• Base model(s). All main experiments use QwenViT (the vision encoder from Qwen2.5VL) paired with the Qwen2.5-7B LLM as the VLM backbone (Section 3.1). The ViT is frozen during training to fix visual representations; the connector and LLM backbone are unfrozen. This choice of backbone is motivated by the paper's goal of isolating the effects of RoPE modifications: "This strategy is designed to isolate the effects of our proposed RoPE modifications while adhering to the standard VLM adaptation paradigm of building upon a pre-trained LLM" (Section 3.1). The paper notes this initialization aligns with VideoRoPE and HoPE and differs from alternative architectural explorations such as Apollo or Cambrian-1. For cross-architecture generalization, the paper also evaluates on Qwen3-VL-4B-Instruct and Qwen3-VL-8B-Instruct (Section 3.3, Tables 10–11), which introduce architectural differences including removal of window attention, addition of a DeepStack architecture between connector and backbone, application of QK-Norm, and (for the 4B variant) weight tying between embedding and LM head.

• Metrics. The primary metric is benchmark-specific accuracy or score reported according to each benchmark's official evaluation protocol (Section 3.1). The paper reports raw scores per benchmark (e.g., MMMU accuracy percentage, RefCOCOval AP) and then computes three modality-level averages: Image (average across the 11 image benchmarks), Video (average across the 6 video benchmarks), and Grounding (average across the 8 RefCOCO subsets). An "Overall" average across all benchmarks is not explicitly computed; instead, the paper reports the three modality averages separately. For the ablation study on visual attention (Table 7), the metric is the average attention score on visual tokens across layers 4, 12, 20, and 28, averaged over all attention heads and samples from the DocVQA test set. No confidence intervals, standard deviations, or statistical significance tests are reported anywhere in the paper.

• Baselines. The paper compares seven existing methods plus its two proposed variants (Table 1, evaluated in Tables 2, 3, 10, 11):

  • Vanilla RoPE (Su et al., 2024): 1D sequential position IDs, all modalities treated uniformly with step size 1.
  • MRoPE (Wang et al., 2024a): 3D position tuples with chunked frequency allocation, as implemented in Qwen2-VL.
  • VideoRoPE (Wei et al., 2025): diagonal layout for position design, temporal axis moved to low-frequency channels.
  • HoPE (Li et al., 2025): similar diagonal layout and frequency allocation as VideoRoPE.
  • CircleRoPE (Wang et al., 2025): circular spatial layout orthogonal to text axis, no temporal dimension.
  • V2PE (Ge et al., 2024): 1D sequential with dynamic position scaling for visual tokens (referenced in analysis but not directly in the main comparison tables; appears in Figure 5 for extrapolation).
  • IL-RoPE (Liao et al., 2025) and Omni-RoPE (Wu et al., 2025): mentioned in Table 1 and Figure 3 frequency allocation comparison but not evaluated as full baselines in the main results tables; their design choices (text spatial-reset, partial interleaving) are ablated in Tables 4 and 5.

  The proposed methods are MHRoPE (Multi-Head RoPE) and MRoPE-I (MRoPE-Interleave). All methods are implemented under identical conditions: same training data, model architecture, hyperparameters, and training context length (32K), with "the sole difference being the choice of multimodal RoPE" (Section 3.1).

• Generation budget / compute accounting. The paper does not use generation budgets in the standard LLM inference-scaling sense. All comparisons are made at equal training compute: each method is trained with identical batch size (128), identical optimizer (AdamW with α=0.9, β=0.98, weight decay 0.05), identical learning rate schedule (cosine decay from 1×10⁻⁵ to 3×10⁻⁶), and "approximately 512 NVIDIA A100 GPU hours" per experiment (Section 3.1). The training context length is set to 32K tokens, and the rotary base is 1,000,000 for all experiments. For inference-time evaluation, the maximum number of visual tokens is capped at 24K with "smart resize" to preserve dynamic resolution; video frames are sampled at 2 FPS by default with a cap of 768 frames. For the long-context extrapolation experiments (Figure 5, Appendix D.4), models trained at 32K context are evaluated at 64K, 128K, and 256K context lengths without additional training.

• Cross-validation / statistical protocol. None. The paper does not mention cross-validation, multiple training runs with different seeds, or any form of statistical significance testing. All results in Tables 2, 3, 10, 11 appear to come from single training runs. The paper reports "highest score in bold, second-highest underlined," implying deterministic ranking without error bars. This is a notable methodological limitation (see Critical Assessment).

---

Main Quantitative Results

The paper organizes its quantitative results into three levels: (1) overall performance on the main Qwen2.5-VL-7B backbone (Table 2), (2) generalization across Qwen3-VL architectures (Table 3, full details in Tables 10–11), and (3) targeted ablation studies on position design and frequency allocation (Tables 4–5). Additional results on long-context extrapolation, YaRN compatibility, and stride ablation appear in the Appendix.

Overall Performance on Qwen2.5-VL-7B (Table 2)

Table 2 reports results across 25 individual benchmarks (11 image, 6 video, 8 grounding) for seven methods: Vanilla RoPE, MRoPE, VideoRoPE, HoPE, CircleRoPE, MHRoPE, and MRoPE-I. The paper computes three modality-level averages (Image, Video, Grounding), which I reproduce from Table 2:

MRoPE-I achieves the top Image and Grounding averages; HoPE achieves the top Video average (52.95 vs. 52.58 for MHRoPE and 52.36 for MRoPE-I).

Headline findings at the per-benchmark level:

• MRoPE-I vs. Vanilla RoPE: MRoPE-I outperforms vanilla RoPE on MMMU by +2.67 points (53.22 vs. 50.56), ChartQA by +5.28 points (62.12 vs. 56.84), and RefCOCOval by +3.27 points (80.94 vs. 77.67). These are the largest absolute gains cited in the paper.

• Document benchmark collapse for VideoRoPE and HoPE: VideoRoPE achieves 60.13 on DocVQA and 37.42 on InfoVQA; HoPE achieves 60.12 and 34.80 respectively. Both are dramatically below vanilla RoPE's 82.94 (DocVQA) and 58.85 (InfoVQA). This is the most visually salient failure mode in the entire results table, with DocVQA dropping by ~22 points.

• Vanilla RoPE's grounding weakness: Despite its competitive image average (62.17, second only to the proposed methods), vanilla RoPE underperforms on ChartQA (56.84 vs. 63.56 for MRoPE, 62.44 for MHRoPE, 62.12 for MRoPE-I) and RefCOCOval (77.67 vs. 80.94 for MRoPE-I). The paper attributes this to the "fundamental limitations of its flattened, 1D position design" (Section 3.2).

• CircleRoPE's video weakness: CircleRoPE scores only 51.09 on the Video average, the lowest among all methods, and underperforms particularly on VideoMME (57.70 vs. 59.52 for HoPE) and LVBench (38.80 vs. 40.99 for HoPE). This empirically confirms the paper's analysis that collapsing all video frames onto a single spatial ring eliminates temporal information (Section 2.2.2).

• MRoPE's limitations: Despite being the direct predecessor of the proposed methods, MRoPE is outperformed by MHRoPE and MRoPE-I on almost every benchmark category. Its most visible weaknesses are on long-video understanding (MLVU 63.26 vs. 66.05 for VideoRoPE, LVBench 39.22 vs. 40.99 for HoPE) and visual grounding (RefCOCOval 78.35 vs. 80.94 for MRoPE-I). The paper attributes these respectively to temporal axis being stuck in high frequencies and spatial axes receiving restricted frequency spectra (Section 2.3.2).

• MHRoPE vs. MRoPE-I: MRoPE-I consistently edges out MHRoPE by small margins: Image 63.79 vs. 62.92 (+0.87), Grounding 75.85 vs. 74.92 (+0.93), though MHRoPE has a slight edge on Video (52.58 vs. 52.36, +0.22). The paper attributes MHRoPE's minor deficit to "head-level information partitioning, which prevents the integration of different positional axes within the self-attention mechanism" (Appendix D.1).

What the averages obscure: The per-benchmark variability is substantial. For example, on BLINK (multi-image reasoning), CircleRoPE achieves 40.88, outperforming vanilla RoPE (36.12) and MRoPE (37.93), while MHRoPE and MRoPE-I push this further to 42.80 and 44.08 respectively. This suggests the multi-image setting particularly benefits from the 3D position design, consistent with the paper's emphasis on preserving visual structure. On MMBench, all methods cluster tightly (range 74.06–75.95), suggesting this benchmark is less sensitive to positional encoding choices—the paper does not comment on why.

Generalization Across Architectures (Tables 3, 10, 11)

Table 3 reports modality-level averages for Qwen3-VL-4B-Instruct and Qwen3-VL-8B-Instruct (full per-benchmark results in Appendix D.6, Tables 10–11).

Qwen3-VL-4B-Instruct (Table 3a): MRoPE-I achieves the best Image (48.82) and Grounding (27.52) averages, while MHRoPE achieves the best Video average (50.80). The absolute numbers are substantially lower than the 7B-scale results, consistent with the smaller model size. The ranking of methods is largely preserved: MRoPE-I and MHRoPE outperform all prior methods across modalities, though the margins vary. Notably, the VideoRoPE/HoPE document collapse is replicated: VideoRoPE drops to 25.27 on DocVQA and 16.22 on InfoVQA versus 46.59 and 30.55 for vanilla RoPE (Table 10). The diagonal layout failure is therefore not specific to the 7B backbone.

Qwen3-VL-8B-Instruct (Table 3b): MRoPE-I again achieves top Image (64.82) and Grounding (75.46) scores. HoPE achieves the best Video average (57.81), slightly outperforming MRoPE-I (57.64) and MHRoPE (57.46). The paper notes in Section 2.3.2 that VideoRoPE and HoPE's frequency allocation (temporal → low frequencies) is beneficial for video but harmful for spatial tasks—a trade-off visible here where HoPE leads on Video but trails on Image (59.96) and Grounding (70.19). The 8B results replicate the same qualitative patterns: positional coherence failures are catastrophic (VideoRoPE DocVQA 71.71 vs. vanilla 82.41), full frequency utilization benefits all-around performance, and MRoPE-I provides the best balance.

Key replication findings across architectures:

• "Previously observed phenomena, such as the performance degradation caused by diagonal layouts, are corroborated in these new experiments" (Section 3.3).
• The ordering MRoPE-I ≥ MHRoPE > MRoPE ≈ Vanilla RoPE > VideoRoPE/HoPE (on image tasks) and MRoPE-I ≈ MHRoPE ≈ HoPE > MRoPE ≈ Vanilla RoPE (on video tasks) is consistent across 4B, 7B, and 8B scales.
• The grounding advantage of the proposed methods is particularly pronounced: on Qwen3-VL-8B, MRoPE-I achieves 75.46 Grounding average versus 70.09 for vanilla RoPE (+5.37) and 70.97 for MRoPE (+4.49). This supports the paper's claim that full-spectrum spatial encoding is critical for fine-grained localization.

Long-Context Extrapolation (Figure 5, Appendix D.4)

Figure 5 (Appendix D.4) shows video benchmark performance when models trained at 32K context length are extrapolated to 64K, 128K, and 256K. The paper reports:

• "Apart from LVBench, we do not observe clear performance improvements or degradation when extrapolating to longer sequences" (Appendix D.4).
• The only exception is vanilla RoPE, which "suffers from a sharp performance drop at 128K/256K" attributed to "excessively fast-growing position IDs, which lead to degraded extrapolation capability."
• Methods with temporal-biased frequency allocation (VideoRoPE, HoPE) "exhibit slightly better extrapolation ability in long video scenario," but the paper frames this as a trade-off against their poor image and grounding performance: "when considering performance across images and grounding tasks, MHRoPE and MRoPE-I remain the most comprehensive and balanced designs."
• Table 6 shows MRoPE-I + YaRN at 256K context achieves 43.6 on LVBench and 64.1 on MLVU, versus 41.2 and 63.3 for MRoPE + YaRN, demonstrating the interleaved allocation's compatibility with frequency-scaling extrapolation algorithms.

---

Ablation Studies and Robustness Checks

Position Design Ablation (Table 4)

Ablation setup: MRoPE-I's interleaved frequency allocation is fixed; only the position design is varied. The baseline is vanilla RoPE (Image 65.69, Grounding 73.48, Video 51.64). Four modifications are tested incrementally and against alternative designs:

• + 3D structure (no spatial-reset): Adding 3D position tuples using MRoPE's coordinate assignment (max-plus-1 rule, no spatial-reset) improves Image from 65.69 to 65.87, Grounding from 73.48 to 74.40, but slightly degrades Video from 51.64 to 51.29. The grounding gain (+0.92) confirms that preserving visual geometry matters most for spatial localization tasks. The slight video degradation is unexplained but may relate to the coupling of temporal and spatial coordinates described in Equation 3.

• + 3D + spatial-reset: Adding spatial-reset on top of 3D structure improves all three modalities: Image 65.87 → 66.65 (+0.78), Grounding 74.40 → 75.85 (+1.45), Video 51.29 → 52.36 (+1.07). This is the core evidence for spatial-reset's effectiveness. The paper interprets the gains—particularly the grounding improvement—as evidence that spatial-reset "significantly facilitate(s) the model's focus to visual information" (Section 2.2.3) by aligning visual tokens with the LLM's attention-sink bias.

• + diagonal layout (VideoRoPE/HoPE position design): Applying the diagonal layout while keeping interleaved frequency allocation causes catastrophic degradation on document benchmarks: DocVQA drops from 83.72 (with spatial-reset) to 60.13, InfoVQA from 58.24 to 37.42, ChartQA from 62.12 to 54.88. The Image average drops from 66.65 to 61.20. The Video average actually improves slightly (52.36 → 52.51), consistent with the diagonal layout's video-oriented design motivation. The paper identifies the failure mode as "repetitive, nonsensical text generation (e.g., '1111...'), which occurs even when the layout is applied only at inference time" (Section 3.4.1).

• + enlarged modality interval (RoPE-Tie-like): Artificially increasing the modality gap to match vanilla RoPE's sequential spacing degrades document-related performance: DocVQA drops from 83.72 to 70.43, InfoVQA from 58.24 to 42.18, ChartQA from 62.12 to 51.28. The failure mode is different from diagonal layout: the model generates "fluent but contextually irrelevant text, effectively ignoring the visual input" (Section 3.4.1). This establishes an upper bound on acceptable modality separation: too much distance prevents effective cross-modal attention.

• + text spatial-reset (IL-RoPE/Omni-RoPE design): Setting spatial coordinates to zero for both visual and text tokens causes uniform degradation across all categories: Image 66.65 → 58.27 (-8.38), Grounding 75.85 → 68.20 (-7.65), Video 52.36 → 50.71 (-1.65). This is the single largest degradation in the table and serves as the primary evidence for the "preservation of textual priors" guideline. The paper interprets this as confirmation that "preserving RoPE alignment for text is critical for successfully adapting LLMs into VLMs" (Section 3.4.1).

• + scaling rotary base (spatial axes base = 10K instead of 1M): Scaling the rotary base for spatial dimensions causes Image to drop from 66.65 to 60.15 (-6.50), with Video relatively unaffected (52.36 → 52.11) and Grounding slightly degraded (75.85 → 74.13). The paper frames this as another text-compatibility violation: "even well-intentioned deviations from the base LLM's RoPE formulation can break compatibility and severely impair knowledge transfer" (Section 3.4.1).

Non-obvious findings: The two different mechanisms for breaking positional coherence—diagonal layout (causes position overlap) and enlarged modality interval (causes visual ignoring)—produce similar-magnitude degradation on document benchmarks but via qualitatively different failure modes. The diagonal layout's failure at inference time (without retraining) is particularly revealing: it confirms the problem is architectural (positional ambiguity confusing the attention mechanism) rather than a training data issue. The text spatial-reset's degradation being larger than any position design modification highlights that the LLM's pre-trained positional associations are the most fragile component in the multimodal RoPE stack.

Frequency Allocation Ablation (Table 5)

Ablation setup: The position design is fixed to MRoPE with spatial-reset enhancement; only the frequency allocation strategy is varied. Four strategies are compared:

• VideoRoPE-like: Contiguous chunking with temporal assigned to low-frequency channels (the VideoRoPE/HoPE frequency inversion). Overall: 63.31.

• IL-RoPE-like: Partial interleaving but lowest frequencies reserved for temporal. Overall: 63.07—slightly worse than the VideoRoPE-like allocation, suggesting that partial interleaving without full-spectrum coverage is counterproductive.

• Multi-Head (MHRoPE): Head-level partitioning, full spectrum per axis. Overall: 64.63. Image: 66.40, Video: 52.58, Grounding: 74.92.

• Interleave (MRoPE-I): Fine-grained round-robin channel allocation, full spectrum per axis. Overall: 64.95. Image: 66.65, Video: 52.36, Grounding: 75.85.

Key finding: "A more uniform allocation strategy consistently outperforms alternatives that split the spectrum into partial chunks" (Section 3.4.2). The interleaved and multi-head strategies—both providing full frequency spectrum per axis—outperform the chunked variants by 1.3–1.9 points overall. Within the full-spectrum variants, MRoPE-I's interleaving slightly edges out MHRoPE's head-level partitioning (64.95 vs. 64.63), consistent with the main results. The VideoRoPE-like allocation's relatively poor Image score (65.33 vs. 66.65 for interleave) supports the paper's claim that restricting spatial axes to high frequencies impairs multi-scale spatial reasoning.

Non-obvious finding: The IL-RoPE-like allocation, despite being a form of interleaving (giving spatial axes access to some mid and high frequencies), performs worse than the purely chunked VideoRoPE-like allocation (63.07 vs. 63.31). This suggests that partial interleaving that reserves the lowest frequencies for temporal is actually worse than a clean contiguous split—perhaps because it creates an inconsistent frequency distribution that neither specializes axes (like chunking) nor provides full uniformity (like full interleaving). The paper does not explore this intermediate regime further.

Attention Pattern Analysis (Table 7, Appendix D.5.1)

Visual attention sink quantification: Table 7 reports average attention scores on visual tokens at layers 4, 12, 20, and 28 from the DocVQA test set, comparing MHRoPE and MRoPE-I with and without spatial-reset:

The pattern is consistent: spatial-reset increases attention to visual tokens across almost all layers and methods, with the effect most pronounced in deeper layers. For MHRoPE, the deep layer (28) difference is 19.00 vs. 9.93—nearly a 2× increase. For MRoPE-I, the same layer shows 23.23 vs. 11.69—again roughly 2×. This directly supports the paper's mechanistic claim that spatial-reset works by enhancing visual attention, particularly in later transformer layers where high-level visual-semantic integration occurs.

Non-obvious finding: The attention patterns differ qualitatively between MHRoPE and MRoPE-I. MHRoPE shows higher visual attention in early layers (40.31 at layer 4 vs. 37.48 for MRoPE-I) but lower in deep layers (19.00 at layer 28 vs. 23.23). MRoPE-I's deeper-layer visual attention advantage may explain its slight overall performance edge—the paper does not explicitly make this connection but the data suggest it.

Frequency Allocation Ratio Ablation (Table 8, Appendix D.5.2)

The paper sweeps the t:h:w channel allocation ratio under the interleaved frequency design (MRoPE-I's channel assignment pattern, but varying how many channels each axis receives in the round-robin cycle):

The balanced 24:20:20 ratio achieves the best Overall score (64.95). Eliminating temporal channels entirely (0:32:32) slightly improves Image (+0.23 over 24:20:20) but severely degrades Video (-1.35), confirming temporal information is needed for video. Heavy temporal allocation (48:8:8) degrades Grounding substantially (72.87 vs. 75.85)—spatial axes receive too few channels for fine-grained localization. The 24:20:20 ratio represents a Pareto-optimal balance where slight temporal bias (24/64 channels ≈ 37.5% to temporal, vs. 31.25% per spatial axis) captures the natural asymmetry that temporal ranges can be much larger than spatial ranges, without starving spatial axes of frequencies.

Non-obvious finding: The Image score is relatively insensitive to allocation ratio as long as spatial axes get at least ~20 channels each (scores range 64.07–66.65 across all ratios), while Grounding is highly sensitive (72.87–76.02). This suggests fine-grained localization is the task most dependent on high-frequency spatial channels, consistent with the paper's analysis that spatial frequencies enable precise positional discrimination (Section 2.3.2).

Temporal Stride Ablation (Table 9, Appendix D.5.3)

For MRoPE-I, the temporal stride between video frames is varied: δ = 0.5, 1, 2, and "Dynamic" (V2PE-style variable stride). The Overall video average is 51.11 (δ=0.5), 52.36 (δ=1), 51.10 (δ=2), and 51.80 (Dynamic). δ = 1 is optimal. Smaller strides (0.5, meaning twice as many temporal positions per frame) and larger strides (2, meaning temporal positions advance faster than frame count) both degrade performance. Dynamic stride does not provide significant benefit over fixed δ=1. The paper treats this as a straightforward hyperparameter selection: δ=1 is used throughout all experiments (Section 3.1).

YaRN Extrapolation Compatibility (Table 6, Appendix D.3)

At 256K context with YaRN applied, MRoPE-I improves from 42.0 to 43.6 on LVBench and from 63.2 to 64.1 on MLVU. MRoPE with YaRN shows minimal improvement (LVBench: 41.5 → 41.2, actually a slight degradation; MLVU: 62.9 → 63.3). The interpretation: "MRoPE-I's interleaved design provides a full spectrum across all positional axes, enabling a straightforward and symmetric application of these methods" while MRoPE's "partitioned spectrum complicates the application of a consistent frequency scaling boundary" (Appendix D.3). This is a robustness check confirming that the interleaved allocation's theoretical extrapolation compatibility (Section 2.3.3) translates to empirical gains.

---

Critical Assessment

Does the evidence support the claim that full frequency utilization is the key driver of robust performance?

The evidence is strong but conditional. Table 5 convincingly shows that two different full-spectrum strategies (MHRoPE and MRoPE-I) outperform two different partial-spectrum strategies (VideoRoPE-like and IL-RoPE-like) when position design is held constant. The overall score difference (64.95/64.63 vs. 63.31/63.07) is consistent, and the per-modality breakdown reveals the expected trade-offs: partial-spectrum methods that favor temporal (VideoRoPE-like, IL-RoPE-like) score slightly better on Video (52.11 and 51.15) than MRoPE-I (52.36) but substantially worse on Image (65.33 and 65.26 vs. 66.65). However, the evidence would be stronger with two additions. First, there is no direct comparison of MHRoPE and MRoPE-I to a method that gives every axis the full spectrum through a simpler mechanism—for example, simply using the same frequency assignment for all three axes and relying on different position ID values to disambiguate. The paper's two full-spectrum strategies are both non-trivial (head partitioning, interleaving); a null experiment showing that naive full-spectrum sharing works would strengthen the causal claim. Second, the ablation only tests four allocation strategies. The space of possible allocations is continuous (ratios of channels per axis) and discrete (interleaving patterns), and Table 8 only partially explores the ratio dimension under the interleaved pattern. It is possible that an unexplored strategy (e.g., interleaving with a different ratio, or a hybrid head-level + channel-level approach) would outperform both MHRoPE and MRoPE-I. The paper's claim that full frequency utilization is "the key driver" is supported, but the claim that MRoPE-I's specific interleaving pattern is optimal for doing so is only partially explored.

Does the evidence support the claim that positional coherence is a necessary condition?

Strongly supported for the specific failure modes documented. The diagonal layout ablation (Table 4) demonstrates that positional overlap between modalities causes catastrophic degradation on document benchmarks, with the failure mode (repetitive text generation) occurring even at inference time without retraining. The enlarged modality interval ablation demonstrates that too much separation causes the model to ignore visual input. These are clear existence proofs that the viable region for modality separation is bounded on both sides. However, the paper's "positional coherence" guideline is stated more broadly than what is tested: it encompasses "unambiguous coordinates with a well-defined modality interval" (Section 1), but the experiments only manipulate the modality interval within the MRoPE position design framework (3D tuples with max-plus-1 advancement). The guideline's generality to other position design paradigms (e.g., what if modalities were separated in a different coordinate system, or what if the coordinate advancement rule were different?) is untested. The paper's framework would predict that any position design creating overlap or excessive separation would fail, but no experiment tests this prediction on a design radically different from MRoPE's.

Does the evidence support the claim that text RoPE compatibility is a hard constraint?

Strongly supported through negative results. The two compatibility violations tested (scaling rotary base for spatial axes, applying text spatial-reset) both cause large, uniform degradation across all benchmark categories (Table 4). The text spatial-reset degradation (-8.38 Image, -7.65 Grounding) is the single largest ablation effect in the paper. The interpretation—that these changes break pre-trained knowledge transfer—is consistent with the magnitude and uniformity of the drops. However, the paper tests only two specific violations. It does not establish that all violations of text compatibility are catastrophic, only that these two are. A more systematic test would sweep the degree of violation (e.g., progressively scaling the rotary base and measuring the performance-degradation curve) to establish whether there is a continuous relationship or a threshold effect. Without such a sweep, the claim that compatibility must be exact rather than approximate is inferred from two data points. It is also worth noting that IL-RoPE and Omni-RoPE—which the paper criticizes for breaking text compatibility—were designed for image generation/editing tasks, not for the general image/video/grounding understanding suite used here. Their text spatial-reset modification may provide benefits on generation tasks that the paper's evaluation does not capture, so the claim that breaking text compatibility is "always wrong" is broader than the evidence supports.

Does the evidence support the claim that spatial-reset enhances visual attention?

Supported at the mechanistic level by Table 7, which shows a roughly 2× increase in visual attention scores in deep layers when spatial-reset is applied. The attention pattern analysis is limited to one dataset (DocVQA test set) and four layers, but the effect is consistent across both MHRoPE and MRoPE-I. The performance gains (Table 4: +0.78 Image, +1.45 Grounding, +1.07 Video) are consistent with the attention mechanism but could in principle be caused by the motion disentanglement benefit (Equations 3–4) rather than the attention-sink realignment. The paper does not provide an ablation that isolates these two mechanisms—for instance, testing spatial-reset on static images only (where motion disentanglement is irrelevant) to see if the attention-sink effect alone drives the image benchmark gains. The video gains of spatial-reset (+1.07) are larger than the image gains (+0.78), which is consistent with both mechanisms operating simultaneously, but this is suggestive rather than conclusive.

Genuine methodological weaknesses

No statistical significance reporting. Every number in Tables 2, 3, 10, 11 is a point estimate from what appears to be a single training run. The differences between top methods are often small: MRoPE-I vs. MHRoPE Image average differs by 0.87, Video average by 0.22, Grounding average by 0.93. Without error bars or multiple seeds, it is impossible to know whether these differences are statistically reliable or within training noise. This is a significant limitation given that the paper's recommendation of MRoPE-I over MHRoPE (Appendix D.1) hinges partly on "consistent (albeit slight) performance advantage." The advantage may be consistent across benchmarks but could also be consistent noise if the training variance is larger than the between-method differences.

Single training run, single data mixture. All experiments use the identical ~2M SFT samples, but the composition of this mixture is described only at the category level (image captioning, OCR, visual reasoning, etc.). If the SFT mixture is imbalanced toward certain task types, the benchmark results may reflect training data bias rather than positional encoding quality. For instance, if grounding data is underrepresented, the grounding gains from MRoPE-I might be even larger with balanced data—or smaller, if the gains are partly an artifact of the data distribution interacting with the RoPE design. The paper does not report data mixture ratios or ablate data composition.

No baseline giving the larger model equal treatment. All baselines are implemented under the same training recipe, which is a strength for controlled comparison. However, the evaluation does not include a "best-effort" version of each baseline—for example, VRoPE with hyperparameters tuned for the specific training data, or MRoPE with a different channel allocation ratio than the standard 1:1:1 split. The baselines are implemented as described in their original papers, but those papers may have used different training data, different model scales, or different hyperparameters. Under the specific conditions of this paper's setup, some baselines may underperform their potential, making the proposed methods look better by comparison. The paper acknowledges this implicitly by noting that its initialization "aligns with that of VideoRoPE and HoPE" (Section 3.1), but this alignment may favor those baselines while disadvantaging others.

Limited exploration of MHRoPE's head allocation ratios. The paper does not report how many attention heads are allocated to each axis in MHRoPE. The standard Qwen2.5-7B likely has 32 heads, so the allocation might be, e.g., 10 temporal, 11 height, 11 width, or some other split. The ratio ablation (Table 8) is performed only for MRoPE-I's channel allocation, not for MHRoPE's head allocation, leaving open the possibility that MHRoPE's slight underperformance relative to MRoPE-I is due to suboptimal head allocation rather than the inherent limitation of "head-level information partitioning."

No combination of MHRoPE and MRoPE-I. The paper treats the two methods as alternatives, but they address different levels of the architecture (head partitioning vs. channel interleaving). A hybrid approach—interleaved channel allocation within each head, with different heads dedicated to different axes—is not explored. This hybrid might capture the benefits of both (full frequency resolution from interleaving, axis specialization from head partitioning) while mitigating MHRoPE's limitation (no within-head cross-axis integration) by providing all axes some representation in every head.

Evaluation benchmark suite favors the proposed methods' strengths. The benchmark suite heavily weights document understanding and grounding tasks, where 3D position design and full frequency allocation are most beneficial. The paper does not evaluate on tasks where 1D sequential design might have an advantage—for example, tasks requiring the model to reason about the order of multiple images in a sequence, where vanilla RoPE's natural sequential ordering might outperform coordinate-based designs. The BLINK benchmark (multi-image reasoning) partially addresses this, and the proposed methods do outperform there, but the space of potential tasks where vanilla RoPE excels is underexplored.

Missing experiments that would strengthen the paper

• Multiple training seeds for all methods, with mean and standard deviation reported, to establish whether between-method differences exceed within-method variance.
• Sweep of MRoPE's channel allocation ratios as a stronger baseline. The standard MRoPE uses equal chunking (d/3 channels per axis), but Table 8 shows allocation ratio matters for MRoPE-I. MRoPE could potentially be improved by rebalancing its chunks, and the paper does not test whether an optimized MRoPE (e.g., with more channels to temporal for video, or more to spatial for grounding) would close the gap with the proposed methods.
• Ablation isolating attention-sink realignment from motion disentanglement in spatial-reset, by testing on static-image-only benchmarks where motion is irrelevant.
• Zero-shot evaluation of all methods on a held-out benchmark not present in the SFT mixture, to test whether the proposed RoPE variants genuinely improve multimodal reasoning or merely better fit the training distribution.
• Larger-scale experiments (e.g., Qwen2.5-72B) to test whether the benefits of full-spectrum allocation scale with model size or saturate. The generalization across 4B, 7B, and 8B is suggestive but the range is narrow.

Conditional nature of the claims

The paper's central claim—that following the three guidelines produces robust multimodal RoPE—holds under the specific conditions tested: Qwen-family models, the Qwen2.5-VL training recipe, a 2M-sample SFT mixture heavily weighted toward understanding tasks, and a 32K training context. The generalization experiments (Qwen3-VL-4B and -8B) extend this to different architectural variants within the same model family and training paradigm, but not to different model families (e.g., Llama-based VLMs) or different training paradigms (e.g., models that train ViT and LLM jointly rather than freezing the ViT). The paper's claim that the guidelines "pave the way for future advancements in VLMs" (Section 4) is aspirational and reasonable given the systematic evidence, but the specific methods (MHRoPE, MRoPE-I) should be understood as validated for Qwen-family models with frozen vision encoders—their transferability to other VLM architectures is plausible but unproven, despite the paper's "architectural generalization" framing in Section 3.3.

6. Limitations and Trade-offs

Assumption: Difficulty Estimation is Computationally Free

The paper's compute-optimal framework depends on knowing each prompt's difficulty before allocating the inference budget. The paper's method for estimating difficulty—generating 2048 samples per prompt and averaging their correctness scores (Section 3.2, Section 5.2)—consumes more compute than the largest inference budgets studied in the FLOPs-matched comparison. The paper acknowledges this explicitly:

"Our experiments do not account for this cost largely for simplicity" (Section 3.2)

The consequence. The headline efficiency claims—a 4× improvement over best-of-N (Section 6, Figure 8; Section 5, Figure 4)—are computed after difficulty is known, without amortizing the cost of learning it. In a deployment scenario, total compute would be the sum of difficulty estimation (2048 generations × PRM scoring per prompt) plus strategy execution. Depending on the prompt volume and the target inference budget, the difficulty estimation cost could dominate—potentially erasing or reversing the reported gains. An organization deploying this approach would need to either accept this upfront cost per unseen prompt or pre-compute difficulty for a fixed set of prompts (impractical for open-ended user queries).

What evidence exists. The paper does not report total-cost-inclusive numbers anywhere. The compute-optimal curves in Figure 4 and Figure 8 plot accuracy versus execution budget only. The predicted difficulty bins (which use PRM scores instead of ground-truth labels) are described as having "largely overlapping" performance with oracle bins (Section 5.3, Figure 4), but this comparison is between two methods that both incur the same 2048-sample estimation cost—neither is cost-free. The paper provides no ablation where difficulty is estimated from a small fraction of the full 2048 samples, nor any experiment where the difficulty estimation cost is included in the compute budget.

Mitigation status. The paper explicitly flags this as a key direction for future work (Section 8), suggesting "pretraining or finetuning models to directly predict difficulty of a question." It also proposes "adaptive difficulty estimation"—starting with a small number of samples, assessing the verifier's score distribution as a quick difficulty signal, and allocating the remaining budget accordingly—as an alternative that would subsume estimation into the solution process. Neither approach is implemented or evaluated. The difficulty estimation cost is therefore the single largest practical barrier between the paper's results and deployment, and it is entirely unaddressed.

---

Single Model Family and Single Benchmark

All experiments use PaLM 2-S* as the base model and the MATH benchmark as the evaluation dataset. While this is a deliberate choice—MATH provides clean correctness signals and PaLM 2-S* sits in a useful intermediate capability regime (Section 4)—it means every finding in the paper is conditioned on one model's output distribution and one task domain.

The paper states that it "believe[s] this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is not verified.

The consequence. Several aspects of the paper's results could be model-specific:

• The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties, different error patterns, or different solution styles might exhibit different difficulty-dependent scaling curves. For instance, a model that tends to produce verbose but wrong solutions might produce PRM scores that are harder to calibrate or easier to over-optimize.

• The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families. The paper's finding that revisions work best on easy problems (Figure 7, right) might reverse for a model with stronger self-correction abilities.

• The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning. It is unclear whether the difficulty-dependent patterns—beam search hurting easy problems, revisions helping easy problems, no method helping the hardest problems—generalize to other reasoning domains (code generation, logical deduction, scientific QA) or to tasks requiring factual recall rather than inference. Math problems also have a specific structure (multi-step, deterministic correctness) that favors certain test-time strategies; an open-ended generation task might show entirely different scaling behavior.

What evidence exists. The paper provides no cross-model or cross-domain experiments. All ablation studies (Sections 5.3, 6, 7) are conducted within the PaLM 2-S* + MATH setup. The paper does cite prior work that found contradictory results on self-correction (Huang et al., 2023 found it ineffective; Madaan et al., 2023 found it helpful) and uses its difficulty framework to reconcile these findings (Section 2), which is intellectually satisfying but does not substitute for direct replication on other models or benchmarks.

The test set consists of 500 MATH questions, split into five difficulty quintiles of ~100 each, then further split by two-fold cross-validation (Section 3.2). This means the compute-optimal policy is selected based on approximately 50 questions per fold per bin—a small sample for policy selection. The paper does not report confidence intervals on the compute-optimal scaling curves.

Mitigation status. No mitigation is attempted. The paper does not claim generalizability beyond the tested setup, but the strong normative language used throughout (e.g., "the optimal allocation," "compute-optimal scaling") implies universality that the evidence does not support. A reasonable reader should treat these results as established for PaLM-family models on competition-level math problems, with extrapolation to other settings being plausible but unvalidated.

---

The ~14× Larger Model Baseline is Weak

The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time strategies against a model with approximately 14× more parameters using only greedy decoding. This baseline has three weaknesses that the paper partially acknowledges:

1. The larger model is not compute-optimally trained. The paper scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than Chinchilla-optimal scaling (Hoffmann et al., 2022), which would scale both parameters and data equally. The paper states: "We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work" (Section 7).

2. The larger model gets no test-time compute of its own. All test-time compute budget is allocated to the smaller model. The larger model uses greedy decoding—no majority voting, no best-of-N, no search of any kind. In practice, a 14× larger model with even a modest test-time budget (e.g., best-of-8 or best-of-16) would be a substantially stronger baseline.

3. The FLOP accounting uses approximate formulas (X = 6ND_pretrain, Y = 2ND_inference; Section 7) that are standard but approximate—they ignore attention FLOPs, assume dense computation, and do not account for hardware efficiency differences between training and inference.

The consequence. The reported advantages of test-time compute over pretraining—for example, +27.8% relative improvement on easy questions at R ≪ 1 for revisions (Section 7, Figure 1 bar charts)—likely overstate the advantage that would be observed against a properly compute-optimal larger model with a modest inference-time budget. Since the paper's central policy claim is about the training-inference compute tradeoff, a systematically weak pretraining baseline inflates the case for test-time compute.

The paper's finding that test-time compute cannot help on the hardest problems (difficulty bin 5, Section 7, Figure 9) is largely independent of this baseline weakness—no amount of test-time compute on the smaller model matches the larger model's performance on bin 5. But the finding that test-time compute can substitute for pretraining on easy-medium problems is partially an artifact of the larger model being suboptimally trained and evaluated without any inference-time augmentation.

What evidence exists. The paper's Figure 9 shows the larger model's performance as stars at three x-axis positions (corresponding to the three R values). The compute-optimal scaling curves are above these stars for easy bins (1–2) but not for hard bins (4–5). The stars' vertical positions depend entirely on the larger model's greedy accuracy—if the larger model were Chinchilla-optimally trained or evaluated with best-of-8, all stars would shift upward, potentially reversing some of the "test-time compute wins" cases, especially at R ≈ 1 and R ≫ 1.

The paper does not provide any sensitivity analysis: no experiment with a Chinchilla-optimal larger model, no experiment giving the larger model a test-time compute budget, and no exploration of how the FLOPs equivalence changes under more precise FLOP counting.

Mitigation status. The paper acknowledges the non-Chinchilla-optimal scaling as a deliberate choice but frames it as representative practice rather than a limitation. The absence of test-time compute for the larger model is not discussed as a limitation. A fairer comparison would allocate some fraction of the test-time budget to the larger model as well—for instance, comparing small-model-with-compute-optimal-test-time against large-model-with-best-of-16 under the same total FLOPs.

---

Verifier Over-Optimization is the Scaling Ceiling, and It Is Unsolved

The paper documents verifier over-optimization as the factor that limits continued improvements from additional test-time compute. Beam search degrades easy-problem performance at high budgets (Figure 3, right). Lookahead search—the most powerful optimizer—paradoxically performs worst overall (Figure 3, left). Qualitative examples in Appendix M show search producing degenerate outputs (repetitive low-information steps, overly short 1–2 step solutions) that score highly under the PRM but are incorrect. The paper's compute-optimal policy mitigates this by routing easy problems away from aggressive search, but it does not solve the underlying problem.

The consequence. Test-time compute scaling is fundamentally bounded by verifier quality, not by search algorithm sophistication or compute budget. On medium-difficulty problems where beam search is deployed, the scaling curves in Figure 3 (right, bins 3–4) flatten well before the maximum budget is reached—beam search at 256 generations is only marginally better than at 64 or even 16 generations. This means the compute-optimal approach cannot achieve arbitrary improvements by scaling compute further; it asymptotes at whatever accuracy the verifier can reliably discriminate.

The paper's central finding—that difficulty-adaptive allocation yields 4× improvements—should be understood as achieving the same performance with less compute, not as achieving higher performance with more compute. The absolute accuracy ceiling is still set by the PRM, and that ceiling is reached relatively quickly (within 64–256 generations for most methods in Figure 3).

What evidence exists. Figure 3 (left) shows beam search plateauing around 34% at high budgets while best-of-N weighted continues to improve slowly, reaching approximately 38% at 512 generations—suggesting that aggressive optimization hits the over-optimization wall earlier than simple sampling. Figure 3 (right, bin 1) shows beam search decreasing in accuracy as budget increases (from ~78% at 4 generations to ~77% at 256), the clearest signature of verifier exploitation. Appendix M provides qualitative examples of degenerate search outputs.

The paper does not provide a detailed analysis of why the PRM over-optimizes—whether it is due to imperfect calibration on out-of-distribution solution steps, insufficient coverage of the solution space in PRM training data, or fundamental limitations of the Monte Carlo rollout supervision approach. The paper also does not experiment with techniques to improve verifier robustness, such as ensembling multiple PRMs, adversarial training on search-generated solutions, or regularization that penalizes solutions far from the base model's typical output distribution.

Mitigation status. The paper identifies verifier over-optimization as a bottleneck (Sections 5.3, 8) and the compute-optimal policy can be partly understood as a workaround—staying below the over-optimization threshold on easy problems by using the weaker best-of-N optimizer. But the fundamental problem is unsolved: on medium-difficulty problems where aggressive optimization is deployed, over-optimization still limits the benefit of additional compute. The paper suggests improving verifier robustness as a direction for future work but provides no concrete proposals or preliminary experiments. This limitation is the most significant barrier to achieving unbounded scaling of test-time compute—without it, test-time compute can at best reach the verifier's reliability frontier, not surpass it.

---

Hard Problems Remain Fundamentally Out of Reach

Across every method and every budget level, the hardest questions (difficulty bin 5, corresponding to the lowest 20% of base model pass@1) show near-zero improvement from any test-time compute strategy. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all search methods and all budgets from 4 to 256 generations. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy regardless of the sequential-to-parallel ratio at 128 generations. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling curve is essentially flat near 0–5% for both revisions and PRM search.

The consequence. Test-time compute can amplify existing capability but cannot create new capability. If the base model's probability of generating a correct solution is near zero (pass@1 ≈ 0 on bin 5 problems), then no amount of sampling, search, or revision will find a correct solution—there are simply no correct trajectories in the proposal distribution to discover or refine. This is the fundamental boundary that separates test-time compute scaling from pretraining scaling: pretraining can teach the model entirely new capabilities (new reasoning patterns, new factual knowledge), while test-time compute can only better exploit capabilities already present.

This matters for deployment decisions. The paper's FLOPs-matched comparison (Section 7, Figure 9) shows that on hard problems, the ~14× larger model consistently outperforms the smaller model with test-time compute, regardless of the R ratio. For organizations whose problem distribution skews toward genuinely difficult tasks (those where the base model's pass@1 is negligible), investing in pretraining is the only viable path. Test-time compute is not a substitute—it is a complement that works only within the base model's capability envelope.

What evidence exists. The bin 5 results are consistently flat across all experimental sections—search (Figure 3), revisions (Figure 7), and FLOPs-matching (Figure 9). This is the most robust null result in the paper: no method, no budget, no allocation strategy can overcome the base model's fundamental inability to produce correct answers on these problems. The paper is explicit about this limitation (Section 7 takeaway box, Section 8), noting that test-time compute "cannot compensate for fundamental capability gaps that larger pretraining would address."

Mitigation status. None, and none is possible within the test-time compute paradigm. The paper acknowledges the limitation clearly but does not explore ways to expand the base model's capability envelope—for instance, by using retrieval-augmented generation to inject external knowledge, or by decomposing hard problems into easier sub-problems that the base model can solve. These are fundamentally different approaches that fall outside the scope of inference-compute allocation. The practical implication for practitioners is that their problem distribution's difficulty relative to their chosen base model's capabilities is the single most important factor determining whether test-time compute is worth investing in—a point the paper makes clearly but that deserves emphasis as a deployment consideration.

---

The Revision Model Suffers from a 38% Correct-to-Incorrect Reversion Rate

The revision model is trained exclusively on sequences where all in-context answers are incorrect followed by a correct target (Section 6.1). This creates a distribution shift at inference time: when the revision chain produces a correct answer, the model has never been trained on what to do next—it only knows how to revise incorrect answers into correct ones.

The paper reports that approximately 38% of correct answers get converted back to incorrect ones during sequential revision chains (Section 6.1). The paper mitigates this with post-hoc selection (majority voting or verifier-based selection) across the entire chain rather than taking the final revision output, but this is a patch that adds latency and does not address the root cause.

The consequence. Sequential revisions are inherently unreliable as a chain—the model cannot be trusted to preserve correctness from one step to the next. The reported benefits of sequential revision (Section 6, Figure 6) are achieved despite this reversion, by selecting the best answer from across the chain rather than trusting the final output. This means:

• Latency penalty: generating a chain of N sequential revisions takes N× longer wall-clock time than N parallel samples, yet the chain cannot simply be truncated at the point of apparent correctness (since the model might revert). The selection mechanism requires generating the full chain and then retroactively picking the best answer.

• Wasted compute: approximately 38% of correct answers produced mid-chain are subsequently corrupted, meaning a significant fraction of the sequential compute budget is spent undoing progress.

• Fragility to training methodology: the paper's attempt to optimize the revision model with ReST^EM (Appendix K, Figure 16) caused performance to degrade substantially, suggesting the revision training procedure is sensitive to on-policy data collection in ways that are not fully understood.

What evidence exists. The 38% reversion rate is stated in Section 6.1 without a detailed per-difficulty or per-chain-length breakdown. The ReST^EM failure is documented in Appendix K (Figure 16), where fully sequential performance drops to ~33.5% compared to ~38.5% at the optimal ratio, with the paper hypothesizing that "on-policy data collection in ReST^EM exacerbates spurious correlations in revision data" (Section 6).

Mitigation status. The paper's within-chain selection (majority or verifier-based) is a partial mitigation—it recovers correct answers that would otherwise be lost to reversion—but it does not prevent the waste of compute spent on the reversion steps themselves. A more principled solution, such as training the revision model to recognize when no revision is needed (by including "stop" or "keep" examples in the training data where the current answer is already correct), is not explored. The paper acknowledges this as a design consequence but does not propose a fix or analyze its quantitative impact on the compute-optimal allocation strategy.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the field's understanding of multimodal positional encoding from a collection of ad hoc, modality-specific solutions to a principled design space with diagnosable failure modes. Before this work, the choice between MRoPE, VideoRoPE, HoPE, CircleRoPE, and IL-RoPE was effectively a matter of which prior paper's implementation was most convenient to adopt—there was no shared vocabulary for articulating why one method outperformed another on a given benchmark, nor any framework for predicting when a particular design would break. The paper's decomposition of multimodal RoPE into three independent axes—position design, frequency allocation, and text RoPE compatibility—changes this by making the design space tractable. A practitioner building a new VLM can now reason about these axes separately: fix frequency allocation to full-spectrum interleaving, then optimize position design (spatial-reset, appropriate modality interval) independently, knowing that text RoPE compatibility is a hard constraint that must never be violated.

This is not merely a taxonomy. The decomposition is diagnostic in a way that prior literature could not achieve. The paper demonstrates that VideoRoPE's catastrophic collapse on document benchmarks (DocVQA dropping from ~82% to ~60%, Table 2) is not a mysterious interaction between the diagonal layout and the frequency allocation—it is a pure position design failure, reproducible even when the frequency allocation is fixed to the optimal interleaved strategy (Table 4, diagonal layout row). Similarly, MRoPE's underperformance on long-video understanding (MLVU 63.26, LVBench 39.22) is not a consequence of its position design (which is largely sound) but of its chunked frequency allocation, and improving only the allocation while keeping the position design roughly constant yields measurable gains (MRoPE-I on MLVU 65.46, LVBench 40.54). The ability to isolate failure modes to specific design choices, and then fix them independently, represents a qualitative improvement in how the field can approach multimodal positional encoding.

The paper also resolves a latent contradiction in the prior literature. Methods like VideoRoPE and HoPE were designed for video understanding and showed strong results on video benchmarks in their original papers—yet when evaluated in the paper's controlled setting, they catastrophically degrade on document understanding tasks. The paper provides a mechanistic explanation: the diagonal layout creates positional overlap for high-resolution documents, confusing the model about which tokens are visual versus generated text. This does not mean the original video results were wrong—rather, the original papers' evaluation suites were narrow (video-only or video-heavy), missing the failure modes on other modalities. The paper's holistic evaluation across 20+ benchmarks spanning images, videos, and grounding reveals that designing for one modality in isolation produces fragile solutions that break on others. The implication for the field is clear: multimodal RoPE must be evaluated on comprehensive benchmarks spanning all target modalities, because optimizations for one modality can silently destroy performance on others.

The paper's most consequential empirical finding—that full frequency utilization is the key driver of robust performance—represents a conceptual reframing of the frequency allocation problem. Prior work treated the d/2 frequency pairs as a resource to be divided across positional axes, operating under the implicit assumption that different axes need different frequency characteristics. The paper demonstrates that this assumption is wrong: every axis benefits from access to the complete spectrum, and the axes can be disambiguated through their position ID values rather than their frequency content. This is a genuinely counterintuitive result—one might reasonably expect that assigning the same frequencies to different axes would create ambiguity, but the attention mechanism's reliance on position ID differences (not just frequency similarity) makes full-spectrum sharing not only workable but optimal. The two proposed methods (MHRoPE and MRoPE-I) serve as existence proofs that full-spectrum strategies can be implemented with minimal architectural changes, and their consistent superiority across benchmarks (Tables 2, 3, 10, 11) establishes this finding as robust.

The spatial-reset mechanism introduces a different kind of reframing: multimodal position design should account for the pre-existing biases of the language model backbone, not just the geometry of the visual content. Prior work treated position design as a pure geometric problem—how to assign coordinates that faithfully represent 2D or 3D structure. The paper's discovery of the visual attention sink (Figure 2) and the demonstration that repositioning it via spatial-reset yields measurable gains (Table 4, Table 7) reveals that the LLM's training history imposes constraints and opportunities that geometric reasoning alone would miss. This insight is likely to influence future work beyond RoPE design—for instance, in connector design (where to place visual tokens in the LLM's input space), in training recipe design (whether to train the LLM to attend differently to visual versus text tokens), and in multi-image/video interleaving strategies.

Finally, the paper's demonstration that text RoPE compatibility is a hard binary constraint, not a tunable hyperparameter, establishes an important design principle that prior work (IL-RoPE, Omni-RoPE) violated without fully understanding the cost. The 6–8 point absolute degradation from seemingly minor modifications (Table 4: scaling rotary base, text spatial-reset) serves as a stark warning: the LLM's pre-trained positional associations are the most fragile component in the VLM stack, and any modification to text token encoding will incur a performance penalty that limited multimodal SFT data (~2M samples) cannot recover from. This implies that future multimodal RoPE extensions should be strictly additive for visual tokens, leaving text encoding entirely untouched—a principle that the paper's proposed methods follow but that is not universally respected in the literature.

Follow-Up Research This Work Enables

Systematic replication across model families. The paper's experiments are confined to Qwen-family models (Qwen2.5-VL-7B, Qwen3-VL-4B, Qwen3-VL-8B). While the cross-architecture replication within the Qwen lineage is valuable (Section 3.3), the generalization claim would be substantially strengthened by replicating the full comparison on Llama-based VLMs (e.g., LLaVA, Llama 3-V) and on models with different vision encoders and training paradigms. A strong follow-up would implement MHRoPE, MRoPE-I, MRoPE, VideoRoPE, and Vanilla RoPE on a Llama-3-8B backbone with a CLIP or SigLIP vision encoder, training on the same ~2M SFT mixture and evaluating on the same benchmark suite. The key question is whether the three guidelines (positional coherence, full frequency utilization, text compatibility) produce the same ranking of methods across model families, or whether Qwen-specific architectural details (e.g., the specific ViT, the connector design, the pre-training data distribution) interact with RoPE design in ways that change the optimal strategy. A negative result—finding that full-spectrum allocation is not universally optimal—would be as informative as a positive replication, because it would reveal coupling between RoPE design and model-specific factors that the current paper cannot detect.

Cheap difficulty estimation for adaptive allocation. While not directly a follow-up to the specific methods in this paper, the paper's framework of diagnostic decomposition naturally suggests a meta-learning problem: can a lightweight classifier predict which RoPE design is optimal for a given VLM architecture and training data distribution without running the full benchmark suite? The paper required approximately 512 GPU-hours per experiment to evaluate each RoPE variant, making exhaustive search over the design space (position design parameters, frequency allocation ratios, head allocation patterns) prohibitively expensive. A practical follow-up would train a small prediction model that takes as input (a) architectural specifications of the VLM, (b) a small probing evaluation on ~5 diverse benchmarks after brief training (~10% of full SFT data), and predicts which RoPE variant will achieve the best overall performance. The training data for this predictor would come from the paper's own evaluations plus any replications—at 7 methods × 3 model scales = ~21 data points, which is small but could be augmented with synthetic variations (e.g., training MRoPE with different channel allocation ratios and measuring the resulting performance). Success here would make the paper's insights actionable at scale, enabling VLM developers to select an optimal RoPE design in hours rather than weeks.

MHRoPE with learned head-to-axis assignment. The paper's MHRoPE uses a fixed, hand-specified partitioning of attention heads to positional axes (Section 2.3.3). A natural extension is to make this assignment learned: allow each attention head to learn a soft weighting over the three positional axes (t, h, w), with a regularization that encourages specialization. This would address MHRoPE's stated limitation—that "head-level information partitioning prevents the integration of different positional axes within the self-attention mechanism" (Appendix D.1)—by allowing heads to partially attend to multiple axes, with the relative importance determined by gradient descent rather than hard-coded. A concrete experiment: initialize Qwen2.5-VL-7B with MHRoPE's fixed head assignment, then add a learnable 3D vector per head that parameterizes a softmax over {t, h, w}, and train with the standard SFT mixture. The hypothesis is that some heads will learn to specialize (matching the fixed assignment), while others will learn to integrate multiple axes for tasks requiring joint spatio-temporal reasoning. The primary metric would be whether learned assignment improves over fixed assignment, particularly on video benchmarks where cross-axis integration matters most.

Stress-testing the diagonal layout failure across resolutions and modalities. The paper identifies VideoRoPE/HoPE's diagonal layout as causing catastrophic degradation on high-resolution document images due to positional overlap with generated text tokens (Section 3.4.1, Table 4). A rigorous follow-up would map the boundary of this failure mode by systematically varying (a) image resolution (from 224×224 to 4096×4096 or higher), (b) the number of interleaved images in the sequence, and (c) the length of generated text, while measuring both task accuracy and the frequency of repetitive generation failures. The paper's mechanistic explanation (Section 2.2.2) predicts that the failure should occur precisely when max(shifted spatial coordinates) > position_id_of_first_generated_token, and this prediction can be tested by computing the overlap threshold analytically and verifying it empirically. Such a study would convert the paper's qualitative observation ("diagonal layout causes modalities confusion") into a quantitative design constraint that future position design methods must satisfy.

Frequency allocation ratio as a function of task distribution. The paper's Table 8 sweeps t:h:w allocation ratios under interleaved allocation and finds 24:20:20 to be optimal for the overall benchmark suite. But this optimum likely depends on the task distribution: a VLM deployed primarily for video understanding might benefit from more temporal channels (e.g., 32:16:16), while one deployed for document QA might benefit from fewer temporal channels (e.g., 8:28:28). A follow-up study would train MRoPE-I models with different allocation ratios and evaluate them on individual benchmarks rather than aggregate averages, producing a matrix of [allocation ratio × benchmark] performance that could guide ratio selection based on a target deployment's expected query distribution. This would also test whether the optimal ratio is additive across tasks (i.e., the best ratio for a mixture of tasks can be predicted from the best ratios for each task individually) or if there are interaction effects that require joint optimization.

Combining MHRoPE and MRoPE-I into a hybrid architecture. The paper treats MHRoPE and MRoPE-I as alternatives, but they operate at different levels of the architecture—head partitioning vs. channel interleaving. A hybrid approach would apply MRoPE-I's interleaved channel allocation within each attention head while using MHRoPE's head partitioning to assign different axes to different heads. This could capture the benefits of both: full frequency resolution from interleaving (unlike MHRoPE, which currently applies the full spectrum to a single axis per head, the hybrid would give each head access to all axes at the channel level), and axis-specialized attention computation from head partitioning. The paper's stated limitation of MHRoPE—that heads cannot integrate multiple axes—would be partially addressed because each head would still see all three axes via interleaving, just with limited per-axis frequency resolution. A concrete experiment: take MRoPE-I as the base, but restrict each attention head to use only a subset of channels for each axis (e.g., head 1 uses channels 0–20 for temporal, 21–41 for height, 42–63 for width; head 2 uses a different assignment). This tests whether head-level specialization can extract additional signal beyond what uniform interleaving provides, or whether the paper's finding that MRoPE-I slightly outperforms MHRoPE indicates that uniform interleaving within every head is already sufficient.

Practical Applications and Downstream Use Cases

General-purpose VLM development with plug-and-play RoPE. For teams building VLMs by adapting pre-trained LLMs (the dominant paradigm), MRoPE-I offers a drop-in replacement for existing multimodal RoPE implementations that requires zero architectural changes and consistently improves performance across images, videos, and grounding (Table 2: +1.62 Image, +0.72 Video, +2.37 Grounding over vanilla RoPE). The implementation complexity is minimal—replacing the channel-to-axis mapping function to use interleaved assignment instead of contiguous chunks—and the training cost is identical (all methods in the paper used the same 512 GPU-hours). The paper's demonstration that spatial-reset further improves performance (+0.78 Image, +1.45 Grounding, +1.07 Video in Table 4) and can be implemented by resetting spatial coordinate counters per visual content also requires negligible engineering effort. For a team training a production VLM, adopting MRoPE-I + spatial-reset over vanilla RoPE or standard MRoPE represents one of the highest-ROI changes available—significant, consistent gains across benchmarks for near-zero implementation cost.

Diagnostic debugging of positional encoding failures in existing VLMs. The paper's decomposition of failure modes provides a practical diagnostic toolkit for VLM developers encountering mysterious performance degradations. If a VLM suddenly produces repetitive nonsense when processing high-resolution documents, the paper identifies diagonal-layout-style position overlap as a likely cause (Section 2.2.2, Table 4). If a VLM ignores visual input and generates fluent but irrelevant text, the paper identifies excessive modality interval as a likely cause (Section 3.4.1). If a VLM performs well on images but poorly on long videos, the paper identifies chunked frequency allocation forcing temporal into high frequencies as a likely cause (Section 2.3.2). Each failure mode has a specific, testable mechanism (check whether position IDs overlap; measure visual attention scores per layer; compute attention decay curves for temporal vs. spatial axes), enabling targeted fixes rather than blind hyperparameter sweeps. This diagnostic value may ultimately be more practically impactful than the specific MRoPE-I method itself.

Allocation ratio tuning based on deployment task distribution. Table 8 provides a concrete knob—the t:h:w channel allocation ratio under MRoPE-I's interleaved design—that can be tuned per deployment scenario without architectural changes. A VLM deployed for document understanding (where spatial precision matters most) could use a 12:26:26 ratio to allocate more channels to spatial axes, gaining ~1.5 points on grounding at the cost of ~0.4 points on video (interpolating from Table 8). A VLM deployed for video surveillance or video QA could use 32:16:16 to favor temporal modeling. The paper's finding that the 24:20:20 balanced ratio achieves the best overall performance provides a sensible default, but the ability to shift the ratio based on known deployment requirements—without retraining the vision encoder, the connector, or the LLM from scratch—offers a lightweight customization path that is rare in VLM development.

When to Prefer This Method

The paper explicitly positions MHRoPE and MRoPE-I against a landscape of named alternatives (MRoPE, VideoRoPE, HoPE, CircleRoPE, vanilla RoPE) and discusses trade-offs between its own two proposals (Appendix D.1). The decision rules that emerge from the paper's evidence are:

• Prefer MRoPE-I over MHRoPE when implementation simplicity is a priority, tensor parallelism is used in distributed training, or single-head integration of multiple positional axes is desired. MRoPE-I achieves slightly better overall performance (Table 2: 64.95 vs. 64.63 overall, computed from modality averages in Table 5) and avoids the head-partitioning complexity that interacts with tensor parallelism (Appendix D.1).

• Prefer MHRoPE over MRoPE-I when the model must scale to more than three positional axes in the future (e.g., adding depth, or additional modality-specific dimensions). MHRoPE's head-level partitioning can accommodate new axes by dedicating additional heads without sacrificing frequency resolution per axis, while MRoPE-I's channel-level interleaving would further subdivide an already-fixed channel budget (Section 2.3.3, Appendix D.1).

• Prefer either MHRoPE or MRoPE-I over VideoRoPE or HoPE when the VLM must handle high-resolution documents or interleaved image-text sequences. The diagonal layout's positional overlap causes catastrophic degradation on document benchmarks (Table 2: DocVQA 60.13/60.12 for VideoRoPE/HoPE vs. 83.72 for MRoPE-I) that is structural rather than trainable—it occurs even at inference time without retraining (Section 3.4.1).

• Prefer either proposed method over CircleRoPE when video understanding is required. CircleRoPE collapses all video frames onto a single spatial ring, eliminating temporal information and producing the lowest video average in Table 2 (51.09 vs. 52.36 for MRoPE-I and 52.95 for HoPE).

• Prefer either proposed method over standard MRoPE when long-video understanding or fine-grained grounding are priority tasks. MRoPE's chunked frequency allocation forces temporal into high frequencies (impairing long-range temporal attention: MLVU 63.26 vs. 65.46 for MRoPE-I) and gives spatial axes asymmetric, restricted frequency spectra (impairing grounding: RefCOCOval 78.35 vs. 80.94 for MRoPE-I).

• Prefer vanilla RoPE only when the VLM is deployed for tasks that require no spatial reasoning whatsoever and the 3D structure of visual content can be safely ignored. Even then, the paper shows vanilla RoPE loses to MRoPE-I on almost every benchmark including image understanding (62.17 vs. 63.79 average), making it a rarely-justified choice.

• Never modify text token positional encoding, scale rotary bases for spatial dimensions separately, or apply text spatial-reset unless the modification's benefit on a specific novel task has been demonstrated to outweigh the 6–8 point degradation across all existing benchmarks (Table 4). The paper's evidence indicates these modifications break pre-trained knowledge transfer in ways that limited multimodal SFT data cannot repair.