ArXiv: 2603.03825

🎯 Pitch

Text-only cold-start training surprisingly boosts multimodal reasoning better than training with actual images, because multimodal data paradoxically teaches models to ignore the visual input. The authors show you can fix this by explicitly training attention heads to look at images, yielding a 12% jump on MathVision.

1. Executive Summary

This paper analyzes the cold-start initialization stage in Multimodal Large Reasoning Models (MLRMs), introducing a metric called the Visual Attention Score (VAS) that quantifies how much a model attends to visual tokens relative to system tokens, and uncovers a counterintuitive phenomenon termed Lazy Attention Localization — where multimodal cold-start fails to increase visual attention while text-only cold-start paradoxically does, revealing that reasoning capability scales strongly with visual attention (Pearson r = 0.9616). Building on causal evidence from training-free attention modulation experiments that yield consistent 1–2% gains, the authors propose Attention-Guided Visual Anchoring and Reflection (AVAR) , a comprehensive cold-start framework that integrates three components: visual-anchored reflection data synthesis (embedding explicit image look-back references into reasoning chains), attention-guided training objectives (losses that amplify attention to visual tokens while suppressing redundant attention to system tokens), and visual-anchored reward shaping (an RL reward signal that incentivizes sustained visual grounding). Applied to Qwen2.5-VL-7B, AVAR achieves a 7.0% average gain across seven multimodal reasoning benchmarks, with the strongest improvements on MathVision (+12.2%) and HallusionBench (+8.8%), establishing that explicitly reshaping attention allocation during cold-start training decisively improves multimodal reasoning — while also demonstrating that text-only cold-start data can outperform multimodal cold-start data specifically because it preserves visual grounding patterns rather than from direct multimodal alignment.

2. Context and Motivation

The Core Problem: Multimodal Cold-Start Is Broken, and Nobody Knows Why

The paper addresses a specific, empirically observed paradox in training Multimodal Large Reasoning Models (MLRMs). The standard training pipeline for these models typically involves a two-stage process: first, a cold-start initialization stage where the base vision-language model is fine-tuned on reasoning data to establish initial reasoning patterns, followed by a reinforcement learning (RL) stage that optimizes these patterns further. The problem is that this cold-start stage behaves in a way that contradicts intuition:

"A text-only cold-start yields substantial improvements for MLRMs in subsequent RL tuning, whereas multimodal cold-start provides only marginal gains."

In plain language: if you take a base vision-language model like Qwen2.5-VL-7B and fine-tune it on reasoning data that includes images (multimodal cold-start), the improvement is disappointingly small. But if you fine-tune the same model on text-only reasoning data — stripping out the visual modality entirely — the model becomes substantially better at subsequent multimodal reasoning tasks. This is strange. Why would removing visual information during training make the model better at tasks that require looking at images?

This gap matters for several concrete reasons:

  • Resource inefficiency during training. Multimodal data is expensive to curate and requires significantly more GPU memory and compute to process than text-only data. If multimodal cold-start provides only marginal gains, organizations are wasting substantial resources on data preparation and training that could be redirected elsewhere. The paper explicitly frames this as "inefficient resource use."

  • Ceiling on RL effectiveness. The cold-start stage determines the initial policy that subsequent RL optimization builds upon. If cold-start does not establish proper visual grounding, the RL stage starts from a compromised position, limiting how much improvement is possible regardless of how much RL budget is spent. The paper argues this is "limiting the potential of RL for multimodal reasoning."

  • Lack of mechanistic understanding. Prior to this work, the phenomenon was observed but unexplained. Practitioners knew that text-only cold-start worked better, but they didn't know why, making it impossible to design better cold-start procedures — they could only replicate the accidental success.

  • Deployment implications. If models fail to develop strong visual grounding during training, they will underperform on any task requiring genuine visual understanding — math with diagrams, scientific figure interpretation, visual question answering with subtle perceptual cues — rather than tasks where language priors can compensate.

The Broader Context: The Rise (and Limits) of Multimodal Reasoning Models

The paper situates itself within a rapidly evolving landscape. Recent advances in reinforcement learning, exemplified by DeepSeek-R1 and OpenAI's o1, have demonstrated that LLMs can develop sophisticated reasoning capabilities — chain-of-thought, self-verification, error correction — when trained with appropriate reward signals. This success has naturally prompted researchers to extend the same paradigm to multimodal settings, producing models like R1-OneVision, OpenVLThinker, MM-Eureka, and Vision-R1.

These models follow a common recipe: take a pretrained vision-language model (like Qwen2.5-VL-7B or InternVL2.5), perform cold-start fine-tuning on curated reasoning data (either text-only or multimodal), then apply RL to optimize reasoning trajectories. The collective goal is to produce models that can look at an image — a geometry diagram, a scientific chart, a visual puzzle — and reason step-by-step to reach a correct answer, all while maintaining visual grounding (actually attending to what's in the image rather than relying on memorized patterns).

However, the paper identifies a critical blindspot in this research program: while substantial effort has gone into improving cold-start data curation and exploring RL algorithms, almost no work has investigated the underlying mechanisms of the cold-start stage itself. The paper states this explicitly:

"these works stop short of uncovering mechanisms behind this effect, and they have yet to investigate how multimodal reasoning data, particularly in 'reasoning-with-image' settings, should be optimized."

This is a different kind of gap than the typical "we need better performance" gap. It's a gap in understanding — the field has been optimizing cold-start procedures based on trial and error rather than on a principled theory of what cold-start needs to accomplish. The paper positions its contribution as filling this mechanistic gap: first diagnosing why text-only cold-start works better, then using that diagnosis to design a cold-start procedure that achieves the best of both worlds.

Where Prior Approaches Fall Short

The paper identifies two clusters of relevant prior work, each with specific limitations:

Multimodal Reasoning Model Development

Prior work on MLRMs has focused on two strategies for cold-start data:

Caption-then-reason pipelines. Models like R1-OneVision (Yang et al., 2025d) first generate textual descriptions of images, then build reasoning chains from those descriptions. The problem, as the paper later demonstrates, is that this approach doesn't force the model to maintain visual attention during the reasoning process itself — once the caption is generated, the model can ignore the image and reason entirely from text, which is precisely what the attention analysis shows happens. The paper's fine-grained attention analysis (Appendix D, Figures 4–5) reveals that R1-OneVision and ThinkLite-VL, trained on multimodal thinking data, "do not alter the attention distribution behavior of the base model" — they remain stuck in the base model's pattern of weak visual attention.

Multimodal reasoning data curation. Other approaches like OpenVLThinker, Vision-R1, and Vision-SR1 create reasoning data that includes images, but they don't explicitly ensure that the reasoning chain contains references back to the image. The paper's ablation in Table 3 shows that simply using these datasets for cold-start can actually reduce performance relative to the base model:

"some datasets such as R1-OneVision even reduce performance relative to the baseline (-4.7%), indicating that simply scaling reasoning data is insufficient and can be harmful."

This is a key finding: more multimodal data is not automatically better. If the data doesn't enforce visual grounding, it can reinforce lazy attention patterns, teaching the model that it's acceptable to reason without genuinely looking at the image.

Visual Attention Analysis in MLLMs

A separate research thread has studied how multimodal language models allocate attention across token types. The paper builds on several specific findings:

Modality fusion occurs in middle layers, but with insufficient visual signal. Yin et al. (2025) demonstrated that while vision-language models do integrate visual and textual information, they consistently devote too little attention to visual tokens and over-rely on language priors — learned textual patterns that can answer questions without actually processing the image.

Attention is uneven across heads. Tang et al. (2025) showed that certain attention heads are disproportionately dominated by language priors, with some heads almost entirely ignoring visual tokens. This means the problem isn't just "not enough visual attention overall" but rather that attention is unevenly distributed, with some components of the model effectively blind to images.

Reasoning-oriented MLLMs pay less attention to images, not more. Liu et al. (2025) made the seemingly paradoxical observation that models fine-tuned for reasoning actually allocate less attention to visual tokens than their non-reasoning counterparts. As the reasoning chain gets longer, visual attention degrades further, amplifying hallucinations — the model starts generating content that isn't grounded in the image.

Proposed solutions are inference-time only. Prior work has proposed interventions to reweight attention toward visual tokens, but these are all inference-time modifications — applying masks or scaling factors to attention weights during generation (ClearSight by Yin et al., 2025; adaptive attention calibration by Fazli et al., 2025; dual-level attention intervention by Tang et al., 2025). While these approaches validate that attention allocation matters, they don't address the root cause during training. The model is still fundamentally trained with weak visual attention patterns; inference-time patches are band-aids.

The critical gap this paper identifies is that no prior work has connected these two threads: the failure of multimodal cold-start and the attention allocation patterns of MLLMs. Researchers studying MLRMs weren't looking at attention distributions, and researchers studying visual attention weren't analyzing cold-start training dynamics. The paper bridges this gap by asking: what if cold-start fails specifically because it doesn't reshape attention allocation, and what if the reason text-only cold-start works is that it inadvertently does?

How This Paper Positions Itself

The paper positions itself at the intersection of the two research threads described above, making a move that is simultaneously diagnostic and prescriptive:

Diagnostic contribution: Lazy Attention Localization. The paper introduces VAS as a lens for understanding cold-start behavior and uses it to explain the text-only vs. multimodal cold-start paradox. This is not framed as a new method but as a new understanding — a mechanistic explanation for a phenomenon the field had observed but couldn't explain. The paper explicitly connects VAS to reasoning performance via the 0.9616 Pearson correlation (Figure 1a), establishing that visual attention is not just incidentally correlated with performance but is plausibly causal.

Causal validation: Training-free attention modulation. Before proposing a training framework, the paper runs pilot experiments that directly manipulate attention at inference time (Section 4). This is important positioning: it transforms the correlation into causal evidence. If amplifying visual attention at inference time improves performance without any retraining, then attention allocation is not merely a symptom of good models but a mechanism that can be exploited. The 1–2% consistent gains across Qwen2.5-VL-7B, Revisual-R1-CS, and OVR-CS validate this causal claim.

Prescriptive contribution: AVAR as principled cold-start. With the mechanism established, AVAR is positioned not as an arbitrary combination of techniques but as a framework designed to directly counteract the identified bottleneck — Lazy Attention Localization. Each component targets a specific aspect of the problem:

  • Visual-anchored reflection data synthesis ensures that training data itself models visual grounding behavior, unlike caption-then-reason approaches that allow attention to drift.
  • Attention-guided training objectives provide explicit optimization pressure toward visual attention, unlike standard supervised fine-tuning which only optimizes token prediction.
  • Visual-anchored reward shaping prevents RL from reverting to text-only reasoning patterns, addressing the "more thinking, less seeing" degradation that Liu et al. (2025) documented.

The paper's positioning is ultimately constructive rather than critical: it doesn't argue that prior approaches are wrong, but rather that they are incomplete — optimizing the wrong things (data quantity, RL algorithms) while neglecting the fundamental mechanism of attention allocation. The metaphor in the title — "From Narrow to Panoramic Vision" — captures this framing: the goal is to transform models from narrow-view (VAS < 10, weak visual grounding) to panoramic-view (VAS > 15, sustained visual attention) by explicitly engineering attention patterns during cold-start training, rather than hoping they emerge accidentally from data curation choices.

3. Technical Approach

3.1 Reader Orientation

This paper builds a cold-start training framework for multimodal reasoning models that explicitly reshapes where the model directs its attention during training. The core problem is that standard multimodal cold-start training fails to teach models to actually look at images during reasoning — instead, they develop "lazy" attention patterns where they over-rely on system tokens and language priors while barely attending to visual information. The solution is AVAR, a three-component framework that intervenes at every stage of the cold-start pipeline — data synthesis, supervised fine-tuning, and reinforcement learning — to redirect attention from redundant system tokens toward visual tokens, transforming the model from a narrow-view reasoner that mostly ignores images into a panoramic-view reasoner that sustains visual grounding throughout extended reasoning chains.

3.2 Big-Picture Architecture (Diagram in Words)

The AVAR framework consists of five major components that operate sequentially through the training pipeline:

  1. A three-model data synthesis pipeline — Gemini 2.5-Pro generates high-fidelity visual descriptions, Qwen3-235B-A22B produces reflection-enhanced reasoning chains over those descriptions, and Qwen3-32B inserts explicit visual anchor references (e.g., "look back at the triangle") into the reasoning chains, producing training data with built-in visual grounding.

  2. An attention-guided training objective — during cold-start supervised fine-tuning, the standard language modeling loss is augmented with two additional loss terms: one that encourages the model to allocate more attention to visual tokens ($\mathcal{L}_{\text{enhance-img}}$), and one that penalizes excessive attention to system tokens ($\mathcal{L}_{\text{suppress-sys}}$). Together, these reshape the model's attention distribution toward stronger visual grounding.

  3. A visual-anchored reward shaping mechanism — during the subsequent RL stage (using GRPO), the correctness reward is augmented with a visual attention reward that incentivizes the model to sustain high visual-to-system token attention ratios throughout its reasoning chains. This prevents the model from reverting to text-only reasoning patterns during RL optimization.

  4. The base model (Qwen2.5-VL-7B) — the pretrained vision-language model that serves as the starting point. All training interventions operate on this model.

  5. A visual attention monitoring infrastructure (VAS) — not a component of the training pipeline itself, but the analytical tool used throughout the paper to quantify attention allocation patterns and validate that each training stage is reshaping attention as intended.

Information flows as follows: raw multimodal data → three-stage synthesis pipeline producing visual-anchored reasoning data → cold-start SFT with attention-guided losses → RL fine-tuning with visual-anchored rewards → final AVAR-Thinker model. At each stage, VAS is computed to track how attention allocation evolves from the baseline model's lazy patterns to the final model's panoramic-view patterns.

3.3 Roadmap for the Deep Dive

  • First, the Visual Attention Score (VAS) — how it is defined, computed, and what it operationalizes — since every subsequent component either diagnoses or modifies what VAS measures.
  • Second, the training-free attention modulation experiments — since they establish the causal link between attention allocation and reasoning performance, providing the motivation for building attention-guiding mechanisms into training rather than just applying inference-time patches.
  • Third, the visual-anchored reflection data synthesis pipeline — since the training data is the foundation that everything else builds on, and the design choices here determine what patterns the model learns during SFT.
  • Fourth, the attention-guided training objectives — the loss functions added during cold-start SFT that provide explicit optimization pressure toward the attention patterns observed in panoramic-view models.
  • Fifth, the visual-anchored reward shaping for RL — since the RL stage has its own dynamics (models can revert to text-only reasoning), requiring a separate mechanism to sustain visual grounding.
  • Sixth, the complete training configuration — the hyperparameters, datasets, hardware, and training schedule that tie all components together into the final AVAR-Thinker model.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methods paper with analytical grounding, whose core idea is that the cold-start stage in MLRM training should be explicitly designed to reshape attention allocation toward visual tokens, and that this can be achieved through coordinated interventions at the data, supervised learning, and reinforcement learning levels.

The Visual Attention Score (VAS): Quantifying Visual Grounding

The paper introduces the Visual Attention Score as the central analytical instrument for the entire study. VAS is not just a metric — it is the operational definition of "how much the model is looking at the image" that the paper uses to diagnose the cold-start paradox, validate the training-free interventions, and track the evolution of attention patterns throughout AVAR training.

The VAS is defined at the level of individual attention heads, then aggregated across the model. For a specific layer $l$, head $h$, and query token $i$ (where $i$ belongs to the set of user tokens $U$ — the tokens that make up the user's question), the per-head VAS is:

VASi(l,h)=∑j∈VAi,j(l,h)∑j∈SAi,j(l,h)\text{VAS}_{i}(l,h) = \frac{\sum_{j \in V} A_{i,j}(l,h)}{\sum_{j \in S} A_{i,j}(l,h)}

where $A(l,h) \in \mathbb{R}^{T \times T}$ is the attention matrix at layer $l$ and head $h$, $T$ is the total number of tokens in the sequence, $V$ is the index set of visual tokens (the image patch embeddings that the vision encoder produces and that are fed into the language model), $S$ is the index set of system tokens (the system prompt and instruction tokens that frame the task), and $A_{i,j}(l,h)$ is the attention weight that query token $i$ assigns to key token $j$ at that layer and head.

What it computes: For a single query token in the user's question, the per-head VAS is the ratio of the total attention that token pays to all visual tokens divided by the total attention it pays to all system tokens. If this ratio is large (e.g., 15–20), the token is attending much more strongly to image features than to system instructions — it's "looking at the image." If the ratio is small (e.g., below 5), the token's attention is dominated by system tokens — it's "reading the instructions" rather than processing visual information.

Why this form: The ratio formulation is deliberate. An alternative metric — say, the raw sum of attention to visual tokens — would be confounded by sequence length: longer system prompts would dilute all attention values, making cross-example comparisons meaningless. By normalizing against system token attention, the VAS controls for sequence length effects and isolates the relative preference for visual versus instructional information. The choice of system tokens specifically (rather than all non-visual tokens) is also intentional: prior work had shown that redundant attention to system prompts is a particular problem in MLLMs, and the paper's training-free experiments (Section 4) demonstrate that suppressing system token attention specifically yields gains, validating that system tokens are the right normalization target.

The model-level VAS is obtained by averaging over all query tokens, all heads, and all layers:

VAS=1L⋅H⋅∣U∣∑l=1L∑h=1H∑i∈UVASi(l,h)\text{VAS} = \frac{1}{L \cdot H \cdot |U|} \sum_{l=1}^{L} \sum_{h=1}^{H} \sum_{i \in U} \text{VAS}_{i}(l,h)

where $L$ is the number of transformer layers and $H$ is the number of attention heads per layer.

What it computes: A single scalar that summarizes how much the model, on average across its entire architecture and across all user tokens in the input, favors attending to visual features over system instructions. For Qwen2.5-VL-7B, this value is approximately 7.5 (Table 4 in Section 6.4). For the final AVAR-Thinker, it rises to 18.9.

Why this form: The uniform average over layers and heads treats all components equally, which is appropriate for a summary statistic but deliberately coarse. The paper complements this with layer-wise VAS breakdowns (Figure 1b, Appendix D Figures 4–6) that reveal which layers are most affected by cold-start choices, providing a more nuanced picture than the aggregate VAS alone. The averaging over user tokens specifically (not all tokens) ensures the metric measures attention during the reasoning process — when the model is generating its answer — rather than during prompt encoding.

Computational procedure in practice. For the cross-model analysis in Section 3.1, the authors sample 200 cases from the MathVista benchmark for each model under evaluation, run each model on these cases, extract the attention matrices from every layer and head during the forward pass, compute per-head VAS for every user query token, and average to obtain the model-level VAS. They then evaluate each model's reasoning performance on four multimodal benchmarks (MathVista, MathVision, MathVerse-Vision-Only, DynaMath-WORSE) and compute the Pearson correlation between average benchmark performance and VAS. The resulting correlation of 0.9616 (Figure 1a) is the paper's central empirical finding: VAS is not just loosely associated with reasoning ability — it is almost perfectly linearly correlated across a diverse set of 7B-parameter models trained with different cold-start strategies.

The Narrow-Wide-Panoramic taxonomy. Based on the VAS values, the paper groups models into three qualitative categories:

  • Narrow-View Models (VAS < 10): These include Qwen2.5-VL-7B-Instruct (7.5), R1-OneVision, ThinkLite-VL, and MM-Eureka. They "devote minimal attention to visual features and consistently underperform in reasoning tasks."
  • Wide-View Models (10 ≤ VAS ≤ 15): These include the Revisual-R1 variants. They display "a more balanced distribution between textual and visual modalities and achieve moderate improvements."
  • Panoramic-View Models (VAS > 15): These include OVR-RL, OVR-CS, MiMo-VL-CS, and MiMo-VL-RL. They "sustain strong visual grounding and superior results across benchmarks."

This taxonomy is not just descriptive — it structures the paper's entire narrative. The goal of AVAR is to transform a narrow-view model (Qwen2.5-VL-7B, VAS = 7.5) into a panoramic-view model (AVAR-Thinker, VAS = 18.9).

Relationship to Lazy Attention Localization. The VAS analysis directly explains the cold-start paradox that motivated the paper. When the base model Qwen2.5-VL-7B undergoes multimodal cold-start (as in R1-OneVision), its VAS remains nearly unchanged — Figure 1b shows "nearly identical attention distributions, with persistently weak reliance on visual tokens across all layers." But when it undergoes text-only cold-start (as in OVR-CS), its VAS increases substantially. The paper terms this Lazy Attention Localization: multimodal cold-start doesn't increase visual attention, but text-only cold-start somehow does. This is the phenomenon that AVAR is designed to overcome — by explicitly engineering attention patterns during training rather than hoping they emerge from data characteristics.

Training-Free Attention Modulation: Establishing Causality

Before proposing a training framework, the paper conducts a critical pilot experiment: if VAS is causally related to reasoning performance (not just correlated), then directly manipulating attention weights at inference time — with no retraining — should produce measurable performance changes. Section 4 presents exactly this experiment, and it does so before introducing AVAR to establish that the subsequent training framework is targeting a causal mechanism, not just a statistical pattern.

The modulation procedure. The intervention operates directly on the attention weight computation during the scaled dot-product attention in every transformer layer. For layer $l$ and head $h$, the hidden state $Z_{l,h}$ (the output of the attention mechanism before the output projection) is modified through element-wise operations with attention masks:

Z^l,h=Zl,h+αimg⋅Ml,henh⊙Zl,h−αsys⋅Ml,hsup⊙Zl,h\hat{Z}_{l,h} = Z_{l,h} + \alpha_{img} \cdot M_{l,h}^{enh} \odot Z_{l,h} - \alpha_{sys} \cdot M_{l,h}^{sup} \odot Z_{l,h}

where $M_{l,h}^{enh}$ is a binary mask that is 1 for visual token positions and 0 elsewhere, $M_{l,h}^{sup}$ is a binary mask that is 1 for system token positions and 0 elsewhere, $\odot$ denotes element-wise (Hadamard) product, and $\alpha_{img}$ and $\alpha_{sys}$ are scalar coefficients that control the strength of the modulation.

What it computes: For visual tokens, the modulation adds $\alpha_{img}$ times the original hidden state — effectively scaling up the contribution of visual features to the attention output by a factor of $(1 + \alpha_{img})$. For system tokens, it subtracts $\alpha_{sys}$ times the original hidden state — scaling down the system token contribution by a factor of $(1 - \alpha_{sys})$. Since these operations are applied to the attention output $Z_{l,h}$, they affect how strongly each token type influences the subsequent layers of the transformer.

Why this form: The additive and subtractive formulation (rather than multiplicative scaling) allows the modulation to be selectively applied per token type while leaving other token types (user tokens, generated tokens) unchanged. The use of binary masks means the intervention is applied uniformly across all tokens of a given type — there's no per-token selection, which would require additional computation or heuristics. This simplicity makes the intervention training-free: it requires only access to the attention weights during the forward pass, which any transformer implementation exposes.

Experimental configuration. The paper evaluates three models (Qwen2.5-VL-7B, Revisual-R1-CS, OVR-CS) across three multimodal reasoning benchmarks (MathVista, MathVision, MathVerse-VO). The image enhancement coefficient is fixed at $\alpha_{img} = 0.15$, while the system suppression coefficient is varied across $\alpha_{sys} \in \{0.00, 0.05, 0.40, 0.60\}$.

The System Token Redundancy Zone. The results (Figure 2) reveal a non-monotonic pattern: when $\alpha_{sys} \in \{0.00, 0.40\}$, performance consistently improves by 1–2% across all three models and benchmarks. But when $\alpha_{sys} = 0.60$, the gains diminish or reverse. The paper interprets this as evidence for a System Token Redundancy Zone: there is a range of system token attention that is genuinely redundant — suppressing it and redirecting that attention capacity to other tokens (via the mechanics of softmax normalization) improves performance. But suppressing too aggressively starts to remove useful information from the system prompt, degrading performance.

Why this experiment establishes causality. Prior work had shown correlation between visual attention and performance. But correlation could run in either direction: maybe good models happen to attend to images because they've learned good reasoning strategies, not vice versa. The training-free modulation experiment breaks this ambiguity because the manipulation is exogenous — it's applied by the experimenter, not learned by the model. If performance improves when visual attention is artificially boosted and system attention artificially suppressed, then attention allocation is causally upstream of reasoning quality, not just a symptom of it. This is the empirical justification for building attention-guiding mechanisms into the training process: if you can achieve gains by manipulating attention at inference time, you can achieve even larger gains by teaching the model to allocate attention optimally during training.

Visual-Anchored Reflection Data Synthesis

The first component of AVAR addresses the data level. The paper's diagnosis of Lazy Attention Localization implies that standard multimodal cold-start data — which typically uses caption-then-reason pipelines — fails because the reasoning chains don't force the model to maintain visual attention. Once the image is captioned, the model can ignore it and reason from text alone. AVAR's data synthesis pipeline is designed to produce training data where visual grounding is baked into the reasoning process itself, making it impossible to produce correct reasoning without repeatedly referencing the image.

The pipeline coordinates three separate models in sequence, each with a specialized role:

Stage 1: High-Fidelity Visual Descriptions Generation. The first model, Gemini 2.5-Pro, receives the original image and a specialized prompt (detailed in Appendix G) that asks it to produce a comprehensive visual description. The paper chooses Gemini 2.5-Pro specifically for its "superior perceptual capabilities, which enable the accurate generation of visual element priors" — in other words, it can reliably describe fine-grained visual details that smaller models might miss or hallucinate. Unlike the self-descriptions that a 7B model might generate (which would be limited by its own perceptual weaknesses), Gemini's descriptions provide a high-quality textual representation of the image that serves as the foundation for subsequent reasoning.

The output is a textual description that covers all visually relevant elements: for a geometry problem, this would include the shapes present, their spatial relationships, labeled points, angle measures, and any other visual features that the reasoning process will need to reference. This description is more detailed than what a typical MLLM would produce from a brief captioning prompt, providing "richer scene understanding" that establishes accurate visual priors.

Stage 2: Reflection-Enhanced Reasoning Generation. The second model, Qwen3-235B-A22B (a large language model, not a vision-language model), receives the visual description from Stage 1 along with the original question text and a prompt that instructs it to produce a step-by-step reasoning chain with built-in self-reflection and error checking. The prompt is designed to elicit "iterative self-reflection and error checking, which naturally leads it to leverage the visual context during multi-step reasoning."

This is a crucial design choice: Qwen3-235B-A22B is a text-only model — it cannot see the original image. It reasons entirely from the visual description produced in Stage 1. This might seem like a weakness, but it's actually central to the data synthesis strategy. Because the model can only access visual information through the textual description, every reference to visual content in its reasoning chain must be explicitly grounded in that description. There is no lazy option of "just looking at the image" — the model must verbalize what it's attending to, creating reasoning chains where visual grounding is explicit in the text.

The resulting reasoning chains contain both the logical steps toward the answer and self-reflection statements where the model checks its own work. These reflection patterns are what the paper aims to transfer to the 7B vision-language model during cold-start training — the goal is for AVAR-Thinker to develop the habit of looking back at the image to verify its reasoning, just as the larger model does with the textual description.

Why use a 235B model for this stage: The paper leverages the strong reasoning capabilities of Qwen3-235B-A22B to generate high-quality, complex reasoning chains that smaller models cannot reliably produce. The reasoning quality of the training data is critical — if the cold-start data contains flawed reasoning, the 7B model will learn flawed patterns. By using a much larger model as the "teacher," AVAR distills sophisticated reasoning behaviors into the smaller model.

Stage 3: Visual Anchor Integration. The third model, Qwen3-32B, takes the reasoning chains from Stage 2 and augments them by inserting explicit visual anchor references. The paper describes this as inserting "references such as 'look back at the triangle' or 'check the image again', simulating direct image perception."

This stage addresses a subtle alignment problem: the reasoning chain from Stage 2 was generated from a textual description, so its references are to the description text. But during cold-start training on the 7B vision-language model, the model will have access to the actual image, not the description. The visual anchor integration rewrites the reasoning chain so that references that were implicit become explicit — "the triangle has base length 5" becomes "look back at the triangle in the image — it has base length 5."

The paper notes that Qwen3-32B is sufficient for this task because "the task of visual anchor rewriting is relatively straightforward" — it's essentially a text editing task that requires understanding the reasoning chain well enough to know where visual references are implicitly assumed and inserting explicit "look back" markers. A smaller dense model achieves "sufficient accuracy after manual verification, making it suitable and efficient for this specific task."

Why three separate models instead of a single multimodal model: The paper's design decomposes the data generation problem into three sub-problems — perception (what's in the image), reasoning (how to solve the problem), and grounding (how to connect reasoning steps back to the image) — and assigns each to the model best suited for it. Gemini 2.5-Pro excels at detailed visual perception, Qwen3-235B-A22B excels at complex multi-step reasoning, and Qwen3-32B is sufficient for the relatively simple text editing task of inserting visual anchors. Using a single model for all three would compromise on at least one dimension — either the perceptual quality, the reasoning depth, or the grounding explicitness.

What the final data looks like. The output is a set of training examples where each example consists of an image, a question, and a multi-step reasoning chain that explicitly references the image at key points. A typical reasoning step might read:

"Looking at the diagram, I can see that triangle ABC is a right triangle with the right angle at B. The hypotenuse AC is labeled as 10 units. According to the Pythagorean theorem..."

The "looking at the diagram" and "I can see" phrases are the visual anchors — they create a textual signal that the reasoning step requires visual attention. During cold-start training, these textual patterns teach the model to associate reasoning steps with visual consultation, creating the attention patterns that AVAR aims to establish.

Comparison to caption-then-reason approaches. In R1-OneVision's pipeline, the model first generates a caption, then reasons from that caption — and at inference time, there's no caption generation step, so the model must reason directly from the image. The mismatch between training (reason from caption) and inference (reason from image) creates the conditions for lazy attention: during training, the model never learns to consult the image during reasoning because the caption provides all needed information. AVAR's approach avoids this by making visual references explicit in the training data, so the model learns to "look back" at the image even when reasoning from visual descriptions — a skill that transfers to inference with actual images.

Attention-Guided Training Objectives

The second component of AVAR addresses the supervised fine-tuning stage of cold-start. Standard cold-start SFT trains the model with a language modeling loss — the model learns to predict the next token in the reasoning chain, and the optimization signal comes entirely from token-level accuracy. There is no direct pressure on how the model allocates attention to achieve that accuracy; the attention patterns that emerge are a byproduct of the data and the model architecture.

AVAR augments the standard language modeling loss with two additional attention-based loss terms that provide explicit optimization pressure toward the attention patterns observed in panoramic-view models. The total loss is:

Ltotal=LLM+α⋅Lenhance-img+β⋅Lsuppress-sys\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{LM}} + \alpha \cdot \mathcal{L}_{\text{enhance-img}} + \beta \cdot \mathcal{L}_{\text{suppress-sys}}

where $\mathcal{L}_{\text{LM}}$ is the standard autoregressive language modeling loss (cross-entropy between predicted and actual next tokens), $\mathcal{L}_{\text{enhance-img}}$ is a loss term that encourages attention to visual tokens, $\mathcal{L}_{\text{suppress-sys}}$ is a loss term that penalizes attention to system tokens, and $\alpha$ and $\beta$ are scalar coefficients controlling the relative weight of each auxiliary loss.

The paper sets $\alpha = 0.15$ and $\beta = 0.15$, giving equal weight to both attention guidance components. These values are relatively small compared to the language modeling loss (implicitly weighted at 1.0), indicating that attention guidance is a gentle nudge rather than a dominant training signal — the model is still primarily optimized for token prediction accuracy, with attention patterns shaped as a secondary objective.

The image enhancement loss is designed to increase the attention that query tokens pay to visual token keys. It is defined as:

Lenhance-img=−1∣L∣∑l∈L1H∑h=1Hlog⁡(1∣Q∣⋅∣Kimg∣∑q∈Q∑k∈KimgAq,kl,h)\mathcal{L}_{\text{enhance-img}} = -\frac{1}{|\mathcal{L}|} \sum_{l \in \mathcal{L}} \frac{1}{H} \sum_{h=1}^{H} \log\left(\frac{1}{|\mathcal{Q}| \cdot |\mathcal{K}_{\text{img}}|} \sum_{q \in \mathcal{Q}} \sum_{k \in \mathcal{K}_{\text{img}}} A_{q,k}^{l,h}\right)

where $\mathcal{L}$ is the set of targeted layers (in the paper's configuration, all layers are targeted), $H$ is the number of attention heads, $\mathcal{Q}$ is the set of query tokens, $\mathcal{K}_{\text{img}}$ is the set of image token keys (the image patch embeddings that the model can attend to), and $A_{q,k}^{l,h}$ is the attention weight from query $q$ to key $k$ at layer $l$ and head $h$.

What it computes, step by step:

  1. For each layer and head, compute the average attention weight from all query tokens to all image key tokens — this is the inner double sum divided by $|\mathcal{Q}| \cdot |\mathcal{K}_{\text{img}}|$, giving a single scalar representing the average per-token, per-image-key attention.
  2. Take the natural logarithm of this average — this transforms the value into log-space, which amplifies the gradient signal when the average attention is very small (precisely the regime where lazy attention models operate).
  3. Average this log value across all heads in the layer, then across all layers.
  4. Negate the result — because the loss is minimized, negating the log makes the model increase the attention to image tokens.

Why this form: The logarithm is the critical design choice. A linear penalty on low attention would provide a weak gradient when attention is near zero — the model would receive almost no signal to increase attention from very low levels. The log transform amplifies the gradient for small values: as the average attention approaches zero, $-\log(\text{avg\_attn})$ approaches infinity, creating a strong optimization pressure to move away from near-zero attention. This is precisely what's needed for narrow-view models (VAS < 10) that start with extremely weak visual attention. The double averaging (over queries and image keys, then heads, then layers) ensures the loss is a single scalar that is comparable across different sequence lengths and model configurations.

The system suppression loss is the complement — it penalizes the model for allocating too much attention to system tokens:

Lsuppress-sys=1∣L∣∑l∈L1H∑h=1Hlog⁡(1∣Q∣⋅∣Ksys∣∑q∈Q∑k∈KsysAq,kl,h+ϵ)\mathcal{L}_{\text{suppress-sys}} = \frac{1}{|\mathcal{L}|} \sum_{l \in \mathcal{L}} \frac{1}{H} \sum_{h=1}^{H} \log\left(\frac{1}{|\mathcal{Q}| \cdot |\mathcal{K}_{\text{sys}}|} \sum_{q \in \mathcal{Q}} \sum_{k \in \mathcal{K}_{\text{sys}}} A_{q,k}^{l,h} + \epsilon\right)

where $\mathcal{K}_{\text{sys}}$ is the set of system token keys, and $\epsilon = 10^{-6}$ is a small constant added for numerical stability to prevent the logarithm from being evaluated at exactly zero.

What it computes: The same averaging and log-transform structure as the enhancement loss, but applied to system tokens rather than image tokens, and without the negation — since the loss is minimized, the positive log term encourages the average attention to system tokens to decrease. The $\epsilon$ term prevents $\log(0)$ from producing negative infinity (which would make the loss undefined) when the average system attention is extremely small.

Why this form: The choice to penalize system token attention specifically (rather than all non-visual tokens) is based on the paper's empirical findings from the training-free experiments. Section 4 showed that suppressing system token attention produced consistent gains, identifying system tokens as the primary locus of redundant attention. The paper also notes in Section 6.3 that the enhancement and suppression losses "work synergistically to reshape attention distributions" — the enhancement loss pulls attention toward visual tokens, creating competition for the fixed total attention budget, and the suppression loss simultaneously pushes attention away from system tokens, freeing up capacity. Without the suppression loss, the enhancement loss alone might increase visual attention by pulling from other useful token types (user tokens, reasoning tokens), potentially degrading performance.

Targeted layers. The paper applies both attention-guided losses across all layers of the transformer. This is a deliberate choice: the layer-wise VAS analysis in Figure 1b shows that lazy attention models exhibit weak visual attention across all layers uniformly, not just in specific layers. Targeting all layers ensures the attention reshaping is comprehensive.

Interaction with the standard LM loss. The standard language modeling loss $\mathcal{L}_{\text{LM}}$ remains the dominant training signal — the model is still primarily learning to predict tokens correctly. The attention-guided losses act as regularizers that shape how the model achieves good token prediction. This is important because it prevents the attention losses from degrading language modeling quality: the model learns to allocate attention optimally subject to the constraint that it must still predict tokens accurately, rather than blindly maximizing visual attention at the expense of coherent text generation.

Visual-Anchored Reward Shaping

The third component of AVAR addresses the reinforcement learning stage that follows cold-start SFT. After the model has been fine-tuned on visual-anchored data with attention-guided objectives, it enters an RL stage where it generates reasoning trajectories and receives rewards based on answer correctness. The problem is that RL optimization, if guided only by correctness rewards, can cause the model to revert to text-only reasoning patterns — spending the additional reasoning budget (more tokens, more steps) on linguistic elaboration rather than visual re-examination. This is the "more thinking, less seeing" phenomenon documented by Liu et al. (2025).

To prevent this, AVAR introduces a visual attention reward that provides an auxiliary signal proportional to how much the model sustains visual attention during its reasoning trajectory. This reward is only awarded when the model's final answer is correct — incorrect trajectories receive zero visual reward regardless of their attention patterns, ensuring that the model doesn't learn to "game" the visual reward by attending to images without actually solving problems.

The visual reward for a single rollout is:

rvisual={0if rollout outcome is incorrect1∣T∣∑t∈T(1∣L∣∑l∈L∑k∈KimgAt,kl∑k∈KsysAt,kl+ϵ)if rollout outcome is correctr_{\text{visual}} = \begin{cases} 0 & \text{if rollout outcome is incorrect} \\ \frac{1}{|\mathcal{T}|} \sum_{t \in \mathcal{T}} \left( \frac{1}{|\mathcal{L}|} \sum_{l \in \mathcal{L}} \frac{\sum_{k \in \mathcal{K}_{\text{img}}} A_{t,k}^{l}}{\sum_{k \in \mathcal{K}_{\text{sys}}} A_{t,k}^{l} + \epsilon} \right) & \text{if rollout outcome is correct} \end{cases}

where $\mathcal{T}$ is the set of all generated tokens in the reasoning trajectory (the model's output, not the input), $\mathcal{L}$ is the set of all transformer layers, $\mathcal{K}_{\text{img}}$ and $\mathcal{K}_{\text{sys}}$ are the image and system token key sets respectively, $A_{t,k}^{l}$ is the attention weight from generated token $t$ to key $k$ at layer $l$, and $\epsilon = 10^{-6}$ is the stability constant.

What it computes, step by step:

  1. For each layer, compute the ratio of total attention from a given generated token $t$ to all image tokens divided by the total attention from that token to all system tokens. This is essentially a per-token VAS computed over attention weights rather than the hidden-state-level VAS from Section 3.1 — it measures the same concept (visual vs. system attention) but at the granularity of individual generated tokens.
  2. Average this ratio across all layers to get a single per-token visual attention score.
  3. Average across all generated tokens $t \in \mathcal{T}$ in the reasoning trajectory to get a trajectory-level score.
  4. Multiply by an indicator that is 1 if the final answer is correct and 0 otherwise, so the reward is zero for incorrect trajectories.

Why this form — the correctness gate: The binary gate on correctness is critical because, without it, the model could maximize the visual reward by generating long chains of "looking at the image" statements without actually solving the problem. The gate ensures that visual attention is only rewarded when it contributes to problem-solving, creating a joint optimization: to maximize expected reward, the model must both attend to images and produce correct answers. This is a form of reward shaping where the auxiliary reward guides the model toward desired behaviors (visual grounding) without distorting the primary objective (correctness).

Why this form — the ratio structure: The ratio of image-to-system attention mirrors the VAS definition from Section 3.1, maintaining consistency with the paper's analytical framework. It penalizes trajectories where the model's generated tokens attend heavily to system instructions (which don't change during generation) while rewarding trajectories where generated tokens consistently look back at the image. However, unlike the training objectives in Section 5.2 which use log transforms, the reward uses the raw ratio — because rewards in RL don't need amplified gradients the way loss functions do, and the raw ratio is more interpretable as a reward signal (a VAS of 15 is intuitively better than 10).

The total reward. The visual reward is combined with two other reward components:

rtotal=raccuracy+λv⋅rvisual+λf⋅rformatr_{\text{total}} = r_{\text{accuracy}} + \lambda_v \cdot r_{\text{visual}} + \lambda_f \cdot r_{\text{format}}

where $r_{\text{accuracy}}$ is a binary reward for answer correctness (1 if correct, 0 otherwise), $r_{\text{format}}$ is a reward for complying with the required output structure (e.g., putting the final answer in a specific format), $\lambda_v = 0.3$ weights the visual reward, and $\lambda_f = 0.1$ weights the format reward.

The weighting scheme reveals the relative importance: accuracy is the dominant signal (implicit weight 1.0), visual attention provides a moderate supplementary signal (0.3), and formatting is a minor constraint (0.1). The visual reward weight of 0.3 means that, for correct answers, the total reward can range from 1.0 (accuracy alone) to approximately 1.0 + 0.3 × (typical VAS ratio) ≈ 1.0 + 0.3 × 0.4 ≈ 1.12 for a model with very high visual attention. This is a meaningful but not overwhelming difference — enough to shape behavior without making the visual reward dominate over correctness.

RL algorithm: GRPO. The paper uses Group Relative Policy Optimization (GRPO), a variant of policy gradient methods introduced by Shao et al. (2024) in the DeepSeekMath paper. GRPO stabilizes training by comparing the relative performance of trajectories within groups rather than using absolute reward values or learned value functions.

The procedure for each update step:

  1. Sample a batch of $N$ trajectories $\{\tau_i\}_{i=1}^{N}$ from the current policy $\pi_{\theta_{\text{old}}}$.
  2. Compute the total shaped reward $r_{\text{total}}^{(i)}$ for each trajectory using Equation 8.
  3. Form groups of trajectories (the paper uses quantile-based grouping, though the exact grouping method is not specified in detail) and compute relative advantages within each group:

Ai=rtotal,i−mean({rtotal,1,rtotal,2,…,rtotal,G})std({rtotal,1,rtotal,2,…,rtotal,G})A_i = \frac{r_{\text{total}, i} - \text{mean}(\{r_{\text{total}, 1}, r_{\text{total}, 2}, \dots, r_{\text{total}, G}\})}{\text{std}(\{r_{\text{total}, 1}, r_{\text{total}, 2}, \dots, r_{\text{total}, G}\})}

where $G$ is the group size (all $N$ trajectories or subsets thereof), and $A_i$ is the standardized advantage for trajectory $i$ — positive if the trajectory outperforms the group mean, negative if it underperforms.

What the advantage computes: The advantage measures how much better or worse a trajectory's reward is compared to what the current policy typically achieves. By standardizing with the group standard deviation, GRPO ensures that the advantage scale is consistent across training steps even as the policy improves and absolute rewards increase. This prevents the policy gradient from becoming unstable when rewards drift upward.

The GRPO objective with the visual-anchored reward shaping is:

JGRPO(θ)=E(q,y)∼D,{oi}i=1G∼πθold(⋅∣q)[1G∑i=1G1∣oi∣∑t=1∣oi∣(min⁡(rti(θ)Ai,clip(rti(θ),1−ϵ,1+ϵ)Ai)−βDKLi,t(πθ∥πref))]\mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}_{(q, y) \sim \mathcal{D}, \{o^i\}_{i=1}^{G} \sim \pi_{\theta_{\text{old}}}(\cdot \mid q)} \left[ \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|o^i|} \sum_{t=1}^{|o^i|} \left( \min(r_t^i(\theta) A^i, \text{clip}(r_t^i(\theta), 1 - \epsilon, 1 + \epsilon) A^i) - \beta D_{\text{KL}}^{i,t}(\pi_\theta \parallel \pi_{\text{ref}}) \right) \right]

where $(q, y)$ is a question-answer pair from the dataset, $o^i$ is the $i$-th generated trajectory (output) in the group, $|o^i|$ is the number of tokens in that trajectory, $r_t^i(\theta) = \pi_\theta(o_t^i \mid q, o_{<t}^i) / \pi_{\theta_{\text{old}}}(o_t^i \mid q, o_{<t}^i)$ is the per-token importance sampling ratio comparing the new policy to the old policy, $A^i$ is the advantage from Equation 9, $\epsilon$ is the PPO clipping parameter (controlling how far the policy can deviate from the old policy), $\beta$ is the KL divergence coefficient, and $D_{\text{KL}}^{i,t}(\pi_\theta \parallel \pi_{\text{ref}})$ is the KL divergence between the current policy and a reference policy at token $t$ of trajectory $i$.

What this objective does: It's a standard PPO-style clipped objective with two modifications specific to this work. First, the advantages $A^i$ incorporate the visual attention reward through $r_{\text{total}}$, meaning trajectories that maintain high visual attention while producing correct answers receive higher advantages. Second, the KL penalty term prevents the policy from drifting too far from the cold-start SFT model during RL — this is important because the cold-start SFT already established strong visual grounding through the attention-guided training objectives, and unconstrained RL might undo those gains.

Hyperparameter specifics. The paper's RL configuration (Appendix E, Table 6) uses: learning rate $1 \times 10^{-6}$, weight decay $3 \times 10^{-5}$, batch size 256, KL divergence coefficient $\beta = 0.0$ (meaning the KL penalty is disabled in practice), temperature 0.8 (for exploration during trajectory sampling), and 8 rollouts per question. The RL stage runs for 4 epochs on 17.9K samples using the VeRL framework on 16 A100 GPUs.

The KL coefficient of 0.0 is noteworthy — it means the paper does not actually constrain the policy to stay close to the reference model, relying instead on the combination of a low learning rate ($1 \times 10^{-6}$), small number of epochs (4), and the PPO clipping mechanism to prevent catastrophic deviation. The low learning rate in particular acts as an implicit regularization: with such a small step size, the policy cannot move far from its initialization in only 4 epochs.

Complete Training Pipeline and Hyperparameters

The full AVAR training pipeline proceeds in three stages, with specific hyperparameters and datasets at each stage:

Stage 0: Base model. The starting point is Qwen2.5-VL-7B-Instruct, a 7-billion-parameter vision-language model that has been instruction-tuned but not specialized for reasoning. Its VAS is approximately 7.5, placing it in the narrow-view category.

Stage 1: Cold-start SFT with attention-guided objectives. The model is fine-tuned on 30.6K samples from the visual-anchored reflection data synthesis pipeline. The data comes from five sources: R1-ShareVL (~22.2K samples), Geo3K (~2.1K), M3COT (~3.2K), AlgoPuzzleVQA (~1.8K), and SOLIDGEO (~1.3K). Training uses LlamaFactory on 16 A100 GPUs for 20 epochs with the following hyperparameters:

  • Cutoff length: 30,000 tokens (maximum sequence length)
  • Learning rate: $5 \times 10^{-6}$
  • Warm-up ratio: 0.1 (linear warm-up over first 10% of training)
  • Batch size: 512
  • LR scheduler: cosine decay
  • Precision: bf16
  • Training module: all parameters (full fine-tuning)
  • Attention loss coefficients: $\alpha = 0.15$, $\beta = 0.15$
  • Stability constant: $\epsilon = 10^{-6}$

The output of this stage is the AVAR-CS model, which achieves a VAS of 13.8 (up from 7.5) and an average benchmark score of 52.6% (up from 49.1%, per Table 4).

Stage 2: RL fine-tuning with visual-anchored reward shaping. The AVAR-CS model is further optimized using GRPO on 17.9K public samples from four sources: R1-ShareVL (~12.1K), Geo3K (~2.1K), Super-CLEVER (~2.2K), and AI2D (~1.5K). Before RL training, the dataset undergoes a one-time difficulty filtering process: using the Qwen2.5-VL-7B base model, 8 rollouts are generated per question, and only questions whose 8-rollout accuracy falls between 0.25 and 0.75 are retained. This filtering ensures the RL dataset consists of problems that are neither trivially easy (where RL would provide no learning signal) nor impossibly hard (where the model never succeeds and thus never receives positive reward).

RL hyperparameters (from Appendix E, Table 6):

  • Max response length: 30,000 tokens
  • Weight decay: $3 \times 10^{-5}$
  • Learning rate: $1 \times 10^{-6}$
  • Warm-up ratio: 0.03
  • Batch size: 256
  • KL divergence coefficient: 0.0
  • Number of rollouts per question: 8
  • Temperature: 0.8
  • Epochs: 4
  • Hardware: 16 A100 GPUs using VeRL framework
  • Reward weights: $\lambda_v = 0.3$, $\lambda_f = 0.1$

The output is the final AVAR-Thinker model, which achieves a VAS of 18.9 and an average benchmark score of 56.1%, representing a 7.0% improvement over the base Qwen2.5-VL-7B.

Why this two-stage design: The cold-start SFT establishes the initial reasoning patterns and visual grounding behaviors that the RL stage then optimizes. Without the attention-guided SFT, the RL stage would start from a model with weak visual attention (VAS = 7.5) and would need to both discover good reasoning strategies and reshape attention simultaneously — a harder optimization problem. The SFT stage provides a warm start with already-improved attention patterns (VAS = 13.8), allowing the RL stage to focus on optimizing reasoning quality while the visual reward maintains (and further improves) visual grounding.

Generalization experiment. The paper also applies the same AVAR pipeline to Llama-3.2-11B-Vision-Instruct (Appendix B, Table 5) to demonstrate that the framework generalizes beyond the Qwen architecture. The same training data and hyperparameters are used, with the base model swapped. Results show a similar pattern of incremental gains from each component, with the final AVAR-Thinker achieving a 9.5% average improvement over the Llama baseline (46.7% vs. 37.2%).

The difficulty filtering rationale. The filtering of RL data to keep only examples with accuracy between 0.25 and 0.75 is based on the observation (from Yu et al., 2025b, on DAPO) that RL is most effective when the model has a reasonable chance of success — enough to receive positive reward and learn from it — but also a meaningful chance of failure — enough that improvement is possible. If the model already achieves near-perfect accuracy on a problem, RL provides no learning signal (the advantage is zero because all rollouts are correct). If the model never succeeds, RL provides no positive examples to reinforce. The 0.25–0.75 range targets the "zone of proximal development" where the model can succeed with effort but still has room for improvement, maximizing the expected information gain per RL step.

Why 20 epochs for SFT but only 4 for RL: The SFT stage involves learning entirely new behaviors (visual-anchored reasoning, attention reshaping) from static data, requiring many passes to fully internalize the patterns. The RL stage involves online optimization where the model generates its own trajectories — running too many epochs risks the model over-optimizing the reward signal (reward hacking) or drifting into degenerate behaviors. The small number of RL epochs (4) combined with the very low learning rate ($1 \times 10^{-6}$) reflects a conservative approach to RL that prioritizes stability over maximum optimization.

Summary of Design Choices and Their Justifications

  • Three-model data synthesis pipeline over single-model generation: decomposes the data generation problem into perception (Gemini), reasoning (Qwen3-235B), and grounding (Qwen3-32B), assigning each to the model best suited for it. A single multimodal model would compromise on at least one dimension.

  • Text-only reasoning model (Qwen3-235B) for generating reasoning chains over a multimodal reasoning model: forces all visual references to be explicitly verbalized (since the model can't see the image), creating training data where visual grounding is explicit in the text rather than implicit in the model's internal attention. This transfers better to the target 7B vision-language model.

  • Visual anchor insertion as a separate stage over generating anchors during reasoning: the reasoning generation model is optimized for logical correctness; adding the requirement to insert visual anchors would distract from reasoning quality. A separate smaller model handles the simpler text-editing task of anchor insertion.

  • Log-transformed attention losses over linear attention losses: amplifies gradients when attention is near zero (the regime where narrow-view models operate), creating stronger optimization pressure to escape lazy attention patterns. A linear loss would provide weak gradients in precisely the regime where they're most needed.

  • System token attention as the suppression target over suppressing all non-visual tokens: the training-free experiments (Section 4) showed that system token attention specifically is redundant; suppressing it produces gains without degrading performance. Suppressing attention to user tokens or generated tokens could remove useful information.

  • Correctness-gated visual reward over an unconditional visual reward: prevents the model from gaming the reward by generating long "looking at the image" sequences without solving problems. The gate ensures visual attention is only rewarded when it contributes to task success, aligning the auxiliary reward with the primary objective.

  • GRPO with relative advantages over absolute reward RL: stabilizes training by comparing trajectories within groups, making the optimization invariant to shifts in the absolute reward scale that occur as the policy improves. This is particularly important when the reward is a composite of binary correctness and continuous visual attention — the relative comparison naturally handles the different scales.

  • Cold-start SFT with 20 epochs followed by RL with 4 epochs over joint optimization: the SFT stage establishes foundational patterns from static data, and the RL stage refines them through online exploration. Too much RL risks undoing the attention reshaping achieved during SFT (the "more thinking, less seeing" degradation); the conservative RL schedule maintains the SFT gains while adding further optimization.

  • Difficulty-filtered RL data (accuracy 0.25–0.75) over using all available data: focuses RL on problems in the model's zone of proximal development where learning is possible, avoiding both trivial problems (no learning signal) and impossible ones (no positive examples). This is a variance reduction technique that makes each RL step more informative.

  • All-layer attention guidance over targeting specific layers: the layer-wise VAS analysis shows that lazy attention models exhibit weak visual attention uniformly across all layers, not just in specific ones. Targeting all layers ensures comprehensive attention reshaping rather than partial remediation.

4. Key Insights and Innovations

Innovation 1: Visual Attention as a First-Class Causal Mechanism in Multimodal Reasoning — Not Just a Correlate

The paper's most fundamental contribution is not the VAS metric itself, but the elevation of visual attention allocation from a descriptive correlate to a causal mechanism that governs multimodal reasoning performance. Prior work had documented correlations between attention patterns and model behavior — Yin et al. (2025) showed MLLMs devote insufficient attention to visual signals, Liu et al. (2025) observed that reasoning-oriented models pay less attention to images than non-reasoning counterparts, and Tang et al. (2025) found uneven attention distribution across heads. But these findings existed in an ambiguous epistemic space: was weak visual attention a symptom of poor reasoning (models that can't reason well happen to not look at images) or a cause (failing to look at images prevents good reasoning)?

The paper resolves this ambiguity through a causal chain of evidence that no prior work had assembled:

First, the VAS-reasoning correlation (r = 0.9616, Figure 1a) is not merely high — it's near-perfect across a diverse set of ten models trained with different strategies (text-only cold-start, multimodal cold-start, RL variants). A correlation this strong in a noisy empirical domain (where models differ in architecture, training data, and optimization) strongly constrains the space of possible explanations. It rules out accounts where reasoning quality and visual attention are independently determined by some third factor (e.g., "better training data improves both") because such accounts would predict more scatter than a 0.96 correlation permits.

Second, the training-free attention modulation experiments (Section 4, Figure 2) provide the decisive causal test. By exogenously manipulating attention weights at inference time — amplifying visual token contributions and suppressing system token contributions — the paper demonstrates that changing attention allocation changes reasoning performance, without any retraining. The 1–2% consistent gains across three different models and three benchmarks establish that attention allocation is upstream of reasoning quality: you can take a model with weak visual grounding, mechanically boost its visual attention, and it will reason better. This is the signature of a causal mechanism, not a mere correlate.

Third, the identification of the System Token Redundancy Zone adds mechanistic specificity. The finding that moderate system token suppression ($\alpha_{sys} \in \{0.00, 0.40\}$) helps but aggressive suppression ($\alpha_{sys} = 0.60$) hurts reveals that the attention budget is genuinely constrained — there is a fixed pool of attention that must be allocated across token types, and system tokens consume a portion that could be productively redirected to visual tokens. This shifts the conceptual model from "models don't look enough at images" to "models have a finite attention budget, and system tokens are consuming too much of it."

Why this is fundamental rather than incremental: Prior work treated attention patterns as diagnostic — something to measure and report. This paper treats attention patterns as a control variable — something to engineer. The shift from diagnostic to causal is not a small refinement; it changes what kinds of interventions make sense. If weak visual attention is merely a symptom, you try to improve reasoning and hope attention follows. If weak visual attention is causal, you directly target attention allocation and expect reasoning to improve. The entire AVAR framework follows from this causal claim.

Significance beyond raw performance: The causal framing opens a new axis for model improvement that is orthogonal to scaling model size, data quantity, or RL sophistication. Regardless of how large the model is or how much training data it has seen, if its attention is misallocated, it will underperform. This means attention optimization is a complementary lever — it can be combined with other advances rather than competing with them. The paper's FLOPs-matched analysis in the original example summary shows a parallel insight about test-time compute; here, the insight is about attention allocation during training.

Evidence anchor: The training-free modulation experiments (Figure 2) are the critical evidence for the causal claim. The correlation (Figure 1a) establishes the relationship's strength; the modulation experiments establish its direction.

Innovation 2: Lazy Attention Localization as a Mechanistic Explanation for the Text-Only Cold-Start Paradox

The paper's second distinctive contribution is the diagnosis of why text-only cold-start outperforms multimodal cold-start — a phenomenon the field had observed (Wei et al., 2025a,b; Yue et al., 2025b) and treated as a puzzling empirical regularity rather than an explained mechanism. The paper names this phenomenon Lazy Attention Localization and provides a concrete, attention-based account that explains not just that text-only cold-start works better, but how it achieves its effect through attention redistribution.

What the field assumed before this work: The implicit assumption in multimodal reasoning research was that cold-start data containing images would teach models to integrate visual information into reasoning — that seeing images during training would naturally lead to attending to images during inference. This assumption motivated the development of multimodal cold-start datasets (R1-OneVision, OpenVLThinker, Vision-SR1) that pair images with reasoning chains. The empirical finding that these datasets often underperform text-only alternatives was treated as a curiosity — perhaps text-only reasoning data is simply "higher quality" in some unspecified way, or perhaps multimodal alignment happens later during RL.

What this paper shows instead: The VAS analysis reveals that the mechanism is fundamentally about attention allocation, not data quality. Multimodal cold-start (as in R1-OneVision) produces models with attention distributions "nearly identical" to the base model (Figure 1b, Section 3.3) — the model never learns to look at images more than it already did. Text-only cold-start (as in OVR-CS), paradoxically, increases visual attention compared to the base model. The attention heatmaps in Appendix D (Figures 4–6) show this pattern consistently across multiple model pairs and benchmarks.

The explanatory mechanism: The paper's account is that text-only cold-start teaches structured reasoning patterns — step-by-step analysis, self-verification, error checking — that generalize across modalities. Once a model has internalized these patterns, it deploys them during multimodal inference, and the reasoning process itself creates pressure to consult visual information (because proper reasoning about a visual problem requires looking at the image). In contrast, multimodal cold-start data — especially caption-then-reason approaches — allows the model to satisfy the training objective by attending to captions rather than images, creating no pressure to develop visual attention habits.

Why this is a fundamental reframing: Before this work, the text-only vs. multimodal cold-start question was framed as a data quality problem: "we need better multimodal reasoning data." The paper's diagnosis reframes it as an attention allocation problem: "we need cold-start data that forces visual attention during reasoning, regardless of whether images are present." This reframing changes the optimization target from what data to include to what attention patterns the data induces — a more fundamental level of analysis that produces different design choices (visual anchors embedded in reasoning chains rather than captions prepended to them).

The negative result on existing multimodal data: The finding that some multimodal cold-start datasets reduce performance relative to the baseline (R1-OneVision data degrades Qwen2.5-VL-7B by 4.7%, Table 3) is a significant negative result because it demonstrates that the naive assumption — multimodal data helps multimodal models — is not just insufficient but can be actively harmful. This parallels the paper's earlier finding that aggressive system token suppression ($\alpha_{sys} = 0.60$) degrades performance — there is a regime where more of the "right" thing becomes too much. In this case, more multimodal data that doesn't enforce visual grounding reinforces lazy attention patterns, making the model worse at visual reasoning than if it had received no cold-start training at all.

Significance beyond raw performance: This diagnosis is important because it explains a set of seemingly contradictory empirical findings in the literature — why OVR-CS (text-only) outperforms R1-OneVision (multimodal) despite the latter seeming more "appropriate" for multimodal tasks — and provides a unified explanatory framework for future cold-start design. It tells practitioners: don't worry about whether your cold-start data includes images; worry about whether it teaches the model to sustain visual attention during reasoning.

Evidence anchor: Figure 1b (the VAS distribution comparison of Qwen2.5-VL-7B, R1-OneVision, and OVR-CS across layers), Appendix D Figures 4–6 (attention heatmaps), and Table 3 (comparison of VARD against other cold-start datasets, including the -4.7% degradation from R1-OneVision data).

Innovation 3: A Unified Three-Level Framework for Attention Engineering During Training

The paper's third distinctive contribution is the AVAR framework as a demonstration that attention allocation can be systematically engineered across all three levels of the training pipeline — data, supervised loss, and RL reward — rather than addressed at a single level. This is novel because prior work on visual attention in MLLMs had operated exclusively at inference time (ClearSight, adaptive attention calibration, dual-level attention intervention) or at a single training stage (e.g., attention-based regularization during fine-tuning), but never coordinated interventions across the entire training pipeline.

What the field did before AVAR: Prior attention interventions fell into two categories:

Inference-time patches (Yin et al., 2025; Fazli et al., 2025; Tang et al., 2025): Apply attention masks or scaling factors during the forward pass to reweight attention toward visual tokens. These are training-free and model-agnostic, but they don't change the model's underlying behavior — they're band-aids that must be applied every time the model is used. The paper's own training-free experiments (Section 4) belong to this category and validate the approach, but the paper explicitly positions them as diagnostic rather than prescriptive.

Single-stage training modifications: Some prior work incorporated attention-based regularization into supervised fine-tuning (encouraging certain attention patterns through auxiliary losses) or used attention signals in RL rewards, but these operated in isolation — a single loss term added to one training stage, without coordination across stages.

What AVAR does differently: The framework coordinates three interventions that reinforce each other:

At the data level, visual-anchored reflection data synthesis creates training examples where visual grounding is explicit in the text — the reasoning chains contain phrases like "look back at the triangle" that create a textual signal for when visual attention should occur. This provides the what to learn — the target behavior.

At the supervised learning level, the attention-guided training objectives ($\mathcal{L}_{\text{enhance-img}}$ and $\mathcal{L}_{\text{suppress-sys}}$) provide explicit optimization pressure toward the attention patterns that the data implicitly encodes. This provides the how to learn it — the gradient signal that reshapes attention.

At the RL level, the visual-anchored reward shaping ($r_{\text{visual}}$) ensures that the attention patterns established during SFT are maintained (and further optimized) during RL, rather than being undone by "more thinking, less seeing" drift. This provides the how to keep it — the incentive to sustain visual grounding when the model has the freedom to generate arbitrary reasoning chains.

Why coordination matters: The three levels address different failure modes. Data alone can encode the target behavior but provides no gradient signal for attention allocation specifically — the standard LM loss only optimizes token prediction, and attention patterns are a byproduct. Supervised loss alone can reshape attention but only on the fixed training distribution — it doesn't teach the model to allocate attention adaptively during free-form generation. RL reward alone can incentivize visual attention but starting from a model with lazy attention patterns (VAS = 7.5), the RL optimization problem is much harder — the model must simultaneously discover good reasoning and proper attention allocation, with sparse rewards. The coordinated framework solves each problem at the stage where it's most naturally addressed, producing a cumulative effect (VAS: 7.5 → 10.1 → 13.8 → 18.9, Table 4) that exceeds what any single level could achieve.

Why this is fundamental rather than incremental: The paper is not proposing a new attention loss or a new data augmentation technique — it's proposing a design philosophy for training multimodal reasoning models: treat attention allocation as a first-class optimization target that must be engineered at every stage, not left as an emergent property. The specific loss functions and data synthesis procedures are instances of this philosophy, but the philosophy itself is the innovation. Future work can substitute different data synthesis methods or different attention losses while still operating within the same three-level framework.

Significance beyond raw performance: The step-wise ablation in Table 2 (VARD alone: +1.7%, VARD+AGTO: +3.3%, full AVAR: +6.8%) demonstrates that each component provides cumulative gains rather than redundant contributions. This is evidence that the three levels are complementary — they address different aspects of the attention problem. A framework where components are redundant would show diminishing returns; AVAR shows approximately additive returns, suggesting that data-level, loss-level, and reward-level interventions each capture something the others miss.

Evidence anchor: Table 2 (step-wise ablation showing cumulative gains), Table 4 (VAS evolution across training stages: 7.5 → 10.1 → 13.8 → 18.9), and the comparison of AVAR against other cold-start approaches in Table 3 (showing that VARD alone already outperforms prior cold-start data, establishing the data-level component as independently valuable).

Innovation 4: A Quantitative, Reproducible Diagnostic Instrument for Multimodal Attention

The paper's fourth contribution is the Visual Attention Score (VAS) as a portable, model-agnostic diagnostic metric that quantifies visual grounding in a single scalar and enables systematic comparison across models, training strategies, and architectures. While this might seem like a methodological contribution rather than a conceptual one, the VAS's design embodies specific conceptual choices that make it more than just "another attention metric."

What the field used before VAS: Prior work on visual attention analysis typically relied on qualitative visualizations (attention heatmaps overlaid on images), layer-wise attention distribution plots, or head-level analyses. These are informative for understanding a single model's behavior but difficult to compare across models — there's no common scale, and aggregate patterns must be inferred by eye. Some works computed attention ratios similar to VAS but used them for within-model analysis rather than cross-model comparison.

What makes VAS distinctive:

The choice of system tokens as the normalization target. VAS is not "attention to images divided by attention to everything else" — it's specifically "attention to images divided by attention to system tokens." This choice is empirically grounded in the paper's finding that system tokens are the primary locus of redundant attention (Section 4, Figure 2), but it also makes VAS interpretable: a VAS of 7.5 means the model pays 7.5 times more attention to images than to system instructions. This ratio is meaningful regardless of model architecture, input length, or tokenization — it captures a behavioral propensity (the model's preference for visual vs. instructional information) that generalizes across settings.

The user-token-only query restriction. VAS is computed only over user tokens (the question), not over system tokens or generated tokens. This focuses the metric on the model's information-seeking behavior — when the model is processing the user's question, does it consult the image or the instructions? This is distinct from measuring attention during generation, which would conflate visual grounding with other factors (e.g., the model might attend to images while generating descriptions but not while reasoning).

The uniform layer-and-head aggregation. VAS averages uniformly over all layers and heads, treating all model components as equally important. This is deliberately coarse — it trades fine-grained detail for cross-model comparability. A model with VAS = 15 is not necessarily "twice as visually grounded" as one with VAS = 7.5, but the ordering is reliable, which is what enables the cross-model correlation analysis and the Narrow-Wide-Panoramic taxonomy.

Why this matters beyond this paper: The VAS provides a common language for discussing visual grounding that the field previously lacked. Instead of saying "this model seems to look at images more" (qualitative, subjective), one can say "this model has VAS = 18.9" (quantitative, reproducible). The paper demonstrates the utility of this common language through the Narrow-Wide-Panoramic taxonomy — a categorization that would be impossible without a shared metric. As more multimodal reasoning models are developed, a standardized diagnostic like VAS enables systematic comparison of training strategies across labs and architectures.

The Pearson correlation of 0.9616 as validation: The near-perfect correlation between VAS and reasoning performance is not just a finding — it validates VAS as a diagnostic instrument. A metric that correlates this strongly with downstream performance is useful for model development: researchers can compute VAS on a small sample of examples during training and predict whether their cold-start strategy is working, without waiting for full benchmark evaluation. This accelerates the research cycle by providing a quick, interpretable signal.

Evidence anchor: Figure 1a (VAS vs. performance correlation with r = 0.9616), Figure 1b (layer-wise VAS distributions for different models), Table 4 (VAS evolution tracking across AVAR training stages), and the Narrow-Wide-Panoramic taxonomy in Section 3.2 (which categorizes models based on VAS thresholds that are empirically derived from the cross-model analysis).

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The primary evaluation spans 7 multimodal reasoning benchmarks. For mathematical reasoning, the paper uses MathVista, MathVerse-Vision-Only (MathVerse-VO), and MathVision — all standard benchmarks for evaluating visual math reasoning in MLLMs. For multidisciplinary reasoning, MMMU and MMMU-Pro test understanding across diverse academic disciplines (science, humanities, engineering) requiring both visual perception and domain knowledge. For perceptual understanding, MMStar evaluates general visual comprehension, and HallusionBench specifically diagnoses robustness against visual hallucinations — where language priors might override what is actually in the image. This selection deliberately spans math reasoning, cross-disciplinary reasoning, and perception-focused tasks to test whether visual grounding improvements generalize beyond any single capability domain. The cross-model VAS analysis in Section 3 uses 200 sampled cases from MathVista for attention extraction and four benchmarks (MathVista, MathVision, MathVerse-VO, DynaMath-WORSE) for the performance correlation.

  • Base model(s). All main experiments use Qwen2.5-VL-7B-Instruct, a 7-billion-parameter vision-language model with instruction tuning but no specific reasoning specialization. The model is chosen because it represents "a canonical starting point" — it has non-trivial base performance (allowing cold-start improvements to be measurable) but is not already optimized for reasoning (allowing substantial headroom). Critically, its VAS of 7.5 places it squarely in the narrow-view category, making it the ideal testbed for demonstrating whether AVAR can transform attention patterns from narrow to panoramic. A generalization experiment in Appendix B applies the same AVAR pipeline to Llama-3.2-11B-Vision-Instruct, a different architecture family at a different parameter scale, to test whether the framework transfers beyond the Qwen architecture.

  • Metrics. The primary metric across all experiments is benchmark accuracy — the percentage of questions where the model's final answer matches the ground truth, as determined by each benchmark's standard evaluation protocol. For the VAS-performance correlation analysis, the metric is the average accuracy across MathVista, MathVision, MathVerse-VO, and DynaMath-WORSE. For the training-free modulation experiments, accuracy is reported per-benchmark (MathVista, MathVision, MathVerse-VO) at each modulation setting. All benchmark evaluations use standard grading functions provided by the respective dataset authors. The paper also reports Visual Attention Score (VAS) as a diagnostic metric to track attention evolution across training stages (Table 4), but VAS is not used as a primary evaluation metric — it is an analytical instrument, not a performance claim.

  • Baselines. The paper compares against three categories of models (Table 1). Closed-source models: GPT-4o and Claude-3.7-Sonnet — these represent frontier general-purpose models and provide an upper bound on what is achievable, though the paper does not claim to match them. Open-source general models: Qwen2.5-VL-7B (the base model itself, representing zero cold-start improvement), InternVL2.5-8B, LLaVA-OneVision-7B, and Llama-3.2-11B-Vision-Instruct — these are comparable-scale vision-language models without reasoning specialization. Multimodal reasoning models: Mulberry-7B, R1-OneVision, OpenVLThinker, ThinkLite-VL, MM-Eureka-7B, Vision-R1, VLAA-Thinker-7B, and Vision-SR1 — these are the most direct competitors, as they all apply some form of cold-start + RL pipeline to 7B-scale models. Some (R1-OneVision, OpenVLThinker, Vision-R1) use multimodal cold-start data; others (OVR-CS, OVR-RL, MiMo-VL-CS, MiMo-VL-RL) use text-only cold-start, providing the contrast that motivates AVAR. The paper also compares against prior cold-start data approaches in a separate controlled experiment (Table 3), where the same base model (Qwen2.5-VL-7B) is fine-tuned on different cold-start datasets: R1-OneVision data, OpenVLThinker data, Vision-SR1 data, and the paper's own VARD data — all under identical training conditions to isolate the effect of data composition.

  • Generation budget / compute accounting. The paper does not frame comparisons around a unified compute budget (unlike the test-time compute scaling paper in the reference example). Instead, comparisons are made at equal model scale (7B parameters) under standard training budgets. Cold-start SFT uses 30.6K samples for 20 epochs with batch size 512 on 16 A100 GPUs. RL uses 17.9K samples for 4 epochs with batch size 256 on the same hardware. The paper does not compare AVAR against larger models with matched FLOPs — all comparisons are at equal parameter count, meaning the performance differences are attributable to training methodology rather than model scale. This is both a strength (clean comparison) and a limitation (no evidence about whether scaling model size would be more or less efficient than AVAR's attention engineering).

  • Cross-validation / statistical protocol. The paper does not report cross-validation, confidence intervals, or statistical significance tests. Performance numbers are reported as point estimates (single accuracy figures per benchmark). This is a notable methodological limitation — with test sets ranging from several hundred to a few thousand questions, the reported improvements (e.g., +1.7% from VARD alone in Table 2) could fall within benchmark variance without statistical testing. The VAS-performance correlation analysis uses 200 sampled MathVista cases for attention extraction, and the correlation coefficient of 0.9616 is reported without a confidence interval. The training-free modulation experiments in Section 4 sweep four values of $\alpha_{sys}$ but do not report variance across multiple runs. The difficulty filtering for RL data (keeping samples with 8-rollout accuracy between 0.25 and 0.75) is a one-time preprocessing step with no reported sensitivity analysis.

Main Quantitative Results

Cross-Model VAS-Performance Correlation (Section 3.2)

The paper establishes its foundational finding through a cross-model analysis of 10 representative 7B-scale models spanning different cold-start strategies. The headline result appears in Figure 1a: reasoning performance and VAS are almost perfectly correlated, with a Pearson correlation coefficient of r = 0.9616.

The specific numbers (read approximately from Figure 1a): models with VAS below 10 — including Qwen2.5-VL-7B itself (VAS ≈ 7.5, average benchmark accuracy ≈ 45–48%), R1-OneVision, ThinkLite-VL, and MM-Eureka — cluster at the lower-left of the performance distribution, achieving average accuracy below 50%. Models with VAS between 10 and 15 — the Revisual-R1 variants — achieve intermediate performance around 50–53%. Models with VAS above 15 — OVR-RL, OVR-CS, MiMo-VL-CS, MiMo-VL-RL — cluster at the upper-right with accuracy above 53%.

The layer-wise VAS distributions in Figure 1b reveal the structural pattern behind this correlation. Qwen2.5-VL-7B and its multimodal cold-start variant R1-OneVision show "nearly identical attention distributions, with persistently weak reliance on visual tokens across all layers." Their VAS remains below 10 throughout all transformer layers, with no layer showing strong visual grounding. In contrast, OVR-CS (text-only cold-start) shows consistently higher VAS across all layers, with the gap widening in middle-to-upper layers where modality fusion predominantly occurs — exactly the layers Yin et al. (2025) identified as critical for integrating visual and textual information.

The fine-grained attention breakdowns in Appendix D (Figures 4–6) decompose this further by token type (visual, user, system). The heatmaps show that in narrow-view models (Qwen2.5-VL-7B, R1-OneVision, ThinkLite-VL), system tokens consistently dominate attention across most layers and heads — the model is "reading the instructions" rather than "looking at the image." In panoramic-view models (OVR-CS, MiMo-VL-CS, Revisual-R1-CS), visual tokens capture a larger share of attention, particularly in the middle layers where cross-modal integration happens.

Key interpretive insight: The correlation of 0.9616 is remarkably high for a noisy empirical domain. It implies that VAS alone explains roughly 92.5% of the variance in reasoning performance across these 10 models — leaving very little room for other factors like architecture differences, training data composition, or RL algorithms. This is what makes the correlation compelling as evidence for causality: if other factors were independently important, the correlation would be lower.

Lazy Attention Localization: Evidence from Cross-Model Comparison (Section 3.3)

The paper's diagnostic finding is that multimodal cold-start fails to increase VAS relative to the base model, while text-only cold-start succeeds. This is established through direct VAS comparison of specific model pairs in Figure 1b:

  • Qwen2.5-VL-7B (base, VAS ≈ 7.5) vs. R1-OneVision (multimodal cold-start, VAS ≈ 7–8): The distributions are "nearly identical," confirming that multimodal cold-start did not reshape attention. Despite being trained on reasoning data that includes images, R1-OneVision's attention patterns are essentially unchanged from the base model.

  • Qwen2.5-VL-7B (base, VAS ≈ 7.5) vs. OVR-CS (text-only cold-start, VAS ≈ 15–18): The text-only cold-start model shows substantially higher VAS across all layers. The attention that was previously allocated to system tokens has been redirected toward visual tokens, even though the training data contained no visual information at all.

The paper quantifies the magnitude of this difference: "models initialized with unimodal reasoning data, such as OVR-CS and Revisual-R1-CS, maintain 15–20% higher attention to visual features compared to those trained with multimodal reasoning data such as R1-OneVision and ThinkLite-VL." This 15–20% difference in attention allocation translates to substantial performance gaps — OVR-CS substantially outperforms R1-OneVision on multimodal benchmarks despite having never seen an image during cold-start.

The paper also documents that this pattern generalizes across model families. Appendix D, Figure 5 shows that ThinkLite-VL (multimodal cold-start) preserves the base model's attention distribution, while Revisual-R1-CS (text-only cold-start) increases visual attention. Figure 6 extends this to MathVision specifically, showing Vision-R1 (multimodal cold-start) failing to elicit enhanced visual attention while OVR-CS (text-only) succeeds. This cross-model, cross-benchmark consistency is the basis for naming the phenomenon Lazy Attention Localization: it is not an artifact of a specific model or dataset, but a systematic failure mode of multimodal cold-start as currently practiced.

Key interpretive insight: The term "lazy" is specifically about attention localization, not about the model's effort or reasoning depth. A model can generate long, elaborate reasoning chains (R1-OneVision produces chain-of-thought reasoning) while still being "lazy" in its attention — never genuinely looking at the image. The laziness is in the attention mechanism, not the output.

Training-Free Attention Modulation Results (Section 4)

The paper's causal validation experiment tests whether directly manipulating attention allocation at inference time — with no retraining — can improve reasoning performance. Three models (Qwen2.5-VL-7B, Revisual-R1-CS, OVR-CS) are evaluated on three benchmarks (MathVista, MathVision, MathVerse-VO) under attention modulation with fixed $\alpha_{img} = 0.15$ and varying $\alpha_{sys} \in \{0.00, 0.05, 0.40, 0.60\}$. The headline finding is that moderate system token suppression ($\alpha_{sys} \in \{0.00, 0.40\}$) consistently improves performance by 1–2%, as shown in Figure 2.

The specific patterns:

  • At $\alpha_{sys} = 0.00$ (no system suppression, only image enhancement at $\alpha_{img} = 0.15$): All three models show modest but consistent improvements across benchmarks. For Qwen2.5-VL-7B, MathVista improves from baseline (~68%) to roughly 69%, MathVision from ~25% to roughly 26%, and MathVerse-VO from ~41% to roughly 42%. The gains are small but present across all model-benchmark combinations.

  • At $\alpha_{sys} = 0.40$ (moderate system suppression): Performance peaks for most configurations. The 1–2% absolute improvement is sustained or slightly increased relative to $\alpha_{sys} = 0.00$. The paper terms the regime $\alpha_{sys} \in \{0.00, 0.40\}$ the System Token Redundancy Zone — a range where system token attention can be reduced without losing useful information.

  • At $\alpha_{sys} = 0.60$ (aggressive system suppression): Gains diminish or reverse. The paper does not provide exact numbers for the degradation, but Figure 2 shows that performance at $\alpha_{sys} = 0.60$ is consistently below the peak at $\alpha_{sys} = 0.40$, and in some configurations falls below the unmodulated baseline. This establishes a boundary: beyond a certain suppression threshold, the model starts losing genuinely useful information from the system prompt.

  • At $\alpha_{sys} = 0.05$ (minimal system suppression): Performance is similar to $\alpha_{sys} = 0.00$, suggesting that very weak suppression provides no additional benefit beyond image enhancement alone.

Cross-model consistency: The pattern of improvement at moderate suppression is consistent across Qwen2.5-VL-7B, Revisual-R1-CS, and OVR-CS despite these models having different baseline VAS values and different cold-start histories. This is important because it demonstrates that attention misallocation is not specific to untrained models — even models that already have improved attention from text-only cold-start (OVR-CS) benefit from further attention modulation, suggesting there is room for improvement even in panoramic-view models.

Key interpretive insight: The non-monotonic U-shaped response to $\alpha_{sys}$ is the critical evidence for the System Token Redundancy Zone concept. If system tokens were uniformly redundant, performance would improve monotonically with increasing suppression. If they were uniformly necessary, performance would degrade monotonically. The fact that moderate suppression helps but aggressive suppression hurts implies a genuine trade-off: some system token attention is redundant (the model is re-reading instructions it doesn't need), but some is functional (the model needs to process task framing and format requirements). The optimal $\alpha_{sys}$ represents the point where redundant attention has been eliminated while functional attention remains intact.

Main Benchmark Results: AVAR-Thinker vs. Baselines (Section 6.2, Table 1)

The paper evaluates AVAR-Thinker against all baselines on 7 benchmarks. The headline result: AVAR-Thinker achieves an average improvement of 7.0% over the base Qwen2.5-VL-7B, with improvements on every single benchmark and new state-of-the-art among 7B models.

The per-benchmark improvements over Qwen2.5-VL-7B (Table 1):

BenchmarkQwen2.5-VL-7BAVAR-ThinkerAbsolute Gain
MathVista68.274.7+6.5
MathVision25.237.4+12.2
MathVerse-VO41.150.4+9.3
MMMU-VAL58.163.8+5.7
MMMU-Pro38.342.9+4.6
MMStar62.164.1+2.0
HallusionBench50.759.5+8.8
Average49.156.1+7.0

The strongest improvements occur on benchmarks that the paper's framework predicts should benefit most from enhanced visual grounding. MathVision (+12.2%), which requires multi-step geometric reasoning where each step must reference specific visual features (angle measures, spatial relationships, shape properties), shows the largest gain — exactly what the attention reshaping hypothesis predicts, since geometry reasoning cannot succeed through language priors alone. HallusionBench (+8.8%), which is specifically designed to diagnose visual hallucinations (cases where language priors contradict image content), shows the second-largest gain — again consistent with the hypothesis, since increased visual attention should reduce the model's reliance on language priors and thus reduce hallucinations.

The weakest improvement is on MMStar (+2.0%), a general visual comprehension benchmark. This modest gain may reflect that MMStar includes many questions answerable through recognition rather than reasoning — if the base model already sees well enough for recognition tasks, additional visual attention provides diminishing returns. The paper does not discuss this possibility, but it aligns with the framework: attention reshaping should help most on tasks where sustained visual grounding throughout multi-step reasoning is required, not on single-step recognition.

Comparison to prior reasoning models: AVAR-Thinker at 56.1% average outperforms all prior 7B reasoning models. ThinkLite-VL achieves 53.1%, MM-Eureka 51.2%, OpenVLThinker 50.2%, and R1-OneVision 44.8%. The gap between AVAR-Thinker and the closest competitor (ThinkLite-VL, +3.0%) is comparable to the gap between ThinkLite-VL and the general-purpose InternVL2.5-8B (+5.3%), suggesting that AVAR represents a meaningful step rather than an incremental tweak.

Comparison to closed-source models: AVAR-Thinker at 74.7% on MathVista matches Claude-3.7-Sonnet (74.5%) and substantially exceeds GPT-4o (63.8%). However, on other benchmarks, the closed-source models maintain larger leads — Claude-3.7-Sonnet achieves 58.3% on HallusionBench vs. AVAR-Thinker's 59.5% (a rare win for the 7B model), but Claude likely dominates on MMMU and other knowledge-intensive benchmarks (Claude-3.7-Sonnet: 75.2% on MMMU-VAL vs. AVAR-Thinker: 63.8%). These mixed comparisons are expected given the massive scale difference (7B vs. hundreds of billions of parameters).

Key interpretive insight: The pattern of gains — largest on geometry reasoning and hallucination resistance, weakest on general recognition — is internally consistent with the paper's causal model. This is a form of construct validity: if AVAR works by reshaping visual attention, its effects should concentrate where visual attention most matters, and that is exactly what Table 1 shows.

Data-Centric Comparison: VARD vs. Prior Cold-Start Data (Section 6.3, Table 3)

To isolate the contribution of the data synthesis pipeline from the attention-guided training and reward shaping components, the paper conducts a controlled experiment where the same base model (Qwen2.5-VL-7B) is cold-start fine-tuned on different cold-start datasets under identical training conditions (standard SFT, no attention losses, no RL). The comparison in Table 3 includes:

Cold-Start Data SourceAverage Accuracy
Baseline (no cold-start)49.1%
+ R1-OneVision data44.6% (-4.5%)
+ OpenVLThinker data48.1% (-1.0%)
+ Vision-SR1 data44.8% (-4.3%)
+ VARD (Ours)51.0% (+1.9%)

The most striking finding is that some prior cold-start datasets reduce performance below the baseline. Fine-tuning on R1-OneVision's reasoning data causes a 4.5% average degradation, and Vision-SR1 data causes a 4.3% degradation. The paper states: "simply scaling reasoning data is insufficient and can be harmful." This is a genuine negative result with practical implications: not all reasoning data is beneficial, and using the wrong data can leave the model worse off than no cold-start at all.

In contrast, VARD alone (without attention-guided training or RL reward shaping) provides a 1.9% improvement over the baseline. On MathVision specifically, VARD achieves 32.9% compared to the baseline's 25.2% — a +7.7% improvement, demonstrating that the visual-anchored data design is particularly effective for tasks requiring sustained visual attention.

Key interpretive insight: The degradation from R1-OneVision data (-4.5%) should be understood through the Lazy Attention Localization lens. R1-OneVision's data uses caption-then-reason pipelines that train the model to reason from textual image descriptions rather than from images directly. When fine-tuned on this data, Qwen2.5-VL-7B learns that reasoning can succeed without genuine visual consultation — it reinforces the narrow-view attention pattern rather than disrupting it. The model becomes better at text-only reasoning but worse at visual reasoning because it has been trained to treat the image as optional. This is the "harmful" effect the paper warns about: training data that is misaligned with desired inference behavior can actively damage performance.

Ablation of AVAR Components: Cumulative Gains (Section 6.3, Table 2)

The paper systematically adds AVAR components to the baseline, measuring the cumulative performance impact. Table 2 reports:

ConfigurationComponentsAvg. Accuracy
BaselineNone49.1%
+ VARDVisual-Anchored Reflection Data only51.0% (+1.9%)
+ VARD + AGTO+ Attention-Guided Training Objectives52.6% (+1.6%)
AVAR-Thinker+ Visual-Anchored Reward Shaping (full)56.1% (+3.5%)

Stage 1 (VARD): The data synthesis pipeline alone contributes +1.9% average improvement. The gains are concentrated on MathVision (+7.7%) and HallusionBench (+4.6%), with smaller improvements on MathVista (+2.4%) and MathVerse-VO (+2.4%). Interestingly, MMStar shows a slight decrease (62.1% → 61.1%, -1.0%) and MMMU-VAL also decreases (58.1% → 55.2%, -2.9%). These regressions on general understanding benchmarks while improving on reasoning benchmarks suggest a possible specialization trade-off: visual-anchored data teaches the model to process images differently, which may temporarily disrupt recognition patterns that the base model had already mastered.

Stage 2 (+AGTO): Adding attention-guided training objectives provides an additional +1.6% average gain. The improvements are most pronounced on MathVerse-VO (+2.9% relative to VARD-only), MMMU-VAL (+3.1%), and HallusionBench (+1.9%). Notably, MMMU-VAL recovers from its VARD-only regression and now matches the baseline (58.3% vs. baseline 58.1%), suggesting the attention losses help the model balance its attention allocation more effectively — the VARD data may have over-emphasized visual attention at the expense of useful system token information, and the attention losses correct this imbalance.

Stage 3 (+VARS, full AVAR): The complete framework adds visual-anchored reward shaping during RL, contributing the largest single gain of +3.5% average. All benchmarks improve substantially: MathVista +2.7%, MathVision +3.3%, MathVerse-VO +6.4%, MMMU-VAL +5.5%, MMMU-Pro +3.1%, MMStar +1.3%, HallusionBench +2.3%. The large gain on MathVerse-VO is particularly notable — this benchmark specifically tests whether models "truly see the diagrams" (as the benchmark paper's title asks), and the visual reward during RL appears to strongly reinforce exactly this behavior.

Key interpretive insight: The cumulative gains pattern (+1.9% → +1.6% → +3.5%) is approximately additive rather than showing diminishing returns. If the components were redundant — e.g., if the attention-guided objectives merely reinforced patterns already present in the VARD data — we would expect the second and third stages to contribute less than the first. The fact that each stage provides a substantial increment suggests that the three components address genuinely distinct aspects of the attention problem: data provides the target behavior, attention losses provide the gradient signal to learn it, and RL reward prevents forgetting it during online optimization.

Attention Evolution Across Training Stages (Section 6.4, Table 4)

The paper tracks VAS throughout the training pipeline to validate that AVAR is reshaping attention as intended. Table 4 reports:

Training StageVASAvg. Accuracy
Qwen2.5-VL-7B (base)7.549.3%
+ VARD Data (SFT)10.151.0%
AVAR-CS (+AGTO)13.852.6%
AVAR-Thinker (+VARS)18.956.1%

The VAS progression (7.5 → 10.1 → 13.8 → 18.9) shows that each training stage increases visual attention, and the magnitude of VAS increase correlates with performance improvement. The base model starts in the narrow-view range (VAS < 10). VARD data alone moves it to the boundary of narrow-view (VAS = 10.1). Adding attention-guided objectives pushes it into wide-view territory (VAS = 13.8). The full AVAR framework with RL reward shaping reaches panoramic-view status (VAS = 18.9), surpassing OVR-CS and MiMo-VL-CS in VAS while also exceeding them in benchmark performance.

The final VAS of 18.9 is notable because it exceeds the VAS of OVR-CS (the text-only cold-start model that inspired AVAR) and is comparable to the strongest models in the paper's cross-model analysis. The paper achieves what text-only cold-start achieves accidentally — strong visual grounding — but through explicit engineering rather than happenstance.

Key interpretive insight: The VAS increase from AVAR-CS (13.8) to AVAR-Thinker (18.9) is +5.1, a substantial jump during the RL stage. This confirms that the visual-anchored reward shaping is actively reshaping attention during RL, not merely maintaining the attention patterns from SFT. It also validates the paper's concern that RL without visual rewards could cause attention degradation — the model would not spontaneously increase VAS by 5.1 points during RL without explicit incentive to do so.

Generalization Experiment: AVAR on Llama-3.2-11B-Vision-Instruct (Appendix B, Table 5)

To test whether AVAR generalizes beyond the Qwen architecture, the paper applies the same pipeline (same training data, same hyperparameters) to Llama-3.2-11B-Vision-Instruct. The results in Table 5 show:

ConfigurationComponentsAvg. Accuracy
Llama-3.2-11B-Vision-Instruct (base)None37.2%
+ VARDVARD only43.2% (+6.0%)
+ VARD + AGTO+ AGTO44.0% (+0.8%)
AVAR-Thinker (Llama)+ VARS (full)46.7% (+2.7%)

The overall improvement is 9.5% average, larger in absolute terms than the 7.0% on Qwen2.5-VL-7B, though from a lower baseline. The component contributions follow a similar pattern — VARD provides the largest gain (+6.0%), AGTO provides a smaller increment (+0.8%), and VARS provides a moderate further gain (+2.7%). The reduced contribution from AGTO on Llama (0.8% vs. 1.6% on Qwen) may reflect architectural differences in how attention is structured in Llama vs. Qwen, though the paper does not investigate this.

Key interpretive insight: The generalization experiment is important because it addresses the most obvious concern about AVAR: that it might be specific to Qwen2.5-VL's architecture, tokenization, or pretraining data. The fact that the same pipeline works on Llama — a different architecture from a different organization — suggests the underlying principles (visual anchoring in data, attention-guidance in training, visual rewards in RL) are architecture-agnostic. The larger absolute gain on Llama (9.5% vs. 7.0%) may reflect that the base Llama model had even weaker initial visual grounding, leaving more room for improvement.

Ablation Studies and Robustness Checks

  • Attention-guided loss coefficients ($\alpha$ and $\beta$): The paper sets both to 0.15 but does not systematically vary them. This is a significant missing ablation — we don't know whether the 0.15 values are near-optimal, whether the enhancement and suppression losses contribute equally, or whether different datasets/models benefit from different coefficients. The paper's justification (these values "are set" without sensitivity analysis) means the contribution of attention-guided objectives might be understated if the coefficients are suboptimal, or overstated if they happen to be surprisingly effective at these particular values.

  • Visual reward weight ($\lambda_v$): Set to 0.3, with no sweep reported. The choice is consequential: too high, and the model might prioritize looking at images over solving problems (gaming the visual reward); too low, and the reward might provide insufficient signal to resist the "more thinking, less seeing" RL drift. Without a sweep, we cannot assess whether the 7.0% gain could be higher with better reward weighting, or whether AVAR is robust to this hyperparameter.

  • Stability constant ($\epsilon$): Set to $10^{-6}$ and not varied. This is a minor hyperparameter that primarily affects numerical stability; the paper's choice is standard and unlikely to significantly affect results.

  • KL divergence coefficient ($\beta$): Set to 0.0 for RL, meaning no KL penalty is applied. This is a notable design choice: RL without KL regularization risks catastrophic forgetting or reward hacking. The paper mitigates this through a very low learning rate ($1 \times 10^{-6}$) and few epochs (4), but an ablation comparing $\beta = 0.0$ vs. $\beta > 0.0$ would clarify whether the KL penalty is unnecessary (because the low learning rate provides sufficient stability) or whether adding it could further improve results by preventing subtle attention drift.

  • Difficulty filtering threshold for RL data: The paper filters RL data to keep only samples with base model 8-rollout accuracy between 0.25 and 0.75. This is based on prior work (Yu et al., 2025b) rather than ablated in this paper. We don't know whether different thresholds (e.g., 0.1–0.9, 0.4–0.6) would change results, or whether the filtering is even necessary — an ablation comparing filtered vs. unfiltered RL data would establish the filtering's contribution to the 3.5% gain from the VARS stage.

  • Number of RL epochs: Set to 4, not varied. The paper's rationale (preventing reward hacking and catastrophic forgetting) is sensible, but an ablation showing that 8 or 16 epochs cause performance degradation would strengthen the case that 4 is near-optimal rather than arbitrary.

  • GRPO grouping strategy: The paper mentions "groups (e.g., by quantiles or clustering)" but does not specify the actual grouping method used. The standard GRPO implementation uses all $N$ trajectories as a single group, but the paper's language suggests subgrouping was considered. Without clarification, the exact RL optimization procedure is underspecified.

  • VARD data synthesis model choices: The paper uses Gemini 2.5-Pro for visual descriptions, Qwen3-235B-A22B for reasoning, and Qwen3-32B for anchor integration. The contribution of each model choice is not ablated (e.g., would using Qwen2.5-VL-7B's own descriptions degrade performance? Would a smaller reasoning model produce worse training data?). These ablations would be expensive but informative about the sensitivity of AVAR to teacher model quality.

  • VARD vs. caption-then-reason ablation: The paper compares VARD against other cold-start datasets (Table 3) but does not compare against a version of VARD that uses the caption-then-reason approach — e.g., where the same Gemini descriptions are used to generate reasoning chains without explicit visual anchors. This would isolate the contribution of the visual anchor insertion specifically, separate from the contribution of using Gemini for high-quality descriptions.

  • Training data quantity: The paper uses 30.6K cold-start samples and 17.9K RL samples. No data scaling analysis is performed — we don't know whether 30.6K is sufficient, excessive, or whether more data would yield further gains. The generalization experiment (Llama, Appendix B) uses the same data quantity, so it doesn't provide scaling information.

  • Layer targeting for attention losses: The paper applies attention losses to all layers. An ablation comparing all-layer vs. middle-layer-only targeting would test the hypothesis (from Yin et al., 2025) that middle layers are the critical site for modality fusion, and potentially show that targeting specific layers is more efficient than uniform application.

  • Training-free modulation on AVAR-Thinker: The paper applies training-free modulation to Qwen2.5-VL-7B, Revisual-R1-CS, and OVR-CS (Section 4), but not to AVAR-Thinker itself. If AVAR has already optimized attention allocation, training-free modulation might provide no additional benefit — this would be a strong confirmation that AVAR has reached the attention allocation frontier. Alternatively, if modulation still helps, it would suggest AVAR can be further improved.

Critical Assessment

What the Experiments Genuinely Demonstrate

The VAS-performance correlation (r = 0.9616) is robust — the relationship is visually clear in Figure 1a, spans 10 diverse models, and is reinforced by layer-wise analysis (Figure 1b) and fine-grained attention breakdowns (Appendix D). However, the correlation is computed over a selected set of models that all share the same base architecture scale (7B) and are drawn from recent multimodal reasoning research. It is not a random sample of all possible models or training strategies. The correlation might partially reflect a selection effect: models that end up with high VAS are those where training succeeded, and models where training failed to improve reasoning also failed to reshape attention. The causal direction cannot be fully resolved by correlation alone — which is why the training-free experiments are essential.

The training-free modulation experiments establish causal direction but are limited in magnitude and scope. The 1–2% gains (Figure 2) are consistent but small, and they are achieved by a crude mechanism (uniform scaling of attention by token type). The modulation does not demonstrate that attention allocation alone can achieve the 7.0% gain that AVAR achieves — it demonstrates that attention matters, but not that it matters this much. The gap between 1–2% (inference-time modulation) and 7.0% (full AVAR training) suggests that AVAR is doing more than just reallocating attention — it is likely also improving reasoning quality, data efficiency, and RL optimization in ways that go beyond the attention mechanism.

AVAR's 7.0% average gain over Qwen2.5-VL-7B is clearly demonstrated across 7 diverse benchmarks (Table 1), with every benchmark showing improvement. The gains are larger on benchmarks that specifically require visual grounding (MathVision +12.2%, HallusionBench +8.8%), consistent with the attention-reshaping hypothesis. However, all comparisons are at the same model scale — we don't know whether simply scaling Qwen2.5-VL to a larger size with standard training would match or exceed AVAR's 7.0% gain at lower engineering complexity. A FLOPs-matched comparison (like the reference example's Section 7) would clarify whether AVAR is efficient relative to the alternative of just training bigger models.

The cumulative contribution of AVAR components is demonstrated in Table 2, but the ablation is limited in one important way: the full AVAR-Thinker result includes both VARS and any interaction between VARS and the earlier stages. The 3.5% gain attributed to VARS (the difference between AVAR-CS at 52.6% and AVAR-Thinker at 56.1%) conflates the effect of RL with the effect of visual-anchored reward shaping — we don't have a "standard RL without visual reward" baseline (e.g., AVAR-CS + GRPO with only accuracy reward). This means we cannot determine how much of the 3.5% gain comes from RL itself (which typically improves reasoning models regardless of attention) versus from the visual reward specifically.

Where the Paper's Claims Are Stronger Than the Evidence

Claim: "Cold-start with high-quality text-only data consistently outperforms multimodal cold-start" because of attention allocation. The paper demonstrates that text-only cold-start models (OVR-CS, MiMo-VL-CS) have higher VAS than multimodal cold-start models (R1-OneVision, ThinkLite-VL) and that higher VAS correlates with better performance. But the paper does not directly demonstrate that attention allocation is the mechanism by which text-only cold-start achieves its advantage. Alternative explanations exist: text-only reasoning data might be of inherently higher quality (because text-only reasoning benchmarks are more mature), or text-only cold-start might teach more general reasoning skills that transfer better to multimodal settings. The paper's framework is consistent with the data but not uniquely supported by it — a controlled experiment where attention is artificially equalized between a text-only and multimodal cold-start model (through training-free modulation) and their performance compared would provide stronger mechanistic evidence.

Claim: "AVAR shifts redundant attention from system to visual tokens." Table 4 shows VAS increases from 7.5 to 18.9, confirming that visual attention increases relative to system attention. But "redundant" is an empirical claim — it asserts that the system token attention being reduced was genuinely unnecessary. The training-free experiments provide some evidence (suppressing system tokens at moderate levels helps), but we don't have a direct demonstration that the specific attention reduction achieved by AVAR targets redundant rather than functional system token attention. It's possible that AVAR reduces some functional system attention along with the redundant portion, and that the net gain of +7.0% represents the balance of improved visual grounding against slightly degraded instruction-following.

Claim: AVAR transforms models "from narrow to panoramic vision." The VAS taxonomy (narrow < 10, wide 10–15, panoramic > 15) is empirically grounded in the cross-model analysis but is fundamentally descriptive rather than theoretically derived. The thresholds are chosen post-hoc based on where models naturally cluster in Figure 1a. There is no independent validation that VAS = 15 represents a qualitatively different mode of visual processing rather than a point on a continuum. The "panoramic" label implies a fundamental shift in how the model processes images, but the evidence only shows a continuous increase in VAS along with continuous improvements in benchmark performance.

Genuine Weaknesses in the Experimental Design

Single model scale (7B). All main experiments use 7B-parameter models. The paper provides no evidence about whether AVAR's benefits scale with model size — would a 70B model benefit as much, less, or more from attention reshaping? The generalization experiment uses 11B parameters (Llama-3.2-11B), which is a step toward scale diversity but still within the same order of magnitude. The paper cannot claim that AVAR is beneficial for "MLRMs" in general — only for 7B MLRMs specifically.

No statistical significance reporting. Benchmark accuracies are reported as point estimates with no confidence intervals, standard errors, or significance tests. For benchmarks with 500–1000 test questions, differences of 1–2% could fall within sampling error. The 7.0% average gain is large enough to be practically meaningful, but the incremental contributions in Table 2 (+1.9%, +1.6%) could reflect noise, especially given the small ablation increments and the fact that some benchmarks showed regression at intermediate stages (e.g., MMMU-VAL dropped from 58.1% to 55.2% with VARD alone, then recovered to 58.3% with AGTO).

Cost of attention computation and training. The attention-guided training objectives require computing and storing attention matrices during training, which increases memory usage and computation time. The paper does not report the overhead relative to standard SFT, making it difficult for practitioners to assess the cost-benefit trade-off. Similarly, the visual-anchored reward shaping requires attention extraction during RL rollouts — an additional computational burden not quantified in the paper.

Difficulty estimation is absent. Unlike the reference example (which uses oracle difficulty estimation via 2048 samples per question), AVAR provides no mechanism for distinguishing when visual anchoring is most needed. The training-free experiments show that attention modulation benefits models regardless of their baseline VAS, suggesting the problem is universal, but this also means AVAR applies the same attention-reshaping to every example, regardless of whether that example actually requires sustained visual attention. An adaptive version that modulates attention based on example characteristics might be more efficient.

Missing Experiments That Would Strengthen the Paper

A "standard RL without visual reward" baseline. As noted above, the 3.5% gain attributed to VARS mixes the effect of RL with the effect of visual reward shaping. An ablation where AVAR-CS undergoes the same GRPO training but with only accuracy and format rewards (no $r_{\text{visual}}$) would isolate the contribution of visual reward shaping. If standard RL achieves +2.0% and visual reward adds +1.5%, that's a different story than if standard RL achieves +0.5% and visual reward adds +3.0%.

A FLOPs-matched comparison against larger models. The paper compares AVAR-Thinker (7B) against other 7B models but not against larger models with matched training compute. Would training a 13B or 70B model with standard cold-start (no AVAR) achieve better or worse performance than AVAR-Thinker for the same total compute? This question matters for practitioners deciding whether to invest in AVAR's engineering complexity versus simply scaling up.

A data-only ablation with standard SFT on VARD data but no attention losses. This is partially provided in Table 2 (the VARD row), but the paper does not report the VAS of the VARD-only model. This would clarify whether the data alone reshapes attention (which the VAS of 10.1 in Table 4 suggests it does to some degree) or whether the attention losses are necessary to achieve the full attention transformation.

A "VAS-matched" comparison between text-only and multimodal cold-start models. The paper's central diagnostic claim is that text-only cold-start works because it increases VAS, not because text-only data is inherently better. A controlled experiment would take a multimodal cold-start model with low VAS (e.g., R1-OneVision), apply training-free attention modulation to boost its VAS to match OVR-CS, and compare performance. If the performance gap closes, the mechanism is confirmed; if not, other factors are at play.

Sensitivity analysis of the VAS computation. The paper defines VAS with respect to user tokens as queries, system tokens as the normalization target, and uniform averaging over layers and heads. Different choices (e.g., using all tokens as queries, normalizing by all non-visual tokens, weighting layers by their information content) would produce different VAS values. An analysis showing that the cross-model ranking is robust to these choices would strengthen confidence that VAS measures a genuine property rather than an artifact of the specific formulation.

Where the Claims Hold Conditionally

AVAR's benefits are demonstrated only on multimodal reasoning benchmarks that have clear ground-truth answers (math, science, visual QA). The paper does not evaluate on open-ended generation, dialogue, or creative tasks. The attention-reshaping approach may be less beneficial — or even detrimental — when the task does not have a single correct answer that depends on precise visual grounding.

AVAR is demonstrated on static-image reasoning, not video. The benchmarks all use single images. The paper does not address whether sustained visual attention is equally important for video understanding (where attention must track objects across frames) or whether the visual-anchored data synthesis pipeline can be extended to temporal visual grounding.

AVAR's generalizability is shown for exactly one additional model family (Llama) at one additional scale (11B). The paper claims the approach "generalizes," but the evidence is limited to a single out-of-domain test. Llama and Qwen share fundamental architectural similarities (transformer-based, similar attention mechanisms); testing on models with substantially different attention patterns (e.g., Mamba-based, or models with different vision encoder architectures) would be needed for a strong generalizability claim.

The 7.0% gain is relative to a specific base model (Qwen2.5-VL-7B) at a specific point in its development. Future versions of Qwen (or other base models) may already incorporate better visual grounding, reducing the headroom for AVAR. The framework's value depends on base models having the Lazy Attention Localization problem; if base models improve, AVAR's marginal benefit may shrink.

6. Limitations and Trade-offs

The Difficulty Estimation Bottleneck: AVAR Treats All Examples as Equally in Need of Attention Reshaping

The assumption or constraint. AVAR applies the same three-component framework — visual-anchored data synthesis, attention-guided training objectives, and visual-anchored reward shaping — uniformly to all training examples and across all layers. Unlike the reference example's compute-optimal scaling framework, which allocates test-time compute adaptively based on estimated problem difficulty, AVAR makes no distinction between examples where attention reshaping is critical (e.g., geometry problems requiring precise visual feature extraction at each reasoning step) and examples where it may be unnecessary or even counterproductive (e.g., recognition tasks that the base model already handles well through language priors). The paper does not introduce any difficulty estimation mechanism, example-level strategy selection, or adaptive attention modulation based on problem characteristics.

The consequence. This uniformity means AVAR potentially applies attention-reshaping pressure to examples and model components that do not benefit from it. The ablation in Table 2 provides circumstantial evidence: adding VARD data alone caused regressions on MMStar (−1.0%) and MMMU-VAL (−2.9%) relative to the baseline, with MMMU-VAL only recovering to baseline levels (58.3% vs. 58.1%) after adding attention-guided training objectives. This pattern — a temporary regression followed by recovery — suggests that the VARD data introduced an attention distribution that was suboptimal for some task types, and the attention losses partially corrected this. But the paper does not investigate whether a selective application of attention guidance (e.g., applying attention losses only to middle layers where modality fusion occurs, or applying visual rewards only to examples where the base model's VAS is below a threshold) would outperform the uniform approach. In the worst case, forcing strong visual anchoring on all examples could interfere with efficient processing of tasks where language priors are genuinely sufficient — the model wastes attention capacity "looking at the image" when it doesn't need to.

What evidence exists in the paper. The MMMU-VAL regression under VARD-only training (58.1% → 55.2%) followed by recovery under VARD+AGTO (55.2% → 58.3%) is reported in Table 2 but not discussed in the text as evidence of potential interference. The paper hypothesizes that the regression reflects the model needing to "balance its attention allocation more effectively" after VARD data over-emphasized visual attention — but this hypothesis implies exactly the kind of example-dependent optimal attention allocation that AVAR's uniform approach cannot achieve. The generalization experiment (Appendix B, Table 5) shows a different pattern on Llama: MMMU-VAL improves monotonically (50.7% → 55.2% → 56.2% → 58.6%), suggesting the interference pattern may be model-specific, further complicating any claim that uniform attention reshaping is universally beneficial.

Mitigation status. The paper does not address this limitation. There is no difficulty estimation, no example-level strategy selection, no layer-specific targeting (the attention losses are applied to all layers), and no analysis of whether different benchmark categories respond differently to the same attention-reshaping pressure. The paper's discussion of limitations (Section 7) does not mention this uniformity assumption. Future work could introduce adaptive attention modulation based on estimated task visual-dependence, analogous to the difficulty-conditioned test-time compute allocation in the reference example.

The 7.0% Gain Is Measured Without Accounting for the Cost of Attention Computation During Training

The assumption or constraint. AVAR's headline 7.0% improvement over Qwen2.5-VL-7B is measured purely in terms of benchmark accuracy, with no accounting for the additional computational cost of the attention-guided components during training. Specifically: (1) the attention-guided training objectives ($\mathcal{L}_{\text{enhance-img}}$ and $\mathcal{L}_{\text{suppress-sys}}$) require computing and storing full attention matrices across all layers during SFT, which increases GPU memory usage and per-step computation time compared to standard SFT that only requires the forward pass for token prediction; and (2) the visual-anchored reward shaping ($r_{\text{visual}}$) requires extracting attention weights during RL rollouts to compute the visual-to-system attention ratio for every generated token, adding overhead to each GRPO trajectory generation step. The paper does not report wall-clock training time, peak GPU memory, or total FLOPs for any of its training stages, nor does it compare these costs against the standard cold-start + RL pipeline.

The consequence. A practitioner cannot assess whether AVAR's 7.0% accuracy gain represents a genuine efficiency improvement or simply reflects that AVAR uses more training compute. It is possible — and the paper provides no evidence to the contrary — that simply training Qwen2.5-VL-7B for more epochs with standard SFT on the same data, or using a larger batch size, or applying more RL steps, would achieve comparable gains at similar or lower computational cost. The attention losses require materializing and operating on $L \times H$ attention matrices of size $T \times T$ (where $T$ can be up to 30,000 tokens with the paper's cutoff length), which is a $O(T^2)$ memory and computation cost that standard SFT avoids. For long sequences — and multimodal reasoning chains can be very long — this overhead could be substantial, potentially making AVAR's SFT stage multiple times more expensive per step than standard SFT. Similarly, extracting attention during RL rollouts adds latency to each trajectory generation, reducing the number of rollouts that can be processed per unit time. Without cost accounting, the paper's claim that AVAR "achieves" a 7.0% gain is accurate but incomplete — it achieves this gain at an unspecified additional training cost.

What evidence exists in the paper. The paper reports hardware (16 A100 GPUs), training epochs (20 for SFT, 4 for RL), and batch sizes (512 for SFT, 256 for RL) in Sections 6.1 and Appendix E, but does not report training time, memory usage, or FLOP counts. The paper does not compare training cost against any baseline — standard SFT on the same data, RL without visual rewards, or simply training for more epochs. The attention loss hyperparameters ($\alpha = 0.15$, $\beta = 0.15$) are reported in Section 5.2, and the stability constant ($\epsilon = 10^{-6}$) and all-layer targeting are specified, but the computational implications of these choices are not discussed. There is no ablation comparing AVAR at $N$ epochs against standard training at $N + k$ epochs to determine whether the attention-guided approach is more sample-efficient or simply more compute-intensive.

Mitigation status. The paper does not address this limitation. The cost of attention computation is never mentioned, and the training configuration (hardware, epochs, batch size) is presented as standard implementation detail rather than as a cost that should be weighed against the accuracy gains. The authors do not suggest that future work should benchmark AVAR against compute-matched baselines. This is a significant practical gap: a deployment team deciding whether to adopt AVAR needs to know whether the 7.0% gain justifies the engineering complexity and computational overhead, and the paper provides no basis for that decision.

The Hardest Multimodal Reasoning Problems Remain Largely Unsolved by Attention Reshaping Alone

The assumption or constraint. AVAR operates by reshaping attention allocation during cold-start training, but it does not address the fundamental capability ceiling of the base model on problems that are genuinely outside its competence. If Qwen2.5-VL-7B fundamentally lacks the perceptual resolution, world knowledge, or reasoning depth to solve certain problems, no amount of attention reallocation will compensate — the model will fail regardless of how effectively it attends to visual tokens. This is directly analogous to the reference example's finding that test-time compute provides "near-zero improvement" on the hardest difficulty bin (bin 5) regardless of budget, because the base model simply cannot produce correct solutions. AVAR shares this fundamental limitation: it amplifies existing capability by ensuring the model uses its visual input effectively, but it does not create new capability.

The consequence. AVAR's improvements are likely concentrated on problems of easy-to-medium difficulty where the base model has the necessary knowledge and reasoning capacity but fails due to lazy attention — it "knows" how to solve the problem but doesn't "look" properly. On genuinely hard problems — those requiring spatial reasoning beyond the model's capacity, domain knowledge the model lacks, or multi-step inference chains that exceed its reasoning depth — AVAR's attention reshaping would provide minimal benefit because there is no correct solution in the model's output distribution to "unlock" through better visual grounding. The paper provides indirect evidence for this: the largest gains occur on MathVision (+12.2%) and HallusionBench (+8.8%), which specifically test visual grounding and hallucination resistance — problems where lazy attention is plausibly the primary failure mode. The smallest gain is on MMMU-Pro (+4.6%), which tests expert-level multidisciplinary understanding that may require knowledge beyond what a 7B model possesses regardless of how well it attends to images.

What evidence exists in the paper. The paper does not perform a difficulty-stratified analysis of AVAR's gains, so this limitation is inferred rather than directly demonstrated. Table 1 shows substantially different gains across benchmarks (+12.2% on MathVision vs. +2.0% on MMStar), which is consistent with the difficulty-dependence hypothesis but does not prove it — the benchmarks differ in task type as well as difficulty, so the varying gains could reflect task-specific effects rather than difficulty per se. The paper's difficulty filtering for RL data (Section 6.1, keeping only examples with base model accuracy between 0.25 and 0.75) implicitly acknowledges that AVAR's RL stage is most effective in a specific difficulty band, but this filtering is a data preprocessing choice rather than an experimental analysis. Unlike the reference example, which explicitly bins questions by difficulty quintiles and shows stratified results (Figure 3 right, Figure 7 right, Figure 9), this paper provides no per-question difficulty breakdown of its gains. We do not know whether the +12.2% on MathVision comes primarily from easy/medium questions with modest gains on hard ones (consistent with the "amplification" hypothesis) or is distributed evenly across difficulty levels (which would challenge it).

Mitigation status. The paper does not address this limitation. Section 7 (Conclusion) does not mention difficulty-dependent efficacy, and the difficulty filtering for RL data is presented as a standard practice rather than as evidence that AVAR's benefits are bounded. The analogy to the reference example's Section 7 finding — that test-time compute cannot substitute for pretraining on hard problems — is never drawn, despite the structural similarity. Future work could replicate the reference example's difficulty-bin analysis to map precisely where AVAR's gains concentrate, providing practitioners with a clear boundary condition for when attention reshaping is worth the training overhead.

The Visual-Anchored Reward's Contribution Is Confounded with RL's General Benefit

The assumption or constraint. The ablation in Table 2 attributes the gain from AVAR-CS (52.6%) to AVAR-Thinker (56.1%) — a +3.5% increment — to the addition of visual-anchored reward shaping (VARS). However, this comparison conflates two distinct effects: the effect of applying RL at all (GRPO optimization with accuracy and format rewards) and the marginal effect of adding the visual attention reward ($r_{\text{visual}}$) on top of standard RL rewards. The paper does not include a "standard RL without visual reward" baseline — i.e., training AVAR-CS with GRPO using only $r_{\text{accuracy}} + \lambda_f \cdot r_{\text{format}}$, with $\lambda_v = 0$. Without this ablation, the 3.5% gain attributed to VARS could be largely or entirely due to the general benefit of RL optimization (which typically improves reasoning models by 2–5% on multimodal benchmarks, as demonstrated by R1-OneVision, OVR-RL, and other prior work), with the visual attention reward contributing only a small fraction.

The consequence. The paper's claim that visual-anchored reward shaping is an effective attention-reshaping mechanism during RL — and specifically that it "prevents the model from reverting to text-only reasoning patterns" — is not directly tested. It is possible that standard RL without visual rewards would achieve nearly the same gain (e.g., +3.0% instead of +3.5%), in which case the visual reward would be a minor refinement rather than a critical component. Alternatively, standard RL might achieve a smaller gain (e.g., +1.5%) and the visual reward provides the remaining +2.0%, which would strongly validate the paper's claim. Without this ablation, we cannot assess whether the engineering complexity of extracting attention during RL rollouts and computing the visual reward is justified by the marginal benefit over standard RL. This is particularly important because the training-free modulation experiments (Section 4) achieved only 1–2% gains from direct attention manipulation — if standard RL already provides 2–3% gains, the visual reward's marginal contribution might be comparable to what a simple inference-time attention modulation could achieve without any RL at all.

What evidence exists in the paper. The paper's evidence for the visual reward's contribution is entirely indirect. Table 4 shows VAS increasing from 13.8 (AVAR-CS) to 18.9 (AVAR-Thinker), confirming that attention does shift during RL — but this shift could be caused by the RL process itself (the model generating longer reasoning chains and naturally attending more to images as it engages in deeper reasoning) rather than by the visual reward specifically. The paper's references to Liu et al. (2025)'s finding that reasoning-oriented MLLMs allocate less attention to images (the "more thinking, less seeing" phenomenon) provide motivation for adding the visual reward, but this motivation assumes that standard RL would cause attention degradation — an assumption that is never tested. The relevant ablation — AVAR-CS + GRPO with $\lambda_v = 0$ — is conspicuously absent from both Table 2 and the discussion.

Mitigation status. The paper does not acknowledge this confounding. The ablation in Table 2 is presented as a clean decomposition of AVAR's components, but the VARS row conflates "RL exists" with "visual reward exists." The paper's description of VARS in Section 5.3 states that the visual reward "ensures the model not only produces correct answers but also maintains strong visual grounding," implying that standard RL would fail to maintain visual grounding — but this is a claim, not a finding. Future work should include the missing ablation to establish the marginal contribution of visual-anchored reward shaping over standard RL, and ideally compare against a baseline that uses inference-time attention modulation (as in Section 4) applied to an AVAR-CS model to determine whether the RL stage is even necessary.

The Training Data Synthesis Pipeline Depends on Proprietary, Closed-Source Teacher Models

The assumption or constraint. AVAR's visual-anchored reflection data synthesis pipeline (Section 5.1) relies on three external models, two of which are proprietary: Gemini 2.5-Pro (used for high-fidelity visual descriptions) is a closed-source Google model accessible only through an API, and Qwen3-235B-A22B (used for reflection-enhanced reasoning generation) is a 235B-parameter model that, while open-weight, requires substantial computational resources to run — far more than the 7B target model being trained. Only Qwen3-32B (used for visual anchor integration) is both open-weight and manageable on modest hardware. The paper states that Gemini 2.5-Pro was chosen for its "superior perceptual capabilities," and Qwen3-235B-A22B for its "strong mathematical and scientific reasoning abilities" — both capabilities that the 7B target model lacks, which is precisely why they are needed as teachers.

The consequence. This dependence creates a barrier to reproducibility and independent validation. A research group that does not have API access to Gemini 2.5-Pro (due to cost, geographic restrictions, or organizational policy) cannot replicate the VARD data synthesis pipeline as described. Even with API access, the cost of generating 30,600 high-fidelity visual descriptions through Gemini 2.5-Pro could be substantial — the paper does not report the API cost, making it impossible for practitioners to estimate the total expense of replicating AVAR. Furthermore, the quality of the synthetic training data is bounded by the teacher models' capabilities: if Gemini 2.5-Pro misses important visual details, those details are lost from the training data entirely; if Qwen3-235B-A22B produces flawed reasoning chains, those flaws are distilled into the 7B student model. The paper's validation that the synthetic data is high-quality relies on "manual verification" (mentioned in Appendix G for the Qwen3-32B anchor integration stage) and the inference that the resulting model's strong performance implies data quality — but there is no direct evaluation of the synthetic data's accuracy, completeness, or freedom from hallucination. If future base models or tasks require perceptual or reasoning capabilities beyond what Gemini 2.5-Pro and Qwen3-235B-A22B can provide, the data synthesis pipeline would need to be upgraded to even stronger (and likely even less accessible) teacher models.

What evidence exists in the paper. The paper does not report the cost, success rate, or quality metrics of the data synthesis pipeline. The total dataset size (30.6K samples) is reported in Appendix C, but the number of API calls, token counts, or generation time is not. The paper does not discuss alternatives to Gemini 2.5-Pro (e.g., using open-source vision-language models for visual descriptions, or having the target model generate its own descriptions) beyond noting that Gemini was chosen for "superior perceptual capabilities." The generalization experiment in Appendix B uses the same VARD data synthesized with the same teacher models, so it does not test whether AVAR's benefits transfer when different teacher models are used.

Mitigation status. The paper partially mitigates this limitation by open-sourcing the generated training data. The abstract states: "The code, data, and models are available at [the project repository]." This means that practitioners who want to replicate AVAR's training can use the pre-generated VARD data without running the synthesis pipeline themselves — they avoid the API dependency and cost. However, this mitigation is partial: if a practitioner wants to apply AVAR to a new domain (e.g., medical imaging, document understanding, video reasoning), they would need to generate new training data for that domain, which would require either replicating the synthesis pipeline with the same teacher models or developing alternative synthesis procedures. The paper provides the prompts used for data synthesis (Appendix G), which aids replication, but does not provide an analysis of how sensitive AVAR's performance is to teacher model quality — e.g., would visual descriptions from a smaller open-source model produce comparable results, or is Gemini 2.5-Pro's quality essential? Without this analysis, the data synthesis pipeline is specified but not characterized.

Benchmark-Only Evaluation with No Real-World Deployment Analysis

The assumption or constraint. All of AVAR's evaluation is conducted on static benchmark datasets with clear ground-truth answers: MathVista, MathVision, MathVerse-VO, MMMU, MMMU-Pro, MMStar, and HallusionBench. These benchmarks share several properties that may not generalize to real-world deployment: they have unambiguous correct answers evaluable by string matching; they consist primarily of academic-style problems (math competitions, textbook exercises, diagnostic visual puzzles); they present single images with focused questions rather than multi-image, conversational, or interactive scenarios; and they have fixed evaluation protocols that the paper follows exactly. The paper does not evaluate AVAR-Thinker on any open-ended generation task, any interactive or multi-turn reasoning scenario, any task where correctness is subjective or multi-dimensional, or any real-world application (e.g., assisting humans with visual analysis, document understanding in a production setting, or visual reasoning in a conversational agent).

The consequence. We do not know whether AVAR's attention reshaping transfers to deployment settings or whether it introduces subtle failure modes that benchmarks cannot detect. Several specific concerns arise from the paper's own findings. First, the temporary regressions on some benchmarks during intermediate training stages (MMMU-VAL dropped 2.9% under VARD-only, MMStar dropped 1.0%) suggest that AVAR's training process temporarily disrupts some capabilities before rebuilding them — in a deployment context where the model is continuously updated, this pattern could cause regressions on customer-facing tasks that are not captured by offline benchmarks. Second, AVAR's attention reshaping makes the model "look at the image" more — but in real-world scenarios, images may contain distracting, irrelevant, or misleading visual information. A model trained to sustain strong visual attention might be more susceptible to visual distractors than a model with more flexible attention allocation. Third, the HallusionBench improvement (+8.8%) suggests AVAR reduces visual hallucinations, but HallusionBench is a curated diagnostic benchmark — we cannot assume that hallucination reduction on HallusionBench translates proportionally to hallucination reduction in the wild, where image-text mismatches are more diverse and subtle.

What evidence exists in the paper. The paper's evaluation covers 7 benchmarks, which is a reasonable breadth for an academic study but is entirely within the standard multimodal benchmark ecosystem. There is no evaluation on tasks that test attention flexibility — e.g., whether AVAR-Thinker can appropriately reduce visual attention when the image is irrelevant to the question (as in some adversarial or trick-question settings), or whether it can dynamically modulate attention based on task demands. The case study in Appendix F shows one successful example of AVAR-Thinker performing visual reflection on MathVerse-VO, but this is a cherry-picked positive example rather than a systematic analysis of failure modes. The paper's ablation (Table 2) shows that performance on MMStar — the benchmark most similar to general visual comprehension — improves by only +2.0%, which is consistent with AVAR's benefits being narrow (specific to reasoning tasks requiring sustained visual attention) rather than broad (improving all visual processing), but the paper does not discuss this implication.

Mitigation status. The paper does not address this limitation. There is no discussion of deployment considerations, no robustness evaluation (adversarial images, out-of-distribution visual inputs, multi-image scenarios), and no analysis of whether the attention patterns AVAR induces are adaptive or rigid. The paper's framing — "from narrow to panoramic vision" — implies that stronger visual attention is uniformly better, but in real-world use, optimal attention allocation is likely context-dependent. Future work should evaluate AVAR on tasks that require flexible attention modulation (e.g., answering questions about specific image regions while ignoring distractors), interactive scenarios (where the model must update its visual understanding based on user feedback), and diverse image types (photographs, documents, charts, diagrams, medical images) to establish the boundaries of where panoramic vision helps versus where it may interfere with task-appropriate processing.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the research conversation around multimodal reasoning from a data-centric framing — "we need better reasoning data" — to an attention-centric framing — "we need to engineer how models allocate attention during training." This is not a paradigm shift in the sense of overturning established theory, but it is a substantive reframing that changes what researchers optimize and how they diagnose failures.

Before this work, the dominant assumption in multimodal reasoning research was that cold-start data quality was the primary lever: curate high-quality reasoning chains, pair them with images, fine-tune, and good multimodal reasoning would emerge. The paper's Lazy Attention Localization diagnosis dismantles this assumption with concrete evidence. Multimodal cold-start data (R1-OneVision, OpenVLThinker, Vision-SR1) did not increase models' visual attention at all (Figure 1b, Appendix D) and could even degrade performance below the untrained baseline (Table 3: R1-OneVision data caused a 4.5% average drop). The mechanism of improvement was not data quality in the abstract — it was whether the cold-start procedure reshaped attention allocation toward visual tokens, which text-only cold-start achieved accidentally through structured reasoning patterns that generalize across modalities.

This reframing has three concrete consequences for how the field operates:

First, it redirects diagnostic effort from outputs to internal mechanisms. The paper demonstrates that a simple attention-derived metric (VAS) can predict multimodal reasoning performance with near-perfect correlation (r = 0.9616) across diverse models without running full benchmark evaluations. This makes VAS — or metrics like it — a practical tool for rapid model development: researchers can compute VAS on a few hundred examples during training and immediately assess whether their cold-start strategy is working, rather than waiting for benchmark results that may take hours or days. This accelerates the research cycle in a domain where training runs are expensive and evaluation is slow.

Second, it elevates attention allocation from a descriptive variable to a design target. Prior work on visual attention in MLLMs (Yin et al., 2025; Tang et al., 2025; Liu et al., 2025) analyzed attention patterns to understand model behavior, but did not propose training procedures to reshape those patterns. Inference-time interventions (ClearSight, adaptive attention calibration) treated attention as something to patch, not something to train. AVAR demonstrates that attention allocation can be systematically engineered across all three levels of the training pipeline — data, supervised loss, and RL reward — with cumulative, approximately additive gains (+1.9% from data, +1.6% from attention losses, +3.5% from visual rewards, totaling +7.0%). This opens a new axis for model improvement that is orthogonal to scaling parameters, data quantity, or RL sophistication — it operates on how the model uses its existing capacity rather than on increasing capacity itself.

Third, it reconciles the contradictory finding that text-only cold-start outperforms multimodal cold-start. This paradox had been observed empirically (Wei et al., 2025a,b; Yue et al., 2025b) but lacked a mechanistic explanation, leading to confusion about whether multimodal data was inherently problematic or whether existing multimodal datasets were simply poorly curated. The paper provides a clear, testable mechanism: text-only cold-start teaches structured reasoning patterns that create implicit pressure to consult visual information during multimodal inference, while caption-then-reason multimodal cold-start allows the model to satisfy training objectives by attending to textual descriptions rather than images. This explanation is specific enough to be falsifiable (if a multimodal cold-start dataset were constructed with explicit visual anchoring — as VARD does — it should match or exceed text-only cold-start) and constructive (it tells practitioners how to fix multimodal cold-start rather than abandoning it).

What becomes more attractive as a research direction: Attention-aware training objectives that go beyond simple token-type reweighting — the paper's log-transformed enhancement and suppression losses (Equations 5–6) are a first step, but the principle of directly optimizing attention allocation during training opens the door to more sophisticated mechanisms like attention routing, head-specific specialization, or dynamic attention budgets that adapt to example difficulty. Verifier and reward design for multimodal RL also becomes more attractive — the paper shows that adding a visual attention reward to standard correctness-based RL prevents the "more thinking, less seeing" degradation that Liu et al. (2025) documented, suggesting that reward shaping is a general tool for maintaining desired attention patterns during online optimization.

What becomes less attractive: Naive scaling of multimodal cold-start data without attention-aware design. The paper's finding that some multimodal datasets actively harm performance (Table 3: -4.5% from R1-OneVision data) is a cautionary signal against the assumption that more multimodal training data is always beneficial. Caption-then-reason pipelines, in particular, look suspect: if the training procedure allows the model to succeed without genuine visual consultation, it may reinforce lazy attention patterns rather than correct them. Research effort is better directed at data synthesis methods that embed visual grounding into the reasoning process itself (as VARD does) rather than at scaling existing pipeline outputs.

Follow-Up Research This Work Enables

Difficulty-stratified analysis of attention reshaping: where does AVAR help and where does it fail? The paper reports aggregate benchmark gains (+7.0% average) but does not break down improvements by question difficulty, a critical omission. Following the reference example's approach, a strong follow-up would compute per-question pass@1 rates for Qwen2.5-VL-7B on each benchmark, bin questions into difficulty quintiles, and measure AVAR's gain separately per bin. The paper's own framework predicts that gains should concentrate on easy-to-medium questions where the base model has the necessary knowledge but fails due to lazy attention; gains on the hardest bin should be minimal because no amount of attention reallocation can compensate for fundamental capability gaps. Confirming this prediction would establish a clear boundary condition for attention reshaping and tell practitioners exactly when AVAR is worth the training overhead. Disconfirming it — finding uniform gains across difficulty levels — would suggest AVAR is doing something more than attention reallocation, requiring a revision of the paper's causal model.

Ablation isolating the visual reward's marginal contribution over standard RL. The missing baseline is AVAR-CS trained with GRPO using only accuracy and format rewards (λ_v = 0), keeping all other hyperparameters identical. The paper attributes the +3.5% gain from AVAR-CS (52.6%) to AVAR-Thinker (56.1%) to visual-anchored reward shaping, but this gain conflates the general benefit of RL with the specific benefit of the visual reward. If standard RL without visual rewards achieves +3.0% (making the visual reward's contribution only +0.5%), the engineering complexity of extracting attention during RL rollouts is poorly justified. If standard RL achieves +1.0% and the visual reward provides +2.5%, the paper's claim is strongly validated. A secondary comparison — applying the paper's own training-free attention modulation (Section 4) to AVAR-CS during inference, without any RL — would further clarify whether the RL stage is necessary at all, or whether the same attention improvements can be achieved more cheaply at inference time on the already-improved AVAR-CS model.

Attention flexibility: can AVAR models appropriately reduce visual attention when images are irrelevant? AVAR is designed to increase visual attention uniformly — it applies the same enhancement and suppression losses across all layers and all examples. But in real-world deployment, not all images are equally relevant to their accompanying questions. A model that has been trained to sustain strong visual attention might be more susceptible to visual distractors (irrelevant or misleading image content) or might waste computational resources "looking" at images when language priors are sufficient. A stress-test would construct adversarial examples where the image contains misleading visual information that contradicts the text, or where the question is answerable from text alone with the image serving as a distractor. If AVAR-Thinker is more easily misled by visual distractors than Qwen2.5-VL-7B (because it trusts visual input more), this would reveal a significant deployment risk and motivate research into adaptive attention mechanisms that modulate visual grounding based on estimated image relevance. The HallusionBench result (+8.8%) suggests AVAR may actually be more robust to certain types of visual manipulation, but HallusionBench tests a specific form of hallucination (language-prior override) rather than visual distraction.

Layer-specific and head-specific attention guidance: can more targeted intervention achieve better results with lower overhead? AVAR applies attention-guided training objectives uniformly across all layers and heads. However, prior work (Yin et al., 2025) established that modality fusion occurs predominantly in middle layers, and Tang et al. (2025) showed that attention is unevenly distributed across heads, with certain heads disproportionately dominated by language priors. A follow-up could apply the attention enhancement and suppression losses only to the middle layers (e.g., layers 12–24 in a 28-layer model) or only to heads with the lowest baseline visual attention. The hypothesis is that targeting intervention where it is most needed would achieve comparable (or better) attention reshaping with lower computational overhead and reduced risk of interfering with layers/heads that already have healthy attention patterns. The paper's all-layer approach is a reasonable first attempt, but it likely wastes optimization effort on layers where attention is already well-allocated and may introduce unnecessary regularization in early layers (which tend to process low-level features) or late layers (which tend to integrate information for prediction).

Cross-architecture and cross-scale validation: does AVAR's benefit scale with model size? The paper demonstrates AVAR on two model families at two scales (Qwen2.5-VL-7B and Llama-3.2-11B), but provides no evidence about whether the 7.0% gain would be larger, smaller, or similar at 1B, 70B, or 400B parameters. This matters because larger models typically have more attention heads and layers, potentially providing more "slack" in attention allocation — a 70B model might have even more redundant system token attention that could be productively redirected, yielding larger absolute gains. Alternatively, larger models might already allocate attention more effectively (having more capacity to learn good attention patterns from pretraining alone), leaving less room for AVAR's improvements. A scaling study applying AVAR to Qwen2.5-VL at 1B, 7B, and 72B (or to a comparable model family with multiple available scales) would characterize the relationship between model scale and attention-reshaping benefit, answering the practical question of whether AVAR is most valuable for small models (where efficiency matters most), large models (where absolute gains might be largest), or all scales equally.

Data synthesis robustness: how sensitive is VARD to teacher model quality? The paper's data synthesis pipeline depends on Gemini 2.5-Pro (visual descriptions) and Qwen3-235B-A22B (reasoning chains) — both substantially more capable than the 7B target model. A systematic ablation would replace these teacher models with progressively weaker alternatives (e.g., Gemini 2.5-Flash, Qwen2.5-VL-72B, Qwen2.5-VL-7B itself) and measure the resulting VARD data quality and downstream AVAR performance. This would answer two practical questions: (1) Can AVAR be replicated without API access to proprietary models like Gemini, using only open-weight alternatives? (2) Is AVAR fundamentally a distillation technique (transferring capability from strong teachers to a weak student) or a genuine attention-reshaping technique (where the attention patterns matter more than the absolute quality of the reasoning)? If AVAR with self-generated VARD data (using Qwen2.5-VL-7B as its own teacher) still achieves significant gains, the attention-reshaping mechanism is confirmed as the primary driver. If gains collapse without strong teachers, AVAR is better understood as a distillation method with attention regularization.

Practical Applications and Downstream Use Cases

Cost-efficient visual reasoning for educational technology. AVAR-Thinker's strongest gains occur on MathVision (+12.2%), a benchmark of competition-level geometry problems that require sustained visual attention across multi-step reasoning — precisely the type of problems students encounter in geometry coursework and standardized tests. A 7B model running on consumer-grade hardware (a single A100 or even a high-end consumer GPU) can now approach the MathVista performance of Claude-3.7-Sonnet (74.7% vs. 74.5%), a closed-source model likely two orders of magnitude larger. For educational platforms that need to provide step-by-step geometry tutoring at scale, deploying AVAR-Thinker locally eliminates per-query API costs while maintaining competitive accuracy on the visual reasoning tasks that most strain current open-weight models. The HallusionBench improvement (+8.8%) is equally practical: in educational settings, a model that hallucinates visual content (claiming to see geometric features that aren't in the diagram) provides actively harmful tutoring. AVAR-Thinker's reduced hallucination rate directly addresses a known failure mode of MLLMs in pedagogical applications.

Visual document understanding with reduced hallucination risk. The HallusionBench result (+8.8%) measures robustness against exactly the failure mode that makes current MLLMs unreliable for document understanding: the model overrides what it sees in the image with what it expects based on language priors. In legal document review, medical image analysis, or scientific figure interpretation, a hallucinated "reading" of a chart or diagram can have serious consequences. AVAR's attention reshaping provides a training-based solution to hallucination reduction that does not require inference-time guardrails or retrieval augmentation — the model itself becomes more grounded in visual evidence. For organizations handling sensitive visual data where sending images to cloud APIs raises privacy concerns, a locally-deployed AVAR model that both runs on-premise and hallucinates less is a meaningful improvement over both cloud-based alternatives (which may hallucinate less but raise data residency issues) and existing open-weight models (which can run locally but hallucinate more).

Cold-start recipe for domain-specific multimodal reasoning. The paper's generalization experiment (Llama-3.2-11B, +9.5% average) and the open-sourcing of both the VARD data and the training code mean that practitioners can apply AVAR to new base models and new domains without designing their own attention analysis or loss functions from scratch. A team building a multimodal reasoning system for a specific domain — architectural blueprint analysis, satellite imagery interpretation, medical scan reading — can take their domain-specific image-question pairs, run them through the VARD synthesis pipeline (or adapt the prompts in Appendix G for their domain), fine-tune with the attention-guided objectives at the specified hyperparameters (α = 0.15, β = 0.15), and apply GRPO with visual reward shaping at λ_v = 0.3. The paper's value here is not just the 7.0% gain on general benchmarks but the demonstration that the framework transfers across model families (Qwen → Llama) and that each component provides cumulative benefit — practitioners can adopt the full pipeline with confidence that each stage adds value and that the hyperparameters are not catastrophically sensitive (given that the same settings worked on two different architectures without tuning).

Inference-time attention modulation as a lightweight diagnostic for model selection. The training-free modulation experiments (Section 4, +1–2% gains) are not just a causal validation for the paper's framework — they are a practical tool. When evaluating whether a candidate model has a visual attention problem, a practitioner can apply the modulation (α_img = 0.15, moderate α_sys) to a small test set. If performance improves by 1–2%, the model has unused visual reasoning capacity that could be unlocked through more thorough attention reshaping (cold-start retraining with AVAR, or additional RL with visual rewards). If performance does not improve or degrades, the model's attention is already well-allocated, and further investment in attention engineering would have diminishing returns. This diagnostic is training-free, takes minutes to run, and works across model architectures — making it a practical addition to any model evaluation pipeline for multimodal reasoning deployments.