Think Visually, Reason Textually: Vision-Language Synergy in ARC

🎯 Pitch

Giving vision-language models images of ARC grids makes them worse at reasoning—unless you leverage the complementary strengths of vision and text at different stages. By letting VLMs visually perceive spatial patterns first, then reason about precise rule application in text, and finally use vision to self-correct errors, the authors unlock up to a 7.25% accuracy gain over text-only baselines on the stubborn ARC-AGI benchmark.

1. Executive Summary

This paper proposes two training-free strategies that strategically integrate visual and textual modalities to solve ARC-AGI abstract reasoning tasks, demonstrating that vision and language possess complementary strengths across distinct reasoning stages. Using flagship models including o4-mini, Gemini-2.5-Pro, GPT-4o, and Qwen3-VL on the official ARC-AGI evaluation set as well as BARC and Re-ARC benchmarks, the authors introduce Vision-Language Synergy Reasoning (VLSR — decomposing ARC-AGI into visual rule summarization that perceives 2D spatial patterns holistically, followed by textual rule application for precise element-wise manipulation) and Modality-Switch Self-Correction (MSSC — using visual consistency verification to detect errors in text-based outputs and trigger iterative refinement, breaking the confirmation bias that cripples same-modality self-correction). The combined approach yields up to a 7.25% improvement over text-only baselines and a 4.33% average gain across diverse models, establishing that visual abstraction provides complementary benefits that text-based memory retrieval cannot capture, while hard problems where the model lacks fundamental capability remain unsolved regardless of modality.

2. Context and Motivation

The Core Problem: ARC-AGI Remains Unsolved Despite Frontier Model Advances

The Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI), introduced by Chollet (2019), represents one of the most stubbornly persistent challenges in modern AI evaluation. Unlike benchmarks that test domain-specific knowledge — natural language understanding, image classification, or even mathematical reasoning — ARC-AGI measures the ability to learn how to learn: a system must induce abstract transformation rules from a minimal set of input-output matrix pairs (typically 2–5 examples) and apply those rules to entirely novel test instances. Each task involves 2D grids with cell values ranging from 0–9, where grid dimensions can reach up to 30×30. The underlying transformations span a vast space of conceptual operations: symmetries, rotations, object manipulations, color mappings, counting, path-finding, and complex compositional rules that combine multiple primitives.

The benchmark is deliberately designed to be "priors-minimal" — it does not assume the solver possesses any specific world knowledge, domain expertise, or pre-trained patterns. Instead, it tests whether an intelligence can construct new reasoning procedures from scratch given only a handful of demonstrations. This makes ARC-AGI fundamentally different from standard supervised learning benchmarks where models can exploit statistical regularities in large training corpora. Humans achieve over 97% accuracy on ARC-AGI tasks, finding them intuitive and often trivial. Yet even the most capable frontier models — GPT-5, Grok 4, Gemini-2.5-Pro, and o4-mini — struggle substantially, with text-only approaches typically achieving pass@1 accuracy well below 50% on the official 400-task evaluation set (Table 2, baseline column).

The paper opens by positioning ARC-AGI performance as a litmus test for artificial general intelligence, noting that "state-of-the-art models (e.g., GPT-5 and Grok 4) now highlight ARC-AGI performance as a key indicator of their reasoning capabilities." This framing matters: ARC-AGI is not just another benchmark to be saturated through scale — it is explicitly designed to resist memorization and require genuine abstraction. Progress on ARC-AGI therefore carries outsized significance for the field's broader ambitions toward general-purpose reasoning systems.

Why This Problem Matters

The significance of ARC-AGI extends well beyond the benchmark itself, touching on several fundamental questions in AI research:

Evaluating reasoning vs. retrieval. Most AI benchmarks conflate two distinct capabilities: (1) retrieving and applying memorized knowledge, and (2) constructing novel reasoning chains from first principles. ARC-AGI cleanly isolates the latter by ensuring tasks cannot be solved through pattern matching against pretraining data. This matters because real-world intelligence — particularly in scientific discovery, mathematical theorem proving, and novel engineering design — depends far more on the second capability than the first. A system that performs well on ARC-AGI genuinely demonstrates abstraction, not just retrieval.

Measuring sample efficiency. Humans can induce complex rules from 2–5 examples; LLMs typically require thousands or millions of examples to learn comparable patterns through fine-tuning. ARC-AGI's few-shot evaluation paradigm directly tests this sample efficiency gap, which remains one of the most salient differences between human and machine learning. Closing this gap has implications for few-shot learning in data-scarce domains (medical diagnosis, rare language translation, novel scientific problem-solving).

Testing compositional generalization. ARC-AGI tasks often require combining multiple primitive operations — "rotate the blue object 90 degrees clockwise, then fill all empty cells adjacent to red objects with yellow." The compositional explosion means that even a system exposed to all individual primitives during training must generalize to unseen combinations at test time. This is a fundamental challenge for neural networks, which often struggle with systematic compositionality beyond their training distribution.

Implications for AGI progress. Because ARC-AGI explicitly targets fluid intelligence (the ability to solve novel problems) rather than crystallized intelligence (accumulated knowledge), it serves as one of the few benchmarks where performance improvements cannot be trivially attributed to scale or data. A breakthrough on ARC-AGI would represent genuine progress toward general-purpose reasoning, not just incremental scaling of existing paradigms. This is why the paper positions ARC-AGI as "a key indicator of their reasoning capabilities" and why both model developers and the broader research community track ARC-AGI scores closely.

The Text-Only Paradigm and Its Blind Spots

The paper's central observation is that the overwhelming majority of existing ARC-AGI approaches share a common assumption: they treat ARC-AGI as a purely textual reasoning task. Input-output matrix pairs are represented as nested lists (e.g., [[0,1,2],[3,4,5],[2,3,5]]), and all reasoning — rule induction, transformation application, output generation — occurs in the textual modality. This design choice, while computationally convenient (it allows standard LLMs to process ARC-AGI without architectural modifications), fundamentally contradicts human problem-solving intuition.

The paper articulates this tension in Section 1:

"when humans approach ARC-AGI tasks, they naturally visualize the patterns: a color-coded 2D grid immediately reveals spatial relationships like symmetries, rotations, or shape transformations that are tedious to infer from textual coordinate descriptions."

This is not merely a philosophical point about cognitive fidelity. The paper argues that the text-only paradigm introduces specific, identifiable limitations that constrain model performance:

Loss of 2D spatial structure. In textual representation, two vertically adjacent cells in the same column — which are perceptually contiguous in a visual grid — may be separated by dozens of tokens in a nested list representation. The paper gives the concrete example (Section 3.2.2, point 2): [[0,1,2],[3,4,5]] where elements 1 and 4 (vertically adjacent in the same column) are separated by multiple brackets and commas in the token sequence. This means that spatial relationships that are immediate and obvious to visual perception (symmetry across a horizontal axis, connected components, diagonal patterns) require explicit, error-prone sequential reasoning when encoded as text. The paper notes that "rules extracted through textual reasoning tend to lack 2D structural characteristics and perform poorly when tasks require capturing inter-row or diagonal regularities."

Inefficient encoding for large grids. ARC-AGI grids can reach 30×30, containing 900 cells. Textual representation requires encoding each cell value (one token) along with delimiters — brackets, commas, and whitespace — ballooning to thousands of tokens for a single matrix. When example pairs and test inputs are concatenated, the prompt length grows prohibitively large, consuming context window budget and introducing long-range attention challenges. The paper contrasts this with visual encoding, where a 30×30 grid is represented as a single image consuming only a few hundred vision tokens (Section 3.2.2, point 3), citing work on visual-based compression for document understanding as a parallel development.

Element-level processing bias. The paper's qualitative analysis reveals that text-based reasoning tends to process elements independently or through type-level statistics (e.g., frequency counts, row-by-row operations), rather than perceiving holistic spatial patterns. The example in Figure 4 exemplifies this failure mode: GPT-4o, given textual matrices, incorrectly summarizes the rule as "select the number with the highest count" — a frequency-based heuristic that ignores spatial structure entirely. The correct rule ("retain large connected color blocks") requires recognizing contiguous regions, which is immediate in visual perception but obscure in textual representation.

Statistical vulnerability to transposition. The paper makes a particularly insightful observation (Section 3.2.2, point 2): if both input and output matrices in a task are transposed, a rule discovered through visual representation remains essentially invariant (spatial transformations are preserved), while a rule derived from textual representation may be significantly affected because the token ordering changes. This sensitivity to what should be a representationally neutral transformation reveals the brittleness of text-centric reasoning for spatial tasks.

The Paradox of Naive Visual Integration

Given these limitations of text-only approaches, the natural response might be: why not simply render ARC-AGI grids as images and use vision-language models? The paper reveals that this straightforward approach produces a counterintuitive negative result that motivates the entire methodological framework.

The authors' "preliminary experiments" (Section 1, expanded in Section 3.2) show that "naively rendering ARC-AGI grids as images actually degrades performance compared to text-only baselines." This is the paper's critical empirical pivot: vision is not uniformly superior; its utility depends dramatically on the specific reasoning stage. The quantitative analysis in Table 1 quantifies this paradox precisely. Using Gemini-2.5-Pro as the base model:

For rule summarization (extracting transformation patterns from examples), visual representation provides a clear advantage: 40.75% accuracy versus 37.25% for text-only — a 3.5 percentage point gain.
For rule application (applying the extracted rule to a new input), visual representation causes a dramatic performance collapse: accuracy drops from 40.75% (textual application) to 23.75% (visual application) — a 17 percentage point degradation.

The paper attributes this to a fundamental tension in how vision processes information. Section 3.2.2, point 4 explains:

"Images represent matrices as an integrated whole rather than encoding individual elements separately. While this provides a global perspective advantageous for rule summarization, it becomes inadequate when element-wise processing is required during rule application."

The failure mode is concrete: when a 20×20 grid is presented as an image, the model may "confuse the value at position (5,7) with a nearby cell" or fail to reliably identify specific grid coordinates. Vision excels at holistic pattern recognition but lacks the discrete, indexable precision that text naturally provides through explicit coordinate-value pairs. This is why vision helps for answering "what is the overall transformation pattern?" but hurts for answering "what value should go in cell (5,7)?"

This paradox — that vision can simultaneously improve one reasoning stage while catastrophically degrading another — is the paper's central empirical motivation. It transforms the research question from "should we use vision?" to "when and how should we strategically combine vision and text?" The answer, as the paper develops throughout Section 3, is that the two modalities have complementary strengths that map naturally onto distinct reasoning subtasks.

Where Prior Approaches Fall Short

The paper situates its contribution against a landscape of existing ARC-AGI strategies, identifying specific limitations in each category:

Training-based approaches. A dominant paradigm generates large-scale synthetic ARC-AGI tasks by permuting and combining predefined transformation rules, then fine-tuning LLMs on this synthetic data. The paper cites approaches including Li et al. (2024), Hodel et al. (2024), Moskvichev et al. (2023), and several others (Section 2). While these methods achieve non-trivial performance through scale (the paper's own fine-tuning experiments in Section 4.4 use 200k synthetic tasks from ARC-Heavy-200k), they share a common limitation: they "treat ARC-AGI as a pure text task for both training and reasoning, neglecting image representations that demonstrate the rules more intuitively." The paper argues that even when fine-tuned on massive synthetic corpora, text-only models lack access to the 2D structural cues that visual representations provide, fundamentally constraining their ability to learn spatial transformation patterns.

Test-time training. Another family of approaches (Akyürek et al., 2024; Franzen et al., 2024; Pourcel et al., 2025) uses the provided example pairs not only as context for reasoning but as training data for additional fine-tuning before generating answers. While effective, these methods are computationally expensive (requiring gradient updates at inference time) and still operate purely in text. The paper does not directly compare against test-time training approaches, focusing instead on training-free methods.

Memory-augmented reasoning. A more recent class of strategies (Suzgun et al., 2025; Ho et al., 2025) builds external memory from past problem-solving experiences: observations, strategies, and reusable concepts are stored and selectively retrieved to guide reasoning on new problems. The paper directly compares against two such methods — Dynamic Cheatsheet (Suzgun et al., 2025) and ArcMemo-PS (Ho et al., 2025) — in Table 3, using o4-mini as the base model. Despite the sophistication of these memory-based approaches, the paper demonstrates that they are outperformed by VLSR+MSSC: on the full ARC-AGI-400, VLSR+MSSC achieves 28.75% versus 27.25% for ArcMemo-PS. The paper's explanation is that "while ArcMemo-PS and Cheatsheet use retrieved past experiences for text-only reasoning, they cannot access the global 2D structure and spatial pattern information that visual representations provide." Memory retrieval helps the model recall what transformations might be relevant, but it does not provide the perceptual immediacy of seeing the spatial pattern directly.

Rule library and DSL approaches. Some strategies pre-define transformation rules using domain-specific languages (DSLs) or enumerated rule libraries, then search over these primitives to find a program that explains the examples. The paper cites Wind et al. (2020), Qiu et al. (2023), and Singhal et al. (2025). These approaches are limited by the expressiveness of the pre-defined rule set — any transformation not representable in the DSL cannot be discovered — and typically require careful engineering of the rule primitives.

Self-correction. The paper engages with a broader literature on self-correction that is directly relevant to its MSSC component. It cites Huang et al. (2023) and Zhang et al. (2024) as establishing that "intrinsic self-correction (without external feedback) is difficult because models struggle to distinguish correct from incorrect outputs when using the same reasoning modality." This prior work documents a fundamental limitation: when a model is asked to check its own answers using the same modality it used to generate them, it exhibits confirmation bias — it tends to endorse its prior outputs rather than identifying errors. The paper's MSSC framework directly addresses this limitation by switching modalities between generation and verification, a design choice explicitly motivated by this prior finding.

How the Paper Positions Itself

The paper positions itself along two key dimensions relative to existing work:

1. Training-free, modality-aware inference. Unlike the dominant synthetic-data fine-tuning paradigm, VLSR and MSSC are training-free strategies that can be applied to any vision-language model at inference time. This makes them immediately deployable with existing frontier models without requiring task-specific training data or compute-intensive fine-tuning. The paper emphasizes this property: "Our approach does not require any training or external data" (even though Section 4.4 later shows the framework extends naturally to fine-tuning with additional gains).

2. Resolving the vision-vs-text tension through strategic decomposition. The paper does not argue that vision is uniformly superior to text — an argument that would be falsified by the catastrophic performance drop when applying rules visually. Instead, it argues for a nuanced position: vision and text have identifiable, complementary strengths that can be exploited through deliberate task decomposition. This positions the paper as offering a resolution to the apparent paradox of naive visual integration, rather than simply advocating for modality substitution.

The paper's central hypothesis, stated in Section 1, captures this positioning:

"vision and language possess complementary strengths across distinct reasoning stages: vision supports global pattern abstraction and verification, whereas language specializes in symbolic rule formulation and precise execution."

This hypothesis is operationalized through the VLSR framework (visual rule summarization → textual rule application) and the MSSC framework (textual generation → visual verification). The empirical evidence in Table 1 provides the foundation for these design choices, demonstrating that the modality-task alignment is not arbitrary but reflects genuine differential capabilities.

A subtle positioning nuance: the paper does not claim to have solved ARC-AGI. The improvements are genuine but modest in absolute terms — the best model (Gemini-2.5-Pro with VLSR+MSSC) achieves 31.25% on ARC-AGI-400, far below human performance of over 97%. The paper is transparent about this, framing its contribution as establishing the principle of vision-language synergy for abstract reasoning and demonstrating its efficacy across multiple models and benchmarks, rather than claiming a breakthrough in absolute ARC-AGI scores. This positions the work as foundational — opening a new research direction — rather than as a final solution.

The Gap This Paper Fills

Synthesizing the above, the specific gap this paper addresses can be stated as:

Prior work treats ARC-AGI as a purely textual reasoning task, despite the fundamentally visual-spatial nature of the benchmark. This choice introduces systematic limitations in 2D structure preservation, encoding efficiency, and holistic pattern recognition that constrain performance. While vision-language models exist and could theoretically process ARC-AGI grids as images, naive visual integration causes catastrophic performance degradation on rule application due to lack of element-wise precision. No prior work has systematically analyzed the complementary strengths of visual and textual modalities in ARC-AGI reasoning stages, nor developed principled methods for strategically combining them. The paper fills this gap by (1) providing the first systematic comparative analysis of modality effects across reasoning subtasks (Table 1, Section 3.2), (2) deriving a set of testable characteristics that explain modality-specific performance patterns (Section 3.2.2), and (3) translating these insights into training-free strategies (VLSR, MSSC) that achieve consistent improvements across diverse models and benchmarks.

3. Technical Approach

3.1 Reader Orientation

This paper introduces a reasoning system for ARC-AGI tasks — not a new model architecture or training procedure — built from two training-free strategies that can be applied to any existing vision-language model at inference time. The core problem it addresses is that visual and textual modalities have fundamentally complementary strengths for abstract reasoning (vision excels at holistic pattern recognition; text excels at precise element-wise manipulation), and the system solves this by decomposing ARC-AGI into modality-aligned subtasks and using cross-modal verification for intrinsic error correction.

3.2 Big-Picture Architecture (Diagram in Words)

The system has two major components that can be used independently or combined, both operating on top of an existing Large Vision-Language Model (LVLM) such as GPT-4o, Gemini-2.5-Pro, o4-mini, or Qwen3-VL:

Vision-Language Synergy Reasoning (VLSR) — a two-phase inference pipeline that routes each reasoning subtask to its optimal modality:
- Phase 1 (Visual Rule Summarization): The example input-output grid pairs are rendered as color-coded images. The LVLM analyzes these images to produce a natural-language description of the transformation rule (e.g., "rotate each connected component 90 degrees clockwise").
- Phase 2 (Textual Rule Application): The extracted rule, along with the example pairs and test input represented as nested lists (text), are fed to the same LVLM. The model performs element-wise manipulation guided by the rule to produce the predicted output matrix in text format.
Modality-Switch Self-Correction (MSSC) — an iterative refinement loop that uses visual verification to correct errors in text-based reasoning:
- Step 1 (Visualization): The predicted output from textual rule application (a nested list) is converted back into a color-coded image. The test input is also visualized.
- Step 2 (Visual Consistency Verification): The visualized test input-output pair is presented alongside the visualized example pairs to the LVLM. The model acts as a critic, judging whether the test pair exhibits the same transformation pattern as the examples.
- Step 3 (Iterative Refinement): If the verification step detects inconsistency, the model receives explicit feedback and re-executes textual rule application with awareness of the previous error. This loop repeats until consistency is achieved or a maximum of 3 iterations is reached.

Information flows as follows: example grid pairs → visual rendering → LVLM → natural language rule → textual example pairs + test input + predicted rule → LVLM textual inference → predicted output (text) → visual rendering of prediction + test input → LVLM visual critic → consistency judgment → (if inconsistent) feedback → LVLM textual re-inference → ... → final predicted output.

3.3 Roadmap for the Deep Dive

First, the formal problem setup and notation (Section 3.1), which establishes the precise mathematical language for ARC-AGI tasks, defines the modality conversion functions ( $\mathcal{T}$ for text, $\mathcal{V}$ for vision), and formalizes the standard text-only baseline — this provides the vocabulary needed to understand why the proposed methods differ.
Second, the comparative modality analysis (Section 3.2), because it is the empirical foundation justifying every design choice in VLSR and MSSC. Understanding the quantitative results (Table 1) and the four qualitative characteristics of modality-specific processing is prerequisite to understanding why the pipeline routes rule summarization to vision and rule application to text.
Third, Vision-Language Synergy Reasoning (VLSR, Section 3.3), which translates the empirical insights into a concrete two-phase inference strategy. I will walk through the modality-switching mechanism, the role of explicit rule extraction, and the precise interface between the visual and textual components.
Fourth, Modality-Switch Self-Correction (MSSC, Section 3.4), which addresses the intrinsic limitation that models cannot effectively verify their own reasoning within a single modality. I will explain the visualization step, the consistency verification prompt, the iterative refinement loop, and why cross-modal verification succeeds where same-modality verification fails.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodology paper whose core idea is that visual and textual modalities have complementary, stage-dependent strengths for abstract reasoning, and that these can be exploited through strategic task decomposition (VLSR) and cross-modal verification (MSSC) without any model training or fine-tuning.

3.1 Problem Setup and Notation (Formalizing the ARC-AGI Task)

Before presenting the method, the paper establishes a precise mathematical framework that makes the distinction between modalities explicit. This formalism is essential because it defines the interface points where modality switching occurs — without it, VLSR and MSSC would be described only in vague terms.

Task formulation. Each ARC-AGI task is defined by a hidden transformation rule $r$ that maps input matrices to output matrices:

$m^{\text{output}} = r(m^{\text{input}})$

where both $m^{\text{input}}$ and $m^{\text{output}}$ are 2D integer matrices with dimensions $H \times W$ where $H, W \leq 30$ , and each cell value belongs to the set $\{0, 1, \ldots, 9\}$ . The model receives $K$ example pairs (typically 2–5 demonstrations):

$\{(m_1^{\text{input}}, m_1^{\text{output}}), (m_2^{\text{input}}, m_2^{\text{output}}), \ldots, (m_K^{\text{input}}, m_K^{\text{output}})\}$

What this means operationally: The system sees a few demonstrations of a transformation (e.g., a grid with colored shapes gets rotated), must infer what the transformation is, and then apply it to a new test input $m_{\text{test}}^{\text{input}}$ to produce $m_{\text{test}}^{\text{output}}$ — without ever being told the rule explicitly. This is few-shot rule induction with zero prior knowledge about the specific transformation class.

Modality encoding functions. The paper introduces two invertible transformation functions that convert matrices between modalities — a critical abstraction that makes modality switching possible:

$\mathcal{T}$ : converts a matrix $m$ to its textual representation $t = \mathcal{T}(m)$ , which is a nested list of integers (e.g., [[0,1,2],[3,4,5],[2,3,5]]).
$\mathcal{V}$ : converts a matrix $m$ to its visual representation $i = \mathcal{V}(m)$ , which is a color-coded 2D grid image where each cell value 0–9 maps to a distinct color (detailed in Appendix B: 0 → black, 1 → blue, 2 → red, 3 → green, 4 → yellow, 5 → grey, 6 → pink, 7 → orange, 8 → light blue, 9 → brown). White dividing lines separate adjacent cells to clarify element boundaries.

Both transformations are invertible: $\mathcal{T}^{-1}(t) = m$ and $\mathcal{V}^{-1}(i) = m$ . This invertibility is essential for MSSC because the text-based output must be converted back to an image for visual verification — without it, the cross-modal loop would have no way to visualize the predicted output.

Previous approaches (text-only baseline). The paper formalizes the standard text-only paradigm as a direct mapping from example pairs and test input to a predicted output, all in text:

$t_{\text{pred}} = f(t_1^{\text{input}}, t_1^{\text{output}}, \ldots, t_K^{\text{input}}, t_K^{\text{output}}, t_{\text{test}}^{\text{input}})$

where $f$ is the language model operating purely in the textual modality, and $t_{\text{pred}}$ is the predicted output matrix as a nested list.

What this equation computes: The function $f$ takes all demonstration pairs (as text) and the test input (as text), concatenates them into a prompt, and autoregressively generates the predicted output matrix. The model may internally perform rule induction and application, but these are conflated into a single forward pass — there is no explicit decomposition into subtasks.

Why this form is limiting: (1) The textual representation $t = \mathcal{T}(m)$ flattens 2D spatial structure into a 1D token sequence, destroying the perceptual contiguity that visual representations preserve. Two vertically adjacent cells in a grid become separated by arbitrary numbers of tokens depending on row width and delimiters. (2) The single-step formulation conflates rule induction (identifying what transformation is occurring) with rule application (computing the output for the test case), preventing the model from leveraging modality-specific strengths for each subtask independently. (3) There is no mechanism for the model to verify its own output against the example patterns — it must get both the rule and its application right in one attempt.

3.2 Comparative Analysis of Vision and Text (The Empirical Foundation)

This section is not yet part of the method — it is the diagnostic experiment that motivates every design choice in VLSR and MSSC. The paper decomposes ARC-AGI into two subtasks (rule summarization and rule application), evaluates each modality's performance on each subtask independently across multiple models, and extracts four qualitative characteristics explaining the observed performance patterns.

Why this analysis matters for the method design. Without this section, VLSR's pipeline (visual summarization → textual application) would be an arbitrary choice. The comparative analysis proves that the mapping from modality to subtask is not arbitrary — it reflects genuine, measurable differential capabilities that are consistent across model families (GPT-4o, Gemini-2.5-Pro, o4-mini).

Experimental Design for the Comparative Analysis

Rule Summarization Phase (testing which modality is better for extracting rules). The paper isolates the rule extraction subtask by varying only the modality used for presenting example pairs:

The example matrix pairs $\{(m_i^{\text{input}}, m_i^{\text{output}})\}$ are presented to the model either as text (nested lists) or as images (color-coded grids).
The model is asked to summarize the transformation rule $r$ in natural language.
To ensure a fair comparison of rule quality independent of application method, both extracted rules are then applied in the textual modality — meaning any performance difference must arise from the quality of the rule itself, not from the application step.
Final accuracy is measured on the test matrix output.

Rule Application Phase (testing which modality is better for applying known rules). The paper isolates the rule application subtask by fixing the rule quality:

The same high-quality rule $r$ — derived from visual summarization (which the previous experiment showed to be superior) — is used for all conditions.
The comparison is between presenting the example matrices and test input as images versus as text when applying this rule.
The question is: given an equally good rule, does visual or textual representation lead to more accurate element-wise execution?

Baseline. A standard text-only approach without explicit rule extraction serves as the reference point — this is the dominant paradigm in prior work.

Quantitative Results (Table 1)

The paper reports results for Gemini-2.5-Pro (the most detailed example), with supporting evidence across other models:

Text-only baseline (no explicit rule extraction): 37.25% accuracy on the evaluation set.
Visual rule summarization + textual rule application: 40.75% accuracy — a 3.5 percentage point improvement over baseline, with the visual rule extraction contributing a 3.0 percentage point gain over text-only rule extraction.
Textual rule summarization + visual rule application: 23.75% accuracy — a catastrophic 17.0 percentage point degradation from the 40.75% achieved with textual application, demonstrating that the quality of the rule cannot compensate for the imprecision of visual execution.

The paper states these results generalize: "using the visual modality for rule summarization provides a clear advantage, yielding an average improvement of 3.2% across models" while "when applying rules using visual representations instead of textual ones, performance drops dramatically by an average of 15.0%." These are not subtle effects — they are large, consistent, and directional, directly informing the VLSR design.

Qualitative Analysis: Four Characteristics of Modality-Specific Reasoning

The paper goes beyond quantitative results to explain why vision and text behave differently, identifying four characteristics through analysis of model outputs (Section 3.2.2). These characteristics are the conceptual backbone of the entire method design, so I will explain each in detail:

Characteristic 1: Holistic vs. independent processing. Visual reasoning demonstrates a bias toward encoding relational properties anchored on contiguous spatial structures — it naturally perceives central blocks, checkerboard patterns, connected components, and other gestalt-like groupings. Textual reasoning, by contrast, relies more on type-level statistics (frequency counts, row-by-row operations) that treat each element largely independently. The paper describes this as visual thinking exhibiting "a systematic bias toward encoding relational properties anchored on contiguous spatial structures," while textual thinking "relies more on type-level statistics (e.g., frequency counts) to identify patterns."

Why this matters for the method: Rule summarization requires identifying global spatial relationships — is the transformation a rotation? A symmetry? A shape-removal operation based on connectivity? The holistic nature of visual perception aligns directly with this requirement, which is why visual rule summarization outperforms textual rule summarization. This characteristic also explains the failure case in Figure 4: GPT-4o in text mode identifies the rule as "select the number with the highest count" (a frequency-based heuristic) because it lacks the holistic perception to see that the task actually requires retaining large connected color blocks (a spatial property).

Characteristic 2: 2D structure preservation vs. loss in text. The paper makes a concrete, measurable claim about tokenization: "In a matrix, two vertically adjacent elements in the same column are perceptually contiguous in a visual representation, yet may be separated by dozens of tokens in a textual representation (e.g., [[0,1,2],[3,4,5]] where elements in different rows are far apart in the token sequence)." This means that spatial relationships that are immediate to visual perception — adjacency, collinearity, enclosed regions — require explicit, error-prone sequential reasoning to extract from text.

The transposition sensitivity argument: The paper further notes that "when both input and output matrices are transposed, rules discovered via visual representation remain essentially invariant, while those derived from textual representation may be significantly affected by the change in token ordering." A rotation or reflection of a grid is perceptually the same transformation whether the grid is transposed or not — but the token sequences change dramatically, and a text-based rule learner may latch onto spurious positional patterns that break under transposition.

Why this matters for the method: VLSR routes the structural pattern recognition phase to vision precisely because text cannot be trusted to preserve 2D relationships. The rule summarization prompt explicitly asks the model to describe spatial transformations, and visual input makes this natural.

Characteristic 3: Encoding efficiency for large grids. ARC-AGI grids can be 30×30 = 900 cells. Textual representation requires encoding each cell value plus delimiters: the example in Appendix A shows [[0, 1, 0, …], [0, 1, 1, …], …], where each row adds brackets, commas, and whitespace. A 30×30 grid could require thousands of tokens, consuming context window and introducing long-range attention challenges. Visual encoding, by contrast, represents the entire grid as a single image using "only a few hundred vision tokens." The paper cites DeepSeek-OCR's work on visual compression for document understanding as corroborating evidence that vision can be more token-efficient for structured spatial content.

Why this matters for the method: This characteristic makes visual encoding not just more accurate but more practical for large grids — a 30×30 task might be infeasible in text due to context length constraints but trivially processable as an image. The VLSR pipeline's visual phase naturally handles these cases without special treatment.

Characteristic 4: Lack of fine-grained element-wise precision in vision. This is the critical limitation that prevents vision from being used for rule application. The paper observes: "Images represent matrices as an integrated whole rather than encoding individual elements separately. While this provides a global perspective advantageous for rule summarization, it becomes inadequate when element-wise processing is required." Concrete failure modes include "confusing the value at position (5,7) with a nearby cell" — the model literally cannot reliably index into a specific grid coordinate from a visual representation.

Why this matters for the method: This characteristic is the reason VLSR must switch to text for rule application. The rule "rotate the blue object 90 degrees clockwise" requires computing, for every cell in the output grid, what value should go there based on spatial transformations of input cells. This demands per-element indexing and value assignment that text handles naturally (the i-th element of the j-th row is unambiguously the integer at that position in the nested list) but vision handles poorly. The MSSC framework further exploits this limitation by using vision's strength (pattern consistency checking) to verify text's output, rather than asking vision to produce the output directly.

3.3 Vision-Language Synergy Reasoning (VLSR)

Motivation and Design Principle

VLSR is the direct operationalization of the empirical findings from Section 3.2. The design principle is stated explicitly:

"Our key design principle is to route each sub-task to its optimal modality: visual reasoning for rule summarization (exploiting global pattern recognition and 2D structural understanding) and textual reasoning for rule application (exploiting precise element-wise manipulation)."

The paper frames this as addressing two limitations of the text-only baseline (Equation 1): (1) the baseline conflates rule summarization and application into a single step, preventing the model from leveraging modality-specific strengths, and (2) the baseline uses text for everything, losing 2D structural information. VLSR addresses both through task decomposition and modality matching.

Phase 1: Visual Rule Summarization

Input preparation. All example matrix pairs $\{(m_i^{\text{input}}, m_i^{\text{output}})\}$ are converted to visual form using the visualization function $\mathcal{V}$ . The paper's Appendix B specifies the visualization process:

Each cell value 0–9 maps to a distinct color: 0 → black, 1 → blue, 2 → red, 3 → green, 4 → yellow, 5 → grey, 6 → pink, 7 → orange, 8 → light blue, 9 → brown.
White dividing lines are added between adjacent cells to "more clearly indicate the specific number and the structure of the elements contained within a block." This is a subtle but important implementation detail: without dividing lines, adjacent cells of the same color would merge into an undifferentiated block, losing the grid structure. The lines preserve the discrete nature of the grid while maintaining the spatial layout.

Rule extraction. The LVLM analyzes these visualized examples and produces an explicit transformation rule $r_{\text{pred}}$ expressed in natural language:

$r_{\text{pred}} = f^{\text{vision}}_{\text{sum}}(i_1^{\text{input}}, i_1^{\text{output}}, i_2^{\text{input}}, i_2^{\text{output}}, \ldots, i_K^{\text{input}}, i_K^{\text{output}})$

where $f^{\text{vision}}_{\text{sum}}$ is the LVLM operating in visual mode with a rule summarization prompt, $i_j^{\text{input}} = \mathcal{V}(m_j^{\text{input}})$ is the visualized input grid for example $j$ , $i_j^{\text{output}} = \mathcal{V}(m_j^{\text{output}})$ is the visualized output grid for example $j$ , and $K$ is the number of demonstration pairs.

What this equation computes: The function takes $K$ images of input grids and their corresponding $K$ images of output grids, processes them through the LVLM's vision encoder and language decoder, and produces a natural language string $r_{\text{pred}}$ that describes the transformation rule. The model is explicitly prompted to output the rule within \boxed{} delimiters, as shown in Appendix A: "Output the rule you learned within \boxed{}."

Why extract the rule explicitly (not implicitly)? The standard text-only approach in Equation 1 does not require the model to articulate the rule — it can produce the output directly through implicit reasoning. VLSR deliberately forces explicit rule extraction for three reasons: (1) it creates a clean interface between the visual summarization and textual application phases (the rule is the "message" passed between them), (2) it enables the textual application phase to verify the rule against the examples before applying it (the prompt in Appendix A says "You need to first check the correctness of the rule based on the examples"), and (3) it makes the reasoning process interpretable — human evaluators can inspect the extracted rule to diagnose failures.

Why visual input for this phase? The answer comes directly from Characteristics 1–3 in Section 3.2.2. Visual representation provides holistic perception of spatial patterns (Characteristic 1), preserves 2D structure (Characteristic 2), and efficiently encodes large grids (Characteristic 3). These properties align with the requirements of rule summarization: identifying global transformation patterns like rotations, symmetries, shape manipulations, and connectivity-based operations.

The prompt structure. Appendix A shows the vision-centric rule summarization prompt:

"I will now provide you with several input and output images about 2D grids. You need to summarize the grid-changing rule from it. Output the rule you learned within \boxed{}."

Example Input 1: <Input Image 1> Example Output 1: <Output Image 1> ...

The prompt is deliberately minimal — it asks for rule summarization without providing any strategy hints or decomposition instructions. This tests the model's genuine ability to extract spatial transformations from visual examples, rather than following a pre-specified reasoning template.

Phase 2: Textual Rule Application

Input preparation. The extracted rule $r_{\text{pred}}$ (a natural language string) is combined with the example pairs and test input, all converted to text format using $\mathcal{T}$ :

$t_{\text{pred}} = f^{\text{text}}_{\text{app}}(r_{\text{pred}}, t_1^{\text{input}}, t_1^{\text{output}}, \ldots, t_K^{\text{input}}, t_K^{\text{output}}, t_{\text{test}}^{\text{input}})$

where $f^{\text{text}}_{\text{app}}$ is the same base LVLM — not a different model — operating in text mode with a rule application prompt, $t_j^{\text{input}} = \mathcal{T}(m_j^{\text{input}})$ is the textual input grid for example $j$ , $t_j^{\text{output}} = \mathcal{T}(m_j^{\text{output}})$ is the textual output grid for example $j$ , $t_{\text{test}}^{\text{input}} = \mathcal{T}(m_{\text{test}}^{\text{input}})$ is the textual test input, and $t_{\text{pred}}$ is the predicted output matrix as a nested list.

What this equation computes: The function takes the natural language rule, $K$ demonstration pairs as nested lists, and the test input as a nested list, processes them through the LVLM's text-only reasoning capability, and produces a predicted output grid as a nested list. The model is prompted to verify the rule against the examples first, then apply it to the test input.

Why textual input for this phase? The answer comes from Characteristic 4 in Section 3.2.2: visual reasoning lacks fine-grained element-wise precision. Textual representation provides unambiguous indexing — the value at row $i$ , column $j$ is explicitly the $j$ -th integer in the $i$ -th sublist, with no possibility of confusing it with nearby cells. Applying a rule like "fill all cells adjacent to red objects with yellow" requires computing for each cell whether it satisfies the adjacency condition, which demands precise per-element access that text provides natively.

The prompt structure and rule verification. The text-centric rule application prompt in Appendix A is more elaborate than the summarization prompt, reflecting the complexity of the application step:

"I will provide you with several input and output matrices. You need to find the matrix-changing rule from it and apply it to the new input. Put the output matrix within \boxed{}."

"Here is a possible rule for your reference. Rule: [THE VISUALLY EXTRACTED RULE] ... Note that the rule is described in color and each color represents a value in the matrix: [0:black; 1:blue; 2:red; 3:green; 4:yellow; 5:grey; 6:pink; 7:orange; 8:light blue; 9:brown]. You need to first check the correctness of the rule based on the examples. If the rule is correct, apply it to the new input. Otherwise, summarize a new rule and apply it to the new input."

This prompt design incorporates several important features:

Rule verification gate: The model is explicitly instructed to check the visually extracted rule against the textual examples before applying it. This is a safety mechanism — if the visual rule extraction produced an incorrect rule, the textual phase can detect the inconsistency and recover by summarizing a new rule.
Color-to-value mapping: Since the rule was extracted from visual representations using color descriptions (e.g., "retain large connected color blocks"), the prompt provides the explicit mapping from color names back to numeric values. This is essential because the textual application phase operates on integers, not colors — without this mapping, the rule would be inapplicable.
Fallback to independent reasoning: The instruction "Otherwise, summarize a new rule and apply it to the new input" means that even if the visual rule extraction fails, the system degrades gracefully — the textual phase can function as a standalone text-only reasoner, ensuring performance never falls below the baseline.

Key difference from the baseline (Equation 1). The baseline $t_{\text{pred}} = f(t_1^{\text{input}}, t_1^{\text{output}}, \ldots, t_{\text{test}}^{\text{input}})$ provides no explicit rule and no modality switching. VLSR's formulation $t_{\text{pred}} = f^{\text{text}}_{\text{app}}(r_{\text{pred}}, \ldots)$ adds the rule $r_{\text{pred}}$ as an explicit input, where $r_{\text{pred}}$ itself was produced by a different modality pass ( $f^{\text{vision}}_{\text{sum}}$ ). This is the mechanism by which VLSR harnesses complementary modalities: the visual pass contributes better pattern recognition, and the textual pass contributes better element-wise execution, with the rule serving as the information bridge between them.

Key Advantages of the Two-Phase Design

The paper identifies two benefits over the single-phase text-only baseline:

Task decomposition reduces individual subtask complexity. Instead of asking the model to simultaneously induce a rule and apply it (a cognitively demanding composite operation), VLSR separates these into distinct stages, each with a focused prompt and clear objective. The divide-and-conquer strategy means errors in rule extraction can be caught at the verification gate before they propagate to application.
Modality matching exploits each modality's strengths. The paper's empirical analysis (Section 3.2) demonstrated that rule summarization benefits from visual input while rule application benefits from textual input. VLSR implements this directly — visual for Phase 1, text for Phase 2 — rather than forcing a compromise modality that would be suboptimal for both subtasks.

A subtle implementation note: The paper emphasizes that $f^{\text{vision}}_{\text{sum}}$ and $f^{\text{text}}_{\text{app}}$ are "the same base model; only the input modality and prompting strategy differ." This means VLSR does not require model ensembling or separate specialized models — it is a purely inference-time strategy that can be applied to any LVLM without modification. The gains come from how the model is used, not from changing what the model is.

3.4 Modality-Switch Self-Correction (MSSC)

Motivation: The Failure of Same-Modality Self-Correction

MSSC addresses a well-documented limitation in LLM reasoning: intrinsic self-correction (having a model check its own work without external feedback) is largely ineffective because "models struggle to distinguish correct from incorrect outputs when using the same reasoning modality" (Huang et al., 2023; Zhang et al., 2024). The paper's Table 4 quantifies this for ARC-AGI: text-only self-correction (TOSC) applied to GPT-4o produces only a 0.5 percentage point improvement over three rounds (8.25 → 8.75), with some rounds actually degrading performance (Round 2 drops to 8.0).

The confirmation bias problem. When a model generates a candidate output through textual reasoning and then re-examines that output using textual reasoning, it tends to endorse its prior reasoning rather than identify errors. The paper describes this as "confirmation bias" — the same reasoning patterns that produced the error are recruited to evaluate it, creating a self-consistent but wrong loop.

The core insight of MSSC is that using a different modality for verification than for generation disrupts confirmation bias. The paper explains:

"The key insight is that visual and textual modalities have complementary verification capabilities: while text excels at forward rule application, vision excels at pattern consistency verification. By switching modalities, the model gains a fresh perspective that enables it to identify errors that are imperceptible in the original modality."

This is not merely speculative — it is grounded in the four characteristics from Section 3.2.2. Vision's holistic perception (Characteristic 1) makes it naturally suited to checking whether a transformation "looks right" — whether the predicted output maintains the same spatial patterns, symmetries, and gestalt properties as the example outputs. Text, which processes elements independently, may generate a cell-by-cell correct output that nonetheless violates global pattern constraints, and then fail to detect this violation when re-reading its own text output.

The MSSC Pipeline (Figure 3b)

The self-correction loop has three steps, executed iteratively:

Step 1: Visualization of the predicted output. The candidate output $t_{\text{pred}}$ from textual rule application (a nested list string) is parsed back into matrix form $m_{\text{pred}}$ , then converted to a visual representation using the same visualization function $\mathcal{V}$ :

$i_{\text{test}}^{\text{input}} = \mathcal{V}(t_{\text{test}}^{\text{input}}), \quad i_{\text{pred}} = \mathcal{V}(t_{\text{pred}})$

where $i_{\text{test}}^{\text{input}}$ is the visualized test input grid and $i_{\text{pred}}$ is the visualized predicted output grid.

What this step computes: It takes two textual nested lists (the test input and the candidate output) and renders them as color-coded grid images using the same color mapping and white dividing lines specified in Appendix B. This creates a visual pair $(i_{\text{test}}^{\text{input}}, i_{\text{pred}})$ that can be compared against the example visual pairs.

Why this step is necessary: The verification step operates in vision, so the text output must be converted back to an image. The invertibility of $\mathcal{V}$ and $\mathcal{T}$ (established in Section 3.1) guarantees this conversion is lossless — the image represents the exact same grid as the text, just in a different format. There is no information lost or added in the conversion.

Step 2: Visual consistency verification. The visualized test pair is presented alongside the visualized example pairs to the LVLM acting as a critic:

$s_{\text{consistent}} = f^{\text{vision}}_{\text{critic}}(i_1^{\text{input}}, i_1^{\text{output}}, \ldots, i_K^{\text{input}}, i_K^{\text{output}}, i_{\text{test}}^{\text{input}}, i_{\text{pred}})$

where $s_{\text{consistent}} \in \{\text{yes}, \text{no}\}$ is a binary judgment of whether the test pair follows the same transformation pattern as the example pairs.

What this equation computes: The critic function takes all $K$ example pairs as images plus the test input-output pair as images, and produces a binary consistency judgment. The model is prompted (Appendix A) to "determine whether the new pair also follows the same changing rule" and output its judgment within \boxed{True} or \boxed{False}.

Why visual verification? The paper argues that visual verification can detect pattern inconsistencies that text-based verification misses. For example, if the transformation is "rotate the blue object 90 degrees clockwise and then mirror across the vertical axis," a text-based output might correctly place individual elements but produce an overall arrangement that "looks wrong" — the global gestalt mismatches the examples. Vision, with its holistic perception, can flag this inconsistency even when each local element is technically correctly placed.

Step 3: Iterative refinement with error awareness. If $s_{\text{consistent}} = \text{no}$ , the model receives explicit feedback about the inconsistency and performs another round of textual reasoning:

$t_{\text{pred}} \leftarrow f^{\text{text}}_{\text{inf}}(r_{\text{pred}}, t_1^{\text{input}}, t_1^{\text{output}}, \ldots, t_{\text{test}}^{\text{input}}, \text{feedback}_{\text{prev}})$

where $\text{feedback}_{\text{prev}}$ contains information about the previous attempt's inconsistency, and the arrow $\leftarrow$ denotes updating the predicted output with the new candidate.

What this equation computes: The same textual inference function is called again, but with additional information — feedback indicating that the previous output was visually inconsistent with the example patterns. This feedback steers the model away from the prior incorrect reasoning path, effectively implementing a form of error-aware regeneration.

Iteration limit and termination. The process repeats — visualize new prediction, verify visually, refine if inconsistent — until $s_{\text{consistent}} = \text{yes}$ or a maximum number of iterations $N_{\max} = 3$ is reached. The final prediction is the last generated textual output. The choice of $N_{\max} = 3$ represents a compute-quality tradeoff: more iterations could potentially yield further improvements but at linearly increasing inference cost, and the paper's results in Table 4 show diminishing returns (e.g., for GPT-4o: +2.0 in Round 1, +1.25 in Round 2, +0.5 in Round 3).

Why MSSC Works When Same-Modality Self-Correction Fails

The paper attributes MSSC's success to two factors:

Fresh perspective through modality switching. When the model verifies its textual output using vision, it engages fundamentally different perceptual mechanisms — spatial pattern recognition rather than sequential token comparison. This breaks the confirmation bias loop because the verification "perspective" is genuinely different from the generation perspective, making errors that were invisible in text become visible in vision. The paper's qualitative examples (Appendix C, Figures 5–8) illustrate this: Figure 7 shows a case where Gemini-2.5-Pro's visual thinking adopts a "global perspective and thus processes information based on internal and external paths; in contrast, textual thinking focuses more on local information and processes individual elements with reference to their 8-neighbor context." The visual verifier can detect that the local element-by-element approach produced a globally inconsistent result, even though each local operation was individually correct.
No external information needed. MSSC is purely intrinsic — it uses only the model's own multimodal capabilities, without ground-truth labels, external verifiers, or additional training data. This is important because it makes MSSC applicable in exactly the same settings as the base model: any task where an LVLM can process both images and text can use MSSC, with no additional infrastructure.

Comparison with Text-Only Self-Correction (Table 4 Results)

The paper's empirical comparison in Table 4 demonstrates the dramatic difference between same-modality and cross-modal correction:

GPT-4o: TOSC improves from 8.25 → 8.75 across three rounds (total +0.5), with Round 2 degrading to 8.0 before recovering. MSSC improves from 8.25 → 10.25 → 11.5 → 12.0 (total +3.75), with monotonic gains at every round.
Gemini-2.5-Pro: TOSC shows minimal improvement. MSSC achieves +1.5 total across three rounds.
o4-mini: TOSC shows minimal improvement. MSSC achieves +2.5 total across three rounds.

The paper's explanation, directly from Section 4.3:

"When TOSC uses the same textual modality for both generation and verification, the model exhibits confirmation bias and cannot effectively spot its own errors. By contrast, MSSC's visual verification stage detects pattern violations (e.g., missing symmetry, incorrect spatial relationships) that the textual reasoning stage overlooked, enabling genuine iterative improvement without external feedback."

A subtle point about the verification task: The visual critic is not asked to identify what is wrong with the output — only whether it is consistent with the examples. This is a binary classification task, which is substantially easier than the generative task of producing the correct output. The critic can flag inconsistency without being able to solve the problem itself, which is precisely the asymmetry that makes self-correction possible: verification is easier than generation, but only when using a modality that is aligned with the verification subtask's requirements (holistic pattern matching).

4. Key Insights and Innovations

Innovation 1: The Modality Complementarity Thesis as an Empirically Grounded Diagnostic Framework

The paper's most fundamental intellectual contribution is not VLSR or MSSC as mechanisms—those are the engineered consequences—but rather the systematic, empirically validated thesis that vision and language possess identifiable, complementary strengths mapping onto distinct reasoning stages, and that ignoring this complementarity explains both the failure of text-only approaches (loss of spatial structure) and the failure of naive visual approaches (loss of element-wise precision). This transforms a vague intuition ("maybe vision would help") into a precise, testable framework with four enumerated characteristics.

What the field assumed before this work. The dominant paradigm treated modality choice as binary and uniform: either process ARC-AGI as text (the overwhelming default, from synthetic data fine-tuning approaches like Li et al. 2024 and Hodel et al. 2024 to memory-augmented methods like ArcMemo-PS and Dynamic Cheatsheet) or render it as images and hope vision-language models perform well. The possibility that different stages of the same task might benefit from different modalities—and that a single-modality approach necessarily underperforms because no single modality dominates across all reasoning substages—was not articulated, let alone empirically characterized. Prior work on aiding reasoning with images (Visual Sketchpad, ViLaSR) demonstrated that visual information could augment specific reasoning types (geometry, spatial planning), but did not provide a general diagnostic framework for when vision helps versus hurts, or why.

Why this is a reframing, not just a finding. The paper's four characteristics (Section 3.2.2)—holistic vs. independent processing, 2D structure preservation vs. loss, encoding efficiency, and element-wise precision trade-offs—constitute a diagnostic vocabulary that generalizes beyond ARC-AGI. Any reasoning task involving spatial, relational, or structural information can now be analyzed through this lens: where does the task require holistic pattern recognition? Where does it require precise element-wise indexing? The answer dictates modality allocation. This is analogous to how the Chinchilla scaling laws (Hoffmann et al., 2022) provided a vocabulary for reasoning about pretraining compute allocation—not just a specific recipe, but a conceptual framework that changes how researchers design experiments and interpret results.

Evidence anchoring. The quantitative demonstration in Table 1 is essential: the same base model (Gemini-2.5-Pro) shows a 3.5 percentage point gain from visual rule summarization but a 17 percentage point loss from visual rule application. These are not subtle, overlapping confidence-interval effects—they are large, directionally opposite effects that could only arise from genuinely different modality-task alignments. The fact that this pattern replicates across models (GPT-4o, o4-mini, Gemini-2.5-Pro) and the qualitative examples in Appendix C (Figures 5–8) showing systematic failure modes in text-only reasoning (frequency-based heuristics instead of spatial pattern recognition, row-by-row processing instead of block-by-block, inability to capture long-range spatial correlations) provide converging evidence that the four characteristics describe real processing differences, not idiosyncratic model behavior.

Distinguishing incremental from fundamental. This is a fundamental conceptual advance, not an incremental refinement. Prior work had no framework for reasoning about modality-specific strengths in abstract reasoning tasks. The paper provides one, and demonstrates that operationalizing it (via VLSR and MSSC) yields consistent improvements across diverse models and benchmarks without training. This is the kind of contribution that reshapes how a research community thinks about a problem, regardless of the absolute performance numbers achieved.

The paper's second major conceptual move is identifying that the well-documented failure of intrinsic self-correction is modality-dependent, and that switching modalities between generation and verification can enable effective self-correction without external feedback. This reframes self-correction from a model capability problem ("models aren't good enough at self-critique") to a representation alignment problem ("models can't critique their own outputs when using the same perceptual encoding they used to generate them").

What the field assumed before this work. The dominant narrative, established by Huang et al. (2023) and reinforced by Zhang et al. (2024), was that LLMs cannot effectively self-correct on reasoning tasks because they exhibit confirmation bias—they tend to endorse their own outputs rather than identify errors. This was treated as a fundamental limitation of current architectures: if a model could identify its errors, the reasoning goes, it would have generated the correct answer in the first place. The practical consequence has been that self-correction research has largely focused on external feedback sources (verifier models, execution traces, ground-truth checking) rather than intrinsic mechanisms. The paper cites these works explicitly in Sections 2 and 3.4, positioning MSSC as a direct response to this apparent impasse.

The reframing: confirmation bias is modality-specific, not absolute. MSSC's core insight is that confirmation bias arises not from an inherent inability to self-critique, but from the model processing its own output through the same perceptual pathway that generated it. The text-based reasoning that produced an error engages the same sequential, element-wise processing patterns when re-reading the output, and thus "sees" the same (incorrect) structure it originally computed. Vision, with its holistic spatial perception, provides a genuinely different perspective—one that can detect global pattern violations (missing symmetries, spatial inconsistencies, gestalt mismatches) that are invisible to text-based re-reading.

This is not merely a "different prompt" approach to self-correction. It's a claim about representational independence: the verification signal must come from a processing pathway that is not a mechanistic duplicate of the generation pathway. This connects to broader principles in machine learning (ensemble diversity, adversarial detection via representation disagreement) but applies them to the specific architecture of multimodal reasoning in a novel way.

Why this matters beyond ARC-AGI. The principle that cross-modal verification enables self-correction has implications for any domain where (a) tasks can be represented in multiple modalities and (b) the modalities have different failure modes. Code generation with visual execution trace verification, mathematical reasoning with geometric diagram checking, and structured planning with spatial layout validation all fit this pattern. The paper's specific finding—that verification is easier than generation when using an appropriate modality—suggests a general design principle for self-improving systems: invest in diverse representational capabilities not just for solving tasks, but for verifying solutions.

Evidence anchoring. Table 4 is the critical evidence: same-modality self-correction (TOSC) produces essentially zero improvement (GPT-4o: +0.5 across three rounds, with degradation in Round 2) while cross-modal correction (MSSC) produces consistent, monotonic gains (GPT-4o: +3.75 across three rounds, every round improving). The monotonicity is as important as the magnitude—it demonstrates that the cross-modal verification signal is genuinely informative, not just randomly perturbing the model into occasionally better outputs. The qualitative examples in Appendix C (particularly Figure 7, showing visual thinking detecting global pattern violations that text-based local processing produced) provide the mechanistic explanation for why the verification signal works.

Distinguishing incremental from fundamental. This is a fundamental reframing of the self-correction problem. Prior work asked "how can we make models better at self-critique?" and largely concluded they can't (intrinsically). This paper asks "under what conditions can models self-critique effectively?" and answers: when the critique modality differs from the generation modality. This transforms self-correction from a binary capability that models either possess or lack into a design problem—one of selecting appropriate verification representations.

Innovation 3: The Paradox of Naive Visual Integration as a Productive Negative Result

Most papers treat negative results as obstacles to overcome or explain away. This paper treats its negative result—that naively rendering ARC-AGI grids as images degrades performance—as a productive diagnostic that reveals the non-uniformity of modality effects and directly motivates the decomposition framework. The negative result is not a failure; it is the empirical pivot that transforms the research question.

What the field assumed before this work. The field has a strong default assumption when introducing visual representations to reasoning tasks: if vision helps at all, it helps uniformly. Visual Sketchpad (Hu et al., 2024) and ViLaSR (Wu et al., 2025) both demonstrate that adding visual information improves spatial reasoning, and the natural extrapolation would be that rendering ARC-AGI grids as images should provide similar benefits. This assumption is so ingrained that the paper's counter-result—visual rule application causes a catastrophic 17 percentage point drop—would be genuinely surprising to most practitioners. The fact that no prior work had systematically tested this (despite the obviousness of the idea "just use a vision-language model") suggests the assumption was simply accepted without empirical scrutiny.

Why the negative result is intellectually productive. The paradox—vision helps rule summarization but devastates rule application—forces a refinement of the research question from "should we use vision?" to "when and how should we combine vision and text?" This is a qualitative shift in the nature of the inquiry, from a binary choice to a resource allocation problem. Without the negative result, VLSR's decomposition would appear arbitrary or over-engineered; with it, VLSR appears as the only coherent response to the empirical evidence. The negative result also provides the cleanest possible demonstration of the modality complementarity thesis: if vision were uniformly helpful, there would be no complementarity to exploit—just modality substitution. The fact that vision simultaneously helps one subtask and hurts another is what creates the opportunity for strategic combination.

A pattern for future research. The paper models a methodological approach that is underutilized in ML research: systematically decompose a task into subtasks, test each modality on each subtask independently, and use the pattern of results to design a modality-routing strategy. This "diagnostic decomposition" is generalizable to any multimodal reasoning problem and provides a template for moving beyond "should we add modality X?" to "where in the reasoning pipeline does modality X add value, and where does it subtract?"

Evidence anchoring. Table 1 is the central exhibit. The 17 percentage point drop on rule application is large enough that it cannot be attributed to noise or experimental variation—it demands explanation. The four characteristics in Section 3.2.2 provide that explanation, specifically Characteristic 4 (lack of fine-grained element-wise precision in vision) which directly accounts for the application-phase degradation. The qualitative examples in Appendix C, Figures 5–8, provide additional mechanistic evidence by showing specific cases where text-only reasoning produces systematically different (and incorrect) rule interpretations compared to visual reasoning.

Distinguishing incremental from fundamental. This is a methodological contribution rather than an algorithmic one: the paper demonstrates the value of systematic modality ablation in understanding reasoning system design. The negative result itself is not fundamental (it's specific to current vision-language model capabilities and might change as models improve at fine-grained visual indexing), but the method of using negative results to diagnose complementarity is a generalizable research practice that the field would benefit from adopting more broadly.

Innovation 4: Explicit Rule Extraction as a Modality-Interface Mechanism

A subtle but important conceptual move in VLSR is the deliberate externalization of the transformation rule as a natural language string that serves as the interface between the visual and textual reasoning phases. This is not merely a prompt engineering detail—it is a claim about how to architect multimodal reasoning systems so that different modalities can contribute their complementary strengths without requiring end-to-end multimodal fusion.

What the field assumed before this work. Most multimodal reasoning approaches fall into two categories: (1) end-to-end fusion, where vision and language representations are combined in a shared latent space and reasoning proceeds jointly (standard LVLM operation), or (2) modality substitution, where one modality is converted to another and all reasoning occurs in a single modality (the text-only ARC-AGI paradigm). VLSR introduces a third category: modality-sequential reasoning with explicit symbolic intermediate representations. The visual phase produces a symbolic output (the natural language rule), which then serves as privileged input to the textual phase. The rule is not a latent vector or an attention pattern—it is a human-readable, verifiable, and editable artifact that can be inspected, validated, and potentially corrected.

Why this architectural choice matters. The explicit rule extraction serves multiple functions that are invisible in end-to-end approaches: (1) it creates a clean interface between modality-specific processing stages, decoupling the visual pattern recognition from the textual manipulation so that improvements to one phase do not require retraining the other; (2) it enables verifiable intermediate outputs—the rule can be checked against the examples (which the prompt explicitly instructs the model to do) before being applied, catching errors at the summarization stage before they propagate; (3) it provides interpretability and debuggability—when VLSR fails, a human can inspect the extracted rule to determine whether the error originated in summarization or application; and (4) it enables graceful degradation—the textual phase is instructed to verify the rule and fall back to independent reasoning if the rule is incorrect, meaning VLSR's worst-case performance is bounded below by the text-only baseline.

This design principle—using natural language as an explicit modality bridge—connects to broader ideas in AI about symbolic interfaces between subsymbolic components, but applies them specifically to the problem of multimodal reasoning stage decomposition.

Evidence anchoring. The prompt structures in Appendix A are the key evidence. The rule summarization prompt explicitly asks for the rule "within \boxed{}," and the rule application prompt explicitly instructs the model to "first check the correctness of the rule based on the examples. If the rule is correct, apply it to the new input. Otherwise, summarize a new rule and apply it." This is not an accident of prompt phrasing—it's a deliberate architectural decision that creates a verification gate and a fallback mechanism. The fact that VLSR (with its explicit rule) outperforms the text-only baseline (which may implicitly induce a rule but doesn't externalize it) by an average of 3.02% (Table 2) provides evidence that the externalization contributes to performance, not just interpretability.

Distinguishing incremental from fundamental. This is a design principle contribution—it identifies a general architectural pattern (explicit symbolic intermediate representations as modality bridges) that is likely applicable to other multimodal reasoning tasks. While not as conceptually novel as the modality complementarity thesis or cross-modal verification insight, it is an important engineering contribution that addresses a practical challenge: how do you actually build systems where different modalities contribute to different reasoning stages without requiring monolithic end-to-end fusion models?

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary benchmark is the official ARC-AGI evaluation set (Chollet, 2019), consisting of 400 tasks. The paper also evaluates on two supplementary benchmarks to test generalization: 100 randomly sampled tasks from Re-ARC (Hodel et al., 2024) and 100 randomly sampled tasks from BARC (Li et al., 2024). Each sampled task from Re-ARC and BARC contains four input-output pairs: three examples and one test instance. For the fine-tuning experiments in Section 4.4, training data comes from ARC-Heavy-200k (Li et al., 2024), which provides approximately 200,000 synthetic ARC-AGI tasks with ground-truth rules, with a held-out 100-task test set.
Base model(s). The paper evaluates both open-source and closed-source vision-language models spanning a wide capability range. Closed-source models: GPT-4o (Hurst et al., 2024), Gemini-2.5-Pro-Thinking-8192 (Comanici et al., 2025), and o4-mini (OpenAI, 2025). Open-source models: Qwen3-VL-235B-A22B-Instruct (Yang et al., 2025) for inference experiments, and Qwen3-VL-8B-Instruct / Qwen3-8B for fine-tuning experiments. The paper does not explicitly justify why these specific models were chosen beyond their status as "flagship reasoning models" (Section 1) and the desire to demonstrate consistency across diverse model families and scales. The choice of o4-mini for the comparison against training-free methods (Table 3) is justified by it being a state-of-the-art reasoning model at the time of writing.
Metrics. All experiments report Pass@1 accuracy — the fraction of tasks for which the model's first generated output matches the ground-truth answer exactly, with no retries or selection among multiple candidates. Accuracy is computed at temperature 0.7 across all experiments. For the fine-tuning experiments in Section 4.4, the metric is similarly accuracy on the held-out test set. The paper does not report confidence intervals, statistical significance tests, or variance estimates for any reported numbers, which is a notable omission given the relatively small test set sizes (400 tasks for ARC-AGI, 100 each for Re-ARC and BARC).
Baselines. The paper evaluates against several categories of baselines:
- Text-only reasoning: The standard approach formalized in Equation 1 — processing ARC-AGI purely through nested list representations with no explicit rule extraction and no visual information. This is the primary baseline used throughout Tables 1, 2, and 4.
- Text-only self-correction (TOSC): Applying traditional same-modality self-correction where the model checks its own textual output and regenerates if errors are detected. Compared against MSSC in Table 4. The paper does not specify the exact prompt used for TOSC, which makes the comparison somewhat opaque — TOSC's poor performance could reflect suboptimal prompting rather than a fundamental limitation.
- Training-free memory-augmented methods (Table 3): Dynamic Cheatsheet (Suzgun et al., 2025), which stores strategies and findings from past problem-solving as retrievable memory, and ArcMemo-PS (Ho et al., 2025), which builds concept-level external memory through program synthesis and selectively retrieves reusable abstractions. Both operate purely in text.
- Text-only fine-tuning (Table 5): Fine-tuning Qwen3-8B on the same 200k ARC-Heavy-200k tasks using textual matrix representations, serving as the baseline for the vision-language synergy fine-tuning experiment.
- Closed-source models without fine-tuning (Table 5): GPT-4o (8.25%) and other models serve as baselines to contextualize the fine-tuned open-source model's performance.
Several natural baselines are notably absent: (1) a vision-only baseline where both rule summarization and rule application use visual representations, which would provide the full picture of single-modality performance; (2) an ablation where VLSR uses text for rule summarization and vision for rule application (the reverse of the proposed pipeline), which would test whether the specific routing direction matters or merely the decomposition itself; (3) a random-modality-switching baseline to control for the possibility that any perturbation of the reasoning process (not specifically vision-language synergy) produces the observed gains.
Generation budget / compute accounting. The paper does not provide a formal compute budget or FLOPs accounting for the inference-time methods. This is a significant gap because VLSR requires two separate model calls (visual summarization + textual application), and MSSC adds up to three additional visual verification + textual re-inference cycles. Compared to the text-only baseline (one model call), VLSR at minimum doubles the inference compute, and VLSR+MSSC can multiply it by up to 5× (one visual summarization, then up to four rounds of verification + re-inference). The paper reports accuracy improvements without explicitly accounting for this increased inference cost, making it impossible to assess whether the gains represent genuine efficiency improvements or simply reflect additional computation. The paper's framing of these as "training-free" methods obscures the fact that they are not computation-free — they trade training cost for inference cost, but the inference cost multiplier is never quantified.
Cross-validation / statistical protocol. The paper reports no cross-validation, statistical significance testing, or confidence intervals for any experimental results. All numbers are reported as point estimates. For the fine-tuning experiments (Section 4.4), the paper uses a held-out 100-task test set from ARC-Heavy-200k. For the main ARC-AGI evaluation, the full 400-task official evaluation set is used. Given that a 1% accuracy difference on a 400-task set represents only 4 tasks, the reported improvements — while consistent across models and benchmarks — could be sensitive to small variations in task sampling. The absence of variance estimates or significance tests means readers cannot assess whether, for example, the 1.5% improvement over ArcMemo-PS on ARC-AGI-400 (Table 3, 28.75% vs. 27.25%) is statistically reliable or within the range of run-to-run variation at temperature 0.7.

Main Quantitative Results

VLSR and MSSC Performance Across Models and Benchmarks (Table 2)

The headline result is that VLSR and MSSC each provide consistent improvements over text-only baselines across all tested models and benchmarks, with the combination yielding the largest gains.

Individual strategy contributions. On ARC-AGI-400, VLSR alone improves over the text-only baseline by an average of 3.02 percentage points across models. MSSC applied on top of VLSR provides an additional 1.82 percentage point average improvement. The full VLSR+MSSC pipeline yields improvements of:

GPT-4o: 8.25% (text-only) → 14.5% (VLSR+MSSC), a gain of 6.25 percentage points.
Gemini-2.5-Pro: 24.0% (text-only) → 31.25% (VLSR+MSSC), a gain of 7.25 percentage points. This is the largest absolute improvement reported.
o4-mini: 25.25% (text-only) → 29.75% (VLSR+MSSC), a gain of 4.5 percentage points.
Qwen3-VL: 18.75% (text-only) → 21.0% (VLSR+MSSC), a gain of 2.25 percentage points.

Benchmark generalization. The improvements generalize beyond ARC-AGI to the supplementary benchmarks:

Re-ARC (100 tasks): Average improvement of 3.02% for VLSR and 1.82% additional for MSSC. For GPT-4o, text-only achieves 19.5% while VLSR+MSSC achieves 25.5% (+6.0). For Gemini-2.5-Pro: 36.0% → 42.0% (+6.0). For o4-mini: 38.5% → 42.5% (+4.0).
BARC (100 tasks): Similar pattern, with GPT-4o improving from 17.0% to 24.0% (+7.0), Gemini-2.5-Pro from 39.0% to 45.5% (+6.5), and o4-mini from 40.5% to 47.0% (+6.5).

Key observations from Table 2. (1) The gains are remarkably consistent across model families (GPT, Gemini, Qwen) and model scales, suggesting the modality complementarity principle is robust to model architecture and training differences. (2) The strongest models (o4-mini, Gemini-2.5-Pro) show larger absolute gains than weaker models (GPT-4o), indicating that better base reasoners benefit more from strategic modality allocation — they can more effectively exploit the complementary information. (3) Both VLSR and MSSC contribute independently and additively, with no evidence of negative interaction: MSSC's additional gain is similar whether applied to the baseline or to VLSR. (4) The paper's claim of "up to a 4.33% improvement" in the abstract is an average across all models and benchmarks, not a per-model guarantee — individual model improvements range from 2.25% (Qwen3-VL) to 7.25% (Gemini-2.5-Pro).

Comparison Against Training-Free Memory-Augmented Methods (Table 3)

Table 3 benchmarks VLSR+MSSC against two recent training-free strategies that use external memory: Dynamic Cheatsheet (Suzgun et al., 2025) and ArcMemo-PS (Ho et al., 2025), using o4-mini as the base model across three test sets.

Results on ARC-AGI-400: VLSR+MSSC achieves 28.75%, compared to 27.25% for ArcMemo-PS and 23.5% for Dynamic Cheatsheet. The 1.5 percentage point margin over ArcMemo-PS represents approximately 6 additional correctly solved tasks out of 400.

Results on ARC-AGI-100 (the ArcMemo subset): VLSR+MSSC achieves 28.0% versus 26.0% for ArcMemo-PS and 24.0% for Dynamic Cheatsheet.

Results on Re-ARC: VLSR+MSSC achieves 42.5% versus 40.5% for ArcMemo-PS and 38.0% for Dynamic Cheatsheet.

Interpretation. The paper argues that these results demonstrate visual information provides benefits that text-based memory retrieval cannot capture. However, several factors complicate this interpretation: (1) The comparison is not compute-matched — VLSR+MSSC uses multiple model calls while memory-augmented methods likely have their own overhead for memory retrieval and prompt construction, and neither side's total inference cost is quantified. (2) The 1.5 percentage point gap on ARC-AGI-400 (6 tasks) is small enough that without confidence intervals, it is unclear whether the difference is statistically reliable. (3) ArcMemo-PS and Dynamic Cheatsheet are both text-centric by design; the fact that adding visual information improves performance is consistent with the paper's thesis but does not isolate whether the gain comes from vision specifically or from any additional information source. A stronger comparison would include a memory-augmented method that also has access to visual representations.

Modality-Switch vs. Text-Only Self-Correction (Table 4)

Table 4 provides the direct comparison between traditional text-only self-correction (TOSC) and Modality-Switch Self-Correction (MSSC), with both applied for up to three iterative rounds starting from the same base text-only performance.

GPT-4o results (most dramatic difference):

Starting point: 8.25% (text-only baseline).
TOSC trajectory: Round 1 → 8.75%, Round 2 → 8.0%, Round 3 → 8.75%. Total gain: +0.5 percentage points, with Round 2 showing a performance degradation.
MSSC trajectory: Round 1 → 10.25%, Round 2 → 11.5%, Round 3 → 12.0%. Total gain: +3.75 percentage points, with monotonic improvement at every round.

Gemini-2.5-Pro results:

Starting point: 24.0%.
TOSC shows minimal change across rounds (the paper states "minimal improvement" without exact per-round numbers in the main text, but Table 4 provides the data).
MSSC achieves approximately +1.5 total improvement across three rounds.

o4-mini results:

Starting point: 25.25%.
TOSC shows minimal improvement.
MSSC achieves approximately +2.5 total improvement across three rounds.

Key patterns. (1) MSSC produces monotonic gains at every round for every model, while TOSC either stagnates or oscillates with degradation in intermediate rounds. This monotonicity is important because it suggests the visual verification signal is genuinely informative rather than randomly perturbing the output distribution. (2) The marginal gain decreases with successive rounds (GPT-4o: +2.0, +1.25, +0.5), indicating diminishing returns consistent with the iterative refinement eventually converging. (3) The paper does not report what fraction of tasks are flagged as "inconsistent" by the visual critic at each round, which would be informative for understanding how often MSSC actually triggers refinement versus simply accepting the initial output.

Fine-Tuning Results (Table 5)

Section 4.4 extends the VLSR principle to the training paradigm, fine-tuning separate models for visual rule summarization and textual rule application rather than using them as inference-time strategies.

Experimental setup. A vision-language model (Qwen3-VL-8B-Instruct) is fine-tuned for rule summarization using visual inputs, and a text-only model (Qwen3-8B) is fine-tuned for rule application using textual inputs. Training data comes from ARC-Heavy-200k (Li et al., 2024), with approximately 200k training tasks and a held-out 100-task test set. The text-only baseline fine-tunes Qwen3-8B on the same 200k tasks using purely textual matrix representations.

Results on ARC-AGI (100-task test set from ARC-Heavy-200k):

Qwen3-8B before fine-tuning: 3.5% (text-only baseline without training)
Text-only fine-tuning (Qwen3-8B on 200k text tasks): 9.75% (improvement of 6.25 percentage points from training)
VL synergy fine-tuning (Qwen3-VL-8B-Instruct for rules + Qwen3-8B for application): 13.25% (improvement of 9.75 percentage points from the pre-fine-tuning baseline)
The VL synergy approach outperforms text-only fine-tuning by 3.5 percentage points and surpasses the closed-source GPT-4o baseline (8.25%, from Table 2) by 5.0 percentage points.

Results on BARC-100: VL synergy fine-tuning achieves 16.5% versus 13.5% for text-only fine-tuning (+3.0). On Re-ARC: 22.5% versus 18.5% (+4.0).

Interpretation and caveats. (1) The 3.5 percentage point improvement over text-only fine-tuning demonstrates that the vision-language synergy principle transfers from inference-time strategies to the training paradigm — visual information helps even when models are explicitly trained on the task. (2) However, the absolute numbers are modest: 13.25% on ARC-AGI tasks from the same distribution as training data, compared to the inference-time VLSR+MSSC results of 29.75% for o4-mini. This gap reflects the difference between fine-tuning relatively small open-source models (8B parameters) versus using large frontier models (o4-mini, Gemini-2.5-Pro) with inference-time strategies. (3) The paper does not report an ablation where a single Qwen3-VL-8B-Instruct performs both visual rule summarization and textual rule application (using the same model with modality switching at inference time). Such an ablation would isolate whether the specialized training of separate models is necessary or whether the VLSR inference strategy alone would suffice with the fine-tuned model. (4) The comparison against "several open-source and closed-source models with much larger parameter sizes" (Section 4.4 claim) is valid but slightly misleading — GPT-4o's 8.25% is without fine-tuning or specialized inference strategies, not a like-for-like comparison of model capabilities.

Ablation Studies and Robustness Checks

Modality assignment for rule summarization vs. rule application (Table 1): The paper's foundational ablation tests which modality should be assigned to each subtask. Using Gemini-2.5-Pro, textual rule summarization achieves 37.25% accuracy while visual rule summarization achieves 40.75% (a 3.5 percentage point gain). Critically, when the same high-quality visual rule is then applied using visual representations, accuracy drops to 23.75% — a catastrophic 17 percentage point degradation from the 40.75% achieved with textual application. This ablation directly justifies VLSR's routing decisions: vision for summarization, text for application. The paper reports this pattern is consistent across models, though per-model breakdowns are not provided in Table 1 (only the "average" improvements across models are stated: +3.2% for visual summarization, -15.0% for visual application).

Iterative self-correction depth (Table 4): The three-round comparison between TOSC and MSSC serves as an ablation on the number and modality of correction rounds. The key finding is not just that MSSC outperforms TOSC, but that the performance trajectory is qualitatively different: MSSC shows monotonic improvement (GPT-4o: 8.25 → 10.25 → 11.5 → 12.0) while TOSC is non-monotonic with degradation in intermediate rounds (GPT-4o: 8.25 → 8.75 → 8.0 → 8.75). This ablation demonstrates that cross-modal verification is not merely "better" than same-modality verification — it produces a fundamentally different correction dynamic where each round adds genuine value rather than oscillating.

Model scale and family robustness (Table 2): The paper implicitly ablates across model families and scales by testing four different models (GPT-4o, Gemini-2.5-Pro, o4-mini, Qwen3-VL) and three benchmarks (ARC-AGI, BARC, Re-ARC). The consistent improvements across all 12 model-benchmark combinations (VLSR always improves over text-only; MSSC always provides additional gains) serve as a robustness check on the modality complementarity hypothesis. However, the paper does not systematically vary model scale within a single family (e.g., GPT-4o-mini vs. GPT-4o vs. a hypothetical larger GPT variant), which would more cleanly isolate whether the benefits of vision-language synergy scale with model capability or are relatively constant.

Training paradigm transfer (Table 5): Section 4.4 functions as an ablation on whether the VLSR principle requires large frontier models and inference-time strategies, or whether it also benefits smaller fine-tuned models. The 3.5 percentage point improvement of VL synergy fine-tuning over text-only fine-tuning on the same training data demonstrates that the principle transfers to training-time applications with smaller models.

Missing ablations that would strengthen the paper:

Decomposition without modality switching: Does VLSR's gain come from the explicit two-stage decomposition, or specifically from routing subtasks to different modalities? A text-only two-stage baseline (textual rule summarization → textual rule application) would isolate the decomposition effect from the modality-matching effect.
Modality-switching without decomposition: A single-stage approach that interleaves visual and textual information (e.g., presenting both the image and the nested list in the same prompt) would test whether the sequential two-phase structure is necessary or whether VLSR's gains could be achieved through richer single-pass multimodal prompting.
Reverse modality assignment (text for summarization, vision for application): This would serve as a negative control — if VLSR's routing is genuinely optimal, the reverse assignment should perform worse than both the proposed VLSR and the text-only baseline. The paper has the data to compute this from Table 1 (textual summarization + visual application is partially reported) but does not explicitly present it as an ablation of VLSR.
MSSC with oracle verification: Replacing the visual critic with ground-truth correctness checking would establish an upper bound on self-correction performance and quantify how much of MSSC's limitation comes from critic errors versus fundamental unrecoverability of initial mistakes.
Prompt sensitivity: The paper does not ablate prompt design choices — for example, whether the explicit instruction to "first check the correctness of the rule" in the textual application prompt is necessary for VLSR's performance, or whether the specific wording of the visual verification prompt affects MSSC's accuracy.

Critical Assessment

The experiments provide substantial evidence for the paper's core thesis — that visual and textual modalities have complementary strengths in ARC-AGI reasoning and can be strategically combined for improved performance — but the strength of support varies across the specific claims, and several experimental design choices limit the definitiveness of the conclusions.

Claim: "Vision and language possess complementary strengths across distinct reasoning stages." This claim is very strongly supported by the quantitative results in Table 1 and the qualitative analysis in Section 3.2.2. The 17 percentage point performance swing between visual rule summarization (+3.5%) and visual rule application (-17.0%) on the same model (Gemini-2.5-Pro) is a large, directionally opposite effect that can only arise from genuine modality-task interaction, not from general model quality differences. The four characteristics identified in Section 3.2.2 (holistic vs. independent processing, 2D structure preservation, encoding efficiency, element-wise precision) provide a mechanistic explanation that goes beyond correlation. The replication across models (mentioning "an average improvement of 3.2% across models" for visual summarization) further strengthens the claim.

However, the evidence comes exclusively from ARC-AGI and its variants. The paper's claim about "distinct reasoning stages" is operationalized specifically as "rule summarization" and "rule application" in the context of grid transformation tasks. Whether these characteristics generalize to other types of reasoning (deductive logic, causal inference, mathematical proof) or other task structures (non-grid-based spatial reasoning, temporal reasoning) is untested. The paper's framing implies broader applicability, but the experiments only directly support the claim for ARC-AGI-style abstract reasoning.

Claim: "VLSR yields up to a 4.33% improvement over text-only baselines." This claim is supported by Table 2, but the "up to" framing masks substantial model-dependent variation. The 4.33% figure is described as an average, with individual improvements ranging from 2.25% (Qwen3-VL on ARC-AGI) to 7.25% (Gemini-2.5-Pro on ARC-AGI). More importantly, VLSR approximately doubles the inference compute relative to the text-only baseline (two model calls instead of one), and the paper does not account for this cost multiplier. The claim of "improvement" is about accuracy, not efficiency — readers should understand that the 4.33% gain comes with at minimum a 2× increase in inference computation.

Claim: "MSSC provides consistent iterative improvements while text-only self-correction fails or degrades performance." Table 4 provides the strongest evidence in the paper, with a clean, well-controlled comparison. The qualitative difference in trajectories — MSSC's monotonic gains versus TOSC's stagnation and oscillation — is compelling and aligns with the paper's theoretical explanation (confirmation bias in same-modality verification). The within-model, within-task design (same starting accuracy, same number of rounds) makes this comparison unusually rigorous.

However, the paper does not specify the exact prompt used for TOSC, which is a significant omission. Text-only self-correction is an active research area with many prompt design choices (e.g., chain-of-thought verification, step-by-step error checking, self-consistency checks), and TOSC's poor performance could partially reflect suboptimal prompt engineering rather than a fundamental limitation. A more convincing demonstration would include multiple TOSC prompt variants to show that the effect is robust to how text-only verification is elicited. Additionally, the paper does not ablate whether the visual critic is actually necessary for MSSC's gains, or whether simply regenerating with a different random seed (at temperature 0.7) after an arbitrary "inconsistency" flag would produce similar improvements — an important control for whether the visual signal specifically is driving the improvement.

Claim: "VLSR+MSSC outperforms memory-augmented text-only strategies." Table 3 supports this claim for the two specific baselines tested (Dynamic Cheatsheet and ArcMemo-PS), but the comparison has important limitations. The margin over ArcMemo-PS on ARC-AGI-400 is 1.5 percentage points (28.75% vs. 27.25%), representing 6 tasks out of 400. Without confidence intervals or statistical tests, it is impossible to assess whether this difference is reliable or within run-to-run variation. Furthermore, the comparison is not compute-matched — VLSR+MSSC uses multiple model calls with distinct prompts, while the memory-augmented methods have their own computational overhead (memory retrieval, program synthesis) that is not quantified. A reader cannot determine whether VLSR+MSSC achieves higher accuracy per unit of compute, or simply achieves higher accuracy through higher compute expenditure.

Claim: "Vision-language synergy fine-tuning enables small open-source models to surpass closed-source models like GPT-4o." Table 5 shows VL synergy fine-tuning achieves 13.25% versus GPT-4o's 8.25% on ARC-AGI. This claim is technically accurate but misleading in its framing: GPT-4o's 8.25% is its zero-shot performance without fine-tuning or specialized inference strategies, while the fine-tuned Qwen3 models are trained on 200k synthetic tasks from the same distribution as the test set. The appropriate closed-source comparison would be GPT-4o with the same VLSR+MSSC inference strategy (which achieves 14.5% from Table 2), or GPT-4o fine-tuned on the same data. The paper's claim narrowly construed ("on this specific test set, with our specific training data, our fine-tuned 8B model beats GPT-4o's untrained performance") is true but substantially weaker than the headline implies.

Genuine experimental weaknesses:

No compute-matched comparisons. The paper's central methodological contribution — training-free inference strategies — is evaluated purely on accuracy, with no accounting for the 2–5× increase in inference calls relative to baselines. For training-free methods where the only cost is inference, this omission makes it impossible to determine whether the strategies represent genuine improvements in capability or simply reflect additional computation. A proper evaluation would include accuracy-vs-compute curves showing that VLSR and MSSC achieve higher accuracy at the same compute budget, not just higher accuracy at higher compute.
No confidence intervals or statistical tests. All results are reported as point estimates. On a 400-task evaluation set, a 1% difference represents 4 tasks — small enough that run-to-run variation at temperature 0.7 could plausibly produce differences of this magnitude. The consistency across models and benchmarks provides some reassurance, but the absence of any statistical framework means readers cannot assess the reliability of individual comparisons.
Limited ablation of the decomposition itself. The paper never tests whether the two-phase decomposition (separate rule extraction and application) provides benefits independently of the modality assignment. A text-only two-phase baseline would isolate the decomposition effect from the modality-matching effect, answering whether VLSR's gains come from the structured reasoning approach or specifically from using vision for rule extraction.
No prompt sensitivity analysis. All results use the specific prompts shown in Appendix A. The performance of LLM-based methods is often highly sensitive to prompt wording, and without variations or robustness checks, it is unclear whether the reported gains depend on particular prompt formulations that may not generalize.
Single benchmark domain (grid-based abstract reasoning). All experiments are on ARC-AGI and its derivatives. While this is appropriate for the paper's scope, the broader claims about vision-language complementarity in reasoning would be substantially strengthened by even a single experiment on a different type of spatial reasoning task (e.g., geometry problems, block world planning, navigation).

Missing experiments that would strengthen the paper:

An accuracy-vs-inference-compute analysis plotting performance against the number of model calls or total tokens generated, showing that VLSR and MSSC are on a superior scaling curve rather than simply using more compute.
A control experiment where the visual critic in MSSC is replaced with a random "inconsistency" trigger at the same frequency, testing whether the benefit comes from the visual verification signal or from the additional generation opportunities.
An ablation where VLSR's two phases use the same modality (text for both or vision for both) to isolate the contribution of decomposition from the contribution of modality switching.
Evaluation on a non-ARC-AGI spatial reasoning benchmark to test generalization of the modality complementarity thesis beyond grid transformations.
A human evaluation of the extracted rules from visual versus textual summarization to validate that the accuracy difference reflects genuine rule quality rather than downstream application artifacts.

6. Limitations and Trade-offs

1. Inference Compute Cost Is Unaccounted For in All Headline Accuracy Numbers

The assumption or constraint. The paper presents VLSR and MSSC as "training-free" strategies, which is technically accurate — no gradient updates or fine-tuning are required. However, the paper never quantifies the inference-time compute cost of these strategies relative to the text-only baseline, despite the fact that VLSR at minimum doubles the number of model calls (one visual summarization pass + one textual application pass), and MSSC can multiply inference cost by up to 5× when all three refinement rounds are triggered (one visual summarization, then up to three rounds of visual verification + textual re-inference, plus one final application). The paper's abstract claims "up to a 4.33% improvement over text-only baselines" and Section 5 claims "up to 7.25% improvement" without once mentioning that these gains come with a proportional increase in inference computation.

The authors acknowledge this implicitly when describing the MSSC pipeline: "This process repeats until consistency is achieved or the iteration limit is reached (we use $N_{\max} = 3$ )" (Section 3.4), but frame the iteration limit purely as a convergence parameter rather than a cost-quality tradeoff. The cost multiplier is never expressed or tabulated alongside the accuracy gains.

The consequence. A practitioner reading the headline results cannot determine whether VLSR+MSSC represents a genuine improvement in reasoning capability or merely reflects additional computation. Consider two concrete scenarios:

VLSR vs. text-only at the same compute budget: The text-only baseline uses 1 model call. If VLSR uses 2 calls (visual summarization + textual application), a fair comparison would give the text-only baseline 2 calls as well (e.g., majority voting over 2 samples, or best-of-2). The paper never runs this comparison, so it is unknown whether VLSR's ~3% average gain (Table 2) exceeds what could be achieved by simply running the text-only baseline twice and selecting the best output.
VLSR+MSSC vs. text-only at the same compute budget: In the worst case, VLSR+MSSC uses 1 (summarization) + 3 × (verification + re-inference) + 1 (final application) = up to 8 model calls per task. A text-only baseline with 8 calls could use majority voting, best-of-8, or other ensembling strategies. The paper never benchmarks text-only performance at these higher compute budgets, making it impossible to determine whether the gains come from the cross-modal mechanism or simply from additional sampling.

The paper frames its methods in opposition to "training-based" approaches (which require expensive data generation and fine-tuning), but fails to account for the fact that training-free methods shift cost from training to inference. For high-volume deployment (many tasks), the inference cost multiplier may dominate the total cost, making the "training-free" label misleading as a statement about total resource requirements.

What evidence exists in the paper. None — the paper provides no compute accounting whatsoever. The generation budget (temperature, number of tokens, number of model calls, total FLOPs) is never tabulated or discussed as a dependent variable. Table 2 reports accuracy only. Table 3 compares against memory-augmented methods without quantifying the inference cost on either side. Table 4 reports accuracy per self-correction round, which implicitly shows that additional rounds improve accuracy, but never expresses this as a cost-quality tradeoff curve.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation, does not provide compute-matched baselines, and does not discuss the cost-quality tradeoff in any section. The choice of $N_{\max} = 3$ is presented as a design parameter rather than a cost constraint, and the fact that VLSR uses two distinct model calls is treated as an architectural detail rather than a cost consideration. This is the most significant omission in the paper's experimental design because it undermines the central practical claim — that VLSR+MSSC provides a better way to solve ARC-AGI tasks.

2. Hard Problems (Where Base Models Have Near-Zero Pass@1) Remain Unsolved Regardless of Modality Strategy

The assumption or constraint. The paper's thesis is that visual and textual modalities have complementary strengths that can be exploited for improved ARC-AGI reasoning. However, this thesis implicitly assumes that the base model possesses at least some minimal capability on a given task — that there is a signal to amplify. When the model fundamentally lacks the capability to solve a task, no amount of modality routing or cross-modal verification can help.

The paper's experimental results reveal this boundary condition clearly. Table 2 shows that the strongest model (Gemini-2.5-Pro) with the full VLSR+MSSC pipeline achieves only 31.25% on ARC-AGI-400 — meaning nearly 70% of tasks remain unsolved. Table 5 shows that even with 200k synthetic training examples and specialized vision-language synergy fine-tuning, the fine-tuned model achieves only 13.25% on ARC-AGI tasks from the same distribution.

The paper does not provide a per-difficulty breakdown of results (in contrast to the reference example paper on test-time compute scaling, which explicitly analyzed performance by difficulty quintile), so it is unclear whether the gains are concentrated on easier tasks (where the base model already has some traction) or distributed across the difficulty spectrum. The qualitative examples in Appendix C (Figures 5–8) all show cases where the model successfully extracts the correct rule — these are naturally selected to illustrate the benefit of visual reasoning, not cases where both modalities fail.

The consequence. For the hardest ARC-AGI tasks — those requiring novel compositional reasoning, deep abstraction, or conceptual leaps that exceed the base model's training distribution — VLSR and MSSC offer no path forward. The framing of the paper (particularly the abstract and introduction) emphasizes the positive gains without clearly bounding where those gains occur, potentially creating a misleading impression that the approach broadly improves ARC-AGI reasoning. In reality, the approach amplifies existing capability but cannot create capability that isn't there.

This limitation is practically significant because it mirrors the "scaling vs. capability" distinction that appears in other reasoning domains. For example, the compute-optimal test-time scaling paper (Snell et al., 2024) explicitly showed that test-time compute helps on easy-to-medium problems but provides zero benefit on the hardest problems where the base model's pass@1 is near zero. This paper never makes that distinction explicit, despite the fact that its results are consistent with the same pattern: the largest absolute gains appear on models that already perform best (Gemini-2.5-Pro gains +7.25% from a 24.0% baseline, while GPT-4o gains +6.25% from an 8.25% baseline), suggesting the approach adds the most value where there is already substantial capability to amplify.

What evidence exists in the paper. Indirect evidence only. The absolute performance ceiling (31.25% on ARC-AGI-400 for the best model) shows that the majority of tasks are unsolved. The fine-tuning results (13.25% after 200k training examples) show that even extensive task-specific training leaves most problems unsolved. However, the paper does not provide any difficulty-stratified analysis, does not characterize which types of tasks benefit most from VLSR/MSSC, and does not acknowledge the hard-problem failure as a systematic limitation.

Mitigation status. Not addressed. The paper does not discuss the difficulty distribution of gains, does not identify task categories where VLSR/MSSC fails, and does not bound the scope of applicability. Section 5 (Conclusion) states the approach "achieves an average performance improvement of 4.3%" without noting that this is an average over tasks where many are completely unaffected. This is a transparency gap — the paper would be stronger if it explicitly characterized where the method helps and where it does not.

3. Single Benchmark Domain (Grid-Based Abstract Reasoning) with No Evidence of Generalization

The assumption or constraint. All experiments in the paper — both inference-time strategies and fine-tuning — are conducted exclusively on ARC-AGI and its derivatives (Re-ARC, BARC). These are all variants of the same underlying task: inducing transformation rules from 2D grid input-output pairs with discrete cell values (0–9) and applying them to novel test inputs. While ARC-AGI is designed to be diverse in the space of possible transformation rules (rotations, symmetries, object manipulations, color mappings, counting, path-finding, compositional operations), it is a single task family — all instances share the same input-output structure, the same representation format (2D grids), and the same reasoning paradigm (few-shot rule induction from demonstrations).

The paper makes broader claims about vision-language complementarity that are not qualified by this domain limitation. The abstract states: "Our findings suggest that unifying visual abstraction with linguistic reasoning is a crucial step toward achieving generalizable, human-like intelligence in future foundation models." Section 1 claims: "we systematically decompose ARC-AGI into two sub-tasks: rule summarization and rule application" and "vision and language possess complementary strengths across distinct reasoning stages." The four characteristics identified in Section 3.2.2 (holistic vs. independent processing, 2D structure preservation, encoding efficiency, element-wise precision trade-offs) are presented as general properties of modality-specific reasoning, not as ARC-AGI-specific observations.

The consequence. A practitioner considering whether to apply VLSR/MSSC to a different reasoning domain — geometry problems, block world planning, code generation from visual specifications, scientific diagram interpretation — has no evidence that the approach generalizes. The four characteristics may or may not apply to other domains. For example, in geometry, the rule summarization might involve identifying which theorem applies (benefiting from visual diagram perception) while rule application might involve algebraic manipulation (benefiting from textual precision) — an analogous structure. But in code generation from visual wireframes, the "rule" is the specification-to-code mapping, and the modality assignment may be entirely different.

Additionally, the specific decomposition into "rule summarization" and "rule application" is natural for ARC-AGI (where an underlying transformation must be induced) but may not map cleanly onto other reasoning tasks. For causal reasoning, the decomposition might be "hypothesis generation" and "hypothesis testing." For mathematical proof, it might be "strategy selection" and "step-by-step derivation." The paper provides no framework for determining whether VLSR's decomposition generalizes, or how a practitioner would adapt it to a new domain.

The claim that results on "multiple ARC-AGI benchmarks" (Section 1) demonstrates generalization is misleading — Re-ARC and BARC are constructed from the same task distribution as ARC-AGI, differing in how tasks are generated (synthetic permutations of pre-defined rules) but not in the fundamental task structure. Showing that VLSR improves performance on Re-ARC and BARC is a robustness check, not a generalization test.

What evidence exists in the paper. All experiments are on grid-based abstract reasoning tasks. The paper provides no evaluation on any non-ARC-AGI benchmark. The supplementary benchmarks (Re-ARC, BARC) are variants of the same task family. There is no discussion of whether the findings might generalize to other spatial reasoning tasks, code generation, mathematical problem-solving, or any other domain.

Mitigation status. Not addressed. The paper does not acknowledge the single-domain limitation, does not qualify its broader claims with scope boundaries, and does not suggest domain-generalization experiments as future work. A brief acknowledgment that "validation on other spatial reasoning benchmarks (e.g., geometry, block world planning) remains future work" would substantially improve the paper's intellectual honesty about the scope of its contribution.

4. No Statistical Significance or Confidence Intervals on a Modest-Sized Test Set

The assumption or constraint. All experimental results in the paper are reported as point estimates of accuracy, with no confidence intervals, standard errors, statistical significance tests, or variance decomposition. The primary evaluation set (ARC-AGI-400) contains 400 tasks. The supplementary benchmarks (Re-ARC, BARC) contain 100 tasks each. At these sample sizes, a 1 percentage point difference represents 4 tasks on ARC-AGI or 1 task on Re-ARC/BARC.

The paper reports many comparisons where the margin between methods is small relative to the test set size:

Table 3, ARC-AGI-400: VLSR+MSSC achieves 28.75% vs. ArcMemo-PS at 27.25% — a difference of 1.5 percentage points, representing 6 tasks out of 400.
Table 2, across models: The average improvement of VLSR over text-only is reported as 3.02%, but per-model differences range from 2.25% (Qwen3-VL, 9 tasks on 400) to 7.25% (Gemini-2.5-Pro, 29 tasks). The variation across models is larger than the average effect.
Table 2, BARC-100: Many comparisons involve differences of 2–4 percentage points (2–4 tasks out of 100).

All experiments use temperature 0.7, which introduces stochasticity in model outputs. Without multiple runs or variance estimates, it is unclear whether the reported differences are statistically reliable or within the range of run-to-run variation. A difference of 1.5 percentage points on ARC-AGI-400 could plausibly arise from sampling noise at temperature 0.7, especially for comparisons where the model's internal reasoning process is sensitive to prompt formulation and generation randomness.

The consequence. Several of the paper's specific comparative claims — particularly the claim that VLSR+MSSC "outperforms the strongest baseline ArcMemo-PS by 1.5% on ARC-AGI" (Section 4.2) — cannot be evaluated for reliability by the reader. This is especially problematic for the memory-augmented comparison (Table 3), which the paper uses to argue that "visual information offers complementary benefits that text-based memory retrieval alone cannot capture." If the 1.5 percentage point margin is not statistically significant, this claim is unsupported.

The absence of variance estimates also makes it impossible for practitioners to assess expected performance variability in deployment. If a method achieves 28.75% on average but varies by ±3% across runs due to temperature sampling, the practical reliability is much lower than the point estimate suggests. This matters for high-stakes applications where consistent performance is as important as average performance.

What evidence exists in the paper. None. The paper reports no variance estimates, no confidence intervals, no statistical tests, and no mention of run-to-run variability. The experimental setup (Section 4.1) specifies temperature 0.7 but does not report whether results are averaged over multiple runs, whether the reported numbers are from a single run, or whether any statistical framework was used.

Mitigation status. Not addressed. This is a straightforward methodological gap that could be partially remedied by reporting results averaged over multiple runs with standard errors, or by computing confidence intervals based on the binomial distribution (since accuracy on a fixed test set is a binomial proportion). The paper does not discuss this limitation.

5. The Text-Only Self-Correction Baseline (TOSC) Is Under-Specified and Potentially Under-Engineered

The assumption or constraint. Table 4 provides the critical comparison between Modality-Switch Self-Correction (MSSC) and Text-Only Self-Correction (TOSC), showing MSSC producing consistent monotonic gains while TOSC stagnates or degrades. This comparison anchors the paper's claim that cross-modal verification "breaks the model's confirmation bias" and enables effective intrinsic self-correction (Section 3.4). The paper cites Huang et al. (2023) and Zhang et al. (2024) to establish that same-modality self-correction fails, and presents TOSC's poor performance as evidence that MSSC solves this failure mode.

However, the paper never specifies the exact prompt used for TOSC. Appendix A provides prompts for VLSR's rule summarization, rule application, and visual consistency verification, but does not include the TOSC prompt. The reader cannot determine whether TOSC's failure reflects a fundamental limitation of same-modality verification or suboptimal prompt design. Text-based self-correction can be implemented many ways — asking the model to "check your answer," asking it to "verify step by step," asking it to "identify any errors in your reasoning," requiring it to "provide a confidence score," or using chain-of-thought verification where the model re-derives the solution from scratch — and the effectiveness varies substantially with prompt design.

The paper's description of TOSC is limited to Section 4.3: "TOSC uses the same textual modality for both generation and verification," with no further specification of the verification mechanism. This makes it impossible to assess whether TOSC represents a fair baseline or a straw-man comparison.

The consequence. If TOSC's poor performance is partially attributable to suboptimal prompting, then the paper's core claim about MSSC — that it uniquely enables intrinsic self-correction through cross-modal verification — is weaker than the evidence suggests. A more robust demonstration would show that MSSC outperforms a well-engineered TOSC baseline with multiple prompt variants, or would include an ablation where the "verification" modality is something other than vision (e.g., a different text-based representation, or a different language for code-based verification) to isolate whether the benefit comes from cross-modal processing specifically or simply from introducing any perturbation to the reasoning loop.

Furthermore, without the TOSC prompt specification, the results in Table 4 are not reproducible by other researchers, which undermines their value as a benchmark comparison.

What evidence exists in the paper. Table 4 reports TOSC performance numbers across three rounds and three models, demonstrating stagnation and occasional degradation. However, the mechanism by which TOSC operates is not described, the prompt is not provided, and the paper does not discuss alternative text-only self-correction implementations that might perform better.

Mitigation status. Partially addressed through citation. The paper cites Huang et al. (2023) and Zhang et al. (2024) as prior evidence that intrinsic same-modality self-correction is difficult, establishing a literature baseline for TOSC's expected poor performance. However, these citations establish that self-correction is difficult, not that any specific TOSC implementation is optimal or that the specific TOSC used in this paper is representative of the best possible text-only approach. Providing the TOSC prompt and testing multiple text-only verification strategies would substantially strengthen the claim.

6. The Difficulty Estimation Overhead and Dynamic Allocation Problem Are Not Addressed

The assumption or constraint. VLSR applies a fixed decomposition strategy to every ARC-AGI task: visual rule summarization followed by textual rule application. This assumes that the modality-subtask mapping (vision → summarization, text → application) is optimal for all task types in ARC-AGI, regardless of the specific transformation involved. However, the paper's own analysis suggests that modality effectiveness varies with task characteristics. Section 3.2.2 identifies four characteristics that explain modality differences, and these characteristics are not uniformly present in all tasks. For example, Characteristic 2 (2D structure preservation) matters most for tasks requiring spatial reasoning, while Characteristic 3 (encoding efficiency) matters most for large grids. Tasks that involve purely local, element-wise transformations (e.g., "increment each cell value by 1 if it is even") might not benefit from visual summarization at all — vision's holistic perception adds no value when the rule is inherently element-wise.

Additionally, MSSC applies visual verification to every output regardless of whether visual verification is likely to detect an error. If the transformation is purely about cell values (e.g., a mathematical mapping like $f(x) = (x + 3) \bmod 10$ ), visual consistency checking may be less informative than textual re-derivation, and the extra model calls for visual verification add cost without benefit.

The paper never addresses the question of when to use VLSR versus a text-only approach, or when MSSC is likely to help versus when it wastes computation. The strategies are applied uniformly to all tasks, which implicitly assumes the modality complementarity benefits are universal across ARC-AGI rather than task-dependent.

The consequence. In a deployment setting, the cost of VLSR+MSSC (2–8× the inference calls of the text-only baseline, as discussed in Limitation 1) is paid on every task, including tasks where the approach provides no benefit. A more efficient system would estimate task characteristics upfront — perhaps by analyzing grid size, spatial structure, or initial model confidence — and route tasks to the appropriate strategy. Easy element-wise tasks could use text-only (avoiding the visual summarization overhead), while spatially complex tasks could use the full VLSR+MSSC pipeline. The current uniform application leaves substantial efficiency gains on the table.

This limitation is directly analogous to the difficulty estimation problem in the compute-optimal test-time scaling paper (Snell et al., 2024), which demonstrated that the optimal test-time strategy varies substantially with problem difficulty and that a difficulty-conditioned allocation policy recovers large efficiency gains. VLSR+MSSC as presented here is analogous to applying beam search uniformly to all problems — better than the baseline on average, but leaving efficiency on the table by not adapting to task characteristics.

What evidence exists in the paper. The paper does not provide any task-level analysis of where VLSR or MSSC helps versus hurts. There is no breakdown by task category (e.g., spatial vs. non-spatial transformations, small vs. large grids, single-object vs. multi-object manipulations). The paper does not discuss whether some tasks are solved correctly by the text-only baseline but incorrectly by VLSR (false modality-switching errors), which would indicate that the fixed strategy sometimes degrades performance for specific task types. The consistency of improvement across benchmarks (Table 2) provides aggregate evidence that VLSR helps on average, but averages can conceal task-level heterogeneity where the method helps on some tasks and hurts on others.

Mitigation status. Not addressed. The paper does not discuss task-adaptive strategy selection, does not analyze where the method fails, and does not frame the fixed VLSR decomposition as a design choice that could be made adaptive. The paper's framing treats the uniform application as the method itself, rather than as a specific instantiation of a broader principle (modality-complementarity-aware reasoning) that could be applied adaptively. This represents a missed opportunity to connect the paper's diagnostic framework (the four characteristics) to a practical deployment strategy that uses those characteristics to make per-task routing decisions.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a diagnostic framework for modality-aware reasoning system design rather than a specific model or training procedure. The landscape change is not that ARC-AGI performance improves by a few percentage points—that is the evidence, not the contribution—but rather that the paper provides a principled vocabulary and methodology for reasoning about when and why different modalities help different reasoning substages.

The key reframing: from modality choice as binary to modality allocation as a design problem. Prior to this work, the implicit assumption in ARC-AGI research was that modality choice is uniform: either process everything as text (the dominant paradigm across synthetic data fine-tuning, test-time training, and memory-augmented methods) or render everything as images (naive visual integration). The paper's central empirical finding—that vision improves rule summarization by 3.2% on average while degrading rule application by 15.0% (Table 1)—demonstrates that neither single-modality approach can be optimal. This transforms the research question from "should we use vision?" to "which reasoning substages should receive visual input, and which should receive textual input?" The field now has a concrete demonstration that modality allocation is a resource optimization problem analogous to compute allocation in scaling laws, where the optimal strategy depends on matching modality characteristics to subtask requirements.

The four characteristics as a transferable diagnostic vocabulary. The paper's enumeration of modality-specific processing properties (holistic vs. independent processing, 2D structure preservation, encoding efficiency, element-wise precision trade-offs; Section 3.2.2) provides a vocabulary that generalizes beyond ARC-AGI. Any reasoning task involving spatial, relational, or structural information can now be analyzed through this lens: where does the task require holistic pattern recognition? Where does it require precise element-wise indexing? The answers dictate modality routing. This is analogous to how the Chinchilla scaling laws (Hoffmann et al., 2022) provided a vocabulary for reasoning about pretraining compute allocation—not just a specific recipe for a specific model, but a conceptual framework that changes how researchers design experiments and interpret results across the field.

Reconciling conflicting intuitions about visual reasoning. The paper resolves a latent tension in the multimodal reasoning literature. On one side, work like Visual Sketchpad (Hu et al., 2024) and ViLaSR (Wu et al., 2025) demonstrated that visual information can substantially improve spatial reasoning, creating an expectation that vision should help uniformly. On the other side, the dominant ARC-AGI paradigms persisted with text-only approaches, implicitly suggesting that vision was not beneficial. The paper's finding—that naively rendering ARC-AGI grids as images degrades performance overall—explains this apparent contradiction: vision does help, but only for specific reasoning stages. Prior work that tested vision uniformly (and found it unhelpful) was asking the wrong question by not decomposing the task. The paper thus provides a unified explanation for why some studies find vision beneficial and others do not, with the key moderating variable being whether the task decomposition aligns modality with subtask.

What becomes more attractive as a research direction. The paper makes cross-modal reasoning pipeline design a first-class research problem. Instead of treating LVLMs as monolithic systems that either process images or text, the paper demonstrates that explicitly managing modality at the pipeline architecture level—with natural language rules as modality bridges—yields consistent gains. This opens the door to more sophisticated routing strategies (dynamic modality selection based on task characteristics, hierarchical decomposition where sub-subtasks receive different modalities) and to the study of modality complementarity in other reasoning domains. The paper also makes verifier design for reasoning systems more attractive, but specifically cross-modal verifiers: MSSC's success (Table 4) suggests that verification representations should be deliberately chosen to differ from generation representations, not merely to be "more capable."

What becomes less attractive. The paper casts doubt on two approaches: (1) purely text-only ARC-AGI methods that do not attempt to incorporate spatial information, since the paper demonstrates that text alone systematically underperforms on rule summarization (losing ~3% on average), and (2) naive end-to-end multimodal approaches where vision and text are simply concatenated without explicit reasoning-stage decomposition, since the paper shows that vision applied indiscriminately can hurt more than it helps (the 15% degradation on rule application). The field should move toward architected multimodal reasoning where modality choice is a deliberate design parameter rather than a default or an afterthought.

Distinguishing paradigm shift from incremental refinement. This is a methodological reframing, not a paradigm shift. The underlying models (LVLMs) are unchanged; what changes is how they are used. The paper does not introduce new architectures, training objectives, or capabilities. Instead, it introduces a new way of thinking about how to deploy existing capabilities—specifically, the insight that modality allocation is a design dimension that can be optimized independently of model scale or training. The magnitude of the empirical gains (4.3% average, up to 7.25% for the best model) is modest in absolute terms, and the majority of ARC-AGI tasks remain unsolved (the best model achieves 31.25%). The paper's lasting impact will likely be the diagnostic framework and the demonstration that strategic modality decomposition matters, rather than the specific VLSR or MSSC algorithms, which are straightforward instantiations of the framework.

Follow-Up Research This Work Enables

1. Dynamic modality routing via task-characteristic prediction. The paper applies VLSR uniformly to all tasks, but the four characteristics in Section 3.2.2 imply that modality benefits should vary with task properties. A direct extension would train a lightweight classifier to predict, from the text of the examples alone (or from a small number of initial visual samples), whether a task is primarily spatial (benefiting from visual summarization) or element-wise (potentially not benefiting). This classifier would route tasks: spatially complex tasks → full VLSR+MSSC; element-wise tasks → text-only (avoiding the 2×+ inference cost overhead of VLSR). The experiment would measure whether adaptive routing recovers the VLSR accuracy gain at substantially lower average inference cost, and would use the held-out 100-task set from ARC-Heavy-200k or the full ARC-AGI-400 for evaluation. The key metric is accuracy-per-inference-call, not absolute accuracy. This directly addresses the unaccounted inference cost limitation identified in Section 6, Limitation 1.

2. Cross-modal self-correction in non-spatial reasoning domains. MSSC's core mechanism—using a different modality for verification than for generation to break confirmation bias—is not inherently spatial. A critical stress-test would apply the same principle to code generation with visual execution trace verification (render data structure states or algorithm steps as diagrams, verify visual consistency), or to mathematical reasoning with geometric diagram verification (render algebraic relationships as geometric figures, check visual coherence). The experiment would use established benchmarks (HumanEval for code, Geometry3K or similar for math), compare same-modality self-correction against cross-modal verification, and measure whether the monotonic improvement pattern from Table 4 replicates. A negative result—that MSSC only works for inherently spatial tasks—would bound the generality of the confirmation-bias-breaking mechanism. A positive result would establish cross-modal verification as a general design principle for self-improving reasoning systems.

3. Training-time modality decomposition for specialized small models. Section 4.4 demonstrates that VL synergy fine-tuning (separate models for visual summarization and textual application) improves over text-only fine-tuning on the same data. A natural next step is to study the scaling properties of this decomposition: as the rule summarization model and rule application model are scaled independently, where do diminishing returns set in? Does a very small summarization model (1B parameters) plus a large application model outperform a single medium model doing both? This experiment would use the ARC-Heavy-200k dataset, vary model sizes systematically within the Qwen3 family (0.5B, 1.8B, 4B, 8B, 32B), and measure whether the optimal allocation of parameters between summarization and application differs from a 50/50 split. This connects the paper's inference-time decomposition insight to the training-time resource allocation problem and would provide practical guidance for practitioners building specialized ARC-AGI systems.

4. What does the visual critic actually detect? MSSC's visual verification step (Equation 5) produces a binary consistency judgment, but the paper never analyzes what specific types of errors the visual critic detects that text-based verification misses. A detailed error analysis would categorize the model's failures on a subset of ARC-AGI tasks, then measure the visual critic's detection rate for each error category (e.g., spatial transformation errors vs. arithmetic errors vs. object identification errors). The hypothesis from the four characteristics is that the visual critic should excel at detecting global pattern violations (missing symmetries, incorrect spatial relationships, gestalt mismatches) but perform no better than text at detecting element-wise value errors. This experiment would require manual annotation of error types on, say, 100 model failures, followed by measuring per-category visual critic recall. The results would refine the modality complementarity thesis by specifying the boundary conditions more precisely than "vision for pattern, text for precision."

5. Adversarial evaluation: can visual patterns deceive the visual summarizer? The paper's qualitative examples (Appendix C, Figures 5–8) show cases where visual summarization succeeds and text fails. A stress-test would construct adversarial ARC-AGI tasks where the visual pattern is misleading—for example, a task where the spatial arrangement looks like a rotation but the actual rule is a counterintuitive element-wise mapping, or where the most salient visual feature is a distractor unrelated to the transformation. The experiment would measure whether VLSR's visual summarization phase is more susceptible to such visual distractors than the text-only baseline is to textual distractors. This would test whether the holistic perception advantage (Characteristic 1) comes with a corresponding vulnerability to holistic misdirection, and would inform when VLSR should not be used even for spatially complex-looking tasks.

6. Scaling the inference budget: does MSSC benefit from more than 3 rounds? Table 4 shows diminishing returns across three MSSC rounds (GPT-4o: +2.0, +1.25, +0.5), but the ceiling of $N_{\max} = 3$ was chosen as a compute-quality tradeoff without systematic justification. A scaling experiment would run MSSC out to 5, 7, or 10 rounds on a subset of ARC-AGI-400 (to manage cost), measure whether accuracy eventually plateaus or continues to improve, and characterize which tasks benefit from deeper refinement versus which converge quickly. The experiment would also track the fraction of tasks flagged as "inconsistent" at each round—if this fraction drops sharply after round 2 or 3, it would justify the $N_{\max}=3$ choice empirically rather than heuristically. This addresses the unquantified inference cost limitation by characterizing the accuracy-vs-rounds tradeoff curve.

Practical Applications and Downstream Use Cases

1. Inference-time augmentation for LVLM-based reasoning APIs. VLSR+MSSC is immediately deployable as a wrapper around any LVLM API (GPT-4o, Gemini, Claude, Qwen-VL) without model modification, fine-tuning, or access to internal representations. The prompts in Appendix A are self-contained, and the visualization procedure (Appendix B) is deterministic and implementation-light. A practitioner building an ARC-AGI evaluation pipeline or a general-purpose abstract reasoning system could integrate VLSR+MSSC today and expect an average ~4.3% accuracy improvement over the text-only default, with larger gains (up to 7.25%) for the most capable models. The cost is 2–8× the per-task inference calls, which may be acceptable for low-volume, high-stakes reasoning tasks where accuracy dominates cost concerns.

2. Data generation for self-improving reasoning systems. The paper's finding that text-only self-correction fails while cross-modal verification succeeds (Table 4) has direct implications for automated data generation pipelines. When using LVLMs to generate training data for reasoning tasks—for instance, producing solution trajectories for fine-tuning—MSSC-style cross-modal verification could serve as a quality filter that accepts or rejects generated solutions without ground-truth labels. The specific workflow: generate candidate solution via textual reasoning → render output visually → verify spatial/logical consistency against visual examples → accept only if verified. This is particularly valuable for domains like ARC-AGI where ground-truth labels are scarce or expensive, and where same-modality self-verification is known to be unreliable (confirmed by the TOSC results). The paper's reported MSSC improvement trajectory (GPT-4o: 8.25% → 12.0% over three rounds) suggests that cross-modal filtering could substantially improve the quality of generated training data.

3. Specialized small-model deployment via modality-decomposed training. The fine-tuning results in Table 5 (13.25% for VL synergy vs. 9.75% for text-only, both from 8B-class models) demonstrate that the modality decomposition principle transfers to training. For organizations deploying ARC-AGI solvers on resource-constrained hardware (edge devices, mobile, embedded systems), this means they can train two specialized small models (a vision model for rule extraction, a text model for rule application) that together outperform a single small model doing both, and can even surpass closed-source models like GPT-4o (8.25%) without the associated API costs or latency. The training recipe—200k synthetic tasks from ARC-Heavy-200k, separate fine-tuning of vision and text components, rule as explicit interface—is fully specified and reproducible with open-source models (Qwen3 family).

4. Diagnostic tool for evaluating LVLM spatial reasoning capabilities. The comparative analysis methodology in Section 3.2—systematically ablating modality per reasoning substage—provides a template for evaluating any LVLM's spatial reasoning capabilities. Instead of reporting a single "ARC-AGI score," model developers could report a modality profile: accuracy on visual rule summarization, accuracy on textual rule application, accuracy on visual rule application (the degradation metric), and self-correction gain from cross-modal verification. This profile would reveal whether a model's weaknesses lie in spatial perception, symbolic manipulation, or both, guiding targeted improvement efforts. The paper's Table 1 for Gemini-2.5-Pro is a prototype of such a profile, and the methodology could be applied systematically across model releases to track progress in modality-specific reasoning capabilities.

When to Prefer This Method

The paper does not articulate an explicit tradeoff framework positioning VLSR/MSSC against named alternatives with decision boundaries. The comparisons in Tables 2–5 are empirical demonstrations of improvement rather than prescriptive guidance about when to choose one strategy over another. The paper also does not characterize where VLSR/MSSC fails relative to baselines—no task-level analysis identifies cases where the text-only baseline outperforms VLSR, which would be necessary for a decision rule.

Since the paper lacks this explicit positioning, a forced "Prefer A when... Prefer B when..." matrix would be fabricated rather than derived from the paper's own analysis. The closest the paper comes to conditional guidance is the implicit observation that VLSR helps more for stronger base models (Gemini-2.5-Pro gains +7.25% vs. Qwen3-VL gains +2.25%; Table 2), suggesting that the method amplifies existing capability but does not create it, and that the visual critic in MSSC is only as good as the model's visual perception capabilities. However, the paper does not develop these observations into explicit deployment recommendations, so no decision matrix is included here.