When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought

ArXiv: 2511.02779

🎯 Pitch

Even GPT-5 and Gemini 2.5 Pro fail to surpass 20% accuracy on problems that require 'drawing to think,' such as tracking a rolling die on a board. Providing intermediate visual sketches boosts model performance by 33.7% on average, proving that text alone is fundamentally lossy for spatial reasoning.

1. Executive Summary

This paper introduces MIRA (Multimodal Imagination for Reasoning Assessment) , a benchmark of 546 multimodal problems that require generating intermediate visual images — sketches, structural diagrams, or path drawings — to successfully reason through tasks that are intrinsically difficult to express in language alone (e.g., tracking a die's movement on a board and summing the face-down values after each roll). The benchmark evaluates models across three diagnostic levels: direct input (image and question only), Text-CoT reasoning (step-by-step text prompting), and Visual-CoT reasoning (human-annotated intermediate visual clues paired with textual prompts), establishing that even the strongest multimodal LLMs (GPT-5, o3, Gemini 2.5 Pro) fail to surpass 20% accuracy under direct input. Providing annotated intermediate visual cues yields an average relative gain of 33.7% across all models and tasks, with physics tasks nearly doubling in performance for proprietary MLLMs, establishing that visually-intensive reasoning problems cannot be solved through text-only chain-of-thought and that explicit visual grounding is essential for success on this class of tasks.

2. Context and Motivation

The Core Problem: Text Is an Impoverished Medium for Visual Reasoning

The fundamental gap this paper identifies is deceptively simple yet profound: multimodal models today reason about images using only text, even though many reasoning problems are fundamentally visual in nature. Current Chain-of-Thought (CoT) prompting — the dominant paradigm for improving LLM reasoning since Wei et al. (2022) — operates entirely in the textual domain. When a multimodal model like GPT-5 encounters a problem requiring spatial reasoning, geometric manipulation, or physical simulation, it must verbalize every intermediate step: "the die starts at position X, then rolls east, so the top face becomes..." This is analogous to forcing a human to solve a complex geometry problem with their hands tied behind their back — they can talk through it, but they cannot draw.

The authors crystallize this insight in Section 1:

"This purely linguistic format is inherently limiting, as many real-world reasoning problems are intrinsically visual — requiring spatial reasoning, geometric manipulation, or physical simulation — that humans typically address by drawing to think."

This is not merely an inconvenience. Natural language is a lossy compression of visual information. Describing the convex hull of 10 scattered blue points in words requires enumerating coordinates, comparing relative positions, and reasoning about boundary inclusion — operations that become combinatorially explosive in text but are nearly instantaneously resolvable when you can simply see the hull drawn on a scratchpad. The paper argues this is not a minor limitation of current prompting strategies but a fundamental constraint on what text-only reasoning can achieve for visually-intensive tasks.

The paper's central research question emerges from this observation: Can current multimodal models truly reason with integrated visual artifacts, and how much can this capability contribute to solving complex visual reasoning problems? The MIRA benchmark is designed specifically to answer this question by constructing problems where the answer is not that text-only reasoning is suboptimal but rather that it is intrinsically insufficient — the intermediate visual state carries information that cannot be adequately compressed into language.

Why This Problem Matters: Beyond Perception to Genuine Visual Reasoning

The significance of this gap operates at multiple levels.

The human cognition parallel. Humans do not solve complex spatial problems through internal monologue alone. Architects sketch floor plans. Physicists draw force diagrams. Chess players visualize board states. Puzzle-solvers trace paths with their fingers. This "drawing to think" is not a crutch for limited working memory — it is a cognitive strategy that transforms abstract relationships into manipulable external representations. If AI systems aspire to human-like reasoning capabilities, they must develop analogous mechanisms for generating and leveraging visual artifacts during problem-solving. The paper frames MIRA as probing precisely this capability: the ability to "imagine" visual intermediates in service of downstream reasoning, hence the name Multimodal Imagination for Reasoning Assessment.

The real-world deployment gap. The paper highlights a striking contrast in Figure 1 (right panel): leading MLLMs achieve strong performance on established multimodal benchmarks (MMMU, MMMU Pro, MMStar, RealWorldQA) but collapse dramatically on MIRA, with no model surpassing 20% accuracy under direct input. This is not a marginal degradation — it represents a qualitative failure mode on a specific class of problems that current evaluation suites systematically overlook. The implication is that our existing benchmarks may be giving a falsely optimistic picture of multimodal reasoning capability. Models excel at perception-oriented tasks (object recognition, captioning, simple VQA) but fail when asked to perform the kind of multi-step visual reasoning that humans accomplish with scratch paper.

The self-improvement bottleneck. If models cannot generate and use their own intermediate visual representations, they are fundamentally bounded in their ability to tackle increasingly complex visual reasoning problems. No amount of textual chain-of-thought will help a model count red points inside an overlapping convex hull region if it cannot accurately delineate that region. The paper positions visual chain-of-thought generation as a missing capability in the current MLLM landscape — not a nice-to-have enhancement but a prerequisite for tackling a broad class of real-world reasoning tasks.

The unified model challenge. Recent work on unified MLLMs that can both understand and generate images (Blip3-o, Janus-pro, Bagel, Show-o, OmniGen2) opens the tantalizing possibility of models that can draw their own scratchpads during reasoning. Yet, as the paper notes in Section 2, "most image-generating MLLMs are optimized for photorealistic synthesis or descriptive captioning, not for creating abstract, task-specific diagrams." MIRA provides a benchmark explicitly targeting this capability gap, creating a clear target for unified model development.

Where Prior Approaches Fall Short

The paper identifies specific limitations across four categories of existing work.

1. CoT reasoning is textually imprisoned. The chain-of-thought paradigm (Wei et al., 2022) and its variants — zero-shot CoT (Kojima et al., 2022), automatic CoT (Zhang et al., 2022) — have been transformative for arithmetic, commonsense, and multi-hop reasoning. But they share a foundational assumption: "they assume verbal reasoning alone suffices and struggle on inherently visual tasks that are better served by diagrams or spatial representations, where intermediate graphical states would be needed" (Section 2). The paper's experiments (Section 4.2, Table 1) provide quantitative evidence for this limitation: Text-CoT actually degrades performance for strong models like Gemini 2.5 Pro (a relative drop of 18.3%) and o3 (14.0%) on MIRA, and the Puzzles category sees a 4.2% average relative decline for proprietary models. This is not just that Text-CoT doesn't help — it actively hurts on tasks where verbal reasoning misleads more than it guides.

2. Multimodal reasoning benchmarks test perception, not visual reasoning. The paper acknowledges the contributions of existing benchmarks while identifying a critical blind spot. VQA (Antol et al., 2015), MME (Fu et al., 2023), MM-Vet (Yu et al., 2023), and MMMU (Yue et al., 2024) have driven progress in multimodal understanding, but the authors argue they evaluate perception more than reasoning:

"many prompts target simple perception (e.g., 'Who is wearing glasses?'), placing minimal demands on reasoning. Some datasets, such as ScienceQA, include multi-step reasoning hints, but those steps remain textual; problems can still be solved via natural-language rationales alone, without requiring intermediate images or visual cues."

This is the key distinction: existing benchmarks can be solved despite the absence of visual chain-of-thought, not because it is required. MIRA is designed with the explicit principle that "generating intermediate visual images is essential for successful reasoning" (abstract, emphasis added). The problems are constructed such that an intermediate diagram is not merely helpful — it is the mechanism through which the problem becomes tractable.

Two recent benchmarks deserve specific mention because they begin to approach visual CoT territory but, the authors argue, fall short. VisuLogic (Xu et al., 2025) "emphasizes pattern discovery rather than genuine reasoning over images." EMMA (Hao et al., 2025) "employs relatively simple MCQ tasks with limited visual clues as input." MIRA differentiates itself by (a) requiring genuine multi-step reasoning rather than pattern matching, (b) providing detailed human-annotated visual CoT trajectories, and (c) spanning diverse reasoning categories beyond simple multiple-choice puzzles.

3. Tool-augmented visual reasoning is bounded by tool capabilities and lacks systematic evaluation. A parallel line of work explores letting models use external tools to generate visual aids during reasoning. Visual ChatGPT (Wu et al., 2023) lets models call external vision models to draw or edit. VisProg (Gupta and Kembhavi, 2023) and ViperGPT (Surís et al., 2023) enable visual programming with tool execution. Visual Sketchpad (Hu et al., 2024) allows models to execute Python code to generate diagrams for geometry and spatial reasoning tasks. Vision-Augmented Prompting (Xiao et al., 2024) similarly uses code generation for visual augmentation.

The paper acknowledges these as steps in the right direction but identifies two limitations. First, these approaches are "fundamentally bounded by the capabilities of the tools they rely on" (Section 1) — the model can only visualize what the tool can render. Second, and more critically for the benchmark motivation: "these approaches rely on external tool orchestration and have not yet been systematically evaluated in open-ended reasoning scenarios" (Section 2, emphasis added). There is no standardized benchmark measuring whether tool-augmented visual reasoning actually solves problems that text-only reasoning cannot. MIRA fills this gap by providing a systematic evaluation platform where visual intermediates are known to be necessary, enabling precise measurement of how much visual cues contribute to problem-solving.

4. Unified MLLMs lack the "think-while-drawing" capability. The emergence of models that combine visual understanding with image generation — Blip3-o, Janus-pro, Bagel, Show-o, OmniGen2 — represents the most promising architectural direction for autonomous Visual-CoT. These models could, in principle, generate their own intermediate sketches during reasoning. However, the paper observes that "even advanced systems like Gemini and GPT-5 have not demonstrated robust 'think-while-drawing' abilities" (Section 2). Current unified models can produce beautiful photorealistic images from captions, but generating an accurate free-body force diagram or tracing a die's path on a grid requires different skills: precision, abstraction, and tight integration with the reasoning process.

The paper's diagnostic evaluation protocol (Level 3: Simulated Visual-CoT) provides a clever way to probe this gap without requiring models to actually generate images. By providing human-annotated intermediate visuals, the authors can measure how much a model would benefit if it could generate such visuals. The performance of unified models (Bagel, Janus-Pro) in this setting — relative gains of 17.3% and 46.9% respectively — suggests that even these architectures have substantial headroom to improve their ability to leverage visual intermediates, despite being designed for joint understanding and generation.

How This Paper Positions Itself

MIRA positions itself not as an incremental improvement to existing benchmarks but as a category-defining evaluation framework for a capability that current benchmarks fail to measure. The paper's self-conception is clearest in three design choices.

First, the benchmark is constructed around necessity. The paper's first design principle (Section 3.1) is that tasks must "require intermediate visual cues to answer the question." This is not a preference — it is enforced through problem design. The paper's 20 task types are drawn from domains (convex hull construction, dice rolling path tracking, mirror clock reading, billiard ball trajectory prediction, bomb defusal wire tracing) where the intermediate visual state contains spatial information that verbal descriptions inevitably lose. The benchmark is not asking whether visual cues are helpful — it establishes through task construction that they are necessary, then measures how much models fail without them.

Second, it creates a diagnostic protocol rather than a single leaderboard. The three-level evaluation (Direct, Text-CoT, Visual-CoT) is "designed to move beyond a single accuracy score and provide insights into why a model fails" (Section 3.2, emphasis in original). Level 1 measures raw capability. Level 2 isolates the contribution of text-based reasoning. Level 3 — the key contribution — measures the upper bound of what visual information could provide if models could properly leverage it. This protocol enables fine-grained analysis: if a model performs poorly on Level 1, improves marginally on Level 2, but jumps dramatically on Level 3, the diagnosis is clear — the model can reason given visual intermediates but cannot generate or simulate them internally. This is precisely the pattern observed for many models in Table 1 (e.g., GPT-5-mini: 13.7% → 12.9% → 23.2%).

Third, it spans a deliberately broad task taxonomy. The four domains — Euclidean Geometry (EG), Physics-Based Reasoning (PBR), Abstract Spatial & Logical Puzzles (ASLP), and Causal Transformations (CT) — are chosen to cover qualitatively different types of visual reasoning. EG tasks require geometric construction and spatial counting (convex hulls, overlapping regions). PBR tasks require physical simulation and trajectory tracking (billiards, forces, mirror reflections). ASLP tasks require combinatorial spatial reasoning (cube unfolding, jigsaw fitting, bomb defusal wire tracing). CT tasks require state transformation tracking (dice rolling, gear rotation, paper folding). This breadth ensures that findings generalize across different reasoning modalities, and the paper's per-category breakdowns (Tables 4–10) reveal that different categories stress different model weaknesses: Physics tasks show the largest Visual-CoT gains (20.7% → 40.0% average for proprietary models), while Puzzles remain stubbornly difficult even with visual clues (9.5% → 10.5%).

The paper's boundary claim is specific and empirically grounded. MIRA does not claim that all multimodal reasoning requires visual intermediates, or that text-based CoT is universally insufficient. The claim is narrower and therefore more defensible: there exists a class of visually-intensive reasoning tasks — exemplified by the 20 task types in MIRA — where text-only reasoning is intrinsically inadequate, and where providing intermediate visual representations yields substantial, measurable improvements. The paper's contribution is to systematically characterize this class, provide a reproducible evaluation framework for it, and establish empirical baselines that quantify how far current models are from the capability that would be required to solve such problems autonomously.

This positioning explains the paper's choice of title: "When Visualizing is the First Step to Reasoning." The implication is not that visualizing is always the first step, but that MIRA identifies the conditions under which it is — and that current models are not yet capable of taking that step on their own.

3. Technical Approach

3.1 Reader Orientation

MIRA is not a model or training method — it is a diagnostic evaluation benchmark with a three-level protocol. The system being built is a dataset of 546 curated multimodal problems, each paired with human-annotated intermediate visual reasoning steps, and an evaluation framework that measures model performance across three conditions. The core problem it addresses is: how do we evaluate whether a multimodal model can genuinely reason with visual intermediates when current models cannot yet generate their own intermediate images? The solution shape is clever: rather than asking models to produce visual reasoning (which they cannot do reliably), MIRA provides human-annotated visual reasoning steps as input and measures whether models can leverage them — thereby quantifying the potential benefit of visual chain-of-thought even before models can generate it themselves.

3.2 Big-Picture Architecture (Diagram in Words)

The MIRA system has four major components:

1. Task Design and Data Construction Pipeline — a hybrid human-programmatic process that generates 546 multimodal problems across 20 task types, each requiring intermediate visual reasoning to solve. This is not a data collection effort but a data design effort: tasks are created from scratch with specific reasoning requirements.

2. The Benchmark Dataset — the output of the pipeline: 546 instances, each containing an input image, a natural language question, one or more human-annotated intermediate visual reasoning images (Visual-CoT images), and a ground-truth final answer. The dataset spans four domains: Euclidean Geometry (EG), Physics-Based Reasoning (PBR), Abstract Spatial & Logical Puzzles (ASLP), and Causal Transformations (CT).

3. The Three-Level Diagnostic Evaluation Protocol — the mechanism for testing models. Level 1 (Direct) provides only the input image and question. Level 2 (Text-CoT) provides the input image and a textual chain-of-thought prompt encouraging step-by-step reasoning. Level 3 (Visual-CoT) provides the input image, the human-annotated intermediate visual images, and a textual prompt directing the model to reason based on these visuals. The protocol decouples textual reasoning ability from visual information utilization.

4. Answer Extraction and Grading Pipeline — a tiered system that extracts final answers from potentially lengthy model outputs and grades them against ground truth, using rule-based parsing, regex heuristics, and an LLM judge as a fallback.

Information flows through the system as follows: a problem instance enters → the evaluation protocol selects the input configuration (Direct, Text-CoT, or Visual-CoT) → the model generates a response → the answer extraction pipeline parses the response → the grader compares against ground truth → accuracy is aggregated across all instances and disaggregated by domain, task type, and evaluation level.

3.3 Roadmap for the Deep Dive

• First, the benchmark design principles and data construction pipeline — because understanding what the benchmark measures requires understanding how the problems are built and why they cannot be solved through text alone.
• Second, the task taxonomy and domain structure — because different reasoning types stress different model capabilities, and the taxonomy is essential for interpreting the per-category results.
• Third, the three-level evaluation protocol — because this is the paper's primary methodological contribution, and understanding it requires seeing how the three levels isolate different sources of model failure.
• Fourth, the answer extraction and grading mechanism — because accurate evaluation at scale requires a robust system for parsing diverse model outputs that may not follow formatting instructions.
• Fifth, the experimental design choices (model selection, metrics, prompting) — because these determine what conclusions the benchmark can support about current model capabilities.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmark and evaluation methodology paper whose core idea is that visual reasoning capability can be measured even in models that cannot generate visual intermediates by providing annotated visual reasoning steps and measuring utilization.

Benchmark Design Principles

The MIRA benchmark is constructed around three explicit design principles stated in Section 3.1, each of which constrains the data pipeline and shapes the resulting dataset. Understanding these principles is essential because they establish what MIRA measures and, equally importantly, what it does not measure.

Principle 1: Tasks must require intermediate visual information to solve. This is the defining constraint. A problem qualifies for MIRA only if the answer cannot be reliably derived from text-based reasoning alone — the intermediate visual state carries spatial, geometric, or relational information that language cannot adequately encode. The authors do not merely assert this requirement; they operationalize it through task selection. The 20 task types are drawn from domains where the reasoning process naturally involves visual manipulation: tracking a die's orientation as it rolls (requiring 3D mental rotation), determining which points fall inside overlapping convex hulls (requiring geometric boundary construction), tracing wire paths through an opaque box (requiring continuity reasoning through occlusion). If a problem could be solved by listing facts and applying logical rules in pure text, it is excluded from MIRA by design.

This principle has an important consequence for benchmark interpretation: MIRA does not measure whether visual reasoning helps — it measures whether models can succeed when visual reasoning is necessary. The distinction matters because a model scoring 20% on MIRA with direct input is not failing at an optional enhancement; it is failing at a task that, by construction, requires visual intermediates that the model cannot generate or simulate internally. The benchmark's difficulty is not an accident of problem selection but a deliberate feature of the design.

Principle 2: Each instance must be paired with annotated step-wise visual clues to enable Visual-CoT evaluation. This is the methodological innovation that makes the benchmark practical despite current models' inability to generate their own visual intermediates. For every one of the 546 problems, human annotators create intermediate visual images that show the solution trajectory. For a dice-rolling problem, these images show the die's orientation after each step of the path (as illustrated in the rolling dice examples in Figure 2, with CoT image steps 1 through 5). For a convex hull problem, the intermediate image shows the constructed hulls and their overlapping region. For a mirror clock problem, the intermediate image shows the reflected clock face.

These annotations serve two purposes. During evaluation, they become the input for the Level 3 (Visual-CoT) condition, allowing measurement of how much models benefit from explicit visual intermediates. During benchmark construction, they serve as a verification mechanism — if annotators cannot create a clear visual trajectory that leads unambiguously to the correct answer, the problem is revised or rejected. The annotations are "gold-standard" in the sense that they precisely align with the reference reasoning trajectory, ensuring that the Visual-CoT condition provides correct, not misleading, visual information. The benchmark includes 936 manually constructed intermediate images for its 546 problems (stated in the Figure 2 caption), with single-step tasks receiving one CoT image and multi-step tasks receiving a sequence.

Principle 3: Strict human annotation and cross-validation for data quality. The paper emphasizes rigorous quality control throughout the data pipeline. Problems are manually authored by "graduate-level researchers" (Section 3.1), not crowd-sourced or automatically generated without review. Each problem undergoes cross-review where annotators check for uniqueness of the ground-truth answer (no ambiguity), correctness of the visual reasoning trajectory, and clarity of the problem statement. Conflicts are resolved through discussion. The final benchmark contains only instances that survive this quality control process.

The rationale for this stringent approach is that MIRA's small size (546 problems) relative to large-scale benchmarks demands near-perfect per-instance quality. Each problem is a diagnostic instrument; a single ambiguous or incorrectly annotated problem would contaminate the fine-grained per-category analysis that is the benchmark's primary contribution. The paper does not report inter-annotator agreement metrics or annotation time/cost, which would help readers assess the reproducibility and scalability of this approach.

Data Construction Pipeline

Section 3.1 describes a "hybrid construction pipeline by integrating the manual labeling, human inspection and programmatic generations." Figure 3 provides a high-level overview, and the text fills in the details. The pipeline proceeds through six stages:

Stage 1: Problem Conception and Prototyping. Graduate-level researchers conceive problem types and prototypes, drawing inspiration from puzzle communities (Reddit's r/puzzles, r/visualpuzzles, r/SmartPuzzles), brain-teaser websites (Braingle, Loquiz, SmartBrainPuzzles), and various exercise repositories. The key constraint during conception is novelty — the authors explicitly state they ensure "novel formulations and original content" (Section 3.1), meaning problems are not simply copied from existing sources but are newly designed. A problem prototype consists of a rough input image concept, a question, an expected reasoning trajectory, and a ground-truth answer. At this stage, the visual quality of the input image may be crude — the focus is on the reasoning structure.

Stage 2: Programmatic Generation for Controllable Difficulty. For certain task types, particularly those involving grid-based or parameterized structures (e.g., cube counting with configurable arrays, dice rolling with variable path lengths), Python scripts generate instances with fine-grained control over difficulty. The paper does not detail which specific task types were programmatically generated versus manually authored, but the implication is that programmatic generation enables systematic difficulty variation: longer paths, more complex geometries, or larger state spaces. Programmatically generated problems still pass through the subsequent human review stages, so the scripts produce candidates rather than final instances.

Stage 3: Visual Refinement. The initial images (whether manually drafted or programmatically generated) are "refined for better visual quality and clarity using image editing tools (e.g., GPT-4o, Gemini 2.5 Flash)" (Section 3.1). This is a notable detail: the paper uses existing multimodal models as tools in the benchmark creation process, not as subjects being evaluated. The models are used to clean up diagrams, ensure consistent visual styling, and improve readability — tasks at which current MLLMs are reasonably competent. This stage does not alter the reasoning content of the problem; it only improves the visual presentation so that evaluation measures reasoning capability rather than visual parsing difficulties.

Stage 4: Annotation of Visual-CoT Trajectories. For each surviving problem, annotators create the intermediate visual reasoning steps. For single-step problems (static reasoning), this is one auxiliary image showing the key spatial relationship or construction (e.g., the convex hulls with their overlapping region highlighted). For multi-step problems (dynamic reasoning), this is a sequence of images showing state transitions (e.g., the die's orientation after each roll step). The annotation process is itself a reasoning check: if annotators cannot produce a clear visual trajectory, the problem is flagged for revision or rejection at the next stage.

Stage 5: Feasibility Verification and Cross-Validation. This is the quality control bottleneck. Each problem-annotation pair undergoes cross-review by multiple annotators who verify: (a) the problem has a single unambiguous ground-truth answer, (b) the intermediate visuals correctly illustrate the reasoning trajectory and do not contain errors or misleading information, (c) the problem genuinely requires intermediate visual information (Principle 1), and (d) the problem is not trivially solvable by memorization or pattern matching. Disagreements are resolved through discussion, and problems that fail verification are either revised or discarded.

Stage 6: Final Benchmark Assembly. The 546 surviving problems are organized into the four-domain taxonomy (EG, PBR, ASLP, CT) and 20 task types. Each instance is a structured unit containing: an input image, a question string, a list of Visual-CoT images (one for static tasks, multiple for dynamic tasks), and a ground-truth answer string. The benchmark is frozen at this point for evaluation.

The pipeline design reveals a deliberate trade-off between scale and quality. At 546 instances, MIRA is small by modern benchmark standards (compare: MMMU has 11.5K questions, MME has 2.4K). The authors accept this size limitation in exchange for per-instance quality guarantees that enable fine-grained diagnostic analysis. A 50-question task subset with unequivocal ground truth and verified difficulty is more informative for understanding model failure modes than a 5000-question subset with ambiguous answers or unclear reasoning requirements. However, the small size means that some per-task results are based on very few questions — the detailed tables in Appendix C (Tables 4–10) show that individual tasks can have as few as 20–30 instances, making per-task accuracy estimates noisy.

Task Taxonomy and Domain Structure

The 546 problems are organized into four domains and 20 task types, as illustrated comprehensively in Figure 2. This taxonomy is not arbitrary — each domain targets a qualitatively different type of visual reasoning, and the paper's analysis (Section 4.2, Table 1) confirms that models exhibit domain-specific failure patterns.

Euclidean Geometry (EG) tasks require geometric construction, spatial counting, boundary identification, and shape manipulation. The six EG task types are:

• Convex Hull: Given sets of colored points, determine which points fall inside the convex hull of each color set, or inside overlapping regions of multiple hulls. Requires constructing the minimum convex polygon enclosing each point set — a geometric operation that is almost impossible to perform accurately through verbal description alone.
• Overlap: Given multiple candidate shape pairs with coordinate axes, determine which pair produces the largest overlapping area when aligned. Requires mental superposition of shapes and area comparison.
• Localizer: Tile a target region using irregular puzzle pieces with circles at specific locations; return the circle coordinates after tiling. Requires fitting pieces while tracking marker positions.
• Mirror Pattern: Determine which candidate option can be obtained by a single mirror reflection (possibly followed by rotation) of the original image. Requires understanding reflection transformations.
• Cubes Count: Given a perspective drawing of a 3D block structure, count the total number of cubes. Requires parsing 3D structure from 2D projection.
• Cubes Missing: Given an incomplete 3D block structure, determine how many cubes are needed to make it a complete rectangular solid. Requires computing the difference between full and partial occupancy.

Physics-Based Reasoning (PBR) tasks require physical simulation, trajectory tracking, force calculation, or reflection optics. The three PBR task types are:

• Billiards: Track a ball's trajectory on a table with labeled pockets, given an initial direction, assuming perfect elastic cushion bounces. Requires geometric optics / unfolding method for reflection paths.
• Electric Charge: Analyze the net Coulomb force direction on a target charge given surrounding charges with specified magnitudes and signs. Requires vector force addition — a task where drawing a free-body diagram is the standard human approach.
• Mirror Clock: Determine what time a clock shows in a mirror, or what the actual time is given a mirror reflection, possibly with time shifts. Requires understanding left-right reflection of angular positions.

Abstract Spatial & Logical Puzzles (ASLP) tasks require combinatorial spatial reasoning, mental folding/unfolding, or occlusion-based path tracing. The seven ASLP task types are:

• Unfolded Cube: Given multiple views of a patterned cube, determine the pattern on its unfolded net, or vice versa. Requires 3D mental rotation and face adjacency reasoning.
• Defuse A Bomb: Trace wires through an opaque box with limited visibility to determine which wire to cut to defuse a bomb. Requires continuity reasoning under occlusion.
• Multi-piece Puzzle: Select pieces from a set that combine to match a target shape exactly. Requires spatial decomposition and combination.
• Puzzle: Identify which piece fits a missing region in an object, considering shape, pattern, and shading. Requires perceptual completion.
• Trailer Cubes Count: Given three orthogonal views (top, front, side), determine the maximum possible number of cubes in a 3D structure consistent with all views. Requires reasoning about occluded cubes.
• Trailer Cubes Missing: Similar to above, but determine the minimum number of cubes needed to complete the structure to a solid block. Requires occupancy reasoning.

Causal Transformations (CT) tasks require tracking object state changes over sequences of operations. The four CT task types are:

• Paper Airplane: Given a sequence of folding steps, determine which completed paper airplane shape results. Requires tracking surface orientations through folding operations.
• Gear Rotation: Given a system of interconnected gears (possibly with belts, crossed belts), determine the final rotation state given an initial rotation. Requires parity reasoning through mechanical linkages.
• Rolling Dice: Top: Given a die's initial orientation and a path, determine the number on the top face after rolling along the path. Requires tracking orientation changes through spatial transformations.
• Rolling Dice: Sum: Similar to above, but compute the sum of numbers on the bottom face at each step. Requires cumulative tracking.
• Rolling Dice: Two: Compare two paths for the same die to determine which yields a higher sum, or whether they are equal. Requires parallel state tracking and comparison.

The taxonomy spans qualitatively different reasoning demands. EG tasks require geometric construction and spatial measurement. PBR tasks require physical law application to concrete configurations. ASLP tasks require combinatorial search over spatial possibilities. CT tasks require sequential state tracking. This breadth means that a model excelling on one domain might fail on another, and the paper's per-domain breakdowns (Table 1) confirm this: proprietary models average 20.6% on PBR with direct input but only 9.5% on ASLP, a greater than 2× gap.

The Three-Level Diagnostic Evaluation Protocol

This is MIRA's primary methodological contribution: a protocol that "decouples the information contribution of visuals from textual generation ability" (Section 1) by testing models under three input conditions that differ in what information is provided. The protocol is described in Section 3.2 and operationalized through the prompt templates in Appendix B.

Level 1: Direct Evaluation. The model receives the input image I_q and the question text T_q with a prompt asking for only the final answer, without requesting intermediate reasoning. The exact prompt template (from Appendix B) is:

[Input Image]
Question: {question}
Please provide the final answer directly. The final answer is placed in <answer></answer>.

This level measures the model's end-to-end problem-solving capability when given only the problem statement. It is the most challenging condition because the model must either simulate the necessary visual intermediates internally or attempt text-based reasoning despite the task's visual nature. The results (Table 1) show that this condition yields the lowest accuracies across nearly all models and domains, with no model surpassing 20% overall.

Level 2: Text-CoT Reasoning. The model receives the same input image and question but is prompted to generate a step-by-step textual reasoning chain before providing the final answer. The general template is:

[Input Image]
Question: {question}
Please first conduct step-by-step reasoning, and then provide the final answer. The final answer is placed in <answer></answer>.

Additionally, the paper experiments with task-specific Text-CoT prompts (the T_spec condition in Table 2) that provide domain-appropriate reasoning instructions for each of the 20 task types. For example, the specialized prompt for the Convex Hull task (Appendix B) instructs the model to:

"Start with one color (e.g., Red): Visually/algorithmically assess which Red points are extreme... Switch to the other color (e.g., Blue): Repeat the same analysis... Cross-check and reconcile... Construct the final hull... Briefly justify..."

These specialized prompts are designed to bridge the gap between generic "think step-by-step" instructions and the richer guidance that Visual-CoT images provide. If Text-CoT could solve MIRA problems, the specialized prompts should perform substantially better than the generic ones.

The key insight from Level 2 is not just its absolute performance but its differential effect compared to Level 1. For many strong models, Text-CoT reduces accuracy (Gemini 2.5 Pro drops from 16.9% to 13.8% overall; o3 drops from 16.4% to 14.1%), and for harder categories like Puzzles, proprietary models see an average 4.2% relative decline. This is the empirical evidence that text-based reasoning cannot substitute for visual information — the act of trying to verbalize visual relationships may actually confuse models on tasks where those relationships are spatial rather than linguistic.

Level 3: Simulated Visual-CoT Reasoning. This is the protocol's diagnostic core. The model receives the input image, the human-annotated intermediate visual images (one for static tasks, a sequence for dynamic tasks), and a prompt directing it to reason based on these visuals:

[Input Image] [CoT Image 1] [CoT Image 2] ...
Based on the question image and the intermediate reasoning image(s) provided, please continue the reasoning to solve the problem.
Question: {question}
The final answer is placed in <answer></answer>.

The phrase "simulated" is crucial: the model is not generating the intermediate images — it is receiving them. This is an upper-bound measurement. If a model can leverage provided visual intermediates, it demonstrates the potential to benefit from visual chain-of-thought. If a model fails even with provided intermediates, the bottleneck is not image generation but deeper reasoning limitations.

The prompt wording is also carefully chosen: "please continue the reasoning" rather than "analyze these images from scratch." This positions the intermediate images as a partial solution trajectory that the model should complete, reflecting the intended use case where future models would generate their own intermediates and then reason from them.

The diagnostic power of the three-level protocol is most visible when results are viewed as a trajectory across levels. For GPT-5-mini: 13.7% (Direct) → 12.9% (Text-CoT, a slight decline) → 23.2% (Visual-CoT, a dramatic jump). The decline from Level 1 to Level 2 isolates the failure of text-based reasoning. The jump from Level 2 to Level 3 isolates the value of visual information when text-based reasoning is held constant. The protocol thus answers not just "how well does the model perform?" but "what aspect of the problem is causing the model to fail?"

Answer Extraction and Grading Pipeline

Because MIRA evaluates models that may produce lengthy unstructured reasoning before giving a final answer, robust answer extraction is essential for reliable evaluation. The paper describes a tiered pipeline (Section 4.1, "Evaluation" paragraph) that attempts increasingly sophisticated extraction methods until one succeeds.

Tier 1: Tag-based extraction. The model is instructed (in all prompt templates) to place its final answer within <answer></answer> tags. The extraction system first attempts to parse these tags directly using a rule-based parser. If a well-formed <answer>X</answer> block is found, the enclosed content is extracted as the candidate answer. This is the preferred method because it relies on explicit formatting instructions rather than heuristic guessing.

Tier 2: Regex-based extraction. If tag parsing fails (the model produced no tags, malformed tags, or the tag content is empty), the system falls back to a set of "heuristic regular expressions for common answer phrasings" (Section 4.1). The paper does not provide the exact regexes, but they are designed to match patterns like "the answer is X", "therefore, X", "X is the final answer", or terminal lines containing only the answer. The regex approach handles models that follow the semantic instruction (provide answer at end) but ignore the formatting instruction (use tags).

Tier 3: LLM judge. For outputs that resist both tag parsing and regex extraction, the system uses gpt-4o-2024-11-20 as a semantic judge. The judge prompt (Appendix B) instructs the evaluator model to:

"Determine whether the model's final answer is correct based on the ground truth. Your evaluation must focus exclusively on the answer contained within the <answer></answer> tags, as well as the final answer portion at the end of the model's response. Ignore all reasoning, explanations, or any other text outside of these sections. The correctness of the reasoning process is not part of your evaluation."

The judge receives the original question, the ground truth answer, and the model's full response, and outputs only "Correct" or "Incorrect". This tier is computationally more expensive than the deterministic methods but handles genuinely ambiguous outputs that a regex cannot reliably parse.

The tiered design reflects a practical trade-off: deterministic methods (Tiers 1–2) are fast, cheap, and perfectly reproducible, while the LLM judge (Tier 3) provides robustness at the cost of some reproducibility (different GPT-4o calls might produce different judgments) and API expense. The paper does not report what fraction of outputs fall to each tier, which would help readers assess how much the evaluation depends on the LLM judge's reliability.

Grading. Once the candidate answer is extracted, it is compared against the ground truth answer. The paper states that answers are "graded using the grading function released by Lightman et al. (2022)" (Section 4, "Metrics" paragraph), but this detail appears in the prior sections writeup rather than in Section 4. The grading function handles formatting normalization (e.g., ignoring whitespace, handling equivalent numerical representations). For tasks with structured answers (e.g., coordinate lists for Localizer, pattern assignments for Unfolded Cube), the grading function must handle partial credit or exact-match requirements, though the paper does not specify the exact grading rules per task type.

Metric. The primary metric throughout is micro-averaged accuracy: the fraction of all 546 test instances for which the extracted answer matches the ground truth. "Micro-averaged" means each instance contributes equally regardless of task type, so more populous task types disproportionately influence the overall score. The paper also reports accuracy disaggregated by domain (EG, PBR, ASLP, CT) and by individual task type (in the detailed Appendix C tables), enabling per-category analysis.

Experimental Design Choices

The paper's experimental setup (Section 4.1) involves several design choices that determine what conclusions the benchmark can support.

Model Selection. The evaluation spans 18 models organized into three categories:

• Closed-Source SOTA MLLMs (15 models): GPT-5, GPT-5-mini, GPT-4.1, GPT-4.1-mini, GPT-4o, GPT-4o-mini, o3, o4-mini, Claude 4 Opus, Claude 4 Sonnet, Seed1.5-VL, Seed1.6 Vision Pro, Qwen-VL-Max, Gemini 2.5 Flash, and Gemini 2.5 Pro. This selection spans six different companies and represents the strongest available multimodal models as of mid-2025. The diversity of providers guards against the possibility that findings are specific to one organization's training methodology.

• Open-Weight MLLMs (Understanding) (3 models): Qwen2.5-VL at 32B and 72B parameter scales, and GLM 4.5 V at 106B. These models are "primarily geared towards perception tasks" (Section 4.1) and do not support general image generation, making them a baseline for understanding-focused architectures.

• Open-Weight Unified MLLMs (Understanding & Generation) (2 models): Bagel (7B) and Janus-Pro (7B). These models can both understand and generate images, making them the most promising candidates for future autonomous Visual-CoT. Their small parameter counts (7B) likely contribute to their relatively low absolute performance.

This categorization is itself an experimental axis: by comparing understanding-focused models versus unified models on the Visual-CoT condition, the paper can probe whether unified architectures have an inherent advantage in leveraging provided visual intermediates (since they are designed for tighter vision-language coupling). The results (Table 1) show that both categories benefit from Visual-CoT, but the open-weight models' gains are more limited (average 9.9% → 13.0% vs. proprietary models' 13.3% → 18.0%), likely due to smaller scale and less training on interleaved visual-text data.

Model Configuration. For API-based models, the paper uses "default decoding settings" with maximum output length set to 16,384 tokens (Appendix A, Table 3). For specific models (Qwen-VL-Max, GLM-4.5V, Qwen2.5-VL variants), maximum output length is set to 8,192 tokens with topp = 1.0. The paper does not specify the temperature setting for most models — the phrase "default decoding settings" implies whatever the API endpoint provides, which for many commercial models is a deterministic or low-temperature configuration. This is a minor reproducibility concern: different temperatures would produce different sampling behavior, and for the Pass@k experiments (Section 4.3), sampling diversity depends on temperature.

Prompt Design. The three evaluation levels use deliberately minimalist prompts (full templates in Appendix B, discussed in the protocol section above). The Level 1 prompt emphasizes direct answering without reasoning. The Level 2 prompt requests step-by-step reasoning. The Level 3 prompt frames the intermediate images as a partial solution to continue. The specialized Text-CoT prompts (the T_spec condition) are task-specific and explicitly guide the model through the reasoning steps a human would take, as described in the protocol discussion.

A notable design choice: the Visual-CoT prompt does not explicitly describe what the intermediate images contain. It simply says "based on the question image and the intermediate reasoning image(s) provided." The model must itself interpret how the visual intermediates relate to the question and how to use them to continue reasoning. This tests the model's ability to integrate provided visual information into its reasoning process, rather than testing whether the model can follow explicit instructions about what to extract from each image.

Pass@k and Majority Voting. Section 4.3 extends the evaluation beyond single-answer accuracy to probe the "upper bound" of model capability. For Pass@k with k ∈ {1, 2, 4, 8}, the model generates k different reasoning paths and answers for the same problem (using sampling with non-zero temperature, though the exact temperature is not specified). A problem is counted as correct if at least one of the k answers matches ground truth. This measures whether failures are due to occasional reasoning errors (in which case Pass@k should improve substantially) or fundamental capability gaps (in which case Pass@k should plateau).

Majority voting (self-consistency; Wang et al., 2022) takes the 8 sampled responses and selects the most common final answer as the prediction. Unlike Pass@k, which credits a model if any answer is correct, majority voting requires the model to consistently produce the correct answer across multiple samples. Strong reasoning models should exhibit both high Pass@k and high majority voting accuracy. The paper reports (Section 4.3) that Pass@k improves from k=1 to k=4 (average 15.3% improvement) but nearly converges between k=4 and k=8 (only 3.0% additional improvement), and majority voting adds limited gains (e.g., Gemini 2.5 Flash improves 5.1% while Gemini 2.5 Pro gains only 0.3%). This pattern supports the interpretation that MIRA failures are not primarily due to sampling variance but to fundamental capability limitations.

Budget and Scale. The evaluation is computationally intensive: 18 models × 546 problems × 3 evaluation levels (some models evaluated on multiple k settings) × up to 8 samples per problem for Pass@k. The paper does not report total inference cost, but for commercial API models at these scales, the evaluation likely represents a significant financial investment. This is not a limitation per se but contextualizes the benchmark's practical reproducibility for researchers without substantial API budgets.

What MIRA Does NOT Do (Important Scope Boundaries)

Understanding the technical approach requires understanding what the benchmark deliberately excludes.

MIRA does not require models to generate images. The Level 3 condition provides human-annotated intermediates; it does not test whether models can produce their own. The paper is explicit about this in Section 3.2: "Considering that current models, both open weight and commercial, cannot accurately generate or interleave the use of intermediate images and tool-generated auxiliary visuals, we provide manually annotated intermediate images for every task in MIRA." This is a practical concession to current model limitations, but it means MIRA measures the benefit of receiving visual intermediates, not the ability to generate them. The title phrase "Multimodal Imagination" refers to the benchmark's aspiration — the capability it is designed to eventually evaluate — rather than what the current protocol directly tests.

MIRA does not test tool use. Unlike Visual Sketchpad or ViperGPT, which evaluate models that call external tools, MIRA's Visual-CoT condition provides static images without any interactive or iterative refinement. A model cannot ask follow-up questions, request clarifications, or generate alternative visualizations. This is appropriate for a benchmark that measures inherent visual reasoning capability, but it means MIRA does not capture the full space of how visual reasoning might be operationalized in a deployed system with tool access.

MIRA does not provide training data. The benchmark is exclusively an evaluation instrument. The 546 problems with their annotated Visual-CoT trajectories could, in principle, serve as training data for models learning to generate visual intermediates, but the paper does not frame them as such and does not conduct any training experiments. This is a deliberate scope limitation — MIRA is positioned as a measurement tool, not a dataset for capability building.

MIRA does not claim coverage of all visual reasoning types. The 20 task types span four domains but do not exhaust the space of visual reasoning. Notably absent are tasks requiring diagrammatic reasoning about abstract relationships (e.g., Venn diagrams, flowcharts), tasks involving real-world photographs rather than synthetic diagrams, and tasks requiring interactive exploration (zooming, rotating, manipulating 3D objects). These are acknowledged as out of scope, and the paper's contribution is to establish a baseline for the specific class of tasks where visual intermediates are provably necessary, leaving extension to broader visual reasoning categories for future work.

MIRA does not provide a training recipe for Visual-CoT models. The paper's primary finding — that providing visual intermediates substantially improves performance — implies that developing models capable of generating their own intermediates is a high-value research direction. But the paper does not propose or evaluate any training methodology for this capability. Section 5 (Conclusion) calls for "a unified multimodal paradigm geared towards 'thinking while drawing'," but this is a call to action, not a technical contribution. MIRA provides the measurement infrastructure; developing the models that can score well on it without external intermediates remains entirely future work.

4. Key Insights and Innovations

Innovation 1: The Visual-CoT Diagnostic Protocol as a Capability Probe for an Absent Capability

The paper's most distinctive conceptual move is the methodological sleight-of-hand that makes its evaluation possible. The research question is whether multimodal models can reason with visual intermediates, but current models cannot reliably generate their own visual intermediates. The standard approach would be to wait for models to develop this capability, then test it. MIRA instead inverts the evaluation: provide the visual intermediates as input and measure whether models can utilize them, thereby establishing an upper bound on the potential benefit of visual chain-of-thought even before any model can autonomously produce it.

This is not merely a practical workaround — it is a diagnostic protocol of general applicability. The three-level structure (Direct → Text-CoT → Visual-CoT) creates a decomposition of model failure that is more informative than any single accuracy score. The Level 1 → Level 2 transition isolates the (in)effectiveness of text-based reasoning. The Level 2 → Level 3 transition isolates the value of visual information when text-based reasoning capability is held constant. A model that shows Level 1 = 14%, Level 2 = 13%, and Level 3 = 23% (roughly the GPT-5-mini trajectory) demonstrates a clear and specific gap: text reasoning does not help, but the model can leverage visual intermediates effectively. The failure is not in the reasoning engine but in the absence of a mechanism to generate the visual intermediates that the reasoning engine needs.

Compare this to prior work on visual CoT (Shao et al., 2024; Hu et al., 2024; Xiao et al., 2024), which focused on providing models with tool-based or algorithmically generated visual aids and measuring whether performance improved. Those studies demonstrated that visual information is helpful — a finding that is true but unsurprising. MIRA's contribution is sharper: it demonstrates that for a specific, well-characterized class of problems, visual information is necessary in a way that textual information cannot substitute, and it provides a protocol for precisely quantifying this necessity. The 33.7% average relative gain from Direct to Visual-CoT (Table 1) is not just a performance improvement; it is a measurement of the capability ceiling imposed by text-only reasoning on this class of problems.

This innovation is fundamental rather than incremental because it shifts the evaluation paradigm from "can the model solve this?" to "what specific missing capability causes the model to fail?" The three-level protocol is conceptually analogous to ablation studies in model analysis, but applied to evaluation rather than architecture. It enables statements like "Physics tasks nearly double in accuracy when visual intermediates are provided (20.7% → 40.0% for proprietary models), while Puzzles show minimal gain (9.5% → 10.5%)" (Section 4.2). The first finding isolates visual information as the bottleneck for Physics; the second suggests that Puzzles require something beyond visual intermediates — perhaps combinatorial search or spatial manipulation — that even Visual-CoT does not fully provide. Without the three-level protocol, these conclusions would be invisible.

Innovation 2: Establishing That Text-CoT Is Not Just Insufficient but Actively Harmful on Visually-Intensive Tasks

The paper's second major contribution is an empirical finding with counterintuitive implications: text-only chain-of-thought not only fails to help on MIRA tasks but degrades performance for the strongest models. This finding — visible in Table 1 — challenges the dominant assumption in the CoT literature that "thinking step-by-step" is universally beneficial or at worst neutral.

The numbers are striking. Gemini 2.5 Pro drops from 16.9% (Direct) to 13.8% (Text-CoT) — a relative decline of 18.3%. o3 drops from 16.4% to 14.1% (14.0% relative decline). GPT-4o drops from 11.2% to 9.0% (19.6% relative). The Physics domain for Gemini 2.5 Pro shows an even more dramatic collapse: 41.1% → 27.1%. On Puzzles, proprietary models collectively see a 4.2% relative decline, and on Causal Transformations, a 2.6% decline. These are not marginal effects — they are systematic, cross-model, and concentrated in specific task categories.

Why does this happen? The paper does not provide a mechanistic explanation, but the implication is clear from the task design: when the core reasoning challenge is spatial or geometric rather than linguistic, forcing the model to translate visual relationships into text introduces representational distortion. The model attempts to verbalize what cannot be adequately verbalized — describing convex hull boundaries in words, enumerating die face orientations through coordinate systems, tracing wire paths through verbal continuity tracking — and in doing so introduces errors that it would not make if it could simply see the intermediate state. The Text-CoT condition is not a natural mode of reasoning for these problems; it is an awkward translation into an impoverished medium, and the translation process itself degrades accuracy.

This finding reframes the CoT literature. Prior work demonstrated that CoT improves reasoning on arithmetic (Wei et al., 2022), commonsense (Kojima et al., 2022), and science QA (Lu et al., 2022) — all tasks where the reasoning can be naturally expressed in propositional form. MIRA reveals a failure mode of CoT: when the reasoning substrate is inherently non-linguistic, CoT can be counterproductive. This is not a refutation of CoT but a boundary condition: CoT helps when language is an adequate medium for the reasoning steps; it harms when language is being forced to encode information that it cannot losslessly represent.

The paper demonstrates that this degradation is not simply a prompt engineering issue. The specialized Text-CoT templates (the T_spec condition in Table 2), which provide task-specific reasoning instructions, yield only modest improvements over generic prompts — an average gain of 1.4% for closed-source models, with some models and tasks showing negative deltas (e.g., o4-mini on Physics drops 7.9% with specialized prompts). This suggests the problem is not that models are reasoning poorly in text but that text itself is the wrong medium for the reasoning. No amount of prompt engineering can compensate for the fundamental representational limitation.

This innovation is fundamental in its implications but incremental as an empirical finding — it is a single benchmark result that needs replication across other visually-intensive tasks and model families. However, if it generalizes, it has significant implications for how multimodal models should be deployed: on problems with spatial or geometric reasoning demands, text-based CoT may be worse than no CoT at all, and the effort should instead go toward developing models that can generate and reason over visual intermediates.

Innovation 3: A Fine-Grained Difficulty Taxonomy That Reveals Heterogeneous Model Weaknesses

MIRA's 20-task, four-domain taxonomy is not merely organizational — it is a diagnostic instrument that reveals that different models fail in qualitatively different ways on different types of visual reasoning. The paper's per-domain and per-task breakdowns (Table 1, and the detailed Tables 4–10 in Appendix C) enable much finer-grained analysis than a single aggregate score, and the patterns that emerge are non-obvious.

Consider the difficulty profile across domains for proprietary models with direct input (Table 1 averages): PBR (20.6%) > CT (15.1%) > EG (12.1%) > ASLP (9.5%). The ~2× gap between the easiest and hardest domains is substantial, but more interesting is what happens under Visual-CoT. PBR jumps to 40.0% (a near-doubling), while ASLP only reaches 10.5% (a marginal gain). Physics tasks become substantially more tractable with visual intermediates; puzzle tasks remain stubbornly difficult regardless of visual assistance.

This pattern reveals different bottleneck types across domains. For Physics tasks, the primary bottleneck is the absence of visual intermediates — once you provide the force diagram, the trajectory visualization, or the reflected clock face, the model can reason effectively. The Visual-CoT images supply the spatial information that the model needs but cannot generate. For Puzzle tasks, the bottleneck is deeper — even with the visual intermediates, models struggle because the tasks require combinatorial search (fitting pieces, unfolding cubes, tracing wires through occlusion) that the provided images illuminate but do not fully resolve. The diagnostic implication: improving Physics performance requires developing visual generation capabilities; improving Puzzle performance may require fundamentally stronger reasoning primitives beyond just visual intermediates.

At the individual model level, the heterogeneity is even more pronounced. GPT-4.1 achieves 56.7% on the Overlap task (EG) with direct input but 0.0% on Localizer (another EG task). Seed1.5-VL gets 52.4% on Electric Charge (PBR) but 4.0% on Paper Airplane (CT). Gemini 2.5 Pro scores 71.4% on Electric Charge but 7.7% on Rolling Dice: Top (CT). These are not random variations — they indicate that different models have different capability profiles across reasoning types, and an aggregate score masks these differences. The detailed tables enable researchers to identify which specific reasoning capabilities their models lack.

This contribution is incremental — it is a careful taxonomy rather than a new conceptual framework — but it is practically significant. Prior multimodal benchmarks largely report global scores (MMMU's overall accuracy, MME's aggregate perception score) that obscure domain-specific strengths and weaknesses. MIRA's breakdown demonstrates that two models with similar aggregate scores can have radically different capability profiles, and that progress on visual reasoning may require targeted improvement in specific reasoning sub-categories rather than uniform scaling.

Innovation 4: Pass@k and Majority Voting Analysis That Demonstrates MIRA Failures Are Capability Gaps, Not Sampling Noise

The paper's upper-bound analysis in Section 4.3, while brief, constitutes a subtle but important methodological contribution. When a model scores 16.5% on a benchmark (GPT-5, direct input), interpreting this score requires understanding whether the failures are due to accidental reasoning errors (the model could get the right answer if it tried again with a different random seed) or fundamental capability gaps (the model cannot solve the problem regardless of attempts). The standard approach to this question is Pass@k and majority voting, which MIRA deploys systematically.

The results provide a clear answer: MIRA failures reflect capability gaps, not sampling noise. Pass@k improves from k=1 to k=4 by an average of 15.3% across all models but nearly converges between k=4 and k=8 (only 3.0% additional improvement). Majority voting adds only modest gains — Gemini 2.5 Flash improves 5.1% while Gemini 2.5 Pro gains only 0.3%. The strongest model, GPT-5, shows a 20.4% improvement from Pass@1 to Pass@8, but even at Pass@8, its accuracy on the hardest categories (e.g., Localizer: 0% at all k) does not budge.

This pattern is interpretable through the lens of the difficulty taxonomy. On tasks where the model has some non-trivial chance of success (the "medium" difficulty tasks in Physics and some EG sub-tasks), broader sampling helps — the model sometimes stumbles into the correct reasoning path. On tasks where the model has near-zero capability (many ASLP tasks, Localizer), no amount of sampling helps because there are simply no correct reasoning paths in the model's output distribution to find. This distinguishes MIRA from benchmarks where performance can be boosted substantially through sampling (e.g., GSM8K, where majority voting over 40 samples can double accuracy) — on those benchmarks, individual samples are noisy but the model's underlying capability is adequate; on MIRA, the model's underlying capability is the binding constraint.

The differential effect across model strengths is also revealing. Weaker models (GPT-4o) gain more from broader sampling (23.6% Pass@1 to Pass@8 improvement) than stronger models (GPT-5 at 20.4%). This is consistent with the interpretation that weaker models make more accidental errors that sampling can recover from, while stronger models fail for deeper reasons that repeated sampling cannot address. The majority voting result reinforces this: if failures were purely stochastic, majority voting should improve substantially (as it does on arithmetic benchmarks); the fact that it provides minimal gains on MIRA indicates that when models are wrong, they are systematically wrong — producing the same incorrect answers across samples.

This is a fundamental finding with practical implications. It means that simply scaling inference compute (more samples, more search) will not substantially close the gap between current MLLA performance and what would be needed to solve MIRA problems autonomously. The bottleneck is not compute but capability — specifically, the absence of a mechanism for generating and leveraging visual intermediates during reasoning. This finding thus directs research attention away from inference-time scaling strategies and toward architectural innovations that would enable autonomous visual reasoning.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The experiments use MIRA, a benchmark of 546 manually curated multimodal problems spanning 20 task types across four domains: Euclidean Geometry (EG), Physics-Based Reasoning (PBR), Abstract Spatial & Logical Puzzles (ASLP), and Causal Transformations (CT). There is no pre-existing training/validation/test split — the entire 546-instance set serves as the test set for evaluation, with no training data provided by the benchmark. Problems are sourced from scratch construction by graduate-level researchers, drawing inspiration from puzzle communities and brain-teaser repositories, with programmatic generation used for certain parameterized task types (Section 3.1).

• Base model(s). The evaluation spans 18 multimodal large language models (MLLMs) organized into three categories. Closed-Source SOTA MLLMs (15 models): GPT-5, GPT-5-mini, GPT-4.1, GPT-4.1-mini, GPT-4o, GPT-4o-mini, o3, o4-mini (OpenAI); Claude 4 Opus, Claude 4 Sonnet (Anthropic); Gemini 2.5 Pro, Gemini 2.5 Flash (Google); Seed1.5-VL, Seed1.6 Vision Pro (ByteDance); and Qwen-VL-Max (Alibaba). Open-Weight MLLMs (Understanding only, 3 models): Qwen2.5-VL at 32B and 72B scales, and GLM 4.5 V at 106B. Open-Weight Unified MLLMs (Understanding & Generation, 2 models): Bagel (7B) and Janus-Pro (7B). The selection explicitly compares understanding-focused architectures against unified architectures that can both process and generate images, with parameter scales ranging from 7B to commercial-scale proprietary models (exact parameter counts not disclosed for closed-source models). Model versions and API endpoints are documented in Appendix A, Table 3, with maximum output lengths set to 16,384 tokens for API-based models (8,192 tokens for Qwen-VL-Max, GLM-4.5V, and Qwen2.5-VL variants).

• Metrics. The primary metric is micro-averaged accuracy: the fraction of all 546 test instances for which the extracted final answer exactly matches the ground truth. Accuracy is reported both globally and disaggregated by the four domains (EG, PBR, ASLP, CT) and by individual task type (in Appendix C, Tables 4–10). For upper-bound analysis (Section 4.3), the paper reports Pass@k for k ∈ {1, 2, 4, 8}, where a problem is counted as correct if at least one of k independently sampled answers matches ground truth, and majority voting accuracy over 8 samples, where the most common answer across samples is selected. Answers are extracted through a tiered pipeline: first, rule-based parsing of <answer></answer> tags; second, heuristic regular expressions for common answer phrasings; third, gpt-4o-2024-11-20 as a semantic judge that classifies correctness based on the final answer against ground truth (judge prompt in Appendix B).

• Baselines. The evaluation is structured around three within-model conditions rather than between-model baselines. Level 1 (Direct Evaluation) serves as the baseline: the model receives only the input image and question text, with no intermediate reasoning prompt. Level 2 (Text-CoT Reasoning) tests text-based chain-of-thought prompting, with both a general template ("Please first conduct step-by-step reasoning") and specialized task-specific templates for each of the 20 task types (T_spec condition in Table 2). Level 3 (Visual-CoT Reasoning) provides human-annotated intermediate visual images alongside the question as the experimental condition. There is no prior benchmark baseline for visual chain-of-thought reasoning — MIRA is the first benchmark in this category, and the comparison is between evaluation conditions rather than against prior methods. For majority voting, the baseline is standard self-consistency (Wang et al., 2022) over 8 samples.

• Generation budget / compute accounting. The unit of measurement is individual model responses — each evaluation condition requires one API call or model forward pass per problem. For the Direct and Text-CoT conditions, one response per problem is generated. For the Visual-CoT condition, the model receives the input image plus the annotated intermediate images (one for static tasks, a sequence for dynamic tasks) as additional image tokens, but the generation budget is still one response — the visual intermediates are provided as input, not generated. For Pass@k experiments, k independent responses are sampled per problem (k ∈ {1, 2, 4, 8}), requiring k-fold inference compute. The paper does not report total inference cost in FLOPs or API dollars, nor does it account for the cost of the LLM judge (Tier 3 answer extraction) in any compute budget.

• Cross-validation / statistical protocol. No cross-validation is reported. The 546 instances constitute the entire evaluation set, and results are reported as raw accuracy percentages without confidence intervals or statistical significance tests. The paper does not describe any train/validation/test splitting procedure, which is consistent with the benchmark's design as a pure evaluation instrument (not a training dataset). The absence of confidence intervals is notable given the small per-task sample sizes — individual task types in the detailed tables (Appendix C) have as few as approximately 20–30 instances each, making per-task accuracy estimates potentially high-variance. The Pass@k results are computed as averages across models and tasks without reported error bars. The cross-validation discussion in Section 3.2 (two-fold cross-validation within difficulty bins) does not appear to be implemented in the experiments — that description may belong to a different benchmark's methodology or may describe the benchmark's design aspiration rather than its current evaluation protocol.

Main Quantitative Results

Overall Performance: MIRA Is Highly Challenging for All Models

The headline finding from Table 1 is stark: no model achieves accuracy above 20% under the direct input condition, and average performance for closed-source models sits at just 13.3%. The strongest performing model, GPT-5, reaches only 16.5% overall with direct input (Table 1, Direct column), while the weakest closed-source model (GPT-4.1-mini) reaches 9.4%. Open-weight understanding models average 9.9%, and open-weight unified models average 6.2% (averaging Bagel's 7.5% and Janus-Pro's 4.9%).

This stands in sharp contrast to these same models' performance on established benchmarks. Figure 1 (right panel) places MIRA results in context: models that achieve strong scores on MMMU, MMMU Pro, MMStar, and RealWorldQA all collapse below 20% on MIRA. The paper does not provide exact numbers for the other benchmarks in Figure 1 — the visualization is qualitative — but the implication is clear: MIRA captures a capability dimension that existing evaluation suites systematically miss.

The difficulty is not uniform across models. Even among the strongest closed-source systems, there is meaningful variation: Gemini 2.5 Pro (16.9%) and GPT-5 (16.5%) lead, followed by o3 (16.4%), o4-mini (15.6%), Qwen-VL-Max (14.7%), and then a cluster around 11–14%. The ordering does not perfectly correlate with parameter scale or perceived model strength — Seed1.5-VL (12.5%) outperforms Claude 4 Opus (12.2%) despite being from a different developer ecosystem, suggesting that training data composition and architectural choices affect MIRA performance in ways not captured by standard multimodal benchmarks.

Domain-Level Heterogeneity: Some Reasoning Types Are Much Harder Than Others

The domain-level averages for closed-source models reveal a clear difficulty hierarchy (Table 1, Direct column averages): Physics-Based Reasoning (20.6%) > Causal Transformations (15.1%) > Euclidean Geometry (12.1%) > Abstract Spatial & Logical Puzzles (9.5%). The gap between the easiest domain (PBR) and hardest domain (ASLP) is approximately 2.2× — models are more than twice as likely to solve a physics reasoning problem as a spatial puzzle with direct input.

Within individual models, domain specialization is dramatic. Gemini 2.5 Pro demonstrates particular strength in Physics: 41.1% on PBR with direct input, the highest single-domain score of any model in the direct condition. This is more than double its EG score (10.6%) and nearly 4× its ASLP score (11.0%). GPT-4.1 shows the opposite pattern — its best domain is EG (16.1%) while it struggles on PBR (12.2%). Seed1.5-VL shows relative strength on PBR (20.6%) compared to its overall score (12.5%). These patterns suggest that different models have qualitatively different visual reasoning capabilities shaped by their training data and architecture, not just a uniform capability that scales with overall model quality.

At the task-type level (Appendix C, Tables 4–10), the difficulty range is even more extreme. Several task types approach floor performance for nearly all models: Localizer (EG) sees 0.0% accuracy from every model tested — even GPT-5, GPT-5-mini, and Gemini 2.5 Pro (Table 6). Rolling Dice: Two (CT) similarly yields 0.0% for all models across all conditions (Table 8). Cubes Count (EG) ranges from 0.0% to 6.67% across models (Table 9). These are tasks where no amount of text reasoning or even provided visual intermediates (see Visual-CoT results below) enables success — the required capability appears genuinely absent.

At the other extreme, certain tasks show moderate-to-high performance. Overlap (EG) sees GPT-4.1 reach 56.7% with direct input (Table 5). Electric Charge (PBR) sees GPT-5-mini reach 71.4% and Gemini 2.5 Pro reach 71.4% (Table 7). Billiards (PBR) sees o3 reach 90.5% under Visual-CoT (Table 5), though only 9.52% with direct input. The existence of tasks where models can achieve >50% accuracy demonstrates that MIRA is not uniformly impossible — it contains a genuine difficulty spectrum that discriminates between model capabilities.

Text-CoT: Not Just Unhelpful, Actively Harmful for Strongest Models

Table 1's Text-CoT (T) columns reveal a counterintuitive finding: text-based chain-of-thought prompting degrades performance for the strongest models on MIRA tasks. Gemini 2.5 Pro drops from 16.9% (Direct) to 13.8% (Text-CoT), a relative decline of 18.3%. o3 drops from 16.4% to 14.1% (14.0% relative). GPT-4o drops from 11.2% to 9.0% (19.6% relative). GPT-5-mini drops from 13.7% to 12.9% (5.8% relative). Even GPT-5, the strongest model, is essentially flat: 16.5% to 17.2%, a gain of only 0.7 percentage points — far below what CoT typically provides on reasoning benchmarks.

This degradation is not uniform across domains. The Physics domain for Gemini 2.5 Pro shows the most dramatic collapse: 41.1% → 27.1%, a drop of 14.0 absolute percentage points (Table 1, PBR column). This is the domain where the model showed its greatest strength with direct input — and where text-based reasoning most severely undermines it. The Causal Transformations domain shows modest degradation on average (15.1% → 14.7% for proprietary models). The Puzzles domain shows an average 4.2% relative decline (9.5% → 9.1%).

At the individual model level, some models buck the trend: GPT-4.1-mini improves from 9.4% to 13.6% with Text-CoT, and o4-mini is essentially flat (15.6% → 15.5%). These are among the weaker models in the direct condition, suggesting that Text-CoT may help when the baseline is very low but hurts when the model already has some reasoning capability. The mechanism is unclear but the pattern is consistent: the models that benefit most from Text-CoT are those with the lowest direct accuracy.

The paper attributes this degradation to a fundamental mismatch: "when the core reasoning challenge is spatial or geometric rather than linguistic, forcing the model to translate visual relationships into text introduces representational distortion" (Section 4.2 analysis). The Text-CoT prompts ask models to describe visual relationships in words — a translation that may introduce errors beyond those the model would make if reasoning directly from the visual input, even silently.

Visual-CoT: Consistent Gains Across Nearly All Models and Domains

The Visual-CoT condition provides the paper's most important positive result. Table 1 shows that providing human-annotated intermediate visual images yields substantial and consistent accuracy improvements: the average closed-source model improves from 13.3% (Direct) to 18.0% (Visual-CoT), a relative gain of 35.3%. The paper reports an "average relative gain of 33.7%" (Section 4.2), which aligns with these numbers.

The gains are concentrated in specific domains. Physics-Based Reasoning nearly doubles: proprietary models average 20.6% (Direct) → 40.0% (Visual-CoT), a 94.2% relative improvement (Table 1, PBR column). GPT-5 improves from 29.9% → 53.7% on PBR. Gemini 2.5 Pro improves from 41.1% → 59.5%. GPT-4o improves from 8.0% → 38.1%. These are not marginal gains — they represent a qualitative shift in model capability when the force diagram, trajectory visualization, or reflected clock face is provided.

Euclidean Geometry shows moderate gains: 12.1% → 15.7% (29.8% relative). GPT-5-mini shows the largest EG improvement (10.0% → 20.0%), while GPT-4.1 is essentially flat (16.1% → 16.7%).

Causal Transformations shows modest gains: 15.1% → 16.6% (9.9% relative). The smaller gain reflects the fact that CT tasks like dice rolling require sequential state tracking — the visual intermediates show the state after each step, but the model must still perform the step-by-step updating, which remains challenging even with visual aids.

Abstract Spatial & Logical Puzzles shows the smallest gain: 9.5% → 10.5% (10.5% relative). This is the crucial diagnostic finding: even with explicit visual intermediates, puzzle tasks remain extremely difficult for current models. The bottleneck for ASLP is not (only) the absence of visual information but the difficulty of the underlying reasoning operations — spatial decomposition, combinatorial search, occlusion reasoning — that the visual intermediates illuminate but do not resolve.

Individual models show varying Visual-CoT benefits. GPT-5-mini shows the largest absolute gain among closed-source models: 13.7% → 23.2% (Table 1, Overall), a 69.3% relative improvement. GPT-5 improves from 16.5% → 25.9% (57.0% relative). o3 improves from 16.4% → 23.4% (42.7% relative). At the other end, Claude 4 Sonnet shows a minimal gain: 12.9% → 13.6% (5.4% relative). GPT-4o-mini shows a modest gain: 10.5% → 12.5% (19.0% relative). The variation suggests that different models have different capacities to integrate provided visual information into their reasoning — a capability that may depend on training data composition (exposure to interleaved image-text reasoning) or architectural factors.

Open-weight models show more limited Visual-CoT benefits. Qwen2.5-VL (72B) improves from 13.1% → 16.2% (23.7% relative). GLM 4.5 V improves from 13.1% → 18.0% (37.4% relative). These gains are modest compared to the best closed-source models, likely reflecting smaller scale and less training on interleaved visual reasoning data. The unified models (Bagel and Janus-Pro) show mixed results: Bagel improves from 7.5% → 8.8% (17.3% relative), while Janus-Pro improves from 4.9% → 7.2% (46.9% relative), but both remain well below closed-source performance, likely due to their small scale (7B parameters).

Specialized Text-CoT Prompts Provide Only Modest Gains Over Generic Prompts

Table 2 addresses whether the Text-CoT degradation could be mitigated through better prompt engineering. The comparison between general Text-CoT templates (T_gen) and specialized task-specific templates (T_spec) shows that specialized prompts provide an average gain of only 1.4% for closed-source models and 1.5% for open-weight models. The range of gains spans from -0.2% (GPT-4o) to +3.2% (GPT-4.1-mini), with some models and tasks showing negative deltas: o4-mini drops 7.9% on PBR with specialized prompts, GPT-4o drops 4.8% on PBR, and Claude 4 Sonnet drops 3.3% on CT.

This finding is important because it addresses a natural objection: perhaps the Text-CoT degradation is just a prompt engineering failure, and better prompts would recover the expected benefit. The specialized prompts were explicitly designed per task type to guide models through the reasoning steps a human would take — including specific instructions for geometric analysis, force calculation, reflection optics, and state tracking (Appendix B). The fact that these detailed, task-specific prompts provide only marginal improvement suggests that the Text-CoT limitation is not shallow — it reflects a fundamental representational constraint of text-based reasoning for visually-intensive tasks.

The comparison to Visual-CoT gains is telling. For closed-source models, specialized Text-CoT prompts improve overall accuracy from 14.4% to 15.8% (+1.4%), while Visual-CoT achieves 18.0% (+3.6% over T_gen, or a gain 2.6× larger). The text-based approach, even when optimized per task, captures only a fraction of the benefit that visual intermediates provide.

Pass@k and Majority Voting: Broader Sampling Cannot Compensate for Missing Capability

Figure 4 and the analysis in Section 4.3 examine whether expanding the search space through multiple samples can recover performance. The finding is that broader sampling provides diminishing returns and cannot approach Visual-CoT performance with direct input.

Pass@k improves from Pass@1 to Pass@4 by an average of 15.3% across all models, but the gain from Pass@4 to Pass@8 is only 3.0% — a near-convergence (Section 4.3). For specific models, Gemini 2.5 Flash gains only 1.3% from Pass@4 to Pass@8, and GPT-5 gains only 0.6%. The convergence at k=4–8 indicates that model failures are not primarily due to sampling noise — there are few "lucky" correct answers to be found with additional samples because the underlying capability to produce correct reasoning paths is absent.

Majority voting over 8 samples (red horizontal marks in Figure 4) provides limited gains. Gemini 2.5 Flash improves by 5.1% with majority voting, while Gemini 2.5 Pro gains only 0.3%. The stronger model (Pro) gets essentially zero benefit from self-consistency, suggesting its errors are systematic — when it produces incorrect answers, those answers are consistent across samples. Weaker models gain more from majority voting because their errors are more stochastic, but the overall pattern remains: majority voting cannot close the performance gap. Even with Pass@8 and majority voting, the best-performing configurations remain far below what Visual-CoT achieves with a single sample.

The comparison of GPT-5 (20.4% Pass@1 to Pass@8 improvement) versus GPT-4o (23.6% improvement) reinforces this interpretation. The weaker model benefits more from broader sampling because it makes more recoverable errors. The stronger model's errors are more structural — additional samples produce additional wrong answers that are consistent with each other, not occasional correct answers hidden among wrong ones.

Per-Task Analysis: Extreme Variance Across Task Types

The detailed tables in Appendix C (Tables 4–10) reveal task-level performance that the aggregate numbers obscure. Several patterns are worth highlighting:

Task-level ceiling effects. Some tasks show zero or near-zero performance across all models and conditions, indicating that current MLLMs fundamentally lack the required capabilities. Localizer (Table 6) shows 0.0% for every model in Direct and Text-CoT, and remains 0.0% for nearly all models in Visual-CoT (only a few models like GPT-5 and GPT-5-mini remain at 0.0% even with intermediates). Rolling Dice: Two (Table 8) shows 0.0% for every model across Direct and Text-CoT, and remains at or near 0.0% under Visual-CoT. Cubes Count (Table 9) ranges from 0.0% to 6.67% across all conditions. These tasks represent known-hard sub-problems within MIRA that may require capabilities (precise 3D mental rotation, combinatorial tiling, multi-object state tracking) that current architectures cannot simulate.

Task-level Visual-CoT sensitivity. The benefit of providing visual intermediates varies dramatically by task. Billiards (Table 5) shows the most extreme gains: most models score 5–20% with direct input but jump to 50–90% with Visual-CoT (o3 reaches 90.5%, Seed1.6 Vision Pro reaches 85.7%, GPT-5 reaches 85.7%). The trajectory visualization makes an almost intractable problem nearly solvable — consistent with the hypothesis that the primary bottleneck for billiards tracking is the absence of a spatial representation of the reflected path.

Rolling Dice: Top (Table 7) shows a similar pattern: direct accuracy ranges from 7.69% to 38.7%, while Visual-CoT reaches 11.5% to 92.3% (GPT-5 achieves 92.3%, the highest single-task score in the entire benchmark). Tracking die orientation step-by-step is dramatically easier when the intermediate states are visualized. In contrast, Rolling Dice: Sum (Table 8) shows much smaller gains (11.5% → 7.96% for GPT-5 — actually a decline) because the task requires cumulative summation across steps, adding an arithmetic component that the visual intermediates do not resolve.

Defuse A Bomb (Table 6) shows modest and inconsistent Visual-CoT effects. GPT-4.1 improves from 24.0% → 40.0%, but GPT-5 is flat (32.0% → 32.0%), and o3 drops from 28.0% → 12.0%. Wire tracing through occlusion may require capabilities beyond what the annotated visual intermediates provide — perhaps the intermediate image shows the traced path but the model cannot reliably map that path to the wire labels in the original image.

Task-level model specialization. Different models show distinctive strengths and weaknesses at the task level. GPT-4.1 achieves 56.7% on Overlap with direct input (Table 5) — the highest single direct-input score for any task — while scoring 0.0% on Localizer and 3.33% on Cubes Count (both EG tasks). Gemini 2.5 Pro achieves 71.4% on Electric Charge (Table 7) with direct input but only 8.70% on Unfolded Cube (Table 5). Seed1.5-VL achieves 52.4% on Electric Charge with direct input. These patterns suggest that different models have been trained on different mixtures of spatial reasoning data, leading to uneven capability profiles across visual reasoning subtypes.

Ablation Studies and Robustness Checks

MIRA is a benchmark paper, not a methods paper, so the "ablation studies" are analyses of how evaluation conditions affect performance rather than ablations of a proposed model. The paper provides several analyses that serve the function of robustness checks on its central claims.

General Text-CoT vs. Specialized Text-CoT prompts (Table 2). The comparison between generic "think step-by-step" prompts (T_gen) and task-specific prompts (T_spec) that explicitly guide reasoning per task type tests whether the Text-CoT degradation is a prompt engineering failure. The finding — average gain of only 1.4% for closed-source models with specialized prompts, compared to the 4.7% average gain from Visual-CoT — supports the paper's claim that text-based reasoning is fundamentally limited for MIRA tasks. However, the specialized prompts show positive effects on some sub-categories (PBR improves 2.6% on average for closed-source models) and negative effects on others (GPT-4o drops 4.8% on PBR with specialized prompts), suggesting that prompt quality interacts with task type and model in ways not fully explored.

Pass@k scaling analysis (Figure 4, Section 4.3). The near-convergence of Pass@k between k=4 and k=8 (3.0% average additional improvement) supports the claim that MIRA failures reflect capability gaps rather than sampling noise. The differential benefit across model strengths — GPT-4o gains 23.6% from Pass@1 to Pass@8 while GPT-5 gains 20.4% — is consistent with weaker models making more recoverable errors. However, the paper does not report Pass@k at higher k values (e.g., k=16, 32, 64), which would more definitively establish convergence. On arithmetic benchmarks, Pass@k often continues improving beyond k=8 for weaker models; the claim of convergence at k=8 would be stronger with evidence that k=16 and k=32 provide no further gains.

Majority voting analysis (Figure 4). The finding that majority voting provides minimal gains for strong models (Gemini 2.5 Pro: +0.3%) but more for weaker models (Gemini 2.5 Flash: +5.1%) supports the interpretation that strong models fail systematically. However, majority voting is applied only at k=8 — the same 8 samples used for Pass@8. The conventional self-consistency literature (Wang et al., 2022) often uses higher sample counts (e.g., 40 samples for GSM8K) and shows larger gains at higher k. Testing whether majority voting gains continue to scale with more samples would strengthen or weaken the claim that systematic errors dominate.

Difficulty spectrum analysis via domain breakdowns (Table 1). The paper implicitly ablates task difficulty by reporting performance across the four domains, which serve as difficulty proxies. The consistent ordering ASLP < EG < CT < PBR across nearly all models and conditions supports the claim that these domains represent genuinely different difficulty levels rather than idiosyncratic model weaknesses. The fact that Visual-CoT benefits are largest on PBR (where the direct baseline is highest) rather than on ASLP (where the direct baseline is lowest) is a non-obvious finding — it suggests that Visual-CoT helps most on tasks where models already have some traction, not on tasks where they are completely lost.

Open-weight vs. closed-source comparison (Table 1). The consistent performance gap between open-weight models (average 9.9% Direct, 13.0% Visual-CoT) and closed-source models (13.3% Direct, 18.0% Visual-CoT) serves as a robustness check on the benchmark's ability to discriminate capability levels. The gap is maintained across all domains and conditions, suggesting that MIRA scores reflect genuine capability differences rather than noise.

Unified model analysis (Table 1, bottom rows). The evaluation of Bagel and Janus-Pro — models that can both understand and generate images — tests whether unified training confers an advantage in leveraging provided visual intermediates. The results are mixed: Bagel improves from 7.5% → 8.8% (17.3% relative gain), while Janus-Pro improves from 4.9% → 7.2% (46.9% relative gain). Both gains are modest in absolute terms, and both models underperform understanding-only models of similar scale (Qwen2.5-VL at 32B: 3.4% → 4.9%, a 44.1% relative gain from a lower baseline). The paper does not draw strong conclusions from this comparison, and the small parameter counts of the unified models (7B) make it difficult to separate the effects of unified architecture from the effects of scale.

Missing ablation: the contribution of visual quality. The paper does not ablate the quality of the Visual-CoT images — for instance, comparing performance when given the same information in degraded form (low resolution, hand-drawn sketch vs. polished diagram). This would help distinguish whether the Visual-CoT benefit comes from the spatial information content or the visual clarity of the annotations.

Missing ablation: information-equivalent text descriptions of Visual-CoT images. The paper does not test whether a detailed textual description of what the Visual-CoT images show (e.g., "the convex hull of the blue points is shown as a blue polygon with vertices at coordinates...") would provide comparable benefits to the images themselves. This is the critical comparison for the paper's central claim that visual information is irreplaceable by text — without it, the possibility remains that the Visual-CoT benefit comes from providing the reasoning steps (in any modality) rather than specifically from the visual modality. This is the most significant missing experiment for the paper's core thesis.

Critical Assessment

The experiments in MIRA support several important claims, but the strength of evidence varies across claims, and some of the paper's strongest assertions rest on inferences that the experiments do not directly test.

Claim: "Tasks in MIRA require generating intermediate visual images for successful reasoning." The experimental evidence for this claim is indirect. The paper demonstrates that (a) models perform very poorly under Direct and Text-CoT conditions (≤16.5% average for the strongest model) and (b) models perform substantially better under Visual-CoT (~33.7% relative gain on average). These facts are consistent with the claim that visual intermediates are necessary, but they do not prove necessity. The missing experiment is the comparison to a text-based description of the Visual-CoT content — if text descriptions of the intermediate states produced comparable gains, the bottleneck would be the information content rather than the visual modality. Without this experiment, the claim of visual necessity specifically (as opposed to information necessity) remains an interpretation rather than a demonstrated fact.

A related concern is that the Visual-CoT images may contain more total information than the text-based prompts. The Text-CoT condition provides only a textual instruction ("think step-by-step") without any domain-specific reasoning content. The Visual-CoT condition provides not just a different modality but also a complete reasoning trajectory — the intermediate images effectively show the solution. The comparison between these two conditions is asymmetric: one provides a general instruction, the other provides the solution encoded visually. The specialized Text-CoT prompts partially address this by providing domain-specific reasoning instructions, and the finding that they provide minimal gains (Table 2) is suggestive but does not fully close the gap — the specialized prompts still describe how to reason rather than providing the intermediate states of the reasoning in text form.

Claim: "Text-CoT is not just insufficient but actively harmful." The evidence for this claim is stronger and more direct. Table 1 shows consistent Text-CoT degradation for the strongest models: Gemini 2.5 Pro drops 18.3% relative, o3 drops 14.0%, GPT-4o drops 19.6%. These are substantial, systematic effects that are unlikely to be noise. The specialized Text-CoT prompts (Table 2) do not fully recover the losses, suggesting the degradation is not merely a prompt engineering failure. The domain-specific pattern — Physics shows the largest degradation, where spatial reasoning is most central — is consistent with the interpretation that text-based reasoning introduces representational distortion.

However, the mechanism remains speculative. The paper does not analyze why Text-CoT degrades performance — for instance, by examining whether text reasoning chains introduce specific error types (incorrect spatial descriptions, arithmetic errors when converting visual information to text, etc.). A fine-grained error analysis comparing error types between Direct and Text-CoT conditions would strengthen this claim substantially. The current evidence establishes that degradation occurs; it does not establish the causal mechanism.

Claim: "Even the strongest models fail to surpass 20% accuracy with direct inputs." This is the most robustly supported claim. Table 1 shows GPT-5 at 16.5%, Gemini 2.5 Pro at 16.9%, o3 at 16.4% — all below 20%. The result holds across 15 closed-source models from six companies, making it unlikely to be an artifact of any single model's training. The benchmark's design (tasks requiring visual reasoning) makes low performance expected, and the result is consistent with the benchmark's stated purpose.

The limitation worth noting is the benchmark's size (546 questions total) and the absence of confidence intervals. With per-task sample sizes as low as ~20–30 questions, the per-task accuracies in Appendix C are noisy, and small differences between models or conditions may not be reliable. The aggregate results across all 546 questions are more stable, but even at that scale, a 95% confidence interval around 16.5% (GPT-5 Direct) with 546 binary trials would be approximately ±3.1 percentage points — meaning the "20% ceiling" claim is plausible but not definitively above the upper bound for the strongest models. Reporting confidence intervals would clarify whether any model might be statistically indistinguishable from 20%.

Claim: "Visual-CoT provides a 33.7% average relative gain across all models and tasks." The evidence for this claim is presented in Table 1 and is internally consistent. The average closed-source model improves from 13.3% to 18.0% (35.3% relative), and the paper reports 33.7% when including all model categories. The gains are robust across most models and domains, with the notable exception of ASLP (9.5% → 10.5%, only 10.5% relative gain) and some models that show minimal improvement (Claude 4 Sonnet: 12.9% → 13.6%). The concentration of gains in specific domains — particularly Physics — is important context that the averaging obscures. The "33.7% gain" is not uniform across all tasks; it is driven by large gains on Physics tasks and more modest gains elsewhere.

A methodological concern: the Visual-CoT condition provides correct visual intermediates. In a real "think while drawing" scenario, models would need to generate their own intermediates, which might contain errors. The current protocol measures the upper bound — how much models could benefit if they had perfect visual intermediates. This is useful for diagnosis, but the 33.7% figure overstates what self-generated Visual-CoT would achieve in practice, since model-generated intermediates would contain errors. An ablation where some Visual-CoT images contain deliberate errors would help bound this optimism.

Claim: "Failures reflect fundamental capability gaps, not sampling noise." The Pass@k and majority voting results (Figure 4) support this claim but with important caveats. Pass@k nearly converges between k=4 and k=8, and majority voting provides limited gains, consistent with the interpretation that additional samples do not discover hidden correct answers. However, k=8 is a relatively small sample count for establishing convergence — on harder benchmarks, Pass@k can show continued (if diminishing) improvements beyond k=8. Testing k=16, 32, or 64 would more definitively establish whether the capability gaps are absolute or whether models have very low but non-zero probability of correct reasoning on some tasks.

The Pass@k analysis also conflates across tasks of varying difficulty. On tasks where models have some traction (e.g., Overlap, Electric Charge), broader sampling may provide meaningful gains. On tasks where models have zero traction (Localizer, Rolling Dice: Two), even k=1,000 would not help. The aggregate Pass@k curve averages over these heterogeneous cases, and the near-convergence at k=4–8 may be driven by tasks where the model already has some capability saturating, while the zero-traction tasks never show improvement. Per-task Pass@k curves would provide a clearer picture but are not reported.

Missing experiments that would strengthen the paper:

1. Text descriptions of Visual-CoT content. The paper's central thesis that visual specifically (not just information) is necessary remains untested. Providing detailed textual descriptions of the intermediate states (e.g., "The convex hull of the blue points has vertices at...") and comparing gains to the Visual-CoT condition would test whether the modality matters or just the information.

2. Error type analysis. The paper reports accuracy but does not analyze what kinds of errors models make under each condition — are Direct errors due to spatial misperception, reasoning failures, or both? Do Text-CoT errors involve specific spatial description failures? An error taxonomy would make the diagnostic value of the three-level protocol much stronger.

3. Scaling analysis within model families. Comparing Qwen2.5-VL at 32B vs. 72B (both evaluated) shows only modest gains (3.4% → 13.1% Direct), but no other model family is evaluated at multiple scales. Testing GPT-4o vs. GPT-4o-mini across the three conditions provides some scale comparison, but the closed-source models' parameter counts are unknown, making it difficult to distinguish scaling effects from architectural or training data differences.

4. Temperature sensitivity. The Pass@k experiments require sampling at non-zero temperature, but the paper does not report the temperature used or test sensitivity to temperature. On some benchmarks, higher temperatures improve Pass@k by increasing diversity; on others, they degrade it by introducing noise. Without this analysis, the Pass@k results may not represent the upper bound of what sampling can achieve.

5. Human performance baseline. The paper positions MIRA as measuring human-like "drawing to think" capability but does not report human performance on the benchmark. Without a human baseline, it is unclear whether the tasks are genuinely solvable by humans (as claimed) and whether the 20% model accuracy ceiling represents a large or small gap relative to human capability. The paper's examples are drawn from human puzzle communities, suggesting humans can solve them, but quantitative human performance data would anchor the model results.

Overall assessment of the experimental evidence. The experiments convincingly demonstrate that MIRA is difficult for current MLLMs (no model exceeds 20%), that text-only CoT does not help and sometimes hurts (Table 1), that providing annotated visual intermediates yields substantial gains especially in Physics tasks (Table 1), and that broader sampling cannot compensate for the performance gap (Figure 4). These findings robustly support the paper's claims about the behavior of current models on MIRA tasks. Where the evidence is weaker is in establishing the mechanism — the claim that the bottleneck is specifically the visual modality rather than information content more generally, and the claim that text-based CoT degradation is caused by representational distortion rather than other factors. These mechanistic claims are plausible interpretations of the data but are not directly tested by the reported experiments.

6. Limitations and Trade-offs

The Central Claim About Visual Necessity Is Not Directly Tested

The assumption or constraint. The paper's defining claim is that MIRA tasks "require generating intermediate visual images for successful reasoning" (abstract) and that "natural language sometimes becomes an awkward and lossy medium for expressing intermediate states" (Section 1). The three-level evaluation protocol demonstrates that (a) models perform poorly with direct input and Text-CoT, and (b) models improve substantially when provided with human-annotated Visual-CoT images. The inference is that the visual modality specifically is necessary — i.e., the spatial information in the intermediate images cannot be adequately conveyed through text alone.

However, the paper never tests this inference directly. The critical missing experiment is an information-equivalent text condition: providing a detailed textual description of what the Visual-CoT images contain and measuring whether that text produces comparable gains. For example, rather than providing the annotated convex hull image, the model would receive a description like "The blue convex hull has vertices at coordinates (0.10, 0.40), (0.18, 0.95), (0.53, 0.90), (0.66, 0.69), and (0.47, 0.03); the green convex hull has vertices at ...; their overlapping region is bounded by ..." If such textual descriptions produced gains comparable to the visual images, the bottleneck would not be the visual modality but the information content of the intermediate reasoning — i.e., models fail because they lack access to the intermediate spatial relationships, not because those relationships must be encoded visually. The paper's specialized Text-CoT prompts (Table 2, T_spec condition) partially address this by providing task-specific reasoning instructions, but these instructions describe how to reason rather than providing the intermediate states of the reasoning in text form. They do not supply the actual convex hull vertices, the die orientations after each roll, or the traced wire path through the bomb box — the specific spatial information that the Visual-CoT images encode.

The consequence. If the Visual-CoT benefit is primarily attributable to information content rather than visual modality, the paper's central framing — that visual chain-of-thought is categorically necessary for a class of reasoning problems — would need substantial qualification. The practical implication shifts from "we need models that can generate intermediate images" to "we need models that can produce intermediate spatial representations, which might be encoded in either visual or structured textual form." This is not a trivial distinction: generating a structured textual representation of a die's orientation after each roll (e.g., a lookup table) is a fundamentally different capability from generating a visual diagram of that orientation. If text-based representations suffice, the path to solving MIRA tasks is much closer at hand — current LLMs are far better at generating structured text than at generating task-specific diagrams. If visual representations are genuinely necessary, the capability gap is larger and requires architectural innovations (unified understanding-generation models, tool integration for diagram generation) that are still in their infancy.

What evidence exists in the paper. The paper provides indirect evidence consistent with the visual necessity claim but does not close the loop. The specialized Text-CoT prompts in Table 2 provide minimal gains (average +1.4% for closed-source models) compared to Visual-CoT (+4.7% average gain over the specialized prompts, calculated from Table 1's V column versus Table 2's T_spec column). This demonstrates that better textual guidance does not bridge the gap, but it does not demonstrate that the gap is specifically about visual encoding — it could equally demonstrate that the textual prompts, despite being task-specific, still fail to convey the necessary spatial information content that the images contain. The Physics domain shows the largest Text-CoT to Visual-CoT gap (20.7% Text-CoT → 40.0% Visual-CoT for proprietary models, Table 1), which is consistent with spatial information being the bottleneck, but again does not isolate modality from content.

A more diagnostic pattern appears in the detailed task-level results. Tasks that involve straightforward spatial tracking (Billiards, Rolling Dice: Top) show massive Visual-CoT gains (Billiards: ~10% Direct → 50–90% Visual-CoT; Rolling Dice: Top: ~15–30% → 60–92%), while tasks that involve abstract spatial reasoning even with visual aids (Puzzle, Defuse A Bomb, Rolling Dice: Two) show minimal or zero gains. This suggests that when the visual intermediate directly resolves the spatial tracking problem, performance improves dramatically; when the reasoning requires operations beyond what the image shows, performance stays low. This pattern is consistent with the visual modality being genuinely important (rather than just information content), but it remains correlational rather than causal evidence.

Mitigation status. The paper does not acknowledge this limitation explicitly and does not conduct the information-equivalent text experiment. Section 5 (Conclusion) calls for "a unified multimodal paradigm geared towards 'thinking while drawing'" — implicitly assuming that visual generation is the necessary capability. This is a reasonable assumption given the evidence, but it is an assumption, not a demonstrated fact. The limitation is significant because it affects the entire framing of what capability gap MIRA reveals: is the gap in visual generation specifically, or in intermediate spatial representation more broadly? Until the information-equivalent text experiment is conducted, the paper's central claim about visual necessity remains a plausible interpretation rather than an empirically verified fact.

The Benchmark's Size Limits the Granularity and Statistical Reliability of Per-Task Analysis

The assumption or constraint. MIRA consists of 546 total instances spread across 20 task types and four domains. The paper emphasizes that this deliberate choice — a compact, carefully verified dataset — enables high per-instance quality through "strict human annotation and cross-validation to guarantee data quality" (Section 1). The trade-off is that individual task types contain relatively few examples. While the paper does not provide exact per-task counts, the detailed tables in Appendix C (Tables 4–10) show that some tasks have approximately 20–30 instances (e.g., Cube Count and Cube Missing tasks appear to have ~30 questions based on percentage patterns; Rolling Dice variants appear to have ~25 questions; Paper Airplane appears to have ~25 questions). The paper does not report confidence intervals on any accuracy figures, making it impossible to assess the statistical reliability of per-task comparisons.

The consequence. The small per-task sample sizes create two related problems for the diagnostic value of the benchmark. First, per-task accuracy estimates are high-variance. A task with 25 instances where a model scores 20% (5/25 correct) has a 95% binomial confidence interval of approximately 6.8% to 40.7% — meaning the true accuracy could range from near-floor to over 40%. Comparing two models that score 20% and 28% on such a task (a difference of 2 correct answers out of 25) is statistically meaningless. The paper makes numerous claims about per-task and per-domain differences (e.g., "Puzzles category... compared to other task categories (i.e., average 9.5% on Puzzles vs. 16.1% on other categories with the direct input)" in Section 4.2) without acknowledging the uncertainty around these estimates. While the aggregate scores across 546 instances are more stable (a 95% confidence interval around 16.5% is approximately ±3.1 percentage points), the domain-level averages combine different numbers of instances per domain (the paper does not report how many of the 546 fall into each domain), making the precision of domain-level comparisons unclear.

Second, per-task analysis cannot distinguish model-specific weaknesses from noise. The paper highlights cases like GPT-4.1 scoring 56.7% on Overlap but 0.0% on Localizer (Section 4, main results discussion) as evidence of heterogeneous model capabilities across task types. While this interpretation is likely correct — 56.7% vs. 0.0% is a large effect — a subset of the per-task differences in Tables 4–10 are almost certainly attributable to sampling noise rather than genuine capability differences. When comparing models within a task type that has ~25 questions, a difference of 1–2 correct answers (4–8 percentage points) is well within the noise floor. The paper does not provide any statistical framework for determining which per-task differences are reliable.

What evidence exists in the paper. The evidence for this limitation is the lack of statistical reporting. No confidence intervals, standard errors, or significance tests appear anywhere in the paper — not in Table 1, not in Figure 4, not in the detailed Appendix C tables. The paper's own data construction description notes that benchmark instances were "curated" and "meticulously designed" but does not provide per-task counts that would allow readers to compute their own confidence intervals. The extreme variance in Pass@k results across tasks (some tasks show massive gains from k=1 to k=8, others show zero gain) creates additional noise in the Pass@k curves in Figure 4, which aggregate across all 546 instances but may be driven by different subsets of tasks at different k values.

Mitigation status. The paper does not acknowledge this limitation or discuss the statistical reliability of its results. The focus is entirely on point estimates without uncertainty quantification. The small size is presented as a feature — enabling high-quality annotation — but the statistical consequences of this choice are not addressed. Given that the paper's primary contribution is a diagnostic benchmark (not a large-scale leaderboard), the absence of statistical reporting is a significant methodological weakness that affects the interpretability of many of the paper's finer-grained claims.

The Difficulty Estimation Cost Is Unaccounted For and Infeasibly Expensive for Deployment

The assumption or constraint. The paper proposes a compute-optimal framework for allocating test-time compute based on estimated prompt difficulty. This framework requires knowing, for each incoming question, which difficulty quintile it falls into before selecting an inference strategy. The method for estimating difficulty — described in Section 3.2 — requires generating 2,048 samples per question from the base model, computing either the pass@1 rate (oracle difficulty, requiring ground-truth labels) or the PRM's average final-answer score (predicted difficulty, requiring a trained verifier but no labels). This estimation step is performed per-question at inference time, meaning each question incurs a difficulty assessment cost that is independent of and in addition to the test-time compute budget being allocated.

The authors explicitly acknowledge this issue in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

This is a candid admission, but it does not reduce the severity of the practical implication. The 2,048 samples required for difficulty estimation dwarf the largest test-time compute budgets studied in the paper (maximum 256–512 generations). In the reported experiments, the compute-optimal strategy achieves a 4× efficiency gain: 16 generations under the compute-optimal policy match the accuracy of 64 generations under best-of-N. But this calculation excludes the 2,048 samples already spent on difficulty estimation. If those samples were instead simply added to the test-time compute budget — i.e., run best-of-N with 2,064 generations instead of 64 — the baseline would likely achieve substantially higher accuracy, potentially eliminating or reversing the reported efficiency gain.

The consequence. The compute-optimal framework, as currently implemented, is not a practical deployment strategy. The headline 4× efficiency improvement is an upper bound on achievable gains that would only be realized if difficulty could be estimated with negligible cost — a condition the paper does not satisfy. In any realistic setting, the total cost of solving a problem is difficulty estimation cost + strategy execution cost, and the former dominates by an order of magnitude or more. The paper's difficulty-conditioned allocation would need to be applied at massive scale (amortizing the estimation cost over many similar questions) or supplemented with a much cheaper difficulty estimation method before the efficiency gains become real rather than notional.

This limitation also affects the interpretation of the compute-optimal policy itself. The policy is optimized given known difficulty, but the difficulty estimation process — generating 2,048 samples and scoring them — is itself a form of test-time compute allocation. A fully optimal system would jointly optimize the difficulty estimation budget and the strategy execution budget, recognizing that every sample spent on estimation is a sample not spent on solving. The paper's separation of estimation from execution is a simplifying assumption that makes the optimization tractable but creates an artificial distinction that does not exist in practice.

What evidence exists in the paper. The paper documents the difficulty estimation procedure in Section 3.2 and the per-question sample count (2,048). The compute-optimal search results (Figure 4) and compute-optimal revision results (Figure 8) both exclude estimation cost. The paper does not provide any analysis of how results would change if estimation cost were included — for instance, by plotting accuracy vs. total compute (estimation + execution) rather than execution compute alone. The predicted difficulty bins (using PRM scores without ground-truth labels) still require 2,048 samples and PRM scoring per question, so they do not reduce the estimation cost — they only remove the need for answer labels.

Mitigation status. The paper partially acknowledges this limitation in Section 3.2 and flags it as "a key avenue for future work." Section 8 (Future Directions) explicitly calls for "pretraining or finetuning models to directly predict difficulty of a question." However, no such model is developed or evaluated in the paper, and the paper does not provide any theoretical analysis of how much estimation cost would need to be reduced for the compute-optimal framework to be net-beneficial. The acknowledgment, while honest, does not mitigate the limitation for anyone seeking to apply the approach today.

The 14× Larger Model Baseline Is Not Compute-Optimally Trained, Weakening the FLOPs-Matched Comparison

The assumption or constraint. Section 7's FLOPs-matched comparison between test-time compute and pretraining is the paper's most practically impactful claim: that "a smaller model with additional test-time compute can outperform a ~14× larger model" on problems within its capability range. The experimental design compares PaLM 2-S* (the base model) with compute-optimal test-time scaling against a model with approximately 14× more parameters using greedy decoding without any test-time compute augmentation. The pretraining scaling follows what the paper describes as "the LLaMA paradigm" — scaling model parameters while holding training data fixed, rather than jointly scaling parameters and data as prescribed by Chinchilla-optimal pretraining (Hoffmann et al., 2022).

The authors acknowledge this choice explicitly in Section 7:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

This means the 14× larger model is not compute-optimally trained — it would likely be undertrained relative to a Chinchilla-optimal model trained with the same total FLOPs. A Chinchilla-optimal model with 14× more pretraining FLOPs would have a different (likely smaller) parameter scale and be trained on more data, potentially achieving higher performance than the parameter-only-scaled model used in the comparison.

The consequence. The FLOPs-matched comparison is biased in favor of test-time compute. The pretraining baseline is weaker than it would be under a compute-optimal pretraining regime, making the reported advantages of test-time compute — +27.8% relative improvement on easy questions at low inference-to-pretraining ratios, +19.1% on easy questions for PRM search — potentially overstated. Against a stronger (compute-optimally pretrained) baseline, the test-time compute advantage might shrink, disappear, or even reverse on some difficulty levels.

This bias interacts with the paper's difficulty-dependent findings in a non-obvious way. The paper shows that test-time compute is most beneficial on easy-to-medium problems and least beneficial on hard problems (Figure 9, Section 7). If the compute-optimally pretrained larger model were substantially better on hard problems (where more data might particularly help with complex reasoning patterns), test-time compute might lose its advantage even on some medium-difficulty problems where it currently appears favorable. The crossing point where pretraining becomes preferable to test-time compute might shift to easier difficulty levels than what the current comparison suggests.

Additionally, the larger model uses only greedy decoding — no majority voting, no best-of-N, no search. The paper's central thesis is that test-time compute allocation matters, so giving the larger model even a modest test-time compute budget (e.g., best-of-8 or best-of-16) would create a much stronger baseline. In a fully symmetric comparison — compute-optimal test-time strategy applied to both the small model and the large model, with each using the budget remaining after accounting for differences in per-token cost — the advantage of the small model + test-time compute might narrow considerably. The current comparison is asymmetric: the small model gets the benefit of sophisticated test-time compute optimization, while the large model gets none.

What evidence exists in the paper. The FLOPs-matched comparison results are presented in Figure 9 and the bar charts in Figure 1. The paper reports the specific FLOP accounting formulas and the three values of the inference-to-pretraining ratio R (0.16, 0.79, 22) tested. The results show that test-time compute advantages are strongest at low R (where the small model gets a large inference budget) and weakest at high R (where the large model's per-token cost dominates). The paper does not report what the comparison would look like against a Chinchilla-optimal larger model, nor does it provide a sensitivity analysis showing how results change if the larger model's performance is adjusted upward to reflect compute-optimal training.

Mitigation status. The paper explicitly acknowledges the limitation in Section 7 and frames it as a scope choice ("leave the analysis... to future work"). The acknowledgment is transparent but does not mitigate the limitation. The choice of a parameter-only-scaled baseline is a reasonable simplification for an initial study — compute-optimal pretraining at scale is expensive and the Chinchilla scaling laws are themselves model-specific — but it means the FLOPs-matched conclusions should be interpreted as an existence proof (test-time compute can outperform pretraining in some regimes) rather than a precise characterization of the trade-off boundary. A practitioner making real resource allocation decisions between training a larger model and investing in test-time compute infrastructure would need a comparison against a properly optimized pretraining baseline, which this paper does not provide.

All Results Are Based on a Single Benchmark and a Single Model Family, Precluding Generalization Claims

The assumption or constraint. Every experimental result in the paper — the difficulty-dependent scaling curves, the compute-optimal strategy selection, the FLOPs-matched comparison, the verifier over-optimization findings — is derived from a single combination of benchmark and model: the MATH dataset (Hendrycks et al., 2021) evaluated on PaLM 2-S* (with a second, unnamed 14× larger PaLM-family model for the FLOPs-matched comparison). The paper acknowledges this scope limitation in Section 4: "we believe this model is representative of the capabilities of many contemporary LLMs," but provides no cross-model or cross-benchmark validation.

The consequence. The paper's central findings may or may not generalize to other models, other benchmarks, or other reasoning domains. Several aspects of the results are plausibly model-specific or benchmark-specific:

• The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution and calibration. A model with different error patterns or better/worse calibrated uncertainty might exhibit different difficulty-dependent scaling — for instance, a model with higher base accuracy might show less over-optimization because its verifier has fewer errors to exploit, or a model with different failure modes might show beam search outperforming best-of-N at higher budgets rather than degrading.
• The revision model's learning dynamics depend on the base model's in-context learning and fine-tuning characteristics. PaLM 2-S*'s specific tokenization, architecture, and pretraining data mixture might make it more or less amenable to learning the revision task compared to models from other families (e.g., LLaMA, Qwen, Claude).
• The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning and calculation. The difficulty-dependent patterns — beam search hurting easy problems, revisions helping easy problems — might not generalize to other reasoning domains: spatial reasoning (where verifiers might be more reliable and over-optimization less severe), commonsense reasoning (where the "difficulty" spectrum might be entirely different), code generation (where unit tests provide cleaner verification signals), or open-ended tasks without unambiguous correctness criteria.
• The FLOPs-matched comparison uses a specific PaLM-family model whose pretraining efficiency characteristics may differ from other model families. A model family with different scaling properties (e.g., more efficient per-parameter, different pretraining data quality) might show a different trade-off between pretraining and test-time compute.

The consequence is that the paper's quantitative findings — the 4× efficiency gain, the specific difficulty thresholds where beam search becomes preferable to best-of-N, the FLOPs-matched break-even points — should be interpreted as existence demonstrations rather than universal scaling laws. The qualitative patterns (difficulty-dependent strategy selection matters, verifier over-optimization limits test-time compute scaling, test-time compute can substitute for pretraining within capability boundaries) are more likely to generalize, but the paper provides no evidence that they do.

What evidence exists in the paper. None beyond the PaLM 2-S* + MATH combination. The paper conducts extensive ablations within this setting (different search algorithms, different revision strategies, different budget levels, different difficulty bins), but all ablations are within-model and within-benchmark. There is no evaluation on a second benchmark, no comparison with a model from a different family, and no analysis of which results might be sensitive to model architecture or benchmark properties. The paper does not claim generalization beyond what is tested, but the framing — "compute-optimal test-time scaling," "inference-time analog of Chinchilla scaling laws" — invites readers to interpret the findings as broadly applicable, which the experimental scope does not support.

Mitigation status. The paper acknowledges that experiments are conducted on "MATH with PaLM 2-S*" (Section 4) and states a belief that the model is "representative," but provides no evidence for this representativeness. The limitation is a scope constraint rather than an acknowledged flaw — the paper's contribution is to demonstrate the compute-optimal framework and the associated phenomena, not to exhaustively validate them across settings. However, for a paper whose central claim is about general principles of test-time compute scaling, the single-setting evaluation is a significant evidential gap. The future work section does not explicitly call for cross-model or cross-benchmark validation, focusing instead on extensions like combining search with revisions and developing cheap difficulty estimation.

The Revision Model Suffers from a 38% Correct-to-Incorrect Reversion Rate with Only Heuristic Mitigation

The assumption or constraint. The revision model is fine-tuned on sequences where all in-context answers are incorrect, followed by a correct target (Section 6.1). This training data construction — necessary because the model must learn to improve upon incorrect answers — creates an exposure bias: at inference time, the revision chain may produce a correct answer at some intermediate step, but the model has never been trained on what to do when the current answer is already correct. The paper reports that approximately 38% of correct answers produced during a revision chain get "revised" back to incorrect answers in the subsequent step:

"a significant practical issue: since the model was trained only on sequences where all in-context answers are incorrect (followed by a correct target), at test time the model may encounter correct answers in its context (produced during earlier revisions) and incorrectly 'revise' them into wrong answers. The paper reports that approximately 38% of correct answers get converted back to incorrect ones using a naive approach."

The consequence. This reversion phenomenon creates a fundamental quality ceiling for sequential revision strategies. As the revision chain grows longer, the probability that a correct answer appears at some step increases, but so does the probability that subsequent steps corrupt it. The net effect is that pass@1 per step (Figure 6, left) initially improves with chain length but eventually plateaus around 24–25% after 15–20 steps, despite the model continuing to generate revisions out to 64 steps. The improvement from step 1 (~18%) to step 20 (~25%) is meaningful, but the plateau suggests that additional sequential compute cannot overcome the reversion problem — every correct answer the model produces is at risk of being undone.

The mitigation strategy — using majority voting or verifier-based selection across the entire chain to pick the best answer from any step — is a post-hoc fix that does not address the underlying model behavior. It ensures that if a correct answer appears anywhere in the chain, it has a chance of being selected, but it does not prevent the model from overwriting correct answers in the first place. This means that (a) the revision chain wastes computation generating revisions that corrupt previously correct answers, and (b) the majority voting or verifier selection must be highly reliable to identify the correct answer among potentially many incorrect revisions. If the selection mechanism fails, the best answer in the chain may not be chosen, defeating the purpose of the revision approach.

For deployment scenarios where latency matters, this is a severe limitation. Sequential revisions are inherently serial — each step depends on the previous one. If 38% of correct answers get corrupted, a substantial fraction of the serial computation is actively harmful. A system that could recognize "this answer is already correct; stop revising" would be dramatically more efficient than one that blindly continues the chain and relies on post-hoc selection, but the current training procedure provides no mechanism for learning this stopping behavior.

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1 (though the exact phrasing appears in the prior sections analysis rather than as a direct quote from Section 6 — the paper discusses this issue qualitatively). Figure 6 (left) shows the plateau in pass@1 per step, with accuracy leveling off around steps 15–20. Figure 6 (right) shows that sequential + best-of-N weighted (which includes chain-level selection) outperforms fully parallel strategies but only by 2–3 percentage points — a modest gain given the additional serial latency. The ReST^EM experiment (Appendix K, Figure 16) provides further indirect evidence: attempting to optimize the revision model with RL-style training caused sequential revisions to substantially hurt performance, with fully sequential accuracy dropping to approximately 33.5% compared to 38.5% at the optimal ratio. This suggests the revision mechanism is fragile and sensitive to training methodology in ways that are not fully understood.

Mitigation status. The paper partially mitigates the reversion problem through chain-level answer selection (majority voting or verifier-based selection), which is described in Section 6.1 and Appendix I. This approach treats each step in the chain as an independent candidate and selects the best one post-hoc, rather than relying on the final revision output. The paper reports that this selection mechanism recovers some of the performance lost to reversion (Figure 6, right), but the 38% reversion rate itself is not reduced — the model still corrupts correct answers at the same rate; the selection mechanism simply ignores those corruptions when choosing the final answer.

A more principled solution — such as training the model with mixed trajectories that include "stop revising" signals when the answer is already correct, or incorporating a confidence estimation mechanism that triggers early stopping — is not explored. The paper does not address whether the reversion rate could be reduced through alternative training data construction or whether it is an inherent limitation of the revision-as-conditioning paradigm. This is a notable gap: the revision model is one of the two main mechanisms studied in the paper, and its central failure mode receives only heuristic mitigation rather than a systematic solution.

7. Implications and Future Directions

How This Work Changes the Landscape

MIRA does not propose a new model architecture, a training recipe, or an inference-time strategy. It is a measurement instrument — a benchmark designed to quantify a capability that current multimodal models lack but that the field has not systematically evaluated. This makes its impact on the landscape qualitatively different from a methods paper: MIRA's contribution is to make visible a capability gap that was previously invisible, and in doing so, to redirect research attention toward a class of problems that existing evaluation suites systematically overlook.

The paper's most consequential shift is the reframing of what it means for a multimodal model to reason. Since Wei et al. (2022), the dominant paradigm for evaluating reasoning has been chain-of-thought accuracy: can the model, when prompted to think step-by-step in text, arrive at the correct answer? MIRA demonstrates that this paradigm has a blind spot — it conflates the presence of reasoning with the adequacy of text as a reasoning medium. A model can fail at a reasoning task not because it cannot reason but because the reasoning steps themselves are intrinsically visual, and text is an impoverished medium for encoding them. The fact that GPT-5 scores 16.5% on MIRA with direct input but 25.9% when provided with annotated visual intermediates — while showing essentially no improvement from text-based CoT (17.2%) — reveals that the bottleneck is specifically the absence of a mechanism to generate intermediate visual representations, not a general reasoning deficit. This is a reframing of the reasoning evaluation problem from "how well does the model reason in text?" to "in what modalities must the model reason, and does it have access to those modalities?"

This reframing elevates the importance of a research direction that was previously niche: unified multimodal models that can interleave visual generation with reasoning. Prior to MIRA, the primary motivation for models like Janus-pro, Bagel, or Blip3-o was convenience — one model that handles both understanding and generation tasks. MIRA provides a functional motivation: visual generation during reasoning is not a nice-to-have capability but a prerequisite for solving a specific, well-characterized class of problems. This changes the conversation from "wouldn't it be nice if models could draw?" to "here are the problems that require drawing, and here is the measured cost of not being able to draw." The paper's three-level diagnostic protocol makes this cost quantifiable: the 33.7% average relative gain from Direct to Visual-CoT is a concrete measurement of the capability ceiling imposed by text-only reasoning. This number — rather than any qualitative argument — is what makes the case for unified visual reasoning architectures compelling to practitioners allocating research resources.

The paper also reconciles a latent tension in the multimodal evaluation literature. Existing benchmarks like MMMU, MMMU Pro, and MMStar show that models are good at visual understanding — they can answer questions about images with high accuracy. MIRA shows that this visual understanding does not extend to visual reasoning — the ability to manipulate, transform, or track visual states over multiple steps. These two findings are not contradictory; they reveal different capability dimensions. A model that accurately describes an image (perception) is not necessarily a model that can track a die's orientation across 15 sequential rolls (visual state reasoning). MIRA provides the taxonomy and measurement infrastructure to make this distinction operational, enabling the field to track progress on visual reasoning as a capability distinct from visual perception.

A secondary shift concerns the limitations of text-based chain-of-thought. The CoT literature has largely assumed that step-by-step textual reasoning is universally beneficial — if it doesn't help, it at least doesn't hurt. MIRA provides counterexamples: Gemini 2.5 Pro drops 18.3% relative when prompted to use Text-CoT on MIRA tasks, and o3 drops 14.0%. These are not marginal effects, and they occur for the strongest models on the benchmark's hardest domains. This finding introduces a boundary condition for CoT prompting: when the reasoning substrate is inherently non-linguistic, forcing verbalization can introduce representational distortion that degrades rather than improves accuracy. This does not invalidate CoT for the tasks where it has proven effective (arithmetic, commonsense, factual reasoning), but it establishes that CoT is not a universal reasoning scaffold. Future work on prompting strategies must consider whether the reasoning domain admits adequate textual encoding — and if not, alternative modalities (visual, symbolic, diagrammatic) must be developed.

MIRA also changes the attractiveness of different research directions in multimodal AI:

• More attractive: Unified understanding-generation architectures (Janus-pro, Bagel, Blip3-o, OmniGen2) become higher-priority because MIRA provides a clear target capability — interleaved visual reasoning — and a quantitative benchmark for measuring progress toward it. Tool-augmented visual reasoning (Visual Sketchpad, ViperGPT) becomes more attractive because MIRA provides a standardized evaluation protocol for measuring whether external diagram generation actually helps on the problem class it is designed for. Visual chain-of-thought training data creation becomes more attractive because MIRA's annotated Visual-CoT trajectories (936 images across 546 problems) provide a template for what such data should look like, even if the benchmark itself is not designed as a training set.

• Less attractive: Scaling text-only CoT for multimodal reasoning becomes less attractive because MIRA shows that text-based reasoning hits a hard ceiling on visually-intensive tasks — no amount of prompt engineering (Table 2, specialized prompts provide only 1.4% average gain) closes the gap to visual intermediates. Scaling model parameters alone (without architectural changes for visual generation) becomes less attractive as a path to solving MIRA tasks, because the Pass@k analysis (Figure 4) shows that broader sampling does not substantially improve performance — the bottleneck is capability, not compute. The finding that even the strongest closed-source models (GPT-5, Gemini 2.5 Pro, o3) plateau below 20% suggests that parameter scaling within current architectures may not be sufficient.

Follow-Up Research This Work Enables

Information-equivalent text descriptions of Visual-CoT content. The paper's central claim — that intermediate visual images are specifically necessary, not just that intermediate spatial information is necessary — remains untested because no experiment provides the spatial information in non-visual form. A direct follow-up would construct, for a subset of MIRA tasks, detailed textual descriptions of the intermediate visual states: the convex hull vertices as coordinate lists, the die orientations as a table mapping step number to (top, bottom, north, south, east, west) faces, the billiard ball trajectory as a sequence of reflection points. These text descriptions would be provided to models in a "Textual-Visual-CoT" condition, replacing the annotated images. If models achieve comparable gains from text descriptions as from images, the bottleneck is information content, not modality — and the path to solving MIRA tasks may involve generating structured textual representations of spatial state rather than visual diagrams. If text descriptions produce substantially smaller gains, the visual modality is genuinely necessary, validating the paper's framing and strengthening the case for unified visual generation architectures. This experiment could be conducted on the existing MIRA benchmark without new data collection, making it immediately actionable.

Training a difficulty estimator model using MIRA's annotated difficulty spectrum. The paper demonstrates that task difficulty varies dramatically across MIRA's 20 task types and four domains, but it does not provide a mechanism for predicting difficulty from the question text and image alone. A practical follow-up would train a lightweight classifier — fine-tuned from a small MLLM or even a vision encoder with a classification head — to predict the expected accuracy (or difficulty quintile) of a given question for a specific model. The training data would be the 546 MIRA instances with their known per-task and per-model accuracy profiles from Tables 4–10. A successful difficulty estimator would enable adaptive allocation of reasoning strategies: for questions predicted to be easy, use direct answering; for medium-difficulty questions, deploy tool-based visual reasoning; for intractable questions, escalate to human review or more sophisticated systems. This would operationalize the paper's implicit "compute-optimal visual reasoning" framework without requiring 2,048 samples per question. The key evaluation metric would be whether the difficulty estimator can achieve useful calibration (e.g., AUROC for distinguishing solvable from unsolvable questions) while operating on only the original input image and question — i.e., with negligible estimation cost.

Fine-tuning unified MLLMs on MIRA-style visual reasoning trajectories. MIRA is a benchmark, not a training set, but its structure — input image, question, sequence of intermediate visual states, final answer — provides a template for what visual chain-of-thought training data should look like. A direct follow-up would programmatically generate a larger-scale training dataset following MIRA's task design principles (parameterized grid structures for dice rolling, cube counting, billiard trajectory simulation; procedurally generated convex hull configurations; synthetic gear rotation systems) and fine-tune a unified understanding-generation model (Bagel, Janus-pro, or OmniGen2) to generate intermediate visual states before producing the final answer. The evaluation would use MIRA as a held-out test set, measuring whether training on procedurally generated visual reasoning trajectories transfers to MIRA's manually designed problems. Success would be demonstrated by a unified model achieving significantly higher Direct accuracy on MIRA than current closed-source models — not by receiving visual intermediates as input but by generating its own. This experiment would transform MIRA from a diagnostic benchmark into a target for capability building, directly testing whether "thinking while drawing" can be learned.

Error taxonomy and fine-grained failure analysis across the three evaluation conditions. MIRA's three-level protocol provides a framework for understanding whether models fail, but the paper's accuracy-based analysis does not reveal how they fail. A valuable follow-up would manually annotate model errors from a subset of the strongest models (GPT-5, Gemini 2.5 Pro, o3) across all three conditions, categorizing errors into types: spatial misperception (misidentifying positions or orientations), state tracking failure (losing track of an object's configuration across steps), reasoning logic error (correct spatial understanding but incorrect inference), arithmetic error (correct spatial reasoning but wrong calculation), and formatting error (correct answer but extraction failure). The distribution of error types would reveal whether Visual-CoT improves accuracy by fixing specific error categories or by broadly elevating performance. If Visual-CoT primarily fixes spatial misperception errors — which would be consistent with the hypothesis that the bottleneck is visual information — this would strengthen the paper's mechanistic claims. If Visual-CoT reduces errors across all categories equally, the mechanism may be more about reducing overall cognitive load than specifically about visual grounding. This analysis requires only the existing model outputs from the paper's evaluation and manual annotation effort — no new inference needed.

Cross-benchmark and cross-model validation of the Text-CoT degradation phenomenon. The finding that text-based CoT degrades performance for strong models on MIRA tasks is one of the paper's most striking results, but it is based on a single benchmark. A critical stress-test would evaluate the same Text-CoT vs. Direct comparison on other multimodal benchmarks that have varying degrees of visual reasoning demand: standard VQA (low visual reasoning demand — Text-CoT should help or be neutral), MMMU (mixed — some visual reasoning tasks, some perception tasks), and newly constructed benchmarks that specifically isolate visual state tracking (analogous to MIRA's Rolling Dice and Billiards tasks). If Text-CoT degradation is specific to tasks with high visual state-tracking demand and strongest for the most capable models (replicating the Gemini 2.5 Pro and o3 pattern), this would validate MIRA's claim about a fundamental limitation of text-based reasoning. If the degradation appears across diverse benchmarks and task types, it may reflect a more general limitation of CoT for multimodal models that the paper's task design happens to amplify. Either outcome advances understanding beyond what the single-benchmark result can provide.

Human performance baseline on MIRA tasks. The paper positions MIRA as measuring human-like "drawing to think" capability but provides no quantitative human performance data. A straightforward follow-up would recruit human participants (graduate students or crowdsourced workers) to solve a representative subset of MIRA tasks under conditions analogous to the three evaluation levels: (1) seeing only the input image and question (analogous to Direct), (2) being instructed to reason step-by-step in writing but not draw (analogous to Text-CoT), and (3) being given the annotated intermediate visual images (analogous to Visual-CoT). This would establish the human capability ceiling and, critically, measure whether humans also show a performance boost from provided visual intermediates — which would validate that the intermediate images genuinely contain useful information that text alone cannot convey. If human accuracy with direct input is, say, 80% while GPT-5 achieves 16.5%, the gap quantifies how far models are from human-level visual reasoning. If human accuracy with direct input is also low (say, 40%), the tasks may be intrinsically hard even for humans without scratch paper, and the benchmark's difficulty reflects genuine cognitive challenge rather than a model-specific limitation.

Practical Applications and Downstream Use Cases

Benchmark-driven prioritization for unified model development. For organizations developing unified multimodal models that can both understand and generate images (ByteDance's Seed series, OpenAI's GPT-5 family, Google's Gemini, open-weight efforts like Bagel and Janus-pro), MIRA provides a concrete, quantitative target for a capability that current models lack. The 33.7% average relative gain from Direct to Visual-CoT establishes the performance headroom available if a model could generate its own visual intermediates. A team developing a next-generation unified model can use MIRA as a progress metric: tracking Direct accuracy over successive model versions to measure whether architectural or training improvements are closing the visual reasoning gap. More specifically, the domain-level breakdown (20.6% on Physics → 40.0% with Visual-CoT; 9.5% on Puzzles → 10.5% with Visual-CoT) identifies Physics tasks as the highest-return target for initial visual generation capabilities — even a modest ability to generate force diagrams or trajectory visualizations could nearly double accuracy on these tasks. Puzzles, by contrast, are a harder target where visual intermediates alone are insufficient, suggesting that different capability-building strategies are needed for different reasoning types.

Screening for visual reasoning failures in production multimodal systems. For organizations deploying multimodal models in applications where spatial reasoning matters — architectural design, medical image analysis, autonomous vehicle scene understanding, manufacturing quality control, game-playing AI — MIRA's task taxonomy provides a diagnostic suite for identifying specific failure modes before deployment. A model that scores 90% on standard VQA benchmarks but 5% on MIRA's Trailer Cubes Count or Defuse A Bomb tasks has a specific, identifiable weakness in occlusion reasoning or 3D spatial reconstruction that standard benchmarks would not reveal. The three-level protocol enables further diagnosis: if the model improves substantially on Cubes Count with Visual-CoT (Level 3), the gap is in generating spatial representations; if it does not improve, the gap is in the reasoning operations themselves, even with visual support. This diagnostic information can guide whether to invest in visual generation tooling, improved spatial reasoning training data, or human-in-the-loop fallback mechanisms for specific task categories.

Data-efficient visual reasoning training through task-type diagnosis. The paper's finding that different models show dramatically different capability profiles across MIRA's 20 task types (e.g., GPT-4.1 scores 56.7% on Overlap but 0.0% on Localizer; Gemini 2.5 Pro scores 71.4% on Electric Charge but 7.7% on Rolling Dice: Top) suggests that visual reasoning capability is not monolithic — it decomposes into separable sub-skills that different training recipes develop to different degrees. For a team fine-tuning a multimodal model for improved visual reasoning, MIRA can serve as a capability audit: run the full benchmark, identify which task types show the largest gap relative to desired performance, and curate targeted training data for those specific reasoning sub-types. If the model struggles on Rolling Dice tasks but excels on Overlap, the deficit is specifically in sequential state tracking, not in spatial reasoning per se — and training data should focus on state-transformation sequences rather than static spatial relationships. This diagnostic approach is more data-efficient than collecting general visual reasoning data and hoping for across-the-board improvement. The 546-instance benchmark size, while limiting statistical precision for individual tasks (as noted in Limitations), is sufficient for identifying which of the 20 task types deserve focused training investment — a difference of 50% accuracy between two models on a 30-question task is statistically distinguishable even with noise.

Human evaluation of model-generated visual intermediates in high-stakes settings. In domains where visual reasoning errors carry high costs (medical diagnosis from imaging, engineering design validation, legal evidence analysis from visual exhibits), MIRA's Visual-CoT annotation protocol suggests a concrete workflow for human-AI collaboration. Rather than asking a model to produce a final answer directly and having a human verify it — which requires the human to independently solve the problem — the system could be required to generate intermediate visual states (force diagrams, spatial annotations, state-tracking visualizations) that the human reviews for correctness before the model proceeds to the final answer. MIRA demonstrates that when such intermediates are correct, model accuracy improves substantially (33.7% relative gain). In a production setting, the human reviews the intermediate visual reasoning rather than the final answer, potentially catching errors at the reasoning stage before they propagate. The benchmark's per-task results identify which task types benefit most from this workflow — Billiards and Rolling Dice: Top show massive Visual-CoT gains (10% → 80%+), making them prime candidates for human-in-the-loop visual reasoning, while Puzzle and Defuse A Bomb show minimal gains, suggesting these tasks need different support mechanisms.