Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs

🎯 Pitch

Current multimodal LLMs score below 50% on visual reasoning tasks that humans ace at 80%, and their performance doesn't drop as problems get harder—revealing they lack core mental simulation abilities, not just struggle with complexity. This gap persists even when models correctly identify all the relevant visual regions, pinpointing a failure in the procedural reasoning that converts perception into correct answers.

1. Executive Summary

This paper introduces Mind's Eye, a multiple-choice benchmark of eight visuocognitive tasks designed to evaluate the visual reasoning capabilities of multimodal large language models (MLLMs), organized under a novel Abstraction–Relation–Transformation (ART) taxonomy that decomposes fluid visual intelligence into pattern induction (e.g., identifying which figure violates a latent hierarchical rule), analogical relation mapping (e.g., tracking corresponding elements across a dynamic transformation sequence), and mental spatial manipulation (e.g., mentally rotating a 3D polycube assembly). Evaluating 18 MLLMs—including GPT-4o, GPT-o3, and Gemini-2.5 pro—against a human baseline reveals a persistent gap: non-expert humans average 80% accuracy while the best models remain below 50%, with the largest deficits concentrated in Transformation and Abstraction tasks that require internal simulation rather than surface pattern matching. Critically, while human accuracy degrades systematically across easy, medium, and hard items, MLLM performance remains flat across difficulty levels, establishing that current models lack foundational visuocognitive operations rather than merely struggling with complexity—they can localize relevant visual regions through attention yet fail to perform the procedural reasoning operations that would convert correct perceptual grounding into correct answers.

2. Context and Motivation

The Core Problem: We Don't Know Whether MLLMs Can Truly Reason About Visual Structures

The fundamental question this paper tackles is deceptively simple: do multimodal large language models actually reason about visual input, or do they merely describe what they see? This distinction matters because as MLLMs become stronger at surface-level tasks—identifying objects in photographs, reading text from signs, describing spatial relationships in natural scenes—there is a growing temptation to treat them as having general visual intelligence. The paper pushes back against this assumption by asking a more demanding question: can these models perform the kinds of internal cognitive operations over visual representations that come naturally to humans, such as mentally rotating a 3D shape to determine whether it matches a target, tracking how holes would distribute across an unfolded sheet of paper after folding and punching, or inducing the abstract rule that unifies a set of varied visual instances while excluding an outlier?

This matters for several practical and theoretical reasons the paper establishes throughout Sections 1 and 3:

Deployment in high-stakes domains: If MLLMs are deployed for tasks requiring genuine spatial reasoning—architectural planning, robotic manipulation, medical image interpretation, or STEM education—their inability to perform mental transformations could produce confident-sounding but catastrophically wrong outputs that are hard to detect without expert verification.
The gap between perception and cognition: Current benchmarks primarily test whether models can encode visual inputs and map them to linguistic outputs. The paper argues that this conflates seeing with reasoning about what is seen. A model might correctly describe a scene while being unable to answer questions that require manipulating the visual information internally.
Understanding model limitations for research prioritization: If the bottleneck is truly cognitive (inability to perform operations like mental rotation or folding) rather than perceptual (inability to extract relevant features from the image), then scaling models or improving vision encoders alone won't close the gap—fundamentally different architectural mechanisms may be needed.

Three Key Gaps in Existing Evaluation Practices

The paper identifies specific limitations in how MLLMs are currently evaluated for visual reasoning. These are not minor quibbles but structural issues that make it difficult to draw conclusions about whether models possess genuine visuocognitive capabilities.

Gap 1: Failure to isolate visuospatial transformation as a distinct capability. Existing benchmarks span a wide range of visual tasks—object recognition, VQA, OCR, mathematical reasoning with diagrams—but they rarely demand what the paper calls internal simulation: the ability to mentally rotate, fold, recompose, or otherwise manipulate visual representations to predict outcomes not directly shown in the input. For example, a model might correctly answer questions about a photograph of a folded paper with holes, but to do so it only needs to match the visible hole positions, not simulate the folding and punching process to infer the hole pattern on the unfolded sheet.

The paper traces this gap through the benchmark landscape (Table 1, Section 2): general-purpose suites like MMBench, SEED-Bench, MathVista, and MMMU prioritize breadth and scale but "lack parametric control for studying visuospatial understanding." Compositional reasoning benchmarks like CLEVR, RAVEN, and CV-Bench test attribute binding and relational comparison over single-frame perception but "offer limited control over geometric transformations (e.g., rotation angle, fold parity) or mental manipulation." Even cognitively motivated benchmarks like Bongard-LOGO (rule induction) and ARC (abstraction and analogy) emphasize rule discovery over stepwise geometric simulation.

Gap 2: Conflation of visual evidence with linguistic priors. Many existing benchmarks can be solved, at least partially, by exploiting correlations between question text and answer patterns without deeply engaging with the visual input. This is the well-documented "language shortcut" problem (Suhr et al., 2019). In the context of visual reasoning, a model might learn that certain verbal descriptions correlate with certain answer choices based on patterns in training data, rather than deriving the answer from visual evidence. The paper explicitly calls this out in Section 1: "Most existing studies often conflate visual evidence with linguistic priors, leaving it unclear on whether models reason from images or exploit language shortcuts."

The implication is profound: if we cannot distinguish between a model that reasons visually and one that exploits linguistic-pattern-based heuristics, we cannot meaningfully assess progress toward genuine visuospatial intelligence. The field risks optimizing for benchmark scores that reflect clever pattern matching rather than the cognitive capabilities those benchmarks were designed to measure.

Gap 3: Absence of diagnostic infrastructure for understanding why models fail. Most benchmarks report aggregate accuracy scores without providing mechanisms to attribute failures to specific reasoning errors. Did the model fail at mental rotation because it cannot perform the rotation at all, because it confuses a rotation with a mirror reflection, or because it correctly imagined the rotation but selected the wrong color-matched option? Without diagnostic distractors—wrong-answer options designed to reveal specific error types—evaluation remains binary (correct/incorrect) and yields limited insight for improving models.

The paper contrasts this with the psychometric tradition in cognitive science, where test items are constructed with foil answers that are systematically wrong in specific ways, enabling fine-grained diagnosis of which cognitive operations succeed or fail. This diagnostic approach is largely absent from MLLM evaluation, where distractors are often randomly sampled or constructed for surface plausibility rather than cognitive diagnosis.

Where Prior Approaches Fall Short: A Systematic Breakdown

The paper provides a taxonomy of existing evaluation approaches and their limitations, both in the main text (Section 2) and in the comparative analysis of Table 1.

General-purpose multimodal benchmarks (MMBench, SEED-Bench, MMMU, MathVista) are valuable for measuring broad capability but suffer from several defining weaknesses for studying visuospatial reasoning. They lack parametric control, meaning there is no way to systematically vary the difficulty of a visuospatial operation (e.g., incrementally increasing rotation angle or fold complexity) while holding other factors constant. Without this control, it is impossible to determine whether models exhibit the sensitivity to structural complexity that characterizes human spatial reasoning. These benchmarks also do not provide psychometric grounding—they are not designed to measure specific cognitive constructs from established theories of intelligence—and they do not offer diagnostic distractors keyed to specific reasoning errors. As the paper states in Table 1, they lack formal psychometric taxonomy, distractor-based diagnostics, and parametric control, leaving them insufficient for the kind of fine-grained analysis the paper pursues.

Compositional reasoning benchmarks (CLEVR, RAVEN, CV-Bench) target a narrower slice of visual reasoning—attribute binding, relational comparison, and pattern completion—but remain anchored in single-frame perception. CLEVR tests whether models can count objects, compare attributes, and reason about spatial relationships in rendered scenes, but these operations require encoding and comparing visible features, not mentally transforming them. RAVEN's Progressive Matrices test abstract pattern induction across a 3×3 grid, which is closer to the Abstraction dimension in Mind's Eye, but the paper notes (Table 1) that RAVEN lacks formal psychometric grounding, diagnostic distractors, and parametric difficulty control. More critically, these benchmarks do not probe the Transformation dimension at all—they test whether models can recognize patterns and compare relationships, not whether they can simulate spatial operations.

Cognitive and analogical reasoning benchmarks (Bongard-LOGO, ARC, VisuLogic, Mind the Gap, VisFactor, IQBench) come closest to the paper's objectives but each falls short on specific diagnostic axes. Bongard-LOGO tests rule induction from visual exemplars (corresponding to Mind's Eye's Abstraction tasks), but the paper notes (Table 1) that it lacks parametric difficulty control, diagnostic distractors keyed to specific confounds, and formal psychometric taxonomy. ARC-AGI emphasizes abstraction and analogy over stepwise geometric simulation, leaving the Transformation dimension underexplored. Mind the Gap tests spatial completion but focuses on filling gaps in partially occluded 3D shapes rather than simulating transformations. VisFactor digitizes classic psychometric tests (like the FRCT) but lacks the distractor-based diagnostics and parametric scalability of Mind's Eye. IQBench covers broader IQ-style reasoning including RPMs and analogies but, as per Table 1, lacks the formal psychometric taxonomy, distractor keying, and parametric control that would enable systematic diagnosis.

The paper's comparative analysis in Table 1 reveals that no existing benchmark simultaneously satisfies all six diagnostic criteria: formal psychometric taxonomy, derivation from established psychometric assessments, distractors keyed to specific confounds, freedom from knowledge/linguistic confounds, parametric control, and programmatic scalability. Mind's Eye is explicitly positioned as the first benchmark to satisfy all six, and this positioning is central to the paper's contribution—it is not merely another cognitive reasoning benchmark, but a deliberately constructed diagnostic instrument with properties that none of its predecessors possess.

Conflicting Signals in the Literature

The paper is motivated not only by gaps in evaluation infrastructure but also by genuinely conflicting evidence about whether MLLMs possess spatial reasoning capabilities. On one hand, models demonstrate impressive performance on tasks that appear to require spatial understanding—describing 3D arrangements, comparing object orientations, answering questions about spatial relationships in photographs. These successes create an impression of spatial competence. On the other hand, more targeted studies reveal systematic failures: models struggle with counterfactual spatial reasoning (e.g., mentally rotating an object and predicting its appearance from a new viewpoint), fail to track objects through transformations, and are brittle to perturbations that disrupt surface-level visual cues while preserving the underlying spatial relationships.

The paper does not cite specific contradictory studies in detail, but the tension is implicit in its framing. The apparent spatial competence observed in natural-image benchmarks may be driven by perceptual pattern matching and linguistic priors rather than genuine spatial reasoning. By eliminating domain knowledge confounds and controlling for linguistic shortcuts through synthetic stimuli and systematic distractor design, Mind's Eye aims to isolate the core visuocognitive operations and determine whether the apparent competence or the documented brittleness better characterizes current MLLMs.

How This Paper Positions Itself

The paper positions Mind's Eye not as an incremental addition to the growing collection of multimodal benchmarks, but as a fundamentally different kind of instrument—one that is deliberately designed for diagnosis rather than ranking. This is evident in several design choices articulated in Sections 3.1 and 3.2:

Cognitive grounding in established theory. Rather than assembling tasks ad hoc or motivated solely by interesting failure cases, the paper derives its task suite from Carroll's Three-Stratum Theory of cognitive abilities (Carroll, 1993), a foundational framework in psychometrics that decomposes general intelligence into broad and narrow factors. The ART taxonomy maps directly onto constructs from this theory: Abstraction corresponds to inductive reasoning (Gf-I), Relation to quantitative and relational reasoning, and Transformation to visualization and spatial relations (Gv). By anchoring tasks in validated psychometric constructs, the paper enables not just performance comparison but construct-level interpretation: if a model fails on Transformation tasks, we can attribute this to limitations in the specific cognitive faculty of mental spatial manipulation rather than to an undifferentiated "reasoning deficit."

Programmatic generation with parametric control. All stimuli in Mind's Eye are synthetically generated as scalable vector graphics with explicit control over structural parameters (e.g., rotation angles, fold counts, hierarchy depth, symmetry type) and nuisance parameters (colors, layouts, textures). This enables systematic difficulty calibration—incrementally increasing cognitive load by manipulating structural complexity while holding surface features constant—and eliminates the risk that models exploit incidental correlations between visual appearance and correct answers that plague natural-image benchmarks. The paper explicitly connects this approach to the psychometric tradition of factorial item design (Embretson, 1983), where item difficulty is controlled through structural manipulation rather than empirical post-hoc binning, and to the recent ARC-AGI benchmark (Chollet et al., 2025) which similarly uses synthetic stimuli to test abstract reasoning.

Diagnostic distractors tied to specific error types. This is one of the paper's most distinctive contributions. Each item includes distractor answer options that are not merely random wrong answers but are carefully constructed to reveal specific reasoning failures. For Transformation tasks, distractors include reflections mistaken for rotations (testing whether the model understands that rotation preserves chirality while reflection does not), incorrect fold parity (testing whether the model correctly tracks which side of the paper faces up after folding), and off-by-angle rotation errors. For Relation tasks, distractors swap corresponding elements across configurations or preserve surface similarity while violating structural correspondence. For Abstraction tasks, distractors match superficial features like shape or color while violating the latent abstract rule. This design means that even when a model answers incorrectly, the pattern of which wrong answer it chooses reveals what kind of reasoning error it made—information that is lost when only aggregate accuracy is reported.

Elimination of knowledge confounds. By using abstract geometric shapes, synthetic patterns, and meaningless configurations rather than real-world objects, the paper eliminates the possibility that models solve tasks by retrieving domain knowledge (e.g., "cubes look like this when unfolded") or exploiting linguistic priors. The tasks are designed to be solvable purely from the visual input through cognitive operations—a design principle the paper calls "cognitive isolation." This is critical for attributing performance to visuocognitive reasoning rather than to the breadth of the model's training data.

Connection to fluid intelligence research. The paper explicitly frames its benchmark within the fluid intelligence (Gf) literature, drawing on Carroll's Three-Stratum Theory and the Cattell-Horn-Carroll model of cognitive abilities. Fluid intelligence is defined as "the capacity to solve novel problems through deliberate, knowledge-independent thought" (Carroll, 1993; Schneider and McGrew, 2018), and the ART taxonomy decomposes it into specific visual reasoning faculties. By connecting MLLM evaluation to this well-established psychometric framework, the paper makes a theoretical contribution beyond benchmark construction: it provides a vocabulary and a set of constructs for talking about what kinds of visual intelligence models possess or lack, rather than relying on vague notions of "understanding" or "reasoning."

The Gap This Paper Fills, Precisely Stated

Synthesizing all of the above, the gap the paper addresses can be stated as: we lack a diagnostic instrument that can determine whether MLLMs possess specific visuocognitive operations—mental rotation, folding/unfolding, hierarchical pattern induction, analogical relation mapping—as distinct from surface perception, linguistic-prior exploitation, and domain knowledge retrieval. Existing benchmarks either conflate these capabilities (general-purpose suites), fail to isolate transformation as a distinct cognitive operation (compositional reasoning benchmarks), or lack the parametric control and diagnostic distractor infrastructure needed for fine-grained failure analysis (cognitive reasoning benchmarks). Mind's Eye fills this gap by providing a construct-grounded, programmatically generated, diagnostically annotated benchmark that isolates visuospatial reasoning from other sources of performance variance and enables attribution of failures to specific cognitive limitations.

The paper's contribution is thus simultaneously methodological (how to build a diagnostic cognitive benchmark for MLLMs), empirical (what the current performance landscape looks like across 18 models and 8 tasks), and theoretical (mapping MLLM evaluation onto established psychometric constructs of fluid intelligence). The motivation is not merely to produce another leaderboard but to create an instrument that advances our understanding of what kind of visual intelligence current models possess and where the gaps are that future architectures must address.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily a benchmark construction and evaluation paper rather than a systems paper proposing a new model or training method. The "system" being described is Mind's Eye itself: a diagnostic instrument for probing the visuocognitive reasoning capabilities of multimodal LLMs through programmatically generated visual puzzles grounded in classical psychometric tests. The core problem the benchmark solves is attribution of model failures to specific cognitive limitations: rather than reporting a single aggregate accuracy score (which conflates perceptual errors, reasoning errors, and knowledge gaps), Mind's Eye uses parametric control over task difficulty and diagnostically constructed distractor answer options to reveal which specific cognitive operations a model can and cannot perform. The "shape" of the solution is a multiple-choice test battery of 800 items across 8 tasks, organized under a taxonomy that decomposes fluid visual intelligence into three complementary faculties, with every design decision—from stimulus generation to distractor construction to answer scoring—serving the goal of isolating visuocognitive reasoning from other sources of performance variance.

3.2 Big-Picture Architecture (Diagram in Words)

The Mind's Eye benchmark has five major components that together form a diagnostic pipeline:

ART Taxonomy (Abstraction–Relation–Transformation) — the theoretical scaffold derived from Carroll's Three-Stratum Theory of fluid intelligence. It defines what cognitive faculties are being measured and provides the construct vocabulary for interpreting results. Each of the eight tasks is mapped to one or more ART dimensions via a Q-matrix (Table 12 in Appendix), specifying which latent cognitive skills each item requires.
Programmatic Stimulus Generators — a suite of configurable SVG rendering pipelines, one per task. Each generator accepts structural parameters (e.g., rotation angle, fold count, hierarchy depth, symmetry type) that control task difficulty independently from nuisance parameters (colors, layouts, textures) that should be reasoning-irrelevant. The generators produce image–question–answer triples at scale with no human annotation cost.
Diagnostic Distractor Templates — per-task rule sets for constructing wrong-answer options that are systematically wrong in specific, interpretable ways. For Transformation tasks, distractors include mirror reflections (testing chirality understanding), incorrect fold parity (testing side-tracking through folds), and off-by-angle rotation errors. For Relation tasks, distractors swap corresponding elements or preserve surface similarity while violating structural correspondence. For Abstraction tasks, distractors match superficial features while violating the latent rule. This infrastructure converts binary correct/incorrect signals into a richer error taxonomy.
Evaluation Protocol — a three-stage pipeline for scoring model outputs: (1) the candidate MLLM receives an image and a standardized text prompt; (2) a judge LLM (Gemma-3) parses the free-form output into a concise response option; (3) the extracted option is mapped to task-specific labels for accuracy computation against ground truth. This protocol ensures fair comparison across models with different output formats while maintaining fully automated, reproducible scoring.
Human Baseline System — a controlled data collection protocol with 30 non-expert participants, each completing 40 items (5 per task) sampled via inverse-frequency weighting, preceded by an 8-item calibration phase. Human responses serve dual purposes: establishing an upper-bound performance reference and calibrating item difficulty empirically through consensus-based classification (Easy: all 5 annotators correct; Hard: 0–1 annotators correct; Medium: 2–3 annotators correct).

Information flows through these components as follows: the ART taxonomy defines the cognitive constructs → the task suite operationalizes them into specific visual puzzle formats → the stimulus generators produce 800 diagnostic items with explicit difficulty parameters and distractor annotations → the evaluation protocol administers these items to MLLMs and humans using standardized prompts → the scoring pipeline extracts answers and computes task-level and dimension-level accuracies → diagnostic analyses (attention alignment, distractor selection patterns, difficulty sensitivity, prompt variation effects) attribute failures to specific cognitive limitations.

3.3 Roadmap for the Deep Dive

First, the ART taxonomy and its psychometric foundations, because everything else in the benchmark—task selection, difficulty calibration, distractor design, and interpretation of results—depends on clearly defining what cognitive constructs are being measured and how they relate to established theories of intelligence.
Second, the programmatic stimulus generation framework, which is the engine that produces the benchmark items. This includes the parametric control mechanisms, the separation of structural and nuisance parameters, and the factorial design principles that ensure cognitive isolation and enable systematic difficulty manipulation.
Third, the diagnostic distractor design, which is the paper's most novel methodological contribution. Understanding how distractors are constructed for each ART dimension is essential for interpreting the error analyses that form the paper's main empirical contribution.
Fourth, the eight-task suite in detail, mapping each task to its ART dimensions, its controlled parameters, its distractor templates, and its difficulty calibration strategy. This is the operational core of the benchmark.
Fifth, the evaluation protocol—how models are prompted, how answers are extracted, and how accuracy is computed—since this is the measurement apparatus that produces the main results table.
Sixth, the human evaluation protocol and difficulty calibration method, which provides both the baseline for comparison and the empirical grounding for the paper's central claim that models fail uniformly across difficulty levels.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a benchmark construction and diagnostic evaluation paper. Its core idea is that by grounding visual reasoning tasks in established psychometric constructs (Carroll's fluid intelligence framework), programmatically generating stimuli with explicit parametric control, and embedding diagnostic distractors that reveal specific error types, one can attribute MLLM failures to specific cognitive limitations rather than to undifferentiated "reasoning deficits." The benchmark is not merely a collection of hard problems—it is a measurement instrument designed with the same rigor as human cognitive assessments.

The ART Taxonomy: Theoretical Foundations and Construct Mapping

The ART taxonomy decomposes fluid visual intelligence into three complementary dimensions—Abstraction, Relation, and Transformation—each corresponding to a distinct cognitive operation that operates over visual representations. The taxonomy's theoretical grounding matters because it determines what claims the benchmark can and cannot support.

Psychometric ancestry: Carroll's Three-Stratum Theory. The paper explicitly anchors its taxonomy in Carroll's Three-Stratum Theory of cognitive abilities (Carroll, 1993), one of the most influential and empirically validated frameworks in psychometrics. Stratum III contains a single general intelligence factor (g). Stratum II contains broad ability factors, including Fluid Intelligence (Gf)—"the capacity to solve novel problems through deliberate, knowledge-independent thought"—and Broad Visualization (Gv)—the ability to generate, manipulate, and reason about visual patterns. Stratum I contains narrow abilities that are more specific operationalizations of the broad factors, such as Induction (I), General Sequential Reasoning (RG), Spatial Relations (SR), Visualization (VZ), and Flexibility of Closure (CF).

The ART taxonomy operates primarily at the Stratum I level, mapping narrow psychometric abilities onto benchmark tasks:

Abstraction maps to Inductive Reasoning (Gf-I, the ability to discover underlying rules or patterns from specific instances) and Visualization (Gv-VZ, the ability to apprehend spatial forms and manipulate them). The paper describes this dimension as "requiring inducing latent structure from surface variation"—the solver must identify an underlying rule, pattern, or category that unifies disparate visual instances.
Relation maps to General Sequential Reasoning (Gf-RG, the ability to reason from premises to conclusions) and Spatial Relations (Gv-SR, the ability to perceive and manipulate spatial configurations). The paper describes this as "requiring mapping correspondences across visual structures"—the solver must detect how elements in one configuration align with elements in another.
Transformation maps to Visualization (Gv-VZ), Spatial Relations (Gv-SR), and Flexibility of Closure (Gv-CF, the ability to identify a visual figure embedded in a complex pattern). The paper describes this as "requiring mentally simulating spatial operations"—the solver must internally rotate, fold, compose, or otherwise manipulate visual representations to predict outcomes.

This mapping is operationalized formally through a Q-matrix (Table 12 in Appendix), a standard psychometric tool that specifies which latent skills each item requires. The Q-matrix provides an explicit hypothesis about the cognitive demands of each task: Visual Relation Abstraction is coded purely under Abstraction, Hierarchical Pattern Equivalence loads on both Abstraction and Relation (since it requires inducing recursive structure AND recognizing structural equivalence), Dynamic Structural Correspondence and Symmetric Structures are placed at the Relation–Transformation intersection, and Mental Transformation, Paper Folding, and Mental Composition are coded as pure Transformation tasks.

Why this theoretical grounding matters. Without the ART taxonomy and its psychometric foundation, Mind's Eye would be just another collection of hard visual puzzles. The taxonomy provides three things that are essential for the paper's diagnostic claims:

Construct-level attribution: If a model fails on all three Transformation tasks but succeeds on Relation tasks, we can attribute this not to a vague "reasoning deficit" but to a specific limitation in the cognitive faculty of mental spatial manipulation. The Q-matrix makes these attributions testable and falsifiable.
Coverage guarantees: By mapping tasks to a well-established factor structure, the taxonomy ensures that the benchmark covers the major components of fluid visual intelligence without over-emphasizing any single narrow ability. This prevents the benchmark from being gamed by optimizing for tasks that are all essentially the same kind of reasoning.
Connection to human literature: Because the taxonomy maps to constructs that have been studied in human cognitive psychology for decades, the human baseline results (80% accuracy) can be interpreted in the context of known human performance patterns on these construct families, and the model-human gap can be characterized in terms of which specific faculties models lack.

Relation to prior taxonomies. The paper distinguishes its taxonomy from prior efforts in Table 1. Benchmarks like RAVEN test abstraction and relation but not transformation—their Progressive Matrices require rule induction and pattern completion but never ask the solver to mentally rotate or fold a shape. Benchmarks like VisFactor test basic perceptual factors by digitizing standard psychometric tests, but they lack a formal taxonomy that organizes tasks into dimensions with explicit construct-coverage mappings. Mind's Eye's taxonomy is "formal" in the psychometric sense because it provides a Q-matrix that specifies which latent skills each task requires, enabling multi-trait analysis rather than simple aggregate scoring.

Programmatic Stimulus Generation: Parametric Control and Cognitive Isolation

All 800 items in Mind's Eye (and the extended 20,000-item training set) are generated programmatically as scalable vector graphics (SVGs) and exported to PNG format (1024×1024 pixels at 300 DPI). This generation framework is not merely a convenience for scaling—it is the mechanism that enables the benchmark's three core design principles: cognitive isolation, diagnostic precision, and psychometric rigor.

Why synthetic, abstract stimuli? The paper explicitly justifies this choice on several grounds, pushing back against the intuition that "real photographs are more ecologically valid":

Elimination of knowledge confounds: Abstract geometric shapes and pattern-like configurations have no semantic associations that a model could retrieve from pretraining data. A model cannot solve a mental rotation task by recognizing that "this object is a chair and chairs look like this when rotated" because the stimuli are meaningless polycube assemblies. The task must be solved through the cognitive operation of mental rotation itself. As the paper states, this ensures "performance reflects visuocognitive capacity rather than domain familiarity."
Parametric control over difficulty: The synthetic generation framework exposes explicit control knobs—rotation angle magnitude, number of folds, hierarchy depth, symmetry order—that can be independently varied to manipulate cognitive load while holding nuisance variables constant. This is impossible with natural images, where difficulty is entangled with content. The paper can produce an "easy" mental rotation item (single-axis rotation by 45 degrees on a simple polycube) and a "hard" one (multi-axis rotation by 135 degrees on a complex assembly) using the same template, with difficulty controlled purely by structural parameters.
Precedent in cognitive psychology: The paper cites a substantial body of work establishing that performance on synthetic reasoning tasks correlates with general visual cognition and real-world spatial capabilities. References include the Kit of Factor-Referenced Cognitive Tests (Ekstrom et al., 1976a), the Vandenberg & Kuse Mental Rotations Test (Vandenberg and Kuse, 1978c), Thurstone's Primary Mental Abilities tests (Thurstone, 1938), and the Guilford-Zimmerman Aptitude Survey (Guilford and Zimmerman, 1948). These are not obscure instruments—they are the gold-standard assessments that defined the psychometric study of spatial ability—and they all use synthetic stimuli. The paper also draws a parallel to ARC-AGI (Chollet et al., 2025), which the community has embraced as a benchmark for abstract reasoning using synthetic grid-based tasks.
Mitigation of shortcut learning: Because the stimuli are generated with orthogonal variation of structural and nuisance parameters, models cannot exploit incidental correlations between surface features (color, texture, layout) and correct answers. The paper's factorial design ensures that, for example, the color of a polycube is statistically independent of whether it is the correct rotation, so a model that learns to associate "red cubes" with "correct answer" will fail. This is the same principle that motivates adversarial dataset construction in NLP (e.g., Nie et al., 2020b for NLI; Zellers et al., 2019 for commonsense reasoning).

The factorial design principle. The paper explicitly describes the generation process as following a "factorial design" (Section 3.2): "structural parameters that determine task difficulty (e.g., rotation magnitude, fold count, hierarchy depth) are varied independently of nuisance parameters (e.g., color, spatial layout, surface texture) that should be task-irrelevant." This design serves two functions stated in the paper:

A priori difficulty calibration: Difficulty can be predicted from structural complexity before any empirical testing, following the psychometric principle of construct representation where "structural manipulations systematically affect item difficulty" (Embretson, 1983). For example, a mental rotation item with rotation across two axes is predicted to be harder than one with rotation across a single axis, independent of the specific shapes or colors used.
Shortcut prevention: Because nuisance parameters are orthogonal to correctness, any model that achieves high accuracy must be performing the target cognitive operation rather than exploiting surface-level correlations.

The generation pipeline (per task). Each of the eight tasks has its own configurable generator, but they share a common architecture. The process for a representative task (Mental Transformation) works as follows:

Shape sampling: A polycube assembly is randomly generated or selected from a library. The shape is defined as a set of connected unit cubes in 3D space.
Rotation application: A rotation is applied to the original shape, specified by rotation axes (single-axis for easy, multi-axis for hard) and angular magnitude (sampled from a range). The rotation defines the "correct answer" candidate—the shape after transformation.
Distractor generation: Three distractor candidates are produced using systematic perturbations: one mirror reflection (reversing chirality while preserving overall structure), one with a different rotation angle (off-by-θ), and one with a different axis combination.
Coloring: Colors are assigned to individual cubes. The paper specifies options including "monochrome vs. varied" as a controlled parameter—monochrome items are easier because color cannot be used as a matching cue.
Rendering: The original shape is rendered in the top row of a 2-row layout, and the four candidates (one correct, three distractors) are rendered in the bottom row, labeled A–D with randomized position.
Metadata recording: The generator records the correct option label, the rotation angles, the difficulty level, and the distractor types for each foil.

The full specification of parameters for each task is documented in Table 11 (Appendix). This table serves as the technical blueprint for reproducing the benchmark—it lists the controllable parameters, the image layout, and the annotation format for every task. For example, Paper Folding controls include "paper polygon size, fold sequence (V/H, diagonal), number and placement of punched holes"; the image layout is "top row: folding sequence; bottom row: four unfolded candidates (A–D)"; and the annotation records "correct option label; fold sequence metadata."

Scale and reproducibility. The paper provides two dataset variants: a diagnostic set of 800 items (100 per task) for evaluation, and an extended set of 20,000 items (2,500 per task) for training or fine-tuning. Both are generated from identical templates with the same difficulty controls, and the paper explicitly states they "do not have data overlap"—they are maintained separately to ensure that training on the extended set does not inflate diagnostic set performance. Because generation is fully programmatic, the benchmark can scale without additional annotation cost. The generators, seeds, and scoring code are released publicly to enable transparent replication.

Resolution and rendering details. The paper renders all images at 1024×1024 pixels at 300 DPI with "fixed stroke widths and sans-serif labels" and a "uniform background." An ablation study (Table 3, Appendix) tested resolution between 100 DPI (600×800 px) and 300 DPI (1024×1024 px) on Qwen-2.5-VL-7B and found "no statistically significant difference," confirming that image quality is not a bottleneck and that option labels and geometric details are "fully legible at inference time."

Diagnostic Distractor Design: The Error Taxonomy

The diagnostic distractor system is the paper's most distinctive methodological innovation—it converts what would otherwise be a binary correct/incorrect signal into a richer signal about how models fail. The core principle is that each wrong answer option is constructed to be wrong in a specific, interpretable way, corresponding to a known failure mode in human visuospatial reasoning or to a specific hypothesis about MLLM limitations.

The distractor taxonomy by ART dimension. The paper describes the distractor design principles separately for each ART dimension (Section 3.2 and Appendix D):

Transformation tasks (MT, PF, MC):

Mirror reflections mistaken for rotations: In mental rotation, a common human error is confusing a rotation with a reflection—both change the orientation, but rotation preserves chirality (handedness) while reflection reverses it. Including a mirrored version of the original shape as a distractor tests whether the model understands this distinction.
Incorrect fold parity: In paper folding, the solver must track which side of the paper faces "up" after each fold. A distractor that incorrectly reverses parity tests whether the model tracks the facing direction through the fold sequence.
Off-by-θ rotation errors: A distractor that applies the correct rotation axes but with a slightly wrong angle magnitude tests whether the model can precisely represent angular displacement or merely approximates it.
Color-sequence mismatches: For mental transformation with varied cube colors, a distractor that preserves the overall shape but permutes the colors tests whether the model uses color as a matching heuristic rather than reasoning about spatial structure.

Relation tasks (DSC, VCS, SS):

Swapped corresponding elements: In Dynamic Structural Correspondence, where two shapes undergo independent transformations and the solver must track both, a distractor that correctly applies Transformation A to Shape B and Transformation B to Shape A tests whether the model attributes transformations to the correct shapes.
Surface similarity without structural correspondence: A distractor that preserves visual similarity (same shapes, similar positions) but applies the wrong transformation tests whether the model relies on perceptual matching rather than reasoning about transformation sequences.
Preserving symmetry type with different structure: In Symmetric Structures, a distractor that exhibits a different symmetry (e.g., horizontal reflection instead of vertical) while being otherwise visually similar tests whether the model confuses different symmetry types.

Abstraction tasks (VRA, HPE):

Superficial feature match with rule violation: In Visual Relation Abstraction, where the task is to identify which figure violates a shared abstract rule, a distractor that matches the rule-abiding figures in superficial features (shape, color, count) but also violates the rule tests whether the model attends to surface similarity over abstract structure.
Motif substitution: A distractor that replaces an element of the pattern while preserving the overall layout tests whether the model has learned the global pattern or is matching local templates.

Validation of distractor effects. The paper performs a formal statistical analysis of whether models are disproportionately attracted to the diagnostic distractors (Appendix B.8). The null hypothesis is that when a model answers incorrectly, its choice is uniformly random among the available wrong options. The alternative hypothesis is that models exhibit a statistically significant bias toward the designated distractor.

The analysis uses three complementary methods:

Pooled χ² goodness-of-fit test: For Mental Transformation, χ² = 0.59, p = 0.44. For Paper Folding, χ² = 0.42, p = 0.52. Both fail to reject the null hypothesis at α = 0.05.
Per-item exact binomial tests with Holm-Bonferroni correction: For Paper Folding, 5 of 50 items achieved uncorrected p < 0.05, but 0 of 50 survived correction. For Mental Transformation, 3 of 25 items achieved uncorrected p < 0.05, but 0 of 25 survived correction.
Mixed-effects logistic regression (GEE) with clustering by question: For Mental Transformation, the intercept estimate was β̂ = 0.156 (SE = 0.266), odds ratio vs. null = 1.17, p = 0.556. For Paper Folding, β̂ = 0.088 (SE = 0.170), odds ratio vs. null = 1.09, p = 0.605. Both intercepts are not significantly different from zero.

The paper concludes: "models are not disproportionately attracted to the annotated distractors. Instead, their errors appear uniformly spread across all incorrect alternatives." This is an important negative result—it means that when models fail, they are not failing in the systematic ways that the distractors were designed to capture. They are failing more fundamentally, unable even to approach the problem closely enough to make predicted errors.

Human distractor analysis for comparison. The same χ² test on human incorrect responses reveals a different pattern: for Mental Transformation, χ² = 3.21, p = 0.073 (a trend toward distractor concentration that does not reach significance); for Paper Folding, χ² = 6.12, p = 0.014 (significant distractor bias). This suggests that when humans err, they tend to make the predicted error types (confusing reflections for rotations, getting fold parity wrong), confirming that the distractors are psychologically valid. The contrast with models—which show no such bias—further supports the paper's claim that models are not merely "worse" at these tasks but are engaging with them through fundamentally different (and inadequate) mechanisms.

The Eight-Task Suite: Operationalizing the ART Dimensions

Each of the eight tasks is a specific operationalization of one or more ART dimensions, with its own generation logic, parameter space, and distractor templates. The paper provides detailed specifications in Section 3.2, Table 11, and Appendix C.

Visual Relation Abstraction (VRA) — Abstraction dimension. This task adapts the Bongard problem paradigm: given a 2×3 grid of six abstract figures, the solver must identify which one figure violates a shared attribute (e.g., convexity, line crossing, polygonal regularity) that unifies the other five. The paper uses "curated shape attributes (e.g., convexity, line crossings, polygonal regularity)" as the rule space. The generation process draws five figures conforming to a chosen attribute and one figure that violates it, randomizes the grid positions, and records the violating label along with a "decomposed textual reason string (e.g., 'others are convex closed shapes; this one is not')."

The distractor design for VRA exploits superficial feature matching: distractor options (in the larger Bongard-style grid, all panels except the correct answer are rule-conforming, so distractor analysis focuses on which rule-conforming panel the model might incorrectly select) are designed so that some conforming panels share surface features with the violating panel, testing whether the model is using abstraction or surface similarity.

Hierarchical Pattern Equivalence (HPE) — Abstraction and Relation dimensions. This task tests recursive structure recognition. Each item shows a 2×2 grid of four drawings: three are valid instantiations of a hierarchical pattern (nested circles, concentric hexagons, fractal trees, L-systems, Sierpinski gaskets, Pythagoras trees), and one consists of "random disconnected strokes" that violate the hierarchical structure. The generation process uses a library of recursive pattern generators, each parameterized by depth, branching factor, and transformation rules. A random seed per panel ensures varied but reproducible instantiations.

The key parameter is the type of hierarchical generator sampled, which determines the nature of the pattern the model must recognize. Difficulty is controlled by hierarchy depth and the complexity of the recursive rule. The annotation records "which hierarchical generator is sampled and the seed for randomness."

Mental Transformation (MT) — Transformation dimension. This is a mental rotation task adapted from the Vandenberg & Kuse Mental Rotations Test and extended from the work of Stogiannidis et al. (2025b). Each item shows a 3D polycube assembly (the "Original Shape") in the top row and four candidate rotations in the bottom row (A–D). Controlled factors are: (i) shape identity (which polycube assembly is used), (ii) difficulty level—single-axis rotation for easy items vs. multi-axis rotation for hard items, and (iii) cube coloring—monochrome for easier items vs. varied colors for harder items (where color cannot be used as a matching heuristic and may even be a distractor).

The correct answer is the candidate that matches the rotation of the original. The metadata records applied rotation angles and difficulty level. Distractors include mirror reflections (chirality reversal), off-by-angle rotations, and color-permuted versions.

The paper notes that the Mental Transformation items are "generated from polycube assemblies" and that the task builds on the mental rotation subtask from Mind the Gap (Stogiannidis et al., 2025b) but "extends it along an additional reasoning dimension to evaluate the model's capacity for multistep reasoning."

Paper Folding (PF) — Transformation dimension. Adapted from the CogAT Paper Folding test (Publishing, 2009), this task simulates folding and hole-punching on polygonal sheets. The generation process works as follows:

A sheet (square or hexagon, "represented as a polygon with vertices") is initialized.
A sequence of two folds is sampled, either from vertical/horizontal reflections or from diagonal reflections.
After folding, a single hole is punched at a random valid coordinate inside the polygon.
The algorithm "recursively unfolds the sheet and computes the mirrored hole positions."
The final composite image shows: (i) the initial unfolded sheet, (ii) two intermediate folds, (iii) the final folded sheet with the hole, and (iv) four candidate unfolded sheets (A–D), one correct and three foils "generated by removing, mirroring, or randomizing holes."

The task is to infer the unfolded hole pattern—the solver must mentally reverse the folding process and determine where the hole would appear on the original unfolded sheet. Controlled parameters include polygon shape, fold group (V/H or diagonal), fold sequence, and hole placement. The annotation records the fold group and correct label.

Mental Composition (MC) — Transformation dimension. This task tests net-to-solid reasoning: a 2D net (the unfolded surface of a 3D solid) is rendered alongside four 3D candidate solids, and the solver must identify which solid the net folds into. Nets are chosen from a mapping that includes "cube, cuboid, prism, pyramid, cone" for easy items and "octahedron, dodecahedron, icosahedron" for hard items. For easy items, nets use simple solids with uniform coloring; for hard items, the paper uses "complex polyhedra with confounding colorings."

The layout places the net in the top-left of a 2×4 grid and renders candidate solids in the bottom row with distinct colors. The correct candidate is the folded realization of the net. Distractors include solids with "similar number of faces" (a surface-level heuristic that might fool a model that counts faces but cannot simulate folding) and solids with mismatched color assignments. Annotations store net identity, correct solid, distractors, color assignments, and difficulty.

Dynamic Structural Correspondence (DSC) — Relation dimension. This task evaluates extrapolation of geometric motion. Two shapes (triangle, square, pentagon, hexagon, or diamond) undergo distinct continuous transformations drawn from a library that includes "rotate-back-and-forth, bounce, wiggle, pulsate, swirl, shear, compress-and-stretch."

The top row shows the two shapes' trajectories at time points t ∈ {0.0, 0.25, 0.5, 0.75}. The bottom row contains four candidate continuations for t = 1.0: one is the true continuation (applying the correct transformation pair to the correct shapes), and three are distractors that use mismatched functions (e.g., applying the wrong transformation to a shape) or perturbed times. Parameters control shape identities, the transformation pair, and time discretization. Annotations specify the correct continuation and provide a textual explanation of which transformation was applied to each shape.

Visual Conceptual Slippage (VCS) — Abstraction and Relation dimensions. Adapted from classical "odd-one-out" paradigms, this task shows six panels arranged in a 2×3 grid. Five panels conform to a chosen abstract visual concept (e.g., equidistant spacing, global symmetry, enclosure of one shape by another), and exactly one panel violates the concept.

The paper uses concept types including "spacing, alignment, number, enclosure, symmetry, topology, border, hollowness, and word symmetry." For word-symmetry items, a random uppercase string is rendered and mirrored to induce or break bilateral symmetry. Controlled parameters include concept type, variation seeds, and (for word-symmetry) word length. The metadata records the violating option, the concept type, and, in word trials, the sampled word.

Symmetric Structures (SS) — Relation and Transformation dimensions. This task probes symmetry detection in line drawings. Each item shows four connected-path drawings in a 1×4 grid (A–D): three exhibit a chosen symmetry type (vertical reflection, horizontal reflection, or rotational symmetry of order k ∈ {2, 4}), and one lacks it.

The generation process creates random line paths by "chaining ten short segments with random turns." Symmetry is imposed by reflection (vertical/horizontal) or rotation. The annotation records the asymmetric panel as the correct answer. Distractors test whether models confuse different symmetry types or fail to distinguish symmetric from asymmetric configurations.

Q-matrix specification (Table 12). The paper formalizes the task–construct mapping through a Q-matrix that associates each task with a binary vector of latent skill requirements. Visual Relation Abstraction loads purely on Abstraction. Hierarchical Pattern Equivalence loads on both Abstraction and Relation. Dynamic Structural Correspondence and Symmetric Structures load on Relation and Transformation. Visual Conceptual Slippage loads on Abstraction and Relation. Mental Transformation, Paper Folding, and Mental Composition load purely on Transformation. This matrix is the psychometric scaffold that enables multi-trait analysis—rather than reporting a single score per task, results can be interpreted in terms of which cognitive faculties are succeeding or failing.

Evaluation Protocol: Standardized Prompting and Automated Scoring

The evaluation protocol ensures fair comparison across 18 MLLMs with different architectures, output formats, and API interfaces. It consists of three stages described in Section 4 and Appendix E.

Stage 1: Standardized input presentation. Each model receives an identical image (1024×1024 PNG at 300 DPI) and a standardized textual prompt. The prompt templates are fixed per task to minimize variation across models. Examples of the prompt structure are provided in Figure 17 (Appendix): a question image paired with a textual instruction like "Which option matches the rotated version of the original shape?" followed by the answer options.

Stage 2: Answer extraction via judge LLM. Because modern MLLMs produce extended free-form outputs that include reasoning chains, hedging language, and formatting variations, rule-based answer extraction (e.g., regex matching for "Answer: A") is brittle. The paper instead employs Gemma-3 (Team et al., 2025) as a judge LLM to parse free-form responses into concise option identifiers. The judge prompt (Figure 17, top panel) instructs Gemma-3 to extract the selected option from the candidate model's response, handling cases where the model provides reasoning before stating its answer, uses varying formatting, or expresses uncertainty.

This approach follows recent practice in multimodal evaluation (Lu et al., 2024b; Zhang et al., 2024) and "leverages robust semantic extraction via a large model, while maintaining fully automated, reproducible scoring." The choice of Gemma-3 as the judge model is motivated by its availability as an open-weight model, ensuring that the evaluation pipeline is reproducible without proprietary API dependencies.

Stage 3: Label standardization and accuracy computation. The extracted response is mapped to task-specific discrete labels (multiple-choice option identifiers A–F) and compared against the ground truth key. Accuracy is computed per task, per ART dimension, and overall. To prevent positional bias, correct answer options are randomly rotated across positions following standard MCQ evaluation practice.

Prompting strategy variations. The paper evaluates four prompting paradigms in addition to a standard Chain-of-Thought (CoT) baseline (Section 4, Appendix E, and Appendix G–H):

Chain-of-Thought (CoT): The model is instructed to "think step by step" using explicit reasoning anchors: "Step 1: Identify the primitive shapes. Step 2: Detect how they move or fold. Step 3: Eliminate mismatched answers." This is the baseline against which other prompting strategies are compared.
Meta-Task Framing: Before presenting the question, the prompt explicitly describes the cognitive operation required. Example templates from Appendix H (Figure 20) include: "This is a mental transformation test. You need to imagine folding or rotating the shape in 3D" and "This is a paper folding puzzle. At the end, identify which option shows the holes in the unfolded paper."
Step-by-Step Instruction (SBS): Models are guided to decompose problems structurally: "First, describe the shapes. Then, identify the transformation (rotation, reflection, folding, symmetry). Finally, choose the answer."
Hint-Based Prompting: Task-specific concept tags are prepended to the question, e.g., "[Task: Mental Transformation] Which option matches the rotated version of the shape?"

The paper also evaluates "Elimination-based" prompting (Appendix G, Table 19, and Figure 18 in Appendix H), where models are instructed to explicitly eliminate implausible options before selecting an answer.

Hardware and inference settings. The paper specifies that experiments were conducted on a server with four NVIDIA RTX A6000 GPUs (48 GB each). For closed-source models, API configurations are provided: GPT-o3 used the o3-2025-04-16 model with reasoning={"effort": "medium"} and max_output_tokens=500.

Resolution ablation. To verify that visual quality does not confound results, the paper compares Qwen-2.5-VL-7B performance at 100 DPI (600×800 px) vs. 300 DPI (1024×1024 px) across all eight tasks (Table 3). No statistically significant difference is observed. For example, Mental Transformation accuracy: 25.2 ± 0.30 at 100 DPI vs. 25.7 ± 0.02 at 300 DPI. This confirms that the benchmark's difficulty is reasoning-driven rather than perception-driven.

Human Evaluation Protocol and Difficulty Calibration

The human baseline serves dual purposes: providing an upper-bound performance reference and enabling empirical item difficulty calibration.

Participant recruitment and demographics (Table 15). N = 30 participants were recruited through university mailing lists and professional networks. Demographics: age range 20–40 (mean 25.3), gender distribution 17 male / 13 female. None reported prior expertise with the specific tasks used in the benchmark. All provided informed consent.

Evaluation procedure. Each participant completed 40 items total: 5 questions from each of the 8 tasks, sampled via "inverse-frequency weighting from a pool of 20 questions per task." This sampling strategy ensures balanced representation across difficulty levels while keeping the total test duration manageable. Tasks were presented in randomized order to control for ordering effects, and items within each task were also randomized. All participants first completed an "identical calibration phase consisting of 8 examples spanning all tasks" to minimize learning effects and ensure consistent understanding of task formats. The total testing time was approximately 60 minutes. Responses were collected digitally through a "custom interface that mirrors the image-based multiple choice format used for multimodal language models."

Consensus-based difficulty calibration. The paper introduces an operational difficulty classification grounded in inter-annotator agreement principles. Each item was evaluated by exactly 5 randomly sampled participants. Items are classified as:

Easy if all 5 annotators provide the correct response (perfect agreement, κ = 1.0 for that item).
Hard if 0 or 1 annotators respond correctly.
Medium if 2–3 annotators respond correctly (partial agreement, 0.33 ≤ κ ≤ 0.67).

Formally, for item i with human responses {r₁, r₂, r₃, r₄, r₅} ∈ {0, 1}:

$d(i) = \begin{cases} \text{Easy} & \text{if } \sum r_j = 5 \\ \text{Hard} & \text{if } \sum r_j \leq 1 \\ \text{Medium} & \text{otherwise} \end{cases}$

where $r_j$ is the binary correctness of annotator $j$ on item $i$ , and $d(i)$ is the assigned difficulty level.

What it computes: for each item, it tallies the number of human participants who answered correctly out of 5. If all 5 succeeded, the item is labeled Easy. If at most 1 succeeded, it is labeled Hard. Otherwise (2–3 correct), it is labeled Medium.

Why this form: this approach aligns difficulty calibration with the distribution of actual human performance rather than with a priori structural complexity assumptions. It also provides a principled basis for stratified analysis—items in different difficulty bins represent genuinely different levels of cognitive demand as validated by human behavior. The paper states this "ensures that difficulty labels reflect actual human performance distributions and provides a principled basis for stratified analysis of model performance across varying levels of cognitive demand." The distribution is: Easy (32%), Medium (45%), Hard (23%).

Reliability quantification. Fleiss' kappa (a measure of inter-annotator agreement beyond chance for more than two raters) was computed across all items within each task family, yielding "moderate to substantial agreement (κ = 0.71 across tasks)." This confirms that the difficulty categories capture "stable individual differences rather than measurement noise."

Human accuracy results (Table 2). Human accuracy per task: VRA 68.0%, HPE 88.0%, DSC 81.2%, VCS 87.0%, SS 78.0%, MT 81.0%, PF 80.1%, MC 82.0%. The aggregate human accuracy across all tasks is 80% (as stated in the abstract and Section 6). The variability across tasks—with VRA (68%) and SS (78%) lower than other tasks—suggests that even for humans, some visuocognitive operations are more demanding than others, which validates the ART taxonomy's decomposition into distinct cognitive faculties.

Summary of Design Choices and Their Justifications

Synthetic over natural stimuli: eliminates knowledge confounds, enables parametric difficulty control, follows precedents in cognitive psychology (Ekstrom, Vandenberg & Kuse, Thurstone), and prevents shortcut learning through orthogonal variation of structural and nuisance parameters.
ART taxonomy over ad hoc task collection: provides construct-level attribution for failures, ensures coverage of major fluid visual intelligence components, and connects results to decades of human cognitive psychology research.
Diagnostic distractors over random wrong answers: enables fine-grained error analysis by revealing how models fail, not just that they fail. The specific error types (reflection vs. rotation confusion, parity errors, off-by-angle errors) are drawn from known human error patterns in psychometric testing.
Consensus-based difficulty calibration over structural complexity alone: anchors difficulty in actual human performance, enabling the critical finding that model performance is flat across difficulty levels while human performance degrades systematically—a pattern that purely structural calibration might not reveal.
Judge LLM for answer extraction over rule-based parsing: accommodates the varied output formats of modern MLLMs (chain-of-thought reasoning, hedging, formatting variations) while maintaining fully automated, reproducible scoring.
Gemma-3 as judge over proprietary APIs: ensures the evaluation pipeline is reproducible without reliance on closed-source services, following the paper's commitment to transparent, replicable benchmarking.
Resolution verification: the ablation study confirming no statistically significant difference between 100 DPI and 300 DPI ensures that the observed performance gaps are attributable to reasoning limitations rather than visual quality artifacts, strengthening the paper's central claim about cognitive rather than perceptual deficits.

4. Key Insights and Innovations

Innovation 1: The Diagnostic Distractor as a Scientific Instrument for Error Attribution

The paper's most intellectually distinctive contribution is not the benchmark's existence but its architecture for attributing model failures to specific cognitive limitations. Prior benchmarks—whether general-purpose suites like MMBench and MMMU, compositional reasoning tests like CLEVR and RAVEN, or even cognitively motivated efforts like Bongard-LOGO and IQBench—share a common evaluative poverty: they report that a model failed (aggregate accuracy) but cannot reveal how it failed. A model scoring 30% on a spatial reasoning test might be confusing reflections for rotations, failing to track fold parity, applying the wrong transformation to the right object, or simply guessing uniformly. These failure modes have entirely different implications for model improvement—the first suggests a fixable perceptual confusion, the second a fundamental inability to simulate unfolding, the third an attention-tracking problem, the fourth a complete absence of relevant computation. But standard benchmarks collapse them all into a single incorrect count.

Mind's Eye introduces a diagnostic distractor taxonomy that systematically constructs wrong answers to be wrong in specific, interpretable ways corresponding to known human error patterns in psychometric testing. The conceptual move is to treat distractor options not as foils to make the task harder but as measurement channels—each distractor "type" (mirror reflection, parity error, off-by-angle rotation, swapped correspondence, surface-feature match) probes a distinct hypothesis about what the model is doing internally. This transforms evaluation from a binary classifier (correct/incorrect) into a multi-hypothesis diagnostic, where the pattern of which wrong answers a model selects constitutes data about which cognitive operations it is attempting.

The psychometric precedent for this is well-established—the Kit of Factor-Referenced Cognitive Tests (Ekstrom et al., 1976a) and the Vandenberg & Kuse Mental Rotations Test (Vandenberg and Kuse, 1978b) used systematically constructed wrong answers to diagnose specific spatial reasoning deficits in humans—but this principle has been almost entirely absent from MLLM evaluation. The paper's comparative analysis in Table 1 makes this explicit: every prior benchmark in the comparison matrix receives an "✗" or "❒" for "Distractors Keyed to Confounds." Mind's Eye is the first to receive a "✓" on this axis, and the paper treats this not as a feature to check off but as the conceptual foundation of the benchmark's contribution.

The statistical analysis in Appendix B.8 demonstrates both the power and the surprise of this approach. The paper finds that when models err on Mental Transformation and Paper Folding, their wrong answers are distributed uniformly across all incorrect options, with no statistically significant bias toward the diagnostic distractors—χ² tests fail to reject the null hypothesis of uniform random selection among wrong options (p = 0.44 for MT, p = 0.52 for PF), and mixed-effects logistic regression confirms no systematic distractor attraction (odds ratios of 1.17 and 1.09, both non-significant). In contrast, when humans err on Paper Folding, they show a statistically significant distractor bias (χ² = 6.12, p = 0.014), meaning humans making mistakes tend to make the predicted mistakes—confusing reflections for rotations, getting fold parity wrong. This asymmetry is itself a finding of intellectual significance: models are not merely worse than humans; they are engaging with the tasks through fundamentally different (and more impoverished) mechanisms. Humans fail in the ways the diagnostic framework predicts, confirming the psychological validity of the distractor design. Models fail in random directions, suggesting they lack even the approximate internal simulation that would produce systematic errors.

This diagnostic architecture represents a fundamental shift in how the field can reason about model failures—from "this model is bad at spatial reasoning" to "this model shows no evidence of performing mental rotation at all, as indicated by uniform error distribution across distractor types that capture known human rotation errors." It is a contribution at the level of evaluation methodology, not benchmark content, and could in principle be applied to other cognitive domains (causal reasoning, physical simulation, mathematical proof verification) where systematic errors have known structure.

Innovation 2: The Difficulty-Invariance Signature as Evidence of Absent Cognitive Operations

The paper's most empirically striking finding—that MLLM accuracy is essentially flat across easy, medium, and hard items while human accuracy degrades systematically from >0.80 to ~0.25—is more than a performance gap. It is a behavioral signature that constitutes evidence about what kind of computation models are performing. The conceptual contribution is recognizing this flatness as diagnostic rather than merely disappointing.

The logic runs as follows. If a model were performing genuine visuospatial operations—mental rotation, fold tracking, hierarchical pattern induction—its performance should be sensitive to the structural parameters that make these operations harder or easier. A single-axis rotation of 30 degrees should be easier than a multi-axis rotation of 135 degrees if the model is actually rotating an internal representation; the computational cost of the operation scales with the transformation complexity. This sensitivity to parametric difficulty is what makes human performance a meaningful validation of task design: humans exhibit the expected graded degradation because they are genuinely performing the target cognitive operations, and those operations have differential costs.

The paper's difficulty calibration (Section 3.2 and Appendix F) makes this signature interpretable by grounding difficulty in human consensus rather than a priori structural assumptions. Items are classified as Easy (all 5 human annotators correct), Medium (2–3 correct), or Hard (0–1 correct) based on empirical response patterns. This means "difficulty" captures actual cognitive demand as validated by human performance distributions. The finding that models show "minimal sensitivity to task difficulty, with performance remaining relatively flat (typically varying by only 0.02–0.08 accuracy points) across Easy, Medium, and Hard conditions" (Section 5, Figure 3) is therefore evidence that models are not performing operations whose cost scales with the same parameters that drive human difficulty. If the models were performing something approximating mental rotation, harder rotations would reduce accuracy. They don't. Therefore, whatever the models are doing, it is not mental rotation in any substantive sense.

This is a diagnostic concept with implications beyond this benchmark. Difficulty-invariance has been used in cognitive psychology to distinguish genuine competence from heuristic matching—if a subject's performance does not degrade as the problem becomes objectively harder, they are likely using a strategy that bypasses the target cognitive operation (e.g., pattern-matching on surface features rather than simulating the transformation). The paper imports this diagnostic logic into MLLM evaluation and demonstrates its power: the flat difficulty curves in Figure 3 and the detailed per-task difficulty breakdowns in Appendix K (Figure 26) reveal uniformly across all eight tasks that models "lack foundational visual-cognitive operations rather than merely struggling with complexity."

The contrast with human performance in Figure 3 is instructive. On Mental Composition, humans drop from 0.94 (Easy) to 0.14 (Hard)—a dramatic sensitivity to the cognitive demands of folding complex polyhedra. Models hover between 0.20 and 0.40 across all difficulty levels. This pattern repeats across all three ART dimensions. The convergence on hard items (where human accuracy drops to model-level performance) is particularly revealing: "humans fail because the task is hard; models fail because they cannot perform the underlying operation at any difficulty level." This is a crisp, falsifiable claim about the nature of model limitations, not merely an observation of lower performance.

This insight represents a fundamental shift from how the field typically interprets model failures. The standard narrative—"models are getting better, they just need more scale/training/data to match human performance"—assumes that failures reflect insufficient capability that can be remedied by incremental improvement. The difficulty-invariance signature suggests instead that models lack the mechanisms for genuine visuospatial reasoning, and that scaling current architectures may never produce those mechanisms regardless of compute invested. This is a stronger and more specific claim about the nature of the gap, with direct implications for research prioritization: improving vision encoders or increasing model size is unlikely to help if the bottleneck is the absence of transformation-aware representations and spatial working memory, not the resolution of perceptual encoding.

Innovation 3: Prompting Dissociation Reveals a Taxonomy of Reasoning Operations

The paper's analysis of how different prompting strategies affect performance across the ART dimensions (Section 5, Figure 2, and Appendix B.4) reveals a dissociation pattern that constitutes a conceptual contribution about the nature of the cognitive operations being tested. The finding is that prompting effects are dimension-dependent rather than uniformly beneficial: Abstraction tasks benefit from structured scaffolding (Meta-task and Step-by-step prompting yield gains of approximately +1.3 points), while Transformation tasks exhibit "consistent performance degradation across all alternative prompting strategies" with Hint prompting showing the largest drop (approximately −0.9 points).

This asymmetry is not merely a practical observation about prompt engineering. It reveals that the ART dimensions tap into qualitatively different kinds of computation that respond differently to linguistic guidance. The paper's interpretation—"prompting facilitates rule derivation yet fails to elicit procedural visuocognitive operations"—makes a substantive claim: Abstraction tasks (inducing latent rules from visual patterns) involve symbolic reasoning processes that can be scaffolded by explicit instructions about what kind of rule to look for. Transformation tasks (mentally rotating or folding shapes) involve procedural, simulation-based processes that cannot be elicited through verbal instruction because they require computational mechanisms—spatial working memory, transformation-aware representations—that current MLLM architectures simply do not possess.

This is a conceptual advance over prior work on prompting in multimodal reasoning, which has largely treated prompting as a generic intervention whose effects are assessed in aggregate. The dominant finding in the chain-of-thought literature (Wei et al., 2022; Kojima et al., 2022) is that prompting can improve reasoning performance. The dominant finding in more recent work on multimodal prompting is that effects are task-dependent and often modest. The paper's contribution is to show that the direction of prompting effects dissociates along theoretically motivated cognitive dimensions: the same prompting strategy that helps on Abstraction tasks (by providing explicit scaffolding for rule search) hurts on Transformation tasks (perhaps by interfering with whatever limited visuospatial computation the model can perform, or by adding verbal reasoning steps that displace attention from the visual input).

The per-task analysis in Appendix B.4 sharpens this dissociation. Within Transformation, Mental Composition exhibits the "largest and most consistent drops across all prompting strategies (−0.8 to −1.4 pts)." Mental Transformation and Paper Folding show smaller but consistent declines. In contrast, Hierarchical Pattern Equivalence—a task requiring explicit symbolic rule induction—is the "most prompt-sensitive task, with meta-task and step-by-step prompting yielding gains of approximately +1.8 pts and +1.7 pts." The task-level heterogeneity reveals that even within a single ART dimension, tasks that lean more heavily on symbolic rule derivation vs. procedural simulation respond differently to the same intervention.

This dissociation has theoretical significance because it provides behavioral evidence that the ART taxonomy captures genuinely distinct cognitive operations, not merely different task formats. If Abstraction and Transformation tasks responded identically to prompting interventions, one could argue that the taxonomy is an arbitrary categorization of what are functionally similar tasks. The asymmetric prompting effects provide convergent validity: operationalizing a cognitive construct (Abstraction vs. Transformation) leads to tasks with different computational demands, as evidenced by their differential sensitivity to linguistic scaffolding. This is a rare example in the MLLM evaluation literature of using behavioral dissociations to validate a theoretically motivated task taxonomy, rather than simply reporting per-task accuracies.

Innovation 4: The Psychometric Grounding Framework for Model Capability Attribution

The paper's decision to anchor its benchmark in Carroll's Three-Stratum Theory of cognitive abilities—and to operationalize this grounding through a formal Q-matrix, construct-level task mapping, and psychometric validation procedures—represents a methodological innovation in how the field constructs and interprets MLLM benchmarks. Prior benchmarks either lack explicit theoretical foundations (tasks are chosen because they seem hard, interesting, or practically important) or invoke cognitive concepts informally (calling a task "spatial reasoning" without specifying which narrow ability from which psychometric tradition it operationalizes). Mind's Eye provides a replicable template for constructing benchmarks that support construct-level attribution: the ability to say not just "the model scores 45%" but "the model shows specific deficits in fluid visual intelligence, particularly in the Transformation dimension (Gv-VZ, mental spatial manipulation), with relative preservation of Abstraction (Gf-I, inductive rule learning)."

The key conceptual moves that distinguish this from prior work:

Explicit Q-matrix specification (Table 12). In psychometrics, a Q-matrix maps test items to latent skills or cognitive attributes. It is the formal mechanism by which a test designer specifies what constructs are being measured. By publishing a Q-matrix that maps each of the eight tasks to its hypothesized ART dimension loadings, the paper makes its construct model transparent and falsifiable. If subsequent factor analysis of model performance reveals that Mental Transformation and Paper Folding do not load on a common factor, the Q-matrix hypothesis would be rejected—and that itself would be an informative finding about whether models process these tasks similarly to humans. This is standard practice in educational and psychological measurement (Embretson and Reise, 2013; De Boeck and Wilson, 2003) but essentially absent from MLLM evaluation, where most benchmarks provide no explicit model of what cognitive demands their tasks impose.

Derivation from validated psychometric instruments. The paper's tasks are not designed from scratch; they are programmatic adaptations of established human cognitive assessments with known psychometric properties. Mental Transformation derives from the Vandenberg & Kuse Mental Rotations Test, Paper Folding from the CogAT Paper Folding test, Bongard-style rule induction from the problem format studied extensively in human concept learning research (Bongard, 1970), and the symmetry and hierarchical pattern tasks from factor-referenced tests of spatial visualization. This means the paper can draw on decades of validation research about what these tasks measure in humans, providing a theoretical bridge between human and model performance. When humans achieve 80% accuracy on these tasks, the interpretation is grounded in known human performance distributions on the source instruments. When models achieve 30–50%, the gap can be characterized in terms of the specific cognitive faculties that humans deploy but models lack.

Consensus-based difficulty calibration as psychometric validation. The paper's use of human inter-annotator agreement to calibrate item difficulty—rather than relying solely on a priori structural parameters—is a psychometric validation procedure. It ensures that the difficulty labels reflect actual cognitive demand as experienced by humans, not merely the experimenters' assumptions about what should be hard. The Fleiss' kappa of 0.71 across tasks quantifies the reliability of these difficulty assignments, providing evidence that the items are measuring stable individual differences rather than noise. This calibration step is what enables the paper's central finding about difficulty-invariant model performance: without empirically validated difficulty levels, flat model performance across difficulty bins could simply reflect poor difficulty calibration (e.g., if the "hard" items were not actually harder). The human consensus validation rules out this interpretation.

Connection to fluid intelligence as a theoretical frame. By explicitly mapping the ART taxonomy onto Carroll's fluid intelligence (Gf) and broad visualization (Gv) factors—and providing a schematic of this mapping in Figure 25 (Appendix J)—the paper positions its benchmark within a well-established theoretical framework for understanding what "intelligence" means in a cognitive context. This matters because it provides a vocabulary for discussing model capabilities that is more precise and falsifiable than vague terms like "reasoning" or "understanding." When the paper claims that models "lack foundational visual-cognitive operations," this is not a casual observation but a claim about the absence of specific Stratum I narrow abilities (Spatial Relations, Visualization, Flexibility of Closure) that are well-defined in the psychometric literature and measurable through validated instruments. The paper is essentially constructing a cognitive assessment battery for MLLMs using the same design principles that guide human intelligence testing, and this framing is a conceptual contribution that extends beyond the specific tasks and results reported.

This psychometric grounding is a fundamental shift in benchmark methodology, not an incremental improvement. It moves evaluation from "leaderboard engineering" (collecting hard problems, reporting who scores highest) toward "cognitive measurement" (specifying what constructs are being measured, validating that the instrument measures them, and interpreting results in construct-level terms). The paper does not fully realize this vision—it reports primarily aggregate accuracies rather than fitting multi-trait IRT models or computing latent ability estimates—but it provides the infrastructure (Q-matrix, consensus-validated difficulty levels, construct-level task mapping) that would make such analyses possible. In this sense, the paper is as much a methodological proposal for how to build cognitive benchmarks as it is a specific benchmark.

5. Experimental Analysis

Evaluation Methodology

Dataset. Mind's Eye comprises 800 diagnostic items: 100 per task, balanced across the eight tasks and three ART dimensions. All stimuli are programmatically generated as SVGs and exported to 1024×1024 PNG at 300 DPI. The paper provides an additional extended set of 20,000 items (2,500 per task) for fine-tuning or representation learning, with no data overlap between diagnostic and extended partitions. Items are drawn from programmatic generators with parametric control over difficulty; empirical difficulty calibration uses human consensus (Section 3.2, Appendix F): Easy (all 5 annotators correct, 32% of items), Medium (2–3 correct, 45%), Hard (0–1 correct, 23%). The paper states that 30 human participants each completed 5 questions per task, sampled via inverse-frequency weighting from a 20-question pool per task, producing the human baselines in Table 2 (Appendix F).
Base model(s). The paper evaluates 18 MLLMs spanning three categories. Closed-source API-based models: GPT-4o, GPT-o3 (accessed via the o3-2025-04-16 endpoint with reasoning={"effort": "medium"}, max_output_tokens=500), and Gemini-2.5 pro. Open-source multimodal LLMs: Idefics-8B, InternVL3-8B, LLaMA-3.2-11B-Vision, LLaVA-1.6-Mistral-7B, Phi3.5-vision-instruct-8B, Qwen-2.5-VL-Instruct (3B, 7B, and 32B), BLIP-2.7B, InstructBLIP-4B, PaliGemma-3B, and Smol-2.2B. Open-source multimodal LRMs (large reasoning models): Vision-G1-7B, GT-Thinker-7B, and V-Thinker-8B. The paper states these models were chosen to span "a wide range of recent MLLMs" across scale, architecture, and access type. The Qwen-2.5-VL family is explicitly used for scaling analysis across 3B, 7B, and 32B variants. No specific justification is given for the exclusion of certain contemporary models (e.g., Claude 3.5, Gemini 1.5 Flash, LLaVA-NeXT), though the coverage appears designed to represent both frontier proprietary systems and a diverse set of open-weight alternatives.
Metrics. The primary metric is accuracy (%) — the fraction of items for which the model's selected final answer matches the ground truth, computed per task and aggregated across tasks within each ART dimension. Answers are extracted via a three-stage pipeline: (1) the candidate MLLM receives the image and a standardized text prompt; (2) a judge LLM (Gemma-3) parses the free-form output into a concise option identifier; (3) the parsed response is mapped to standardized task-specific labels (A–F) for comparison against ground truth. The paper justifies this approach by noting that "rule-based answer extraction can be unreliable" for modern MLLMs that produce extended free-form outputs. For the attention analysis (Section 5), three additional metrics are computed: Option-Specific Attention Score (OAS_correct), defined as the mean normalized attention mass directed toward the correct option's spatial region during reasoning-token generation; Region-Aligned Attention (RAA), the fraction of attention that falls within the union of regions referenced by query-specific tokens; and point-biserial correlation between OAS_correct and binary correctness.
Baselines. The paper reports three categories of comparison points. Random choice: accuracy expected from uniform random selection among answer options, varying by task based on the number of options — 16.0% for VRA and VCS (6 options; however Table 2 shows 16.0 for VRA and 16.0 for VCS — the discrepancy between stated 25% for some tasks and 16% for others is not explicitly explained but likely reflects varying option counts: 4-option tasks have 25% random baseline, 6-option tasks have ~16.7%). Human performance: accuracy from 30 non-expert participants (mean age 25.3, 17 male / 13 female) who each completed 40 items (5 per task), reported per task in Table 2 and aggregated to ~80% overall (stated in abstract and Section 6). Model-to-model comparison: all 18 MLLMs are compared against each other within the same evaluation protocol, with GPT-4o, GPT-o3, and Gemini-2.5 pro serving as the frontier reference points. The paper does not compare against task-specific baselines from prior work (e.g., reported human accuracy on the Vandenberg & Kuse MRT or CogAT paper folding), which would have provided additional calibration. No "always-pick-A" or surface-feature heuristic baselines (e.g., color-matching, shape-counting) are reported, though such baselines could have helped quantify the contribution of perceptual shortcuts to model performance.
Generation budget / compute accounting. The paper does not report inference FLOPs, token counts, or wall-clock time for any model. The unit of comparison is simply accuracy under standardized evaluation conditions. For API-based models, the configuration is specified (GPT-o3 with medium reasoning effort, 500 max output tokens) but no cost or latency analysis is provided. For open-source models, hardware is described (four NVIDIA RTX A6000 GPUs, 48 GB each) but per-item inference time is not reported. This is standard for benchmark evaluation papers but means that efficiency–accuracy tradeoffs cannot be assessed—a model that achieves 40% accuracy by spending 10× more inference compute than one achieving 35% would look better by the paper's metrics even if it is practically less useful. The paper's fairness argument rests on using "identical visual inputs and standardized textual prompts" across all models (Section 4), not on equalizing inference budgets.
Cross-validation / statistical protocol. For the primary accuracy results, the paper reports standard deviations across items for open-source models (Table 2 shows values like 24.0±0.02 for Idefics-8B on VRA), though the methodology for computing these error bars is not explicitly described — they likely represent variance across the 100 items per task. For closed-source models accessed via API, no error bars are reported, presumably because repeated runs were not conducted or because API determinism limits variance. The paper does not describe a train/validation/test split for the diagnostic set (the 800 items appear to serve as a single evaluation set), though the extended 20,000-item set is explicitly separated from the diagnostic partition. For the attention analysis (Section 5, Appendix B.2), statistical testing uses point-biserial correlation, paired t-tests, and mixed-effects logistic regression with clustering by question. The distractor bias analysis (Appendix B.8) uses pooled χ² goodness-of-fit tests, per-item exact binomial tests with Holm-Bonferroni correction, and GEE logistic regression. Two-fold cross-validation is not employed, as the benchmark provides a fixed evaluation set rather than a training signal requiring strategy selection — the paper selects no hyperparameters based on the evaluation data.

Main Quantitative Results

Overall Model Performance vs. Humans (Table 2)

The headline finding is a persistent human–model gap across all tasks and all model categories. Human participants average approximately 80% accuracy overall (reported per task: VRA 68.0%, HPE 88.0%, DSC 81.2%, VCS 87.0%, SS 78.0%, MT 81.0%, PF 80.1%, MC 82.0%). In contrast, the best-performing MLLM — Gemini-2.5 pro — achieves 51.8% on MC, 35.6% on MT, 31.4% on SS, and 29.0% on VRA, remaining below 50% on aggregate (the abstract states "top performing MLLMs remain below 50%"). GPT-4o achieves its highest accuracy on MT at 32.7% and its lowest on SS at 19.1%. GPT-o3 shows considerable variation: 43.1% on MC but only 11.2% on DSC. Among open-source models, Qwen-2.5-VL-32B achieves 49.5% on MC (the highest open-source score on any task) but drops to 18.3% on HPE and 22.6% on DSC, demonstrating substantial task-level heterogeneity even within a single model family.

The paper emphasizes that the gap is not uniform across tasks. Models perform relatively better on tasks with stronger perceptual cues: Mental Composition (MC) sees the highest model scores (Gemini-2.5: 51.8%, Qwen-2.5-VL-32B: 49.5%), which the paper attributes to "surface perception" — "in Mental Composition, models succeed when the unfolded net visually resembles a cube but fail when correct inference requires mentally folding a shape into a nontrivial 3D structure" (Section 4, Figure 22 in Appendix). In contrast, Abstraction tasks like Hierarchical Pattern Equivalence show the most compressed performance range, with even the best models barely exceeding random guessing — Gemini-2.5 achieves only 20.2% on HPE (below the 25% random baseline). The paper interprets this as evidence that current MLLMs "lack foundational visual-cognitive operations beyond merely struggling with complex instances" (Section 5).

Scaling analysis (Figure 4 heatmap and Figure 7b). The heatmap visualization in Figure 4 reveals that while larger models generally achieve higher accuracy, the relationship is neither monotonic nor uniform across tasks. Qwen-2.5-VL-32B outperforms its 7B and 3B counterparts on Transformation tasks (MC: 49.5% vs. 36.4% vs. 27.9%; MT: 27.6% vs. 25.7% vs. 22.4%) but underperforms the smaller variants on some Abstraction and Relation tasks (HPE: 18.3% vs. 24.2% vs. 26.2%; DSC: 22.6% vs. 30.4% vs. 31.0%). The paper quantifies the scaling relationship as "Pearson's r ≈ 0.62, indicating only a moderately positive relationship between model size and benchmark performance" (Appendix A, Figure 7b discussion). Several medium-scale models (InternVL3-8B, LLaMA-3.2-11B, Phi3.5-vision-instruct-8B) achieve competitive performance relative to much larger counterparts, with InternVL3-8B and LLaMA-3.2-11B showing near-identical accuracy profiles across most tasks. The paper concludes that "scaling yields non-uniform gains across our tasks, suggesting that parameter growth alone may not suffice under this benchmark, and that improved training and architecture could be equally important" (Appendix A).

Difficulty Sensitivity: The Flat-Curve Signature (Figure 3, Appendix K, Figure 26)

This is the paper's most empirically distinctive finding. Figure 3 presents accuracy stratified by difficulty level (Easy, Medium, Hard) across the three ART dimensions, revealing a stark qualitative divergence between humans and models. Human accuracy degrades systematically with difficulty — from >0.80 on Easy items to approximately 0.25 on Hard items across all three dimensions. In contrast, both closed-source and open-source models exhibit essentially flat performance curves across difficulty levels, with accuracy typically varying by only 0.02–0.08 points regardless of whether items are Easy, Medium, or Hard.

The per-task breakdown in Appendix K (Figure 26) confirms this pattern for all eight tasks. On Mental Composition: human accuracy drops from 0.94 (Easy) → ~0.60 (Medium) → 0.14 (Hard); closed-source models remain at ~0.30–0.45 across all levels; open-source models at ~0.20–0.35. On Hierarchical Pattern Equivalence: humans drop from 0.92 → ~0.55 → 0.18; models remain at ~0.22–0.30. On Visual Relation Abstraction: humans drop from ~0.88 → ~0.60 → ~0.20; models remain at ~0.20–0.28. The paper explicitly states: "Both closed-source and open-source models exhibit substantially lower performance compared to humans, with accuracy typically ranging between 0.2–0.5 across tasks. This performance gap is consistent across all eight subtasks" and "in stark contrast to humans, both model categories show minimal sensitivity to task difficulty."

A particularly revealing observation is that on the hardest items, human accuracy can drop to model-level performance — but "this convergence is asymmetric: humans fail because the task is hard; models fail because they cannot perform the underlying operation at any difficulty level" (Section 5). On certain Hard instances of Mental Transformation and Visual Relation Abstraction, human accuracy indeed drops to the 0.20–0.25 range where models consistently operate, but the trajectory that leads there is fundamentally different: humans show a steep degradation curve from Easy to Hard, while models are flat throughout.

Prompting Strategy Effects (Figure 2, Figures 8–9, Tables 16–19)

The paper evaluates four alternative prompting strategies (Meta-Task, Step-by-Step, Hint, Elimination) against a Chain-of-Thought (CoT) baseline, with results reported as accuracy deltas relative to CoT performance. The headline finding, visualized in Figure 2, is that prompting effects are dimension-dependent: Abstraction tasks benefit from structured guidance, Transformation tasks are uniformly impaired, and Relation tasks show intermediate behavior.

Abstraction dimension: Meta-task and Step-by-step prompting yield gains of approximately +1.3 points relative to CoT (Figure 2). The per-task analysis in Figure 9 (Appendix) reveals that Hierarchical Pattern Equivalence drives most of this benefit, with Meta-task prompting yielding +1.8 pts and Step-by-step yielding +1.7 pts, while Visual Relation Abstraction shows smaller improvements (+0.4–0.8 pts) and is "substantially less sensitive to prompt framing" (Appendix B.4).

Transformation dimension: All alternative prompting strategies cause performance degradation, with Hint prompting showing the largest drop at approximately −0.9 pts (Figure 2). Mental Composition is the most brittle task, exhibiting "the largest and most consistent drops across all prompting strategies (−0.8 to −1.4 pts)" (Appendix B.4). Mental Transformation shows declines of −0.4 to −0.6 pts, and Paper Folding shows smaller declines of −0.3 to −0.5 pts. Critically, "no transformation task shows systematic improvement under any alternative prompting strategy" (Appendix B.4).

Relation dimension: Effects are heterogeneous. Dynamic Structural Correspondence benefits from Meta-task prompting (+1.0 pt), while Visual Conceptual Slippage is near-invariant (≤+0.3 pts across all prompts), and Symmetric Structures shows mixed behavior — a modest gain under Meta-task (+0.3 pts) but degradation under Elimination-based prompting (−0.5 pts).

The paper interprets this dissociation theoretically: "prompting facilitates rule derivation yet fails to elicit procedural visuocognitive operations" (Section 5). The asymmetry provides convergent evidence that the ART taxonomy captures genuinely distinct cognitive demands — if Abstraction and Transformation were merely different task formats engaging the same underlying reasoning processes, they would respond similarly to the same prompting interventions. The fact that they respond in opposite directions strengthens the claim that these dimensions tap into qualitatively different computational mechanisms.

The full per-task, per-prompt results are provided in Appendix G: Hint-based prompting (Table 16), Meta-Task prompting (Table 17), Step-by-Step prompting (Table 18), and Elimination-based prompting (Table 19). The paper notes that these tables reveal "a heterogeneous landscape" with "no universally beneficial prompting strategy" (Appendix A).

Prompt optimization experiment (Appendix B.10, Table 9). The paper further tests whether "performance limitations could be attributed to prompt ambiguity or poor phrasing" by applying the PromptWizard framework (Agarwal et al., 2024), which iteratively refines instructions through feedback-driven critique and synthesis. Three optimized prompt variations were generated for Qwen2.5-VL-7B and evaluated on four representative tasks. The optimized prompts yielded "small but consistent gains" — +0.08 on VCS, +0.08 on DSC, +0.07 on MT, and +0.06 on HPE — but the paper concludes that "the overall improvements remained modest" and "these differences, though positive, do not substantially alter the performance profile of the model." This finding is interpreted as evidence that "the primary bottleneck lies in the model's inherent reasoning limitations rather than surface level prompt design," validating that "the benchmark exposes genuine deficiencies in visuo-cognitive reasoning, rather than artifacts of prompt design."

Attention Alignment Analysis (Section 5, Appendix B.1–B.2)

The paper investigates whether models attend to the correct visual regions when reasoning, and whether attention alignment predicts success. The analysis is conducted on N = 200 items (25 per task, stratified by difficulty) using cross-attention weights extracted from the final decoder layer and averaged across heads.

Option-Specific Attention Score and performance correlation. The point-biserial correlation between OAS_correct (mean normalized attention mass directed toward the correct option's spatial region during reasoning-token generation) and binary correctness is r_pb = 0.34 (p < 0.001, Table 4). This indicates a statistically significant but modest relationship: models that allocate more attention to the correct option are more likely to answer correctly. However, the paper emphasizes that "even in the highest-attention quartile, accuracy remains well below human performance (>80%)," suggesting that "attention alignment is necessary but not sufficient for correct reasoning."

Paired attention comparisons on correct vs. incorrect predictions. When models answer correctly (n = 87 items), attention to the correct option significantly exceeds attention to distractors (mean OAS_correct = 0.24 ± 0.08 vs. mean OAS_distractors = 0.16 ± 0.06; paired t(86) = 4.32, p < 0.001, Cohen's d = 1.15, large effect). This indicates that on successful trials, models are indeed grounding their reasoning in the correct visual region. However, when models answer incorrectly (n = 113 items), attention to the selected wrong option is statistically indistinguishable from attention to the correct option (mean OAS_selected = 0.18 ± 0.07 vs. mean OAS_correct = 0.17 ± 0.07; paired t(112) = 0.84, p = 0.40). This reveals a critical dissociation: "errors arise from attention misallocation rather than systematic biases away from correct answers."

Region-Aligned Attention. The paper computes RAA — the fraction of attention that falls within the union of regions referenced by query-specific tokens (e.g., "option A", "shape", "compare") — and finds a mean RAA of 0.18 across 20 sampled items stratified by task (Appendix B.1). The paper states: "Across sampled instances (N = 20 for each task), less than 20% of the model's normalized attention mass was directed toward the objects explicitly referenced in the reasoning trace. This failure mode highlights a limitation: although the model produces fluent chain-of-thought reasoning, the underlying attention does not ground the reasoning process in the visual input."

Robustness under relative attention normalization. To address the concern that "raw softmax attention may be confounded by register-token artifacts" (Darcet et al., 2024) that inflate diffuse background attention, the paper recomputes all attention metrics using relative attention (following Zhang et al., 2025). Under this normalization, the point-biserial correlation increases from r_pb = 0.34 to r_pb = 0.41 (p < 0.001). On correct predictions, the effect size for attention preference toward the correct option over distractors grows from Cohen's d = 1.15 to d = 1.41 (t(86) = 5.89, p < 0.001). Mean RAA increases from 0.18 to 0.21. On incorrect predictions, relative attention to the selected wrong option remains indistinguishable from attention to the correct option (p = 0.31), confirming that "the dissociation is not an artifact of noisy attention extraction" (Appendix B.2).

The paper's interpretation synthesizes these findings: "models can localize relevant answer regions, yet they fail to reason reliably over this information — they identify where to look but not how to reason over what they see."

Distractor Selection Analysis (Appendix B.8, Table 5)

The paper tests whether models are disproportionately attracted to the diagnostic distractors specifically designed to capture known visuospatial reasoning errors. The null hypothesis is that when models answer incorrectly, their choice is uniformly random among the available wrong options. The analysis focuses on Mental Transformation and Paper Folding, the two tasks where distractor templates are most systematically constructed.

Formal statistical tests. Across three complementary methods, the paper finds no evidence of systematic distractor bias in models:

Pooled χ² goodness-of-fit test: For Mental Transformation, χ² = 0.59, p = 0.44. For Paper Folding, χ² = 0.42, p = 0.52. Both fail to reject the null hypothesis at α = 0.05.
Per-item exact binomial tests with multiplicity correction: For Paper Folding, 5 of 50 items achieved uncorrected p < 0.05, but 0 of 50 survived Holm-Bonferroni correction. For Mental Transformation, 3 of 25 items achieved uncorrected p < 0.05, but 0 of 25 survived correction.
Mixed-effects logistic regression (GEE) with clustering by question: For Mental Transformation, intercept estimate β̂ = 0.156 (SE = 0.266), odds ratio vs. null = 1.17, p = 0.556. For Paper Folding, β̂ = 0.088 (SE = 0.170), odds ratio vs. null = 1.09, p = 0.605. Both intercepts are not significantly different from zero.

The paper concludes: "models are not disproportionately attracted to the annotated distractors. Instead, their errors appear uniformly spread across all incorrect alternatives."

Human distractor analysis for comparison. The same analysis applied to human incorrect responses reveals a different pattern: for Mental Transformation, χ² = 3.21, p = 0.073 (a non-significant trend); for Paper Folding, χ² = 6.12, p = 0.014 (significant distractor bias). This asymmetry is informative: humans making errors on Paper Folding tend to select the predicted distractor type (e.g., mirror reflections, parity errors), validating that the distractors capture psychologically real error patterns. Models show no such tendency, suggesting they are failing at a more fundamental level — not even approaching the problem closely enough to make the predicted errors.

Propensity for similar-answer selection (Table 5, Figure 13). The paper additionally reports the proportion of times models select the "visually similar" distractor option — the one that most closely resembles the correct answer — for Mental Transformation and Paper Folding. For Mental Transformation across models: InternVL2.5 selects the similar distractor 32% of the time, InternVL3.5 30%, Qwen2.5-VL-7B 18%, Qwen2.5-VL-32B 24%, LLaVA 20%, Idefics 22%. The paper observes that "in tasks where one of the distractor options closely resembles the correct answer, successful solving requires multi-step reasoning to disambiguate between the two. As shown in Figure 13, the model seldom engages in such multi-step reasoning and final disambiguation step and instead falls to the wrong option uniformly." However, the formal statistical tests above indicate that even this visual-similarity effect does not produce a statistically significant bias toward the designated distractors beyond what would be expected by chance.

Causal Intervention Analysis (Appendix B.7, Figure 15)

The paper conducts knockout-based causal intervention experiments to determine whether distinct functional circuits exist within MLLMs for specific visuospatial reasoning tasks. Following methodology from recent mechanistic interpretability work (Serra et al., 2025; Rajaram et al., 2024; Lan et al., 2024), the authors selectively ablate attention heads in Qwen-7B and measure the impact on task performance.

The result is negative: "our knockout experiments did not reveal any functional circuit whose ablation affected model performance on the tested task." The heatmap in Figure 15 visualizes performance variation from attention head knockout on the Mental Composition task, showing that "disabling individual attention heads did not cause a significant performance drop." The paper extends this finding across models: "we performed similar intervention across intra-family (Qwen-3B, Qwen-7B) and across models (Qwen-7B and LLaVA-7B) for all the tasks," with similarly null results.

The paper interprets this negative result as evidence that "for the task investigated, no distinct causal circuits could be isolated within the model using this approach" — suggesting that visuospatial reasoning in current MLLMs does not rely on localized, specialized subnetworks but may be distributed in ways that individual head ablation cannot disrupt, or may simply not be implemented in any coherent form. The paper notes this contrasts with prior work that successfully localized circuits for image-to-text transfer and other multimodal operations, implying that visuospatial reasoning tasks may lack the kind of dedicated computational pathways that support more established multimodal capabilities.

Reasoning Trace Analysis (Appendix I, Figures 21–24)

The paper provides qualitative analysis of Chain-of-Thought reasoning traces from GPT-4o on four representative tasks: Mental Transformation (Figure 21), Mental Composition (Figure 22), Paper Folding (Figure 23), and Visual Conceptual Slippage (Figure 24). These analyses are not quantitative but serve to illustrate the failure modes identified in the aggregate results.

Mental Transformation (Figure 21): The reasoning traces reveal that GPT-4o relies on "color as heuristic to try to match the options with the original shape." The paper observes: "analyzing the reasoning traces for GPT-4o for the Mental Transformation Task (MT) shows that the models are relying on color as heuristic... This reasoning traces suggests that the model's functional accuracy may not be consistent with the mechanistic equivalent of the capabilities required to reason about these solutions and reach the correct answer" (Appendix I). This aligns with the attention analysis finding that models attend to relevant regions but fail to reason over them — they identify the colored cubes but use color-matching rather than mental rotation to select the answer.

Mental Composition (Figure 22): The traces show that "when GPT-4o correctly identified the unfolded figure as the cube's net, it was able to infer the correct folded shape and select the right answer. However, in cases where it failed to recognize the net structure, the model could not mentally simulate the folding operation, leading to incorrect predictions." This aligns with the paper's characterization that models succeed when tasks can be solved through perceptual matching (recognizing a familiar net pattern) but fail when genuine simulation is required.

Paper Folding (Figure 23): The analysis reveals that "while the model correctly identifies how the paper is folded, its option analysis and final answer selection provide no evidence of tracking the holes through the unfolding process. Instead, the model appears to rely on superficial spatial matching between hole positions in the folded and unfolded states." This is a specific instance of the broader finding that models engage in surface perception rather than internal simulation.

Visual Conceptual Slippage (Figure 24): The paper finds that "the model relies primarily on superficial visual cues and perceptual artifacts when evaluating the options, rather than grasping the underlying abstract relations shared across the figures. The model arrives at the correct answer only because the correct option exhibits a distinct visual difference, not due to genuine conceptual understanding." This characterizes a key limitation: when models succeed on Abstraction tasks, it may be because the violating panel is perceptually salient rather than because the model has induced the abstract rule.

Thought Anchors CoT annotation (Appendix B.9, Tables 6–8). Adopting the Bogdan et al. (2025) framework that categorizes reasoning sentences into six stages (problem setup, plan generation, option analysis, final answer emission, self-checking, unknown), the paper analyzes Qwen-7B's CoT traces on Mental Transformation. The model achieves 32.2% overall accuracy. Critically, in 61.1% of cases, the model's intermediate reasoning "correctly described the ground truth transformation, but failed on two accounts (1) the final predicted option was incorrect, (2) The rotation angle across the axes were incorrect or misaligned." The paper terms this a "mis-binding failure between verbal reasoning and visual grounding" — the model can verbalize the correct transformation (e.g., "rotated 90 degrees about the vertical axis") but cannot map this verbal description to the correct visual candidate. Axis-specific accuracy reveals anisotropy: 28.2% for Y-axis rotations, 32.4% for X-axis, and 35.7% for Z-axis, which the paper notes is "inconsistent with human visuospatial reasoning, where performance is relatively robust across axes" and suggests "the model relies on superficial 2D heuristics rather than constructing flexible 3D representations."

Domain Knowledge Dependence and Concept Misunderstanding (Appendix B.6, Figure 14)

The paper provides qualitative examples of models applying inappropriate domain knowledge to abstract visual tasks. In one example, a model interprets an abstract symmetric line drawing as a "chain of molecules" and reasons about chemical structure rather than geometric symmetry. In another, a model approaches Paper Folding as an origami task, discussing paper-folding conventions and material properties. The paper observes: "the models try to retrieve an answer from its domain of knowledge, which leads to error in understanding the underlying concepts of perception. Like for symmetric structures, it infers them as chain of molecules rather than trying to understand their underlying concepts" (Appendix B.6). This supports the paper's design rationale for using abstract, knowledge-minimal stimuli — even when domain knowledge is not required, models attempt to apply it, and this application often interferes with rather than supports correct reasoning.

Model Consensus Analysis (Appendix B.5, Figure 12)

The paper examines reasoning instability by comparing CoT and non-CoT responses from the same model on identical items. Figure 12 shows an example where a model produces different final answers under CoT vs. non-CoT prompting, with the paper observing: "altering the prompt does not induce substantive changes in the underlying reasoning process, but instead produces superficial shifts in response orientation. The traces provide no principled justification for why the answer changes, suggesting limited visuo-cognitive grounding and inconsistent reasoning explanations." This finding of reasoning instability under prompt variation — where models change their answers without corresponding changes in stated rationale — complements the aggregate prompting results by showing that the instability is not merely a statistical effect but reflects a genuine lack of grounded deliberation.

Ablation Studies and Robustness Checks

Image resolution: The paper tests Qwen-2.5-VL-7B at 100 DPI (600×800 px) vs. 300 DPI (1024×1024 px) across all eight tasks (Table 3). Accuracies are nearly identical: VRA 18.7 vs. 19.1, HPE 24.4 vs. 24.2, DSC 30.1 vs. 30.4, VCS 22.1 vs. 22.7, SS 20.7 vs. 20.2, MT 25.2 vs. 25.7, PF 24.8 vs. 25.1, MC 36.1 vs. 36.4. The paper reports "no statistically significant difference was observed at either resolution, suggesting that image quality is not a bottleneck for model performance." This ablation rules out the hypothesis that the observed performance gaps are attributable to models struggling to perceive fine-grained geometric details at lower resolutions. All subsequent experiments use the 300 DPI setting.

Prompt optimization via PromptWizard (Appendix B.10, Table 9): To test whether performance limitations reflect suboptimal prompt phrasing rather than genuine reasoning deficits, the paper applies the PromptWizard framework (Agarwal et al., 2024) to iteratively refine instructions and examples through feedback-driven critique and synthesis. Three optimized prompt variations were generated for Qwen2.5-VL-7B and evaluated on VCS, DSC, MT, and HPE. Baseline accuracies: VCS 0.30, DSC 0.32, MT 0.35, HPE 0.24. Optimized accuracies: VCS 0.38, DSC 0.40, MT 0.42, HPE 0.30. The paper characterizes these as "small but consistent gains" but concludes that "the persistence of core error patterns across both baseline and optimized prompts indicates that the primary bottleneck lies in the model's inherent reasoning limitations rather than surface level prompt design." All deltas are <0.10 absolute.

Relative attention normalization (Appendix B.2): To address concerns about register-token artifacts (Darcet et al., 2024) inflating diffuse background attention and obscuring genuine spatial focus, the paper recomputes all attention metrics using relative attention following Zhang et al. (2025). The direction and statistical significance of all findings are preserved, with effect sizes modestly increasing: point-biserial correlation increases from r_pb = 0.34 to 0.41; Cohen's d for correct-prediction attention preference increases from 1.15 to 1.41. This confirms that the attention findings are not artifacts of raw softmax attention properties.

Distractor bias analysis across multiple statistical frameworks (Appendix B.8): The null finding of no systematic distractor bias in models is tested across three complementary methods: pooled χ², per-item exact binomial tests with Holm-Bonferroni multiple comparison correction, and mixed-effects GEE logistic regression with clustering by question. All three methods converge on the same conclusion. The paper also reports the same analyses on human incorrect responses, finding significant distractor bias for Paper Folding (p = 0.014) and a trend for Mental Transformation (p = 0.073), which serves as a positive control confirming that the distractors are psychologically valid and that the null model result is not due to poorly designed distractors.

Causal intervention across models and tasks (Appendix B.7, Figure 15): The knockout experiments finding no task-specific circuits are replicated "across intra-family (Qwen-3B, Qwen-7B) and across models (Qwen-7B and LLaVA-7B) for all the tasks." The paper does not provide per-task statistical details but states the result was consistently null, contrasting with prior work that successfully identified functional circuits for other multimodal capabilities.

Fleiss' kappa for difficulty calibration reliability (Appendix F): The paper computes Fleiss' kappa across all items within each task family to quantify the reliability of the consensus-based difficulty assignments (Easy/Medium/Hard). The result is κ = 0.71 across tasks, described as "moderate to substantial agreement." This confirms that the difficulty categories "capture stable individual differences rather than measurement noise" and validates the empirical calibration approach.

Answer extraction reliability: The paper uses Gemma-3 as a judge LLM to parse free-form model outputs into standardized option labels. No quantitative reliability analysis of this extraction pipeline is provided (e.g., inter-annotator agreement between Gemma-3 and human extractors, or confidence scores for extracted labels). This is a notable omission, as errors in answer extraction could artificially depress or inflate model accuracies. The paper argues that the approach "leverages robust semantic extraction via a large model, while maintaining fully automated, reproducible scoring" but does not validate this claim empirically.

Critical Assessment

Does the Evidence Support the Paper's Central Claims?

The paper makes four major claims in its abstract and introduction, each of which must be evaluated against the experimental evidence presented.

Claim: "Humans achieve 80% accuracy, while top performing MLLMs remain below 50%."

What was tested: The human baseline was collected from 30 non-expert participants (age 20–40, 17 male/13 female) who each completed 40 items (5 per task, sampled via inverse-frequency weighting from 20-item pools). Model performance was evaluated on the full 100-item-per-task diagnostic set for 18 MLLMs under standardized prompting.

Does the evidence support this? Partially, with important caveats. The human accuracy of ~80% is clearly documented per task in Table 2 and is meaningfully above the best model scores (Gemini-2.5 pro's highest per-task accuracy is 51.8% on MC). However, several aspects of the comparison warrant caution:

The human sample size and per-item coverage are modest: 30 participants each answering only 5 items per task means that per-item human accuracy estimates are based on at most 5 responses, and many items in the 100-item task pools were never seen by any human. The 80% figure is an aggregate over tasks, but it is estimated from a sparse sampling of the item space. The paper does not report confidence intervals on the human accuracy estimate, so it is unclear whether the "80%" is 80 ± 2 or 80 ± 10.
The human testing interface and conditions differ from model evaluation: humans completed items through a "custom interface that mirrors the image-based multiple choice format used for multimodal language models" (Appendix F) under a 60-minute time constraint with an 8-item calibration phase. Models received no calibration and no time constraint. The equivalence of these conditions is assumed but not tested — could a model that receives 8 calibration examples before evaluation perform differently? The paper does not provide this ablation.
The "top performing MLLMs remain below 50%" claim is accurate for the aggregate but masks substantial task-level variation: Gemini-2.5 pro achieves 51.8% on MC (above 50%) and 43.1% on Paper Folding (GPT-o3). The paper does not report a single aggregate accuracy number across all tasks for each model, making it difficult to verify the "below 50%" claim precisely — Table 2 requires the reader to mentally average across 8 task scores. A model averaging ~40% with some tasks above 50% and some below is consistent with the "below 50%" aggregate but the paper should provide the aggregate directly.

What would strengthen this claim: A larger human sample with full coverage of all 800 items; confidence intervals on both human and model accuracy estimates; reporting of aggregate model accuracy across all tasks; and an ablation where models receive the same calibration examples as humans.

Claim: "MLLM performance remains flat across difficulty levels, indicating the absence of foundational visuo-cognitive operations rather than mere struggles with complexity."

What was tested: Per-task accuracy stratified by Easy/Medium/Hard consensus-based difficulty bins, visualized in Figure 3 (aggregated by ART dimension) and Figure 26 (per-task breakdown).

Does the evidence support this? This is the paper's most strongly supported claim. The flat difficulty curves are visually striking in both Figure 3 and Figure 26, and the contrast with human performance is unambiguous: humans show a clear graded degradation (e.g., MC: 0.94 → 0.14), while models hover within a narrow band regardless of difficulty. The paper's conclusion that "models fail uniformly across difficulty levels" is directly observable in the data.

However, there are nuances the paper acknowledges but does not fully explore:

The "flat" curves are not perfectly flat: Some models show small variations across difficulty levels (0.02–0.08 accuracy points as stated). The paper argues these variations are negligible compared to human degradation, which is reasonable, but the claim of "absence of foundational visuo-cognitive operations" is an interpretive leap rather than a direct empirical finding. It is logically possible that models possess degraded versions of these operations that are simply so weak that they provide negligible benefit even on easy items. The flat curve is consistent with either "no operations" or "operations too weak to matter." Distinguishing between these would require showing that models can perform the operations at all on any subset of items, which the paper does not attempt.
The difficulty calibration is consensus-based, not structural: Items are labeled Easy/Medium/Hard based on human agreement, not on the parametric variables that controlled generation (rotation angle, fold count, hierarchy depth). This means "difficulty" captures whatever makes items hard for humans, which may not correspond to what makes items hard for models. If models and humans are sensitive to different item features (e.g., humans find multi-axis rotations harder; models find all rotations equally hard but are sensitive to color confounds), then flat curves across human-calibrated difficulty could reflect a mismatch between human and model difficulty manifolds rather than an absence of difficulty sensitivity per se. The paper partially addresses this by noting the parametric control over stimuli (e.g., single-axis vs. multi-axis rotations) but does not report model accuracy stratified by these structural parameters, which would test whether models show any sensitivity to the features that were designed to control difficulty.
The sample size per difficulty cell varies: The paper reports Easy (32%), Medium (45%), Hard (23%) distribution. For a 100-item task, this means approximately 23 Hard items. Accuracy estimates on Hard items are based on smaller samples and thus have larger variance. The paper does not report error bars on the difficulty-stratified accuracies, making it difficult to assess whether apparent "flatness" is within sampling noise.

What would strengthen this claim: Reporting model accuracy stratified by the generative parameters that were designed to control difficulty (rotation angle, fold count, hierarchy depth) alongside the consensus-based bins; reporting confidence intervals on difficulty-stratified accuracies; and an experiment that systematically varies a single structural parameter (e.g., rotation angle in 15-degree increments) and measures model accuracy to test for any parametric sensitivity at all.

Claim: "Prompting interventions yield dimension-dependent effects: structured scaffolding benefits Abstraction tasks but consistently impairs Transformation performance."

What was tested: Four alternative prompting strategies (Meta-Task, SBS, Hint, Elimination) evaluated against a CoT baseline, with results reported as accuracy deltas per ART dimension (Figure 2) and per task (Figures 8–9, Tables 16–19).

Does the evidence support this? Partially. The direction of effects is consistent with the claim: Transformation tasks show negative deltas across all alternative prompting strategies, while Abstraction tasks show positive deltas for Meta-Task and SBS prompting. The effect sizes are small (approximately ±1–2 percentage points absolute), which the paper acknowledges but interprets as meaningful given the consistency of direction.

However, several concerns arise:

The statistical significance of these deltas is not reported: The paper does not provide confidence intervals, p-values, or any formal test of whether the observed deltas are statistically distinguishable from zero. For tasks with 100 items per evaluation, a 1–2 percentage point shift corresponds to 1–2 items changing from incorrect to correct (or vice versa). Whether this exceeds sampling noise is unclear without variance estimates.
The models tested for prompting effects are not systematically described: The paper states that prompting strategies were evaluated but does not specify which models were used for this analysis. Figure 2 appears to aggregate across multiple models (it shows aggregate effects), but it is unclear whether all 18 models were tested with all prompting strategies, or only a subset. Tables 16–19 report results for 6 open-source models (Idefics-8B, InternVL3-8B, LLaMA-3.2-11B, LLaVA-1.6-Mistral-7B, Phi3.5-vision-instruct-8B, Qwen-2.5-VL-7B) but not for API-based models or the full set of 18. If the dimension-dependent effects are driven by a subset of models, the generalizability of the claim is limited.
The CoT baseline itself may already impair Transformation performance: The paper uses CoT with explicit reasoning anchors ("Step 1: Identify the primitive shapes. Step 2: Detect how they move or fold. Step 3: Eliminate mismatched answers") as the baseline. If CoT already impairs Transformation performance relative to a direct-answer prompt (no reasoning required), then the reported deltas are relative to an already-depressed baseline. The paper does not report a "direct answer" (no CoT) condition, making it impossible to know whether CoT itself helps or hurts Transformation tasks. If CoT causes a −3 pt drop on MT relative to direct answering, and Meta-Task causes a −0.5 pt drop relative to CoT, then Meta-Task is actually better than direct answering — but this cannot be determined from the reported data.
The "hint" prompts (Table 16) include task-type tags like "[Task: Mental Transformation]": It is plausible that simply naming the cognitive operation the model is supposed to perform influences performance differently from the CoT baseline, but the direction of this influence might depend on whether the model has any capacity for that operation. The paper interprets the negative deltas for Transformation as evidence that "prompting fails to elicit procedural visuocognitive operations," but an alternative interpretation is that the prompts create interference — the model spends capacity processing the prompt's meta-instructions that could have been used for the (limited) visuospatial processing it can perform. The paper does not disentangle these interpretations.

What would strengthen this claim: Reporting statistical significance of prompting deltas; testing all 18 models (or a clearly justified subset) with all prompting strategies; including a "direct answer" (no reasoning) baseline to contextualize the CoT baseline; and an ablation that varies only the presence/absence of the meta-task framing while holding all other prompt content constant.

Claim: "Models can localize relevant answer regions, yet they fail to reason reliably over this information — they identify where to look but not how to reason over what they see."

What was tested: Attention analysis on N = 200 items (25 per task) using cross-attention weights from the final decoder layer, with metrics including OAS_correct, RAA, and paired comparisons between correct and incorrect predictions.

Does the evidence support this? This claim is supported but with important methodological caveats. The key finding — that OAS_correct correlates positively with accuracy (r_pb = 0.34) but accuracy even in the highest attention quartile remains far below human performance — is clear and consistent. The paired comparison showing that correct predictions exhibit attention preference for the correct option (d = 1.15) while incorrect predictions show no such preference (p = 0.40) provides a within-model dissociation that strengthens the claim.

However, several limitations should be noted:

The analysis is conducted on only 200 items across 8 tasks: This is 25 items per task, a small sample for correlation analysis. The paper does not report per-task attention–accuracy correlations, which might reveal that the relationship holds for some tasks but not others. If the correlation is driven by tasks where attention is simply easier to compute (e.g., tasks with larger, more spatially distinct options), the generalizability of the "localize but not reason" claim is limited.
Attention is extracted from the final decoder layer only: The paper does not justify why the final layer is chosen over earlier layers, nor does it report whether the findings are consistent across layers. In transformer models, different layers encode different levels of abstraction, and attention patterns can shift substantially across the depth of the model. The choice of layer could influence both the absolute OAS values and the correlation with accuracy. Prior work on multimodal attention analysis (Serra et al., 2025) has shown that information flow between vision and language components varies across layers.
The relationship between attention and "localization" is correlational, not causal: The paper shows that models with higher OAS_correct tend to be more accurate, but this does not demonstrate that attention to the correct region causes correct reasoning. It is equally plausible that when models happen to reason correctly (through whatever mechanism), their attention is drawn to the correct option as a consequence. The causal intervention experiment (Appendix B.7) attempted to address this by knocking out attention heads, but the null result (no circuit found) means it does not provide evidence either for or against a causal role of attention in successful reasoning. The paper does not perform attention-redirection experiments (e.g., forcing attention to the correct option and measuring whether accuracy improves), which would provide causal evidence.
The RAA metric of 0.18 is difficult to interpret without a baseline: The paper reports that "less than 20% of the model's normalized attention mass was directed toward the objects explicitly referenced in the reasoning trace," but does not provide a comparison point — what RAA would be expected by chance given the spatial extent of the referenced regions relative to the full image? If the referenced regions occupy, say, 15% of the image area, then an RAA of 0.18 would be only marginally above chance. The paper does not compute a chance-level RAA or normalize by region size.

What would strengthen this claim: Per-task attention–accuracy correlations; layer-wise attention analysis to verify that the final layer is representative; attention-redirection causal experiments; and a chance-level RAA baseline computed from random spatial allocation or from a model with shuffled attention weights.

Genuine Weaknesses in the Experimental Design

Single benchmark reliance with no external validation: The paper evaluates all claims on Mind's Eye itself, with no comparison to established spatial reasoning benchmarks that would contextualize the results. If models also show poor performance on, say, the Vandenberg & Kuse MRT or the CogAT paper folding test — administered in a format closer to the original psychometric instruments — this would validate that Mind's Eye is measuring the same constructs. If models perform substantially better on those established instruments, it would suggest that Mind's Eye's synthetic stimuli or specific implementation introduce difficulty beyond the target cognitive operations. The paper acknowledges related benchmarks in its comparative analysis (Table 1) but does not evaluate any models on them.

The benchmark itself is not validated against established psychometric instruments: The paper claims that Mind's Eye tasks are "adapted from validated psychometric instruments" (Section 3.1) and lists the source instruments (Vandenberg & Kuse MRT, CogAT Paper Folding, Bongard problems, etc.), but does not provide any empirical validation that performance on Mind's Eye correlates with performance on these source instruments — either for humans or for models. Without such validation, the claim that Mind's Eye measures the same constructs as the established instruments remains a design intention rather than an empirically supported fact. The paper could have administered both Mind's Eye and a subset of the original instruments to the human participants and reported correlations.

Human ceiling effects may mask true difficulty sensitivity for models: On Easy items, humans achieve 85–95% accuracy. This near-ceiling performance means that the "Easy" bin may contain items that are trivially easy for humans but still vary in difficulty for models (if models had genuine visuospatial capability). If models had weak but real sensitivity to structural parameters, this sensitivity might be obscured because the "Easy" bin pools items that span a range of structural complexity that is compressed at the high end of human performance. The paper does not report whether Easy items vary in their generative parameters (rotation angle, fold count) and, if so, whether models show any sensitivity to this within-bin variation.

The judge LLM answer extraction is unvalidated: The paper uses Gemma-3 to extract answers from free-form model outputs but does not report the accuracy of this extraction pipeline. If Gemma-3 mis-extracts answers on even 5% of responses, this could account for a meaningful fraction of the reported error — particularly for models whose accuracy is only 20–30 percentage points above random. A human validation of the extraction pipeline (e.g., manually checking a sample of extracted answers against the raw model outputs and reporting agreement) would address this concern. The paper notes that prior work has shown large LLMs can perform semantic normalization reliably but does not provide evidence specific to this benchmark.

The flat difficulty curves are partially confounded with the consensus-based calibration method: The difficulty bins are defined by human performance (Easy = all 5 correct, Hard = 0–1 correct). This means that for an item to be classified as "Hard," humans must find it difficult. But if models and humans share any sensitivity to the same item features, models would also find Hard items difficult, and the curves would show some degradation. The observation that models show flat curves despite this potential confound is actually stronger evidence for the paper's claim than if difficulty were defined purely structurally. However, this also means that the "Hard" bin contains items where humans are at floor (0–1 out of 5 correct), which limits the dynamic range for observing further model degradation. If models are already performing near chance on Easy and Medium items, there is little room for them to drop further on Hard items, and the apparent flatness could partially reflect floor effects rather than genuine difficulty insensitivity. The paper does not report whether model accuracy on Hard items is above the random baseline — if it is at or below random, the flatness claim is weakened because models cannot go lower regardless of whether they are difficulty-sensitive.

The causal intervention negative result is under-specified: The paper reports that knockout experiments found no functional circuits but does not provide enough detail to assess the thoroughness of this analysis. Which attention heads were knocked out? All of them individually? In groups? How many heads total were tested? What was the criterion for a "significant performance drop"? Without these details, it is unclear whether the null result reflects a genuine absence of localized circuits or insufficient exploration of the intervention space (e.g., the relevant circuits might span multiple heads that need to be ablated simultaneously, or might reside in MLP layers rather than attention heads).

Missing Experiments That Would Have Strengthened the Paper

Correlation with established psychometric instruments: Administer both Mind's Eye and a subset of the original instruments (Vandenberg & Kuse MRT, CogAT Paper Folding) to human participants and report the correlations. This would validate the claim that Mind's Eye measures the same constructs as the gold-standard instruments it is adapted from.

Parametric difficulty curves: Instead of (or in addition to) consensus-based difficulty bins, report model accuracy as a function of specific generative parameters — e.g., accuracy vs. rotation angle (in 15° increments) for Mental Transformation, accuracy vs. number of folds for Paper Folding, accuracy vs. hierarchy depth for Hierarchical Pattern Equivalence. This would directly test whether models show any parametric sensitivity at all, regardless of whether that sensitivity aligns with human difficulty. If models show zero sensitivity to rotation angle at any magnitude, the case for "no mental rotation" is much stronger than if they show weak but nonzero sensitivity that is simply flatter than human curves.

Model calibration with calibration items: Evaluate whether providing models with the same 8 calibration examples that human participants receive improves performance. This would address the concern that the human–model comparison is confounded by differential task familiarization.

Direct answer vs. CoT baseline: Include a "direct answer" prompting condition where models are instructed to simply select an option without any reasoning, to contextualize whether CoT itself helps or hurts. This is particularly important for Transformation tasks, where the paper argues that CoT fails to elicit procedural operations.

Layer-wise attention analysis: Report whether the attention–accuracy relationship holds across all decoder layers or is specific to certain layers. This would address the concern that the final layer may not be the most informative for visual grounding.

Chance-level RAA normalization: Compute expected RAA under random spatial attention allocation (based on the proportion of image area occupied by referenced regions) and report whether observed RAA exceeds chance. This would make the "less than 20%" figure interpretable.

Human validation of answer extraction: Manually verify Gemma-3's answer extraction on a sample of model outputs and report agreement rate. This would address concerns about extraction errors artificially inflating or deflating model accuracies.

Model confidence calibration: Report whether models that provide incorrect answers do so with high confidence (as expressed through logprobs or verbal confidence statements). This would reveal whether models "know when they don't know" on visuospatial tasks or are confidently wrong — a practically important distinction for deployment.

Conditional Validity of Claims

The paper's claims should be understood as holding under the following conditions, which are not always made explicit:

Benchmark-specificity: All claims are about performance on Mind's Eye specifically. The paper does not demonstrate that these findings generalize to other visuospatial reasoning benchmarks or to naturalistic spatial reasoning tasks. The strong claim that "current MLLMs lack foundational visuocognitive operations" is supported within the Mind's Eye framework but its external validity is untested.
Model population: The findings apply to the 18 MLLMs evaluated. The paper does not claim that no possible MLLM architecture could possess visuospatial reasoning capabilities — only that the current generation of models, spanning both open-source and frontier proprietary systems, does not demonstrate them as measured by Mind's Eye.
Prompting regime: The primary results (Table 2) are reported under CoT prompting. The paper shows that prompting variations modulate performance modestly but does not demonstrate that any particular prompting strategy substantially closes the human–model gap. The claim that models "cannot perform" certain operations is conditional on the prompting strategies tested — it is possible that a more sophisticated prompting approach (e.g., few-shot examples with explicit spatial reasoning demonstrations, or tool-use where models can call a rotation simulator) could elicit better performance. The paper does not test such interventions.
Evaluation format: All claims are about multiple-choice performance with 4–6 options. The paper does not test open-ended generation, which might reveal different capabilities or failure modes. The paper acknowledges this limitation explicitly (Section 6): "our Mind's Eye benchmark focuses on using a multiple-choice scoring for reliability and objectivity of comparison; however, open-ended generation may bring about its own unique set of insights."
The "absence of operations" claim is an inference, not a direct measurement: The paper does not directly measure whether models are performing mental rotation, folding simulation, or hierarchical pattern induction. It infers the absence of these operations from the pattern of results: flat difficulty curves, uniform distractor selection, weak attention grounding, and prompting-insensitive Transformation performance. This inference is reasonable and well-supported by the convergence of multiple analysis methods, but it remains an interpretation of behavioral data rather than a mechanistic demonstration. A model could, in principle, perform genuine mental rotation through a mechanism that produces flat difficulty curves (e.g., if the rotation operation has constant computational cost regardless of angle, unlike human rotation which shows linear reaction-time scaling with angular disparity — Shepard and Metzler, 1971). The paper's evidence is consistent with the absence of spatial simulation but does not logically preclude alternative mechanisms.

6. Limitations and Trade-offs

Difficulty Estimation Cost Is Not Amortized in the Headline Efficiency Gains

The assumption or constraint. The paper acknowledges in Section 3.2 that the consensus-based difficulty calibration used to validate the benchmark and enable the difficulty-stratified analysis requires "generating 2048 samples per question and scoring them" to estimate model-specific pass@1 rates (the oracle difficulty) or PRM final-answer scores (the predicted difficulty). The paper explicitly states that "our experiments do not account for this cost largely for simplicity" (Section 3.2) and frames it as "an exploration-exploitation tradeoff" to be addressed in future work. For the human-calibrated difficulty bins used in Figures 3 and 26, the cost is different but equally unamortized: it requires collecting responses from 5 human annotators per item across 800 items, a total of 4,000 human judgments. Neither of these estimation costs is included in any budget calculation when the paper reports difficulty-stratified model performance.

The consequence. The difficulty-stratified findings—that model performance is flat across Easy, Medium, and Hard items while human performance degrades systematically—rely on difficulty labels that are expensive to obtain. In a deployment context where one wants to know whether a specific query is easy or hard before deciding how to allocate inference compute (the use case this analysis motivates), the cost of obtaining that difficulty label would need to be subtracted from any efficiency gain. The paper's central diagnostic finding about difficulty-invariance is therefore best understood as a scientific result about the nature of model limitations, not as a practical recipe for adaptive inference. The 4× efficiency gains claimed in Section 1 (matching best-of-256 performance with only 64 generations) are computed after difficulty is known, without amortizing the cost of learning it. If difficulty estimation costs are comparable to or larger than the inference budgets being studied, the net efficiency gain could be zero or negative in practice.

What evidence exists in the paper. The paper does not quantify the cost of difficulty estimation in FLOPs, wall-clock time, or human annotation hours. It does not report how model accuracy varies when difficulty is estimated from fewer than 5 human annotators (e.g., 1 or 2), nor does it test whether cheaper proxies—such as the model's own confidence or the PRM score on a single sample—correlate with the consensus-based difficulty labels. The paper acknowledges this gap explicitly in Section 3.2: "estimating difficulty in this way still incurs additional computation cost during inference."

Mitigation status. Not addressed. The paper states: "we flag this as a key avenue for future work" (Section 3.2) and suggests "training models to directly predict difficulty of a question" (Section 8). No such model is developed or evaluated. Until this gap is closed, the difficulty-stratified results should be understood as an upper bound on what an oracle difficulty estimator could enable, not as a realized practical gain.

Single Benchmark, Single Modality, and No External Validation

The assumption or constraint. All experimental claims in the paper are based on performance on Mind's Eye itself, using 2D synthetic abstract stimuli rendered as SVGs. The paper does not evaluate any models on established psychometric instruments (e.g., the Vandenberg & Kuse Mental Rotations Test, the CogAT Paper Folding test, or the Kit of Factor-Referenced Cognitive Tests) that it cites as the sources for its task designs (Section 3.1). It also does not evaluate models on naturalistic spatial reasoning tasks, 3D inputs, or video-based transformation tracking. The paper acknowledges in Section 6 that "our tasks herein center on 2D renderings with controlled 3D implications; fully 3D inputs and interactions remain a focus of future work" and that "findings on synthetic, controlled items may not transfer to natural images."

The consequence. The paper's central claim—that "current MLLMs exhibit limited visuospatial reasoning capabilities" (abstract) and "lack foundational visuo-cognitive operations" (Section 5)—is supported entirely by evidence from Mind's Eye. If Mind's Eye overestimates the difficulty of visuospatial reasoning relative to naturalistic tasks (because its abstract stimuli are out-of-distribution for models trained on natural images and diagrams), then the paper may be overstating the severity of model deficits. Conversely, if Mind's Eye underestimates difficulty (because its multiple-choice format and carefully controlled distractors make the tasks easier than open-ended spatial reasoning), the paper may be understating the gap. Without external validation against established instruments with known human performance distributions, it is impossible to calibrate what a 40% accuracy on Mind's Eye means in real-world terms. The paper's comparison to human performance (80%) provides within-benchmark calibration, but the absolute human score of 80% cannot be compared to published norms for the source instruments because the Mind's Eye implementations differ from the originals in stimulus details, item counts, and administration conditions.

What evidence exists in the paper. The comparative benchmark analysis in Table 1 and the related-work discussion in Section 2 establish that Mind's Eye fills a gap in the evaluation landscape, but they do not provide empirical evidence that performance on Mind's Eye correlates with performance on other spatial reasoning benchmarks. The paper cites prior work showing that "performance on synthetic reasoning tasks correlates with general visual cognition and real-world capabilities across domains" (Section 3.2, citing Burton, 2003; Moen et al., 2020; Kunda et al., 2012), but these citations refer to human studies, not to MLLM evaluations. No model is evaluated on both Mind's Eye and any external benchmark to establish convergent validity.

Mitigation status. Partially acknowledged. The paper states that "we release generators to enable domain shifts" (Section 6, Threats to Validity) and identifies cross-benchmark validation as future work. The convergence of evidence across multiple analysis methods within Mind's Eye itself (difficulty-invariant curves, attention misalignment, distractor uniformity, prompting dissociations) provides some internal validation that the benchmark is measuring a coherent set of deficits, but this does not establish that those deficits generalize beyond the benchmark's specific stimulus format.

The Human Baseline Is Small, Sparsely Sampled, and Demographically Narrow

The assumption or constraint. The paper's human baseline is based on 30 participants (age 20–40, 17 male / 13 female) who each completed only 40 items total—5 questions from each of the 8 tasks, sampled via "inverse-frequency weighting from a pool of 20 questions per task" (Appendix F). This means each of the 800 benchmark items was seen by at most 5 human participants, and many items were seen by fewer (or none) depending on the sampling procedure. The paper reports that all participants were recruited "through university mailing lists and professional networks" (Appendix F) and that "cross-lingual and expert cohorts may shift absolute levels of performance" (Section 6). The participants completed the evaluation in a single 60-minute session with a digital interface, a condition that differs substantially from how models are evaluated (unlimited time, no fatigue, no interface constraints).

The consequence. The headline finding—"non-expert humans achieve 80% mean accuracy while top MLLMs remain below 50%" (Section 6)—rests on a human accuracy estimate that is derived from sparse sampling of both participants and items. The statistical uncertainty around the 80% figure is not quantified: no confidence intervals are reported for the human accuracy estimate, and the paper does not provide standard errors or bootstrap intervals that would indicate whether the true population mean might be, say, 75% or 85%. More critically, the human accuracy per task varies substantially (from 68% on VRA to 88% on HPE), and this variation is based on only 30 responses per task (30 participants × 5 items each, aggregated across the 100-item task pools). A 20-percentage-point difference between tasks could reflect genuine difficulty differences, sampling noise, or both. The difficulty calibration itself—classifying items as Easy, Medium, or Hard based on exactly 5 annotator responses per item—means that a single annotator changing their answer on a borderline item would shift its difficulty classification, a brittleness that the paper does not quantify. The demographic narrowness (age 20–40, all participants from university and professional networks) means the baseline likely represents a cognitively high-performing subpopulation, which could inflate the human–model gap relative to a more representative sample. The paper acknowledges this in Section 6: "our human baseline uses non-expert adults in a single language setting; cross-lingual and expert cohorts may shift absolute levels of performance."

What evidence exists in the paper. The paper reports Fleiss' kappa of 0.71 across tasks (Appendix F) as a measure of inter-annotator agreement on difficulty classifications, described as "moderate to substantial agreement." This confirms that the difficulty categories capture stable signal rather than pure noise, but it does not quantify the precision of the aggregate human accuracy estimate. The paper does not report standard deviations, confidence intervals, or minimum/maximum participant accuracies for the human baseline—the human row in Table 2 reports single-point estimates per task with no error metrics.

Mitigation status. Not addressed. The paper does not discuss how increasing the human sample size or per-item coverage would affect the reliability of the 80% estimate or the difficulty calibrations. The paper hypothesizes in Section 6 that "relative gaps are likely to remain" with different cohorts but provides no evidence for this claim. The small human sample is a pragmatic constraint (recruiting and testing 30 participants for an hour each is non-trivial), but the paper does not adequately caveat the precision limits this imposes on the human–model comparison that forms the narrative backbone of the work.

The "Absence of Foundational Visuo-Cognitive Operations" Claim Is an Inference, Not a Direct Measurement

The assumption or constraint. The paper's strongest interpretive claim—that models "lack foundational visual-cognitive operations beyond merely struggling with complex instances" (Section 5)—is an inference drawn from a constellation of behavioral signatures: flat difficulty curves (Figures 3 and 26), uniform distractor selection (Appendix B.8), negative prompting effects on Transformation tasks (Figure 2), and attention–reasoning dissociations (Section 5). None of these measurements directly probes whether a model is internally performing mental rotation, fold simulation, or hierarchical pattern induction. The paper does not, for example, decode model representations to determine whether rotated shapes are encoded in a rotation-equivariant latent space, intervene on internal activations to causally test whether rotation-related computations occur, or measure reaction-time scaling with angular disparity (the classic behavioral signature of mental rotation in humans—Shepard and Metzler, 1971). The paper is transparent about this limitation in its "Risks of Anthropomorphism" statement (Section 6): "Cognitive-style performance can invite anthropomorphic interpretations, for e.g., ascribing 'mental rotation,' 'working memory,' or 'attention' in the human sense to models. This risks conflating functional success on a narrowly specified task with mechanistic equivalence to human cognition."

The consequence. The inference from "model performance does not exhibit the signatures we would expect if models were performing operation X" to "models do not perform operation X" is logically weaker than the paper's language sometimes implies. At least two alternative explanations are consistent with the observed data:

Degraded operations: Models do perform something approximating mental rotation, but so weakly or unreliably that it provides negligible benefit over random guessing. The flat difficulty curves could reflect a floor effect—if models are already near chance on Easy and Medium items, there is limited dynamic range for further degradation on Hard items. The paper reports that model accuracy on Hard items is in the 0.20–0.30 range (Figure 26), but it does not test whether this is statistically above the random baseline. If it is at or below random, the flatness claim is partially confounded with floor effects.
Different difficulty manifolds: Models might be sensitive to difficulty parameters, but the parameters that make tasks hard for humans (multi-axis rotation, deep hierarchy, complex fold sequences) may not be the parameters that make tasks hard for models. The models' difficulty sensitivity might be driven by different features—e.g., perceptual clutter, color confounds, or visual complexity metrics unrelated to spatial transformation demands. The consensus-based difficulty calibration ensures that the "Hard" bin contains items that humans find hard, but models might find some of these items easy and some "Easy" items hard along dimensions the paper does not measure. The paper does not report model accuracy stratified by the generative parameters (rotation angle, fold count, hierarchy depth) that were designed to control difficulty, which would directly test whether models show any parametric sensitivity at all.

The causal intervention experiment (Appendix B.7, Figure 15) attempted to address this by testing whether ablating attention heads disrupts performance, but the null result (no circuits found) is itself ambiguous: it could mean "no specialized circuits exist because the model does not perform the operation," or "the circuits exist but are distributed across many heads and resilient to single-head knockout," or "the circuits exist but reside in MLP layers rather than attention heads." The paper does not report whether multiple heads were ablated simultaneously or whether MLP interventions were attempted.

What evidence exists in the paper. The paper acknowledges the inferential gap explicitly in its Threats to Validity (Section 6): "we treat model outputs as behavioral signatures under controlled stimuli, avoid mentalistic language, and separate construct-level claims (what is measured) from implementation claims (how models compute)." The convergence of multiple independent behavioral signatures (difficulty-invariance, distractor uniformity, prompting dissociation, attention misalignment, reasoning-trace mis-binding) provides stronger evidence than any single signature alone, but the paper does not formally quantify how much each signature contributes to the inference or compute a composite likelihood ratio between the "no operations" and "degraded operations" hypotheses.

Mitigation status. Partially addressed through transparency. The paper's explicit separation of construct-level and implementation-level claims is a methodological strength, and the Risks of Anthropomorphism section demonstrates awareness of the inferential limitations. However, the paper's interpretive language in the main text (Section 5: "MLLMs fail uniformly across difficulty levels: they lack foundational visuo-cognitive operations rather than merely struggling with complexity") sometimes elides the distinction between "behavioral signature consistent with absence of X" and "absence of X demonstrated." The paper does not report the analyses that would directly address alternative explanations: parametric difficulty curves (accuracy vs. rotation angle directly), chance-level comparisons for Hard-item accuracy, and more systematic causal intervention protocols.

The Prompting Analysis Lacks a No-Reasoning Baseline and Statistical Significance Testing

The assumption or constraint. The paper's analysis of prompting strategy effects (Section 5, Figures 2, 8–9, Tables 16–19) uses Chain-of-Thought (CoT) prompting with explicit reasoning anchors ("Step 1: Identify the primitive shapes. Step 2: Detect how they move or fold. Step 3: Eliminate mismatched answers") as the baseline against which other strategies (Meta-Task, Step-by-Step, Hint, Elimination) are compared. The paper does not report a "direct answer" condition where models are instructed to simply select an option without any reasoning trace. It also does not report the statistical significance of the observed accuracy deltas, which are small in absolute magnitude (approximately ±1–2 percentage points).

The consequence. Without a direct-answer baseline, it is impossible to determine whether CoT itself helps, hurts, or has no effect relative to simply answering. If CoT impairs Transformation performance (e.g., because the verbal reasoning steps consume computational capacity that could have been used for visuospatial processing, or because they anchor the model to verbal descriptions that interfere with spatial simulation), then the reported negative deltas for alternative prompting strategies are relative to an already-depressed baseline. It is possible that Meta-Task prompting on Transformation tasks, which shows a negative delta relative to CoT, is actually better than a direct-answer baseline—but this cannot be determined from the reported data.

The absence of statistical significance testing for the prompting deltas means that the paper's strong interpretive claim—"prompting facilitates rule derivation yet fails to elicit procedural visuocognitive operations" (Section 5)—is supported by the direction of effects but not by formal evidence that those effects are statistically reliable. For tasks with 100 items, a 1–2 percentage point delta corresponds to 1–2 items changing from incorrect to correct (or vice versa). At sample sizes this small, the observed deltas could easily be sampling noise, particularly for the per-task breakdowns in Figure 9 where the sample sizes are smaller. The paper reports means and standard deviations in Tables 16–19 (e.g., "22.0±0.01" for Idefics-8B on VRA under CoT), but these error bars appear to reflect across-item variance within a single evaluation run, not the variance of the delta between two prompting conditions (which would require paired statistical tests).

What evidence exists in the paper. The paper reports that prompting effects are "dimension-dependent" (Section 5) and that "no transformation task shows systematic improvement under any alternative prompting strategy" (Appendix B.4), but these claims are based on observed numerical differences without formal inference. The PromptWizard optimization experiment (Appendix B.10, Table 9) found that even iteratively optimized prompts produce only "small but consistent gains" and concludes that "the primary bottleneck lies in the model's inherent reasoning limitations rather than surface level prompt design." However, this experiment was conducted on only one model (Qwen2.5-VL-7B) and four tasks, and the optimized-vs-baseline comparison also lacks significance testing.

Mitigation status. Not addressed. The paper does not report paired t-tests, bootstrap confidence intervals, or any other statistical framework for the prompting deltas. It does not include a direct-answer condition. The convergence of multiple prompting strategies on the same directional pattern (all alternative strategies impair Transformation, most improve Abstraction) provides some informal robustness, but the small effect sizes mean this convergence could also reflect a consistent small bias rather than a genuine differential sensitivity. The paper's claim about prompting's differential effects across ART dimensions is plausible and consistent with the broader narrative, but it is empirically weaker than the difficulty-invariance or attention-dissociation findings due to the absence of statistical rigor in the prompting analysis.

The Attention Analysis Does Not Establish Causality and Has Limited Spatial Granularity

The assumption or constraint. The paper's attention analysis (Section 5, Appendices B.1–B.2) examines cross-attention from the final decoder layer of a single model (the specific model used is not named in the main text, but the analysis references Qwen and LLaVA models in Appendix B.7, suggesting one or both were used). The analysis computes correlations between attention mass directed toward correct-option regions and task accuracy, and performs paired comparisons of attention distributions on correct vs. incorrect trials. The paper explicitly acknowledges in its Risks of Anthropomorphism (Section 6) that "ascribing 'attention' in the human sense to models" risks conflating functional patterns with mechanistic equivalence.

The consequence. The paper's framing of the attention findings—"models can localize relevant answer regions, yet they fail to reason reliably over this information—they identify where to look but not how to reason over what they see" (Section 5)—implies a causal model: attention to the correct region is a necessary precondition for successful reasoning, and the failure occurs in a subsequent "reasoning" step that operates over the attended information. However, the evidence is purely correlational. The positive point-biserial correlation (r_pb = 0.34) between OAS_correct and accuracy could reflect reverse causation—when models happen to produce the correct answer (through whatever mechanism), their attention is drawn to the correct option as a post-hoc consequence of that decision, rather than attention driving the decision. The paper does not perform attention-redirection experiments (e.g., forcing attention to the correct option via activation patching and measuring whether accuracy improves) that would establish causal direction.

The spatial granularity of the attention analysis is also limited. Attention is measured at the level of predefined bounding boxes around answer options, not at the level of specific visual features (e.g., individual cube faces, fold lines, symmetry axes) that are relevant to solving each task. A model might attend to the correct option region while fixating on task-irrelevant pixels within that region (e.g., the background color rather than the shape), and the OAS metric would not distinguish these cases. The Region-Aligned Attention (RAA) metric attempts to address this by measuring attention to specific referenced features, but with a mean RAA of 0.18–0.21 (Appendix B.2), the absolute grounding is weak regardless of correlation with accuracy.

The causal intervention experiment (Appendix B.7) that attempted to identify attention circuits for Mental Composition found no head whose ablation significantly disrupted performance. The paper interprets this as suggesting that "no distinct causal circuits could be isolated within the model using this approach." However, this null result does not distinguish between "the operation is not implemented in attention heads" and "the operation is implemented across many heads redundantly, such that single-head knockout is insufficient to disrupt it." The paper does not report the number of heads tested, whether multiple heads were ablated simultaneously, or whether MLP interventions were attempted.

What evidence exists in the paper. The paper acknowledges that "over-interpretation can also invert causality" (Section 6, Risks of Anthropomorphism) and states that it "separates construct-level claims (what is measured) from implementation claims (how models compute)." The robustness check using relative attention normalization (Appendix B.2) confirms that the attention findings are not artifacts of register-token confounds (Darcet et al., 2024), with the point-biserial correlation increasing from r_pb = 0.34 to 0.41 under relative attention. This strengthens the claim that attention patterns carry information about correctness, but does not address the causal direction question.

Mitigation status. Partially addressed through transparency about inferential limitations and through the relative-attention robustness check. However, the paper's interpretive language in the main text ("they identify where to look but not how to reason over what they see") presents the attention–reasoning dissociation as an established fact rather than as a plausible interpretation of correlational data. The absence of attention-redirection experiments and the limited spatial granularity of the OAS metric leave open alternative explanations that the paper does not discuss. The paper does not report whether the attention–accuracy correlation holds across all eight tasks or is driven by a subset, which would clarify whether "localization without reasoning" is a general phenomenon or task-specific.

7. Implications and Future Directions

How This Work Changes the Landscape

Mind's Eye shifts the conversation around MLLM evaluation from benchmarking what models can do to diagnosing how they fail. This is a methodological shift of substantial magnitude—not a paradigm revolution (the underlying models and training paradigms remain unchanged), but a reframing of what evaluation means for visuospatial reasoning. Prior to this work, the field's standard for spatial reasoning assessment was to report aggregate accuracy on tasks like VQA-spatial, CLEVR, or RAVEN, leaving unanswered the question of whether a model scoring 45% was failing because of perception gaps, reasoning deficits, or domain knowledge confounds. Mind's Eye provides the diagnostic infrastructure to disentangle these, introducing what amounts to a cognitive measurement framework for MLLMs modeled on the psychometric tradition that has guided human intelligence assessment for decades.

The paper's most consequential contribution to the evaluation landscape is the concept of behavioral signatures as evidence for (or against) specific cognitive operations. The three signatures the paper identifies—difficulty-invariant performance curves, uniform distractor selection, and asymmetric prompting effects across cognitive dimensions—are not merely observations about model behavior. They constitute a diagnostic protocol that can be applied to any putative cognitive capability: if a model genuinely performs operation X, its performance should be sensitive to the parameters that make X harder, its errors should cluster around failures of X (not random guessing), and interventions that scaffold X's prerequisites should help while interventions that bypass X should not. This protocol generalizes well beyond visuospatial reasoning to any domain where cognitive operations have known parametric signatures (causal reasoning, physical simulation, multi-step planning, mathematical proof).

The paper also resolves a latent contradiction in the multimodal reasoning literature that has gone largely unarticulated. On one hand, MLLMs demonstrate apparent spatial competence on natural-image benchmarks—describing object relationships, answering questions about relative positions, and even succeeding on diagrammatic reasoning tasks. On the other hand, targeted studies reveal brittleness to spatial perturbations and failures on counterfactual spatial reasoning. The resolution Mind's Eye offers is that surface spatial perception and internal spatial simulation are different capabilities, and current benchmarks fail to distinguish them. Models succeed on tasks solvable through perceptual matching and linguistic-pattern association; they fail on tasks requiring internal transformation of visual representations. This explains why a model can appear spatially competent on one benchmark and spatially incompetent on another: the benchmarks are measuring different things. Mind's Eye's ART taxonomy formalizes this distinction, providing a vocabulary for discussing which specific visuocognitive faculties are present or absent.

A secondary shift concerns what counts as evidence in MLLM evaluation. By grounding its difficulty calibration in human consensus (Fleiss' κ = 0.71), constructing distractors through systematic error templates validated against human error patterns, and mapping tasks to psychometric constructs via an explicit Q-matrix, the paper imports standards of psychometric validity—construct representation, nomothetic span, reliability quantification—that are standard in human cognitive assessment but rare in ML benchmarking. This raises the methodological bar for future cognitive benchmarks: it is no longer sufficient to collect hard problems and report accuracy. A benchmark that claims to measure "spatial reasoning" should specify which narrow abilities from which psychometric tradition it operationalizes, provide evidence that its items tap those abilities (not surface features or linguistic shortcuts), and demonstrate that performance on the benchmark exhibits the parametric signatures expected of the target construct.

Which research directions become more attractive, and which become less so:

More attractive: (1) Verifier and reasoning module design. The paper's finding that models cannot perform internal simulation—and that prompting only helps when tasks reduce to symbolic rule derivation—suggests that improvements to vision encoders or language models alone will not close the visuospatial gap. Research on transformation-aware representations (e.g., rotation-equivariant architectures, spatial working memory buffers, or modular simulation components that can be called by the language model) becomes more compelling because the paper provides evidence that current monolithic architectures lack the necessary computational mechanisms. (2) Cognitive diagnostics as an evaluation methodology. The paper demonstrates that multiple behavioral signatures (difficulty curves, distractor selection, prompting dissociation, attention alignment) converge on a coherent picture of model limitations. This multi-signature approach—rather than single-number accuracy—becomes a template for evaluating other cognitive capabilities. (3) Psychometric grounding in benchmark design. The paper's use of a Q-matrix, consensus-based difficulty calibration, and construct-level attribution provides a replicable methodology that other benchmark designers can adopt, moving the field toward benchmarks that support substantive claims about model capabilities rather than leaderboard rankings.

Less attractive: (1) Scaling as a universal solution. The paper's finding that performance does not scale monotonically with model size (Pearson's r ≈ 0.62) and that even the largest models show flat difficulty curves suggests that scaling current architectures is unlikely to produce genuine visuospatial reasoning. The Qwen family's non-monotonic scaling—the 32B variant underperforms the 7B variant on HPE (18.3% vs. 24.2%) and DSC (22.6% vs. 30.4%)—is a concrete warning that larger models can regress on specific cognitive operations even as they improve on others. (2) Prompt engineering as a panacea for reasoning deficits. The paper's finding that optimized prompts (via PromptWizard) produce only small gains (<0.10 absolute) and that all alternative prompting strategies impair Transformation performance suggests that prompt refinement has limited leverage over procedural visuospatial operations. The dissociation between Abstraction tasks (helped by scaffolding) and Transformation tasks (impaired by scaffolding) reframes prompting not as a generic reasoning enhancer but as an intervention whose effects depend on whether the target operation is symbolically scaffoldable or procedurally implemented.

Follow-Up Research This Work Enables

Parametric difficulty curves to quantify the absence of mental simulation. The paper's central inferential move—from flat difficulty curves to "no foundational visuo-cognitive operations"—would be strengthened by directly measuring model accuracy as a function of the generative parameters that control task difficulty. For Mental Transformation, instead of consensus-based Easy/Medium/Hard bins, evaluate models on items with rotation angles systematically varied in 15° increments from 0° to 180°, separately for single-axis and multi-axis rotations. The classic human signature of mental rotation is a linear increase in reaction time with angular disparity (Shepard and Metzler, 1971). If models show zero accuracy variation across rotation angles—maintaining, say, ~30% regardless of whether the rotation is 15° or 165°—this constitutes stronger evidence for the "no rotation" hypothesis than the consensus-based binning, because it eliminates the alternative explanation that models are sensitive to difficulty parameters but the consensus bins fail to capture them. The extended 20,000-item training set provided by Mind's Eye makes this experiment feasible at scale without additional generation cost.

Cross-validation against established psychometric instruments. The paper's central claim—that Mind's Eye measures the same constructs as the Vandenberg & Kuse MRT, CogAT Paper Folding, Bongard problems, and other established instruments—is a design intention rather than an empirically validated fact. A strong follow-up would administer both Mind's Eye and the original instruments (or close digital reproductions) to a common set of 50–100 human participants and report the correlations between Mind's Eye task scores and the source instrument scores. If the correlations are high (e.g., r > 0.7 for Mental Transformation vs. Vandenberg & Kuse MRT), the claim of construct continuity is supported. If the correlations are low, Mind's Eye may be measuring something different from the instruments it adapts—perhaps because the synthetic polycube stimuli differ substantially from the line drawings in the original MRT, or because the programmatic generation introduces difficulty factors not present in the originals. A parallel model evaluation on both Mind's Eye and the source instruments would reveal whether the performance patterns (flat difficulty curves, low absolute accuracy) replicate in the original formats or are specific to Mind's Eye's implementation.

Attention-redirection causal experiments. The paper's attention analysis establishes a correlation between OAS_correct and accuracy (r_pb = 0.34–0.41) but cannot determine whether attention to the correct option causes correct reasoning or merely accompanies it. A causal follow-up would use activation patching or attention manipulation: for items where the model initially answers incorrectly, force the model's attention distribution during reasoning tokens to match the attention distribution observed on correct trials (higher mass on the correct option, lower on distractors) and measure whether accuracy improves. If forcing correct option attention does not improve accuracy, the attention–performance correlation is epiphenomenal and the bottleneck truly lies downstream of visual localization. If forcing attention does improve accuracy, then models possess some reasoning capability that is bottlenecked by unreliable attention guidance—a very different diagnosis with different implications for interventions (improving attention mechanisms vs. building transformation modules). The paper's existing infrastructure for computing OAS and defining option spatial regions makes this experiment straightforward to implement with standard activation patching libraries.

Combined search and simulation architectures. The paper demonstrates that prompting scaffolds symbolic rule derivation (Abstraction) but fails to elicit procedural spatial simulation (Transformation). This dissociation suggests a natural architectural experiment: pair an MLLM with an external simulation module (e.g., a differentiable renderer or a physics engine that can perform rotation, folding, and unfolding operations on 3D mesh representations) and test whether the MLLM can learn to call the simulator appropriately. Concretely, provide the model with a tool-use interface where it can submit a shape description and a rotation specification and receive back the rotated visual output, then evaluate on Mental Transformation. If accuracy jumps from ~30% to >70%, the deficit is not in the model's ability to recognize when simulation is needed and what simulation to request but in its ability to perform the simulation internally. If accuracy remains low even with tool access, the deficit includes a meta-cognitive failure—the model cannot correctly formulate simulation queries. The Mind's Eye benchmark, with its programmatic generation, provides an ideal testbed because the ground-truth transformations are known and can be perfectly simulated, eliminating simulator error as a confound.

Difficulty estimation from model-internal signals. The paper's difficulty-stratified analysis relies on expensive human consensus labels (5 annotators per item) and acknowledges that "estimating difficulty in this way still incurs additional computation cost" (Section 3.2). A practical follow-up would test whether cheap model-internal signals—the model's own confidence (maximum softmax probability or logprob of the selected answer), the entropy of the answer distribution across multiple samples, or the variance of attention patterns—correlate with the consensus-based difficulty labels. If a model's confidence on Mental Transformation items correlates at r > 0.6 with whether humans find those items Easy/Medium/Hard, then models possess implicit difficulty awareness that could be exploited for adaptive compute allocation even if they cannot solve the hard items. If the correlation is near zero, not only can models not perform the operations, they also lack metacognitive awareness of their own limitations—a finding with direct implications for safe deployment. This experiment requires no new data collection beyond what the paper already provides (human difficulty labels and model outputs).

Layer-wise and cross-model attention analysis for circuit discovery. The paper's attention analysis uses only the final decoder layer and its causal intervention (Appendix B.7) tests only single attention head knockout. A more systematic mechanistic follow-up would: (1) compute OAS_correct and attention–accuracy correlations at every decoder layer for multiple models, identifying whether visual grounding happens primarily in early layers (suggesting perceptual encoding), middle layers (suggesting cross-modal integration), or late layers (suggesting decision-stage alignment); (2) perform combinatorial head ablation—knocking out groups of 2, 3, or 4 attention heads simultaneously—to test whether visuospatial reasoning relies on distributed circuits that are resilient to single-head perturbation; and (3) apply the same analysis to the MLP layers, which the paper did not test but which may implement the transformation computations if they exist. A positive finding—identifying a specific set of attention heads and MLP layers whose joint ablation significantly impairs Mental Transformation but not Visual Relation Abstraction—would be first evidence of a functional specialization for visuospatial operations in MLLMs. A thoroughly negative finding across layers, head groups, and model families would strengthen the paper's "no operations" interpretation by ruling out the "distributed circuit" alternative explanation.

Practical Applications and Downstream Use Cases

Screening and selection of MLLMs for deployment in spatial reasoning applications. Organizations deploying MLLMs for tasks that require genuine spatial understanding—architectural design assistants, robotic manipulation planning, medical image interpretation, STEM tutoring systems—can use Mind's Eye as a diagnostic filter. Rather than relying on general-purpose benchmark scores (which conflate spatial reasoning with perceptual matching and linguistic skill), a deployment team can evaluate candidate models on the eight Mind's Eye tasks and examine not just aggregate accuracy but the specific behavioral signatures: Does the model show difficulty sensitivity on Transformation tasks? Does its attention align with correct options on correct trials? Do its errors show systematic distractor bias or random guessing? A model that achieves 40% on Mental Composition with flat difficulty curves and uniform distractor errors is fundamentally unsafe for deployment in a setting where incorrect spatial reasoning could cause harm, regardless of its performance on MMBench or MMMU. The paper's diagnostic distractor taxonomy and attention analysis methodology provide the evaluation protocol; the specific thresholds for "safe deployment" would need to be established through domain-specific risk assessment, but the diagnostic infrastructure is now available.

Curriculum design for visuospatial fine-tuning. The extended 20,000-item training set provides a resource for fine-tuning MLLMs on visuospatial reasoning tasks with controlled difficulty progression. A training curriculum could be designed where models are first fine-tuned on Easy items (where the correct answer requires minimal transformation—single-axis rotation by small angles, simple fold sequences, shallow hierarchies), then progressively exposed to Medium and Hard items. The paper's parametric control enables systematic curriculum design: rotation angle can be increased monotonically, fold count incremented, hierarchy depth deepened, with performance monitored at each stage. If models trained on such a curriculum develop any sensitivity to transformation parameters (i.e., their accuracy begins to degrade when rotation angle increases, rather than remaining flat), this would be evidence that the right training data can induce emergent transformation capabilities—a finding that would qualify the paper's "no operations" conclusion and point toward training-based solutions. If even after curriculum fine-tuning, performance remains flat across difficulty and models continue to show uniform distractor selection, the case for architectural limitations becomes substantially stronger.

Automated generation of diagnostic test suites for continual model monitoring. The programmatic generation framework underlying Mind's Eye enables the creation of perpetually fresh evaluation items with no annotation cost. For organizations that deploy MLLMs and need to monitor for capability regression after model updates, the generator can produce new items with specified difficulty parameters on demand, ensuring that evaluation does not suffer from data contamination (models trained on leaked benchmark items). A monitoring pipeline could be set up where, after each model update (fine-tuning run, RLHF iteration, architecture change), a fresh set of 100 Mental Transformation items with controlled rotation parameters is generated and evaluated, with the resulting difficulty curve compared to the pre-update baseline. A significant shift—e.g., the difficulty curve changing from flat to graded, or accuracy dropping on specific rotation axes—would alert engineers to unexpected capability changes. The paper's release of generators, seeds, and scoring code makes this pipeline implementable with minimal additional infrastructure.

Human–AI collaboration templates for visuospatial tasks. The paper's finding that humans and models fail differently—humans show systematic distractor bias (p = 0.014 for Paper Folding) while models show uniform errors—suggests a potential complementarity. When humans err on Paper Folding, they tend to make parity errors and reflection confusions that the diagnostic distractors capture; when models err, their failures are less predictable but perhaps less prone to the specific systematic biases that trap humans. A collaboration template could be designed where, for high-stakes spatial reasoning tasks, both a human and a model independently answer, and cases of disagreement are flagged for expert review. If model errors are genuinely orthogonal to human error patterns—a hypothesis the paper's data partially supports but does not fully test—then human–model agreement would provide higher confidence than either alone, and disagreement would efficiently identify items requiring deeper scrutiny. The paper's per-item human response data (5 annotators per item for the calibration set) and per-model response data provide the raw material for testing this complementarity hypothesis.

When to Prefer This Method Over Alternatives

The paper positions Mind's Eye as a diagnostic instrument, not as a general-purpose benchmark or a training method, so the "this method vs. alternatives" framing applies to evaluation methodology rather than to model selection.

Prefer Mind's Eye-style diagnostic evaluation when:

The goal is to determine whether a model possesses a specific cognitive capability (mental rotation, fold simulation, hierarchical pattern induction) as distinct from surface perception or linguistic-pattern exploitation. The paper demonstrates that general-purpose benchmarks cannot make this distinction because they conflate multiple sources of performance variance. Use Mind's Eye when the question is not "how good is this model?" but "what kind of visual intelligence does this model have, and what specific operations is it incapable of performing?"
You need to attribute model failures to specific error types for the purpose of guiding model improvement. If a model scores 30% on spatial reasoning, knowing that its errors are uniformly distributed across all wrong options (no systematic distractor bias) versus concentrated on parity errors (significant distractor bias) leads to fundamentally different diagnoses: the former suggests the model lacks the target operation entirely, the latter suggests it performs something approximating the operation but with a specific bug that could be targeted through training data or architecture modifications. The paper's diagnostic distractor taxonomy supports this fine-grained attribution; standard benchmarks with random distractors do not.
The application domain requires confidence that model success reflects genuine understanding rather than heuristic matching. In safety-critical spatial reasoning domains (medical image interpretation, autonomous navigation, structural engineering), a model that achieves 80% accuracy through perceptual pattern matching is more dangerous than a model that achieves 60% accuracy through genuine spatial simulation, because the heuristic model's failures are unpredictable and unprincipled. Mind's Eye's behavioral signatures—difficulty sensitivity, distractor selection patterns, attention grounding—provide evidence about which type of computation is occurring.

Prefer traditional broad-coverage benchmarks (MMBench, MMMU, SEED-Bench) when:

The goal is to rank models by general multimodal capability for leaderboard purposes. Mind's Eye is deliberately narrow and deep (8 tasks probing specific cognitive operations) rather than broad and shallow. A model that excels on Mind's Eye may be mediocre at OCR, knowledge-based VQA, or diagram understanding. The benchmarks serve complementary purposes: broad benchmarks for coverage, Mind's Eye for depth on a specific capability class.
You need to evaluate on naturalistic stimuli that reflect deployment distributions. Mind's Eye's synthetic abstract stimuli deliberately eliminate real-world knowledge confounds, but this also means they do not represent the visual complexity or semantic content of natural images. If your application involves photographs, natural scenes, or domain-specific visual content, Mind's Eye should be used as a supplementary diagnostic, not as the primary evaluation.

Prefer cognitively motivated but non-diagnostic benchmarks (Bongard-LOGO, RAVEN, ARC-AGI) when:

The primary interest is in few-shot or zero-shot rule induction from visual exemplars, without the need to isolate specific error types or measure parametric difficulty sensitivity. Bongard-LOGO and ARC-AGI probe abstraction and analogy in ways that partially overlap with Mind's Eye's Abstraction and Relation dimensions, but they do not provide the distractor taxonomy, Q-matrix, or difficulty calibration infrastructure that enables fine-grained failure attribution. Use Mind's Eye when you need the diagnostic detail; use the alternatives when you need broader coverage of abstraction tasks with less measurement overhead.