KiVA: Kid-inspired Visual Analogies for Testing Large Multimodal Models

🎯 Pitch

State-of-the-art LMMs like GPT-o1 fail to match even a three-year-old's ability to reason about simple visual transformations like rotation or adding objects. While models can often name what changed, they collapse when asked to quantify how it changed or to apply the rule to a new image, revealing a fundamental gap in abstract, relational understanding that text-and-2D-image training cannot fill.

1. Executive Summary

This paper introduces KiVA (Kid-inspired Visual Analogies), a benchmark of 4,300 visual transformations of everyday objects designed to test visual analogical reasoning—the ability to infer an abstract rule from one image and apply it to another—in large multimodal models (GPT-o1, GPT-4V, LLaVA-1.5, MANTIS) and compare them to children ages three to five and to adults. The evaluation decomposes analogical reasoning into three stages: verbal classification of what changed (e.g., identifying a transformation as a color or number change), verbal specification of how it changed (e.g., "objects become bigger"), and visual extrapolation of that rule to a novel object. While GPT-o1, the strongest tested model, nears adult performance in the color domain of the more challenging KiVA-adults extension, all models struggle with quantifying the "how" and extrapolating the rule to new objects—particularly in spatial and numerical domains like rotation, reflection, and number—establishing that state-of-the-art LMMs do not match even the analogical reasoning capabilities of a three-year-old child when transformations require deeper understanding of extrinsic spatial properties and numerical operations.

2. Context and Motivation

The Core Problem: LMMs Can "See" but Cannot Reason About Visual Analogies

The fundamental question this paper tackles is: can large multimodal models perform the kind of visual analogical reasoning that even young children do effortlessly? A visual analogy involves detecting an abstract rule from one image transformation—for instance, noticing that an object has rotated 90 degrees or changed from red to blue—and then applying that same relational rule to predict how a completely different object would transform under the same principle. This is not merely object recognition or property detection; it requires abstracting a relational structure from one instance and mapping it onto a novel target while preserving the underlying rule (Gentner, 1983).

The paper identifies a specific and consequential gap: while LMMs have demonstrated impressive capabilities in object recognition, image captioning, and visual question answering, no existing benchmark tests whether these models can perform the basic visual analogical reasoning that developmental psychology has established is present in children as young as three years old. This is not a gap of incremental capability—it is a gap at the foundation of flexible, adaptive visual intelligence. As the paper argues in Section 1, analogical reasoning is "what enables us to be flexible, adaptive and robust learners across a wide variety of settings, finding meaning in patterns and making out-of-distribution generalizations." Without it, models remain brittle and tethered to their training distributions rather than genuinely understanding visual relationships.

Why This Matters: Practical and Theoretical Significance

The importance of this problem operates on multiple levels:

Theoretical significance for AI cognition. Analogical reasoning has been described as "a hallmark of human intelligence and learning" (Section 1). The paper explicitly invokes this framing, citing Holyoak (2012), Gentner (1983), and Mitchell (2021) to establish that the capacity to draw systematic parallels between situations is not a peripheral skill but central to what makes human cognition generalizable. If LMMs cannot perform even the basic visual analogies that three-year-olds solve, this exposes a fundamental limitation in how these models represent and manipulate relational knowledge. The paper frames this as a test of whether models have truly abstracted relational structures from their training data or are merely pattern-matching on surface features.

Practical implications for real-world deployment. The five transformation domains tested—color, size, number, rotation, and reflection—are not arbitrary. They correspond to visual skills essential for interacting with the physical world: detecting changes in object state (color), planning actions based on spatial relationships (size, rotation), distinguishing chiral objects like left and right shoes (reflection), and tracking quantities (number). A model that cannot reliably perform these analogies cannot be trusted in embodied applications, robotic manipulation, or any setting where understanding how objects relate and transform is necessary. As the paper notes (Section 3.2), these transformations are "crucial for object and scene recognition, scene segmentation, and detecting significant changes in the environment."

A principled developmental comparison baseline. Rather than comparing models against an arbitrary human performance ceiling, the paper anchors its evaluation in a cognitively meaningful reference point: what children aged three to five can do, and what adults can do. This is significant because developmental psychology provides a well-established trajectory of when specific visual reasoning abilities emerge. Color and size perception emerge early in infancy (Ross-sheehy et al., 2003; Day & McKenzie, 1981). Rotation and reflection understanding develops between ages three and five (Frick et al., 2013; Quaiser-Pohl, 2003). Small-number reasoning emerges in early childhood (Cherian et al., 2023; Levine et al., 1992). By benchmarking against this trajectory, the paper can pinpoint precisely which abilities LMMs lack rather than making blanket statements about model inadequacy.

Where Prior Benchmarks Fall Short

The paper identifies three critical limitations in existing visual analogy and reasoning benchmarks (Section 2, Figure 2):

Limitation 1: Abstraction divorced from real-world relevance. Prior benchmarks like the Abstraction and Reasoning Corpus (ARC; Chollet, 2019), ConceptARC (Mitchell et al., 2023; Moskvichev et al., 2023), and Raven's Progressive Matrices variants (Huang et al., 2024; Ahrabian et al., 2024) rely on abstract geometric shapes, grids, and pixel-level transformations. As the paper argues:

"This abstraction of stimuli neither aligns with the training data of large multimodal models nor effectively mimics the complexity and variability found in everyday visual tasks, making it less suitable for assessing how well AI models can perform analogical reasoning in practical contexts."

The issue is twofold. First, LMMs are trained predominantly on natural images of real-world objects—testing them on abstract grids measures something different from their trained capabilities and may underestimate transfer. Second, humans themselves do not learn analogical reasoning through abstract grid puzzles in early development; they learn by observing transformations of familiar objects in grounded contexts (Christie & Gentner, 2010; Goddu et al., 2020). A benchmark that uses abstract stimuli may not be testing the same cognitive faculty.

Limitation 2: Testing only the final answer, not the reasoning process. Existing evaluations ask models to select the correct extrapolation but provide no insight into where in the reasoning chain the model fails. Does the model detect that a transformation involves rotation but confuse 90° with 180°? Does it correctly identify the transformation rule but fail to apply it to a novel object? Without decomposing the task, we cannot distinguish between perception failures (not seeing what changed), specification failures (not articulating the rule), and mapping failures (not applying the rule to a new target). The paper explicitly contrasts this with Sternberg's (1977) component processes model of analogical reasoning, which decomposes the task into encoding, inference, mapping, and application—and uses this framework to design the three-stage evaluation.

Limitation 3: Excluding basic visual cognition that children possess. Prior benchmarks focus on advanced pattern recognition and complex logical sequencing. As the paper notes:

"The transformations examined involve conjunctions of visual concepts such as extracting and transposing pixels according to some arbitrary rule, which do not tap into basic visual cognition. Humans do not require the ability to solve these specific tasks to function effectively in their daily lives nor to demonstrate their capacity for visual analogical reasoning."

This is a pointed critique: existing benchmarks may be measuring something specialized rather than general visual reasoning capacity. A model that fails ARC might still possess the basic analogical abilities that matter for real-world tasks. Conversely, a model that passes some RPM-based tests might still lack the fundamental spatial and numerical reasoning that children demonstrate. The paper's solution is to create a benchmark where the "floor" is explicitly set by what developing humans can do.

Limitation 4: Missing developmental baselines. No prior visual reasoning benchmark systematically compares models against young children. This means we lack a calibrated understanding of model capabilities relative to human cognitive development. A model scoring 70% on a benchmark tells us little about whether that 70% represents three-year-old, five-year-old, or adult-level reasoning. By including both child and adult participants, KiVA provides a developmental ruler against which model performance can be measured in terms that have established psychological meaning.

How This Paper Positions Itself

The paper positions KiVA as filling the gap between overly simplistic property-detection benchmarks and overly abstract puzzle-solving benchmarks, creating an evaluation that is simultaneously grounded, developmentally meaningful, and diagnostic.

A grounded, real-world approach to analogical reasoning. By using 3D models of household objects from Downs et al. (2022) and infant-familiar toys from Stojanov et al. (2021), the benchmark ensures that stimuli align with both human visual experience and LMM training distributions. The transformations tested—color changes, size changes, rotation, reflection, number addition/subtraction—are the same types of changes that Goddu et al. (2020) used to demonstrate analogical reasoning in preschoolers. This is not an arbitrary selection but a direct translation of a validated developmental psychology paradigm into a computational benchmark.

A diagnostic three-stage evaluation inspired by cognitive science. Rather than following the standard approach of presenting a puzzle and measuring end-to-end accuracy, the paper decomposes analogical reasoning into the component processes identified by Sternberg (1977): encoding the transformation (verbal classification), inferring the rule (verbal specification), and applying the rule to a novel target (visual extrapolation). Critically, the visual extrapolation step can be attempted regardless of performance on the verbal steps, preserving the core nonverbal analogy task that has been the focus of prior benchmarks while adding diagnostic granularity. The paper writes:

"KiVA retains the core nonverbal extrapolation task (last step) from previous benchmarks and the verbal questions do not replace the core nonverbal tasks."

This means KiVA can serve both as a direct comparison to prior work (using only visual extrapolation accuracy) and as a richer diagnostic tool (using the full three-stage pipeline).

Two-tiered difficulty for developmental benchmarking. The paper introduces both KiVA (solvable by three-year-olds) and KiVA-adults (solvable by adults but not children under five) to create a developmental gradient. KiVA-adults adds more abstract generalization demands—the starting values of test objects differ from those in the given transformations, requiring deeper rule abstraction—and includes more varied transformations (stretching, multiplication/division, 45°/135° rotations). This tiered structure means the benchmark can assess whether models progress along a human-like developmental trajectory or show a qualitatively different pattern of strengths and weaknesses.

An explicit challenge to the "scale is all you need" assumption. By demonstrating that GPT-o1—a model with chain-of-thought reasoning capabilities—still falls short of three-year-old performance in number and reflection domains (Figure 1, bottom), the paper provides evidence that simply scaling models and adding reasoning mechanisms is insufficient. As Section 4.2 argues, the difficulty lies not merely in perception or language but in the fundamental challenge of "mapping a transformation from a source object to a target while preserving relational structure." This is a claim about the architecture of intelligence, not just about current model limitations—and it motivates research directions beyond simply training on more data.

Connecting model and human difficulty patterns. The paper presents a novel finding that GPT-o1's error patterns correlate with children's errors and adult response times (Figure 6), suggesting that what is cognitively demanding for humans is also computationally challenging for models. This convergence—despite fundamentally different architectures and training regimes—suggests that the difficulty of certain visual analogies may reflect intrinsic properties of the problems themselves rather than peculiarities of human or machine learning. This positions KiVA not only as a benchmark but as a tool for investigating the nature of visual analogical reasoning across biological and artificial systems.

3. Technical Approach

3.1 Reader Orientation

This paper constructs a diagnostic evaluation framework — not a model or a training method — for systematically measuring visual analogical reasoning in both large multimodal models and humans. The problem it solves is that prior visual reasoning benchmarks test only end-to-end accuracy on abstract puzzles, providing no insight into where reasoning breaks down, while also lacking grounding in the real-world visual transformations that even young children can reason about. The solution takes the form of a curated dataset of 4,300 real-world object transformations paired with a three-stage query protocol that probes the component processes of analogical reasoning — classification, specification, and extrapolation — enabling fine-grained comparison against developmental baselines.

3.2 Big-Picture Architecture (Diagram in Words)

The KiVA framework has four major components, arranged as a pipeline from stimulus construction through evaluation:

Stimulus Generation Engine — A procedural pipeline that takes 3D object models from two existing datasets (household objects and infant-familiar toys) and applies parameterized visual transformations across five domains (color, size, number, rotation, reflection) to produce image pairs showing systematic changes. This engine also generates three candidate extrapolations per trial (one correct, two distractors) by applying the same transformation rule to a novel object.
Difficulty Tier System — Two calibrated difficulty levels: KiVA (designed to be solvable by children aged three to five, with aligned starting values between given and test transformations) and KiVA-adults (requiring deeper abstraction where test objects start from different values than the demonstrated transformation, and including more varied transformation subdomains). Children under five perform at chance on KiVA-adults, establishing it as a developmentally-gated harder tier.
Three-Stage Query Protocol — Every trial proceeds through three sequential questions, each building on the previous: (Stage 1) Verbal Classification identifies what visual domain changed, (Stage 2) Verbal Specification identifies how it changed within that domain, and (Stage 3) Visual Extrapolation requires applying the inferred rule to predict a novel object's transformation. Critically, extrapolation can be attempted regardless of performance on verbal stages, preserving the core nonverbal analogy task.
Evaluation Harness — A unified testing infrastructure that administers identical trials to LMMs (via API calls with controlled prompting), to human adults (via Prolific with JsPsych), and to children (via in-person experiments with experimenter-read instructions). All participants see the same images and answer the same multiple-choice questions, with answers scored automatically for direct comparison.

Information flows as follows: a 3D object model enters the generation engine → a specific transformation rule is applied to produce a "given" image pair → three test candidate pairs are generated (one correct extrapolation of the same rule to a new object, two distractor transformations) → the complete trial (given pair + three candidates) is formatted for the target participant type → the three-stage queries are administered sequentially → responses are scored and aggregated by transformation domain and question type.

3.3 Roadmap for the Deep Dive

First, the stimulus generation methodology — how objects are selected, what transformation operations are applied, and how distractor candidates are constructed — since the benchmark's validity depends entirely on the controlled, systematic nature of these transformations.
Second, the five transformation domains and their subdomains — what each domain tests perceptually and cognitively, the specific parameterizations used, and why these particular transformations were chosen based on the developmental psychology literature.
Third, the three-stage query protocol — the exact wording of each question, the multiple-choice structure, how correctness propagates between stages, and why this decomposition maps onto Sternberg's (1977) component processes model of analogical reasoning.
Fourth, the difficulty tier design (KiVA vs. KiVA-adults) — the specific differences in generalization demands and transformation variety that make KiVA-adults unsolvable by young children, creating a developmental gradient for benchmarking.
Fifth, the human and model evaluation procedures — how trials are administered to children, adults, and LMMs, including prompting strategies, randomization, scoring, and the specific hyperparameters used for model inference.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmark construction and evaluation paper whose core idea is that visual analogical reasoning can be diagnostically decomposed into three component processes — detecting what changed, specifying how it changed, and extrapolating the rule to a novel object — and that testing on grounded, real-world transformations with developmental baselines reveals specific failure modes in LMMs that prior abstract benchmarks miss.

Stimulus Generation: Object Sourcing and Transformation Pipeline

The benchmark's stimuli are procedurally generated from 3D object models rather than collected from natural images. This design choice is deliberate and has several important consequences: it ensures controlled, systematic transformations where the exact rule is known by construction, it enables infinite expansion of the benchmark by applying transformations to any new object model, and it eliminates the confounding variables present in natural images (lighting variation, background clutter, viewpoint changes) that would make it unclear whether a model's failure is due to analogical reasoning limitations or low-level perceptual difficulties.

Object sourcing from two established datasets. The paper draws from two complementary 3D model sources (Section 3.2):

Google Scanned Objects (Downs et al., 2022): A dataset of high-quality 3D scans of common household items. These objects — furniture, kitchenware, tools, electronics — represent the kind of everyday items that adults and LMMs encounter in real-world visual data. Using this source ensures that the stimuli align with the natural image distributions on which both computer vision models and humans are trained.
Infant-familiar objects (Stojanov et al., 2021): A dataset specifically curated to include toys and objects commonly encountered by infants and young children. This source ensures that the objects themselves are developmentally appropriate — a three-year-old tested on KiVA sees transformations of familiar items like toy blocks, stuffed animals, and simple shapes, not abstract laboratory stimuli.

The paper does not specify the exact number of distinct object models used from each source, but the scale is implied by the total trial counts: 1,400 object transformations in KiVA and 2,900 in KiVA-adults, with 100 transformations per subdomain (Section 3.2).

Transformation operations as parameterized image manipulations. Each transformation domain is implemented as a specific image-processing operation with fixed parameters. The generation pipeline takes a rendered 2D view of a 3D object model, applies the transformation operation to produce a "before" and "after" pair, and then applies the same operation (with the same parameters) to a different object model to produce the three candidate extrapolation pairs. The key properties of this generation approach are:

Deterministic rules: Every transformation is defined by an explicit mathematical operation (e.g., scaling by a factor of 2, rotating by 90 degrees clockwise, changing the RGB color to red). There is no ambiguity about what constitutes the "correct" answer — it is defined by construction.
Object independence: Because the same parameterized rule is applied to different objects, the analogical mapping requires abstracting the rule from the specific visual features of the source object. A size transformation that doubles both height and width of a coffee mug must be recognized as the same rule when applied to a toy block, even though the absolute pixel changes differ.
Distractor construction: For each trial, the three candidate extrapolations (labeled A, B, C) include one correct application of the rule and two distractors. The paper states (Figure 3 caption) that distractors are other plausible transformations within the same or different visual domains. For instance, if the given transformation is a 90° clockwise rotation, the distractors might be a 90° counterclockwise rotation and a 180° rotation, or a size change and a color change applied to the same test object. The specific distractor generation algorithm is not detailed in the main text, but the multiple-choice format with randomized option labels ensures that models cannot exploit label-position biases.

Output format for different participant types. The final stimulus for each trial is rendered differently depending on whether it will be presented to LMMs or humans:

For LLaVA-1.5, which can only process a single image, the given transformation and three candidate extrapolations are stitched into a composite image (examples shown in Appendix A.1, Figure 10).
For MANTIS and the GPT models (which support multi-image input), the given transformation and each candidate extrapolation are presented as four separate images. The paper explicitly references Campbell et al. (2025) in adopting this approach "to reduce the chance of visual binding errors" (Section 4).
For human participants, the stimuli are rendered within a JsPsych web interface (De Leeuw, 2015) with standardized image dimensions, though the exact rendering parameters are not specified.

The Five Transformation Domains: Developmental Rationale and Parameterization

The paper selects five visual transformation domains, each motivated by specific findings in the developmental psychology literature demonstrating that children can reason about these types of changes at specific ages. For each domain, the paper defines specific subdomains (the "how" of the transformation) that serve as the multiple-choice options in the verbal specification stage.

Color changes. Color perception emerges very early in human development — infants as young as one month can discriminate colors (Milewski & Siqueland, 1975), and visual short-term memory for color develops rapidly in the first year (Ross-sheehy et al., 2003). The paper cites Maule et al. (2023) to motivate the ecological importance of color change detection: noticing that a fruit has changed color can signal ripeness or spoilage, and detecting color changes in the environment can indicate hazards.

Transformation rule: Objects change to a single target color. The rendered object's RGB values are modified such that the object uniformly takes on the specified color while preserving texture and shading cues where possible (though the exact rendering method is not detailed).
KiVA subdomains: Three target colors — red, green, and blue. These are the primary additive colors and are developmentally among the first color categories children learn.
KiVA-adults additions: Two additional colors — yellow and grey. This increases the number of possible specifications from 3 to 5, making the verbal specification task harder by increasing the distractor space. Yellow is a secondary color (red + green) that is perceptually distinct but learned slightly later in development. Grey is an achromatic color that tests whether models can identify the absence of hue as a color category.
Domain-level classification options (verbal classification): When asked "what changed," the correct answer is "change in color of objects." Distractor options are sampled from the other four transformation domains (size, number, rotation, reflection), plus "no change" and "doesn't apply" are always included.

Size changes. Size constancy — the ability to perceive an object's true size despite changes in retinal image size — emerges in infancy (Day & McKenzie, 1981; Slater et al., 1990). The paper cites Giudice (2018) to motivate the ecological importance of size perception for navigation, object manipulation, and obstacle avoidance.

Transformation rule: Objects uniformly scale in both height and width by a specified factor. The scaling is isotropic (preserving aspect ratio) in KiVA, meaning the object's shape proportions remain constant.
KiVA subdomains: Two scaling operations — objects become bigger (scaled by a factor of 2 in both dimensions) and objects become smaller (scaled by a factor of 0.5, i.e., halved in both dimensions). These correspond to the transformations used by Goddu et al. (2020) in their developmental studies of preschoolers.
KiVA-adults additions: Object stretching — independent scaling of height or width by a factor of 2 while keeping the other dimension constant. This introduces an anisotropic transformation that changes the object's shape proportions. Stretching is perceptually and cognitively distinct from uniform scaling because it alters the aspect ratio, requiring the reasoner to notice that only one dimension changed rather than both.
Domain-level classification options: "Change in size of objects" is the correct classification. The verbal specification options in KiVA are "objects become smaller," "objects become bigger," "no change," and "doesn't apply." KiVA-adults adds "objects become stretched (taller or wider)."

Number changes. Numerical reasoning, including small-number addition and subtraction, is present in early childhood (Levine et al., 1992; Cherian et al., 2023). The paper cites Chattopadhyay et al. (2017) and Cohen (2005) to motivate the importance of quantity monitoring for daily activities. Critically, the paper restricts the total number of objects in any image to under 8 — this keeps the counting task within the subitizing range (the ability to immediately perceive small quantities without counting) for humans and avoids large-number counting difficulties that are a separate capability from analogical reasoning.

Transformation rule: The number of identical object instances in the image changes by a specified arithmetic operation. For addition and subtraction, objects are added to or removed from the scene. For multiplication and division, the set of objects is duplicated or partitioned.
KiVA subdomains: Four basic arithmetic operations — +1 (add one object), +2 (add two objects), −1 (remove one object), and −2 (remove two objects). These correspond to the small-number transformations that young children can reason about causally and analogically (Goddu et al., 2020).
KiVA-adults additions: Multiplication and division operations — ×2 (double the number of objects, equivalent to duplicating the set), ×3 (triple the number of objects), ÷2 (halve the number, keeping only half the objects), and ÷3 (divide into thirds, keeping one third). These operations go beyond what young children can typically handle and test whether models can reason about multiplicative relationships between quantities. The constraint that output images must contain fewer than 8 objects limits which input counts can be used for multiplication trials.
Domain-level classification options: "Change in number of objects" is the correct classification. The verbal specification in KiVA involves choosing among the four addition/subtraction operations plus "no change" and "doesn't apply." KiVA-adults adds the four multiplication/division options.

Rotation. Mental rotation — the ability to recognize an object from different viewpoints by mentally simulating its rotation — is a well-studied cognitive ability that develops between ages three and five (Frick et al., 2013; Quaiser-Pohl, 2003) and is classically measured using the Shepard-Metzler paradigm (Shepard & Metzler, 1971). The paper motivates rotation as essential for "object manipulation, spatial orientation and navigation" (Section 3.2, citing Pinto et al., 2008).

Transformation rule: The object is rotated in the 2D image plane by a specified angle about its center. This is a 2D rotation (in the picture plane), not a 3D rotation in depth, which simplifies the transformation to a well-defined image operation and avoids ambiguities about 3D shape from a single 2D view.
KiVA subdomains: Three rotation angles — +90° (clockwise 90-degree rotation), −90° (counterclockwise 90-degree rotation), and 180° (half-turn rotation, which is equivalent to both +180° and −180°). These angles are adapted from standard human psychometric instruments like the Purdue Visualization of Rotations Test (Bodner & Guay, 1997).
KiVA-adults additions: Two oblique rotation angles — +45° (clockwise 45-degree rotation) and −45° (counterclockwise 45-degree rotation), as well as +135° and −135°. These non-orthogonal angles are perceptually harder to discriminate and require more precise mental rotation. The paper notes this but does not provide a detailed rationale for why 45° and 135° specifically were chosen beyond increasing difficulty.
Domain-level classification options: "Change in orientation of objects (where things face)" is the correct classification. The verbal specification involves distinguishing among the specific rotation angles. An important subtlety: the paper refers to "orientation" in the classification prompt but "rotation" in the domain description — these are treated as equivalent in this context, both referring to changes in the object's angular position.

Reflection. Reflection (mirroring) along an axis tests the ability to appreciate object symmetry and chirality — the property that an object and its mirror image cannot be superimposed by rotation alone (Lin et al., 2020). The everyday relevance is clear: distinguishing left from right shoes, recognizing that text appears reversed in a mirror, or detecting that an object has been flipped.

Transformation rule: The object is reflected (mirrored) across a specified axis passing through its center. A reflection along the x-axis flips the object top-to-bottom. A reflection along the y-axis flips the object left-to-right. For chiral objects, the reflected version is a distinct, non-superimposable configuration.
KiVA subdomains: Two reflection axes — x-axis reflection (vertical flip, where top becomes bottom) and y-axis reflection (horizontal flip, where left becomes right). These correspond to the transformations used in Goddu et al. (2020).
KiVA-adults additions: Both axes simultaneously — the object is reflected along both the x-axis and y-axis, which is equivalent to a 180° rotation for non-chiral objects but produces a distinct result (a reflection through the origin) that cannot be achieved by rotation alone for chiral objects. This tests whether models can distinguish compound spatial transformations from simple rotations.
Domain-level classification options: "Change in reflection of objects" is the correct classification domain. The specification involves identifying the axis of reflection. An important note: because a 180° rotation and a double reflection (x then y) produce identical results for symmetric objects, the benchmark must use objects with distinguishable orientations (chiral or asymmetric objects) for these transformations to be well-defined.

Why these five domains? The paper's selection is explicitly grounded in developmental psychology rather than computer vision tradition:

"other visual properties, such as depth, spatial compositionality, and physical affordances are also crucial for such purposes, but we prioritized these five domains for our benchmark in particular because young children can solve these visual analogies, as already shown in developmental psychology literature (Goddu et al., 2020; Harris et al., 2013)"

This is a methodological choice: by selecting transformations with known developmental trajectories, the benchmark can place model performance on a human cognitive timeline. When GPT-o1 performs at 50% accuracy on number extrapolation, this can be contextualized as "below three-year-old level" rather than as an abstract score.

The Three-Stage Query Protocol: Decomposing Analogical Reasoning

The central methodological innovation of KiVA is not the dataset itself but the three-stage evaluation protocol that decomposes analogical reasoning into sequentially probed component processes. This design is explicitly motivated by Sternberg's (1977) component processes model of analogical reasoning, which identifies encoding, inference, mapping, and application as distinct cognitive operations in solving analogies. The paper maps these to: encoding → verbal classification (detecting what changed), inference → verbal specification (formulating how it changed), and mapping + application → visual extrapolation (applying the rule to a new target).

Trial structure and information flow. Each trial presents the participant with:

A given transformation: Two images displayed side-by-side (or as separate images for multi-image models), separated by a black line, showing an object before and after a transformation. The instruction identifies this as "the left-to-right transformation on top of the puzzle where the picture on the left transforms to the picture on the right."
Three candidate extrapolations: Below the given transformation, three additional left-to-right transformation pairs (labeled A, B, and C) showing a different object undergoing potential transformations. Exactly one of these matches the rule applied in the given transformation. A fourth option (D) allows the participant to indicate that none of the candidates match.

The three stages are then administered sequentially:

Stage 1: Verbal Classification ("what changed"). The paper presents the following prompt to models and adults (Section 4, Figure 3; full prompts in Appendix A.2):

"Which one of the following rules {(1) change in orientation of objects (where things face), (2) change in number of objects, (3) change in size of objects, (4) no change, (5) doesn't apply} best describes the left-to-right transformation on top of the puzzle where the picture on the left transforms to the picture on the right? Answer with the correct rule number surrounded by parentheses, then provide a 'step-by-step' reasoning for your choice."

The specific list of rules varies by trial. For a trial involving a color change, the options would be drawn from {change in color, change in size, change in number, change in orientation, change in reflection}, with "no change" and "doesn't apply" always included. The correct answer is always one of the five domain-level categories plus the two universal options, making 7 total options. However, the paper states (Section 3.1) that "chance level is 25% for Verbal Classification (4 choices)" because "no change" and "doesn't apply" are excluded from the chance calculation — the effective chance level depends on whether those are treated as genuine options or catch-all responses.

The step-by-step reasoning request serves a dual purpose: for humans, it encourages explicit reflection on the visual evidence. For models, it elicits chain-of-thought reasoning (Wei et al., 2022) that may improve accuracy and provides qualitative insight into the model's perceptual process — though the paper does not systematically analyze the content of these reasoning traces.

Stage 2: Verbal Specification ("how it changed"). This stage is only administered if the participant correctly answered Stage 1 (green arrow in Figure 3). This conditional structure is a practical feature of the human experimental design, not a fundamental requirement — the paper notes that for models, the full three-stage protocol is always administered, and performance conditional on prior correctness is analyzed post-hoc (Appendix B.2, Figure 12).

The specification prompt (Section 4, Figure 3) presents options specific to the correctly identified domain. For example, if the transformation was correctly classified as a size change:

"Which one of the following rules {(1) objects become smaller, (2) objects become bigger, (3) no change, (4) doesn't apply} best describes the left-to-right transformation in the top of the puzzle where the picture on the left transforms to the picture on the right?. Answer with the correct rule number surrounded by parentheses. Then provide a 'step-by-step' reasoning for your choice."

The options presented are the subdomain-specific transformations for the classified domain. Chance level is reported as 33% for Verbal Specification (3 choices, excluding "doesn't apply"). The paper explicitly includes "no change" and "doesn't apply" even at the specification stage to accommodate cases where the participant correctly identified the domain but believes no actual transformation occurred within it, or that the specific transformation doesn't match any of the provided options.

Stage 3: Visual Extrapolation. This is the core analogical reasoning task — and critically, it can be attempted regardless of performance on Stages 1 and 2. The paper states (Section 3.1):

"Critically, KiVA retains the core nonverbal extrapolation task (last step) from previous benchmarks and the verbal questions do not replace the core nonverbal tasks. Even without correct verbal responses, humans and models can still tackle the independently-assessed visual extrapolation tasks."

This is a crucial design choice. It means the extrapolation accuracy can be compared directly to prior benchmarks that only test end-to-end puzzle solving. At the same time, by tracking whether the participant got the verbal stages right, the paper can condition extrapolation performance on prior verbal reasoning success and identify where in the chain the failure occurs.

The extrapolation prompt (Figure 3):

"Which one of three left-to-right object transformations (marked by either (A), (B) or (C)) on the bottom of the puzzle is the same as the left-to-right transformation on the top of the puzzle? Answer with the correct letter surrounded by parentheses (or (D) if none of the options apply), then provide a 'step-by-step' reasoning for your choice."

Chance level is 33% (3 choices, excluding the "none" option D). The distractor transformations are designed to be plausible — they involve the same test object undergoing different transformations, not obviously wrong changes. This prevents the task from being solvable by simple perceptual matching (e.g., "option B has a red object and the given transformation also has a red object") and requires actual rule abstraction.

Why a three-stage decomposition? The paper offers two justifications:

Diagnostic precision. Without decomposition, a model that scores 50% on extrapolation could be failing because (a) it cannot detect what changed, (b) it can detect the change domain but cannot quantify the specific rule, or (c) it can fully specify the rule but cannot map it to a novel object. Each failure mode suggests a different remediation: (a) requires better low-level visual perception, (b) requires better quantification or within-domain discrimination, and (c) requires better relational mapping — the core of analogy. The paper's results (Section 4.1, Figure 4) show that all three failure modes occur, with mapping being the most severe.

Alignment with cognitive theory. Sternberg's (1977) component processes model posits that analogical reasoning involves sequential encoding, inference, mapping, and application stages. The three-stage protocol approximately maps to these: classification ≈ encoding + early inference (detecting the relevant features), specification ≈ late inference (determining the specific rule), and extrapolation ≈ mapping + application (carrying the rule to a new context). This alignment means that performance patterns on KiVA can be interpreted within an established cognitive framework rather than as atheoretical benchmark scores.

Child-appropriate prompting. For human children, the verbal classification and specification prompts are paraphrased into developmentally appropriate language (Appendix A.3):

"Here are two pictures separated by a black line in the middle. The picture on the left turns into the picture on the right. Do you think there is a change? What do you think the change is?" (Verbal Classification)

"Can you say more about the change from the left to the right?" (Verbal Specification)

The extrapolation prompt for children:

"Here is another picture that goes through the same change from the left to right. Can you find the box that shows the same change?"

The paper notes that "the prompt used for children did not improve model or human adult performance" (Appendix A.3), so the standard prompts were retained for non-child participants. This is methodologically important — it means the benchmark tests the same underlying ability across participant types despite using different surface-level language, and the child prompts are a necessary adaptation rather than a confound.

The Difficulty Tier System: KiVA vs. KiVA-adults

The paper introduces two calibrated difficulty levels that create a developmental gradient for benchmarking. This tiered structure allows the benchmark to assess whether models follow a human-like developmental trajectory (mastering KiVA-level analogies before KiVA-adults-level ones) or show qualitatively different patterns of relative strength and weakness.

KiVA: The child-solvable tier. KiVA is designed to be solvable by children as young as three years old. The paper states (Section 1): "It is solvable by a three-year-old child." This claim is empirically validated by the human data — children aged three to five perform significantly above chance on all five transformation domains in KiVA (Figure 4, top panel), with the lowest child performance being on rotation (still well above the 25-33% chance levels).

The key design property that makes KiVA developmentally accessible is aligned starting values between the given transformation and the test items. In a KiVA trial, the starting state of the test object (its initial color, size, orientation, or quantity) matches or is similar to the starting state of the object in the given transformation. For example, if the given transformation shows a red mug turning blue, the test object might start as a red ball — the initial color is the same (red), and the rule "change to blue" can be applied by simple feature matching ("make the test object blue because the given object became blue"). This reduces the abstraction demand: the reasoner does not need to represent the rule as "change to the target color regardless of starting color" but can instead use a simpler heuristic of "copy the output color."

KiVA-adults: The adult-only tier. KiVA-adults increases the difficulty by introducing misaligned starting values between the given and test transformations. The paper states (Figure 1 caption):

"Unlike KiVA, the starting color, size, orientation and number of test objects in KiVA-adults further differ from the starting values of the given transformations."

This design change forces a deeper level of rule abstraction. Consider a KiVA-adults color trial where the given transformation shows a green car turning red, but the test object starts as a blue chair. A simple "copy the output color" heuristic would incorrectly predict the chair should become red. The correct answer requires understanding the abstract rule: "change the object's color to the target color regardless of what color it started as." This is the relational mapping that Gentner (1983) identifies as the core of analogy — preserving the relationship (becomes red) rather than the attributes (the specific colors involved).

The paper empirically validates that this manipulation creates a genuine difficulty gap: children under five perform at chance on KiVA-adults (Section 4: "We evaluated an additional 10 children and 40 adults on KiVA-adults and found that none of the children performed better than chance"). Adults, by contrast, maintain near-perfect performance on KiVA-adults (Figure 5), demonstrating that the increased abstraction demand is within mature human cognitive capabilities.

Expanded transformation variety in KiVA-adults. In addition to the misaligned starting values, KiVA-adults introduces more subdomain variations within each transformation domain (detailed in Section 3.2 and the domain descriptions above):

Color: adds yellow and grey (5 colors vs. 3)
Size: adds stretching (height-only or width-only scaling)
Number: adds multiplication and division (×2, ×3, ÷2, ÷3)
Rotation: adds oblique angles (45°, 135°)
Reflection: adds compound reflection (both axes simultaneously)

These additions increase the number of possible specifications within each domain, making the verbal specification stage harder even if the domain is correctly classified. They also introduce transformations that are perceptually and cognitively more complex — stretching requires attending to two dimensions independently, oblique rotation requires finer angular discrimination, and division requires partitioning a set rather than simply adding or removing items.

Scale of the benchmarks. The paper reports the following trial counts (Section 3.2):

KiVA: 100 object transformations per subdomain. With 5 domains and an average of approximately 2.8 subdomains per domain (3 color + 2 size + 4 number + 3 rotation + 2 reflection = 14 subdomains), this yields 1,400 total transformations.
KiVA-adults: An additional 2,900 transformations, reflecting the expanded subdomain count (5 color + 3 size + 8 number + 5 rotation + 3 reflection = 24 subdomains, though the paper reports 2,900 not 2,400, suggesting some subdomains have more than 100 trials or there are additional variations not enumerated in the main text).
Total benchmark: 4,300 transformations across both tiers.

No-change trials as a control condition. The paper embeds a special class of trials where the transformation involves no change at all (Section 4.1, Figure 7). For each transformation domain, 10% of trials are randomly sampled and assigned a null transformation — the "before" and "after" images are identical. These trials serve as an important control: they test whether models can recognize the absence of change, and whether they hallucinate transformations when none exist. The correct response for these trials is "no change" at all three stages (classification, specification, and extrapolation), with the correct extrapolation being the option where the test object also does not change.

Human and Model Evaluation Procedures

Human adult evaluation (Section 4). A total of 250 adults aged 21 to 40 were recruited via Prolific, an online participant recruitment platform. The paper specifies the following procedural details:

Compensation: Adults were paid at least $12 per hour with a bonus of$ 0.01 per correct response. This performance-contingent bonus incentivizes genuine effort rather than random responding.
Coverage: Each trial was annotated by 3 to 13 adults. This variable coverage reflects the adaptive nature of online recruitment — some trials received more annotators than others based on participant availability and task routing.
Practice and screening: All participants completed a practice trial involving an "unrelated" transformation (adding a dot to geometric shapes) and received feedback. Participants who failed to pass the practice trial within three attempts were excluded from the main experiment. This screening ensures that participants understand the task format before contributing data.
Test conditions: Participants who passed the practice proceeded to test trials without feedback. They were informed that rewards depended on their performance, maintaining motivation.
Platform: The experiment was implemented using JsPsych (De Leeuw, 2015), a JavaScript library for running behavioral experiments in web browsers.

Human child evaluation (Section 4). A total of 42 children aged three to five years (mean age = 4.07 years, standard error = 0.11 years) were recruited from "early childhood centers and ChildrenHelpingScience," an online platform for developmental research. Key procedural details:

In-person administration: Unlike adults who completed the task online, children were tested by human experimenters who read instructions aloud. This adaptation is necessary because young children cannot read complex written prompts.
Trial sampling: Each child completed a random subset of 10 trials, with 2 trials per transformation domain. This design choice balances the need for statistical power (420 total responses across 42 children × 10 trials) against the limited attention span of young children, who cannot complete hundreds of trials like adults or models.
Motivation: Children were told they would receive coins for correct answers, which could be exchanged for stickers at the end. This gamification is standard practice in developmental psychology to maintain engagement with young participants.
Practice trial: Children completed a practice trial with feedback and had to succeed to proceed, mirroring the adult procedure.
KiVA-adults exclusion: An additional 10 children were tested on KiVA-adults, and none performed above chance. This empirical validation confirms that KiVA-adults genuinely exceeds the capabilities of children under five, establishing it as a valid adult-level tier rather than simply a harder version of KiVA.

Model evaluation (Section 4). Four large multimodal models were tested on the full KiVA and KiVA-adults benchmarks:

GPT-o1 (o1-2024-12-17): OpenAI's reasoning model, which incorporates chain-of-thought reasoning as an intrinsic capability rather than requiring explicit prompting.
GPT-4V (gpt-4-vision-preview): OpenAI's vision-capable model, tested in the configuration available at the time of the study.
LLaVA-1.5 (Liu et al., 2024): An open-source model that integrates a vision encoder (CLIP-based) with a language model (Vicuna), designed for general-purpose visual and language understanding through visual instruction tuning.
MANTIS (Jiang et al., 2024a): An open-source model built on modified architectures from models like LLaVA, specifically designed to support interleaved multi-image input — a capability that is important for KiVA because the given transformation and three candidate extrapolations can be presented as separate images rather than stitched together.

Inference hyperparameters. The paper specifies the following settings (Section 4):

Temperature: Set to 1 for all models (no temperature cap for GPT-o1, as noted parenthetically). This relatively high temperature introduces stochasticity in the outputs, enabling the measurement of response consistency across repeated trials.
Maximum token size: Set to 300 tokens for all models except GPT-o1 (no cap specified). This is sufficient for the model to produce a multiple-choice answer plus step-by-step reasoning.
Seeds: Each experiment was randomized over three seeds. Each trial was run three times with the order of test choices shuffled. This repetition enables the measurement of choice inconsistency (Section 4.1, Figure 8) — how often a model selects different responses in identical repeated trials.
Scoring: Correct choices are scored as 1, incorrect as 0. The mean score across the three seeds is calculated per trial. To evaluate performance per transformation domain, the overall mean and standard error are calculated across all trials in that domain.

Image presentation format. The paper addresses a technical challenge specific to multi-image analogical reasoning: how to present the given transformation and candidate extrapolations to models with different input architectures.

Single-image presentation (LLaVA-1.5): The given transformation and three candidate extrapolation pairs are stitched into a single composite image. Appendix A.1 (Figure 10) shows examples of these composite images, where the given pair appears at the top and three candidate pairs (labeled A, B, C) appear below. This format is necessary because LLaVA-1.5 can only process a single image input.
Multi-image presentation (MANTIS, GPT-4V, GPT-o1): The given transformation and each of the three candidate extrapolation pairs are presented as four separate images. The paper explicitly cites Campbell et al. (2025) to justify this choice: presenting transformations as separate images reduces the chance of "visual binding errors" — the phenomenon where models incorrectly associate visual elements that are spatially proximal but conceptually distinct. When all transformations are stitched into one image, a model might confuse which object belongs to which transformation pair or incorrectly bind features across the composite layout.
GPT-o1 comparison between formats: Appendix B.1 (Figure 11) reports an ablation comparing multi-image vs. single-image presentation for GPT-o1. Multi-image presentation significantly improves extrapolation accuracy for color, size, and number domains, but not for rotation and reflection. The paper interprets this as evidence that the challenges in rotation and reflection "go beyond a visual binding problem described in Campbell et al. (2025)" — even when binding errors are eliminated, these spatial domains remain difficult.

Prompt engineering experiments (Section 4.1, Appendix A). The paper explores whether model performance can be improved through sophisticated prompting strategies, a common approach for eliciting better reasoning from LMMs. Four text-based and two visual prompting methods were tested, though the paper reports that "none of these approaches improve performance, which points to the challenging nature of our benchmark."

The four text-based methods (detailed in Appendices A.4–A.6):

Reasoning through code (Appendix A.6): The model is first prompted to generate Python code (using the Pillow library) that would produce each transformation. It is then asked to answer the KiVA questions using insights from the generated code. This approach (inspired by Sharma et al., 2024) tests whether translating visual transformations into procedural code improves reasoning by leveraging the model's programming capabilities.
Reasoning after Reflection (Appendix A.4): After producing an initial answer, the model is prompted to "reflect on your answer and provide a revised response if necessary." This reflection step is repeated three times per question, with the model instructed to start each response with its updated answer. This approach (inspired by Valmeekam et al., 2023) tests whether iterative self-critique can correct initial errors — a strategy that has shown mixed results in prior work.
Reasoning through instruction (Appendix A.5): The model is first prompted to generate "brief instructions on how to establish if a transformation involves [specific change]" and then uses those self-generated instructions to answer the KiVA question. This approach (inspired by Wei et al., 2022) tests whether having the model externalize its reasoning strategy as explicit instructions before applying them improves performance.
In-Context Learning (Section 4.1): The model is given two randomly sampled examples with their correct solutions before each trial. This tests whether few-shot demonstration can induce better analogical reasoning, a capability that has been demonstrated for many language tasks (Dong et al., 2022) but may not transfer to visual analogy.

The two visual prompting methods (Section 4.1):

Background color manipulation: The image background (initially transparent) is altered to black or white. This is inspired by Bai et al. (2023), who showed that visual model performance can be sensitive to background color.
Image size variation: A center crop is applied to vary the image size between 0.9 and 1.0 times the original. This tests whether model performance is sensitive to the absolute size of objects in the input image.

The failure of all these prompting strategies to improve performance is a significant negative result. It suggests that the difficulty of KiVA is not primarily due to suboptimal prompting but reflects genuine limitations in the models' visual analogical reasoning capabilities. This distinguishes KiVA from benchmarks where better prompt engineering unlocks latent model abilities.

Large Vision Model evaluation (Appendix B.4). The paper additionally tests a Large Vision Model (Bai et al., 2024) — a model trained entirely on visual data without any linguistic supervision — on KiVA to investigate whether language is necessary for visual analogical reasoning. Because this model has no text interface, the three-stage query protocol cannot be applied. Instead, the model is prompted using a visual completion format (Figure 14a): the given transformation and the left side of each candidate extrapolation are provided, and the model must generate (predict) the right side. The candidate whose predicted right side has the lowest perplexity (best matches the model's prediction) is selected as the answer.

This experiment serves as an important control: if the Large Vision Model can solve KiVA analogies despite having no language, this would suggest that visual analogical reasoning does not require the verbal scaffolding that the three-stage protocol provides. The results (Figure 14b) show that the Large Vision Model performs above chance on color and size extrapolations but at or below chance on number, rotation, and reflection — a pattern qualitatively similar to the LMM results. This suggests that the verbal reasoning stages in KiVA are not the source of the difficulty for spatial and numerical domains; rather, these domains are fundamentally harder for current vision architectures regardless of linguistic support.

Summary of Design Choices and Their Justifications

3D object models over natural images: Ensures controlled, deterministic transformations with known ground-truth rules. Natural images would introduce confounding variables (lighting, background, viewpoint) that obscure whether failures are due to analogical reasoning or low-level perception.
Five specific transformation domains grounded in developmental psychology: Rather than selecting arbitrary visual changes, the paper tests transformations with established developmental trajectories, enabling meaningful comparison against human cognitive milestones.
Three-stage decomposition: Separates change detection, rule specification, and rule application into sequentially probed components, enabling diagnosis of where reasoning fails rather than just whether it fails.
Conditional administration with independent extrapolation: The verbal stages provide diagnostic information, but the extrapolation task (the core analogy) can be attempted regardless of verbal performance, preserving comparability with prior benchmarks that test only end-to-end accuracy.
Aligned vs. misaligned starting values (KiVA vs. KiVA-adults): Manipulates the abstraction demand of the analogical mapping. Aligned starting values allow surface-level feature matching (easier, solvable by three-year-olds). Misaligned starting values require relational abstraction (harder, solvable only by adults).
Multiple-choice format with "doesn't apply" and "no change" options: Enables automatic scoring while accommodating edge cases where the participant's reasoning does not match any provided option. The "no change" trials serve as a critical control for hallucinated transformations.
Three repeated trials with shuffled choices for models: Enables measurement of response consistency — an important diagnostic because inconsistent responses to identical stimuli suggest unreliable reasoning rather than systematic errors.
Multi-image presentation where supported: Reduces visual binding errors that can occur when multiple transformations are composited into a single image, ensuring that failures are due to reasoning limitations rather than perceptual conflation of spatially proximal but logically distinct elements.
Child-adapted prompts that do not improve adult/model performance: Ensures that the benchmark tests the same underlying ability across participant types despite using developmentally appropriate surface language for children.

4. Key Insights and Innovations

Innovation 1: Decomposing Analogical Reasoning Into Diagnosable Component Processes, Not Just End-to-End Accuracy

The dominant paradigm in visual reasoning benchmarks — from ARC (Chollet, 2019) to Raven's Progressive Matrices variants (Huang et al., 2024; Ahrabian et al., 2024) — has been to present a puzzle, collect a final answer, and report a single accuracy score. This approach treats analogical reasoning as a black box: a model that scores 60% could have perfect change detection but fail at rule application, or it could have perfect rule application but fail at perceiving the change at all — and the benchmark provides no way to distinguish these scenarios.

KiVA's three-stage query protocol is not merely an incremental improvement in evaluation granularity. It is a fundamental reframing of what it means to evaluate analogical reasoning, shifting from endpoint measurement to process diagnosis. The paper explicitly grounds this in Sternberg's (1977) component processes model, which posits that analogical reasoning involves sequential encoding, inference, mapping, and application stages. By operationalizing these as separately-probed questions — verbal classification (encoding/inference), verbal specification (inference refinement), and visual extrapolation (mapping/application) — the benchmark reveals where in the reasoning chain models fail, not just whether they fail.

The diagnostic value of this decomposition is demonstrated concretely in the results. Figure 4 shows that LMMs achieve high accuracy on verbal classification across most domains (blue bars, often well above chance) but performance drops sharply at verbal specification (orange bars) and drops further at visual extrapolation (green bars). This pattern — high change detection but poor quantification and mapping — would be invisible in a benchmark that only measures extrapolation accuracy. A model could detect that a transformation involves rotation (Stage 1) but fail to specify whether it is 90° or 180° (Stage 2), and even if it specifies correctly, it might fail to apply that 90° rotation to a novel object (Stage 3). Each failure mode implies a different remediation: poor specification suggests inadequate within-domain discrimination, while poor extrapolation despite correct specification (documented in Appendix B.3, Figure 13b) suggests a fundamental challenge in relational mapping that goes beyond perception.

What makes this innovation distinctive is not the idea of multi-step evaluation per se — plenty of benchmarks use multiple questions — but the conditional, diagnostic structure where each stage builds on the previous while remaining independently assessable. The paper explicitly preserves the core nonverbal extrapolation task (Stage 3) that prior benchmarks test, meaning KiVA can serve both as a direct comparison to prior work (using only extrapolation accuracy) and as a richer diagnostic tool (using the full pipeline). The finding that models' extrapolation performance improves when they correctly answer prior verbal stages (Figure 12), but that giving them the correct answer to prior stages does not substantially improve subsequent performance (Figure 13), reveals a nuanced picture: verbal reasoning is correlated with visual reasoning success but does not cause it in any simple way — the underlying analogical mapping ability is partially independent of verbal articulation.

This is a fundamental shift in benchmark design philosophy. It moves from "build harder puzzles" toward "build diagnostic instruments that reveal the structure of model (and human) cognition." The paper itself embodies this by using the decomposition to compare failure patterns across models and humans, finding that GPT-o1's errors correlate with children's errors and adult response times (Figure 6) — a finding that would be inaccessible without diagnostic granularity.

Innovation 2: Anchoring AI Evaluation in the Developmental Psychology of Human Cognition

Prior visual reasoning benchmarks have compared models against either an abstract notion of "human-level performance" (usually measured on adult Mechanical Turk workers or graduate students) or against no human baseline at all. ARC (Chollet, 2019) does not report systematic human performance data stratified by developmental stage. ConceptARC (Moskvichev et al., 2023) includes human adult baselines but no child data. RPM-based benchmarks (Huang et al., 2024) use puzzles that are developmentally gated — children under six typically cannot solve them — but do not frame this as a meaningful calibration point.

KiVA's most distinctive methodological move is to calibrate its benchmark against a specific developmental milestone: what children aged three to five can do. This is not merely "adding a human baseline." It is a fundamental reconceptualization of what the benchmark measures. Rather than asking "Can models solve hard puzzles?" — a question with no clear endpoint — KiVA asks "Do models possess the basic visual analogical reasoning that emerges in early childhood?" This is a theoretically motivated, answerable question grounded in developmental psychology literature showing that children can reason analogically about color, size, number, rotation, and reflection transformations of real-world objects (Goddu et al., 2020).

The implications of this framing are significant. When the paper reports that GPT-o1 "does not match even the capabilities of a three-year-old child in reasoning about number and reflection" (Section 1), this is not an arbitrary threshold — it is a claim that current LMMs lack abilities that developmentally precede more complex reasoning. The fact that children outperform GPT-o1 on number extrapolation (Figure 4, top panel) despite GPT-o1's vastly greater training data and parameters is not just an interesting failure case; it is evidence that the developmental trajectory of visual analogical reasoning is not being replicated by current training paradigms. Models are not progressing along a human-like cognitive trajectory; they show qualitatively different patterns of relative strength and weakness.

The two-tiered structure (KiVA and KiVA-adults) further operationalizes this developmental framing. KiVA is solvable by three-year-olds, establishing a floor. KiVA-adults is not solvable by children under five, establishing a developmental gate. The fact that children perform at chance on KiVA-adults (Section 4) validates this gate empirically rather than assuming it. This creates a developmental ruler: a model that masters KiVA but not KiVA-adults is operating at roughly the level of a young child; a model that masters both is operating at adult level; a model that masters neither has not yet reached the cognitive milestones of early childhood. GPT-o1 currently occupies the last position, with the paper noting it only reaches adult-level reasoning in the color domain of KiVA-adults (Figure 5).

This innovation matters beyond this paper because it provides a template for developmentally-grounded AI evaluation. Rather than building benchmarks by iterating on what current models find hard — a recipe for chasing ever-more-arbitrary puzzle designs — developmental grounding anchors benchmarks in an independently validated cognitive progression. The specific transformation domains, the abstraction gradients, and the baseline age ranges are not arbitrary choices but are drawn from existing psychological literature. This makes the benchmark's difficulty interpretable in cognitive terms rather than merely in terms of model performance metrics.

Innovation 3: Demonstrating That Visual Analogical Difficulty Is Intrinsic to Certain Problem Domains, Not an Artifact of Architecture or Training

A dominant assumption in much LMM research is that scaling — more parameters, more data, more compute — will eventually address most cognitive shortcomings. Under this view, a model's failure at a particular task reflects insufficient scale or suboptimal training, not a fundamental architectural limitation. The paper's results challenge this assumption by showing a convergence in difficulty patterns between fundamentally different systems: GPT-o1 (a transformer-based language model with chain-of-thought reasoning), a Large Vision Model trained without any linguistic data (Appendix B.4, Figure 14b), human children (Figure 4), and human adults (Figure 6, response times).

The evidence for this convergence comes from multiple angles:

Domain-level difficulty ordering is consistent across architectures. All tested systems — GPT-o1, GPT-4V, LLaVA-1.5, MANTIS, the Large Vision Model, children, and adults (as measured by response time) — find color and size transformations easier than rotation, reflection, and number transformations. This is visible in Figure 4 (bottom) for LMMs and adults, Figure 4 (top) for children, and Figure 14b for the Large Vision Model. If difficulty were an artifact of a specific architecture or training regime, we would expect different systems to show different difficulty orderings. The consistency suggests that certain visual analogies are inherently harder, likely because they require cognitive operations — mental rotation, sequential tracking, set manipulation — that are more computationally demanding regardless of the substrate.

GPT-o1 error scores correlate with children's error scores and adult response times. Figure 6 reports positive correlations between GPT-o1's mean error scores on KiVA visual extrapolation and both (a) children's mean error scores and (b) adults' mean response times, broken down by specific transformation subdomain (e.g., +90° rotation, x-axis reflection, +2 addition). This is a striking finding: the relative difficulty of specific transformation rules across the benchmark is similar for a large language model trained on internet-scale data and for developing humans. What is cognitively demanding for humans — and what takes longer to process — is also computationally challenging for models.

The paper does not overclaim this convergence. It explicitly states (Section 4.2):

"Although this convergence in performance between LMMs and human children does not imply that they are built or function identically, it is intriguing that similar trends emerge from such fundamentally different systems."

This is appropriately cautious, but the implication is significant: some of the difficulty in visual analogical reasoning may reflect intrinsic properties of the problems themselves — the computational complexity of mental rotation, the sequential processing demands of counting, the relational abstraction required for reflection — rather than peculiarities of human neural architecture or transformer attention mechanisms. This is a fundamental conceptual shift away from the purely architectural critique of LMM limitations toward a recognition that some cognitive operations are objectively harder regardless of implementation.

Prompt engineering and reasoning strategies do not close the gap. The paper tests four text-based prompting strategies (reasoning through code, reflection-based self-critique, instruction generation, and in-context learning) and two visual prompting strategies (background color, image sizing) — and reports that "none of these approaches improve performance" (Section 4.1). This negative result is methodologically important because it distinguishes KiVA from benchmarks where clever prompting unlocks latent capabilities. The failure of chain-of-thought, reflection, and code-generation strategies to help suggests that the difficulty is not in eliciting existing reasoning but in performing the reasoning at all — the models genuinely cannot do these analogies, not just fail to express their ability.

The Large Vision Model result (Appendix B.4) further isolates the source of difficulty. This model, trained without any linguistic data, performs above chance on color and size extrapolation but at or below chance on number, rotation, and reflection — the same pattern as the language-equipped LMMs. This suggests that language is not the bottleneck for spatial and numerical analogies. Even a purely visual system with no text interface struggles with the same domains. The difficulty is in the visual processing and relational mapping, not in the verbal mediation.

This innovation positions KiVA not just as a benchmark but as a tool for investigating the nature of visual analogical reasoning itself — the patterns of difficulty that transcend implementation details and point toward fundamental cognitive structure.

Innovation 4: Methodologically Rigorous Distinction Between "Seeing" a Change and "Reasoning Analogically" About It

The paper's separation of visual change detection from analogical mapping — operationalized through the conditional performance analyses in Figure 12 and the "given correct answer" experiments in Figure 13 — reveals a critical and previously unmeasured distinction between perceptual success and reasoning success. This is not merely a methodological refinement; it is a reconceptualization of what visual analogical reasoning entails that challenges how prior benchmarks have been interpreted.

Prior benchmarks implicitly assume that if a model selects the correct extrapolation, it has performed analogical reasoning. KiVA's decomposition reveals that this inference can be wrong in multiple ways. Figure 12 shows that GPT-o1's visual extrapolation accuracy is substantially higher when its prior verbal classification or specification was correct — across all domains — compared to when it was incorrect. This means that extrapolation success is not independent of perception success: models that correctly identify what changed are more likely to correctly extrapolate the rule. However, this relationship is not deterministic — even with correct verbal specification, extrapolation accuracy remains far from perfect, especially in spatial domains.

Figure 13b then provides the crucial control: when models are given the correct verbal specification (the experimenter tells them exactly what the rule is), their subsequent visual extrapolation performance improves only marginally and remains well below human levels. For GPT-o1, given the correct specification, number extrapolation rises to approximately 55-60% (still well below the adult near-ceiling), and rotation and reflection extrapolation remain around 40-60%. This means that even when the model knows the rule, it cannot reliably apply it to a novel object. The failure is specifically in the mapping process — carrying a relational structure from a source domain to a target domain while preserving the relationship.

This finding has important implications for how we interpret prior benchmark results. A model that scores highly on RPMs or ARC might be succeeding through perceptual pattern matching rather than genuine analogical mapping. If the test items share surface features with the source items (as they often do in abstract puzzles using the same geometric vocabulary), a model can succeed by detecting feature-level similarities without ever performing the structural alignment that defines analogy. KiVA's design partially mitigates this by using different objects for source and target transformations, but the paper's conditional analyses go further by showing that even when the rule is explicitly communicated, mapping to a novel object remains hard. This is not a failure of perception — it is a failure of analogy specifically.

The hallucination results on no-change trials (Section 4.1, Figure 7) provide additional evidence for this distinction. LLaVA-1.5 and MANTIS "hallucinate a change in 100% of the no-change trials during verbal classification" (Section 4.1), meaning they fail at the most basic perceptual task: detecting that nothing happened. GPT-4V correctly identifies no-change in some domains at the classification stage but struggles at specification. GPT-o1 correctly identifies no-change at both verbal stages but "struggles to extrapolate this to new objects when distractors involve reflection or number change." This progression — from hallucinating change (LLaVA, MANTIS) to detecting absence of change but failing to extrapolate it (GPT-o1) — maps onto the perception-to-reasoning spectrum and shows that different models fail at different points along this continuum.

By making this distinction empirically measurable, the paper provides a framework for categorizing model failures along a perception-reasoning gradient rather than treating all errors as equivalent. This is a fundamental contribution to evaluation methodology that extends beyond this specific benchmark.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper constructs two benchmark tiers: KiVA with 1,400 object transformations across five visual domains (color, size, number, rotation, reflection), each with 100 transformations per subdomain, and KiVA-adults with 2,900 transformations incorporating expanded subdomain variety and misaligned starting values between given and test transformations, totaling 4,300 transformations. Objects are drawn from 3D model datasets of household items (Downs et al., 2022) and infant-familiar toys (Stojanov et al., 2021), rendered as 2D images with parameterized transformations applied procedurally. The paper reports these numbers in Section 3.2.
Base model(s). Four large multimodal models are tested: GPT-o1 (o1-2024-12-17), GPT-4V (gpt-4-vision-preview) (OpenAI, 2023), LLaVA-1.5 (Liu et al., 2024) — an open-source model integrating a CLIP-based vision encoder with a Vicuna language model through visual instruction tuning — and MANTIS (Jiang et al., 2024a), which builds on LLaVA-style architectures to support interleaved multi-image input. The paper additionally tests a Large Vision Model (Bai et al., 2024) trained entirely without linguistic data (Appendix B.4). Human comparison groups include 42 children aged 3–5 years (mean = 4.07, se = 0.11) and 250 adults aged 21–40 recruited via Prolific.
Metrics. The primary metric is accuracy — the fraction of trials where the model or human selects the correct multiple-choice answer — computed separately for each of the three stages (Verbal Classification, Verbal Specification, Visual Extrapolation) and reported by transformation domain (color, size, number, rotation, reflection). For models, accuracy is the mean score across three repeated trials with shuffled choice orders. Additionally, choice inconsistency is measured as the proportion of repeated identical trials where a model selects different responses (Section 4.1, Figure 8). For human adults, response time is recorded during the visual-extrapolation-only variant (Section 4.1, Figure 9). All metrics are reported with standard errors across object variations within each transformation domain.
Baselines. The primary human baselines are children aged 3–5 (developmental floor for KiVA) and human adults (ceiling for both KiVA and KiVA-adults). Within-model baselines include: (1) visual-extrapolation-only performance — a version of KiVA where verbal questions are removed, tested on 200 additional adults, 20 children, and GPT-o1 (Section 4.1, Figure 9), replicating the format of prior visual analogy benchmarks; (2) single-image vs. multi-image presentation for GPT-o1 (Appendix B.1, Figure 11); (3) prompt-engineered variants — four text-based strategies (reasoning through code, reflection-based self-critique, instruction generation, in-context learning) and two visual strategies (background color manipulation, image size variation), described in Section 4.1 and Appendices A.4–A.6; and (4) the Large Vision Model tested via visual completion rather than verbal querying (Appendix B.4, Figure 14).
Generation budget / compute accounting. The paper does not measure compute in FLOPs or GPU-hours. Instead, all models are tested under identical conditions: temperature set to 1 (no cap for GPT-o1), maximum token size 300 (no cap for GPT-o1), and each trial repeated three times with shuffled choice orders across three random seeds (Section 4). Open-source models (LLaVA-1.5, MANTIS) run on a single A6000 48 GB GPU for under 12 hours. The fairness of comparison rests on all models receiving the same stimuli and prompts, not on equalized compute budgets.
Cross-validation / statistical protocol. No cross-validation is used, as this is a benchmark evaluation rather than a model training procedure. The primary statistical protocol is: (1) repeated trials: each model completes every trial three times with choice order shuffled to measure consistency; (2) standard error reporting: error bars in all figures represent standard errors across object variations within each transformation domain; (3) chance level calibration: chance is explicitly reported as 25% for Verbal Classification (4 choices, excluding "doesn't apply" and "no change") and 33% for Verbal Specification and Visual Extrapolation (3 choices, excluding the "none" option and "doesn't apply"); (4) human participant screening: adults and children complete a practice trial with feedback and are excluded if they fail within three attempts (Section 4); (5) variable human annotation density: each KiVA trial is annotated by 3–13 adults, while each child completes 10 randomly sampled trials (2 per domain), yielding 420 total child responses.

Main Quantitative Results

Overall Model vs. Human Performance Across the Three Reasoning Stages

The central finding, visible in Figure 4 (KiVA) and Figure 5 (KiVA-adults), is that LMM performance declines monotonically from verbal classification to verbal specification to visual extrapolation, whereas human performance remains stable or improves across stages.

In KiVA (Figure 4), across all five transformation domains:

Verbal Classification (blue bars): GPT-o1 and GPT-4V achieve high accuracy — near ceiling for color and size, and substantially above chance for rotation and number. GPT-o1 and GPT-4V even outperform children in categorizing rotation and color changes (Section 4.1). LLaVA-1.5 and MANTIS perform above chance but notably lower, particularly for number and reflection.
Verbal Specification (orange bars): Performance drops for all models relative to classification. GPT-o1 maintains the highest specification accuracy but shows substantial declines in number, rotation, and reflection domains. GPT-4V, LLaVA-1.5, and MANTIS drop toward or below chance in several domains, especially rotation and reflection.
Visual Extrapolation (green bars): Performance declines further. No model matches children's extrapolation accuracy in the number or reflection domains (top panel of Figure 4). Adults perform near ceiling across all domains and stages (bottom panel).

The paper quantifies this pattern qualitatively: "models' success in verbally describing transformations does not guarantee their success in extrapolation" (Section 4.1). Even when given the correct verbal specification of the transformation rule (Appendix B.3, Figure 13b), model extrapolation performance improves only marginally and remains far below human levels — for GPT-o1, extrapolation accuracy with correct specification given reaches approximately 55–60% for number and 40–60% for rotation and reflection.

In KiVA-adults (Figure 5), the gap widens further. GPT-o1 "nears adult performance only in the color domain" (Section 4.2), while adults maintain near-ceiling performance across all domains. Children perform at chance on KiVA-adults overall (Section 4), empirically validating the developmental gate.

Domain-Dependent Difficulty: Color and Size vs. Number, Rotation, and Reflection

The paper reports a consistent difficulty ordering across models and humans: color and size transformations are substantially easier than number, rotation, and reflection transformations.

In KiVA (Figure 4, bottom panel), GPT-o1's visual extrapolation accuracy by domain (read approximately from the green bars):

Color: ~80–85% (approaching adult ceiling)
Size: ~70–75%
Number: ~50–55%
Rotation: ~45–50%
Reflection: ~40–45%

GPT-4V shows a similar pattern at lower absolute accuracy. LLaVA-1.5 and MANTIS perform near or below chance on number, rotation, and reflection extrapolation. Children (Figure 4, top panel) show the same domain ordering — performing best on color and size, worse on rotation — with the paper noting that "children perform slightly worse on rotation compared to other domains" (Section 4.1).

In KiVA-adults (Figure 5), GPT-o1's color extrapolation nears adult performance (~90% vs. adult near-100%), but its number, rotation, and reflection extrapolation remain in the 40–60% range — far below adult ceiling. The paper states explicitly: "GPT-o1 only reaches adult-level reasoning in the color domain" (Section 5).

The paper reports a novel correlational finding in Figure 6: GPT-o1's error scores on visual extrapolation correlate positively with both children's error scores and adults' response times, broken down by specific transformation subdomain. This means that specific transformations that children find hard (e.g., −2 subtraction, 180° rotation) are also the ones GPT-o1 finds hard, and transformations that take adults longer to process (higher response times) are also harder for GPT-o1. The paper states (Section 4.1): "What is cognitively demanding to humans is also more computationally challenging for GPT-o1."

Model Hallucination on No-Change Trials

For each transformation domain, 10% of trials involve no change at all — the "before" and "after" images are identical. The results, presented in Figure 7, reveal a sharp model-dependent pattern:

GPT-o1: Correctly selects "no change" in both verbal classification and verbal specification across all visual domains. However, it "struggles to extrapolate this to new objects when distractors involve reflection or number change" (Section 4.1) — meaning that even when it knows nothing changed, it fails to identify that the test object should also remain unchanged when distractors involve transformations in the harder domains.
GPT-4V: Only accurately identifies "no change" in the verbal classification stage for the size domain. When it does correctly classify a trial as no-change, it consistently specifies that no change is involved (the orange bars in Figure 7 are tall for domains where classification succeeds), but extrapolation remains problematic.
LLaVA-1.5 and MANTIS: "Hallucinate a change in 100% of the no-change trials during verbal classification" (Section 4.1). They consistently fail to detect the absence of any transformation. Their visual extrapolation on no-change trials is no better than chance, even though they are occasionally correct by random guessing.

This is a striking finding: the two open-source models cannot perceive the absence of change at all, GPT-4V can detect it only for certain domains, and GPT-o1 can detect it reliably but cannot reliably extrapolate the no-change rule to novel objects when distractors involve transformations in harder domains.

Model Response Inconsistency

Figure 8 presents a heat map of choice inconsistency — the proportion of repeated identical trials (out of three repetitions) where models selected different responses. Key findings:

All models are most consistent in Verbal Classification and least consistent in Visual Extrapolation, suggesting that the extrapolation step involves genuine uncertainty rather than systematic errors.
GPT-o1 and GPT-4V are substantially more consistent than LLaVA-1.5 and MANTIS across all domains and question types.
Inconsistency is highest for number, rotation, and reflection domains — the same domains where absolute accuracy is lowest. This convergence suggests that models are not just wrong in these domains; they are unreliably wrong, producing different answers to the same stimuli on different runs.

The paper does not report exact inconsistency percentages per domain-model combination (the heat map in Figure 8 uses color intensity rather than numeric labels), but the visual pattern shows GPT-o1 with the darkest cells (least inconsistency) and LLaVA-1.5/MANTIS with the lightest (most inconsistency), particularly at the visual extrapolation stage.

Impact of Verbal Reasoning Questions on Visual Extrapolation

To assess whether the verbal classification and specification questions improve or interfere with visual analogical reasoning, the paper tests a visual-extrapolation-only variant of KiVA — removing the first two verbal stages — on 200 additional adults, 20 children, and GPT-o1, focusing on the three harder domains (number, rotation, reflection). Results are reported in Section 4.1 and Figure 9:

Human adults: Accuracy remains similar with and without verbal questions, but response times are significantly slower without the verbal questions. The paper interprets this as the verbal questions facilitating more efficient reasoning: adults reason about what changed and how implicitly, but making it explicit speeds up the mapping process.
Human children: Accuracy is worse without verbal questions. The paper states that children "performed worse in extrapolation" when the verbal scaffolding was removed, suggesting that young children benefit from explicitly articulating what changed and how before attempting extrapolation.
GPT-o1: The effects are mixed and domain-dependent. GPT-o1 "performed equally well in the number domain, it is better at extrapolating object rotations but worse at extrapolating reflections when asked to reason about what changed and how it changed beforehand" (Section 4.1). The paper speculates that "reasoning models like GPT-o1 already reason about 'what changed' and 'how it changed' independently of our verbal queries" (Section 4.1), which would explain why adding explicit verbal questions does not consistently help and may even interfere for some domains.

Multi-Image vs. Single-Image Presentation

Appendix B.1 (Figure 11) reports an ablation on GPT-o1 comparing multi-image presentation (given transformation and three candidate extrapolations as four separate images, as recommended by Campbell et al., 2025) against single-image presentation (all transformations stitched into one composite image, as provided to LLaVA-1.5). Results:

Multi-image presentation significantly improves visual extrapolation accuracy for color, size, and number domains. The improvement magnitudes are visible in Figure 11 as differences between the two bar sets for these domains.
Multi-image presentation does NOT improve performance for rotation and reflection. The bars for these two domains overlap between the two presentation formats.

The paper interprets this as evidence that "challenge in the latter two domains goes beyond a visual binding problem described in Campbell et al. (2025)" (Appendix B.1). Even when visual binding errors are eliminated through separate image presentation, rotation and reflection analogies remain difficult — suggesting the bottleneck is in the spatial reasoning itself, not in perceptual confusion across composite layouts.

Prompt Engineering and In-Context Learning Do Not Help

Section 4.1 reports that four text-based prompting strategies and two visual prompting strategies were tested, and "none of these approaches improve performance." The specific strategies (detailed in Appendices A.4–A.6) are:

Reasoning through code (Sharma et al., 2024): Generating Python/Pillow code for each transformation.
Reasoning after Reflection (Valmeekam et al., 2023): Self-critique with three revision iterations per question.
Reasoning through instruction (Wei et al., 2022): Generating step-by-step instructions before answering.
In-Context Learning (Dong et al., 2022): Two randomly sampled examples with solutions per concept.
Visual: Background color (Bai et al., 2023): Altering transparent background to black or white.
Visual: Image size (Bar et al., 2022): Center crop varying image size between 0.9 and 1.0.

The paper does not report exact accuracy numbers for each prompting strategy — they are summarized with the blanket statement of no improvement. This is a significant negative result: it suggests that the difficulty of KiVA is not a prompting problem that can be solved by better elicitation of latent capabilities. The models genuinely struggle with the visual analogical reasoning task itself, not with expressing their reasoning.

Large Vision Model Performance (Without Language)

Appendix B.4 (Figure 14b) reports results from a Large Vision Model (Bai et al., 2024) trained entirely on visual data without any linguistic supervision. Because it has no text interface, it cannot be tested on the three-stage protocol. Instead, it is evaluated via visual completion: given the source transformation and the left side of each candidate extrapolation, it predicts the right side, and the candidate with lowest perplexity is chosen.

Key results from Figure 14b:

The Large Vision Model performs above chance on color and size extrapolation, with accuracy visibly above the 33% chance line.
It performs at or below chance on number, rotation, and reflection extrapolation.

This pattern — easier for surface features (color, size), harder for spatial and numerical transformations — mirrors the pattern observed in LMMs (Figure 4). The paper interprets this as evidence that "even in the absence of any language to reason about what changed, how it changed, and how to extend the change to a new object, the large vision model can solve some visual analogies" (Appendix B.4) — but only the simpler ones. The difficulty of spatial and numerical analogies is not specific to language-equipped models; it appears to be a more fundamental challenge for current vision architectures.

Conditional Performance: Extrapolation Given Prior Verbal Success or Failure

Figure 12 (Appendix B.2) reports models' visual extrapolation accuracy conditional on whether the preceding verbal reasoning was correct or incorrect, broken down by transformation domain.

GPT-o1: Extrapolation accuracy is substantially higher when prior verbal classification or specification was correct (green bars) compared to when it was incorrect (red bars), across all domains. This means that for GPT-o1, successful extrapolation is contingent on correctly identifying what and how the object changed. The difference between green and red bars is particularly large for size and color domains — when GPT-o1 correctly identifies a size change, it nearly always extrapolates correctly; when it misclassifies the change, extrapolation drops toward chance.
GPT-4V: Shows the conditional dependence for color and size domains but not for number, rotation, or reflection — in the harder domains, extrapolation accuracy is similarly low regardless of whether verbal reasoning was correct.
LLaVA-1.5 and MANTIS: Show "no conditional dependence of prior verbal reasoning on subsequent visual extrapolation" (Appendix B.2). Their extrapolation accuracy is near chance regardless of whether they correctly classified or specified the transformation — further evidence that these models are not solving the task systematically.

The paper notes an important caveat: "verbal specification is only asked if verbal classification is correct" (Figure 12 caption), so the conditional-on-specification analysis applies only to the subset of trials where classification was already successful.

Ablation Studies and Robustness Checks

Multi-image vs. single-image presentation format (GPT-o1 only): Multi-image presentation significantly improves visual extrapolation for color, size, and number domains, but provides no benefit for rotation and reflection, indicating that challenges in spatial domains go beyond visual binding errors (Appendix B.1, Figure 11).

Visual-extrapolation-only variant (removing verbal questions): Removing the verbal classification and specification stages does not harm adult accuracy but significantly increases adult response times and reduces child accuracy; effects on GPT-o1 are mixed — neutral for number, positive for rotation, negative for reflection — suggesting that explicit verbal reasoning about the transformation rule interacts differently with model reasoning than with human reasoning (Section 4.1, Figure 9).

Conditional extrapolation given correct prior verbal reasoning: GPT-o1's visual extrapolation accuracy is substantially higher when its preceding verbal reasoning is correct across all domains, while LLaVA-1.5 and MANTIS show no such conditional dependence, indicating that the latter models are not systematically linking verbal and visual reasoning (Appendix B.2, Figure 12).

Extrapolation given experimenter-provided correct verbal specification: Supplying the ground-truth transformation rule does not substantially improve model extrapolation performance — for GPT-o1, extrapolation with correct specification given reaches approximately 55–60% for number and 40–60% for rotation and reflection — demonstrating that the primary failure is in mapping the known rule to a novel object, not in discovering the rule (Appendix B.3, Figure 13b).

No-change trials (control for hallucination): LLaVA-1.5 and MANTIS hallucinate a change in 100% of no-change trials; GPT-4V detects no-change only in the size domain at classification; GPT-o1 correctly identifies no-change at verbal stages but struggles to extrapolate no-change to novel objects when distractors involve reflection or number change (Section 4.1, Figure 7).

Choice inconsistency across repeated trials: All models are least consistent in visual extrapolation and most consistent in verbal classification; inconsistency is highest in number, rotation, and reflection domains; GPT-o1 and GPT-4V are substantially more consistent than LLaVA-1.5 and MANTIS across all conditions (Section 4.1, Figure 8).

Text-based prompt engineering (four methods): Reasoning through code, reflection-based self-critique, instruction generation, and in-context learning all fail to improve model performance, indicating that difficulty is not due to suboptimal prompting (Section 4.1, Appendices A.4–A.6; exact accuracy numbers not reported for each method).

Visual prompt engineering (two methods): Altering background color (transparent → black/white) and varying image size via center crop both fail to improve model performance (Section 4.1; exact numbers not reported).

Large Vision Model (no language) as control: A model trained without linguistic data performs above chance on color and size extrapolation but at or below chance on number, rotation, and reflection, indicating that spatial and numerical reasoning challenges are not artifacts of the verbal query format but reflect fundamental architectural limitations (Appendix B.4, Figure 14b).

Developmental validation of KiVA-adults: An additional 10 children tested on KiVA-adults performed no better than chance, empirically validating that KiVA-adults genuinely exceeds the capabilities of children under five and operates as a valid adult-level tier (Section 4).

Correlation between GPT-o1 errors and human difficulty measures: GPT-o1's error scores correlate positively with both children's error scores and adults' response times across specific transformation subdomains, suggesting that the relative difficulty of transformations is consistent across biological and artificial systems (Section 4.1, Figure 6).

Critical Assessment

The experiments reported in this paper provide convincing evidence for a specific and well-demarcated set of claims, but several important limitations temper the generality and strength of the conclusions.

Claim: "LMMs do not match even the capabilities of a three-year-old child in reasoning about number and reflection." This claim is directly supported by the data in Figure 4 (top panel), where children perform substantially above chance on number and reflection extrapolation while GPT-o1 — the strongest tested model — performs at or near chance in these domains. However, the comparison warrants careful qualification. Children completed only 10 trials each (2 per domain), with the task administered in-person by experimenters using simplified language, while models completed hundreds of trials via API with standardized prompts. The child data represents 420 total responses from 42 children — a modest sample size. More importantly, children's motivation and engagement in an in-person gamified setting (coins for stickers) differs fundamentally from model evaluation. The claim that LMMs are "below three-year-old level" on specific domains is substantiated by the data shown but should be understood as a claim about performance on this specific benchmark under these specific conditions, not necessarily about generalizable cognitive capability. A child who fails KiVA number trials might still competently count cookies in a real-world context; a model that fails KiVA number trials might still succeed at other numerical reasoning tasks.

Claim: "Performance declines monotonically from verbal description to visual extrapolation." This is consistently observed across models and domains (Figures 4, 5), with the important nuance that GPT-o1 maintains relatively high verbal specification accuracy in color and size but still drops at extrapolation. The monotonic decline is genuine, but the paper's interpretation — that this represents a fundamental difficulty with analogical mapping — is complicated by the fact that the three stages are not independent in difficulty. The verbal specification stage inherently requires finer discrimination than classification (distinguishing +1 from +2 rather than distinguishing number change from color change), and the extrapolation stage requires both the verbal reasoning of the prior stages and the additional mapping step. The decline could partially reflect increasing task difficulty rather than a specific mapping deficit. The conditional analyses in Figure 12 partially address this by showing that extrapolation is low even when specification is correct, but a cleaner decomposition would require controlling for the inherent difficulty of each stage independently.

Claim: "What is cognitively demanding to humans is also more computationally challenging for GPT-o1." The correlation evidence in Figure 6 supports this claim, but the strength of the correlation is not quantified with a correlation coefficient or p-value in the main text — the figure shows a scatter plot with a positive trend, but the paper only states that there is a "positive correlation." The specific subdomains plotted in Figure 6 (labeled with transformation types like "Red," "Blue," "+1," "+2," "X," "Y," "180°," etc.) appear to cluster by domain, which could inflate the correlation if domains are treated as independent points despite sharing underlying difficulty factors. Additionally, the correlation with adult response times comes from the visual-extrapolation-only variant (Figure 9), not the full three-stage protocol, meaning the response time data and the GPT-o1 error data come from slightly different experimental conditions.

Claim: "Prompt engineering and in-context learning did not improve model performance." This claim is reported but not quantitatively substantiated — the paper states that none of the six prompting strategies improved performance but does not provide accuracy numbers for each strategy, making it impossible to assess whether improvements were genuinely zero or merely statistically insignificant at the sample size. This is a surprising omission given the strength of the claim. For a benchmark paper that is otherwise thorough in reporting quantitative results, the absence of a table comparing prompting strategies is a genuine weakness. A reader cannot determine whether chain-of-thought prompting improved rotation extrapolation from 45% to 47% (a small but real gain) or left it unchanged.

Missing experiment: Combining the best conditions. The paper tests multi-image presentation (which helps GPT-o1), verbal scaffolding (which helps children but not consistently GPT-o1), and various prompting strategies independently, but does not report a comprehensive "best-case" configuration combining all favorable conditions. What accuracy does GPT-o1 achieve on KiVA when given multi-image presentation, correct verbal specification from the experimenter, and chain-of-thought reasoning? This upper-bound estimate would clarify whether the failures are absolute (the model genuinely cannot do the mapping under any conditions) or can be partially mitigated through optimal experimental design.

Missing experiment: Fine-tuning on the transformation domains. The paper tests off-the-shelf models with prompting but does not explore whether models can learn to perform visual analogies when fine-tuned on transformation data. Given that humans learn analogical reasoning through experience with object transformations (Christie & Gentner, 2010; Goddu et al., 2020), it would be informative to know whether LMMs can acquire these capabilities through domain-specific training even if they lack them in zero-shot evaluation. The code release enabling "infinite expansion of the benchmark" (Section 3.2) makes such fine-tuning experiments feasible, but they are not attempted.

Limitation: Single benchmark, no cross-domain validation. All results come from KiVA/KiVA-adults. While the benchmark is carefully constructed, the paper does not validate its findings against any external measure of analogical reasoning — for instance, correlating KiVA performance with ARC or RPM performance to establish that it measures the same underlying construct. The claim that KiVA tests "visual analogical reasoning" rather than a narrower set of visual discrimination skills would be strengthened by showing that KiVA performance predicts performance on established analogical reasoning benchmarks.

Limitation: Child and adult sample sizes and representativeness. The child sample (42 children, 420 responses) is adequate for demonstrating above-chance performance and domain-level trends, but the between-domain comparisons — for instance, the claim that "children perform slightly worse on rotation compared to other domains" — are based on small per-domain samples (~84 responses per domain, 2 trials per child per domain). The standard errors in Figure 4 (top) reflect variance across object variations, not across children. A larger and more diverse child sample would strengthen the developmental comparisons. The adult sample (250 participants, 3–13 annotators per trial) is more robust but still modest for a benchmark intended to establish human performance ceilings.

Limitation: The Large Vision Model comparison is suggestive but incomplete. While the Large Vision Model result (Appendix B.4) is interesting as a control for language effects, the visual completion task it performs is fundamentally different from the multiple-choice task administered to LMMs and humans. The Large Vision Model must generate pixel-level predictions; LMMs select among pre-rendered options. These are not equivalent tasks, and the comparison should be interpreted as qualitative evidence that language is not strictly necessary for some visual analogies rather than as a controlled experiment isolating the role of language.

What the experiments do demonstrate, robustly: (1) Current LMMs show a consistent and substantial gap compared to human children and adults on visual analogical reasoning tasks involving real-world objects, with the gap concentrated in spatial and numerical domains. (2) The three-stage decomposition reveals that models can often detect that a change occurred (classification) but progressively fail at quantifying the change (specification) and applying it to new objects (extrapolation). (3) The difficulty ordering across domains — color/size easier than rotation/reflection/number — is consistent across models, humans, and even a language-free vision model, suggesting this reflects intrinsic task difficulty rather than architectural peculiarities. (4) No tested prompting or presentation strategy closes the human-model gap, and the failure persists even when models are given the correct transformation rule explicitly. These findings collectively establish that visual analogical reasoning of the kind young children perform remains a significant challenge for state-of-the-art LMMs.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Makes the Full KiVA Protocol Impractical for Automated Deployment at Scale

The assumption or constraint. The human evaluation protocol — three-stage querying with conditional branching, in-person child administration by trained experimenters, and performance-contingent compensation — is deliberately designed for controlled experimental comparison, not for efficient benchmarking. The paper does not claim otherwise, but the practical consequence is worth articulating explicitly: KiVA cannot be deployed as an automated evaluation pipeline in the way that, for example, MMLU or VQA-v2 can be. Human children cannot be tested via API. Adults require compensation, screening, and quality control via Prolific. The three-stage protocol with conditional branching (verbal specification only asked if classification is correct) requires real-time decision logic that is trivial for human experimenters but constitutes a custom evaluation harness for models. The paper releases code and data (Section 1, footnote 1), but the human baseline data — the defining feature that distinguishes KiVA from prior benchmarks — cannot be reproduced by a third party without replicating the full human subjects protocol, including IRB approval, participant recruitment, and in-person child testing.

The consequence. KiVA's primary value proposition is its developmental grounding — the ability to say "GPT-o1 performs below three-year-old level on number analogies." But this comparison depends on the human data the paper collected. A researcher who wants to benchmark a new model against KiVA's human baselines must either (a) trust the published human data and compare their model's scores to the paper's reported means, or (b) replicate the human study at significant cost and logistical complexity. Option (a) is what most users will do, but it introduces a methodological asymmetry: the model is tested under the researcher's specific conditions (prompt format, temperature, hardware), while the human baseline was collected under the paper's specific conditions (JsPsych interface, Prolific recruitment, in-person child testing). Drift in model evaluation protocols — a well-documented problem in NLP benchmarking — could produce apparent "improvements" that reflect changes in evaluation rather than genuine capability gains. The paper's child data (42 children, 420 responses) is particularly vulnerable here: a subsequent study with a slightly different child demographic or experimental setup might produce different human baselines, making "model X surpasses KiVA child performance" a fragile claim.

What evidence exists in the paper. The paper is transparent about its human data collection (Section 4): 250 adults on Prolific, 42 children from early childhood centers and ChildrenHelpingScience, IRB approval (protocol 2020-10-13755), and per-trial annotation densities of 3–13 adults and 2 trials per domain per child. The paper also releases the benchmark code and data. However, it does not provide a streamlined "model-only" evaluation protocol that would allow fully automated benchmarking against the published human baselines without re-implementing the conditional administration logic. The three-stage conditional structure (Figure 3) — where Stage 2 is only asked if Stage 1 is correct, and Stage 3 is always asked — is described clearly but requires custom evaluation code beyond standard VQA evaluation harnesses.

Mitigation status. Not addressed. The paper focuses on the scientific contribution of the human-model comparison and releases the dataset, but does not discuss the practical challenge of maintaining this comparison as models evolve. Future work would need to establish a standardized evaluation server or frozen test set with published human reference scores to make KiVA a sustainable benchmark in the way that, e.g., ImageNet or MMLU have become.

Single Benchmark Domain (Everyday Object Transformations) Limits Claims About "Visual Analogical Reasoning" as a General Capability

The assumption or constraint. The paper operationalizes visual analogical reasoning as the ability to infer transformation rules (color, size, number, rotation, reflection) from image pairs of everyday 3D objects and apply them to novel objects. This is a specific, well-motivated instantiation grounded in developmental psychology (Goddu et al., 2020). However, it is one instantiation among many possible forms of visual analogical reasoning. The paper acknowledges this scope limitation implicitly by focusing on transformations that "young children can solve" (Section 3.2) and by noting that other visual properties like "depth, spatial compositionality, and physical affordances" are excluded. But the paper's claims — particularly "LMMs still cannot solve visual analogies like humans can" (Section 5) — are stated in general terms that extend beyond the specific transformation types tested.

The consequence. A model that performs poorly on KiVA might fail specifically at mental rotation of 2D rendered objects, or at counting small sets of identical object instances, or at discriminating 90° from 180° rotations in the picture plane — without necessarily lacking "visual analogical reasoning" in a broader sense. The paper's domain-level difficulty ordering (color/size easier than rotation/reflection/number) is internally consistent and cross-validated against human data, but it is unknown whether this ordering generalizes to other forms of visual analogy. For instance, would models also struggle with analogies involving texture changes, transparency, occlusion, lighting, or object part relations? Would the human developmental trajectory for these other transformation types show the same convergence with model difficulty? Without testing on a broader set of visual analogy types, the paper's conclusion that models lack "visual analogical reasoning" could be partly an artifact of selecting transformation domains that happen to expose specific weaknesses in current vision architectures — particularly those relying on 2D image features rather than 3D spatial understanding.

What evidence exists in the paper. The paper explicitly restricts itself to five domains and acknowledges the exclusion of depth, spatial compositionality, and physical affordances (Section 3.2). The consistency of the difficulty ordering across models, children, adults, and even a language-free vision model (Figure 4, Figure 5, Figure 14b) provides some evidence that the measured difficulty reflects genuine task properties rather than arbitrary benchmark design. However, this consistency is entirely within the five tested domains — there is no external validation against other analogical reasoning benchmarks (ARC, ConceptARC, RPMs) that would establish convergent validity. A model that fails KiVA but passes RPMs (or vice versa) would suggest these benchmarks measure different underlying constructs, complicating claims about "visual analogical reasoning" as a unitary capability.

Mitigation status. Partially addressed through the planned KiVA-compositionality extension (Section 5), which will test "multiple object transformations to probe even more complex compositional reasoning." This extends within the same paradigm rather than validating against external benchmarks. No cross-benchmark validation is performed or proposed.

The Child Baseline Sample Is Modest and May Not Support Fine-Grained Between-Domain Comparisons

The assumption or constraint. The paper's headline finding — that LMMs "do not match even the capabilities of a three-year-old child in reasoning about number and reflection" (Section 1) — depends on the reliability of the child performance estimates. The child sample consists of 42 children aged 3–5 years (mean = 4.07 years, se = 0.11 years), each completing 10 randomly sampled trials (2 per transformation domain), yielding 420 total responses (Section 4). This means approximately 84 child responses per domain, aggregated across children who saw different random subsets of trials within each domain. The standard errors reported in Figure 4 (top) represent variance across object variations within each domain, not across children — a legitimate choice for benchmark reporting but one that does not reflect uncertainty in the child population mean.

The consequence. The between-domain comparisons for children — for instance, "children perform slightly worse on rotation compared to other domains" (Section 4.1) — are based on domain-level aggregates with approximately 84 responses per domain. At this sample size, the confidence intervals around child accuracy estimates are substantially wider than the adult error bars (based on 3–13 annotators per trial across 250 adults). A domain-level accuracy difference of, say, 70% vs. 80% between rotation and color (reading approximately from Figure 4 top panel) may or may not be statistically significant given the sample. The paper does not report formal statistical tests (t-tests, ANOVAs, or confidence intervals at the domain level) for the child data. This weakens the claim that children show a specific pattern of domain-dependent difficulty (color ≈ size > rotation > number ≈ reflection) rather than roughly equivalent performance across domains with noise-driven variation. The adult data, by contrast, shows near-ceiling performance with minimal variance, making the domain-level pattern uninformative for adults.

The developmental comparison with models is also affected. When the paper states that GPT-o1 is "below three-year-old level" on number extrapolation, this claim compares a point estimate from 84 child responses in the number domain against GPT-o1's performance aggregated across 300+ number trials (100 per subdomain × 3 number subdomains in KiVA, each run 3 times). The child data may have genuine variance that makes the "three-year-old level" a range rather than a point — and GPT-o1 might fall within that range rather than clearly below it, depending on the confidence interval. The paper does not provide the data needed to assess this.

What evidence exists in the paper. The paper reports the child sample characteristics (Section 4) and presents child performance in Figure 4 (top) with error bars described as "standard errors across object variations." The paper states that children performed "significantly-above-chance" in verbal classification and verbal specification (Section 4.1), suggesting some statistical testing was performed, but these tests are not reported in detail. The KiVA-adults validation — "an additional 10 children tested on KiVA-adults... none performed better than chance" (Section 4) — uses a very small sample (10 children) to establish the developmental gate.

Mitigation status. Not addressed. The paper does not discuss sample size limitations for the child data or acknowledge the uncertainty in between-domain comparisons. The standard errors plotted in Figure 4 (top) give a visual sense of variation across stimuli but not across children. A future replication with a larger child sample, or a meta-analysis combining this data with other developmental studies of analogical reasoning, would strengthen the developmental baselines.

The Difficulty Gap Between Classification and Extrapolation May Partially Reflect Increasing Task Discrimination Difficulty Rather Than a Specific Mapping Deficit

The assumption or constraint. The paper's three-stage protocol is structured such that each successive stage requires finer discrimination within the same visual domain: Stage 1 (classification) asks the participant to distinguish among five broad transformation categories (e.g., "change in color" vs. "change in size" vs. "change in orientation"), Stage 2 (specification) asks them to distinguish among 2–5 subdomain variants within the correctly-identified category (e.g., "rotate 90° clockwise" vs. "rotate 180°"), and Stage 3 (extrapolation) asks them to identify which of three candidate transformations — each involving a different novel object — matches the given rule. The paper interprets the monotonic decline in accuracy from Stage 1 to Stage 3 as evidence that models can detect changes but fail at analogical mapping. However, an alternative (or complementary) interpretation is that each stage is simply a harder perceptual discrimination task, independent of analogical reasoning demands.

The consequence. If the performance decline is largely driven by increasing perceptual discrimination difficulty — distinguishing 90° from 180° rotation is harder than distinguishing rotation from color change, regardless of whether analogy is involved — then the paper's central diagnostic claim (that the failure is specifically in analogical mapping) is partially confounded. The paper's control experiment in Appendix B.3 partially addresses this: when models are given the correct verbal specification and still fail at extrapolation, this isolates mapping as a bottleneck. However, the control shows that given the correct specification, extrapolation accuracy does improve (Figure 13b) — from approximately 40–50% to approximately 55–60% for number, for instance — just not to ceiling. This means part of the extrapolation deficit genuinely is a specification problem (models cannot identify the rule precisely enough), and part is a mapping problem (even with the rule, they cannot apply it). The paper does not quantify the relative contribution of these two failure sources.

Additionally, the extrapolation stage involves a different object from the given transformation, which introduces a perceptual dissimilarity that is not present in the classification or specification stages (which only show the given transformation). A model might correctly specify a rotation rule when looking at the given pair but then fail at extrapolation because it cannot perceptually align the novel test object with the source object to determine what a 90° rotation would look like for that specific shape. This is indeed a mapping failure — but it is a failure of perceptual mapping (what does this specific object look like when rotated?) rather than relational mapping (what is the abstract rule?). The current protocol does not cleanly separate these.

What evidence exists in the paper. The conditional performance analyses in Figure 12 show that GPT-o1's extrapolation accuracy is higher when prior verbal reasoning is correct — consistent with the hypothesis that part of the extrapolation failure is downstream of specification failure. The control experiment in Figure 13b shows that giving the correct specification helps but does not bring models to human-level extrapolation — consistent with a genuine mapping deficit that goes beyond specification. However, the paper does not include a control that isolates perceptual mapping difficulty: for example, asking models to perform a 90° rotation on a novel object without the analogical framing (i.e., as a direct instruction rather than as an inferred rule). If models also fail at direct instruction-based transformations, the difficulty would be perceptual rather than analogical.

Mitigation status. Partially addressed through Figure 13b, but the relative contributions of specification failure and mapping failure to the extrapolation deficit are not quantified. The paper acknowledges this implicitly by stating that "part of the models' failure in analogical reasoning is an inability to correctly recognize the given transformation" and "another part of the model's failure lies in extrapolating the correctly identified transformation to a novel object" (Section 4.1), but does not estimate the magnitude of each part.

The Paper's Negative Prompt Engineering Results Are Unquantified, Making It Impossible to Assess Whether Improvements Were Truly Absent or Merely Small

The assumption or constraint. Section 4.1 reports that four text-based prompting strategies (code generation, reflection-based self-critique, instruction generation, in-context learning) and two visual prompting strategies (background color, image size) were tested, and that "none of these approaches improve performance, which points to the challenging nature of our benchmark." This is a strong claim — that KiVA is robust to prompt engineering and that the measured difficulty reflects genuine capability limitations rather than suboptimal evaluation design. However, the paper provides no quantitative results for any of these prompting experiments. No table, no figure, no accuracy numbers per strategy. The claim is asserted in a single sentence with no supporting evidence presented to the reader.

The consequence. A reader cannot evaluate whether "no improvement" means: (a) accuracy was identical to three decimal places, (b) accuracy changed by +1–2% but the difference was not statistically significant, (c) some strategies improved certain domains while hurting others (producing zero net improvement), or (d) some strategies caused substantial degradation that offset improvements. Each of these scenarios has different implications. Scenario (a) would strengthen the paper's claim that the difficulty is fundamental. Scenario (b) would suggest that prompting matters slightly but is not the bottleneck — a weaker but still valid claim. Scenario (c) would suggest that optimal prompting might be domain-specific, which is a different conclusion. Scenario (d) would suggest that the negative result is partly an artifact of averaging and that some strategies might genuinely help for specific domains.

The absence of quantitative data is particularly notable because the paper is otherwise thorough in reporting numbers for its main experiments (Figures 4, 5, 7, 8, 9, 11, 12, 13, 14). For a paper whose contribution is a benchmark with diagnostic precision, omitting the numbers for the prompt engineering experiments — experiments directly testing whether the benchmark's difficulty is robust — is a significant evidentiary gap. The appendices (A.4–A.6) provide the prompt templates but not the results of using them.

What evidence exists in the paper. The paper provides the prompt text for the four text-based strategies in Appendices A.4–A.6 and mentions the two visual strategies in Section 4.1. No quantitative results are reported for any of them. The claim "none of these approaches improve performance" appears in Section 4.1 without a supporting table or figure reference.

Mitigation status. Not addressed. This is a straightforward omission that could be remedied with a table in the appendix reporting accuracy per strategy per domain for the three evaluation stages. Without this data, the claim about prompt robustness is unsubstantiated, and a practitioner attempting to deploy KiVA evaluation cannot know whether their own prompting strategy might produce different results.

The Benchmark Tests 2D Image Transformations of 3D Objects, Which May Penalize Models That Lack Robust 3D Understanding Rather Than Models That Lack Analogical Reasoning

The assumption or constraint. KiVA renders 3D object models as 2D images and applies transformations in the 2D image plane: rotation is in-plane (picture-plane rotation), reflection is across the image's x-axis or y-axis, size changes are 2D scaling operations, and color changes are RGB value modifications. These are well-defined image operations, but they do not necessarily correspond to how these transformations would appear in the physical world. A 90° in-plane rotation of a 2D rendering of a 3D object is not the same as a 90° rotation of the physical object in 3D space — the latter would produce a different 2D projection with different occlusion patterns, perspective effects, and surface visibility. The paper acknowledges this implicitly by describing the transformations as 2D operations (Section 3.2: "2D rotation by 90 degrees") but does not discuss the implications for models that may have learned to reason about objects in 3D rather than 2D image coordinates.

The consequence. An LMM trained on natural images and text that describe 3D object interactions might have developed internal representations that encode object rotations in 3D rather than 2D. When shown a 2D in-plane rotation of a chair — which, in the physical world, would correspond to rotating the camera around the chair's vertical axis while keeping the chair fixed — the model might interpret this as a 3D rotation and search for a matching 3D rotation among the candidate extrapolations. But the candidate transformations are also 2D in-plane rotations, not 3D rotations in depth. This creates a potential representational mismatch: the model is trying to map a 3D transformation concept onto a 2D transformation task, and failing because the task constraints (2D picture-plane manipulation) don't match the model's learned knowledge (3D object manipulation). If this interpretation is correct, the model's failure on rotation and reflection would reflect not a lack of analogical reasoning per se, but a mismatch between the benchmark's 2D operationalization and the model's 3D world knowledge — a confound rather than a genuine capability deficit.

This concern is partially supported by the paper's own data: the Large Vision Model (Appendix B.4), trained entirely on 2D visual data without linguistic descriptions of 3D object properties, shows the same domain difficulty ordering as the LMMs. This suggests that the difficulty of rotation and reflection analogies is present even for models that have no explicit 3D knowledge to create a representational mismatch. However, the Large Vision Model might also have internal representations that partially encode 3D structure from 2D visual statistics (as deep networks trained on ImageNet are known to do), so the confound cannot be ruled out.

What evidence exists in the paper. The paper describes the transformations as 2D operations (Section 3.2: "featuring 2D rotation by 90 degrees"; "reflected along the x-axis or y-axis"). The object models are 3D (from Downs et al., 2022 and Stojanov et al., 2021), but the paper does not specify the rendering viewpoint or whether the viewpoint is held constant across transformations. It does not discuss potential 2D/3D representational mismatches. The stimulus examples in Appendix A.1 (Figure 10) show objects rendered from a canonical viewpoint with no perspective distortion visible, suggesting orthographic or weak-perspective rendering — but this is not stated explicitly. The Large Vision Model results (Figure 14b) provide partial evidence against the 2D/3D confound hypothesis, but are not designed as a control for it.

Mitigation status. Not addressed. The paper does not discuss the 2D nature of the transformations as a potential limitation or how it might interact with models' 3D understanding. Future work could address this by (a) explicitly testing 3D rotation in depth as a separate domain, (b) comparing 2D in-plane transformations to 3D in-depth transformations to see whether models perform differently, or (c) using rendered views that make the 2D nature of the transformation unambiguous (e.g., by including a visible picture frame or canvas boundary that defines the transformation plane).

7. Implications and Future Directions

How This Work Changes the Landscape

This paper represents a methodological reframing rather than a paradigm shift — it does not introduce a new model architecture, training objective, or reasoning algorithm, but rather changes how the field evaluates visual intelligence by grounding benchmarks in developmental psychology and decomposing reasoning into diagnosable subprocesses. The magnitude is significant for evaluation methodology but indirect for model development: the paper provides a new instrument for measuring progress rather than progress itself.

The most consequential conceptual shift is moving from "can models solve hard puzzles?" to "where do models fall on a human developmental trajectory?" Prior visual reasoning benchmarks — ARC (Chollet, 2019), ConceptARC (Moskvichev et al., 2023), Raven's Progressive Matrices variants (Huang et al., 2024; Ahrabian et al., 2024) — asked whether models could match or approach adult human performance on abstract reasoning tasks. This framing treats human cognition as a single reference point (adult-level) and measures the distance to it. KiVA replaces this with a developmental ruler: the benchmark is calibrated against what three-year-olds can do (KiVA) and what adults but not young children can do (KiVA-adults), creating a cognitive timeline along which model capabilities can be placed. When the paper reports that GPT-o1 falls below three-year-old performance on number and reflection extrapolation (Figure 4, top panel), it is making a claim that is interpretable in terms of human cognitive development, not just as an abstract accuracy deficit. This shifts the conversation from "models are X% below human ceiling" to "models have not yet reached developmental milestone Y," which carries different implications for what kinds of interventions are likely to help: developmental psychology suggests that these abilities are built through interaction with the physical world and experience with object transformations, not merely through more passive observation of static images.

The paper reconciles a latent contradiction in the literature between perception benchmarks and reasoning benchmarks. On one side, models perform impressively on object recognition, property detection, and visual question answering — tasks that suggest they "see" well. On the other side, they struggle on abstract reasoning puzzles like ARC and RPMs — tasks that suggest they cannot "reason." The field has lacked a bridging benchmark that tests whether models can reason about the kinds of visual changes that matter for everyday cognition. KiVA fills this gap by showing that models can detect changes (high verbal classification accuracy in Figure 4) but cannot perform the analogical mapping that constitutes reasoning about those changes (low visual extrapolation accuracy, especially in spatial domains). This reconciles the apparent contradiction: models are good at the perceptual encoding that underlies both object recognition and change detection, but poor at the relational mapping that underlies analogical reasoning. The failure is specifically in applying known transformations to novel objects, not in perceiving the transformations — a distinction that prior benchmarks, which only measure end-to-end accuracy, could not isolate.

The convergence in difficulty patterns across systems — GPT-o1, the Large Vision Model, children, and adult response times (Figure 6, Figure 14b) — challenges the "scale is all you need" assumption in a specific, empirically grounded way. If difficulty were purely an artifact of insufficient training data or model scale, we would not expect GPT-o1 (trained on internet-scale data with chain-of-thought reasoning) to show the same domain ordering as a language-free vision model trained on a completely different objective, or as human children whose visual experience is grounded in 3D physical interaction. The consistency suggests that certain visual analogies — particularly those requiring mental rotation, sequential tracking of quantities, and discrimination of chiral reflections — are intrinsically harder computational problems. This does not mean scaling cannot help, but it does mean that scaling alone, without architectural innovations that address spatial and relational reasoning, is unlikely to close the gap. The paper makes this point implicitly through its data rather than through explicit argument, but the implication is clear: the research agenda should include work on how models represent and manipulate spatial relations, not just how much data they are trained on.

The paper redirects attention toward testing for "analogical mapping" as a distinct capability from "visual perception." By showing that giving models the correct transformation rule (Appendix B.3, Figure 13b) does not bring extrapolation to human levels — GPT-o1 reaches only ~55–60% on number and ~40–60% on rotation and reflection even with the rule provided — the paper establishes that the bottleneck is not in discovering the rule but in applying it to a novel object. This has implications for how future models should be evaluated and trained: a model that performs well on visual property detection benchmarks may still lack the relational mapping capability that KiVA isolates. Benchmarks that conflate perception and reasoning will overestimate the reasoning capabilities of models with strong perceptual abilities. KiVA's three-stage decomposition makes this conflation visible and measurable.

Research directions that become more attractive: (1) Architectures that explicitly represent object-centric relational structures rather than relying on distributed feature representations that may not cleanly separate object identity from object properties; (2) Training objectives that require models to predict the effects of transformations on novel objects, forcing the learning of transformation-invariant relational abstractions; (3) Incorporating 3D geometric priors or explicit spatial reasoning modules that can perform mental rotation, reflection, and set manipulation as structured operations rather than as emergent properties of attention over 2D feature maps; (4) Developmental training curricula that expose models to progressively more complex object transformations in an order that mirrors human cognitive development.

Research directions that become less attractive: (1) Solely scaling model size and training data without changing the training objective or architecture, since the failure patterns persist in GPT-o1 despite its scale and reasoning capabilities; (2) Prompt engineering as a primary solution to visual reasoning deficits, since the paper shows that multiple sophisticated prompting strategies (code generation, reflection, instruction generation, in-context learning) produce no quantitative improvement (Section 4.1); (3) Treating visual analogical reasoning as a problem that will be solved "for free" as a byproduct of better language modeling, since the Large Vision Model trained without language (Appendix B.4) shows the same domain difficulty ordering, suggesting that language is neither the source of the difficulty nor the solution.

Follow-Up Research This Work Enables

Fine-tuning LMMs on procedurally generated transformation data to test whether analogical reasoning can be induced through targeted training. The paper demonstrates that off-the-shelf LMMs lack KiVA-level analogical reasoning, but does not test whether models can acquire this capability through training on transformation analogies. Because KiVA's generation pipeline can produce "infinite expansion of the benchmark" (Section 3.2) — by applying parameterized transformations to any 3D object model — it is straightforward to generate large-scale training data where models must predict the outcome of applying a specified rule to a novel object. A strong follow-up would fine-tune GPT-4V or LLaVA-1.5 on 10,000–100,000 procedurally generated training analogies (with held-out object classes to test generalization) and measure whether extrapolation accuracy on KiVA and KiVA-adults improves. This experiment would distinguish between two hypotheses: (a) LMMs lack analogical reasoning because their training data does not sufficiently emphasize this capability, and it can be acquired through targeted fine-tuning (the optimistic hypothesis), versus (b) current architectures have a fundamental limitation in representing and applying relational transformations that training data alone cannot overcome (the pessimistic hypothesis, which would manifest as near-zero improvement on held-out object classes and hard domains like rotation and reflection). The paper's negative ReST-style result for the revision model in the referenced example paper (though not part of KiVA) provides a cautionary template for how such experiments can backfire.

Systematic comparison of KiVA performance against ARC, RPM, and ConceptARC to establish convergent validity and isolate what "visual analogical reasoning" means across benchmarks. The paper claims to measure visual analogical reasoning but provides no evidence that KiVA performance correlates with performance on established analogical reasoning benchmarks. A study that administers KiVA, ARC, ConceptARC, and an RPM variant (such as the one from Huang et al., 2024) to the same set of LMMs and human participants would reveal whether these benchmarks measure the same underlying capability. If a model scores highly on RPMs but poorly on KiVA rotation (or vice versa), this would indicate that "analogical reasoning" is not a unitary construct and that these benchmarks test different cognitive subcomponents — a finding that would force greater precision in how the field uses the term. The human data would be particularly valuable: do children who perform well on KiVA also perform well on age-appropriate RPM items? Do adult response times on KiVA correlate with adult RPM scores? Such a study would transform KiVA from a standalone benchmark into a validated instrument within a broader nomological network of analogical reasoning measures.

Testing whether 3D-aware architectures (neural radiance fields, 3D vision transformers, or models with explicit depth estimation) show reduced rotation and reflection difficulty compared to 2D-only LMMs. The paper's finding that rotation and reflection are the hardest domains for all tested models (Figures 4, 5) — and that multi-image presentation does not help with these domains while it does help with color and size (Figure 11) — raises the hypothesis that 2D image-plane transformations of 3D objects create a representational mismatch for models that may have learned 3D-consistent object representations from natural images. A study comparing a standard 2D LMM (GPT-4V, LLaVA-1.5) against a model with explicit 3D reasoning capabilities — such as one that constructs a neural radiance field or point cloud from the 2D image and performs the transformation in 3D — would test whether the rotation/reflection difficulty is specific to 2D representations. If a 3D-capable model dramatically improves on rotation and reflection (e.g., from ~45% to ~80% extrapolation accuracy) while maintaining color and size performance, this would suggest that the KiVA difficulty pattern is partly an artifact of 2D operationalization rather than a pure measure of analogical reasoning. If the 3D model shows no improvement, this would strengthen the paper's interpretation that the difficulty lies in analogical mapping rather than spatial representation.

Developmental training curricula that present transformations in the order human children learn them, testing whether LMMs follow a human-like learning trajectory. The paper establishes where models fall on a developmental ruler at a single point in time (off-the-shelf evaluation). A longitudinal training study could establish whether models progress along that ruler in a human-like order when exposed to data in a developmentally motivated sequence. The experiment would start with a base LMM, fine-tune it first on color and size analogies (which children learn earliest; Section 3.2), then on rotation and reflection, then on number, measuring KiVA and KiVA-adults performance after each phase. If models show accelerated learning of later domains after mastering earlier ones — analogous to how children's understanding of object transformations scaffolds later analogical reasoning (Goddu et al., 2020; Christie & Gentner, 2010) — this would support a developmental training paradigm. If learning each domain is independent, this would suggest that the developmental ordering in humans reflects maturational constraints rather than logical prerequisites. This experiment would directly test the paper's implicit claim that a developmental approach "provides a clearer pathway for evaluating and improving cognitive functions in AI" (Section 2, discussing KiVA's motivation).

Using KiVA's three-stage protocol as a diagnostic for where architectural improvements help, by measuring stage-specific improvements after targeted interventions. The paper's decomposition into verbal classification, verbal specification, and visual extrapolation enables a new kind of ablation: rather than measuring whether a new architecture improves end-to-end accuracy, researchers can measure which stage benefits. For example, does adding an explicit object-centric segmentation module improve verbal classification (by isolating the object to be tracked) but not visual extrapolation (because the mapping still relies on distributed features)? Does adding a discrete rotation operation to the vision encoder improve verbal specification of rotation angles but not extrapolation of those rotations to novel objects? Does adding an explicit relational attention mechanism that computes transformations between object representations improve extrapolation specifically? A study that systematically tests architectural modifications and measures their stage-specific effects on KiVA would provide a much richer picture of what each architectural component contributes to analogical reasoning than end-to-end benchmarks allow. The paper's finding that GPT-o1 and GPT-4V show conditional dependence between verbal and visual performance (Figure 12) while LLaVA-1.5 and MANTIS do not already hints that architectures differ in whether they link verbal reasoning to visual mapping — a phenomenon that could be systematically investigated using the stage-specific accuracy decomposition.

Large-scale human developmental study using KiVA to map the precise age trajectory of each transformation domain, establishing a fine-grained developmental ruler for model comparison. The paper's child data (42 children, ages 3–5, mean = 4.07 years) is adequate for establishing that children outperform models on certain domains but too coarse to map when specific abilities emerge. A study with 200+ children evenly distributed across ages 2–7, tested on KiVA and KiVA-adults, would produce age-normed accuracy curves for each transformation domain and subdomain. These curves would serve as precise developmental rulers: a model could be described as performing at the level of a 3.5-year-old on number extrapolation and a 4.2-year-old on color specification, rather than simply "above" or "below" three-year-old level. This would transform KiVA from a binary comparison (model vs. child) into a continuous developmental scale. It would also reveal whether the domain difficulty ordering observed in the paper (color ≈ size > rotation > number ≈ reflection) is stable across development or whether different domains show different growth trajectories — for instance, number reasoning might lag behind color in early childhood but catch up by age 5, which would have implications for interpreting model performance.

Practical Applications and Downstream Use Cases

Diagnostic evaluation of vision-language models before deployment in embodied or safety-critical settings. An organization developing an LMM for robotic manipulation — where the system must recognize that grasping a mug requires rotating the gripper to match the handle's orientation, or that pouring liquid changes the color of the contents — could administer KiVA as part of a capability audit. The paper's finding that GPT-o1, the strongest tested model, extrapolates rotation analogies at only ~45–50% accuracy (Figure 4, bottom panel) means that in a pick-and-place task where the robot must infer a rotation rule from one demonstration and apply it to a novel object of different shape, the model would fail roughly half the time. This is not a theoretical limitation — it is a concrete failure probability that a deployment team can use to decide whether the model is reliable enough for a given application. The three-stage protocol allows the team to diagnose why the model is failing: if verbal classification accuracy is high but extrapolation is low (as is typical for LMMs), the problem is in mapping known rules to new objects, suggesting that the system needs explicit programming for rule application rather than relying on the model's emergent reasoning.

Data generation for training embodied AI systems through analogical demonstration. Rather than requiring a human to demonstrate every possible object manipulation, a system could be trained by showing a single transformation of one object (e.g., "this block rotates 90° clockwise when pushed from the left") and using analogical reasoning to generalize to all other objects. The paper's KiVA generation pipeline — which can apply the same parameterized transformation to any 3D object model (Section 3.2) — provides exactly this capability for generating training data, even if current models cannot solve the analogies. A robotic learning pipeline could use KiVA-style analogies to generate massive datasets where a single transformation rule is applied to thousands of object instances, training the model to recognize that "90° clockwise rotation" is the same rule regardless of whether it is applied to a mug, a block, or a tool. The paper's finding that models struggle with extrapolation (Figure 4) suggests that this kind of data generation is currently necessary — models cannot yet perform the analogical generalization themselves — but the KiVA framework makes the data generation systematic and scalable.

Benchmarking progress in visual reasoning for model cards and capability reporting. As organizations increasingly publish model cards and capability evaluations (following frameworks like those proposed for GPT-4 and o1), KiVA provides a standardized, developmentally-grounded metric for visual analogical reasoning that is more interpretable than abstract benchmarks. Reporting that a model achieves "85% on KiVA color extrapolation (near adult level) and 45% on KiVA rotation extrapolation (below three-year-old level)" communicates capability in terms that non-technical stakeholders — educators, policymakers, parents evaluating AI tools for children — can understand. The developmental framing (three-year-old level, adult level) translates accuracy scores into meaningful reference points that a general audience can interpret, in contrast to benchmarks like ARC or MMLU that report scores without an intuitive human reference. The paper's release of code, data, and models (Section 1, footnote 1) makes this immediately actionable for organizations already publishing model evaluations.

Informing curriculum design for AI literacy and human education by identifying which visual reasoning skills are uniquely human (or currently human-only). The paper's finding that children significantly outperform models on number and reflection extrapolation (Figure 4, top panel) — while models match or exceed children on color and size classification — provides an empirical basis for designing educational activities that emphasize the visual reasoning skills that AI currently lacks. For example, an AI literacy curriculum could use KiVA-style tasks to teach students what AI can and cannot do: "AI can tell you that an object changed color, but it cannot reliably predict how a new object would look if it went through the same rotation that this object did." This has pedagogical value beyond benchmarking — it helps students develop an accurate mental model of AI capabilities by grounding the comparison in a task they themselves can perform (since even three-year-olds solve KiVA). The paper's finding that verbal scaffolding helps children but not consistently models (Figure 9) could also inform teaching strategies: explicit articulation of transformation rules helps children reason analogically, suggesting that human education should continue to emphasize verbal reasoning about visual transformations even as AI capabilities improve in other domains.