MME-CC: A Challenging Multi-Modal Evaluation Benchmark of Cognitive Capacity

🎯 Pitch

Even the best multimodal AI scores just 43% when forced to truly reason about what it sees—not read. This benchmark reveals that models fail catastrophically on basic spatial orientation, cannot maintain object identity across different views, and stubbornly ignore instructions to give counterfactual answers, exposing a fundamental lack of vision-grounded cognition beneath fluent textual outputs.

1. Executive Summary

This paper introduces MME-CC (Multi-Modal Evaluation benchmark of Cognitive Capacity), a vision-grounded benchmark that systematically evaluates the cognitive capacity of multimodal large language models (MLLMs) by organizing 11 representative reasoning tasks into three fundamental categories of visual information — spatial reasoning (e.g., matching street-view images to satellite maps), geometric reasoning (e.g., solving maze shortest-path problems), and visual knowledge reasoning (e.g., following counterfactual instructions that require reversing depicted facts) — and provides fine-grained diagnostic analyses of model behavior across these dimensions. Evaluating 16 MLLMs on 1,173 expert-annotated questions, the study finds that the best closed-source model, Gemini-2.5-Pro, achieves only 42.66% overall accuracy, while spatial and geometric reasoning remain broadly weak at or below 30% across all models, and Chain-of-Thought reasoning typically follows a three-stage layered process — extraction, reasoning, and verification — with visual extraction occurring continuously throughout rather than being confined to an initial perception phase. The paper further identifies recurring failure patterns — orientation and reference-frame confusion, fragile cross-view identity persistence, and poor adherence to counterfactual instructions — establishing that current MLLMs lack comprehensive vision-centric reasoning capacity even on tasks that require no specialized textual knowledge.

2. Context and Motivation

The Core Problem: We Cannot Properly Assess Whether Multimodal Models Actually "See" and "Think"

The fundamental question this paper tackles is deceptively simple: when we test multimodal large language models, are we actually measuring their visual reasoning capacity, or are we measuring something else entirely? The paper argues that existing benchmarks, despite their scale and popularity, systematically fail to isolate vision-centric reasoning — what the paper calls "visual cognitive capacity" — from textual reasoning and superficial pattern matching. This matters because as MLLMs are deployed in increasingly consequential domains (autonomous navigation, medical imaging, scientific diagram interpretation), we need to know whether the models genuinely understand what they're looking at, or whether they're exploiting shortcuts that happen to correlate with benchmark performance.

The gap the paper identifies is not merely that MLLMs score low on hard problems — that would be unsurprising for any benchmark at the frontier. The gap is more fundamental: existing benchmarks are not measuring what they claim to measure. They conflate visual understanding with text-based reasoning, OCR, and format exploitation, giving researchers and practitioners a misleading picture of model capabilities. A model that scores 80% on a "multimodal reasoning" benchmark might in reality be performing strong textual inference on weakly grounded visual inputs — and we would not know it from the aggregate score.

Why This Gap Matters: The Stakes of Capability Assessment

The paper's motivation extends beyond academic benchmarking into several domains where mis-estimation of visual reasoning capacity has direct consequences:

Safety and reliability in deployment. MLLMs are increasingly used in applications where visual understanding failures carry real-world costs. If a model correctly answers a physics diagram question by reading the LaTeX annotations rather than understanding the spatial relationships in the figure, it may seem capable in evaluation but fail catastrophically on unannotated images in deployment. The paper's observation that current benchmarks "permit solutions that rely on textual cues, format priors, or OCR" (Section 5, discussion of related work) is not a minor methodological concern — it means that safety-critical applications (autonomous driving perception, medical scan interpretation, accessibility tools for visually impaired users) cannot rely on benchmark scores as validated capability signals.

The illusion of progress toward visual intelligence. If models improve on benchmarks by getting better at textual reasoning and pattern matching — while their underlying visual cognition remains stagnant — then the field can be misled about how close we are to human-like visual understanding. The paper positions this concern explicitly: while state-of-the-art MLLMs "often demonstrate strong performance" on existing benchmarks, the authors hypothesize that this performance may be largely attributable to non-visual capabilities. MME-CC is designed to strip away those supports and expose what models can do when they must rely primarily on visual information.

The need for fine-grained cognitive diagnostics. Aggregate accuracy scores on a benchmark tell us "model X scores Y%," but they do not tell us why. A model that scores 30% on spatial reasoning might fail for entirely different reasons than another model scoring the same 30% — one might have good object recognition but poor multi-view geometry, while another might have the reverse. Without task-level and error-level diagnostics, benchmark scores cannot guide targeted model improvements. The paper explicitly frames MME-CC as providing "task- and dimension-level diagnostics that are actionable for evaluation and model design" (Section 6).

Theoretical significance: what does "visual reasoning" even mean? A deeper motivation — implicit in the paper's organizational structure — is that the field lacks a coherent taxonomy of what counts as "vision-centric cognition." The paper proposes, through its benchmark design, that visual cognitive capacity decomposes into three separable but interacting dimensions: processing spatial information (where things are relative to each other in 3D space), geometric information (abstract shapes, patterns, and constraints), and visual knowledge (semantic interpretation of visual content combined with rule-following). This is not merely a convenient categorization for benchmark organization — it is a substantive claim about the structure of visual reasoning that, if validated, would give researchers a principled framework for both evaluation and model architecture design.

Where Existing Benchmarks Fall Short

The paper identifies specific, structural flaws in the existing benchmarking landscape that make it impossible to draw confident conclusions about MLLMs' visual reasoning capacity. These are not minor quibbles — they are design-level problems that affect the validity of comparisons and conclusions across the entire field.

The textual reasoning contamination problem. Many widely-used multimodal benchmarks embed task-critical information in text rather than requiring it to be extracted from images. Take the specific example the paper gives: MathVista (Lu et al., 2024) and the MMMU Series (Yue et al., 2024, 2025) are described as being "overly biased toward the text-space-based reasoning capabilities of MLLMs" (Section 1). What this means concretely: in a MathVista question, a geometry problem might include a diagram, but the mathematical relationships, givens, and constraints are predominantly expressed in the accompanying text. A model could solve the problem by performing strong algebraic reasoning on the textual description while making only superficial use of the visual diagram — and the benchmark would have no way to distinguish this from genuine visual-mathematical reasoning.

This is not a hypothetical concern. The related work survey in Section 5 cites several studies that have documented shortcut exploitation in multimodal benchmarks: models answer questions by reading embedded text (OCR), guessing answer formats from common patterns, or exploiting statistical regularities in benchmark construction. The paper notes that NaturalBench, EasyARC, and VLSBench have all shown models taking advantage of "answer-bearing text instead of reasoning over images." When a benchmark's difficulty can be circumvented without looking at the images, its validity as a measure of visual reasoning collapses.

Lack of systematic cognitive taxonomy. Benchmarks like ZeroBench (Roberts et al., 2025) are cited as attempts to probe visual reasoning with "carefully designed queries" that stress spatial and commonsense limits. However, the paper argues that ZeroBench and similar efforts "lack a well-established classification system and in-depth analysis of the cognitive capacity of MLLMs" (Section 1). What this means: ZeroBench enumerates various vision-based reasoning tasks (spatial relations, visual puzzles, etc.) but does not organize them into a coherent framework that captures why certain tasks are cognitively related and how performance on one might inform expectations about another. Without this taxonomic structure, benchmark results become a laundry list of scores with no principled way to aggregate findings, diagnose weaknesses, or track progress on specific cognitive dimensions.

Template predictability and format constraints. The paper identifies that many existing benchmarks rely on "synthetic data, repetitive templates, or narrow formats" (Section 5, discussion of VisuLogic and similar efforts). This is a deeper problem than it might appear. When benchmark questions follow predictable templates — even if the individual instances are novel — models can learn to recognize the template structure and exploit it. For example, if every "spatial reasoning" question in a benchmark follows the format "Object A is to the [left/right] of Object B," a model might learn to produce left/right answers based on linguistic priors rather than visual analysis. The paper's emphasis on diverse, non-templated question construction (which relies on human-crafted scenarios with carefully controlled visual inputs) is a direct response to this limitation.

The vision-based reasoning gap. Perhaps the most telling table in the paper is Table 3 (Section 2.3), which compares representative benchmarks on a simple binary criterion: whether the benchmark is "vision-based." The authors define "vision-based" to mean that "all task information is derived from images rather than text" — that is, you cannot solve the problem without looking at the images. The comparison is stark: MME (Fu et al., 2023), MMMU (Yue et al., 2024), MMBench (Liu et al., 2024), and MMStar (Chen et al., 2024) are all marked as not vision-based, while only ZeroBench and MME-CC qualify. This single column in Table 3 crystallizes the paper's core criticism: the field's most prominent multimodal benchmarks are, by the authors' standard, not actually testing forced visual reasoning. A model could succeed on them with strong language capabilities and weak vision — and we would not know it from the published scores.

Free-form output and the multiple-choice shortcut. Table 3 also shows that leading benchmarks like MME, MMMU, MMBench, and MMStar use multiple-choice answer formats, while MME-CC and ZeroBench use free-form generation. Multiple-choice questions introduce a specific vulnerability: models can eliminate implausible options through textual reasoning, guess based on answer pattern distributions, or exploit the fact that the correct answer is present among the options (a form of recognition rather than generation). Free-form answering removes these shortcuts — the model must produce the answer from scratch, making it a more demanding test of genuine understanding. The paper's choice of free-form output is a deliberate methodological decision to close this shortcut pathway.

How MME-CC Positions Itself Relative to Existing Work

The paper's positioning is not that the existing benchmarks are useless — they serve valuable roles for general capability profiling. Rather, the paper positions MME-CC as filling a specific, undersupplied niche: a benchmark that (1) forces solutions to depend on visual information, (2) organizes tasks into a principled cognitive taxonomy, and (3) provides fine-grained error analysis beyond aggregate scores.

The "vision-based" guarantee as a first principle. MME-CC's construction philosophy centers on ensuring that for every question, the information required to answer correctly is present in the images, not the text prompt. This is achieved through task design choices visible throughout the benchmark taxonomy (Figure 1):

Satellite Image Matching: The prompt contains no location information. The model must extract geometric and landmark cues from the street-view and satellite images to determine correspondence.
Indoor Directional Reasoning: The question asks for an object's orientation, but the reference orientation (e.g., "the window faces south") is given in an anchor image — the model cannot answer without processing the visual spatial layout.
Maze: The maze structure, entrance, exit, and numbered cells are entirely visual. The rules are in the prompt, but the answer (which numbers lie on the shortest path) can only come from analyzing the maze image.
Counterfactual Instruction: The model is told to reverse the depicted facts, but what those facts are (e.g., "the image shows a cat") can only be determined by looking at the image. The textual instruction is generic; the visual content is specific and irreplaceable.

This design philosophy is not just about removing text — it's about guaranteeing that any model which performs well on MME-CC has demonstrated genuine visual information extraction and processing. If a model scores 74.63% on Visual Knowledge Reasoning (as Gemini-2.5-Pro does in Table 4), that cannot be attributed to reading answer-bearing text, because there is no such text to read.

A three-dimensional cognitive taxonomy, not an enumeration. The paper's organization of 11 tasks into three overarching categories — Spatial Reasoning, Geometric Reasoning, and Visual Knowledge Reasoning — is a taxonomic contribution as much as a benchmarking one. Each category captures a qualitatively different cognitive demand:

Spatial Reasoning tasks require the model to infer and reason about 3D spatial relationships from 2D images — understanding camera viewpoints, mapping between allocentric and egocentric perspectives, maintaining object identity across multiple views. These are the tasks where human spatial cognition (mental rotation, spatial updating, perspective-taking) is most directly engaged.
Geometric Reasoning tasks require abstract pattern recognition, constraint satisfaction, and planning — solving mazes, assembling jigsaw puzzles, finding optimal move sequences. These are closer to what one might call "visual logic" — reasoning about abstract patterns instantiated in visual form.
Visual Knowledge Reasoning tasks require integrating semantic understanding of visual content with rule-following and meta-cognitive control — answering exactly one of four questions correctly, providing counterfactual answers, detecting deliberately wrong answers. These probe whether the model can apply higher-order instructions to visual content.

This taxonomy enables a more nuanced analysis than raw scores. For instance, in Table 4, Gemini-2.5-Pro scores 74.63% on Visual Knowledge Reasoning but only 23.80% on Spatial Reasoning. This is not just "Gemini is weak at spatial tasks" — it tells us something specific about the current frontier of MLLM capabilities: semantic understanding of images (knowing what an object is) is far ahead of spatial understanding (knowing where an object is relative to other objects in 3D space). This could inform both evaluation priorities and architectural research: if spatial reasoning is the bottleneck, perhaps models need explicit 3D geometric representations alongside 2D visual features.

Baked-in error analysis, not an afterthought. The paper's error analysis (Section 4.3, supported by extensive case studies in Appendices D and E) is not a post-hoc exploration of interesting failures — it is a core component of the benchmark's design. MME-CC is built to reveal categories of failure, not just failure rates. The three recurring error patterns identified — orientation and reference-frame confusion, fragile cross-view entity persistence, and instruction-following failures — each map to a fundamental cognitive capability:

Orientation/ reference-frame alignment is about the ability to maintain a consistent spatial coordinate system and track how objects relate to it. When Gemini-2.5-Pro scores only 14.3% on Indoor Directional Reasoning (the ablation base in Table 5), the error analysis reveals that this isn't just "the model sometimes gets the direction wrong" — it's that the model systematically fails to propagate directional constraints across rooms, reverting to linguistic heuristics rather than geometric reasoning.
Cross-view identity persistence is about object permanence and recognition under viewpoint change — a capacity that develops early in human infants but that the paper shows is remarkably brittle in MLLMs. The Indoor Deduplication Counting task, where models must identify that the same couch appears in three different photos, exposes this weakness directly.
Instruction adherence probes meta-cognitive control: can the model override its default behavior (describe what it sees) when given explicit counter-instructions? The Sandbagging and Counterfactual Instruction tasks show that models have a strong bias toward "saying what's true" even when explicitly told to do otherwise.

These error patterns are not task-specific quirks — they recur across multiple tasks and multiple model families, suggesting they reflect fundamental limitations in current MLLM architectures rather than training data gaps for specific tasks.

Tension between capability amplification and capability creation. One of the paper's implicit theoretical commitments is that visual cognitive capacity is a distinct capability that cannot be reduced to or recovered from text-based reasoning. This is most visible in the cascade of model performance across difficulty types: models that do well on Visual Knowledge Reasoning (which draws heavily on semantic knowledge and instruction-following skills also used in text tasks) do not necessarily do well on Spatial or Geometric Reasoning (which require specialized visual processing). The framework implies that improving spatial reasoning requires different interventions than improving visual knowledge reasoning — a claim with clear implications for model development priorities.

What MME-CC deliberately does NOT do. The paper is clear about its scope: it is a vision-grounded benchmark that forces visual reasoning, not a general-purpose multimodal evaluation suite. It does not attempt to cover all possible visual tasks (there is no Natural Language Understanding of images, no OCR, no free-form dialogue about visual content). It does not claim that its 1,173 questions are sufficient for training or that the difficulty distribution is calibrated for all model sizes. It does not provide a training set or propose new model architectures. What it does claim is that for the specific and important question of "can MLLMs perform cognitively demanding visual reasoning," existing benchmarks do not provide reliable answers, and MME-CC does.

3. Technical Approach

This is primarily a benchmark construction and evaluation paper whose core idea is that to properly assess the visual cognitive capacity of multimodal large language models, you must (1) design tasks where the answer cannot be derived without deeply processing the visual input, (2) organize those tasks into a principled cognitive taxonomy that isolates different types of visual reasoning, and (3) provide diagnostic tools—not just aggregate scores—that reveal how and why models fail on specific cognitive dimensions.

3.1 Reader Orientation

The "system" being built here is not a model or an algorithm—it is an evaluation instrument: a set of 1,173 carefully crafted questions across 11 distinct task types, supported by a human-in-the-loop data construction pipeline, an LLM-as-judge automated scoring protocol, and a framework for error taxonomy and Chain-of-Thought behavioral analysis. The problem it solves is that existing multimodal benchmarks conflate visual reasoning with text-based reasoning, making it impossible to know whether a model that scores well on a benchmark actually "sees" and "thinks" about images, or simply exploits textual shortcuts. The shape of the solution is a forced vision-centric benchmark with free-form outputs, organized into three cognitive dimensions, where every question embeds the answer-critical information exclusively in the visual modality.

3.2 Big-Picture Architecture (Diagram in Words)

The MME-CC system has four major components, arranged as a pipeline from data creation through evaluation execution to analysis:

Task Design and Definition (Taxonomy). A conceptual framework that partitions visual cognitive capacity into three dimensions—Spatial Reasoning (understanding 3D spatial relationships from 2D images), Geometric Reasoning (abstract pattern recognition, constraint satisfaction, and planning on visual structures), and Visual Knowledge Reasoning (integrating semantic visual understanding with rule-following and meta-cognitive control). Eleven specific tasks are designed to probe these dimensions with minimal overlap.
Data Construction and Quality Assurance Pipeline. A four-stage human-in-the-loop process: (a) task definition with small-scale pilot testing using two models (Doubao and Gemini) to calibrate difficulty and refine prompts; (b) data acquisition from diverse license-compliant sources (photographs, screenshots, web captures) with manual annotation by a 10-person team (6 annotators, 3 subtask leads, 1 task lead); (c) image post-processing (standardized formatting, ID assignment, cropping, stripping of UI overlays and metadata); (d) model-based filtering where the best available model (Gemini-2.5-Pro) screens out items that are trivially easy ((>95%) accuracy), semantically redundant, or lacking discriminative value, removing approximately 50% of the initial data pool. Every sample undergoes at least two rounds of independent review by different annotators, with disagreements escalated to subtask leads for binding decisions. For cognitively demanding sub-tasks (Indoor Directional Reasoning, Unblock Me), 100% of validation is conducted directly by senior subtask leads using an expert-only protocol.
LLM-as-Judge Evaluation Protocol. An automated scoring system using DeepSeek-V3-0324 as a judge model. For each question, the judge compares the model-generated free-form answer against a gold reference answer and assigns a binary correctness score ((0) or (1)). The protocol is validated by manual evaluation of 99 randomly sampled items (33 from each category), achieving a 95% agreement rate with human judgments. Task-specific grading prompts (e.g., for Unblock Me, which requires evaluating both the minimum step count and the set of moved blocks) handle output formats that deviate from standard exact-match criteria.
Cognitive Diagnostic Analysis Framework. Three layers of post-evaluation analysis applied to model outputs: (a) accuracy breakdowns by task type and cognitive dimension, reported in Table 4 and per-subtask in Table 5; (b) Chain-of-Thought behavioral analysis on reasoning model outputs, classifying the structure of reasoning chains into stages (extraction → reasoning → verification) and tracking the distribution of visual information extraction across the chain; (c) error pattern taxonomy (Section 4.3 and Appendices D-E) that categorizes recurring failure modes—orientation/reference-frame confusion, cross-view identity persistence failures, and instruction-following breakdowns—with concrete annotated examples for each.

Information flows as follows: an MLLM receives a text prompt describing the task rules alongside one or more images containing the answer-critical information → the model generates a free-form text response (optionally with visible Chain-of-Thought reasoning) → the LLM judge compares the final answer to the reference and assigns a score → aggregate results are computed per-task and per-dimension → qualitative CoT and error analyses are conducted manually or semi-manually on model outputs.

3.3 Roadmap for the Deep Dive

First, the task taxonomy—how the three cognitive dimensions (Spatial, Geometric, Visual Knowledge Reasoning) are defined, what cognitive demands each entails, and how the 11 subtasks map to these dimensions. This is foundational because every subsequent analysis (accuracy breakdowns, error patterns, CoT behavior) is structured around this taxonomy.
Second, the data construction pipeline in full detail—the four-stage process, the composition and roles of the annotation team, the quality control gates, and the model-based filtering criteria. Understanding this is essential because the benchmark's validity rests entirely on the quality and forced-visual nature of its questions.
Third, the evaluation protocol—how the LLM-as-judge system works, its validation against human judgments, and the task-specific scoring prompts that handle non-standard answer formats. This ensures the reliability of the quantitative results in Table 4.
Fourth, the error analysis framework—how the paper identifies and categorizes failure patterns, the three major error classes, and the concrete evidence for each drawn from model outputs on MME-CC tasks. This is where the paper moves beyond "model X scores Y%" to actionable diagnostic claims.
Fifth, the CoT behavioral analysis—the three-stage reasoning model (extraction → reasoning → verification), the finding of continuous visual extraction, and the role of excessive verification pauses in degrading performance on complex spatial tasks. This connects model behavior to model architecture and training in a way that pure accuracy scores cannot.

3.4 Detailed, Sentence-Based Technical Breakdown

The Three-Dimensional Cognitive Taxonomy

The paper's central design decision is to organize all 11 tasks into three overarching categories that represent qualitatively distinct types of visual cognitive demand. This taxonomy is not merely a convenience for presenting results—it embodies a substantive claim about what visual reasoning decomposes into, and each dimension is operationalized through task designs that force specific cognitive processes.

Spatial Reasoning is defined as the ability to infer and manipulate 3D spatial relationships from 2D visual inputs. The cognitive demands include: understanding camera viewpoint from monocular cues, mapping between allocentric (world-centered) and egocentric (viewer-centered) reference frames, tracking object identity and location across viewpoint changes, and deduplicating entity counts when the same object appears in multiple views. Three tasks operationalize this dimension:

Satellite Image Matching (101 samples): Given two ground-level Google Street View images and one satellite map with seven candidate locations (A–G), the model must determine which of the seven locations corresponds to each street-view photo. The prompt contains no location information—all cues must be extracted from visual landmarks, building geometry, and the spatial relationship between the street-view perspective and the satellite overhead view. To prevent lucky guessing, the two ground truth locations are different (the model must identify two distinct correct answers), making random guessing accuracy approximately (1/(7 \times 6) \approx 2.4%).
Indoor Directional Reasoning (113 samples): Using Lianjia VR tour data that provides floorplan location and camera-facing direction, the model receives an anchor view with a known orientation (e.g., "the window in this image faces North") and one or more query views from the same apartment. It must infer the cardinal orientation of objects (e.g., windows, TV screens) in the query views by tracing the spatial continuity between adjacent camera positions. The task requires propagating directional constraints through a mental map of the indoor layout—a capacity that most models fail at systematically. All compass icons, text overlays, and lighting cues are stripped from images to prevent shortcut solutions.
Indoor Deduplication Counting (105 samples): From a single apartment's VR tour, the model receives a coherent set of views and is asked to count the number of unique instances for two specified object categories (e.g., "dining chairs" and "sofas"). The same physical object appears in multiple photos from different angles, so the model must maintain cross-view object identity persistence—recognizing that the brown couch in Photo 2 is the same physical entity as the brown couch in Photo 5—to avoid overcounting or undercounting. The task provides clear inclusion/exclusion rules (e.g., whether an empty flowerpot counts as "pottery") and the requirement to count two categories per sample reduces the chance of lucky guessing.

Geometric Reasoning is defined as the ability to recognize abstract patterns, satisfy constraints, and plan sequences of actions on visually presented structures. Unlike spatial reasoning—which is fundamentally about mapping 2D inputs to 3D spatial understanding—geometric reasoning operates on visual configurations as abstract patterns and constraints, closer to visual puzzle-solving or visual logic. Five tasks operationalize this dimension:

Gomoku Variation (122 samples): A modified version of the Gomoku (Five-in-a-Row) game played with piece shapes from other board games (Chinese Chess, Checkers) rather than standard Gomoku pieces. The model must analyze endgame positions where exactly one winning move exists and identify that move. The variation in piece appearance prevents models from relying on trained visual priors for standard Gomoku pieces; they must reason about the board state abstractly. Positions with multiple optimal solutions or symmetric layouts are filtered out, and the ground truth is expanded to accept all valid optimal solutions when ambiguity is unavoidable.
Maze (194 samples): The model receives a fixed-size (10 \times 10) grid maze with blocked edges, an entrance at the bottom-left, an exit at the top-right, and digits (1–9) marked on selected cells along the unique shortest path. The task: "find all numbers on the shortest path from entrance to exit and report them in ascending order." Solving this requires constructing a globally optimal path through the maze—not merely following local adjacency cues—and then extracting the subset of labeled cells on that path. Dual human verification and algorithmic checks ensure exactly one unique shortest path exists per maze. This is the task where no model exceeds 2% accuracy in the base setting (Table 5), suggesting that sustained rule-based simulation and path planning push beyond current MLLM capabilities.
Unblock Me (99 samples): A sliding block puzzle where the model must determine the minimum number of moves required to extract a red target block from a constrained board. The model must output both the minimum move count and the set of non-red blocks that need to be moved. Solutions are validated through a dual process: programmatic comparison against the canonical optimal solution data for each puzzle, followed by final manual review by senior leads. The task is classified under the expert-only validation protocol due to the complexity of verifying minimal solutions, and any levels without a unique minimal solution are discarded.
Jigsaw Puzzle (141 samples): Physical jigsaw puzzles are assembled, then 3–6 pieces are removed and placed next to the board, which has labeled empty slots. The model must produce a one-to-one mapping of each candidate piece to its correct empty slot. This tests fine-grained visual pattern matching and spatial alignment under viewpoint variation, as pieces are photographed rather than digitally generated. Symmetric or visually ambiguous pieces are excluded during curation, and image normalization suppresses non-content cues like shadows or lighting gradients.
Chart Modification (49 samples): The model receives a chart image and a textual instruction to modify specific data values (e.g., "swap the scores for DeepSeek-V3 and GPT-4o"). It must extract the underlying data from the visual chart, apply the specified transformation, and output the resulting values. Target answers are computed deterministically from manually annotated ground-truth data tables and validated programmatically.

Visual Knowledge Reasoning is defined as the ability to integrate semantic understanding of visual content with higher-order rule-following and meta-cognitive control. These tasks require the model to recognize what is depicted in an image and then apply a non-standard instruction that overrides its default behavior. Three tasks operationalize this dimension:

Sandbagging (41 samples): The model receives one image with four ordered sub-questions and the instruction: "answer exactly one correctly and three incorrectly." The position of the single correct answer is randomized across questions. Evaluation programmatically enforces the 1-right/3-wrong pattern—the model must demonstrate control over its own output correctness, not just knowledge of the answers. Prompts explicitly forbid staged "first I will answer correctly, then I will answer incorrectly" outputs, requiring a direct final answer.
Counterfactual Instruction (60 samples): Given an image with unambiguous depicted facts (e.g., object presence, color, count, left-right position) and an explicit inversion mapping (e.g., "if the image shows Journey to the West, say 'Water Margin' and introduce one character; if it shows Water Margin, say 'Journey to the West' only"), the model must produce the logically reversed answer. Only items where the counterfactual state is well-defined and checkable are retained, and ambiguous cases where the "opposite" is unclear are filtered out during curation.
Finding Wrong Answer (148 samples): One image is presented with four question-and-answer pairs, of which exactly one answer is deliberately flawed (e.g., an attribute, count, or relation is swapped). The model must detect which of the four answers is incorrect. The three correct answers are independently verifiable from the image, and the flawed answer is created via a minimal, controlled perturbation to a correct answer. Error types are balanced across the dataset to prevent answer-key distribution exploitation.

The Data Construction Pipeline (Four-Stage Human-in-the-Loop Process)

The benchmark's validity depends entirely on the quality and forced-visual nature of its questions. The construction pipeline described in Section 2.1 and detailed in Appendix A is designed to systematically eliminate pathways through which models could answer correctly without genuine visual reasoning.

Stage 1: Task Definition and Preliminary Evaluation. For each of the 11 subtasks, a dedicated subtask lead (one of the three B-role annotators, each with over 3 years of experience in NLP/CV and prior dataset creation experience) defines the task objective and creates a concise, task-specific prompt template. Rather than ad-hoc sampling, construction follows a structured design process that specifies evaluation dimensions and the reasoning steps required.

A pilot set of 5–10 examples is built from carefully selected images for each subtask. The authors then pilot with two representative models—Doubao (ByteDance's model) and Gemini (Google's model)—to test prompt clarity and task feasibility. The key design criterion during this phase is calibration: the task must be solvable by humans (establishing an upper bound on achievable performance) but challenging for the best available models (providing headroom for discrimination). The authors describe iteratively refining "the task scope, prompts, and data handling" based on these pilot results. Reliability controls are specified during this stage, including rules for preventing information leakage, managing solution ambiguity, and ensuring landmark uniqueness.

Stage 2: Data Collection and Manual Quality Control. After task definitions stabilize, data acquisition proceeds from diverse, license-compliant sources:

Real-world photographs taken by the annotation team, primarily for tasks requiring controlled physical setups (Gomoku Variation boards, Jigsaw Puzzle layouts).
Targeted screen captures from online platforms: real-estate listings from Lianjia for Indoor Directional Reasoning and Deduplication Counting; Google Street View and Google Maps for Satellite Image Matching; web images for Sandbagging and Counterfactual Instruction tasks.
Game screenshots covering representative in-game reasoning cases (Unblock Me).
Auto-generated content: mazes are procedurally generated using an algorithm that guarantees a unique shortest path, with digits overlaid at selected empty cells.
Internal dataset images for the Finding Wrong Answer task, supplemented with manual annotations.

The 10-person annotation team follows a structured hierarchy (Table 1). The six annotators (A1–A6) handle question construction and multi-round quality checks. The three subtask leads (B1–B3) own the end-to-end design and oversight for their respective task clusters. The single task lead (C1) is the overall benchmark owner with "a documented history of leading the development of 3+ public benchmarks." All annotators must review a detailed instruction manual (Appendix B) and pass a qualification test before beginning annotation work.

During annotation, each sample undergoes the core quality control protocol: one annotator provides the reference answer, and a different annotator independently verifies it. In cases of disagreement, the sample is escalated to the respective subtask lead for a final binding decision. This two-stage review ensures that every item in the final benchmark has been validated by at least two people, with a third (the subtask lead) serving as the adjudication authority. Additionally, a dedicated QA team conducts periodic audits to catch systematic errors that might escape individual reviewers.

For tasks that are "particularly cognitively demanding and require nuanced judgment"—specifically Indoor Directional Reasoning and Unblock Me—the paper implements a more stringent expert-only protocol: 100% of the validation is conducted directly by the senior subtask leads rather than being delegated to the general annotator pool. The rationale is that these tasks' complexity makes quality "difficult to guarantee via simple rule-based checking," so expert judgment is necessary to catch subtle errors in spatial inference or solution optimality.

Stage 3: Image Post-Processing. All collected images undergo a unified post-processing pipeline:

Identifier assignment: each image receives a systematic ID for tracking through the benchmark.
Cropping and resizing: images are standardized to consistent dimensions appropriate for each task.
Overlay and metadata stripping: compasses, icons, text labels, watermarks, and UI elements are removed to prevent information leakage. For Satellite Image Matching, this means removing map pins and business labels. For Indoor Directional Reasoning, this means removing compass indicators that would directly reveal orientation. For Unblock Me, this means hiding in-game move count displays and hints.
Lighting normalization: for photography-based tasks (Jigsaw Puzzle), lighting gradients and shadows that could provide non-content cues are suppressed.

Stage 4: Model-Based Filtering. The post-processed data pool undergoes a filtering stage where the best available model at the time of construction (Gemini-2.5-Pro) is run on all items. Three categories of items are removed:

Trivially easy items: defined as items where the model achieves accuracy above 95%. These items fail to discriminate between strong and weak models because even the best model's occasional errors are not systematic enough to create meaningful score differences.
Semantically redundant items: items that are near-duplicates or test the same specific skill in the same way as another item, contributing no additional diagnostic value.
Items lacking discriminative value: items where the model's performance does not correlate with expected capability rankings, suggesting the item may have ambiguous ground truth or hidden shortcuts.

The paper reports that "this filtering stage removed approximately 50% of the initial data pool," meaning that the final 1,173 questions represent the most discriminative and cleanly annotated half of a larger candidate set. This is a deliberate quality-over-quantity tradeoff: the goal is 1,173 questions where performance reliably reflects visual cognitive capacity, rather than 10,000 questions where performance is partially confounded by annotation noise or varying item quality.

Exception Handling After Filtering. As an additional check, samples that yield zero accuracy across all screened models after filtering are randomly reviewed. The purpose is to identify whether these universally failed items contain hidden annotation issues (e.g., an incorrect reference answer, an ambiguous prompt) or hidden shortcuts that no model happened to exploit. Any problematic items identified during this review are corrected or removed.

The LLM-as-Judge Evaluation Protocol

Unlike multiple-choice benchmarks where scoring is automatic, MME-CC requires free-form text answers (see Table 3, Output column), making automated evaluation non-trivial. The paper adopts an LLM-as-a-judge protocol with rigorous validation.

Judge Model Selection. DeepSeek-V3-0324 is selected as the judge model. The choice is not arbitrary—the paper validates its reliability by conducting manual evaluation of 99 randomly sampled items, with 33 drawn from each of the three reasoning categories (Spatial, Geometric, Visual Knowledge). The reported agreement rate is 95% with human judgments, which is high enough to support the claim that the automated evaluation faithfully reflects human assessment of answer correctness.

Scoring Mechanism. For each question, the judge receives:

The original question prompt.
The gold reference answer (generated and verified during the annotation process).
The model-generated student answer.

The judge then performs a structured comparison: it first extracts the student's final answer from the generated text (ignoring intermediate reasoning steps), then compares it to the reference answer using a binary rubric ((0) or (1) point). The scoring prompt (reproduced in Appendix C) specifies that:

The score is based "solely on the correctness of the student's final answer" – intermediate steps are not evaluated.
Exact matches earn (1) point; mathematically equivalent expressions (e.g., (1 + 1) written as (1 + 0.5x) when the reference is (1 + 0.5x)) are acceptable.
For multi-part questions, all subparts must be correct for (1) point.
Answers that do not match, including empty answers, earn (0) points.

The judge outputs structured JSON within a code block: {"answer_score": [[score]]} where [[score]] is the integer (0) or (1).

Task-Specific Grading Prompts. For tasks where simple exact-match checking is insufficient, specialized judging prompts are used. The Unblock Me task (Appendix C) has a dedicated grading prompt because the answer has two components: (1) the minimum number of moves, and (2) the set of non-red blocks that must be moved (order-independent). The scoring logic examines both components: the minimum move count must match exactly, and the set of blocks must be identical (ignoring order). If either component is wrong, the entire answer scores (0) points. The prompt includes worked examples of correct and incorrect student answers with the corresponding JSON output to calibrate the judge's behavior.

Why LLM-as-Judge? The free-form output requirement is essential for the benchmark's vision-forcing property (models cannot guess from multiple-choice options), but it creates a scoring challenge that multiple-choice benchmarks avoid. The paper's choice of LLM-as-judge with 95% human agreement at 99-sample scale represents a pragmatic tradeoff: manually scoring all 16 models on 1,173 questions ((18,768) total answers) would be prohibitively expensive, while pure rule-based scoring would fail on answer formats that are semantically equivalent but textually different from the reference. The task-specific prompts for Unblock Me demonstrate a design principle: when the answer format has non-trivial structure, the judging rubric must be tailored to capture the structure rather than relying on generic string matching.

Error Analysis Framework

The error analysis component of MME-CC is not a post-hoc qualitative exploration—it is a structured diagnostic instrument designed to surface specific, recurring failure patterns that aggregate accuracy scores would obscure.

Error Pattern Identification Methodology. The authors analyze failure cases across all 11 subtasks by examining instances where models (primarily Doubao-Seed-1.6-vision-0815, as shown in Appendices D and E) produce incorrect answers. From these failures, three recurring error classes are identified that appear "across tasks and reasoning dimensions" (Section 4.3). The classification is validated by providing concrete, annotated examples from multiple tasks for each error class (Figures 5–14 in Appendices D-E).

Error Pattern 1: Orientation Judgment and Reference-Frame Alignment. The model "often fails to preserve object orientation across views, and viewpoint changes induce mismatches that hinder the establishment of a consistent global reference frame" (Section 4.3). This pattern manifests in:

Satellite Image Matching (Figure 5): The model fails to match ground-level and satellite views by overlooking geometric cues (e.g., building orientation relative to the street grid, visibility of specific architectural features from specific angles). Instead, it relies on superficial textures or isolated landmark features without accounting for how different viewpoints change the appearance of those features.
Indoor Directional Reasoning (Figure 6): The model "does not propagate directional constraints across rooms, defaulting to linguistic heuristics" instead. For instance, if the anchor image shows a window facing North, and the query image is taken from a camera position rotated 90 degrees clockwise, the model should infer that objects that appear "ahead" in the query image are actually facing East in absolute orientation. Instead, the model tends to report relative directions ("left of the camera") rather than computing the absolute orientation transformation.

Error Pattern 2: Cross-View Entity Identity Persistence. When the same physical object appears across multiple images, the model "frequently fails to maintain identity consistency," leading to either double-counting (treating the same object in two views as two different objects) or omission (treating two different objects as the same because they look similar). This manifests most clearly in:

Indoor Deduplication Counting (Figure 7): The model is shown multiple photos from a single apartment and asked to count distinct furniture items. A brown sofa appearing in three different rooms or from three different angles should be counted once, but the model often counts it three times because it fails to match the sofa's identity across viewpoint changes. The annotated error case in Figure 7 shows the model making exactly this mistake: identifying the same physical lounge chair in two different photos as two separate instances.

Error Pattern 3: Over-Reliance on Literal Descriptions Under Instruction Constraints. Faced with non-literal or counterfactual instructions, the model "tends to prioritize the literal visual content while ignoring task-specific counterfactual constraints expressed in text, thereby producing answers that conflict with the required instruction" (Section 4.3). This manifests in:

Sandbagging (Figure 13): The model is told to answer exactly one question correctly and three incorrectly, but it outputs all four answers correctly because it cannot override its default behavior of answering truthfully. In the annotated error case, the model successfully recognizes all four fast-food brand logos but then provides correct answers for all of them, violating the 1-right/3-wrong constraint.
Counterfactual Instruction (Figure 14): When shown an image that clearly depicts characters from the novel "Journey to the West" and instructed to say "Water Margin" and introduce one character if the image is from Journey to the West, the model instead provides a correct-but-non-counterfactual description of the Journey to the West content it sees.
Chart Modification (mentioned in Appendix E.4): The model outputs complete data tables when asked to change a specific value, rather than following the logical constraint to modify only the specified entry.

Why These Patterns Matter. Each error pattern maps to a fundamental cognitive capability that humans possess but current MLLMs lack. Orientation/reference-frame alignment requires maintaining a coherent 3D spatial model from 2D inputs—a capacity that is arguably the core of spatial intelligence. Cross-view identity persistence requires object-level representations that are invariant to viewpoint changes—a problem that the computer vision community has studied for decades under the label "object re-identification." Instruction adherence requires meta-cognitive control that can suppress prepotent responses (the strong tendency to answer truthfully) in favor of task-demanded behavior—a form of executive function.

These error patterns are not specific to any single model. Appendices D and E show failure cases from Doubao-Seed-1.6-vision-0815, but the fact that spatial and geometric reasoning scores are broadly weak (at or below 30% across all models in Table 4) suggests that these are limitations of current MLLM architectures as a class, not quirks of individual training runs.

Chain-of-Thought Behavioral Analysis

The paper provides a structured qualitative analysis of how reasoning-oriented MLLMs (specifically Doubao-Seed-1.6-vision-0815, which has an explicit "Think" mode) produce their reasoning chains on MME-CC tasks. This analysis is based on inspecting the CoT outputs and identifying structural patterns in how models approach visual reasoning problems.

The Three-Stage Reasoning Model. The paper identifies a consistent three-stage structure across CoT outputs (Section 4.2, Figure 3):

Stage 1: Problem Understanding and Information Collection. The model reads the textual prompt to understand the task rules and constraints, then scans the image(s) to identify key objects, spatial relationships, and other task-relevant visual elements. In the Satellite Image Matching example shown in Figure 3, this stage involves identifying the Guggenheim Bilbao building in the street-view photo, noting the presence of the river, the silver sculpture, and the building's architectural features.
Stage 2: Core Analysis and Reasoning. The model proposes candidate answers, checks them against visual evidence and task rules, and updates its assumptions when evidence contradicts initial hypotheses. This is where the bulk of the reasoning work occurs—in the Figure 3 example, this stage involves comparing the observed spatial layout (river position, building orientation, sculpture location) against the candidate positions on the satellite map, eliminating options that don't match, and converging on the correct locations through iterative hypothesis testing.
Stage 3: Conclusion Formation and Verification. The model assembles the confirmed evidence, outputs a final answer with a brief rationale, and performs a final consistency check (e.g., "Wait, let's make sure. Photo 1's elements: silver sculpture (left), ..."). This stage functions as a self-verification mechanism to catch errors before the answer is finalized.

The paper notes that "although task-specific tactics vary, the overall structure remains stable"—suggesting that this three-stage process is a general property of how reasoning-oriented MLLMs approach visual tasks, not an artifact of any particular task design.

Continuous Visual Extraction Throughout Reasoning. A key finding is that "visual information extraction is not confined to the beginning but occurs as needed throughout the reasoning process" (Section 4.2, Finding 2 in Figure 3). In the Satellite Image Matching example, the model repeatedly revisits the original images to re-check building orientation, relative layout, and specific architectural details. It does not extract all visual information upfront and then reason purely symbolically—instead, visual perception and abstract reasoning are interleaved, with reasoning demands triggering new rounds of visual inspection.

This observation has implications for MLLM architecture design. If visual extraction is continuous rather than feedforward, then architectures that process the image once and then reason over a fixed visual embedding may be fundamentally limited on tasks requiring iterative visual re-inspection. The observation may help explain why spatial and geometric reasoning scores are so low: these tasks require precisely the kind of iterative visual re-examination that current architectures do not support natively.

The Efficiency Cost of Self-Interruption. The paper identifies a behavioral pattern that the authors characterize as "excessive verification": the model frequently employs "wait"-style pauses for reflection and re-checks (visible as repeated instances of "wait," "wait wait," and "wait but let's make sure" in the Figure 3 CoT transcript). The authors observe that "moderate pausing can reduce errors; excessive pausing, however, leads to stalling and repetitive verification." This is particularly evident in complex spatial reasoning tasks where the model gets stuck in verification loops—repeatedly re-examining the same visual evidence without making progress toward an answer.

The authors hypothesize that "long reasoning chains dilute attention, obscure crucial visual details, and ultimately degrade outcomes," which they connect to the observation that "in Maze, which requires continued rule-based simulation and path planning, no model exceeds 2%." The Maze task requires sustained, cumulative reasoning over a large search space, and the CoT pattern suggests that models lose coherence over long chains, failing to maintain the global path-planning state while also attending to local visual details.

Textual Anchoring Improves Performance (Ablation). Building on the observation that visual extraction is fragile, the paper tests a simple intervention: adding the instruction "You should first describe the relevant content in the image according to the prompt, and then answer the question" to the original task prompt. The results are reported in Table 5 as an ablation (ablation score = base score ± delta).

The pattern of results shows mostly consistent gains:

Gemini-2.5-Pro improves slightly on average (+0.4% overall, with a +1.2% gain on Spatial Reasoning but a −0.4% drop on Geometric Reasoning).
Doubao-Seed-1.6-vision-0815 improves similarly (+0.3% overall).
o4-mini-high improves +0.3% overall.

The interpretation the paper offers is that "an initial textual description stabilizes subsequent reasoning by anchoring visual perception" and that "the improvements mainly arise from better textual alignment rather than stronger intrinsic visual reasoning." In other words, forcing the model to verbalize what it sees creates a textual memory trace that subsequent reasoning steps can reference, reducing the need for repeated visual extraction that may degrade over long chains. This is not a claim that the model's visual reasoning capability improves—rather, it is a claim about the fragility of visual information propagation through long reasoning chains, and how explicit textual description serves as a stabilizing mechanism.

Computational Cost and Model Configuration

Model Configuration. For proprietary (closed-source) models, the paper uses official inference APIs with default settings—the authors do not control generation hyperparameters. For open-source models, a unified decoding configuration is adopted: temperature set to (1.0) and top-p to (0.7), with all other hyperparameters following their respective defaults. This ensures that open-source models are compared under consistent stochastic decoding conditions, though the paper does not report whether multiple samples per question are generated (the scoring protocol evaluates one response per question per model).

Evaluation Metrics and Aggregation. The primary metric is accuracy (%) on each task category and overall—the fraction of questions for which the LLM judge assigns a score of (1). Per-task and per-dimension accuracy is computed by averaging across all questions in that subset. Since each question contributes exactly one binary score, no normalization or weighting is applied. The human baseline (Table 4) is computed on 99 sampled questions; the reported 95.86–95.92% human scores establish an upper bound that substantially exceeds the best model (Gemini-2.5-Pro at 42.66%), confirming substantial headroom for improvement.

Why This Design and Not Alternatives

Free-form output instead of multiple-choice. The deliberate choice of free-form answering (Table 3) eliminates several shortcut pathways: models cannot eliminate implausible options through textual reasoning, cannot guess based on answer-pattern distributions, and cannot rely on the fact that the correct answer is presented among options (recognition vs. generation). The cost is the need for LLM-as-judge scoring with its 95% agreement rate—a cost the paper accepts as necessary for validity, and validates through the 99-sample human agreement check.

Three-category taxonomy instead of per-task enumeration. By organizing 11 tasks into three cognitive dimensions, the paper enables statements that would be impossible from per-task scores alone: "Spatial and geometric reasoning remain broadly weak ((\leq 30%)) while Visual Knowledge Reasoning is substantially stronger." This structure also enables the error analysis to identify patterns that cross task boundaries within a dimension, supporting the claim that the errors reflect fundamental cognitive limitations rather than task-specific quirks.

Human-in-the-loop quality control instead of automated validation. The annotation team structure with dual review, subtask lead escalation, and expert-only protocols for complex tasks is expensive but necessary because the benchmark's validity depends on ground-truth correctness. If reference answers contain errors, then models that get the "right" answer according to flawed ground truth receive inflated scores, and the error analysis becomes unreliable. The paper's extensive quality control—including the final model-based filtering that removes ambiguous items—represents an investment in ground-truth reliability that is essential for the benchmark's diagnostic claims to be trustworthy.

Forced vision-centric design instead of balanced text-vision tasks. The paper's core methodological commitment is that to measure visual cognitive capacity, you must eliminate the possibility of text-only solutions. This means that every MME-CC question embeds its answer-critical information in images, with text prompts providing only rules and constraints. This design choice is what enables the paper to claim that "a model that scores 74.63% on Visual Knowledge Reasoning cannot be attributed to reading answer-bearing text"—a claim that cannot be made for benchmarks like MMMU or MathVista, where text and images jointly carry the information needed for the answer.

4. Key Insights and Innovations

Innovation 1: The "Forced Visual" Benchmark Design as a Methodological Intervention

The paper's most intellectually distinctive contribution is not that MLLMs struggle with spatial reasoning — that much was already suspected from scattered observations in the literature. Rather, it is the systematic methodological argument that the field's existing benchmarking practices are structurally incapable of measuring what they claim to measure, and that fixing this requires a design philosophy — "vision-based" evaluation — that prior benchmarks have not implemented. The paper operationalizes this philosophy through a specific, falsifiable criterion: can a question be answered correctly without looking at the images? For existing benchmarks like MMMU, MathVista, MMBench, and MMStar, the answer is demonstrably yes — models can exploit textual cues, OCR, format priors, and answer-pattern distributions to achieve scores that misrepresent their visual reasoning capacity. For MME-CC, the answer is designed to be no.

This is a conceptual contribution to benchmark validity theory as applied to multimodal AI, not merely a new dataset. Prior work in the multimodal evaluation space — MME (Fu et al., 2023), MMBench (Liu et al., 2024), MMMU (Yue et al., 2024, 2025), and even more vision-focused efforts like ZeroBench (Roberts et al., 2025) — implicitly assumed that including images in questions was sufficient to test visual reasoning. The field's dominant assumption was that if a benchmark contains images, then performance on it reflects visual capability. MME-CC challenges this assumption directly: including images is necessary but not sufficient; the benchmark must also guarantee that the images carry the answer-critical information and that text-based shortcuts are foreclosed. The "Vision-based" column in Table 3 — where leading benchmarks like MME and MMMU are marked "✗" while only ZeroBench and MME-CC are marked "✓" — is a polemical statement compressed into a single table cell. It asserts that the field's most-cited benchmarks are not actually testing the thing they claim to test.

What distinguishes this from prior critiques of benchmark validity is that it is constructive, not merely diagnostic. The paper does not simply show that models exploit shortcuts (as NaturalBench and similar studies have done) — it builds a benchmark that systematically closes those shortcuts through specific, enumerable design choices: free-form output eliminates multiple-choice guessing; rigid image post-processing removes OCR-able text, compasses, and UI hints; task-specific prompts contain only rules, never location-identifying or answer-bearing information; and the model-based filtering stage removes items solvable through superficial pattern matching. These are not ad-hoc fixes — they constitute a design methodology that future benchmarks can adopt, adapt, or critique. The paper's contribution is thus as much about how to build valid visual reasoning benchmarks as it is about the specific benchmark it built.

The significance of this innovation extends beyond the paper's immediate findings. If the field accepts the premise that "vision-based" in the MME-CC sense is a necessary property of benchmarks that claim to test visual reasoning, then a large fraction of published multimodal model evaluations must be reinterpreted: scores on non-vision-based benchmarks reflect some mixture of visual and textual capability, and improvements over time may reflect advances in the textual component rather than genuine visual understanding. This reframing has direct implications for how researchers report and compare model capabilities — it suggests that vision-based benchmarks like MME-CC should become a standard component of evaluation suites alongside existing general-purpose benchmarks, with the understanding that they measure something qualitatively different. It is a call for evaluation dimension disentanglement, analogous to how the NLP community separated language understanding from world knowledge in reading comprehension benchmarks.

Empirically, the validity of this intervention is supported by the stark discrepancy between model performance on MME-CC's Visual Knowledge Reasoning tasks (where Gemini-2.5-Pro scores 74.63%) and its Spatial/Geometric Reasoning tasks (23.80% and 29.56% respectively, from Table 4). If text-based shortcuts were available, this discrepancy would be smaller — the best models, which are also the best at exploiting textual cues, would perform more uniformly across task categories. The large gap suggests that MME-CC is successfully measuring something that general-purpose benchmarks conflate: the model's ability to extract and reason about spatial/geometric information specifically from visual inputs, separately from its ability to reason about semantic content that happens to be visually presented.

Innovation 2: A Principled Cognitive Taxonomy for Visual Reasoning

Prior to MME-CC, benchmarks that attempted to assess visual cognitive capacity — ZeroBench (Roberts et al., 2025), VisuLogic (Xu et al., 2025), and similar efforts — largely enumerated individual visual reasoning tasks without organizing them into a coherent framework that explains why certain tasks are grouped together and what cognitive dimensions they probe. The implicit taxonomy was flat: spatial puzzles are one thing, visual logic another, common-sense visual reasoning a third, and so on — a list of task types with no principled structure relating them.

MME-CC proposes and operationalizes a three-dimensional cognitive taxonomy: Spatial Reasoning (understanding 3D spatial relationships from 2D images), Geometric Reasoning (abstract pattern recognition, constraint satisfaction, and planning on visual structures), and Visual Knowledge Reasoning (integrating semantic understanding with rule-following and meta-cognitive control). This is not merely a convenient categorization for presenting results — it is a substantive claim about the structure of visual cognitive capacity that carries theoretical weight. The claim is that these three dimensions represent qualitatively distinct cognitive demands that draw on different underlying capabilities, and that performance on one dimension does not necessarily predict performance on another.

What makes this innovation distinctive is that the taxonomy is validated empirically through the benchmark results themselves. If the three dimensions were merely relabeling the same underlying capability, we would expect performance to be highly correlated across them — a model good at Spatial Reasoning would also be good at Geometric Reasoning, and vice versa. The results in Table 4 show a more complex pattern. Gemini-2.5-Pro achieves 74.63% on Visual Knowledge Reasoning but only 23.80% on Spatial Reasoning — a gap of over 50 percentage points. GPT-5 (high) scores 30.63% on Spatial Reasoning but only 23.64% on Geometric Reasoning, reversing Gemini's pattern. Doubao-Seed-1.6-vision-0815 shows the most balanced profile (~22–32% across all three dimensions), while several open-source models show severe asymmetries (e.g., GLM-4.5V at 64.73% on VKR vs. 13.27% on Spatial). These divergent profiles would be impossible if all tasks measured the same latent capability. They suggest that the taxonomy is capturing real, separable dimensions of variation in model capabilities.

The theoretical significance of this taxonomy lies in its potential to redirect research attention. If Spatial Reasoning and Geometric Reasoning are indeed distinct from Visual Knowledge Reasoning — both in cognitive demand and in model performance — then improvements in one may require fundamentally different interventions than improvements in another. Visual Knowledge Reasoning, which the paper shows is substantially stronger across models, likely benefits from the semantic knowledge and instruction-following abilities that are already well-developed in large language models and transfer to the visual domain when vision encoders are added. Spatial and Geometric Reasoning, by contrast, may require architectural innovations — explicit 3D geometric representations, mechanisms for iterative visual re-inspection, or training objectives that specifically target multi-view consistency — that go beyond simply scaling up existing vision-language architectures. The taxonomy thus serves a diagnostic function for research prioritization: it tells the field where to focus improvement efforts, not just that improvement is needed.

This is a fundamental contribution to the conceptual vocabulary of multimodal evaluation. The field can now ask not just "how good is this model at visual reasoning?" but "how good is this model at spatial reasoning specifically, and how does that compare to its geometric and visual knowledge reasoning?" The taxonomy enables fine-grained capability profiling that aggregate scores on flat benchmarks cannot provide. It is not a small refinement of existing categorization schemes (which were largely ad-hoc) — it is a proposed first-principles decomposition that, if adopted, would structure how the community thinks about visual cognitive capacity.

Innovation 3: Error Pattern Diagnosis as a First-Class Evaluation Output

Standard multimodal benchmarks produce two things: an aggregate accuracy score and, sometimes, per-task subscores. These tell you how well a model performs but not how it fails. MME-CC makes a distinctive contribution by treating error pattern diagnosis as a first-class output of evaluation, on equal footing with quantitative scores. The three recurring error patterns identified — orientation and reference-frame confusion, fragile cross-view identity persistence, and over-reliance on literal descriptions under instruction constraints — are not post-hoc observations about interesting failures. They are structured diagnostic categories that emerge from task designs specifically engineered to expose them, and they are validated by concrete, annotated examples drawn from multiple tasks and multiple models (Figures 5–14 in Appendices D and E).

This represents a shift in what evaluation benchmarks are expected to deliver. In the dominant paradigm, a benchmark provides a score, and it is up to subsequent analysis papers to diagnose why models fail. MME-CC integrates the diagnostic analysis into the benchmark itself, providing not just "Gemini-2.5-Pro scores 42.66%" but "here are the specific cognitive failure modes that constitute the missing 57.34%, organized into categories that map to fundamental visual reasoning capabilities." This makes the benchmark actionable for model improvement in a way that pure accuracy scores are not: a model developer who reads the error analysis knows that fixing orientation/reference-frame alignment requires improving the model's ability to maintain consistent 3D coordinate systems across viewpoint changes, not just "improving spatial reasoning" in some vague sense.

The error patterns identified are significant beyond their descriptive value because they point to fundamental architectural limitations rather than training data gaps. Cross-view identity persistence — recognizing that the same couch in three different photos is one object, not three — is not a problem that can be solved by collecting more training data with couches in it. It requires object-level representations that are invariant to viewpoint, lighting, and occlusion changes — a problem that the computer vision community has studied for decades under the label "object re-identification" and that remains partially unsolved even in dedicated vision systems. The fact that MLLMs fail at this systematically (as shown in Figure 7's Indoor Deduplication Counting error case) suggests that current vision-language architectures, which typically process images into fixed embedding vectors that are then attended to by a language model, may lack the representational machinery needed for viewpoint-invariant object tracking. Similarly, the failure to follow counterfactual instructions (Sandbagging, Counterfactual Instruction) points to a limitation in meta-cognitive control — the model's inability to override prepotent truthful responses — that is unlikely to be solved by scaling alone, since larger models in the paper's evaluation (Gemini-2.5-Pro) still exhibit this failure.

This innovation has an important intellectual precedent in the adversarial robustness and failure mode analysis traditions in computer vision and NLP — where systematically characterizing how models fail has driven architectural innovations (e.g., the discovery that CNNs are vulnerable to imperceptible pixel perturbations led to adversarial training). MME-CC adapts this philosophy to multimodal evaluation: rather than waiting for failure modes to be discovered incidentally, build the benchmark to make specific failure modes visible and measurable. This is a conceptual contribution to evaluation methodology that extends beyond this specific benchmark.

Innovation 4: The Fragility of Visual Information Propagation in Chain-of-Thought Reasoning

One of the paper's most striking findings — and one with direct implications for how reasoning-oriented MLLMs are designed and trained — is the observation from the Chain-of-Thought behavioral analysis that visual information extraction is continuous throughout the reasoning process, not confined to an initial perception phase, and that this continuous extraction is fragile, degrading over long reasoning chains. The paper documents this through detailed analysis of Doubao-Seed-1.6-vision-0815's CoT outputs (Figure 3), showing that the model repeatedly revisits images to re-check visual details and that excessive self-interruption ("wait," "wait wait") leads to stalling and repetitive verification without progress.

This finding challenges an implicit assumption in current MLLM architectures: that visual processing can be done once, producing a fixed embedding that subsequent reasoning steps can reference. If visual extraction is genuinely continuous and iterative — if reasoning steps trigger new rounds of visual inspection rather than operating on a static representation — then architectures that encode images into a single vector and then reason over that vector are fundamentally limited. They cannot "look again" at the image when reasoning reveals a need to check a specific detail that was not encoded in the initial forward pass. The paper's observation that performance on the Maze task — which requires sustained spatial simulation and path planning — is catastrophic (no model exceeds 2% in the base setting, Table 5) provides circumstantial evidence for this interpretation: the Maze task requires maintaining a global path-planning state while also attending to local visual details (which edges are blocked, which cells contain numbers), and the CoT pattern suggests that models lose coherence over long chains because the visual information needed for local decisions becomes degraded.

The ablation experiment in Table 5 — where adding "You should first describe the relevant content in the image according to the prompt, and then answer the question" yields consistent, though modest, gains — provides further evidence for this interpretation. The gains are small (0.3–0.4% on average across the three models tested), but their consistency suggests that textual description serves as a stabilizing mechanism: by converting visual information into text early in the reasoning chain, the model creates a textual memory trace that subsequent reasoning steps can reference, partially compensating for the degradation of visual information over long chains. The paper's interpretation — that "the improvements mainly arise from better textual alignment rather than stronger intrinsic visual reasoning" — is an important caveat: this is a mitigation of a fundamental limitation, not a solution to it.

This finding is significant because it connects the paper's evaluation results to an open research question in MLLM architecture design. The current paradigm of vision-language models — encode image, attend to image tokens alongside text tokens, generate text — implicitly assumes that visual information is stable and can be accessed uniformly throughout generation. The paper's CoT analysis suggests that this assumption is violated in practice: visual information degrades, attention dilutes, and the model's ability to reference specific visual details deteriorates as the generation length increases. This is not merely an inference-time problem that can be fixed with better prompting — it may require architectural innovations such as iterative visual grounding mechanisms, explicit memory for visual features, or training objectives that reward sustained visual attention over long contexts. The paper does not propose such innovations, but the finding itself identifies a bottleneck that the field had not previously characterized in the context of multimodal CoT reasoning, making it a genuinely novel empirical contribution with clear implications for future research.

5. Experimental Analysis

Evaluation Methodology

Dataset. MME-CC consists of 1,173 expert-annotated questions organized into 11 subtasks across three cognitive dimensions: Spatial Reasoning (319 questions), Geometric Reasoning (605 questions), and Visual Knowledge Reasoning (249 questions). Data sources include Google Maps/Street View, real-estate VR tours, physical photographs, game screenshots, web images, auto-generated mazes, and an internal dataset, all processed through the four-stage human-in-the-loop pipeline described in Section 3. No formal train/dev/test split exists — the benchmark is purely for evaluation, with all 1,173 questions used for reporting results in Table 4.
Base model(s). Sixteen multimodal large language models are evaluated, spanning both closed-source proprietary systems and open-source releases. The closed-source models include Gemini-2.5-Pro, Gemini-2.5-Flash, GPT-5 (high), GPT-4.1, GPT-4o-1120, o4-mini (high), and two variants of Doubao-Seed-1.6-vision-0815 (Think and Nonthink). The open-source models include GLM-4.5V, Qwen2.5-VL-72B-Instruct, Qwen2.5-VL-32B-Instruct, Qwen2.5-VL-7B-Instruct, MiMo-VL-7B-RL, GLM-4.1V-9B-Thinking, InternVL3-8B, and Keye-VL-8B-Preview. The selection spans a wide parameter range (7B to proprietary-scale), multiple training paradigms (standard VLMs, reasoning-oriented models with explicit CoT training, and RL-tuned variants), and three parameter scales within the Qwen2.5-VL family (7B, 32B, 72B), enabling analysis of scaling trends within a single architecture lineage.
Metrics. The primary metric is accuracy — the fraction of questions for which the model's answer is judged correct. Accuracy is computed per-dimension (Spatial, Geometric, Visual Knowledge) and overall. A human baseline is established on 99 randomly sampled questions (33 per category), achieving 95.86–95.92% accuracy. A secondary metric, reported incidentally in Table 2, is average input/output token length measured via Doubao-Seed-1.6-vision-0815 tokenization, which serves as a proxy for reasoning complexity — longer outputs indicate more extensive reasoning chains. No statistical significance tests, confidence intervals, or inter-annotator agreement metrics are reported for the model evaluation results.
Baselines. The paper does not use algorithmic baselines in the traditional sense — there is no "random guessing" or "simple heuristic" baseline because the benchmark consists of free-form visual reasoning questions with no trainable components. The effective baselines are: (1) human performance, established on the 99-sample subset (95.92% overall), which defines the upper bound; (2) model rankings within the same family (e.g., Qwen2.5-VL 7B vs. 32B vs. 72B), which serve as internal scaling baselines; (3) the reasoning vs. non-reasoning contrast within model families that offer both variants (Doubao Think vs. Nonthink, GPT-5 vs. GPT-4.1), which tests whether explicit Chain-of-Thought training improves visual reasoning specifically.
Generation budget / compute accounting. The paper does not use a FLOPs-based or token-based compute budget for fair comparison. Each model generates one response per question using its default or specified decoding configuration (temperature 1.0, top-p 0.7 for open-source models; API defaults for proprietary models). No multiple-sampling, majority voting, or best-of-N aggregation is employed. The evaluation is thus a pass@1 measurement — what the model produces in a single forward pass (potentially with internal CoT for reasoning-oriented models). There is no attempt to equalize compute across models of different sizes or architectures, meaning that the scores in Table 4 reflect both model capability and the computational cost of generating the answer (longer CoT chains consume more tokens).
Cross-validation / statistical protocol. No cross-validation is used for model evaluation — each model is evaluated once on the full 1,173-question set. The 99-sample human validation of the LLM judge (DeepSeek-V3-0324) uses random stratified sampling (33 from each category) and achieves 95% agreement with human judgments, but this validates the scoring mechanism, not the stability of model rankings. No error bars, bootstrapping, or significance tests are reported for any result in Tables 4 or 5.

Main Quantitative Results

Aggregate Model Performance (Table 4)

The headline result is that no model achieves acceptable performance on the benchmark as a whole. Gemini-2.5-Pro — the best-performing model — scores only 42.66% overall, while the strongest open-source model, GLM-4.5V, scores 30.45%. The best closed-source model (Gemini-2.5-Pro at 42.66) outperforms the best open-source model (GLM-4.5V at 30.45) by 12.21 absolute percentage points, but both fall dramatically short of the human baseline of 95.92%. The worst model, Qwen2.5-VL-7B-Instruct, scores 7.50% overall.

The overall ranking of the top five models is: Gemini-2.5-Pro (42.66) > GPT-5 high (40.25) > Doubao-Seed-1.6-vision-0815 Think (40.08) > Gemini-2.5-Flash (37.57) > o4-mini high (35.00). Notably, the third-place model (Doubao Think, 40.08) and second-place model (GPT-5, 40.25) are separated by only 0.17 percentage points — effectively tied within any reasonable measurement resolution, especially given the absence of reported confidence intervals.

Category-Level Performance: Visual Knowledge Reasoning Far Outstrips Spatial and Geometric

The most salient pattern in Table 4 is the massive performance asymmetry across reasoning dimensions. For Visual Knowledge Reasoning, the best models achieve substantial scores: Gemini-2.5-Pro at 74.63%, Gemini-2.5-Flash at 72.90%, Doubao Think at 66.70%, GPT-5 at 66.47%, and GLM-4.5V at 64.73%. In contrast, spatial reasoning peaks at only 30.63% (GPT-5) and geometric reasoning at 31.50% (Doubao Think). Every model scores dramatically higher on Visual Knowledge Reasoning than on either Spatial or Geometric Reasoning, with gaps ranging from roughly 20 percentage points (GPT-4.1: 56.30% VKR vs. 27.90% SR) to over 50 percentage points (GLM-4.5V: 64.73% VKR vs. 13.27% SR and 13.34% GR).

Within each category, the ranking is:

Spatial Reasoning: GPT-5 high (30.63%) > Gemini-2.5-Pro (23.80%) > GPT-4.1 (27.90%) — wait, this ordering is internally inconsistent in Table 4. The actual top three are: GPT-5 high at 30.63%, GPT-4.1 at 27.90%, and o4-mini high at 25.00%. Gemini-2.5-Pro ranks fourth at 23.80%.
Geometric Reasoning: Doubao Think (31.50%) > Gemini-2.5-Pro (29.56%) > Doubao Nonthink (23.82%) > GPT-5 high (23.64%).
Visual Knowledge Reasoning: Gemini-2.5-Pro (74.63%) ≫ Gemini-2.5-Flash (72.90%) > Doubao Think (66.70%) > GPT-5 high (66.47%) > GLM-4.5V (64.73%).

The divergent profiles are striking. Gemini-2.5-Pro dominates Visual Knowledge Reasoning but is only fourth in Spatial Reasoning and second in Geometric Reasoning. GPT-5 high leads Spatial Reasoning but falls to fourth in Geometric Reasoning and fourth in Visual Knowledge Reasoning. Doubao Think leads Geometric Reasoning but is third in both Spatial and Visual Knowledge. No single model leads across all three dimensions, suggesting that the cognitive capacities measured by each dimension are at least partially independent and that different training recipes optimize for different subsets of these capacities.

The Reasoning Advantage Is Consistent but Modest

Reasoning-oriented models — those with explicit Chain-of-Thought training or inference-time reasoning — consistently outperform their non-reasoning counterparts within the same model family, but the advantage is not large enough to close the gap to the human baseline. In Table 4:

GPT-5 high (reasoning, 40.25 overall) vs. GPT-4.1 (non-reasoning, 32.14 overall): The reasoning model holds an 8.11 percentage point advantage overall, with the gap concentrated in Geometric Reasoning (23.64% vs. 12.22%, a +11.42 point difference) and Visual Knowledge Reasoning (66.47% vs. 56.30%, +10.17), while Spatial Reasoning shows a smaller advantage (30.63% vs. 27.90%, +2.73). This asymmetry — large gains in Geometric and VK Reasoning, smaller in Spatial — suggests that CoT benefits different cognitive dimensions unequally.
Doubao Think (reasoning, 40.08 overall) vs. Doubao Nonthink (25.96 overall): A +14.12 percentage point overall advantage. The gap is largest in Visual Knowledge Reasoning (66.70% vs. 30.43%, an enormous +36.27 point difference), substantial in Geometric Reasoning (31.50% vs. 23.82%, +7.68), but actually reversed in Spatial Reasoning (22.03% vs. 23.63%, a −1.60 point disadvantage for the Think variant). This reversal — the reasoning model performing worse on spatial tasks — is a non-obvious finding that the paper does not comment on directly, but it suggests that CoT training may redirect model capacity away from spatial processing toward semantic reasoning, or that the longer reasoning chains in the Think variant amplify the visual information degradation problem identified in Section 4.2.
GPT-4o-1120 (non-reasoning, 26.88) vs. GPT-4.1 (non-reasoning, 32.14): Even among non-reasoning models within the same family, the newer GPT-4.1 improves over GPT-4o-1120 by +5.26 percentage points overall, with the largest relative improvement in Geometric Reasoning (12.22% vs. 10.12%, a +2.10 absolute gain on a very low base).

The pattern of reasoning advantages being largest in Visual Knowledge Reasoning and smallest (or negative) in Spatial Reasoning is consistent with the paper's observation that CoT "provides additional opportunities for iterative verification of recognition outcomes and intermediate inferences." Verification is most useful when the model already has the content knowledge but needs to avoid careless errors — which describes Visual Knowledge Reasoning tasks where the model must recognize entities and follow rules. Verification is least useful when the fundamental bottleneck is not reasoning errors but the inability to construct a stable spatial representation — which describes Spatial Reasoning tasks where the model's visual extraction itself is flawed regardless of how much verification is applied.

Scaling Laws Within a Model Family (Qwen2.5-VL)

The Qwen2.5-VL family provides a clean test of parameter scaling, with models at 7B, 32B, and 72B parameters evaluated under identical conditions. Performance increases monotonically with scale: Qwen2.5-VL-7B-Instruct (7.50% overall) < Qwen2.5-VL-32B-Instruct (14.39%) < Qwen2.5-VL-72B-Instruct (23.59%). This is an improvement of +16.09 percentage points from 7B to 72B — a 3.15× relative gain on the overall score.

The scaling gains are not uniform across dimensions:

Spatial Reasoning: 4.70% (7B) → 9.03% (32B) → 12.47% (72B), a +7.77 point gain.
Geometric Reasoning: 3.22% (7B) → 8.56% (32B) → 8.96% (72B), a +5.74 point gain but with diminishing returns from 32B to 72B (only +0.40).
Visual Knowledge Reasoning: 14.57% (7B) → 25.57% (32B) → 49.33% (72B), a +34.76 point gain that dominates the overall improvement.

This pattern reveals that scaling predominantly improves Visual Knowledge Reasoning, which nearly triples from 7B to 72B, while Spatial and Geometric Reasoning improve much more modestly. If these trends extrapolate, even models substantially larger than 72B (e.g., 400B+ parameters) would still score well below 50% on Spatial and Geometric Reasoning while potentially saturating Visual Knowledge Reasoning. The paper notes this as evidence that "complex visual perception and reasoning tasks require broader knowledge capacity to support effective inference, while smaller-scale models face inherent limitations in achievable performance," but the differential scaling rates across dimensions suggest that parameter count alone may not be the bottleneck — some dimensions may require qualitatively different architectural capabilities that scaling does not confer.

Human Baseline Establishes a High Ceiling

The human evaluation on 99 randomly sampled questions (Table 4, bottom row) achieves 95.92% overall, with 95.86% on Spatial Reasoning, 95.83% on Geometric Reasoning, and 95.83% on Visual Knowledge Reasoning. This establishes that the benchmark is not impossible — humans with relevant background (the evaluators were "students who did not participate in the question setting, with higher degrees," Section 3.1) can solve these tasks with near-perfect accuracy. The gap between the best model (42.66%) and human performance (95.92%) is 53.26 percentage points — a chasm that underscores how far current MLLMs are from human-like visual cognitive capacity on these specific task types. Critically, the human scores are near-identical across all three dimensions (95.86, 95.83, 95.83), indicating that the benchmark's cognitive dimensions are equally solvable by humans — the large inter-dimension performance gaps observed in models are not an artifact of the tasks being unequally difficult for intelligent agents in general, but rather reflect genuine asymmetries in MLLM capabilities.

Per-Subtask Performance: Where Models Succeed and Fail (Table 5)

Table 5 reports per-subtask scores for three representative models (Gemini-2.5-Pro, Doubao-Seed-1.6-vision-0815, and o4-mini-high) in the base setting (the values in parentheses). These reveal that even within the generally weak Spatial and Geometric categories, performance is highly task-dependent:

Satellite Image Matching (Spatial): Gemini-2.5-Pro scores 28.3%, Doubao Think scores 39.6%, o4-mini scores 30.3%. This is the highest-scoring spatial subtask across all three models.
Indoor Directional Reasoning (Spatial): Scores collapse: Gemini 14.3%, Doubao 5.7%, o4-mini 11.2%. This task — requiring propagation of directional constraints across rooms — is the most difficult spatial subtask.
Indoor Deduplication Counting (Spatial): Gemini 28.8%, Doubao 20.8%, o4-mini 33.5%. Variation across models is substantial (12.7-point spread).
Maze (Geometric): Catastrophic performance across all models: Gemini 1.1%, Doubao 0.8%, o4-mini 1.2%. This is the single most difficult subtask in the entire benchmark — effectively unsolvable by current MLLMs. The requirement for sustained spatial path planning with global optimality constraints defeats all models near-totally.
Gomoku Variation (Geometric): Gemini 34.8%, Doubao 14.9%, o4-mini 12.0%. This is the highest-scoring geometric subtask for Gemini but substantially harder for the other two models.
Jigsaw Puzzle (Geometric): A striking divergence: Doubao scores 70.6% while Gemini scores 30.4% and o4-mini 27.0%. The Doubao model's 40-point advantage on this specific subtask is the largest inter-model gap on any single task, suggesting that Doubao's training includes visual pattern-matching capabilities — perhaps from puzzle-like training data — that other models lack.
Unblock Me (Geometric): Gemini 26.8%, Doubao 28.8%, o4-mini 21.4%. Performance is modest but not catastrophic, unlike Maze — suggesting that the difficulty is not inherent to planning problems but specific to the sustained global optimization required by Maze.

The per-subtask breakdown makes clear that aggregate Geometric Reasoning scores (~23–30%) conceal enormous variation: Jigsaw Puzzle can be solved well by the right model, Maze is unsolvable by any model, and other tasks fall in between. This suggests that "geometric reasoning" as a category is not a unitary capability but a collection of distinct skills that different models possess to different degrees.

Ablation: Textual Description Anchoring (Table 5)

Table 5 presents a controlled experiment: the same models are evaluated with and without an additional instruction — "You should first describe the relevant content in the image according to the prompt, and then answer the question." The ablation score is reported alongside the base score (in parentheses) with the signed difference (±Δ).

The overall effect is consistently positive but small: Gemini-2.5-Pro gains +0.4% overall (23.5% → 23.9%), Doubao Think gains +0.3% (25.4% → 25.7%), o4-mini gains +0.3% (20.2% → 20.5%). These changes are less than 1 percentage point — negligible by any practical standard. However, the consistency of the direction (all three models improve) and the task-level detail reveal more nuanced patterns:

Satellite Image Matching sees the largest consistent gains: +2.2 for Gemini, +1.4 for Doubao, +1.6 for o4-mini. This task, which requires matching ground-level views to satellite positions, may benefit specifically from verbalization of visual landmarks before attempting the spatial correspondence problem.
Gains are not uniform across models and tasks: Doubao Think improves on 5 of 7 reported subtasks but degrades on Indoor Directional Reasoning (−0.9) and Indoor Deduplication Counting (−1.8), and shows no change on Maze (−0.2). Gemini-2.5-Pro improves on 6 of 7 subtasks but degrades on Gomoku Variation (−3.8, the largest single decline observed). o4-mini shows the flattest profile, with changes rarely exceeding ±0.5 points.
The Maze task remains unsolvable: Even with textual anchoring, Maze scores are 1.9% (Gemini), 0.6% (Doubao), and 1.5% (o4-mini). The fact that forcing models to verbalize the maze layout does not rescue performance strongly supports the paper's interpretation that the bottleneck is sustained spatial planning, not initial visual perception.

The paper interprets these results cautiously, noting that "the improvements mainly arise from better textual alignment rather than stronger intrinsic visual reasoning." This is an important qualification: the ablation does not demonstrate improved visual reasoning — it demonstrates that creating a textual memory trace of visual content partially compensates for the degradation of visual information over long reasoning chains, as discussed in the CoT analysis (Section 4.2).

Ablation Studies and Robustness Checks

LLM-as-judge validation: The paper validates the scoring protocol by manually evaluating 99 randomly sampled items (33 from each reasoning category) and comparing human judgments to DeepSeek-V3-0324's automated scores. The reported agreement rate is 95%. This is a critical validation because the entire quantitative evaluation framework rests on the judge's reliability — if the judge mis-scores answers, all numbers in Tables 4 and 5 are unreliable. The 95% agreement is high, but the paper does not report: (1) whether errors are biased (does the judge tend to be lenient or strict relative to humans?), (2) whether agreement varies by task category (33 samples per category may not be enough to detect category-specific biases), or (3) inter-annotator agreement among the human evaluators themselves (if human agreement is 95%, then the judge is perfectly reliable; if human agreement is 98%, then the judge introduces noise). Without these details, the 95% figure is reassuring but incomplete.

Task-specific grading prompts (Unblock Me, Appendix C): The Unblock Me task uses a specialized grading prompt that separately evaluates the minimum move count and the set of moved blocks (order-independent). The prompt includes worked examples of correct and incorrect answers with JSON outputs. This is a robustness measure: generic exact-match scoring would fail on Unblock Me because the answer has structured components, and the specialized prompt is designed to handle this structure. However, no validation of this specialized prompt against human judgments is reported — the 95% agreement figure applies only to the general scoring prompt, and it is possible that the Unblock Me scoring is less reliable if the judge model (DeepSeek-V3-0324) struggles with the structured block-set comparison.

Model-based filtering of the benchmark: As described in the data construction pipeline (Section 2.3), Gemini-2.5-Pro was used to filter the initial data pool, removing approximately 50% of candidate questions that were trivially easy (model accuracy >95%), semantically redundant, or lacking discriminative value. The fact that Gemini-2.5-Pro was also the best-performing model in the final evaluation (Table 4) raises a concern: if the same model was used for both filtering and evaluation, the benchmark may be systematically harder for models that are not Gemini-2.5-Pro because questions that were easy for Gemini were removed, while questions that were hard for Gemini were retained. This is not a fatal flaw — the filtering used a preliminary version of Gemini, and the evaluation used a potentially different checkpoint, and the 50% removal rate is not implausible for a well-designed benchmark — but it is a methodological confound that the paper does not discuss. If a different model had been used for filtering, the difficulty distribution might differ, and the relative rankings in Table 4 might shift.

Human baseline sampling: The human baseline is evaluated on only 99 of the 1,173 questions (8.4% of the benchmark). While 95.92% with near-perfect inter-dimension consistency is strong evidence that the benchmark is solvable by humans, the sample size is modest. If the 99 questions happen to be easier than the average question in the full benchmark (the paper says they are "randomly sampled" but does not describe the sampling procedure), then the 95.92% figure overestimates human performance on the full set. Given that the gap between human and model performance is 53 points, even a substantial overestimate (e.g., true human performance at 90%) would not change the qualitative conclusion, but the absolute ceiling matters for tracking progress — if the true ceiling is lower, models may appear to be making faster progress than they actually are.

No multi-sample or ensemble evaluation: The evaluation protocol uses a single response per model per question. This means that the reported scores conflate model capability (can the model solve this problem?) with model consistency (does the model solve this problem reliably across runs?). For stochastic decoding (temperature 1.0, top-p 0.7 for open-source models), the same model might produce a correct answer on one run and an incorrect answer on another. The pass@1 metric is a valid choice — it measures what a user would get from a single query — but it means that models with higher variance in their outputs (which may include reasoning models with long CoT chains) are penalized relative to models with lower variance. If the paper had reported pass@k for some k > 1 (e.g., generating 4 responses and checking if any is correct), it would provide additional information about whether failures are due to fundamental incapacity or output instability.

No Chain-of-Thought length analysis: While Section 4.2 provides a qualitative analysis of CoT structure, the paper does not report any quantitative metrics on CoT behavior — average chain length per task, correlation between chain length and correctness, or differences in chain structure between correct and incorrect answers. The observation that "long reasoning chains dilute attention" and lead to degraded outcomes is plausible based on the qualitative examples, but without quantitative evidence (e.g., showing that accuracy decreases as chain length increases beyond some threshold, or that correct answers have systematically different chain structures than incorrect ones), it remains a hypothesis rather than a demonstrated finding. The table in the paper that would support this — a breakdown of accuracy by chain length decile — is not presented.

Critical Assessment

Claim 1: "Existing benchmarks overemphasize textual reasoning and fail to systematically capture vision-centric cognitive behaviors"

This claim is supported by design argument rather than direct experiment. The paper does not run MMMU or MathVista questions through a text-only baseline to demonstrate that text suffices for answering them. Instead, it argues from task construction principles: MME-CC embeds answer-critical information in images and strips textual shortcuts, while prior benchmarks do not. The "Vision-based" column in Table 3 encodes this as a binary property, marking MME and MMMU as "✗" — but this is a judgment about benchmark design, not an empirical demonstration that models actually exploit text shortcuts on those benchmarks.

The claim is plausible and well-argued, but a direct experiment that would strengthen it is: take MMMU questions, remove the images, and evaluate whether models still perform above chance. If they do — or if performance with images only modestly exceeds performance without — then the claim is empirically validated. The paper cites prior work (NaturalBench, EasyARC, VLSBench) that has demonstrated shortcut exploitation, but does not itself replicate or extend those findings. The claim is thus supported by citation and design rationale, not directly tested.

Claim 2: "Spatial and geometric reasoning remain broadly weak (≤30% across all models)"

This is directly and unequivocally supported by Table 4. The maximum Spatial Reasoning score is GPT-5 high at 30.63%. The maximum Geometric Reasoning score is Doubao Think at 31.50%. No model, regardless of architecture or training paradigm, exceeds approximately 31% on either category. The finding is robust across 16 models spanning multiple families, scales, and training approaches. The weakness is not a property of particular models or training recipes — it is a property of the current MLLM landscape as a whole.

One caveat: the ≤30% figure applies to the aggregate category scores. As Table 5 reveals, per-subtask performance within categories varies enormously. Doubao Think scores 70.6% on Jigsaw Puzzle (Geometric), far above 30%, while Maze (also Geometric) scores 0.8%, far below. The aggregate masks this heterogeneity. The claim that "spatial and geometric reasoning remain broadly weak" is true as stated (averages ≤30%), but a reader might incorrectly infer that all spatial/geometric tasks are uniformly difficult, when in fact some are tractable for specific models. The per-subtask results suggest that "spatial reasoning" and "geometric reasoning" may not be coherent cognitive capacities that can be summarized by a single score — they are bundles of distinct skills that future benchmarks should disaggregate further.

Claim 3: "Closed-source models consistently outperform open-source counterparts"

Supported, but with important nuance. The top four models in Table 4 are all closed-source: Gemini-2.5-Pro (42.66), GPT-5 high (40.25), Doubao Think (40.08), Gemini-2.5-Flash (37.57). The fifth-place model, o4-mini high (35.00), is also closed-source. The best open-source model, GLM-4.5V (30.45), ranks sixth — behind five closed-source models. The average closed-source score across the 8 closed-source models is approximately 35.12% (computed from Table 4), while the average open-source score across 8 open-source models is approximately 17.34% — roughly half.

However, the open-source category includes models at smaller scales (7B, 8B, 9B, 32B) that are not comparable to the proprietary models at unknown but presumably larger scales. The only open-source model at a competitive scale is Qwen2.5-VL-72B-Instruct, which scores 23.59% — substantially below the best closed-source model (42.66%). This does suggest a genuine capability gap at the frontier, but the paper does not control for parameter count in the comparison. If the best closed-source models have, say, 200B+ parameters while the best open-source has 72B, then part of the "open-source vs. closed-source" gap is actually a "large vs. smaller" gap. A FLOPs-matched comparison (analogous to Section 7 of the reference example paper) would be needed to distinguish these explanations.

Claim 4: "CoT reasoning typically follows a three-stage layered process — extraction, reasoning, and verification"

Supported by qualitative evidence, but not quantitatively validated. Figure 3 provides a detailed annotation of one model's CoT on one task (Doubao on Satellite Image Matching), showing clear Stage 1, Stage 2, and Stage 3 structure. The paper states that "although task-specific tactics vary, the overall structure remains stable," implying that this pattern generalizes across tasks. However, only one example is presented in the main text, and the appendix error cases (Figures 5-14) are curated to illustrate failures, not to test the generality of the three-stage structure.

To validate this claim, the paper would need to: (1) define operational criteria for identifying each stage in an automated or reproducible way, (2) annotate CoT outputs from multiple models across multiple tasks, and (3) report inter-annotator agreement and prevalence rates (e.g., "Stage 1 → Stage 2 → Stage 3 structure is present in 73% of correct responses vs. 41% of incorrect responses"). Without such quantification, the three-stage model is a compelling qualitative observation rather than a validated finding. It may hold for some models, some tasks, and some prompting styles, but its generality is untested.

Claim 5: "Visual extraction is continuous throughout the reasoning process"

Plausible, but the evidence is anecdotal. Figure 3 shows Doubao revisiting visual details at multiple points in its CoT, which is consistent with continuous extraction. However, the paper does not provide metrics such as: frequency of image-reference events per reasoning chain segment, correlation between late-stage visual extraction and correctness, or comparison to models that are forced to extract all visual information upfront (e.g., via a structured "first describe everything, then reason" prompt that prevents mid-chain visual revisiting). The ablation in Table 5, where "first describe everything" yields small gains, is consistent with continuous extraction being important, but it does not directly test whether continuous extraction actually occurs (it tests whether forcing early extraction helps).

Claim 6: "Common error patterns include orientation mistakes, fragile cross-view identity persistence, and poor adherence to counterfactual instructions"

Supported by concrete case studies but not by prevalence statistics. Appendices D and E present 14 annotated error cases across multiple tasks and error types. These cases clearly illustrate the claimed patterns. However, the paper does not report: what fraction of total errors fall into each category? Are some models disproportionately affected by certain error types? Are the error categories exhaustive (do they cover most observed failures) or illustrative (selected examples from a larger and more heterogeneous pool)? Without such statistics, the error analysis is diagnostic in principle but not in practice — a developer reading the paper knows that these error types exist but does not know whether fixing them would address 10% of failures or 80%.

Overall Assessment of the Experimental Design

The experiments effectively demonstrate the paper's central descriptive claims: (1) MME-CC is a challenging benchmark where frontier models score far below human performance, (2) spatial and geometric reasoning are weak points across all current MLLMs, (3) there are large performance asymmetries across the three reasoning dimensions, and (4) specific error patterns recur across models and tasks. The 16-model evaluation provides broad coverage and enables cross-family comparisons that would be impossible with a smaller evaluation suite.

The primary limitations are: (a) Single-pass evaluation with no variance estimates, making it impossible to distinguish real performance differences from sampling noise — the 0.17-point gap between GPT-5 (40.25) and Doubao Think (40.08) is almost certainly within noise. (b) No multi-sample experiments that would reveal whether failures are due to inability or instability — critical for understanding the nature of the errors. (c) Qualitative claims about CoT structure and error prevalence that are supported by well-chosen examples but not by quantitative annotation studies. (d) The confound of using Gemini-2.5-Pro for benchmark filtering — the model that was used to select the hardest questions is also the model that scores highest on those questions, which could inflate its apparent advantage if the filtering inadvertently selected for questions that are differentially easier for Gemini. (e) No control for model scale in the open-source vs. closed-source comparison — the largest open-source model evaluated (72B) is almost certainly smaller than proprietary models, so the "closed-source advantage" may partially reflect scale rather than proprietary training techniques.

The experiments that would have most strengthened the paper are: a systematic, quantitative error categorization study with prevalence rates; a pass@k evaluation to separate capability from consistency; a control experiment demonstrating that performance degrades on prior benchmarks when images are removed (validating the paper's core critique of existing benchmarks); and a FLOPs-matched or parameter-matched comparison between open-source and closed-source models to disentangle scale effects from training-quality effects.

6. Limitations and Trade-offs

Limitation 1: Vision-Based Solving Cannot Be Guaranteed — Only Designed For

The paper's foundational claim is that MME-CC forces vision-based reasoning by embedding answer-critical information in images rather than text. This is a design intent, not a verified property. The paper defines "Vision-based" in Table 3 to mean "all task information is derived from images rather than text," which is an operational claim about task construction — but whether a given model actually solves the task using visual information is an empirical question that the paper does not test. The observation that certain tasks within categories show enormous inter-model variation (Doubao Think scores 70.6% on Jigsaw Puzzle while o4-mini scores 27.0%, Table 5) raises the possibility that some models are using non-visual strategies — if visual reasoning were the only path, the performance spread on identical visual inputs should be narrower.

Consequence. If a model achieves a high score on MME-CC, the paper cannot distinguish between "the model has strong visual reasoning" and "the model has found a non-visual shortcut that the benchmark designers did not anticipate." The confidence that MME-CC scores measure visual cognitive capacity — as opposed to some mixture of visual and non-visual capabilities — rests entirely on the thoroughness of the design process, not on empirical validation. Shortcuts can be subtle: for example, the Maze task (Table 5) might be solvable through recognizing specific digit patterns from training data rather than constructing the shortest path, and no test in the paper rules this out. Without adversarial validation — deliberately testing whether models can solve tasks when visual information is degraded, scrambled, or removed — the paper cannot claim with certainty that its benchmark isolates visual reasoning.

Evidence in the paper. The paper provides no experiment where images are removed, scrambled, or replaced with non-informative substitutes to test whether model performance drops to chance. The 95% human agreement on the LLM judge (Section 3.1) validates scoring, not the vision-based property. The ablation in Table 5, where forced textual description yields small gains, is consistent with visual information being important, but does not prove that visual information is necessary for the solutions models actually produce. The paper cites prior work on shortcut exploitation (NaturalBench, EasyARC, VLSBench) as motivation, but does not itself run shortcut-detection experiments on MME-CC.

Mitigation status. Not addressed. The design process — stripping UI overlays and text from images, using free-form output, embedding answer-critical information in images — reduces the surface area for shortcuts, but cannot eliminate them. The paper acknowledges this implicitly: the model-based filtering stage (Section 2.3) removed "trivially easy" items, but defines "easy" relative to Gemini-2.5-Pro's performance, not relative to shortcut availability. A model different from Gemini might find shortcuts that Gemini did not. The paper does not suggest future work on adversarial validation or shortcut auditing.

Limitation 2: The Gemini-Based Filtering Introduces a Systematic Bias Toward the Best Model

The paper reports that Gemini-2.5-Pro was used to filter the initial data pool, removing approximately 50% of candidates that were "trivially easy (defined as achieving a model accuracy score above 95%), semantically redundant, or lacking discriminative value" (Section 2.3). This means that the benchmark was curated to be difficult specifically for Gemini-2.5-Pro — questions that Gemini found easy were removed, while questions that Gemini found challenging were retained. The same model (or a closely related checkpoint) was then the best-performing model in the evaluation (Table 4, 42.66% overall).

Consequence. The benchmark's difficulty distribution is not model-independent. Questions that are hard for Gemini-2.5-Pro are not necessarily hard for other models in the same way — a question that exploits a specific weakness of Gemini's spatial processing might be retained in the benchmark, while a question that exploits a different model's weakness but happens to be easy for Gemini would be filtered out. This creates a confound where the models used for filtering appear to perform better on the final benchmark than they would if the filtering had been done with a different model or with a model-agnostic difficulty criterion. The consequence is that relative rankings and absolute scores may be biased in favor of the filtering model. A model that is genuinely stronger than Gemini on some visual reasoning dimension might score lower than Gemini on MME-CC because questions that would differentiate in its favor were removed during filtering.

The paper does not discuss this confound, does not report whether the filtering used the exact same checkpoint that was later evaluated, and does not provide any analysis of whether the 50% of removed questions would have changed relative model rankings if included. This is a serious methodological concern because it undermines the benchmark's claim to be a fair, model-agnostic evaluation instrument.

Evidence in the paper. Section 2.3 states the filtering procedure. Table 4 shows Gemini-2.5-Pro as the highest-scoring model. No ablation or analysis examines the impact of the filtering choice on model rankings or score distributions.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation, does not propose alternative filtering strategies (e.g., using an ensemble of diverse models, or using human difficulty ratings rather than model performance), and does not suggest future work on de-confounding the benchmark from the filtering model. This is a fixable problem — re-running the filtering with a different model or releasing the unfiltered question pool — but it requires additional work not present in the paper.

Limitation 3: Single Response Per Question Prevents Distinguishing Capability from Consistency

The evaluation protocol uses a single response per model per question (Section 3.1, pass@1). For open-source models, decoding uses temperature 1.0 and top-p 0.7 — a stochastic setting where the same model can produce different answers on different runs. For proprietary models accessed via API, the decoding settings are whatever the provider defaults to (which may include some stochasticity). This means that the reported scores conflate whether a model can solve a problem with whether it reliably solves it.

Consequence. Two models with identical "capability" (both can produce the correct answer on some fraction of attempts) can have different scores in Table 4 if one is more consistent in generating that correct answer. This is particularly problematic for comparing reasoning models (which generate long CoT chains and may be high-variance) against non-reasoning models (which generate shorter, more deterministic outputs), and for comparing small open-source models (which may be less well-calibrated) against large proprietary models (which may have lower output variance). The paper's claim that "reasoning-oriented models exhibit advantages over non-reasoning models" (Section 3.2) could partially reflect variance differences rather than capability differences — a reasoning model that sometimes succeeds brilliantly and sometimes fails spectacularly might have a lower pass@1 than a more consistent non-reasoning model, even if the reasoning model's "best case" performance is superior.

This limitation also affects the interpretation of per-subtask scores. Doubao Think's 70.6% on Jigsaw Puzzle (Table 5) could mean "Doubao reliably solves 70.6% of Jigsaw puzzles on the first attempt," or it could mean "Doubao can solve nearly all Jigsaw puzzles but only does so on 70.6% of attempts due to decoding variance." These interpretations have very different implications for how the model would perform in deployment (where a user might query it multiple times, or where majority voting could be applied).

Evidence in the paper. The evaluation protocol in Section 3.1 specifies temperature 1.0 and top-p 0.7 for open-source models, indicating stochastic decoding. No pass@k results (where k > 1 samples are generated per question and correctness is assessed by whether any sample is correct) are reported. No analysis of output variance or consistency is provided.

Mitigation status. Not addressed. The paper does not discuss the pass@1 vs. pass@k distinction, does not report variance estimates, and does not suggest that multi-sample evaluation would change the conclusions. This is a standard methodological choice in benchmark papers (pass@1 is common), and the paper's overall conclusions — that models are weak on spatial/geometric reasoning — are unlikely to be overturned by pass@k evaluation (models scoring 5–30% on pass@1 are probably not secretly capable of 80% at pass@10). However, the specific rankings and the reasoning-vs-non-reasoning comparison could be sensitive to this choice, and the paper's silence on the issue means readers cannot assess how much.

Limitation 4: Error Prevalence Is Not Quantified — Diagnosis Without Epidemiology

The paper's error analysis (Section 4.3, Appendices D and E) identifies three recurring error patterns — orientation/reference-frame confusion, cross-view identity persistence failure, and over-reliance on literal descriptions — and supports each with annotated case studies. However, no prevalence statistics are reported. The paper does not state what fraction of total errors across all models fall into each category, whether some models are disproportionately affected by certain error types, or whether the three identified categories cover most observed failures or are a small, illustrative subset of a larger and more heterogeneous error landscape.

Consequence. A model developer reading the paper cannot prioritize which error types to address first — they do not know whether fixing orientation/reference-frame alignment would recover 5% of lost points or 40%. A researcher designing follow-up experiments cannot calibrate their hypotheses: if cross-view identity persistence failures account for only 3% of total errors, then a paper claiming to "address the key failure mode identified in MME-CC" by improving object tracking is solving a minor problem. The error analysis is diagnostic in ambition but illustrative in execution — it shows that certain failures exist and provides compelling examples, but does not provide the epidemiological data needed to assess their quantitative importance.

This limitation is particularly significant because the paper positions error diagnosis as a first-class output of the benchmark. The abstract states that MME-CC "provides fine-grained analyses of MLLMs' cognitive capacity," and Section 4.3 is framed as revealing "recurring errors that appear across tasks and reasoning dimensions." The claim of recurrence is supported by cross-task examples, but recurrence does not imply prevalence — a failure mode can recur across many tasks while affecting only a small fraction of questions on each task.

Evidence in the paper. Appendices D and E present 14 annotated error cases from Doubao-Seed-1.6-vision-0815. No table reports error category frequencies, per-model error profiles, or error category coverage (what fraction of total incorrect answers fall into identified vs. unidentified categories). The paper provides qualitative richness (Figures 5–14) but no quantitative error taxonomy.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation, does not propose future work on systematic error annotation, and does not suggest that quantitative error analysis would strengthen the benchmark's diagnostic value. This omission limits the benchmark's actionability: knowing how models fail is useful, but knowing how often each failure mode occurs is essential for prioritization.

Limitation 5: The Maze Task Is Effectively Unsolvable — A Ceiling at Near-Zero That Provides No Discrimination

The Maze subtask (Table 5) reveals a structural problem in the benchmark's difficulty calibration: no model exceeds approximately 2% accuracy in the base setting, with scores of 1.1% (Gemini-2.5-Pro), 0.8% (Doubao Think), and 1.2% (o4-mini). The ablation with forced textual description improves these only marginally (1.9%, 0.6%, 1.5% respectively). This means that for 194 of the 1,173 questions (16.5% of the benchmark), all models perform at chance or near-chance levels, and the task provides essentially no discriminative power — it cannot distinguish between strong and weak models because no model is strong enough to rise above the floor.

Consequence. The benchmark contains a substantial fraction of questions that contribute noise rather than signal to the aggregate Geometric Reasoning score and the Overall score. Because Maze questions are essentially coin flips for all models (with a heavily biased coin toward "wrong"), differences between models on these questions reflect sampling variance rather than capability differences. This inflates the apparent difficulty of the benchmark — the Overall scores in Table 4 are pulled down by a task that may be too difficult to be useful — and it means that progress in Geometric Reasoning might not register on MME-CC until models cross a capability threshold that is far above the current frontier. If a future model improves substantially on Gomoku Variation, Unblock Me, and Jigsaw Puzzle but still cannot solve Maze, its Geometric Reasoning score will barely budge because Maze constitutes 32% of the Geometric questions (194/605).

The Maze results also complicate the interpretation of the "spatial and geometric reasoning remain broadly weak (≤30%)" claim. If Maze were removed, the Geometric Reasoning average would be higher (since Maze pulls the average down toward zero), potentially altering the category-level narrative. The paper's category-level reporting masks this task-level heterogeneity — a reader sees "29.56% on Geometric Reasoning" for Gemini-2.5-Pro and may not realize that this average includes a 1.1% score on the single largest subtask in the category.

Evidence in the paper. Table 5 reports Maze scores. Table 2 shows Maze as the largest Geometric subtask (194 samples, 32% of Geometric questions). The paper notes in Section 4.2 that "in Maze, which requires continued rule-based simulation and path planning, no model exceeds 2%," but does not discuss whether this level of difficulty is diagnostically useful or whether it constitutes a measurement floor.

Mitigation status. Not addressed. The paper does not discuss whether Maze should be weighted differently in aggregate scores, whether it should be considered a separate "extreme difficulty" tier rather than part of the Geometric average, or whether future versions of MME-CC should recalibrate Maze difficulty (e.g., by using smaller grids, providing partial credit for partial paths, or splitting the task into subproblems). The model-based filtering stage (Section 2.3) removed questions that were "trivially easy" (>95% accuracy), but there was apparently no symmetric filter for "trivially hard" questions that no model can solve. The inclusion of Maze is a design choice that reflects the paper's aspiration to push the difficulty frontier, but it comes at the cost of reduced measurement resolution in the Geometric category.

Limitation 6: The CoT Behavioral Analysis Is Based on a Single Model and Does Not Generalize to Non-Reasoning Systems

The detailed Chain-of-Thought analysis in Section 4.2 — the three-stage reasoning model (extraction → reasoning → verification), the observation of continuous visual extraction, and the finding of excessive verification pauses — is based entirely on one model (Doubao-Seed-1.6-vision-0815 in Think mode) and is illustrated with one task (Satellite Image Matching, Figure 3). The paper states that "although task-specific tactics vary, the overall structure remains stable," but provides no evidence from other tasks, other models, or other model families to support this generalization claim.

Consequence. The CoT findings may be specific to Doubao's reasoning architecture and training recipe, rather than general properties of how MLLMs perform visual reasoning. Different reasoning models (GPT-5, o4-mini, Gemini-2.5-Pro with its own thinking mode) may exhibit different CoT structures — some might extract all visual information upfront, others might do purely symbolic reasoning after an initial visual encoding, and still others might show stage transitions that differ from the three-stage model. Even within Doubao, the CoT structure may differ substantially across task types — the behavior on Satellite Image Matching (a spatial correspondence problem) may not generalize to Maze (a path-planning problem) or Counterfactual Instruction (a rule-following problem).

The finding of "excessive verification" and "stalling" is particularly suspect as a general claim. The repeated "wait" pauses visible in Figure 3 might be an artifact of Doubao's specific training — other reasoning models might be trained to avoid such self-interruption, or might employ different verification strategies (e.g., structured self-checks rather than ad-hoc pauses). Without analyzing CoT outputs from multiple models, the paper cannot claim that excessive verification is a general property of visual reasoning in MLLMs rather than a quirk of one model's generation style.

Consequence for the paper's claims. The CoT analysis is presented as one of the paper's key contributions (listed in the abstract and detailed in Section 4.2). If the findings do not generalize, the paper's contribution reduces from "we discovered how MLLMs reason about visual tasks" to "we observed how one specific MLLM reasons about one specific visual task on one specific example." This is still a valid observation, but it does not support the broad claims the paper makes about "MLLMs" as a class.

Evidence in the paper. Section 4.2 and Figure 3 analyze Doubao-Seed-1.6-vision-0815 on Satellite Image Matching. No CoT outputs from other models are shown or analyzed. The paper does not claim to have analyzed other models' CoT — the scope is implicitly limited to the one model — but the language used ("We analyze the chain-of-thought behavior... and summarize the following observations") does not clearly mark this as a single-model case study.

Mitigation status. Not addressed as a limitation. The findings are presented as general properties of MLLM CoT behavior without a caveat about generalizability. The paper does not suggest future work on cross-model CoT comparison or propose that the three-stage model should be validated on other reasoning architectures. This limitation is particularly consequential because reasoning-oriented MLLMs are a diverse and rapidly evolving category — training techniques, reasoning token budgets, and CoT structures vary enormously across model families, and findings from one model in mid-2025 may not apply to models released even a few months later.

7. Implications and Future Directions

How This Work Changes the Landscape

MME-CC does not propose a new model architecture or training technique — it is a benchmark paper. Its impact on the field will therefore be methodological and diagnostic, not architectural. The paper’s most consequential contribution is the systematic demonstration that the field’s existing multimodal evaluation infrastructure is structurally incapable of measuring visual cognitive capacity, and that fixing this requires a design philosophy — “vision-based” evaluation where answer-critical information is embedded exclusively in images — that the most-cited benchmarks (MME, MMMU, MMBench, MMStar) do not implement. This is not merely a new dataset; it is a validity challenge to the conclusions drawn from prior benchmark results.

The magnitude of this shift should be understood as a reframing rather than a paradigm shift. The field already suspected that multimodal benchmarks had textual shortcut problems — NaturalBench, EasyARC, and VLSBench (cited in Section 5) had documented models exploiting answer-bearing text. What MME-CC adds is a constructive alternative: a benchmark built from first principles to close those shortcuts, with a replicable design methodology (free-form output, stripped images, forced visual information, model-based difficulty filtering) that future benchmarks can adopt. The paper converts a known critique into a positive research program. If the community accepts the premise that “vision-based” benchmarks are necessary for evaluating visual reasoning — a premise the paper defends through the stark performance asymmetries in Table 4, where Visual Knowledge Reasoning scores (up to 74.63%) far exceed Spatial and Geometric Reasoning scores (both ≤30%) — then a significant fraction of published multimodal evaluations must be reinterpreted as measuring a mixture of visual and textual capability, not pure visual reasoning. This reframing shifts the burden of proof: benchmark authors claiming to evaluate visual reasoning must now demonstrate that their tasks cannot be solved through non-visual means, rather than assuming that including images is sufficient.

The paper also resolves — or at least recontextualizes — a latent contradiction in prior multimodal evaluation work. Previous studies had reported both strong performance on multimodal benchmarks (state-of-the-art MLLMs “often demonstrate strong performance” on MME, MMBench, MMMU, as noted in Section 1) and evidence of shortcut exploitation (NaturalBench, EasyARC). The contradiction — how can models be both “strong” and “exploiting shortcuts”? — is resolved by distinguishing benchmark types. MME-CC shows that when visual reasoning is forced, performance collapses: Gemini-2.5-Pro scores 74.63% on Visual Knowledge Reasoning but 23.80% on Spatial Reasoning (Table 4). The prior “strong performance” was not false, but it was measuring the wrong thing — or rather, it was measuring an uninterpretable mixture of visual and non-visual capabilities. MME-CC disentangles them, and the result is that on genuinely vision-dependent tasks, current MLLMs are far weaker than aggregate benchmark scores suggest.

This reframing makes certain research directions more attractive and others less so. More attractive: (1) Architectural research on vision-language models that specifically targets spatial and geometric reasoning — given the ≤30% ceiling on these dimensions, incremental scaling of existing architectures is unlikely to close the gap to human performance (95.92%), and qualitatively different representational mechanisms (e.g., explicit 3D geometric embeddings, iterative visual re-inspection modules, viewpoint-invariant object representations) become high-priority targets. (2) Verifier and self-consistency methods for visual reasoning — the paper’s CoT analysis (Section 4.2) suggests that models can sometimes produce correct reasoning chains but fail to converge on the right answer, implying that verification mechanisms could rescue performance even without improving the base model’s visual processing. (3) Benchmark methodology research that develops adversarial validation protocols — given the paper’s acknowledgment that it cannot guarantee its benchmark is shortcut-free (merely that it was designed to minimize shortcuts), systematic methods for auditing visual benchmarks become essential. Less attractive: (1) Claiming general visual reasoning improvements based solely on non-vision-based benchmarks — MME-CC provides a concrete counter-example showing that improvements on MMMU or MMBench may not transfer to forced-visual tasks. (2) Scaling-alone solutions to spatial/geometric reasoning — the Qwen2.5-VL scaling trend (7B → 32B → 72B, Table 4) shows diminishing returns on Geometric Reasoning (3.22% → 8.56% → 8.96%) and modest gains on Spatial Reasoning (4.70% → 9.03% → 12.47%), suggesting that even 400B+ parameter models would fall well short of human-level performance on these dimensions if the scaling trends hold.

The paper’s identification of specific, recurring error patterns — orientation/reference-frame confusion, cross-view identity persistence failure, and instruction-following breakdowns — provides a diagnostic vocabulary that was previously absent from multimodal evaluation. Rather than saying “models are bad at spatial reasoning,” researchers can now say “models fail at cross-view identity persistence, as measured by the Indoor Deduplication Counting subscore on MME-CC,” and target that specific failure mode. This is a shift from aggregate capability reporting to targeted cognitive deficit identification, analogous to how neuropsychological testing moved from IQ scores to domain-specific assessments. The error taxonomy, while qualitative in the current paper (see Limitation 6 in the prior section), provides a template that future work can quantify and extend.

A subtler shift concerns the role of Chain-of-Thought in visual reasoning. The paper’s observation that CoT follows a three-stage structure (extraction → reasoning → verification) with continuous visual extraction throughout — and that excessive verification in long chains may actually degrade performance — challenges the implicit assumption that “more thinking = better results.” If visual information degrades over long reasoning chains (as suggested by the catastrophic Maze performance, where sustained spatial simulation is required but no model exceeds ~2%), then the optimal CoT length for visual tasks may be substantially shorter than for text-only reasoning, and architectures that allow mid-chain visual re-grounding may outperform those that encode images once and reason symbolically thereafter. This is a hypothesis the paper does not test but that its findings directly motivate.

Follow-Up Research This Work Enables

Adversarial validation of MME-CC to quantify residual shortcut susceptibility. The paper designs MME-CC to minimize textual shortcuts but does not empirically verify that models cannot exploit non-visual cues. A natural follow-up is a controlled experiment where the same questions are presented (a) with the original images, (b) with images replaced by random noise or semantically unrelated images, and (c) with images degraded (blurred, downsampled) but still structurally informative. If model performance on condition (b) is above chance, then non-visual cues remain exploitable — perhaps through linguistic priors in the prompt or answer-format regularities. If performance on condition (c) degrades gracefully with image quality, this provides evidence that models are genuinely using visual information. This experiment would quantify the degree to which MME-CC scores actually reflect visual processing, moving the “vision-based” property from a design claim to an empirically validated metric. A strong study would report per-model shortcut susceptibility scores and correlate them with architectural features (e.g., vision encoder resolution, number of visual tokens) to identify which design choices minimize shortcut reliance.

Quantitative error taxonomy with prevalence rates and per-model error profiles. The paper identifies three recurring error patterns qualitatively but does not report how often each occurs. A direct extension would be a systematic annotation study: sample 200–300 incorrect model outputs from each of 3–4 representative models (e.g., Gemini-2.5-Pro, GPT-5, Doubao Think, Qwen2.5-VL-72B) across all 11 subtasks, have 2–3 annotators categorize each error into the paper’s taxonomy (orientation/reference-frame confusion, cross-view identity persistence failure, instruction-following breakdown, or “other”), compute inter-annotator agreement, and report per-model error profiles. The key questions: (a) Do different models fail in qualitatively different ways, or are the error patterns universal? (b) Does “orientation confusion” account for 15% or 60% of spatial reasoning errors? (c) Are instruction-following failures concentrated in Sandbagging/Counterfactual tasks, or do they also appear in other task types? The answers would tell model developers which failure modes to prioritize and whether different architectures (e.g., native 3D-aware vision encoders vs. 2D-only encoders) produce systematically different error distributions.

Pass@k evaluation to dissociate capability from output variance. The paper reports pass@1 scores (single response per question), which conflates whether a model can solve a problem with whether it reliably does so on the first attempt. A follow-up study would generate k = 4, 8, or 16 responses per question for a subset of models (particularly reasoning vs. non-reasoning pairs like Doubao Think vs. Nonthink, GPT-5 vs. GPT-4.1) and report pass@k curves. The key hypothesis to test: reasoning models may have higher variance in their outputs (long CoT chains can diverge), so their pass@1 scores might understate their capability relative to non-reasoning models. If Doubao Think’s pass@8 is substantially higher than its pass@1 (38.9% overall, Table 4), while Doubao Nonthink’s pass@8 is close to its pass@1 (25.96%), this would imply that the reasoning advantage is partly masked by output instability — and that simple majority voting over multiple samples could close much of the remaining gap to human performance. Conversely, if pass@k curves are flat for all models on Maze (suggesting fundamental incapacity rather than instability), this strengthens the paper’s claim that some tasks require qualitatively different capabilities that current architectures lack.

Cross-model Chain-of-Thought structural comparison to test the generality of the three-stage model. The paper’s CoT analysis (Section 4.2) is based entirely on Doubao-Seed-1.6-vision-0815. A natural extension is to collect CoT outputs from all reasoning-enabled models in Table 4 (Gemini-2.5-Pro, GPT-5, o4-mini, Doubao Think, GLM-4.5V, GLM-4.1V-9B-Thinking) on a standardized subset of MME-CC questions (e.g., 20 questions spanning all three dimensions), and analyze: (a) whether all models exhibit the three-stage structure, (b) whether visual extraction is continuous across all models and task types or specific to certain architectures, (c) whether the “excessive verification” pattern (repeated “wait” pauses) appears in non-Doubao models, and (d) whether chain length correlates with accuracy differently for different models. The findings would distinguish universal properties of visual CoT from model-specific training artifacts, and would inform whether the three-stage model is a useful descriptive framework for the field or merely a characterization of one training approach. A strong negative result — e.g., GPT-5 shows a completely different CoT structure, with visual extraction concentrated in the first 20% of the chain — would be highly informative for understanding how different training recipes produce different reasoning behaviors.

Training interventions that specifically target MME-CC failure modes and measure transfer. The paper’s diagnostic error categories are designed to be actionable — they point to specific cognitive deficits that, if addressed, should improve performance. A follow-up study would take a strong open-source base model (e.g., Qwen2.5-VL-72B-Instruct, which scores 23.59% overall) and apply targeted fine-tuning interventions: (a) multi-view consistency training using synthetic data where the same object appears in different viewpoints with identity labels, targeting the cross-view identity persistence failure; (b) spatial relation prediction training using 3D-rendered scenes with ground-truth camera poses, targeting the orientation/reference-frame failure; (c) instruction-following training with counterfactual and constrained-output scenarios, targeting the instruction adherence failure. The key evaluation: does fine-tuning on these targeted tasks improve performance on the corresponding MME-CC subtasks, and does it transfer to other subtasks within the same cognitive dimension? If (a) improves Indoor Deduplication Counting but not Indoor Directional Reasoning, that suggests the error categories are more fine-grained than the paper’s taxonomy implies. If (a) improves both, it validates the paper’s claim that cross-view identity persistence is a coherent cognitive deficit underlying multiple tasks. The study would simultaneously test the paper’s diagnostic framework and provide concrete training recipes for improving visual reasoning.

FLOPs-matched or parameter-matched comparison between open-source and closed-source models to disentangle scale effects from training-quality effects. The paper reports that Gemini-2.5-Pro (closed-source, 42.66%) substantially outperforms GLM-4.5V (open-source, 30.45%), but the closed-source models are almost certainly larger. A controlled comparison would evaluate the largest available open-source model (Qwen2.5-VL-72B-Instruct, 23.59%) against closed-source models at approximately the same parameter scale, or would use inference FLOPs as a matching criterion. If closed-source models still substantially outperform open-source at matched scale, this suggests that proprietary training techniques (data, recipes, post-training) confer genuine advantages for visual reasoning. If the gap closes at matched scale, it suggests that the apparent open-source deficit is primarily a resource gap rather than a technique gap. Either outcome would inform community strategy: invest in scaling open-source models vs. invest in discovering better training recipes. The MME-CC benchmark is well-suited for this comparison because its per-dimension breakdown can reveal whether any closed-source advantage is uniform (all dimensions) or concentrated in specific dimensions (e.g., perhaps closed-source models have better Visual Knowledge Reasoning due to proprietary knowledge-intensive training data, but perform similarly on Spatial Reasoning where knowledge is less relevant).

Practical Applications and Downstream Use Cases

Risk assessment for MLLM deployment in spatially-grounded applications. The paper’s finding that spatial reasoning peaks at 30.63% (GPT-5 high, Table 4) provides a concrete warning for practitioners considering MLLM deployment in domains where spatial understanding is safety-critical. Specifically: autonomous navigation interfaces that describe environments to users, augmented reality assistants that must reason about 3D object placement, architectural or interior design tools that interpret spatial layouts, and accessibility applications that describe environments to visually impaired users. In all these settings, the ≤30% ceiling on MME-CC spatial tasks means that even the best current MLLMs will fail at basic spatial reasoning (matching viewpoints, maintaining object identity across views, propagating orientation constraints) on roughly 70% of non-trivial queries. The benchmark provides a concrete risk quantification: if your application requires the model to correctly determine object orientation from spatial context (analogous to Indoor Directional Reasoning, where Gemini-2.5-Pro scores 14.3%, Table 5), expect failure rates above 85% even with the strongest available model. This is actionable for deployment decisions — it provides evidence that current MLLMs should not be the sole reasoning engine for spatially-grounded applications and should be supplemented with explicit geometric computation (e.g., SLAM-based localization, 3D reconstruction) when spatial accuracy matters.

Diagnostic profiling for model selection in multi-model pipelines. Organizations building multi-model systems (e.g., a router that sends different queries to different specialized models) can use MME-CC’s per-dimension and per-subtask breakdowns to make granular model selection decisions that aggregate overall scores would obscure. For example, Table 4 and Table 5 reveal that Doubao Think scores 70.6% on Jigsaw Puzzle while Gemini-2.5-Pro scores 30.4% — a 40-point gap. Conversely, Gemini-2.5-Pro scores 34.8% on Gomoku Variation while Doubao Think scores 14.9% — a 20-point gap in the opposite direction. A system that routes Jigsaw Puzzle queries to Doubao and Gomoku queries to Gemini would substantially outperform either model alone on these specific task types. The benchmark enables this kind of fine-grained capability profiling in a way that single-number aggregate benchmarks (e.g., “Model X scores 67.3% on MMMU”) cannot. The practical benefit is direct: for any application with a known distribution of visual reasoning subtasks, MME-CC scores can inform which model to deploy for which query type, potentially recovering 10–40 percentage points of accuracy through routing compared to using any single model.

Filtering and data quality control for visual instruction-tuning datasets. The paper’s data construction pipeline — particularly the model-based filtering stage where Gemini-2.5-Pro was used to remove trivially easy or redundant items — provides a template for curating high-quality visual reasoning training data. Practitioners building visual instruction-tuning datasets can use MME-CC itself (or a model’s performance profile on MME-CC) as a quality filter: include training examples that are analogous to MME-CC task types where target performance is low, and exclude examples that MLLMs already solve at high accuracy. For instance, if a team is building a dataset to improve spatial reasoning, they should prioritize examples that resemble Indoor Directional Reasoning and Indoor Deduplication Counting (where even the best models score ~5–29%, Table 5) rather than Satellite Image Matching (where scores reach ~40%). The benchmark thus serves a dual purpose: evaluation and training data prioritization. This is a practical application of the paper’s core insight that aggregate scores mask task-level heterogeneity: by using MME-CC as a diagnostic instrument rather than just a leaderboard, practitioners can identify exactly which capabilities their training data should target, rather than collecting “more visual data” generically.