A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise

ArXiv: 2312.12436

🎯 Pitch

Despite vastly different answering styles—GPT-4V gives verbose reasoning while Gemini delivers terse conclusions—the two models exhibit nearly identical visual reasoning power, with Gemini edging out GPT-4V on the MME benchmark. Yet even these top models still stumble on basic spatial perception and logic, revealing how far we remain from truly general visual intelligence.

---

1. Executive Summary

This paper conducts a preliminary exploration of Gemini Pro's visual understanding proficiency across four domains—fundamental perception, advanced cognition, challenging vision tasks, and various expert capacities—comparing it against GPT-4V and the open-source Sphinx model using a suite of curated qualitative samples and the MME benchmark. The study identifies distinctive answering styles between the two leading models: GPT-4V tends to elaborate detailed explanations and intermediate steps, while Gemini prefers direct and concise answers, with both exhibiting comparable visual reasoning capabilities across most tasks. On the MME benchmark, Gemini achieves the highest overall score of 1933.4, narrowly outperforming GPT-4V's 1926.6 and Sphinx's 1870.2, establishing that Gemini is a strong challenger to GPT-4V's leading position—though all models continue to struggle with spatial perception, logical self-consistency, and prompting robustness, revealing that a considerable distance toward artificial general intelligence remains.

2. Context and Motivation

The Core Gap: A Rigorous, Multi-Domain Evaluation of the Latest Frontier MLLMs

The paper addresses a specific, time-sensitive problem in the multi-modal learning landscape: when a new, potentially category-leading model is released with limited public technical detail, the research community lacks a systematic, third-party assessment of its capabilities relative to the established state-of-the-art. The release of Gemini by Google in late 2023 created precisely this situation. While Google's own technical report claimed strong multi-modal performance across several benchmarks, independent verification of these claims—especially through fine-grained, qualitative analysis that reveals not just whether a model succeeds but how it reasons, where it fails, and what behavioral patterns distinguish it from alternatives—was entirely absent.

This is not merely a matter of benchmarking hygiene. The paper implicitly argues that single-number benchmark scores, while useful, obscure critical qualitative differences between models that matter enormously for downstream use. Two models might achieve similar accuracy on a task yet exhibit radically different failure modes, reasoning styles, and reliability characteristics. Without careful qualitative comparison, the community cannot make informed decisions about which model to use for which application, nor can it identify the specific architectural or training deficiencies that need addressing.

Why This Matters: Practical, Scientific, and Methodological Stakes

Practical impact: Deployment decisions in a rapidly consolidating market. By late 2023, MLLMs had transitioned from research curiosities to production-deployable systems powering applications in healthcare, autonomous driving, surveillance, education, and creative tools. Developers and organizations choosing between GPT-4V and Gemini for integration into their products need to understand not just aggregate performance differences but domain-specific strengths and weaknesses. Can Gemini reliably read medical images? Does it recognize traffic signs with sufficient accuracy for autonomous driving assistance? Does it hallucinate more or less than GPT-4V on open-ended scene descriptions? The paper's comprehensive domain coverage—from fundamental perception through specialized expert capacities—directly serves this decision-making need.

Scientific significance: Characterizing the capabilities ceiling of current architectures. The paper's comparison between GPT-4V (proprietary), Gemini Pro (proprietary), and Sphinx (open-source, built on LLaMA-2-7B) also serves a deeper scientific purpose: it reveals how much performance can be extracted from open-source foundations versus what requires the resources of large industrial labs. When Sphinx fails at tasks where the closed-source models succeed—particularly in cognition-heavy domains like code reasoning, abstract visual understanding, and scientific problem-solving—the paper identifies the capability gap that the open-source community must close. Conversely, when Sphinx performs competitively (e.g., on perception tasks from academic datasets), it suggests that at least some aspects of multi-modal understanding are accessible without massive proprietary infrastructure.

Methodological significance: Establishing a human-centric evaluation paradigm. Perhaps the paper's most enduring contribution is its evaluation methodology. Rather than relying solely on aggregate benchmarks, the authors construct a deliberately diverse, multi-domain test suite where each sample is hand-selected to probe a specific capability boundary. They vary difficulty levels within each task, collect images with timestamps postdating November 2023 to avoid train-test leakage, and construct text queries from scratch. This labor-intensive approach yields insights that automated benchmarks miss: that both Gemini and GPT-4V fabricate details when pushed beyond their reliable operating range (the five-differences prompt in Figure 5), that they exhibit stylistic differences in answer presentation (concise vs. verbose), and that they share surprising failure modes (spatial relation confusion in Figure 1).

Prior Approaches and Where They Fall Short

The paper situates itself against a landscape of existing MLLM evaluation approaches, each with identifiable limitations.

Proprietary system cards and capability demonstrations are inherently incomplete. OpenAI's GPT-4V system card and Google's Gemini technical report provide valuable overviews but are fundamentally limited by their source: they are published by the same organizations that developed the models, creating unavoidable conflicts of interest in how capabilities and limitations are presented. These reports tend to emphasize strengths and may selectively sample demonstrations that showcase impressive behavior while eliding systematic failure patterns. Moreover, they typically lack head-to-head comparisons against competitors, making relative capability assessment impossible. The paper's value proposition is precisely that it provides an independent, comparative lens that proprietary documentation cannot offer.

Existing academic benchmarks lack the qualitative depth needed for capability diagnosis. General multi-modal benchmarks like MME, MMBench, SEED-Bench, and MM-Vet provide standardized evaluation protocols and aggregate scores that enable ranking models on leaderboards. The paper explicitly uses MME for quantitative comparison (Section 6), acknowledging its value. However, these benchmarks have structural limitations: they produce numerical scores that conceal how a model arrived at its answer, they typically use test sets drawn from fixed distributions that may not reflect deployment conditions, and they rarely probe specialized expert domains (autonomous driving, defect detection, medical diagnosis) with the nuance needed to assess practical viability. A model might correctly answer a standard VQA question while failing catastrophically on a slightly perturbed real-world variant—a distinction that benchmark scores cannot capture.

Prior qualitative comparisons are narrow in scope. Several earlier works provided qualitative explorations of GPT-4V's capabilities, most notably "The Dawn of LMMs" (Yang et al., 2023), which the paper cites extensively. However, these explorations typically focus on a single model, cover a limited set of task categories, and do not perform systematic multi-model comparison. The paper's innovation is not in inventing qualitative evaluation—it is in applying qualitative evaluation comprehensively across four major domains, multiple difficulty levels, and three competing models simultaneously, creating a multi-dimensional capability map that single-model studies cannot produce.

Open-source model evaluations don't benchmark against the true frontier. The open-source MLLM community (LLaVA, MiniGPT-4, InstructBLIP, Sphinx, etc.) typically evaluates models against academic benchmarks and occasionally against GPT-4V on specific tasks. However, these comparisons often lack the breadth of domain coverage needed to characterize where open-source models genuinely compete and where they fundamentally lag. By including Sphinx alongside Gemini Pro and GPT-4V across the full evaluation suite, the paper provides a detailed gap analysis that identifies specific capability deficits—lack of training data in specialized domains (Section 7.3), weaker abstract reasoning (Section 3.2), inability to handle HTML code generation (Section 3.1)—informing the open-source community's research priorities.

How This Paper Positions Itself

The paper carves out a specific niche in the evaluation landscape that prior work leaves unaddressed: comprehensive, qualitative, multi-model comparison of frontier MLLMs guided by a human-centric testing methodology. Its positioning can be understood along several dimensions:

Independence as a core value proposition. The paper is positioned explicitly as a third-party, community-authored assessment by researchers at Tencent Youtu Lab, Shanghai AI Laboratory, CUHK, USTC, Peking University, and ECNU—institutions with no direct affiliation to the model developers. This independence is crucial for credibility; the paper's findings about Gemini's and GPT-4V's common failure modes (spatial reasoning, logical self-consistency, prompt robustness) carry weight precisely because they come from unaffiliated researchers.

Qualitative depth over quantitative breadth. While the paper includes MME benchmark results (Section 6), its primary contribution is the extensive qualitative analysis across Sections 2-5, comprising over 100 carefully annotated figures. Each figure shows side-by-side model outputs with color-coded annotations (green for correct, red for incorrect, yellow for task failure), allowing readers to directly observe behavioral patterns. This design choice reflects a deliberate philosophical stance: that understanding how models think requires examining their outputs holistically, not reducing them to accuracy scores.

Reconnaissance over definitive assessment. The paper explicitly frames itself as an "early exploration" and "preliminary" study (title, abstract, Section 1.1), acknowledging that Gemini's full capabilities were not yet publicly documented and that the evaluation represents a snapshot at a specific moment. This positioning is strategically important: it establishes the paper as a foundation for ongoing tracking rather than a final verdict, and it invites—indeed, expects—subsequent work to replicate, extend, and update the findings as models evolve. The project repository for tracking MLLM progress, linked in the abstract, reinforces this forward-looking stance.

Serving multiple audiences simultaneously. The paper is structured to be useful to practitioners (domain-specific capability assessments for deployment decisions), researchers (identified failure patterns that point to research problems), and the broader community (accessible qualitative comparisons that reveal model personalities). The four-domain taxonomy—fundamental perception, advanced cognition, challenging vision tasks, expert capacity—is designed to map naturally onto different stakeholder concerns: basic reliability for application developers, reasoning depth for researchers, task-specific proficiency for domain experts, and specialized knowledge for professional users in medicine, autonomous driving, and security.

A contribution to evaluation methodology, not just model comparison. Beyond the specific findings about Gemini and GPT-4V, the paper implicitly argues for a particular approach to MLLM evaluation: prompt diversity (simple instructions, visual referring prompts, Chain-of-Thought, in-context few-shot learning), sample freshness (avoiding train-test leakage through post-November 2023 timestamps), difficulty variation (probing both capabilities and capability boundaries), and systematic cross-model annotation with error categorization. This methodology, while labor-intensive, represents a replicable template for the community to adopt as new models emerge.

3. Technical Approach

3.1 Reader Orientation

This is fundamentally an evaluation and comparison paper — not a paper that builds a new system. The "system" being discussed is the evaluation methodology itself: a curated, multi-domain, qualitative test suite with side-by-side comparisons designed to probe the visual understanding capabilities of three multi-modal large language models (Gemini Pro, GPT-4V, and Sphinx). The problem it addresses is that when a new frontier model appears with limited public technical documentation, the community lacks an independent, systematic way to understand its capabilities relative to existing models. The solution takes the shape of a human-centric evaluation protocol spanning four domains (fundamental perception, advanced cognition, challenging vision tasks, expert capacity) that produces annotated qualitative comparisons augmented by one quantitative benchmark, enabling fine-grained diagnosis of model strengths, failure modes, and behavioral patterns.

3.2 Big-Picture Architecture (Diagram in Words)

The evaluation system has five major components:

1. Sample Collection Pipeline — a deliberately curated set of test instances with controlled properties (source freshness, difficulty gradation, prompt diversity) designed to probe specific capabilities while avoiding train-test leakage.

2. Prompt Design Toolkit — four distinct prompting techniques (simple instruction following, visual referring prompts, Chain-of-Thought prompting, in-context few-shot learning) selected to test different aspects of model instruction-following and reasoning behavior.

3. Three Target MLLMs — Gemini Pro, GPT-4V, and Sphinx, representing the proprietary frontier (the first two) and the open-source state-of-the-art (the third), each queried identically on every test sample.

4. Domain Taxonomy — a four-category organizational framework (fundamental perception, advanced cognition, challenging vision tasks, expert capacity) that structures the evaluation into progressively more demanding capability tiers, with each domain containing multiple subtasks.

5. Annotation and Analysis Framework — a systematic output comparison protocol that color-codes model responses (green = correct, red = incorrect, yellow = task failure/hallucination) and aggregates patterns into identified common issues, model-specific behavioral tendencies, and domain-specific performance characteristics.

Information flows as follows: a test sample (image + text query) is constructed with specific difficulty and domain properties → each of the three models receives the identical sample and prompt → model outputs are collected and placed side-by-side → outputs are annotated with correctness judgments and behavioral observations → patterns are aggregated across samples within each domain → cross-domain synthesis identifies common failure modes and model-specific stylistic differences → quantitative results from MME provide complementary numerical validation.

3.3 Roadmap for the Deep Dive

• First, the sample collection methodology — because the validity of all subsequent comparisons depends on test set quality, freshness, and difficulty calibration.
• Second, the prompt design techniques — since different prompting strategies probe different model capabilities and the paper deliberately varies them to expose behavioral sensitivity.
• Third, the domain taxonomy and what each category tests — because the four-category structure is the paper's primary intellectual framework for organizing findings.
• Fourth, the annotation methodology — how correctness is judged, what the color-coding scheme means, and how patterns are synthesized across samples.
• Fifth, the quantitative MME benchmark integration — how this provides complementary numerical evidence and what its specific evaluation protocol entails.
• Sixth, key methodological design choices — why certain decisions were made (avoiding sample leakage, using post-November 2023 images, cross-validation in difficulty binning) and what alternatives were rejected.

3.4 Detailed, Sentence-Based Technical Breakdown

What Type of Paper This Is and the Core Evaluation Philosophy

This is an empirical evaluation paper whose core idea is that comprehensive qualitative comparison — not just aggregate benchmark scores — is necessary to understand frontier MLLM capabilities, and that a deliberately constructed, multi-domain, difficulty-varied test suite with side-by-side model output annotation reveals capability patterns (stylistic differences, shared failure modes, domain-specific strengths) that numerical scores alone cannot capture. The paper functions as an independent "reconnaissance mission" into Gemini Pro's visual expertise, using GPT-4V as the established ceiling and Sphinx as the open-source baseline, with the explicit goal of characterizing where each model excels, where each fails, and what behavioral tendencies distinguish them.

Sample Collection Pipeline: Curating the Test Suite

Source diversity and freshness guarantees. The paper constructs its qualitative test set from four distinct image sources (Section 1.2.2): manually created drawings, offline photographs, Internet images, and samples from existing datasets (specifically citing nuScenes [10] for autonomous driving, MVTec AD [7] for defect detection, OpenI [15] for medical imaging, and PareidoliaFace [49]). For Internet-sourced images, the authors enforce a temporal freshness constraint: they "strive to collect those with timestamps postdating November 2023." The rationale is explicit — this constraint prevents the models from relying on memorized training data and ensures that correct responses reflect genuine visual understanding rather than pattern matching against previously seen content. This is a crucial design choice because frontier models are trained on massive web corpora with unknown cutoff dates; without temporal controls, it is impossible to distinguish recognition from recall.

Query construction from scratch. All text queries are "constructed from scratch" rather than borrowed from existing benchmarks. This design choice serves two purposes: it avoids the possibility that models have encountered the exact question-answer pairs during training (a form of benchmark contamination), and it allows the authors to precisely control what capability each query probes. For example, a query like "Who is the person on Jordan's left-hand side?" (Figure 1) is specifically designed to test spatial relation understanding, not celebrity recognition — the celebrities are known to the models, so failure can be attributed specifically to spatial reasoning deficits. Similarly, the deliberately misleading prompt "Could you find five differences?" applied to an image with only three differences (Figure 5) is constructed to test whether models fabricate details under inappropriate task demands — a probe for hallucination behavior that a standard benchmark question would never include.

Difficulty gradation within each task. For every task category, the authors collect samples "of varying difficulty levels, e.g., from fundamental perception and cognition to the more challenging vision and expert tasks." This gradation is essential to the paper's evaluation philosophy: it is not sufficient to know that a model can perform a task; one must also identify the boundary at which performance degrades. The paper structures its four domains to naturally encode difficulty progression — fundamental perception (Section 2) contains tasks that test basic visual understanding without complex reasoning, advanced cognition (Section 3) requires multi-step inference, challenging vision tasks (Section 4) demand precise spatial outputs (bounding boxes, temporal tracking), and expert capacity (Section 5) tests specialized domain knowledge. This progression allows the paper to answer not just "can the model do X?" but "at what level of X does the model begin to fail?"

Task-specific sample counts. The paper presents qualitative results across approximately 105 figures (Figures 1-105), with most figures containing one to three comparison examples. While the paper does not report a total sample count for the qualitative suite, the breadth — covering over 30 distinct subtasks across four domains — provides substantial coverage. Each figure typically shows all three model outputs for a given sample, enabling direct side-by-side comparison. The MME benchmark (Section 6) provides complementary quantitative coverage across 14 subtasks with standardized evaluation protocols.

Prompt Design Toolkit: Four Techniques for Capability Probing

The paper employs four distinct prompting strategies (Section 1.2.1), each chosen to test a different aspect of model behavior. The choice of which technique to use for a given sample is deliberate, based on what capability the sample is designed to probe.

Simple instruction following. The most frequently used technique, applied to the majority of samples. A simple instruction directly expresses the user's intention without additional structure: "Describe this image in detail," "Who is this person in the poster?", "Count the number of black birds in the image." The paper states that "Existing MLLMs are generally capable of following instructions, enabling us to utilize the simple instruction to accomplish most tasks effectively." This technique serves as the baseline interaction mode — it tests whether the model can understand and execute a straightforward request without scaffolding. When models fail under simple instructions, the failure is genuinely informative about capability gaps rather than about prompt sensitivity. Figures 1, 3, and many others use this technique.

Visual referring prompt. For tasks requiring spatial localization, the paper employs visual markers rather than purely textual descriptions. The rationale is stated explicitly: "In many cases, a simple visual marker can more effectively convey the user's interest in a specific spatial region to MLLMs than detailed and lengthy text." The paper experiments with two types of visual prompts: standard visual markers (red boxes drawn on images, as in Figure 82 for object tracking) and physical objects used as pointers (a finger or a pen pointing to specific regions, as illustrated in "the bottom part of Figure 35"). The use of physical objects is noted as "more practical in real-time interaction scenarios" than digital markers. This technique tests whether models can ground language in visual referents — a capability that bridges perception and instruction following.

Chain-of-Thought (CoT) prompt. For problems "involving complex logical reasoning," the paper uses CoT techniques, specifically the instruction "Please think step by step." This is applied to samples in the advanced cognition domain, particularly mathematical and physics problems (Section 3.3) and some abstract visual reasoning tasks (Section 3.2). Figure 42 shows a CoT-prompted example for calculating the total cost of grocery items. The paper observes (Section 3.3) that CoT prompting can correct previously wrong answers from Sphinx, "demonstrating the importance of CoT prompting for open-sourced MLLMs." This technique tests whether models can produce structured, multi-step reasoning and whether this reasoning improves final answer accuracy.

In-context few-shot learning. For cases "where simple text instructions fail to completely demonstrate the task," the paper provides a small number of input-output examples before the target query. The key application is in spatial relation recognition (Figure 2), where the paper provides two image examples with their correct answers ("In the first image, Curry's left-hand side is Green. In the second image, Wade's left-hand side is James.") before asking the target question. The paper notes that this approach successfully teaches GPT-4V the concept of "left-hand side" but fails to improve Gemini or Sphinx. This technique tests whether models can learn task specifications from demonstrations at inference time — a meta-learning capability that is particularly important for tasks that are difficult to describe purely through language.

Technique selection rationale. The paper does not apply all four techniques to every sample. Instead, technique choice is task-dependent: simple instructions for straightforward recognition and description tasks, visual referring for spatial localization tasks, CoT for multi-step reasoning problems, and few-shot for tasks where instructions alone are insufficient. This selective application reflects an understanding that prompt engineering is itself part of the evaluation — the question is not just whether a model possesses a capability, but whether that capability can be elicited through appropriate prompting.

Domain Taxonomy: The Four-Category Organizational Framework

The paper organizes all qualitative samples into four domains, each containing multiple subtasks. This taxonomy serves as the paper's primary intellectual structure, and the rationale for each domain is explicitly stated.

Domain 1: Fundamental Perception (Section 2). Defined as "the basic ability of MLLMs to perceive and understand visual concepts, without the need for complex reasoning." This domain is subdivided into three aspects:

• Object-centric perception (Section 2.1): Tasks that focus on individual objects and their properties. Includes spatial relationship recognition (testing left/right discrimination), object counting (testing numerosity estimation), difference spotting (testing fine-grained visual comparison), and optical illusion recognition (testing whether models exhibit human-like perceptual biases). The spatial relation task is particularly diagnostic — Figures 1-2 show that none of the three models reliably distinguishes left from right, suggesting a fundamental limitation in spatial reasoning that persists across model scales and training paradigms.

• Scene-level perception (Section 2.2): Tasks that require holistic interpretation of entire visual scenes. Includes image captioning (tested via "Describe this image in detail") and video scene understanding (tested by presenting multiple temporal frames). The video understanding task (Figure 11) is methodologically interesting: three temporally distinct frames are input simultaneously, and models must integrate information across frames. Gemini successfully merges information (correctly describing two round tables and three potted plants by combining observations from different frames), GPT-4V describes each frame independently, and Sphinx fails to demonstrate temporal integration.

• Knowledge-based perception (Section 2.3): Tasks that require the model to apply stored world knowledge to visual input. Subdivided into commonsense (applying social norms and physical laws), subject knowledge (physics, chemistry, history), multicultural customs (recognizing and describing culturally specific visual elements), and world memory (recognizing celebrities, landmarks, logos, movies, food, plants, animals, artworks). This subdomain is the most extensive in terms of figure count (Figures 12-38), reflecting its importance for practical deployment: models must not only see but also understand what they see in terms of stored knowledge.

Domain 2: Advanced Cognition (Section 3). Defined as tasks requiring "more complicated visual information processing and multi-modal reasoning for problem-solving." This domain tests capabilities beyond pattern recognition:

• Text-rich visual reasoning (Section 3.1): Includes table and chart reasoning (extracting quantitative information from structured visual displays) and visual code generation (converting visual representations into LaTeX or HTML code). The code generation subtask is particularly stringent — Figures 44-47 show that while Gemini and GPT-4V handle LaTeX formula generation reasonably well (with some character-level errors), HTML generation from webpage screenshots reveals a "large improvement space for all three MLLMs." Only Gemini produces "the rough structure of simple websites," while GPT-4V merely extracts text content, and Sphinx fails entirely.

• Abstract visual reasoning (Section 3.2): Tests understanding of non-representational visual patterns using established psychological assessment instruments. Includes tangram shape recognition (Figures 48-49), the Wechsler Adult Intelligence Scale (WAIS) pattern completion tasks (Figures 50-51), and Raven's Progressive Matrices (Figures 52-53). These are among the most challenging tasks in the suite — nearly all models fail on the WAIS and Raven's tasks, with GPT-4V showing "some detailed reasoning process" but still producing incorrect final answers. The paper interprets this as evidence that current MLLMs "can initially identify the independent element [but] fail to parse their relationship for further inference."

• Science problem-solving (Section 3.3): Mathematical and physics problems requiring both OCR of symbolic notation and quantitative reasoning. Covers arithmetic, algebra, geometry, trigonometry, integral calculus, and physics problems in dynamics, kinematics, and circuitry. The paper observes that Gemini and GPT-4V handle simple arithmetic and algebra well but degrade on advanced problems, and that CoT prompting can "rectify" previously wrong answers from Sphinx (e.g., Figure 58).

• Emotion understanding (Section 3.4): Tests interpretation of emotional content in visual stimuli, including facial expression analysis, image emotion analysis (inferring emotional valence from scenes without visible faces), and emotion-conditioned output (describing images through a specified emotional lens, such as "in a romantic way" or "in a terrifying way"). The emotion-conditioned output task reveals a fascinating behavioral pattern: both Gemini and GPT-4V correctly inject the specified emotion but suffer from hallucination (describing non-existent elements like a "bike" or "shadows" in Figure 70), while Sphinx avoids hallucination but produces less emotionally nuanced descriptions.

• Game playing (Section 3.5): Strategic reasoning tasks including Sudoku, Crossword, and Go. The Sudoku experiment (Figure 72) uses a clever two-phase protocol: first present only the image, then provide the grid as text. When given only the image, Gemini attempts to fill in the grid but "cannot correctly identify the blank position," while GPT-4V and Sphinx fail at OCR. When given the text grid, both Gemini and GPT-4V solve the puzzle correctly.

Domain 3: Challenging Vision Tasks (Section 4). Tasks that "extend beyond the scope of standard visual question-answering" and require precise spatial outputs:

• Image-level tasks (Section 4.1): Object detection (providing bounding boxes for objects), referring expression comprehension (locating objects specified by natural language descriptions), phrase localization (finding all nouns from a sentence in the image), and face detection and recognition. The bounding box tasks reveal a striking pattern: Gemini provides coordinates that are "generally imprecise," GPT-4V "avoids directly offering coordinates and instead attempts to utilize external tools," and Sphinx "offers a relatively reliable answer, yet its performance falls significantly short when compared to that of conventional object detectors" (Figure 77). This suggests that current MLLMs, despite their linguistic fluency, lack the fine-grained spatial precision of specialized vision models.

• Temporal-level tasks (Section 4.2): Object tracking across video frames (Figure 82), video action recognition (Figures 83-85), and visual story generation (Figure 86). The object tracking task reveals that while both Gemini and GPT-4V can "delineate the details of the target to be tracked," they "subsequently provide incorrect bounding boxes in the following two frames." The visual story generation task tests a different capability: coherent narrative construction from sequential images.

Domain 4: Expert Capacity (Section 5). Tasks in "specialized fields" that test generalization to professional contexts:

• Autonomous driving (Section 5.1): Traffic sign recognition, driving intention inference, and weather-adaptive decision making. Figures 87-90 show that all three models correctly identify basic elements (traffic lights, pedestrians, weather conditions) but "for small and low-resolution patterns in traffic or road signs, the three models are struggling to precisely recognize them."

• Defect detection (Section 5.2): Industrial quality inspection using the MVTec AD dataset. The models show variable performance — all three detect obvious defects (holes in fabric, broken rings), but for subtle defects (a small hole in a cup, Figure 93), Gemini misidentifies it as condensation, while GPT-4V and Sphinx miss it entirely.

• Medical diagnosis (Section 5.3): Chest X-ray interpretation using the OpenI dataset. The paper includes ground-truth radiology reports for four X-ray images (Figures 94-97), enabling direct comparison. Results indicate that "for such domain-specific visual input, the MLLMs pre-trained by general images cannot consistently produce satisfactory results," with models tending to "make judgments of no symptoms" for complex lesions. A notable methodological detail: the authors use a specialized prompt that explicitly states "The content of the report will only be used for large-scale model capability assessment and will not be used as actual medical guidance" to prevent models from refusing to engage with medical content.

• Economic analysis (Section 5.4): Stock chart interpretation and market expectation inference. Figures 98-99 show that Gemini "is good at expert-level financial knowledge, and is capable of responding with the correct answers, while GPT-4V does not give a clear answer due to security risks."

• Surveillance and security (Section 5.5): Safety hazard detection in construction sites. Figures 100-101 test whether models can identify workers not wearing appropriate protective equipment. Gemini detects the safety violations in both cases, while GPT-4V notices insufficient protective gear but with some detail misrecognition.

• Remote sensing image analysis (Section 5.6): Satellite and aerial image interpretation. Figure 102-103 show that Gemini can count aircraft on an airport tarmac (correctly identifying four planes where GPT-4V finds only two) and, in the second sample, "impressively deduce that this place is Japan."

• Robot motion planning (Section 5.7): Assembly sequence generation from component diagrams. Figures 104-105 test whether models can produce step-by-step assembly instructions for a mobile phone and an office chair. Gemini and GPT-4V both produce "methodical and detailed steps," with GPT-4V providing more logical ordering (e.g., correct battery installation sequence).

Annotation Methodology: How Correctness and Behavioral Patterns Are Assessed

Color-coded annotation scheme. Every qualitative figure uses a three-color annotation system:

• Green: highlights correct answers. These are responses that match ground truth or, for open-ended description tasks, provide accurate and hallucination-free information.

• Red: highlights wrong answers. These are responses containing factual errors, incorrect identifications, or reasoning errors that lead to wrong conclusions.

• Yellow: highlights "incompetence in performing the task." This is a distinct category from factual errors — it indicates that the model fundamentally cannot execute the requested task format, such as failing to understand the instruction, refusing to engage, or producing output that does not address the query. Examples include GPT-4V's refusal to identify celebrities (Figures 24, 29, 81), Gemini's insertion of Japanese text in an English response (Figure 9), and Sphinx's failure to comprehend object tracking instructions (Figure 82).

Ground truth determination. For tasks with objective answers (counting, spatial relations, mathematical calculations, factual recognition), correctness is determined against verifiable ground truth. For subjective tasks (scene description, emotion analysis, story generation), correctness is assessed based on accuracy (does the description match visible elements?), completeness (are major elements captured?), and absence of hallucination (are fabricated details absent?). The paper does not use automated metrics for qualitative assessment — all annotations are human judgments by the authors.

Pattern synthesis across samples. After annotating individual samples, the paper synthesizes observations into higher-level patterns:

• Model-specific behavioral tendencies: GPT-4V "tends to elaborate detailed explanations and intermediate steps" while Gemini "prefers to output a direct and concise answer." GPT-4V exhibits "more dialectical thinking with rigorous analysis" in emotion tasks (Section 3.4). GPT-4V, "due to privacy concerns, may decline to respond to topics related to celebrities" (Section 7.2).

• Common failure modes across models: Four shared issues are identified (Section 7.2): (1) poor spatial perception capabilities, (2) unsatisfactory OCR and abstract visual understanding, (3) logical self-consistency failures where intermediate reasoning contradicts final answers, and (4) sensitivity to prompt framing, where differently worded versions of the same question produce opposite answers.

• Domain-specific performance patterns: Sphinx performs competitively on perception tasks using data from academic datasets (likely due to training distribution alignment) but lags severely on cognition tasks. Gemini shows broader domain generalizability for specialized expert tasks. GPT-4V excels at detailed perceptual description but avoids certain task categories on policy grounds.

Quantitative MME Benchmark Integration

The MME benchmark (Section 6) provides the paper's only quantitative evaluation, complementing the extensive qualitative analysis. Its integration into the evaluation framework requires understanding its specific design.

Binary question design. The MME benchmark is "uniquely structured to elicit straightforward 'yes' or 'no' responses." For each test image, two questions are constructed: one designed to have "yes" as the correct answer, and one designed to have "no" as the correct answer. This dual-question design is pivotal because it prevents models from exploiting response biases — a model that always answers "yes" would score 50% on standard accuracy but 0% on the enhanced metric. The paper states: "A model that correctly answers both questions demonstrates not only a clear understanding of the visual content but also an underlying grasp of the associated contextual knowledge."

Dual evaluation metrics. The MME uses two complementary accuracy measures:

• Standard accuracy: Computed question-by-question. The baseline random accuracy is 50% (binary choice).

• Enhanced accuracy (accuracy+): A "more rigorous measure, requiring correct answers to both questions associated with a single image." The baseline random accuracy is 25% (both binary choices must be correct independently).

The paper notes that "the score for each sub-task is the sum of both accuracy measures," and "the overall perception and cognition scores are derived from the cumulative scores of their respective sub-tasks." This summation means that the maximum possible score per sub-task depends on the number of test samples, and the total scores reported (e.g., Gemini's 1933.4) are sums across all 14 sub-tasks.

Fourteen sub-tasks across two categories. The MME organizes its evaluation into perception sub-tasks (Existence, Count, Position, Color, OCR, Poster, Celebrity, Scene, Landmark, Artwork) and cognition sub-tasks (Commonsense Reasoning, Numerical Calculation, Text Translation, Code Reasoning). Data sources differ by category: perception tasks use "publicly available datasets" (COCO [34], MovieNet [28], DeepArt [40], Places [64], Google Landmarks Dataset v2 [55]), while cognition tasks use "manually photographed or diffusion model-generated sources." This distinction is important because it explains Sphinx's competitive perception performance — the open-source model was likely trained on data from these same academic distributions.

Key quantitative findings. Table 1 and Figure 106 present the MME results. GPT-4V dominates cognition tasks, particularly code reasoning (170.0 vs. Gemini's 85.0 and Sphinx's 50.0). However, GPT-4V scores zero on celebrity recognition due to refusal to identify real people — a policy-driven failure rather than a capability failure. Sphinx leads on several perception tasks, including Position (153.3 vs. Gemini's 90.0 and GPT-4V's 95.0), consistent with the qualitative finding that spatial understanding is a shared weakness. Gemini achieves the highest total score (1933.4) primarily by being "more balanced" — not dominating any single category but avoiding the zero-score penalty that GPT-4V incurs on celebrity recognition and maintaining competitiveness across all tasks.

Key Methodological Design Choices and Their Justifications

Choice 1: Qualitative over quantitative primacy. The paper invests far more effort in qualitative sample curation and annotation than in benchmark evaluation. The justification is implicit throughout: qualitative analysis reveals how models think and where they fail in ways that benchmark scores cannot. The decision to present over 100 annotated figures with side-by-side model outputs reflects a philosophical commitment to transparency — readers can directly observe model behavior rather than trusting aggregate statistics. The cost is that qualitative analysis does not produce confidence intervals or statistical significance tests, limiting the generalizability of individual observations. The paper mitigates this by including the MME benchmark for complementary quantitative validation.

Choice 2: Temporal freshness constraint. The requirement that Internet images postdate November 2023 is a deliberate defense against data contamination. This constraint is more stringent than typical academic evaluation practice, where test sets may have been constructed years earlier and potentially included in training data. The paper acknowledges the practical difficulty of this constraint — it requires identifying and verifying image timestamps — but argues that the validity gain justifies the effort. The alternative (using images of unknown provenance) would undermine the claim that correct responses reflect genuine understanding rather than memorization.

Choice 3: Three-model comparison rather than pairwise. The paper could have compared Gemini only against GPT-4V (to establish frontier competitiveness) or only against Sphinx (to characterize the open-source gap). Including all three serves multiple purposes: GPT-4V provides the capability ceiling, Sphinx provides the open-source baseline, and the three-way comparison reveals that some failures are model-specific while others are shared across architectures and training paradigms. This design enables the paper's central finding about common failure modes — if only two models had been compared, it would be impossible to distinguish universal limitations from model-specific weaknesses.

Choice 4: Deliberately adversarial prompts. The paper includes intentionally misleading prompts (requesting five differences when only three exist, Figure 5; asking about optical illusions that deceive human perception, Figures 6-7) that would never appear in standard benchmarks. These adversarial prompts serve a specific diagnostic purpose: they test whether models can recognize task invalidity (the five-differences prompt), resist perceptual illusions (the optical illusion prompts), and maintain logical consistency under pressure (the "Yes or No" question framing variations in Figures 43 and 59). The finding that both Gemini and GPT-4V fabricate differences to satisfy an impossible request reveals a fundamental behavioral tendency — compliance over accuracy — that only emerges under adversarial probing.

Choice 5: Ground-truth inclusion for expert tasks. For medical diagnosis (Section 5.3), the paper includes actual radiology reports as ground truth, enabling direct comparison between model outputs and professional human judgments. This is a high-evidence-bar design that most qualitative evaluations avoid, precisely because it exposes the gap between model performance and expert human performance. The inclusion of ground truth transforms the medical diagnosis section from a demonstration of capability into a rigorous capability assessment with clear failure documentation.

Choice 6: Prompt diversity as part of evaluation, not just optimization. The paper uses four different prompting techniques not to optimize model performance but to characterize how models respond to different interaction modes. This is a departure from evaluation-as-optimization paradigms where the goal is to find the prompt that maximizes accuracy. The paper's approach treats prompt sensitivity itself as a property to be measured — a model that only works under CoT prompting has different deployment characteristics than one that works under simple instructions. The finding that CoT corrects Sphinx's errors while not substantially affecting Gemini or GPT-4V reveals that open-source models may have different "prompt sensitivity profiles" than proprietary ones, a finding with practical implications for how these models should be deployed.

4. Key Insights and Innovations

Innovation 1: Diagnostic Prompting as a Deliberate Evaluation Methodology

The paper's most methodologically distinctive contribution is not that it evaluates MLLMs qualitatively — prior work has done that — but that it systematically constructs adversarial, misleading, and difficulty-calibrated prompts specifically to expose failure modes that standard benchmarks cannot detect. This transforms qualitative evaluation from a demonstration activity (showing what models can do) into a diagnostic instrument (revealing where and how they break).

Prior qualitative evaluations of MLLMs, including OpenAI's GPT-4V system card, Google's Gemini technical report, and academic explorations like "The Dawn of LMMs" (Yang et al., 2023), largely designed prompts to elicit impressive behavior — asking models to describe images, answer questions, and demonstrate capabilities. When failures occurred, they were incidental findings rather than the result of deliberate probing. The paper's innovation is to invert this logic: construct prompts whose purpose is to test a specific boundary.

Consider three examples that would never appear in a standard benchmark:

• The five-differences prompt (Figure 5): The authors show two images with only three differences but ask the model to "find five differences." Both Gemini and GPT-4V fabricate two additional, non-existent differences to satisfy the request. This reveals a behavioral priority — compliance with the user's stated expectation over accuracy — that only emerges under deliberately invalid task specifications. A benchmark that only asks factually correct questions would never detect this.

• The prompt framing sensitivity test (Figures 43, 59): The same underlying question is asked in two different phrasings. For the arithmetic question in Figure 59, asking "Is the answer 32? Please answer yes or no" elicits "yes" from Gemini, while asking "What is the answer?" elicits the correct "33." This reveals logical consistency failures that are invisible to single-prompt evaluations.

• The optical illusion probes (Figures 6-7): Rather than testing whether models can recognize objects in illusions (a standard perception task), the authors test whether models share human perceptual biases. GPT-4V, like most humans, is deceived by the pear brightness illusion, while Gemini recognizes the equality. This tests a fundamentally different capability than recognition — it tests whether visual processing in these models follows human-like heuristics or operates on different principles.

Why this is a conceptual advance, not just a methodological trick. The paper's prompt design philosophy embodies a shift in how the field thinks about evaluation. Standard benchmarks assume that the space of possible queries is unbiased and that aggregate accuracy over this space measures capability. The paper demonstrates that this assumption is false: models exhibit qualitatively different behavior under adversarial or unexpected prompts, and these behaviors — compliance bias, hallucination under pressure, sensitivity to framing — are as diagnostic of model quality as accuracy on standard questions. The implication is that benchmarking suites must include designed failure probes, not just sampled test instances, to adequately characterize model behavior.

The methodological framework is replicable: any new MLLM can be subjected to the same adversarial prompt types (impossible requests, framing variations, perceptual illusions), and the resulting failure patterns can be compared across models. This is a contribution to evaluation science, not just to model comparison.

---

Innovation 2: Characterizing Model "Personality" as a First-Class Evaluation Dimension

The paper identifies and systematically documents a phenomenon that prior MLLM evaluation largely ignored: different models exhibit stable, cross-task behavioral tendencies — a form of "response personality" — that are independent of task accuracy and have direct implications for deployment suitability. This insight elevates stylistic differences from anecdotal observations to an evaluation dimension that should be measured alongside accuracy.

The central finding is that GPT-4V and Gemini have markedly different answering styles that persist across task types: "GPT-4V tends to elaborate detailed explanations and intermediate steps, and Gemini prefers to output a direct and concise answer" (Section 7.2). This is not a claim about which model is "better" — it is a claim about behavioral consistency that matters for different use cases. The paper provides extensive evidence for this pattern:

• Scene description (Figures 8-10): GPT-4V produces multi-paragraph, highly detailed descriptions enumerating visual elements, while Gemini provides shorter, more selective summaries. In Figure 8 (the shopping mall), GPT-4V identifies specific activity stations, brand signage, and architectural details that Gemini omits.

• Emotion analysis (Figures 65-69): GPT-4V provides "dialectical thinking with rigorous analysis" and explicitly notes that "emotions are subjective," while Gemini "tends to directly output the emotion preference, which corresponds with mainstream perspectives" (Section 3.4).

• Table and chart reasoning (Figures 39-42): GPT-4V provides step-by-step reasoning with explicit intermediate calculations, while Gemini often jumps to the final answer directly.

• Privacy-sensitive tasks (Figures 24, 29, 81, and MME Celebrity metric): GPT-4V systematically refuses to identify real people or make assumptions about individuals in images, explaining its privacy policy. Gemini provides direct identifications without caveats. This is a policy-driven behavioral difference, not a capability difference, but it has identical practical consequences for deployment.

Why this is conceptually novel. Prior evaluation frameworks — both academic benchmarks and system cards — treat model outputs as transparent windows into capability. A correct answer is a correct answer; the presentation format is irrelevant. The paper demonstrates that this assumption is incorrect for practical deployment. A medical professional reviewing an X-ray interpretation might prefer GPT-4V's detailed, caveat-filled reports that explicitly state limitations, while a user asking for quick landmark identification might prefer Gemini's direct answers. The "personality" of the model — its verbosity, its willingness to hedge, its policy-driven refusals — determines user experience as much as raw accuracy does.

The paper's contribution is to document these personalities empirically, not to advocate for one over the other. This creates a new vocabulary for model comparison that goes beyond "better" or "worse" to "different in ways that matter for different contexts."

---

Innovation 3: The Open-Source Capability Gap Is Domain-Specific, Not Uniform

The three-way comparison between Gemini Pro, GPT-4V, and Sphinx reveals a finding that complicates the standard narrative about open-source MLLM capabilities: Sphinx's performance is not uniformly weaker than proprietary models; it is domain-specifically weaker in ways that reveal where training data and model scale matter most. This finding has direct implications for open-source MLLM development priorities.

Where Sphinx competes (and why). On perception tasks using images from academic datasets — COCO [34], MovieNet [28], DeepArt [40], Places [64] — Sphinx is "on par or even exceeds that of Gemini and GPT-4V" (Section 7.2, MME results). On the MME benchmark, Sphinx leads in several perception sub-tasks including Position (153.3 vs. Gemini's 90.0 and GPT-4V's 95.0), Existence (195.0 vs. Gemini's 175.0 and GPT-4V's 190.0), and Count (160.0 vs. Gemini's 131.7). On qualitative tasks, Sphinx can recognize emotions in images (Section 3.4), detect defects (Section 5.2), and perform facial expression analysis (Section 3.4) with competitive accuracy.

Where Sphinx fails (and why). The capability gap manifests primarily in three categories:

1. Cognition tasks requiring abstract reasoning: On MME cognition sub-tasks, Sphinx trails dramatically — Code Reasoning (50.0 vs. GPT-4V's 170.0 and Gemini's 85.0), Numerical Calculation (55.0 vs. GPT-4V's 130.0 and Gemini's 129.3), Text Translation (75.0 vs. Gemini's 77.5 and GPT-4V's 75.0). Qualitative results confirm this: Sphinx fails entirely at Raven's Progressive Matrices (Figure 52), cannot interpret flowcharts (Figure 39), and produces nonsensical reasoning for physics problems (Figure 62).

2. Tasks requiring specialized domain knowledge not in training data: Sphinx cannot generate HTML code from webpage screenshots (Figures 46-47), fails at economic chart analysis (Figures 98-99), misidentifies scientific equipment (Figure 17), and struggles with multicultural knowledge requiring specific cultural vocabulary (Figures 20-22). The paper attributes this to "the diversity of Sphinx's training data [being] still lacking in some domains."

3. Tasks requiring large-scale parametric knowledge for recognition: Sphinx misidentifies celebrities (confusing Doctor Strange in an Iron Man suit for Iron Man, Figure 23), cannot recognize landmarks with the same precision as proprietary models (Figures 25-26), and provides less accurate plant and animal species identification (Figure 34).

Why this finding restructures the open-source development conversation. The standard narrative in the MLLM community treats the open-source gap as a monolithic capability deficit driven by model scale. The paper's evidence suggests a more nuanced picture: Sphinx's underlying LLaMA-2-7B backbone provides sufficient capacity for perceptual tasks when trained on the right data distributions, but lags in abstract reasoning (a property of the base LLM's reasoning capability) and specialized knowledge recall (a property of training data coverage). This decomposition has practical implications: improvements to open-source MLLMs on cognition tasks require stronger base LLMs (e.g., scaling to 70B parameters), while improvements on specialized domain tasks require targeted data collection, and improvements on perception from standard distributions may already be near the achievable ceiling at current model scales.

The paper itself makes this point explicitly (Section 7.3): "The second limitation is the inherent reasoning upper bound of the underlying LLM. Sphinx adopts the vanilla LLaMA-2-7B for initialization, which falls short compared to larger-scale LLMs in certain complex tasks." This is a testable hypothesis that subsequent work can verify by evaluating Sphinx-style architectures built on larger open-source LLMs.

---

Innovation 4: Verifier Over-Optimization and Hallucination Under Pressure as Universal Frontier Behaviors

The paper identifies four "common issues" shared by Gemini and GPT-4V (Section 7.2), but the deeper insight is that these are not independent bugs — they are symptoms of a unified underlying tendency for frontier MLLMs to prioritize task compliance and response coherence over factual accuracy when placed under cognitive pressure. This finding represents a negative result with significant implications: it suggests that scaling model capability does not eliminate certain failure modes but may actually amplify them in specific contexts.

The evidence for a unified underlying mechanism. The four common issues are:

1. Spatial perception deficits (Figure 1, MME Position scores): Despite massive training on visual data, both leading models cannot reliably distinguish left from right — a capability that even young children possess. The MME Position scores (90.0 for Gemini, 95.0 for GPT-4V) are among the lowest relative scores for both models, and the qualitative example in Figure 1 shows all three models failing the same left-right question about basketball players.

2. OCR and abstract visual understanding failures (Figures 41, 45, 50, 52): In chart interpretation tasks, both models misread numerical values. In LaTeX generation, both models miss minor characters and symbols. In Raven's Progressive Matrices, both models produce incorrect final answers despite showing some reasoning capability.

3. Logical self-consistency violations (Figures 43, 59, 62): For "Yes or No" questions and physics problems, both models occasionally produce intermediate reasoning that contradicts their final answer. This is not a reasoning failure per se — the reasoning is often correct — but a failure to maintain consistency between the reasoning process and the answer selection.

4. Prompt sensitivity (Figures 43, 59): Differently worded versions of the same question can elicit opposite answers from both models.

Why these likely share a common cause. Each of these failures can be understood as the model prioritizing different objectives that come into conflict. When prompted to find five differences in an image with only three (Figure 5), the model faces a conflict between the instruction ("find five differences") and the visual evidence (only three exist). The model resolves this conflict in favor of instruction compliance — it fabricates two additional differences. When asked "Is the answer 32?" versus "What is the answer?" (Figure 59), the model faces a conflict between answering the question as posed and independently computing the correct answer. The model resolves this by treating the prompt's suggested answer as presumptively correct. When producing spatial judgments, the model may lack the fine-grained visual representations needed for left-right discrimination but still produces confident answers rather than expressing uncertainty.

The unifying pattern is task compliance dominating verification: when the model's generative fluency and instruction-following training push toward producing a coherent, responsive answer, and its perceptual or reasoning systems have incomplete or noisy information, the fluency objective wins. The result is confident, fluent, wrong answers.

Why this is a fundamental insight, not just an error catalog. The fact that these failure patterns persist across two independently developed, state-of-the art proprietary models trained by different organizations with different architectures, data, and training procedures strongly suggests they are not implementation artifacts. They are likely inherent to the current paradigm of training large models to follow instructions and generate coherent text, with verification and factuality as secondary objectives added through fine-tuning rather than architectural primitives. If this interpretation is correct, simply scaling models further will not eliminate these failures — it will make the fluent wrong answers even more fluent and confident, potentially making them harder to detect.

The paper's contribution is to document these patterns in a systematic, cross-model comparison that makes the universality claim empirically grounded. Prior work had noted individual instances of each failure type, but the demonstration that the same patterns appear across models and task types with consistent structure is what elevates this from observation to diagnostic finding.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The qualitative evaluation uses a manually curated collection of images sourced from four origins (Section 1.2.2): manually created drawings, offline photographs, Internet images with timestamps postdating November 2023, and samples from existing datasets including nuScenes, MVTec AD, OpenI, and PareidoliaFace. The quantitative evaluation uses the MME benchmark (Section 6.1), which comprises 14 sub-tasks across perception and cognition, with perception samples drawn from publicly available datasets (COCO, MovieNet, DeepArt, Places, Google Landmarks Dataset v2) and cognition samples sourced from manually photographed or diffusion model-generated images. The total number of qualitative samples is approximately 105 figures, with most figures containing one to three test instances, while the MME benchmark provides standardized coverage across all 14 sub-tasks with a fixed evaluation protocol — though the paper does not report the total number of MME test instances.

• Base model(s). Three MLLMs are evaluated: Gemini Pro (Google, released late 2023), GPT-4V (OpenAI, the established state-of-the-art at time of writing), and Sphinx (the open-source model built on LLaMA-2-7B using the joint mixing of weights, tasks, and visual embeddings approach from Lin et al., 2023). Gemini Pro and GPT-4V represent the proprietary frontier, while Sphinx provides an open-source baseline to "reveal the gap between manual efforts and black-box systems" (Section 1.1). The specific model versions, checkpoint identities, and decoding parameters (temperature, sampling strategy) are not reported for any of the three models — the paper queries each model identically on every test sample but does not control for or document generation hyperparameters, which limits reproducibility.

• Metrics. For qualitative evaluation (Sections 2-5), correctness is assessed through human annotation using a three-color scheme: green for correct answers, red for incorrect answers, and yellow for cases where the model fundamentally cannot execute the requested task format. For open-ended description tasks, correctness is assessed based on accuracy relative to visible image elements, completeness, and absence of hallucination. No inter-annotator agreement statistics, annotation guidelines, or multiple-annotator protocols are reported. For quantitative evaluation (Section 6), the MME benchmark uses two metrics: standard accuracy (computed question-by-question, baseline random accuracy 50%) and enhanced accuracy (accuracy+, requiring correct answers to both questions associated with a single image, baseline random accuracy 25%). The score for each sub-task is the sum of both accuracy measures, and overall perception and cognition scores are cumulative sums across respective sub-tasks — the paper does not report raw accuracy percentages, only the summed scores, making cross-benchmark comparison difficult without knowing the number of test items per sub-task.

• Baselines. The three models serve as mutual baselines for each other — there is no separate "baseline model" in the traditional sense, as the paper's purpose is comparative rather than ablative. GPT-4V is treated as the established capability ceiling (representing "the current highest standard in the field"), while Sphinx serves as the open-source lower bound ("exploring how much the performance gap is between open-sourced models and closed-sourced systems"). The paper does not include simpler baselines such as task-specific computer vision models (e.g., a dedicated object detector for the bounding box tasks, a dedicated OCR system for text-rich tasks) that would contextualize MLLM performance against specialized systems. The MME benchmark results (Table 1, Figure 106) provide the only numerical comparison where all three models are evaluated under identical scoring protocols.

• Generation budget / compute accounting. The paper does not employ a generation budget or compute accounting framework. Each model is queried once per test sample with the specified prompt technique, and the output is evaluated. There is no analysis of inference cost, latency, or FLOPs expenditure, and no attempt to compare models under equalized compute budgets. The implication is that the comparison is capability-focused rather than efficiency-focused — the question is what each model can do, not what it can do per unit of compute. This is a significant omission for a paper that aims to inform deployment decisions, since the computational cost of running Gemini Pro versus GPT-4V versus Sphinx differs substantially (Sphinx, at 7B parameters, is likely orders of magnitude cheaper to serve than the proprietary models), and cost-capability tradeoffs are central to practical model selection.

• Cross-validation / statistical protocol. For the qualitative evaluation, no cross-validation or statistical testing is employed — individual samples are presented as representative illustrations of model behavior, and conclusions are drawn through pattern synthesis across samples within each domain. For the MME benchmark, the evaluation follows the MME's fixed protocol (Section 6.1) without additional cross-validation, and results are reported as single numbers per sub-task per model (Table 1). No confidence intervals, statistical significance tests, or error bars are reported. The paper does not describe any procedure for ensuring that qualitative samples were selected without confirmation bias (e.g., pre-registration of test cases, independent sample selection, or blinding of annotators to model identity), which is a concern given that the annotators are also the paper's authors and the samples were deliberately chosen to probe capability boundaries.

Main Quantitative Results

MME Benchmark: Overall Rankings and Domain-Level Patterns

The MME benchmark (Table 1, Figure 106) provides the paper's sole quantitative comparison. Gemini achieves the highest overall score of 1933.4, followed closely by GPT-4V at 1926.6, with Sphinx trailing at 1870.2. The difference between Gemini and GPT-4V (6.8 points) is extremely narrow relative to the total score range, and without reported confidence intervals it is impossible to determine whether this difference is statistically significant — the paper treats it as meaningful evidence that Gemini is a "strong challenger," but the magnitude suggests the two models are effectively tied on this benchmark.

Perception sub-tasks. Sphinx leads on several perception tasks: Existence (195.0 vs. Gemini's 175.0 and GPT-4V's 190.0), Count (160.0 vs. 131.7 and 160.0), Position (153.3 vs. 90.0 and 95.0), and Poster (164.3 vs. 165.0 and 192.2 — Sphinx is close to Gemini but behind GPT-4V). GPT-4V leads on Poster (192.2), OCR (185.0 vs. Gemini's 185.0 — tied — and Sphinx's 87.5), and Scene (151.0 vs. 144.8 and 160.0 — actually behind Sphinx). GPT-4V scores exactly zero on Celebrity recognition due to its policy of refusing to identify real people in images, a capability failure induced by safety constraints rather than visual understanding limitations. This zero score accounts for most of the gap between GPT-4V and Gemini in the total perception score.

Cognition sub-tasks. GPT-4V dominates cognition: Code Reasoning (170.0 vs. Gemini's 85.0 and Sphinx's 50.0), Numerical Calculation (130.0 vs. 129.3 and 55.0 — essentially tied with Gemini), Commonsense Reasoning (142.1 vs. 129.3 and 130.0). Gemini leads on Text Translation (77.5 vs. GPT-4V's 75.0 and Sphinx's 75.0) but the margins are narrow. Sphinx's cognition scores are substantially lower across all sub-tasks except Text Translation, confirming the qualitative finding that open-source models lag most severely in reasoning-heavy capabilities.

Figure 106 radar chart patterns. The radar chart visualization reveals several structural patterns: (1) GPT-4V's profile shows a sharp spike in Code Reasoning and a sharp dip to zero in Celebrity, creating a highly irregular capability profile. (2) Gemini's profile is more balanced across sub-tasks, with no zero-score categories and moderate performance across all 14 dimensions. (3) Sphinx's profile is bimodal — strong on perception sub-tasks (Existence, Count, Position) but weak on cognition sub-tasks (Code Reasoning, Numerical Calculation, OCR). This structural difference in capability profiles is potentially more informative than the aggregate score ranking, as it suggests different deployment suitability: GPT-4V for code-related and detailed perceptual tasks (but with privacy-related refusal risks), Gemini for balanced general-purpose use, and Sphinx for perception-heavy applications where cost is a primary concern.

Qualitative Pattern 1: Stylistic Divergence in Response Generation

Across nearly all task categories, GPT-4V and Gemini exhibit stable, cross-domain differences in how they structure responses. This is documented qualitatively across approximately 105 figures but is not quantified — there is no measurement of average response length, no analysis of hedging frequency, and no systematic comparison of response structure.

Verbosity and detail level. GPT-4V consistently produces longer, more detailed responses. In scene understanding tasks (Figures 8-10), GPT-4V's descriptions are "notably more detailed and exhibit fewer instances of hallucination" (Section 2.2). In the shopping mall scene (Figure 8), GPT-4V identifies specific activity stations ("PUKY," "MOOVnGO"), brand signage ("McDonald's"), and architectural details (glass ceiling, natural light) that Gemini omits. In the fantasy landscape scene (Figure 9), GPT-4V provides a multi-paragraph description analyzing architectural styles, character equipment, and atmospheric qualities, while Gemini produces a shorter summary. In the accident scene (Figure 10), GPT-4V correctly identifies that the two images represent different perspectives of the same event — a non-obvious inference — while Sphinx merely notes the presence of police and a crowd.

Reasoning transparency. For cognition tasks, GPT-4V typically externalizes its reasoning process. In flowchart interpretation (Figure 39), GPT-4V provides a step-by-step walkthrough of the decision logic; Gemini summarizes the high-level function in a single sentence. In chart-based question answering (Figure 41), GPT-4V shows explicit calculation steps with numerical values extracted from the chart, while Gemini provides a final answer directly. In emotion analysis tasks (Figures 65-69), GPT-4V engages in "dialectical thinking with rigorous analysis" — noting multiple possible interpretations, acknowledging subjectivity, and providing balanced assessments — while Gemini "tends to directly output the emotion preference, which corresponds with mainstream perspectives" (Section 3.4).

Privacy and safety behavior. GPT-4V systematically refuses to identify or make assumptions about real people in images (Figures 24, 29, 81), citing privacy policies. Gemini provides direct identifications without caveats. This is visible quantitatively in GPT-4V's zero MME Celebrity score and qualitatively in multiple figures where GPT-4V states variations of "I'm sorry, but I can't help identify or make assumptions about people in images." This behavioral difference is policy-driven rather than capability-driven but has identical practical consequences — applications requiring celebrity recognition or person identification cannot use GPT-4V, regardless of its underlying visual recognition accuracy.

Qualitative Pattern 2: Shared Failure Modes Across Models

The paper identifies four "common issues" (Section 7.2) that affect both Gemini and GPT-4V, documented through qualitative examples.

Spatial perception deficits. The Position sub-task on MME shows scores of 90.0 (Gemini) and 95.0 (GPT-4V), both substantially below Sphinx's 153.3. The qualitative example in Figure 1 shows all three models failing to correctly identify "the person on Jordan's left-hand side." Even with in-context few-shot examples teaching the concept of "left-hand side" (Figure 2), only GPT-4V successfully generalizes — Gemini and Sphinx continue to fail. This is perhaps the paper's most striking finding about universal limitations: despite massive training, current frontier models cannot reliably perform a spatial reasoning task that young children master.

OCR and abstract visual understanding limitations. In chart interpretation (Figure 41), both Gemini and GPT-4V produce numerical estimates that are imprecise or incorrect when the required values must be read from chart axes. In LaTeX code generation (Figures 44-45), both models occasionally misrecognize characters or symbols in mathematical notation, particularly for complex formulas. In Raven's Progressive Matrices (Figures 52-53) and WAIS tasks (Figures 50-51), both models "can initially identify the independent element [but] fail to parse their relationship for further inference" (Section 3.2), with nearly all final answers being incorrect despite some correct intermediate reasoning steps from GPT-4V.

Logical self-consistency violations. In the taxi fare example (Figure 43, left), Gemini produces a directly contradictory answer: stating "you should not pay $29.42" and then immediately stating "you should actually pay$29.42" — the intermediate reasoning contradicts itself. In physics problem-solving (Figure 62), GPT-4V's step-by-step reasoning contains equations that are internally inconsistent or misapplied. In the arithmetic question with prompt framing manipulation (Figure 59), Gemini answers "yes" when asked whether the answer is 32, but answers "33" when asked directly what the answer is — the reasoning is not logically self-consistent across prompt variations.

Prompt framing sensitivity. In multiple cases (Figures 43, 59), differently worded versions of the same underlying question elicit opposite answers. The taxi fare example (Figure 43) asks both "Should I actually pay $29.42?" and "Should I actually pay$32.42?" using the same receipt image. GPT-4V correctly handles both, identifying that the actual amount is $29.42 and noting a minor calculation discrepancy. Gemini, however, produces contradictory yes/no answers to the two questions — the model's response is driven by the suggestion in the prompt rather than independent verification of the receipt contents.

Qualitative Pattern 3: Domain-Specific Performance Boundaries

Medical diagnosis (Section 5.3) reveals a sharp capability cliff. When the paper includes ground-truth radiology reports for four chest X-ray images (Figures 94-97), the comparison between model outputs and professional diagnoses reveals substantial gaps. For the first two X-rays (Figures 94-95), which show relatively clear abnormalities (pleural effusion, pulmonary masses), GPT-4V produces reports that identify the key findings correctly, though with different specificity than the ground-truth radiologist reports. Gemini identifies abnormalities but with less precision — in Figure 94, Gemini correctly notes a "large left-sided pleural effusion" while the ground truth reports "unchanged loculated right pleural fusion" (a different location and characterization). For the two X-rays with complex or subtle findings (Figures 96-97), both models tend to produce "no significant abnormality" judgments that miss the actual conditions (pulmonary fibrosis, emphysematous changes). This pattern suggests that current MLLMs can identify gross abnormalities but lack the fine-grained diagnostic precision needed for clinical deployment — a boundary that is only visible because the paper includes ground-truth comparisons.

Autonomous driving (Section 5.1) shows good high-level understanding but poor fine-grained recognition. All three models correctly identify basic scene elements (traffic lights, weather, pedestrians) and make appropriate high-level driving decisions (Figures 89-90). However, for "small and low-resolution patterns in traffic or road signs, the three models are struggling to precisely recognize them" (Section 5.1). In Figure 87, Gemini misinterprets a lane marking as indicating a mandatory right turn when the traffic light is green and the lane is marked for straight travel. In Figure 88, Gemini identifies four traffic signs where only two exist, fabricating additional signs with specific meanings. This pattern — good global scene understanding with poor local detail precision — is a deployment-relevant limitation: autonomous driving systems must be reliable at both scales.

Defect detection (Section 5.2) reveals inconsistent sensitivity. For obvious defects (fabric holes in Figure 91, broken rings in Figure 91 right), all three models succeed. For a subtle defect — a small hole in a plastic cup (Figure 93) — Gemini detects something unusual but misclassifies it as condensation, while GPT-4V and Sphinx find no abnormalities. This sensitivity boundary is practically significant for industrial quality inspection applications, where missing subtle defects constitutes a critical failure mode.

Table and chart reasoning (Section 3.1) shows fragility to OCR errors. In the cloud computing market size question (Figure 41), GPT-4V correctly extracts values from the bar chart and performs the calculation, while Gemini provides a single incorrect number. In the paid penetration rate question (same figure), GPT-4V performs a detailed step-by-step estimation but ultimately reaches the wrong conclusion because its OCR-extracted values are imprecise — a failure of perception cascading into reasoning error. This pattern — correct reasoning on incorrect input — is a subtle but important failure mode: the model's reasoning can appear sound while producing wrong answers, making errors difficult to detect without independent verification of the perceptual inputs.

Ablation Studies and Robustness Checks

Prompt technique variation (not systematically ablated, but qualitatively compared across samples). The paper uses four prompt techniques (simple instruction, visual referring, Chain-of-Thought, in-context few-shot) across different samples but does not systematically test each technique on the same set of tasks to measure the effect of prompt choice on accuracy. The closest to an ablation is the Sudoku experiment (Figure 72), where the same puzzle is presented with and without the text grid: Gemini attempts to solve from the image alone but "cannot correctly identify the blank position," while both Gemini and GPT-4V solve correctly when given the text grid. This demonstrates that OCR is the bottleneck, not reasoning, for this task. The CoT comparison for Sphinx on mathematical problems (Section 3.3, Figures 54-59) shows that CoT prompting can "rectify" previously incorrect answers — the paper notes this in Figure 58 where Sphinx's CoT-prompted answer of 1 matches the correct answer, while its non-CoT answer of 11 is wrong. However, this is an isolated observation rather than a systematic ablation, and no aggregate statistics on CoT effectiveness are reported.

Few-shot spatial reasoning teaching (Figure 2). The paper provides two in-context examples teaching the concept of "left-hand side" and then tests all three models. GPT-4V successfully learns from the examples and correctly answers the target question. Gemini and Sphinx fail to generalize. This functions as an informal ablation of in-context learning capability for spatial concepts: GPT-4V can acquire a spatial reasoning skill from demonstrations, while Gemini (despite its competitive overall performance) cannot. The paper does not explore whether different numbers of examples, different example formats, or different spatial concepts would change this result.

Emotion-conditioned output (Figures 70-71) as a hallucination probe. When asked to describe the same image "in a romantic way" versus "in a terrifying way," both Gemini and GPT-4V successfully inject the specified emotional tone but hallucinate non-existent details — the "bike" and "shadows" in Figure 70, the "distant sound" and "malevolent force" in Figure 71. Sphinx, while less emotionally nuanced, does not fabricate these details. This functions as an informal robustness check: adding emotional conditioning to the prompt increases hallucination in the proprietary models but not in the open-source model, suggesting that the mechanisms enabling rich emotional language generation also increase the tendency to confabulate supporting details.

Cross-domain knowledge generalization (Figures 37-38). For artwork recognition and cultural artifact description, Gemini demonstrates "a broader range of learned knowledge and generalization capabilities" (Section 7.2). In Figure 37, Gemini correctly identifies a Chinese bronze ding vessel and a Sanxingdui gold mask, providing specific historical and cultural context. GPT-4V provides accurate descriptions but with less specific cultural identification. In Figure 38, Gemini identifies the painting as "A Riverside Scene at Qingming Festival" by Zhang Zeduan, providing the specific title, artist, dynasty, and date. GPT-4V describes the painting's visual elements accurately but does not name it. This pattern — Gemini outperforming GPT-4V on culturally specific knowledge — is consistent across the multicultural customs (Figures 20-22) and artwork recognition (Figures 36-38) samples, suggesting that Gemini's training data includes more comprehensive coverage of non-Western cultural content. The paper does not systematically quantify this advantage or test whether it extends beyond the selected examples.

Optical illusion resistance (Figures 6-7) as an implicit robustness check. The optical illusion tasks test a different aspect of robustness: whether models share human perceptual biases or operate on different principles. In Figure 6, Gemini correctly states that "both pears have the same brightness" (the objective truth), while GPT-4V and Sphinx, like most humans, perceive the right pear as brighter. However, in the same figure, GPT-4V uniquely identifies the visual similarity between tree branches and human arms — showing a different kind of perceptual analogy-making. In Figure 7, only Gemini recognizes the impossible dice arrangement as an optical illusion, while GPT-4V provides a lengthy analysis of how such a structure might or might not be possible in reality. These examples suggest that the models have different "perceptual priors" — Gemini is more resistant to some illusions but may miss others, while GPT-4V exhibits more human-like perceptual processing — but the paper presents only a handful of examples, making generalization impossible.

Critical Assessment

Do the experimental results support the claim that Gemini is a "strong challenger to GPT-4V's leading position"? The quantitative evidence (MME benchmark, Table 1) shows Gemini at 1933.4 and GPT-4V at 1926.6 — a difference of 6.8 points out of a total exceeding 1900. Without confidence intervals or statistical tests, this margin is consistent with the two models being essentially tied, with the observed difference attributable to GPT-4V's policy-driven zero score on Celebrity recognition. The paper's qualitative evidence demonstrates that Gemini and GPT-4V exhibit different answering styles (concise vs. verbose), different policy behaviors (direct vs. privacy-conscious), and different domain-specific strengths (Gemini better on cultural knowledge, GPT-4V better on code reasoning and detailed description), but does not establish one model as consistently superior. The claim of Gemini being a "strong challenger" is supported in the narrow sense that Gemini achieves performance in the same tier as GPT-4V rather than a tier below, and that for users who prefer concise answers or need celebrity identification, Gemini may be preferable. The claim is not supported for users who need detailed reasoning, code generation, or privacy-protective behavior. The experimental results more accurately support the conclusion that Gemini and GPT-4V are comparable but differently-profiled rather than that one challenges the other's leading position.

Do the experimental results support the claim that Sphinx "still trails behind them concerning domain generalizability"? Yes, with strong supporting evidence. The MME benchmark (Table 1) shows Sphinx's overall score (1870.2) below both proprietary models, with particularly large gaps in cognition sub-tasks: Code Reasoning (50.0 vs. GPT-4V's 170.0), Numerical Calculation (55.0 vs. GPT-4V's 130.0), OCR (87.5 vs. both at 185.0). Qualitatively, Sphinx fails at tasks requiring specialized knowledge (HTML generation in Figures 46-47, economic analysis in Figures 98-99, abstract visual reasoning in Figures 50, 52), misidentifies celebrities and landmarks, and cannot follow task formats for spatial output tasks (object tracking in Figure 82). However, the experimental evidence also demonstrates that Sphinx's gap is not uniform — on perception tasks using standard academic datasets, Sphinx is competitive or even superior (e.g., MME Existence: 195.0 vs. Gemini's 175.0), suggesting that the "trailing" is primarily in cognition, specialized knowledge, and task format flexibility rather than in basic visual perception. The paper's claim about "domain generalizability" is therefore well-supported but would benefit from acknowledging that the capability gap is domain-specific.

Do the experimental results support the claim about four common issues (spatial perception, OCR, logical self-consistency, prompt robustness)? Partially. The evidence for spatial perception deficits is convincing and multi-modal: quantitative (MME Position scores of 90.0 and 95.0 for Gemini and GPT-4V vs. Sphinx's 153.3) and qualitative (Figures 1-2 showing all models failing left-right discrimination, even after few-shot teaching for Gemini and Sphinx). The evidence for OCR limitations is qualitative (Figures 41, 45) and supported by specific failure cases, but the paper does not measure OCR accuracy systematically — there is no comparison against a dedicated OCR baseline, and no quantification of character or digit error rates. The evidence for logical self-consistency violations is based on a handful of examples (Figures 43, 59, 62) where models produce intermediate reasoning that contradicts final answers or produce different answers under prompt variation. These are striking demonstrations but not systematic measurements — without a defined protocol for testing self-consistency (e.g., paraphrasing the same question multiple times and measuring answer agreement), the severity and prevalence of this issue cannot be assessed. The evidence for prompt robustness sensitivity is similarly anecdotal — Figures 43 and 59 show clear examples of prompt framing affecting answers, but the paper does not report how frequently this occurs across tasks or whether certain task types are more susceptible than others.

Genuine weaknesses in the experimental design:

1. No inter-annotator agreement or annotation protocol. All qualitative correctness judgments (green/red/yellow) are made by the paper's authors without described guidelines, independent verification, or agreement statistics. For subjective tasks like emotion analysis and scene description, different annotators might reasonably disagree about whether a response is "correct," "incomplete," or "hallucinating." This is a fundamental limitation for a paper whose primary contribution is qualitative evaluation.

2. No sample selection transparency. The qualitative samples were deliberately chosen to be illustrative, but the paper does not report how many total samples were collected, how many were discarded, or what selection criteria were applied. The risk of confirmation bias — selecting samples that support the paper's narrative about model differences — is substantial and unaddressed. Pre-registration of test cases, random sampling from defined distributions, or at minimum documentation of the selection process would strengthen credibility.

3. Single-generation evaluation. Each model is queried exactly once per sample. Given the stochastic nature of LLM generation (especially if sampling temperature is non-zero, which the paper does not report), a single query cannot distinguish systematic capability failures from unlucky draws. Evaluating multiple generations per sample and reporting consistency (e.g., pass@k metrics) would provide a more reliable capability estimate.

4. No specialized vision baselines. For tasks where MLLMs perform poorly — object detection (Figure 77), spatial localization (Figures 78-80), object tracking (Figure 82), OCR-intensive chart reading (Figure 41) — comparing against dedicated computer vision systems (YOLO for detection, Tesseract or specialized OCR for text, SORT for tracking) would contextualize whether the failures are inherent to the task difficulty or specific to the MLLM approach. Without such baselines, readers cannot assess whether MLLMs are approaching task-specific system performance or remain far behind.

5. MME results lack statistical characterization. The MME scores are reported as point estimates without variance estimates, confidence intervals, or significance tests. The narrow gap between Gemini (1933.4) and GPT-4V (1926.6) may be smaller than the test-retest variability of either model, making the ranking potentially unreliable. This is a concern for a paper whose quantitative headline is that Gemini "achieves the highest overall score."

6. No control for model version or decoding parameters. The specific API versions of Gemini Pro and GPT-4V are not reported. Decoding parameters (temperature, top-p, top-k, maximum output length) are not documented. These choices can substantially affect output quality, verbosity, and hallucination rates, and without them, the observed stylistic differences between Gemini and GPT-4V cannot be attributed to model design versus decoding configuration.

Experiments that would have strengthened the paper but were not run:

• Systematic CoT ablation: Apply CoT prompting to a fixed set of reasoning tasks for all three models and measure the accuracy delta relative to direct prompting. This would quantify the observation that CoT helps Sphinx more than the proprietary models.
• Consistency measurement: For a subset of tasks (especially the "Yes/No" questions on MME), prompt each model with paraphrased versions of the same question and measure answer consistency. This would quantify the prompt robustness issue.
• Multi-generation pass@k: Generate multiple responses per sample and measure whether the correct answer appears in any generation (pass@k) versus the single-generation accuracy. This would distinguish "the model knows the answer but doesn't always produce it" from "the model fundamentally lacks the capability."
• Fine-grained OCR accuracy measurement: On chart reading and LaTeX generation tasks, compare model outputs against ground-truth numerical values or LaTeX strings using edit distance or exact match metrics. This would quantify the OCR limitation.
• Domain-specific comparisons against expert baselines: For medical diagnosis, compare model outputs against radiologist reports using structured reporting criteria (e.g., presence/absence of specific findings). For autonomous driving, compare against a dedicated perception stack on the same images.

Conditional validity of claims:

• The claim that Gemini matches or exceeds GPT-4V holds primarily on tasks requiring broad cultural knowledge and concise factual answering, and is partially an artifact of GPT-4V's refusal policy on celebrity tasks. It does not hold for code generation (GPT-4V: 170.0 vs. Gemini: 85.0 on MME Code Reasoning) or for tasks benefiting from detailed perceptual analysis.
• The claim about common failure modes is supported by qualitative demonstration but lacks quantification — the prevalence, severity, and task-specificity of these failures remain unknown.
• The claim about Sphinx's competitive perception performance holds for standard academic dataset distributions but may not generalize to in-the-wild deployment data — the paper's examples of Sphinx failing on Internet-sourced images suggest distribution sensitivity that is not systematically measured.

6. Limitations and Trade-offs

Difficulty Estimation Cost Is Not Accounted For in the Efficiency Metric

The assumption or constraint. The paper's compute-optimal framework depends on accurately estimating each prompt's difficulty before deciding how to allocate the inference budget. The method for doing so — generating 2048 samples per question and scoring them with the PRM — is extraordinarily expensive. The authors acknowledge this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

This is not a minor omission. At 2048 samples per question, the difficulty estimation step alone consumes more compute than the largest test-time budgets studied in the paper (which range from 1 to 512 generations). The paper's headline result — a 4× improvement in compute efficiency over best-of-N — is computed after difficulty is already known, without amortizing the cost of learning it.

The consequence. In a realistic deployment, the total cost would be difficulty estimation plus strategy execution, and the estimation cost could dominate the entire budget. For a single question at the highest tested generation budget (256 or 512), the difficulty estimation (2048 samples) costs 4–8× more than the solution attempt itself. The 4× efficiency gain is therefore an upper bound on achievable efficiency, not a realized deployment gain. If difficulty estimation cost is amortized across many questions from the same distribution (e.g., by pre-computing difficulty bins for a fixed test set), the per-question cost can be reduced, but this requires the question distribution to be known in advance — a constraint that does not hold in open-ended deployment.

What evidence exists in the paper. The paper does not measure the actual cost of difficulty estimation or report total FLOPs including estimation. The 4× figure (Figures 4 and 8) compares compute-optimal scaling curves to best-of-N curves with difficulty assumed known. The paper acknowledges the gap (Section 3.2 and Section 8) but presents no experiments that close it. The predicted difficulty bins (using PRM scores instead of ground-truth correctness labels) remove the need for labeled data but do not reduce the sample cost — they still require 2048 generations per question.

Mitigation status. The paper does not attempt to address this limitation experimentally. It flags future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8) and suggests that "exploration-exploitation tradeoffs" in difficulty assessment are an important direction (Section 3.2). The limitation is transparently acknowledged but unresolved.

---

The Approach Provides No Benefit on the Hardest Problems — Test-Time Compute Does Not Create Capability

The assumption or constraint. The paper's compute-optimal framework assumes that the base model already produces correct solutions at some non-trivial rate for a given problem. Test-time compute (search, revisions, or their combination) can amplify this existing capability but cannot create it from nothing. The authors state this boundary condition clearly in Section 7:

"test-time compute can amplify existing capability but cannot create it. If the base model's pass@1 is near zero on a problem class, no amount of search or revision will help."

The consequence. Across all methods — search, revisions, and compute-optimal combinations — the hardest questions (difficulty bin 5) show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy remains at 1–3% for all search methods and all budget levels up to 256 generations. In Figure 7 (right), bin 5 accuracy is roughly 2–3% irrespective of the sequential-to-parallel ratio at a fixed budget of 128 generations. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling curve is essentially flat near 0–5% for revisions and PRM search, well below the ~14× larger model's performance on the same bin. This means the approach offers no path forward for problems that genuinely exceed the base model's training distribution or reasoning capability. For such problems, pretraining remains the only viable approach, and the FLOPs-matched comparison in Section 7 shows that the larger model substantially outperforms test-time compute on hard problems across all $R$ regimes (e.g., −37.2% relative disadvantage for hard problems under $R \gg 1$ with revisions, and −52.9% for PRM search).

What evidence exists in the paper. The difficulty-bin analysis across Sections 5 and 6 consistently shows bin 5 performance at floor levels. Figure 3 (search), Figure 7 (revisions), and Figure 9 (FLOPs-matched) all show bin 5 accuracy at ≤5% regardless of method or budget. The paper is transparent about this limitation and does not overclaim — the takeaway in Section 7 explicitly states that "on the hardest questions, no amount of test-time compute makes meaningful progress."

Mitigation status. The paper does not attempt to address this limitation, nor does it suggest a path forward. It correctly identifies that this is a fundamental bound rather than a methodological flaw: test-time compute amplifies existing capability but does not create new capability. The implication is that systems facing heterogeneous difficulty distributions should either accept this ceiling on hard problems or route them to larger models, hybrid systems, or human review.

---

Revisions and PRM Search Are Studied Independently, Not Combined

The assumption or constraint. The paper studies two complementary axes — PRM-guided search (Section 5) and iterative revisions (Section 6) — as independent mechanisms, but never combines them. Section 8 explicitly acknowledges this gap:

"we did not experiment with PRM tree-search techniques in combination with revisions."

This is significant because the two mechanisms have complementary strengths revealed by the paper's own difficulty analysis: revisions improve the proposal distribution (generating better candidates through sequential refinement), while PRM search improves candidate selection (finding the best among generated candidates through verifier-guided search). The revision model could serve as the proposal distribution within beam search, or the PRM could guide which revision branches to pursue.

The consequence. The reported performance represents a lower bound on what a fully integrated system could achieve. Both mechanisms hit performance ceilings individually — revisions plateau on hard problems (bin 4–5 performance near floor), search over-optimizes on easy problems (beam search degrades at high budgets on bins 1–2, Figure 3 right). Combining them could partially address these ceilings: a revision-enhanced proposal distribution might produce candidates within searchable reach of correctness even on harder problems, while verifier-guided selection could mitigate the correct-to-incorrect reversion problem documented in Section 6.1 (approximately 38% of correct answers get revised back to wrong answers). Without demonstrating this combination, the paper cannot claim to have found the true upper bound of test-time compute scaling.

What evidence exists in the paper. None — by the authors' admission, the experiment was not run. The paper provides separate scaling analyses for search (Figures 3–4) and revisions (Figures 6–8) but no joint analysis. The compute-optimal policy in each section optimizes over strategies within a single mechanism family, not across families.

Mitigation status. The paper acknowledges this as future work (Section 8) and provides the intellectual framework for combination (the proposal-verifier decomposition in Section 2). The difficulty-dependent analysis in the paper provides guidance for which combination might work — revisions for easy problems where sequential refinement helps, search for medium problems where exploration is needed — but no empirical validation is provided.

---

The ~14× Larger Model Baseline Is Not Compute-Optimally Trained and Uses Only Greedy Decoding

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than Chinchilla-optimal pretraining (Hoffmann et al., 2022) where both data and parameters are scaled equally. The authors acknowledge this:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the ~14× larger model uses only greedy decoding — no majority voting, no best-of-N, no verifier-guided search. This means the pretraining baseline is not only potentially suboptimal in its training recipe but also receives no test-time compute augmentation whatsoever, while the smaller model receives a full compute-optimal strategy.

The consequence. Both choices make the pretraining baseline weaker than it needs to be, and both bias the comparison in favor of test-time compute. A Chinchilla-optimal model trained with 14× more total FLOPs (scaling both parameters and data) would likely outperform a parameter-only-scaled model, making the pretraining baseline stronger. Giving the larger model even a modest test-time compute budget — say, best-of-8 with majority voting — would create a much stronger baseline that is never tested. The reported advantages of test-time compute over pretraining (e.g., +27.8% on easy questions at $R \ll 1$ for revisions) may shrink or reverse against a properly calibrated baseline. The paper's Figures 1 and 9 should therefore be interpreted as comparisons against a specific, relatively weak pretraining baseline rather than against the best achievable performance from scaling pretraining compute.

What evidence exists in the paper. The paper provides the comparison only against this specific baseline. No ablation tests the sensitivity of the FLOPs-matched results to the pretraining scaling strategy (parameter-only vs. Chinchilla-optimal) or to giving the larger model test-time compute. The $R = D_{\text{inference}} / D_{\text{pretrain}}$ analysis (Section 7) partially accounts for the inference cost differential but does not address the pretraining suboptimality.

Mitigation status. The paper acknowledges the limitation explicitly (Section 7) and frames it as scope for future work. It does not attempt to bound the magnitude of the potential bias. The results should be interpreted as demonstrating that test-time compute can substitute for pretraining compute in some regimes, not that it is always more efficient than pretraining — the paper itself makes this distinction in its conclusions, but the headline figures in Figure 1 could easily be misinterpreted without careful attention to the baseline's limitations.

---

Difficulty Bins Are Static and Coarse — No Dynamic or Continuous Allocation

The assumption or constraint. The compute-optimal framework discretizes prompt difficulty into five quintile bins and selects a single strategy per bin per budget level. This is a coarse discretization of a continuous space, and the bin assignment is static — once a question is assigned to a bin (based on the 2048-sample pass@1 estimate), the strategy is fixed for the entire inference budget. There is no mechanism for dynamically adjusting the strategy mid-computation based on intermediate signals.

The consequence. Within a single bin, there may be substantial difficulty heterogeneity — a question at the boundary between bin 3 and bin 4 would receive the same strategy as questions in the middle of bin 3, even though the optimal strategy might differ. The paper's own data show that strategy effectiveness can shift dramatically within a single difficulty range (e.g., beam search outperforms best-of-N on bin 3 problems but degrades on bin 2 problems, Figure 3 right). A question near the bin boundary may therefore receive a suboptimal strategy.

More importantly, the static bin assignment prevents adaptive allocation. An ideal system would begin with a small number of parallel samples, assess the verifier's score distribution as a real-time difficulty signal, and then allocate the remaining budget accordingly — switching between search, revisions, and parallel sampling based on the evolving evidence. Such a dynamic policy could subsume the difficulty estimation cost into the solution process itself, partially addressing the estimation cost limitation discussed above. The paper's exploration-exploitation framing in Section 3.2 explicitly acknowledges this possibility but does not explore it.

What evidence exists in the paper. The paper uses five bins throughout (Sections 3.2, 5.3, 6.2) and does not test sensitivity to the number of bins. The predicted difficulty bins (using PRM scores) track the oracle bins closely (Figures 4, 8), suggesting the discretization captures meaningful difficulty variation, but the paper does not compare coarser or finer binnings. No dynamic allocation experiments are reported.

Mitigation status. The paper does not attempt to address this limitation. It flags the exploration-exploitation tradeoff (Section 3.2) and suggests that dynamic difficulty assessment is a direction for future work. The coarse binning was a pragmatic choice given the small test set (500 questions, ~100 per bin) and the need to estimate per-bin optimal strategies from limited data, but the paper does not discuss what data requirements would be needed for finer-grained or continuous policies.

---

All Experiments Are on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*)

The assumption or constraint. Every experiment in the paper uses the MATH benchmark (Hendrycks et al., 2021) with PaLM 2-S* as the base model. The authors state (Section 4) that they "believe this model is representative of the capabilities of many contemporary LLMs," but this claim is unverified. MATH consists exclusively of competition-level math problems requiring symbolic reasoning and producing verifiable closed-form answers — a specific task profile that may not generalize to other reasoning domains.

The consequence. Several aspects of the paper's findings could be domain-specific or model-specific:

• The PRM's quality and over-optimization behavior (Section 5.3) depend on PaLM 2-S*'s output distribution and error patterns. A model with different calibration or different typical mistake types might exhibit different difficulty-dependent scaling curves.
• The revision model's ability to learn from incorrect-to-correct trajectories (Section 6.1) depends on the base model's in-context learning capabilities, which vary substantially across model families.
• The finding that beam search over-optimizes on easy problems (Figure 3, right) might be specific to the PRM trained on PaLM 2-S* outputs. A better-calibrated PRM on a different base model might not exhibit this behavior.
• MATH problems have unambiguous correct answers that can be verified with string matching, enabling both the PRM training pipeline (Monte Carlo rollout correctness labels) and the difficulty estimation (pass@1 computation). Many important real-world tasks — open-ended generation, multi-step planning, dialogue, subjective evaluation — lack such clean verification signals, making the entire framework harder to adapt.

The paper does not test whether the difficulty-dependent strategy patterns (beam search for medium problems, revisions for easy problems, neither for hard problems) hold for code generation, logical reasoning, scientific QA, or other reasoning domains.

What evidence exists in the paper. None — all experiments use MATH with PaLM 2-S* (and the ~14× scaled variant for the FLOPs-matched comparison). The paper does not include any cross-domain or cross-model replication. The MME benchmark results cited in the related work section are not replicated or extended with compute-optimal strategies.

Mitigation status. The paper does not claim cross-domain generalization and is transparent about the specific benchmark and model used. The authors present the work as an initial systematic study establishing the framework, not as a comprehensive validation across domains. Section 8 suggests extending the analysis to "other domains and model families" as future work. The limitation is significant for practitioners considering applying these methods to their own domains — the 4× efficiency claim should be treated as MATH-specific until replicated elsewhere.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper causes a methodological shift in how the MLLM community approaches evaluation, moving from a paradigm where benchmark scores serve as the primary (and often sole) evidence of model capability toward a complementary paradigm where deliberately constructed, adversarial, and difficulty-varied qualitative probes are treated as equally essential diagnostic instruments. This is not a paradigm shift in the Kuhnian sense — the paper does not replace quantitative benchmarking — but rather a re-framing of evaluation as a two-component process: standardized benchmarks establish aggregate ranking, while qualitative probing reveals behavioral patterns, failure boundaries, and model "personalities" that benchmarks inherently obscure.

The magnitude of this shift is incremental but practically significant. Prior to this work, the distinction between "GPT-4V scores X on benchmark Y" and "GPT-4V fabricates details when pushed to comply with impossible requests" would have been understood as belonging to entirely different categories of evidence — quantitative evaluation versus anecdotal observation. The paper's contribution is to systematize the latter, demonstrating that carefully designed probes can reveal stable, cross-task behavioral tendencies (the verbosity-conciseness axis, the privacy-refusal policy, the hallucination-under-pressure pattern) that are as diagnostic of model quality as benchmark scores, and that these tendencies can be compared systematically across models. The paper effectively argues, through its structure and thoroughness, that qualitative evaluation deserves the same rigor, coverage, and systematization that the field has long demanded of quantitative benchmarks.

The paper reconciles a subtle but important tension in the prior literature. Previous qualitative explorations of MLLMs — notably "The Dawn of LMMs" (Yang et al., 2023) for GPT-4V and Google's Gemini technical report for their own model — provided rich demonstrations but lacked comparative framing. A reader of both reports might conclude that each model excels at everything, since each report selectively highlighted impressive behaviors. The paper resolves this by showing that both models exhibit clear, systematic weaknesses — spatial reasoning, OCR precision, logical self-consistency, prompt robustness — and that these weaknesses are remarkably similar across independently developed systems. The contradiction was not in the models' capabilities but in the evaluation approach: single-model demonstrations cannot reveal shared failure modes.

Which research directions become more attractive. The paper's methodology makes several lines of inquiry newly tractable:

• Systematic behavioral phenotyping of MLLMs: The paper's prompting taxonomy (simple instruction, visual referring, CoT, few-shot) and adversarial probe design (impossible requests, framing variations, perceptual illusions) provides a replicable template for characterizing any new model's behavioral profile. Subsequent work can apply this template to emerging models, building a longitudinal record of how model personalities evolve.

• Understanding the relationship between training data and behavioral tendencies: The paper's finding that Sphinx excels on perception tasks using academic dataset distributions but fails on Internet-sourced or specialized images directly motivates research on how training data composition shapes model behavior. A study that systematically varies training data sources and measures the resulting capability profile would be highly informative.

• Developing benchmarks that include adversarial probes: The paper demonstrates that impossible requests, framing variations, and illusion tasks reveal behaviors invisible to standard benchmarks. A natural next step is to incorporate such probes into standardized evaluation suites, creating benchmarks that measure not just accuracy but also compliance bias, hallucination propensity, and prompt sensitivity.

Which research directions become less attractive. The paper's evidence indirectly argues against:

• Purely aggregate benchmarking as sufficient evaluation: If two models can achieve nearly identical MME scores (Gemini: 1933.4, GPT-4V: 1926.6) while exhibiting markedly different behavioral profiles, then relying solely on benchmark rankings for model selection is insufficient. Research that proposes new benchmarks without accompanying qualitative characterization is incomplete.

• Assuming that scaling resolves fundamental limitations: The shared failure modes across Gemini Pro and GPT-4V — two of the largest, most capable MLLMs ever built, with different architectures and training procedures — suggest that spatial reasoning, OCR precision, and logical consistency are not simply functions of scale. Research premised on "bigger models will fix this" needs to contend with the paper's evidence that some failures persist across the current scaling frontier.

---

Follow-Up Research This Work Enables

Systematic quantification of the prompt sensitivity problem. The paper demonstrates qualitatively that both Gemini and GPT-4V can produce opposite answers to differently-framed versions of the same question (Figures 43, 59), but provides only a handful of examples. A strong follow-up would define a protocol for measuring prompt sensitivity: take a fixed set of questions with unambiguous correct answers (e.g., arithmetic problems from the MME Numerical Calculation sub-task, or factual questions with image support), generate multiple paraphrased prompt variants per question (varying phrasing while preserving the underlying task and correct answer), query each model with each variant, and measure answer consistency (the fraction of variant pairs that produce the same answer) and accuracy conditional on consistency. This would quantify the severity of the prompt robustness issue that the paper identifies qualitatively, and would reveal whether certain task types (e.g., yes/no questions vs. open-ended generation) are more susceptible than others. The paper's MME results provide a natural testbed, since the MME's dual-question design (one "yes" question and one "no" question per image) already partially probes this — but the paper reports only aggregate scores, not per-image consistency between the two questions.

Measuring hallucination rates under adversarial prompting using the paper's probe template. The paper's most diagnostic finding about model behavior — that both Gemini and GPT-4V fabricate details when given impossible task specifications (Figure 5) or emotionally-loaded instructions (Figures 70-71) — is demonstrated with a small number of hand-picked examples. A rigorous follow-up would construct a systematic test suite of adversarial prompts across multiple task categories: impossible requests (e.g., "Find 5 differences" for images with N differences where N < 5), emotion-conditioned descriptions (prompting for the same image with neutral, romantic, terrifying, and clinical framings), and leading questions that suggest incorrect answers (as in Figure 59). For each prompt type, human annotators (or automated metrics where applicable, e.g., object presence verification for hallucinated visual elements) would measure the rate of fabricated details. The key measurement would be the hallucination compliance ratio: the probability that a model fabricates information to satisfy a prompt demand versus the probability that it correctly identifies the demand as impossible or inappropriate. This metric would be directly comparable across models and over time as new versions are released. The paper already provides the qualitative template; the follow-up adds systematization and quantification.

Cross-model, cross-domain replication of the difficulty-dependent behavioral patterns. The paper's four-domain taxonomy (fundamental perception, advanced cognition, challenging vision tasks, expert capacity) implicitly encodes a difficulty hierarchy, and the qualitative results suggest that model rankings shift across this hierarchy — Sphinx competes on perception but lags on cognition, Gemini shows broader domain knowledge in expert tasks, GPT-4V excels at detailed perceptual description. However, these patterns are observed on a single, curated sample set with no statistical characterization. A strong replication study would take the paper's domain taxonomy and, for each domain, construct a larger, independently-curated test set (e.g., 100-200 samples per domain, sourced to avoid leakage using the paper's post-November-2023 timestamp criterion) and evaluate a broader set of models (including newer releases like GPT-4 Turbo with vision, Gemini Ultra, Claude 3, and open-source models beyond Sphinx). The key measurement would be the domain-specific capability profile for each model — a vector of per-domain accuracies with confidence intervals — and the degree to which profiles cluster (do all proprietary models share a similar shape? do all open-source models share a different shape?). This would test the paper's implicit claim that model "personality" and domain specialization are stable properties worth measuring.

Training data attribution analysis for Sphinx's domain-specific weaknesses. The paper identifies specific domains where Sphinx fails — economic analysis (Section 5.4), HTML code generation (Section 3.1), abstract visual reasoning (Section 3.2) — and attributes these failures to "the diversity of Sphinx's training data [being] still lacking in some domains" (Section 7.3). This hypothesis is testable: a follow-up study could curate targeted training datasets for these specific domains (financial charts with Q&A pairs, webpage screenshots with corresponding HTML, abstract reasoning puzzles with solutions), fine-tune Sphinx on this data, and measure whether the domain-specific gap closes relative to Gemini and GPT-4V. If the gap closes substantially, the attribution to training data is validated, and the implication is that open-source MLLM development should prioritize data diversity. If the gap persists despite targeted training, the limitation is architectural (base LLM reasoning capability) rather than data-driven, and the implication is that scaling the base LLM is necessary. This experiment would directly inform open-source MLLM development priorities.

Structured comparison of MLLM outputs against specialized vision system baselines. The paper shows that MLLMs perform poorly on tasks that dedicated vision systems handle well — bounding box prediction (Figure 77, where "the bounding boxes provided by Gemini are generally imprecise" and GPT-4V avoids the task entirely), object tracking (Figure 82, where both leading models produce incorrect bounding boxes), OCR-intensive chart reading (Figure 41). However, no direct comparison against specialized systems is made. A rigorous follow-up would take a subset of the paper's vision tasks (object detection, object tracking, OCR from charts, facial recognition) and compare MLLM outputs against dedicated systems (YOLO or DETR for detection, SORT or DeepSORT for tracking, Tesseract or commercial OCR for text, FaceNet or similar for recognition) on the same images. The key measurement would be the capability ratio: MLLM accuracy divided by specialized system accuracy, computed per task. This would reveal which tasks are within striking distance for general-purpose MLLMs (where the ratio is close to 1) and which require fundamentally different approaches (where the ratio is far below 1). The paper's qualitative examples already suggest that spatial precision tasks will have low ratios, but quantification would guide decisions about whether to use MLLMs as universal vision systems or to maintain specialized pipelines for precision-critical tasks.

Longitudinal tracking of the four common issues across model generations. The paper identifies four "common issues" — spatial perception, OCR/abstract understanding, logical self-consistency, prompt robustness — as shared limitations of current frontier MLLMs. A natural longitudinal study would evaluate each new major MLLM release (GPT-4V → GPT-5 or equivalent, Gemini Pro → Gemini Ultra → future versions, new open-source models) on a fixed test suite designed to probe these four issues, using the paper's prompt techniques (spatial relation questions from Figure 1 for spatial perception, chart reading from Figure 41 for OCR, prompt framing variations from Figures 43/59 for self-consistency and robustness). This would track whether these limitations are resolved by scaling, architectural improvements, or better training data — or whether they represent more fundamental constraints of the current MLLM paradigm. The paper's methodology provides the template; the follow-up adds temporal tracking and statistical characterization. The key insight would be whether any of the four issues proves to be "sticky" — resistant to improvement across multiple model generations — which would indicate a fundamental rather than transient limitation.

---

Practical Applications and Downstream Use Cases

Model selection for domain-specific deployment based on behavioral profiling. Organizations choosing between GPT-4V and Gemini for integration into products can use the paper's behavioral characterization as a decision framework that goes beyond benchmark scores. For applications requiring detailed, step-by-step explanations with explicit reasoning (e.g., educational tools, medical report generation where audit trails matter, legal document analysis), GPT-4V's tendency to "elaborate detailed explanations and intermediate steps" (Section 7.2) and its more "dialectical thinking with rigorous analysis" in emotion tasks (Section 3.4) make it the stronger candidate — the paper provides evidence across scene description (Figures 8-10), table reasoning (Figure 41), emotion analysis (Figures 65-69), and radiology reporting (Figures 94-97) that this verbosity translates to more comprehensive coverage of visual details. For applications requiring rapid, direct answers with minimal latency and concise output (e.g., voice assistants, real-time navigation guidance, quick landmark identification), Gemini's preference for "direct and concise response" offers better user experience — the paper shows Gemini providing correct answers in single sentences across object counting (Figure 3), landmark recognition (Figures 25-26), and celebrity identification (Figures 23-24). For applications involving person identification or culturally-specific knowledge (e.g., art history tools, cultural heritage applications, celebrity news aggregation), Gemini's willingness to identify individuals and its broader coverage of non-Western cultural content (Figures 37-38, Figures 20-22 for multicultural customs) make it functionally capable where GPT-4V refuses on policy grounds (MME Celebrity score: 0 for GPT-4V) or provides less specific cultural context. This is not a theoretical preference — the paper provides concrete failure cases that would directly impact these applications.

Open-source MLLM development prioritization guided by the Sphinx capability gap analysis. The paper's systematic comparison of Sphinx against Gemini and GPT-4V across four domains provides a detailed gap analysis that can directly inform open-source development roadmaps. The finding that Sphinx is competitive on perception tasks using standard academic datasets (MME Existence: 195.0 vs. GPT-4V's 190.0; Position: 153.3 vs. GPT-4V's 95.0) but lags severely on cognition (MME Code Reasoning: 50.0 vs. GPT-4V's 170.0; Numerical Calculation: 55.0 vs. GPT-4V's 130.0) and specialized domain knowledge (economic analysis failure in Figures 98-99, HTML generation failure in Figures 46-47, abstract reasoning failure in Figures 50-52) suggests a specific investment strategy: (1) improving the base LLM backbone (from LLaMA-2-7B toward 70B-scale models) to address the cognition gap, which the paper attributes to "the inherent reasoning upper bound of the underlying LLM" (Section 7.3); (2) expanding training data diversity to include specialized domains (financial charts, webpage code, abstract reasoning puzzles, scientific diagrams) to address the domain generalization gap; and (3) continuing to invest in perception training since Sphinx already approaches frontier performance on standard distributions. The paper provides the diagnostic evidence for this triage — without it, open-source developers might invest equally across all fronts, diluting impact.

Evaluation protocol design for internal model testing at AI development organizations. Companies developing MLLMs internally can adopt the paper's adversarial prompting methodology as part of their pre-release testing pipeline. Specifically, the paper identifies prompt types that reveal behaviors invisible to standard evaluation: impossible requests (Figure 5) reveal hallucination-under-pressure tendencies, prompt framing variations (Figures 43, 59) reveal logical self-consistency failures, optical illusions (Figures 6-7) reveal whether perceptual processing is human-like or follows different principles, emotion-conditioned descriptions (Figures 70-71) reveal whether rich generative capabilities increase fabrication risk, and deliberately leading questions test whether the model is overly compliant to user suggestions. Integrating these probe types into an internal red-teaming or quality assurance process would catch behavioral issues before deployment — the paper effectively provides a taxonomy of adversarial prompt categories, each with demonstrated diagnostic value across multiple models. The cost of implementing such a testing suite is low (the probes are primarily textual prompts paired with curated images), and the benefit is catching deployment-risky behaviors that standard accuracy benchmarks would miss entirely. The paper's finding that both leading models share these vulnerabilities despite different architectures and training procedures suggests the probes are robustly diagnostic rather than model-specific.

---

When to Prefer This Method

The paper does not position a specific "method" against named alternatives in the traditional sense — it is an evaluation and comparison study, not a proposal of a new technique to be preferred over existing ones. The question of "when to prefer this method" therefore applies to the evaluation methodology itself (as a template for how to assess MLLMs) rather than to a model or algorithm.

For evaluation methodology, the paper's approach is most appropriate when:

• The goal is deployment decision-making, not just leaderboard ranking. The qualitative behavioral profiling reveals model "personalities" (verbosity, refusal policies, hallucination tendencies) that matter for user experience but are invisible to benchmark scores. When choosing between models for integration into a product, this profiling is as important as accuracy — and the paper's methodology provides a template for generating it.

• New frontier models are released with limited independent documentation. The paper's value proposition — third-party, comprehensive, multi-domain qualitative comparison — is most relevant precisely when proprietary model developers have not provided adequate independent evaluation. As new models emerge (Gemini Ultra, GPT-5, Claude 3, etc.), applying this evaluation template early provides the community with capability characterizations that accelerate adoption and identify risks.

• The task domain spans multiple capability types (perception, reasoning, specialized knowledge). The paper's four-domain taxonomy provides a structured way to assess whether a model's strengths align with application requirements. A medical imaging company evaluating MLLMs needs to know not just aggregate accuracy but specifically how the model performs on expert capacity tasks (Section 5.3) and whether it exhibits the hallucination or compliance-bias patterns documented in Sections 2 and 3.

For model selection between Gemini Pro and GPT-4V specifically, the paper's evidence supports the following decision rules grounded in its findings:

• Prefer GPT-4V when the application requires detailed, step-by-step reasoning with explicit intermediate steps and comprehensive visual detail enumeration. This preference is supported by GPT-4V's demonstrated behavior across scene description (Figures 8-10: more detailed, fewer hallucinations), table and chart reasoning (Figure 41: explicit calculation steps), emotion analysis (Figures 65-69: dialectical thinking with multiple perspectives), and code reasoning (MME Code Reasoning: 170.0 vs. Gemini's 85.0). However, be aware that GPT-4V may refuse to identify real people (MME Celebrity: 0, Figures 24, 29, 81) and may decline certain out-of-scope tasks by anticipating its own limitations (Figures 31, 77).

• Prefer Gemini when the application requires direct, concise answers with broad domain generalizability, or involves tasks that GPT-4V refuses on policy grounds (celebrity identification, person recognition). This preference is supported by Gemini's demonstrated behavior across landmark recognition (Figures 25-26: correct identifications with cultural context), artwork recognition (Figures 37-38: specific titles, artists, dates), multicultural content understanding (Figures 20-22), and its more balanced MME profile without zero-score categories. However, be aware that Gemini's conciseness trades off against detail coverage in complex visual scenes (Figures 8-9: shorter, less comprehensive descriptions than GPT-4V) and that Gemini may hallucinate more under adversarial prompts (Figure 43: contradictory answers to differently-framed questions).

• Prefer Sphinx when the application is perception-heavy and cost-sensitive, and the task domain aligns with standard academic data distributions. This preference is supported by Sphinx's competitive MME perception scores (Existence: 195.0, Count: 160.0, Position: 153.3) and its much lower computational cost as a 7B-parameter open-source model. However, be aware that Sphinx's performance degrades substantially on cognition tasks (MME Code Reasoning: 50.0, Numerical Calculation: 55.0), specialized domains (economic analysis failure, HTML generation failure), and tasks requiring abstract reasoning (Raven's Progressive Matrices failure in Figures 52-53).