Gemini vs GPT-4V: A Preliminary Comparison and Combination of Vision-Language Models Through Qualitative Cases

ArXiv: 2312.15011

🎯 Pitch

Google’s Gemini and OpenAI’s GPT-4V might seem neck-and-neck in vision-language tasks, but this paper reveals a critical asymmetry: Gemini Pro consistently stumbles on sequential multi-image reasoning and memory tasks, where GPT-4V shines. The authors further show these two models’ strengths are complementary, and simply combining their raw outputs yields answers more accurate than either can achieve alone.

---

1. Executive Summary

This paper conducts a qualitative, multi-faceted comparative study of two pioneering multimodal large language models—Google's Gemini and OpenAI's GPT-4V(ision)—across diverse visual understanding and reasoning tasks using a sample set derived from the prior "Dawn of LMMs" exploration. The analysis evaluates performance along dimensions such as Vision-Language Capability, Temporal Understanding, and Intelligence and Emotional Quotients, examining specific behaviors like Gemini's detailed, expansive answers with supplementary imagery and links versus GPT-4V's precision and succinctness. It finds that while both models perform comparably on basic recognition, GPT-4V slightly outperforms Gemini Pro overall, with Gemini faltering particularly in tasks requiring multi-image input and sequential memory—establishing that the models possess complementary strengths but that Gemini's single-image, non-memory input mode constitutes a fundamental limitation in integrated, temporal, and interactive applications.

2. Context and Motivation

The Core Problem: No Systematic Qualitative Comparison of Frontier MLLMs Exists

The paper addresses a straightforward but practically significant gap: when two major competing multimodal large language models (MLLMs) are released within months of each other, the research community lacks a systematic, side-by-side qualitative understanding of their respective strengths, weaknesses, and behavioral differences. The authors identify this gap explicitly in Section 1.1, framing the release of Gemini as a direct challenge to GPT-4V's "dominant position" in the MLLM landscape and noting that comprehensive comparative analysis is absent.

This gap matters for several reasons the paper implicitly raises:

• Model selection in practice: Developers, researchers, and organizations deciding which model to adopt for specific applications need more than benchmark scores—they need to understand how each model behaves on real tasks, what failure modes exist, and where each model's design constraints impose practical limitations. A model that excels at creative storytelling but cannot process multiple images sequentially has fundamentally different deployment suitability than one with the inverse profile.

• Understanding design tradeoffs: The two models embody different architectural and interface choices—Gemini's single-image input with web-linked supplementary outputs versus GPT-4V's multi-image sequential processing with text-only responses. A thorough comparison reveals which design decisions matter most for which downstream tasks, informing the next generation of MLLM development.

• Reconciling benchmark claims with practical behavior: Both models are accompanied by technical reports making performance claims (Gemini's report asserting near-parity with GPT-4V on various benchmarks). Independent qualitative evaluation provides a check on whether these quantitative claims translate to observable capability differences in realistic usage scenarios.

The Historical Moment: A Rapidly Shifting Frontier Model Landscape

The paper's timing is significant. As described in Section 1.1, the MLLM field was dominated by GPT-4V (released September 2023) until Google introduced Gemini (released December 2023), creating a new competitive dynamic between two closed-source, proprietary models from the world's leading AI labs. Unlike the open-source MLLM ecosystem (LLaVA, MiniGPT-4, InstructBLIP), where model internals are accessible and systematic benchmarking is more feasible, closed-source models can only be evaluated through their public interfaces. This makes qualitative, case-based comparison studies particularly valuable—they are often the only window into model behavior available to the broader community.

The paper positions itself explicitly as a continuation and expansion of the qualitative exploration begun by Yang et al. (2023) in their "Dawn of LMMs" study of GPT-4V. The authors state directly in Section 1.1:

"It should be noted that the image samples, prompts, and results related to GPT-4V used in our paper are referenced from the study 'The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision).' Our work can be seen as a continuation and expansion of this previous research."

This framing matters: the paper does not attempt to build a new evaluation benchmark or dataset from scratch. Instead, it reuses an established, community-vetted set of qualitative probes to test Gemini under the same conditions GPT-4V was previously evaluated under, enabling direct comparison. The methodological contribution is not in new evaluation design but in the systematic, multi-dimensional application of an existing evaluation framework to a new model, with careful attention to prompt adaptation (Section 1.3) and input modality differences.

The Fundamental Contrast in Model Architecture and Interface Design

One of the paper's key motivating observations—developed throughout Section 1.2 but threaded through every subsequent section—is that Gemini and GPT-4V are not comparable in a straightforward "same input, same output" sense. Their interface differences are deep and consequential:

Gemini's input constraints (Section 1.2):

• Single-image limitation: Gemini cannot natively accept multiple independent images as sequential context. The paper's workaround is to "combine all the images into a single image input" for multi-image tasks, which fundamentally changes the task—the model must parse image boundaries, understand spatial arrangement of sub-images, and handle reduced effective resolution per sub-image.
• No persistent visual memory: Unlike GPT-4V, which can retain context from previous images in a conversation, Gemini's multimodal module "lacks the ability to retain memory of past image inputs and outputs."
• Sensitive information masking: The paper reports that Gemini exhibits obfuscation behavior with facial or medical imagery, refusing to process such inputs—a constraint not observed in GPT-4V.

Gemini's distinctive output capabilities:

• Image and link generation: Gemini can output images related to the content and provide corresponding web links, establishing "a higher level of association similar to search engine functionality." This is a genuinely different capability from GPT-4V's text-only responses and significantly shapes user experience in tasks like product recommendation (Section 10.2, 11.1) and landmark recognition (Section 2.2).
• Verbose, detailed responses: A recurring pattern throughout the paper is that Gemini provides substantially longer, more elaborate answers than GPT-4V's concise responses—even when not explicitly asked for detail.

These differences are not incidental—they constitute the primary structural explanation for the performance patterns the paper documents. Gemini's difficulty with multi-image tasks (interleaved image-text reasoning in Section 6.1, temporal ordering in Section 8.2), its struggles in embodied agent scenarios requiring sequential visual context (Section 10.6), and its reduced performance on Raven's Progressive Matrices requiring comparison across multiple figures (Section 4.5) all trace back to the single-image, non-memory input constraint. Conversely, its strengths in detailed description, web-linked recommendations, and creative generation (Section 11.2) are direct products of its extended output capabilities.

This makes the paper's motivation fundamentally about characterizing the practical implications of architectural design choices rather than simply ranking models. The comparison reveals not just "which model is better" but "which design tradeoffs produce which affordances and limitations."

The Challenge of Fair Comparison Under Asymmetric Interfaces

A methodological tension runs throughout the paper: how do you compare models fairly when one fundamentally cannot perform the task as designed? The paper's approach is pragmatic rather than purist. As described in Sections 1.2 and 1.3:

• For multi-image tasks designed for GPT-4V's sequential input capability, the paper modifies the input format for Gemini rather than excluding those tasks. Images are stitched together into a single composite input, and prompts are adjusted "to align with the input requirements of its architecture" (Section 1.3).
• When Gemini fails on composite inputs (e.g., Fig. 46, calculating tax from three receipts), the paper sometimes further decomposes the task by feeding images individually and asking separate questions for each, then aggregating responses.
• For memory-dependent tasks (e.g., Fig. 52, object localization requiring reference to a prior example), Gemini's inability to retain context means that even with workarounds, some tasks are fundamentally less fair to Gemini.

The paper does not always control for this asymmetry systematically. In the interleaved image-text tasks (Section 6.1), GPT-4V receives images sequentially (its native mode), while Gemini receives a composite image or decomposed queries—fundamentally different tasks that may advantage or disadvantage Gemini in ways the paper does not fully characterize. This is a genuine limitation of the comparison methodology, but it also reflects a real-world constraint: if a model architecturally cannot perform sequential multi-image reasoning, that limitation exists regardless of whether we consider it "fair" to penalize the model for it.

Positioning Against Prior MLLM Evaluation Work

The paper's intellectual lineage is worth parsing:

Direct predecessor: "Dawn of LMMs" (Yang et al., 2023). This prior work established the multi-dimensional qualitative evaluation framework that this paper inherits—testing GPT-4V across image recognition, text understanding, reasoning, temporal understanding, multilingual capabilities, and industry applications using a diverse set of hand-curated examples. The current paper explicitly positions itself as extending this framework to Gemini, using the same images, prompts, and GPT-4V results as baselines. This gives the comparison an unusual degree of replicability: the GPT-4V results are not the current paper's own generation but are drawn from a published, separately-conducted study, reducing the risk of cherry-picking or prompt-engineering to favor one model.

The broader landscape of MLLM evaluation at the time. In late 2023, quantitative MLLM benchmarks existed (MMBench, MME, SEED-Bench, etc.) but were primarily designed for evaluating open-source models with accessible logits or generation probabilities. These benchmarks provide aggregate scores but limited insight into why models fail, what kinds of errors they make, or how their behavior differs qualitatively. The qualitative case-study approach filled a gap: it allowed researchers to observe model behavior on complex, realistic tasks that resist simple scoring rubrics (detective reasoning, emotional intelligence, GUI navigation) and to identify patterns that aggregate metrics obscure.

The paper's contribution is not in proposing a new evaluation methodology but in applying an existing qualitative framework systematically enough to distill behavioral patterns—identifying not just that Gemini performs worse on some tasks, but why (single-image constraint, memory limitation, over-verbosity leading to reduced precision) and when (multi-image contexts, temporal reasoning, abstract pattern matching requiring comparison across sub-images).

The Practical Stakes: Model Selection, Integration, and Development

The paper's findings have direct practical relevance that motivates the comparison:

• For developers and practitioners: Understanding that Gemini cannot natively handle multi-image inputs, masks sensitive facial information, and generates verbose responses with supplementary links directly informs which use cases each model suits. A retail application requiring identification and linking of products (Section 10.2) might prefer Gemini's link-generation capability; a medical imaging or autonomous driving application requiring sequential scene analysis might require GPT-4V's multi-image memory.

• For model developers: The paper's detailed documentation of Gemini's failure modes under the single-image constraint provides concrete evidence for the importance of multi-image input capabilities in next-generation MLLMs. Several tasks where Gemini significantly underperforms—Raven's Progressive Matrices (Section 4.5), temporal ordering (Section 8.2), interleaved image-text reasoning (Section 6.1)—all hinge on the ability to compare or reason across multiple visual inputs. This is actionable design guidance.

• For the research community: The paper's finding in Section 11—that combining both models' outputs (GPT-4V for detailed scene recognition, Gemini for story generation and product linking) can leverage complementary strengths—suggests a research direction toward ensemble or collaborative MLLM systems rather than single-model deployments. This is a preliminary but provocative result.

Summary of the Motivational Logic

The paper is motivated by a confluence of factors: the release of a new frontier MLLM (Gemini) challenging the incumbent (GPT-4V); the absence of systematic qualitative comparison between them; the deep architectural differences between the models that make naive head-to-head comparison misleading; the availability of an established qualitative evaluation framework from prior work; and the practical need for nuanced understanding of model behavior to inform selection, integration, and development decisions. The paper's response is a multi-dimensional case study that documents behavioral patterns, identifies structural explanations for performance differences, and points toward complementary use cases where neither model is universally dominant.

3. Technical Approach

3.1 Reader Orientation

This paper constructs a multi-dimensional qualitative evaluation framework for comparing two frontier multimodal large language models—Google's Gemini and OpenAI's GPT-4V—by systematically running both models through the same set of hand-curated visual tasks and analyzing their behavioral differences. The core problem it solves is that no prior work had provided a systematic, side-by-side qualitative comparison of these two closed-source models, and the solution is a case-study methodology that reuses an established evaluation framework from prior GPT-4V research, adapts prompts for interface differences, and documents behavioral patterns across ten distinct capability dimensions (image recognition, text understanding, reasoning, temporal understanding, multilingual processing, and five industry application domains).

3.2 Big-Picture Architecture (Diagram in Words)

The evaluation framework has five major components:

1. Sample Collection Module — a curated set of 97 test cases (images, prompts, and previously-recorded GPT-4V responses) drawn from the prior "Dawn of LMMs" study (Yang et al., 2023), plus additional images from the internet for the integrated-use section (Section 11). Each test case is designed to probe a specific multimodal capability (e.g., landmark recognition, object counting, emotional intelligence assessment, GUI navigation).

2. Prompt Adaptation Layer — a procedural step where prompts originally designed for GPT-4V's multi-image, sequential-input interface are modified for Gemini's single-image, non-memory constraints. This includes stitching multiple images into composite inputs, decomposing multi-step queries into sequential single-image questions, and adjusting wording to match Gemini's architecture.

3. Model Execution Module — the actual inference calls to both model APIs: GPT-4V (accessed through its vision-enabled chat interface, supporting multi-image sequential input) and Gemini Pro (accessed through its multimodal interface with single-image input, image/link output capability, and YouTube-linked video understanding). Both models receive the same (or adapted) visual inputs and answer the same (or adapted) questions.

4. Response Collection and Aggregation Layer — the outputs from both models are recorded verbatim, including both textual responses and any supplementary outputs (Gemini's generated images, web links, video references). For decomposed tasks (e.g., Fig. 46 where three receipts are fed to Gemini individually), responses are manually aggregated to compute the final answer.

5. Qualitative Analysis Module — human researchers compare responses side-by-side, categorize them as "Satisfactory" (blue highlight) or "Unsatisfactory" (red highlight) based on factual accuracy and task completion criteria, and identify behavioral patterns (Gemini's verbosity, GPT-4V's conciseness, specific error types, interface-constraint-driven failures). The analysis is organized into ten capability dimensions and seven industry application scenarios.

Information flows as follows: a test case (image + prompt) is selected from the sample collection → prompts are adapted for Gemini's interface → both models execute inference → responses are collected and, for decomposed Gemini tasks, aggregated → researchers qualitatively assess and categorize responses → patterns are synthesized into dimension-level findings and overall conclusions.

3.3 Roadmap for the Deep Dive

• First, the sample collection and taxonomy (Section 3.4.1) — what exactly constitutes the test set, how it is organized across capability dimensions, and what makes it a valid basis for qualitative comparison — since the entire analysis flows from these curated cases.
• Second, Gemini's input modality constraints and the adaptation strategies (Section 3.4.2) — the specific interface differences between the two models, the workarounds employed, and the methodological tensions these create — because these structural differences fundamentally shape the comparison and explain many observed performance patterns.
• Third, the prompting methodology and response handling (Section 3.4.3) — how prompts are sourced, adapted, and standardized; how responses are recorded and evaluated; and the criteria for classifying responses as satisfactory or unsatisfactory — since this establishes the evaluative framework.
• Fourth, the capability taxonomy and task design logic (Section 3.4.4) — the ten evaluation dimensions and seven industry scenarios, what each is designed to probe, and how they collectively provide multi-faceted coverage of multimodal intelligence — to understand why specific task types were chosen and what they reveal.
• Fifth, the integrated-use methodology (Section 3.4.5) — the two-step chaining approach where GPT-4V's recognition output feeds Gemini's generation or recommendation capabilities, and the rationale for this combination — since this represents the paper's most distinctive methodological contribution beyond pure comparison.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a qualitative case-study paper whose core idea is that the behavioral differences between Gemini and GPT-4V can be systematically characterized by testing both models on an established, multi-dimensional set of visual tasks and analyzing their responses for accuracy, completeness, style, and failure modes.

---

3.4.1 Sample Collection and Capability Taxonomy

What constitutes the test set. The paper uses 97 figures (images with associated prompts) organized into ten evaluation sections and seven industry application subsections, as itemized in the List of Figures. All test cases, prompts, and GPT-4V responses (except those in Section 11) are sourced from Yang et al. (2023)'s "Dawn of LMMs" study. The authors state explicitly in Section 1.4:

"All our data is sourced from 'The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)' (except for the images in Section 11, which are sourced from the internet). We have utilized their images, GPT-4V's prompts, and corresponding results."

This reuse strategy is methodologically significant: by using an independently-curated, published test set with pre-recorded GPT-4V responses, the paper avoids the risk of cherry-picking examples that favor one model or post-hoc prompt optimization. The GPT-4V results are not the current authors' own generations but rather reference data from a separate research group, providing a degree of replicability unusual in qualitative comparison studies.

The capability taxonomy. The evaluation is organized hierarchically into five levels of multimodal understanding, as described in Section 1.1:

• Level 1: Basic Recognition (Sections 2–3). This involves identifying objects, landmarks, foods, logos, abstract shapes, scenes, text, mathematical formulas, and chart content without further inference. Tasks include object counting, spot-the-difference, scene text OCR, equation-to-LaTeX conversion, and chart text extraction. The goal is to isolate pure perception from reasoning.

• Level 2: Inference and Reasoning (Sections 4–5). Building on recognition, this level requires drawing conclusions—understanding humor, applying scientific knowledge, making deductions from environmental clues, completing pattern-recognition puzzles (Raven's Progressive Matrices), assessing emotional content, solving math problems, reasoning about charts and documents. The distinction between Sections 4 and 5 is that Section 4 focuses on image-based reasoning (understanding visual content and drawing inferences from it) while Section 5 focuses on textual reasoning in images (reading text within images and performing logical or mathematical operations on that text).

• Level 3: Integrated Multi-Image Understanding (Section 6). This level tests the ability to combine information across multiple images or between images and text—calculating total tax from multiple receipts, matching beer types on a table to menu prices, and providing iterative feedback on text-to-image generation outputs. This is where Gemini's single-image constraint creates the most pronounced methodological challenges.

• Specialized Capabilities (Sections 7–9). Three focused evaluations: object localization (providing normalized bounding-box coordinates for specified objects), temporal video understanding (action recognition from keyframes and temporal ordering of video segments), and multilingual capabilities (describing images and recognizing text in diverse languages). These test specific, narrow capabilities not fully covered by the broader recognition-reasoning taxonomy.

• Industry Applications (Section 10). Seven applied scenarios: defect detection on assembly lines, grocery checkout item identification, auto insurance damage assessment, customized object captioning, text-to-image generation quality evaluation, embodied agent navigation in indoor environments, and GUI navigation for web browsing, online shopping, and notification handling. These simulate real-world deployment contexts that a qualitative study can probe more naturally than a benchmark.

Why this taxonomy is designed this way. The hierarchical organization (Recognition → Reasoning → Multi-Image Integration → Specialized Capabilities → Applications) is designed to isolate capability boundaries. If a model fails at Level 1 basic recognition, its failures at Level 2 reasoning are ambiguous—they could reflect perceptual errors rather than reasoning limitations. By testing basic recognition first and separately, the paper can attribute failures more precisely. The specialized sections (7–9) isolate capabilities that are known to challenge different model architectures (spatial grounding, temporal sequence processing, multilingual transfer). The industry section tests whether qualitative behavioral differences observed in controlled tasks translate to practical deployment contexts.

The scoring system. The paper does not use numerical scores. Instead, each response is qualitatively categorized: blue-highlighted responses are "Satisfactory Answers" and red-highlighted responses are "Unsatisfactory Answers." The criteria appear to be: (1) factual accuracy (is the described object/landmark/text actually present and correctly identified?), (2) task completion (does the response directly answer the question asked?), and (3) absence of significant hallucination (are invented details, incorrect quantities, or impossible inferences present?). The categorization is performed by human researchers and displayed visually in the figures side-by-side.

The role of the "Dawn of LMMs" baseline. Because GPT-4V responses are drawn from prior published work rather than generated fresh for this comparison, they represent a fixed reference point. This means the paper cannot control for potential API updates to GPT-4V between the original study's data collection and the current paper's Gemini testing. If GPT-4V's capabilities changed (improved or degraded) between September and December 2023 due to model updates, the comparison would not reflect the current state of both models. The paper does not address this temporal confound.

---

3.4.2 Gemini's Input Modality Constraints and Adaptation Strategies

The fundamental asymmetry. The paper's most critical methodological challenge is that Gemini and GPT-4V have fundamentally different input interfaces, as detailed in Section 1.2. GPT-4V supports "the continuous ingestion of multiple images as context, thereby possessing enhanced memory capabilities." Gemini, in contrast, has the constraints enumerated in Section 1.2:

Constraint 1: Single-Image Input Only. Gemini is "limited to inputting a single image at a time" and "cannot process independent images; instead, it requires accompanying textual instructions." This constraint means that any task designed to test multi-image reasoning—interleaved image-text understanding (Section 6.1), temporal ordering from keyframes (Section 8.2), Raven's Progressive Matrices where patterns span multiple sub-figures (Section 4.5), object localization requiring comparison to a reference example (Section 7.1)—cannot be presented to Gemini in its native form.

Constraint 2: No Persistent Visual Memory. Unlike GPT-4V, Gemini's multimodal module "lacks the ability to retain memory of past image inputs and outputs." This is a stricter constraint than single-image input alone: even if multiple images are presented sequentially in a conversation (rather than within a single turn), Gemini would not retain visual context from earlier turns. This means that multi-step tasks requiring visual reference to previously-seen images (e.g., Fig. 52, where the model must reference a beer bottle localization example from a prior turn to understand the coordinate format) cannot work with Gemini in the sequential manner GPT-4V handles.

Constraint 3: Sensitive Information Masking. The paper reports that Gemini "exhibits some degree of obfuscation when processing images containing explicit facial or medical information, making it unable to recognize these images." This is a content-filtering constraint rather than an interface constraint, but it means certain test cases may produce refusal responses from Gemini that would not occur with GPT-4V. The paper does not quantify how many test cases trigger this behavior.

The primary adaptation strategy: image compositing. For tasks originally designed for GPT-4V's multi-image sequential input, the paper's approach is to "combine all the images into a single image input" (Section 1.2). This means stitching multiple images together into one composite image—arranging video keyframes side-by-side for temporal ordering tasks, placing multiple receipts in a grid for tax calculation, arranging Raven's Progressive Matrices sub-figures into a single panel. The paper states: "This integrated input mode will be used unless explicitly stated otherwise."

This adaptation fundamentally changes the task. The model must now: (1) parse image boundaries within the composite to identify separate sub-images, (2) understand the spatial arrangement (which sub-image corresponds to which position in the sequence or grid), (3) process each sub-image at reduced effective resolution since the composite's total pixel budget is now divided among multiple sub-images, and (4) for temporal tasks, infer temporal order from spatial arrangement rather than from sequential presentation. The paper does not control for the resolution reduction or the additional cognitive burden of parsing composite image structure.

The secondary adaptation strategy: task decomposition. For some tasks, the paper further decomposes the multi-image query into separate single-image queries. For example, in Fig. 46 (calculating total tax from three receipts), after Gemini fails on the composite input, the paper splits the task: "For Gemini, we divide the pictures and ask separately." Each receipt is fed individually with its own prompt ("You paid $3.72 in tax. The information is at the bottom of the receipt, under TOTAL TAX."), and the results are manually aggregated. Similarly, in Fig. 47 (calculating beer cost from a menu), the paper splits the task into two queries: first identifying the beer brand and quantity from the table image, then reading the price from the menu image.

This adaptation changes the task even more fundamentally than compositing. The model no longer needs to integrate information across images—the human researcher performs that integration. The decomposed approach tests single-image recognition rather than multi-image reasoning. The paper documents this distinction implicitly by showing both Gemini's composite-input failure and its decomposed-input success, but does not systematically analyze how the task demands differ between the two modes.

When adaptation is impossible. For true memory-dependent tasks, neither compositing nor decomposition is feasible. Fig. 52 shows a localization task where the model must reference a beer-bottle coordinate example from a previous turn to understand the coordinate format. GPT-4V can do this via sequential multi-image memory. For Gemini, the paper's adaptation is unclear—it appears to include the example and the query image in a composite, but without the conversational context that establishes the example's reference role. The paper reports that Gemini generates a response but its quality relative to GPT-4V is not explicitly rated in that figure.

The video understanding asymmetry. Section 1.2 notes that Gemini has a distinct video understanding capability: it "demonstrates the capability to understand videos and requires a YouTube link as a video input." This is fundamentally different from the keyframe-based temporal understanding tested in Section 8, where the task is to reason about video content from a small set of static keyframes. The paper's temporal understanding evaluation uses GPT-4V's multi-image sequential input (feeding keyframes sequentially) versus Gemini's composite-image input (all keyframes stitched into one image). Gemini's native video understanding mode is not tested in the temporal understanding section, creating a gap: the paper evaluates Gemini on a task that its interface handles poorly (composite keyframe reasoning) but does not evaluate it on a task it was natively designed for (direct video understanding from YouTube links). The reasons for this choice are not explained.

Image and link output as a distinguishing feature. Section 1.2 notes that Gemini, "unlike GPT-4V, which is limited to generating textual outputs, has the ability to create images related to the content and provide corresponding links." This is not a constraint but a capability extension that shapes the comparison throughout the paper. In landmark recognition (Fig. 2, Fig. 3), Gemini outputs Wikipedia links to the identified landmarks; in food recognition (Fig. 4, Fig. 5), it provides recipe links; in grocery checkout (Fig. 67), it generates images of identified products. This capability is not evaluated against a comparable GPT-4V feature (since GPT-4V lacks it), but it is consistently noted as a distinguishing characteristic.

---

3.4.3 Prompting Methodology and Response Handling

Prompt sourcing. As stated in Section 1.3, the prompts used for GPT-4V are taken directly from the "Dawn of LMMs" study. For Gemini, the paper first attempts to use these prompts verbatim: "The prompt design under consideration is tailored for GPT-4V, and direct input into Gemini may yield unsatisfactory responses. In such cases, adjustments to Gemini's prompt are made to align with the input requirements of its architecture."

The paper does not specify which prompts were adjusted, how they were modified, or what criteria triggered adjustment. This introduces a methodological opacity: the reader cannot distinguish between cases where Gemini failed because of capability limitations versus cases where the prompt was suboptimal and could have been improved. The paper's statement that adjustments are made "to align with the input requirements of its architecture" suggests modifications were primarily structural (single-image vs. multi-image format) rather than semantic (rephrasing for clarity), but this is not explicitly verified.

The zero-shot and few-shot prompt patterns. Across the evaluation, prompts follow several recurring patterns:

• Direct queries: "Describe the landmark in the image," "What are all the scene text in the image?", "Count the number of apples in the image." These test basic recognition without providing examples.

• Step-by-step reasoning prompts: For complex tasks like chart reading (Fig. 20), the prompt includes explicit decomposition: "Step 1, read the title of the graph; Step 2 read the legend of the graph... Let's think step by step." This prompt structure tests whether the model can follow decomposing instructions.

• In-context example prompts: For object localization (Section 7), the prompt includes a worked example with specific coordinates: "For example, the (x1,y1,x2,y2) coordinates for that region with the beer bottle is (0.47, 0.48, 0.55, 0.87)." This tests few-shot coordinate generation capability.

• Zero-shot vs. few-shot comparison: Fig. 16 explicitly contrasts zero-shot and few-shot dashboard reading. The zero-shot prompt is "What is the read of the speed meter?" The few-shot version provides two worked examples before asking about a third dashboard. This isolates the effect of in-context examples.

• Multi-turn conversational prompts: The embodied agent (Section 10.6) and GUI navigation (Section 10.7) tasks use multi-turn prompts that reference previous actions: "At the previous turn, you were at the position shown in the below image. The action taken in previous turn is: turn right and move forward towards the hallway. Now you are at the hallway, this is what you can see from there." This tests sequential reasoning with accumulated context.

The localization coordinate format. Section 7 introduces a specific coordinate representation for object localization that serves as an interface convention throughout that section:

"Regions are represented by (x1,y1,x2,y2) coordinates. x1 x2 are the left and right most positions, normalized into 0 to 1, where 0 is the left and 1 is the right. y1 y2 are the top and bottom most positions, normalized into 0 to 1, where 0 is the top and 1 is the bottom."

where $(x_1, y_1)$ is the top-left corner of the bounding box and $(x_2, y_2)$ is the bottom-right corner, both expressed as fractions of image width and height respectively.

What it computes: the four-tuple $(x_1, y_1, x_2, y_2)$ defines an axis-aligned rectangle in normalized image coordinates. For an image of width $W$ and height $H$, the pixel-coordinate rectangle is $(\lfloor x_1 W \rfloor, \lfloor y_1 H \rfloor)$ to $(\lfloor x_2 W \rfloor, \lfloor y_2 H \rfloor)$.

Why this form: normalized coordinates are resolution-independent, meaning the same coordinate tuple describes the same relative region regardless of image scaling. The $(x_1, x_2)$ convention for horizontal extent and $(y_1, y_2)$ for vertical is standard in object detection (e.g., COCO format), making the output machine-parseable. The paper uses this format because it requires the model to understand spatial relationships and produce structured numerical outputs from visual inputs—simultaneously testing perception, spatial reasoning, and format adherence.

Response recording and presentation. Responses are recorded verbatim and displayed side-by-side in figures with color-coded highlighting. Red-highlighted responses indicate unsatisfactory answers; blue-highlighted responses indicate satisfactory answers. Each figure caption identifies which section and task the figure corresponds to. The paper does not describe a formal inter-annotator agreement protocol for the satisfactory/unsatisfactory classification; it appears to be based on author consensus.

The response verbosity asymmetry. A recurring observation throughout the paper is that Gemini's responses are substantially longer and more detailed than GPT-4V's, even when the prompt does not request detail. For example, in Fig. 2 (landmark recognition), GPT-4V provides a single concise paragraph while Gemini provides multiple paragraphs with historical context and a Wikipedia link. In Fig. 7 (in-the-wild logo recognition), GPT-4V's Starbucks description is 4 sentences; Gemini's is 10 sentences with a structured breakdown. This verbosity difference is not controlled for—the paper evaluates responses as-is rather than length-normalizing or explicitly prompting for concise answers from Gemini.

The "Unsatisfactory" classification criteria. Based on examining the figures, unsatisfactory responses fall into several categories: (1) factually incorrect identifications (e.g., Gemini misidentifying the Manhattan Bridge as the Brooklyn Bridge in Fig. 3), (2) numerical errors (e.g., Gemini miscounting objects in Fig. 12), (3) task non-completion (e.g., Gemini refusing to generate an insurance report in Fig. 70), (4) excessive hallucination (e.g., Gemini inventing details about a virtual reality headset in Fig. 54), and (5) format non-compliance (e.g., Gemini failing to follow the requested JSON format in Fig. 70). The paper does not formalize these categories or report their relative frequencies.

---

3.4.4 Capability Taxonomy and Task Design Logic

The five-level hierarchy. Section 1.1 structures the evaluation into a progression of increasing complexity:

Level 1: Image Recognition and Understanding (Section 2). Tasks in this section are designed to isolate visual recognition from reasoning. The paper states that this section "addresses the fundamental recognition and comprehension of image content without involving further inference." Specific tasks include:

• Basic object recognition (Fig. 1): Tests whether models can comply with output format constraints (generating descriptions of exact word counts or with specific initial letters) while accurately identifying scene contents. This probes both visual understanding and instruction-following.

• Recognition of specific visual categories (Figs. 2–9): Landmarks, foods, logos, abstract tangram shapes, and driving scenes. These test domain-specific visual knowledge—can the model identify a specific temple, dish, brand logo, or abstract shape? The choice of categories spans both culturally-specific (Chinese dishes, Japanese landmarks) and universal (logos, abstract shapes) items, testing breadth of visual knowledge.

• Counterfactual examples (Fig. 11): The prompt asks about a non-existent object ("Can you find the red ball near the gray cat?") to test whether the model hallucinates versus correctly identifying absence. This is a critical capability for real-world deployment: models must say "I cannot find X" rather than fabricating X's location.

• Object counting (Fig. 12): Tests precise enumeration rather than approximate description. The paper uses a multi-step prompting strategy for GPT-4V: "Count the number of apples in the image. Let's count the apples row-by-row." GPT-4V succeeds with this guided decomposition; Gemini fails even with guidance, which the paper attributes to its "relatively weaker multimodal memory performance."

• Spot the difference (Fig. 13): Two similar images are presented, requiring careful comparison to identify subtle differences. This tests fine-grained visual discrimination and comparative reasoning from a single composite input.

Level 1b: Text Recognition in Images (Section 3). This section "focuses solely on text extraction and basic understanding, without involving deep logical reasoning." Tasks include:

• Scene text OCR (Figs. 14–16): Extracting text from billboards, stock tickers, shop signs, and informational displays under varied conditions (angles, lighting, fonts). The zero-shot vs. few-shot comparison in Fig. 16 tests whether models can learn to read specific instrument types from examples.

• Equation recognition (Fig. 17): Converting handwritten and printed mathematical formulas to LaTeX. This tests specialized symbol recognition beyond natural language characters.

• Chart text recognition (Figs. 18–20): Reading data from tables, line charts, and trend graphs, and generating reconstruction code or structured output. This bridges pure text extraction and basic data interpretation.

The separation between recognition and reasoning is central to the evaluation design. The paper explicitly states that Level 1 tasks "do not involve further inference," while Level 2 tasks (Sections 4–5) require "more advanced reasoning." This separation allows the paper to attribute failures: if a model fails at chart text recognition (Level 1), its failure at chart reasoning (Level 2) may be due to perceptual errors, not reasoning limitations. Conversely, if a model succeeds at recognition but fails at reasoning, the limitation is in the reasoning capability itself.

Level 2: Image Reasoning (Section 4). Tasks in this section test inference from visual content:

• Humorous image understanding (Fig. 21): Interpreting visual memes that combine images with text to create humor. This requires understanding social conventions, irony, and the relationship between the visual and textual elements.

• Multimodal knowledge and commonsense (Figs. 22–24): Answering science questions (geography, physics, biology) using visual information from maps, diagrams, and food webs. This tests whether the model can extract relevant information from images and apply domain knowledge.

• Detective reasoning (Fig. 25): Inferring personal characteristics (occupation, lifestyle, personality) from a room's contents. This tests abductive reasoning from environmental cues.

• Part-object association (Fig. 26): Combining separate visual elements (cardboard shapes, jeans) into a coherent whole (a person). This tests compositional visual reasoning.

• Intelligence tests (Figs. 27–30): Raven's Progressive Matrices and WAIS-style pattern completion. These are standard IQ-test proxies that require identifying abstract visual patterns and applying them to complete sequences. The paper notes Gemini's particular difficulty here, which it attributes to the composite-image input degrading performance on tasks that require comparing multiple sub-figures.

• Emotional intelligence tests (Figs. 31–35): Assessing emotional responses to images, making aesthetic judgments, generating descriptions to elicit desired emotions (soothing, deterring). This tests social and emotional reasoning rather than logical reasoning.

Level 2b: Textual Reasoning in Images (Section 5). Building on text recognition, these tasks require reasoning about extracted text:

• Visual math (Fig. 36): Solving equations and geometry problems presented as images. This requires extracting mathematical expressions, recognizing geometric relationships, and performing calculations.

• Table and chart reasoning (Figs. 37–41): Answering questions that require aggregating, comparing, or otherwise analyzing data from tables and charts. Fig. 39 asks for profit margins; Fig. 40 asks to identify anomalous trends. This requires reading structured data and performing comparative operations.

• Document understanding (Figs. 42–45): Interpreting floor plans, scientific papers, posters, and web pages. This tests comprehension of document structure, specialized notation (floor plan dimensions), and domain-specific content (academic paper contributions).

Level 3: Integrated Image and Text Understanding (Section 6). This section tests the ability to combine information across multiple inputs:

• Interleaved image-text inputs (Figs. 46–47): Computing total tax from multiple receipts or matching items on a table to menu prices. This requires recognizing relevant information in each image, maintaining it in memory, and performing arithmetic aggregation. The paper explicitly tests this in both GPT-4V's native multi-image mode and Gemini's adapted (composite then decomposed) modes.

• Text-to-image generation guidance (Figs. 48–50): Iteratively improving prompts for a text-to-image model by analyzing generated outputs against desired targets, and using self-consistency to improve counting reliability. This tests the model's ability to provide actionable feedback on visual outputs.

The specialized and applied evaluations. Sections 7–10 extend the evaluation beyond the recognition-reasoning spectrum to capabilities that matter for specific deployment contexts:

• Object localization (Section 7): This is a fundamentally different type of task than description or QA because it requires structured numerical output (bounding box coordinates). The paper tests this in both real-world scenes (cars, beer bottles) and abstract images (tangram shapes). The inclusion of this section reflects the paper's interest in actionable model outputs rather than just descriptive responses.

• Temporal understanding (Section 8): The paper evaluates temporal reasoning through keyframes rather than native video understanding (for Gemini). Action recognition (Fig. 54) tests whether the model can interpret a sequence of motion-capture frames. Temporal ordering (Fig. 55) tests whether the model can correctly sequence scrambled frames of a sushi-making process. Gemini's failure in the latter is particularly notable—it cannot infer the correct temporal order from the composite keyframe arrangement.

• Multilingual capabilities (Section 9): This tests both output language switching (describing images in Chinese, French, Czech, and 20 other languages) and input language recognition (reading scene text in Arabic, Greek, Japanese). The section demonstrates that both models are broadly multilingual, with some minor errors in Gemini's Chinese character recognition.

• Industry applications (Section 10): These seven scenarios (defect detection, grocery checkout, auto insurance, customized captioning, image generation evaluation, embodied agents, GUI navigation) are designed to simulate real-world deployment contexts. Each scenario uses multi-turn interactions where the model must maintain task context across turns, understand domain-specific requirements (insurance report formats, product links, navigation instructions), and produce actionable outputs.

The self-consistency test (Fig. 50). One methodologically interesting task in Section 6.2 tests counting reliability by repeating the same counting query ("Count the number of boats in the image") multiple times. GPT-4V produces three different answers across four attempts (4, 4, 4, and 6 boats), while Gemini consistently outputs 5 boats across all four attempts—but 5 is incorrect (the paper's ground truth appears to be 4 or 6, though it is not explicitly stated which). The paper frames this as demonstrating that "Gemini exhibits stronger consistency in its results" even though consistently wrong. This is a deliberate test of a specific behavioral property (response stability) rather than accuracy.

---

3.4.5 Integrated-Use Methodology

The combination paradigm. Section 11 introduces a methodological contribution beyond pure comparison: the paper explores "a novel paradigm that integrates the functionality of two distinct models" to leverage their complementary strengths. The approach is a two-stage pipeline:

Stage 1: GPT-4V performs detailed scene recognition and description, producing a comprehensive textual inventory of objects, their attributes, and their spatial relationships within the image. This leverages GPT-4V's observed strength in "accurate and comprehensive recognition capabilities" (Section 11 introduction).

Stage 2: The GPT-4V textual output is concatenated with the original image and fed to Gemini, which performs a generative or retrieval task on top of the enriched description. This leverages Gemini's strength in "providing detailed, expansive answers accompanied by relevant imagery and links" and its web-search-like association capability.

Why this specific pipeline direction. The paper is explicit about the complementarity that motivates this architecture: "GPT-4V excels in generating accurate and comprehensive object descriptions. With textual guidance in place, Gemini leverages the combined information of text and images for retrieval and recommendation." The pipeline is not symmetric—GPT-4V output feeds into Gemini, but not vice versa. The paper does not explore the reverse direction (Gemini-generated detailed descriptions feeding GPT-4V for reasoning) or more complex interaction patterns (iterative refinement, debate-style exchange).

Scenario 1: Product Identification and Recommendation (Fig. 96). The pipeline is:

1. Input: A living room image containing multiple furniture and decor items.
2. GPT-4V stage: Prompted with "What is in the room?" GPT-4V produces a detailed inventory: "large flat-screen television on a wooden media console with glass doors... large mirror propped against the wall... round red rug with decorative patterns... woven pouf... modern armchair... plant in a pot..."
3. Gemini stage: Prompted with the GPT-4V description concatenated with the original image: "Can you recommend me all the product links in the picture?" Gemini responds with product recommendations including specific retailer links (Amazon, IKEA, Target) for items matching the described furniture.
4. Result: The combination achieves what neither model could do alone—GPT-4V "struggles to provide recommended links for products, while Gemini, in the initial phase, faces challenges in comprehensively identifying objects."

This scenario demonstrates the core principle: GPT-4V handles fine-grained recognition (identifying specific items in a cluttered scene), while Gemini handles association and retrieval (matching recognized items to purchasable products with working links). The ablation is implied but not formally tested—the paper states that each model alone fails at the other's task, but the single-model baselines are not shown in the figure.

Scenario 2: Multi-Image Recognition and Story Generation (Fig. 97). The pipeline is:

1. Input: A composite image showing the same location across four seasons (spring, summer, autumn, winter).
2. GPT-4V stage: Prompted with "This image depicts the different seasons of a location. Can you point them out one by one?" GPT-4V identifies each seasonal section in order, describing visual characteristics: "Spring: lush, green setting with fully leafed trees... Summer: basketball court in full sunlight... Autumn: warm golden and brown tones... Winter: landscape covered in snow, trees are bare..."
3. Gemini stage: Prompted with the GPT-4V description: "Can you imagine a story based on these pictures? (Makoto Shinkai's style.)" Gemini generates a multi-paragraph narrative in the requested style, weaving together the four seasonal descriptions into a continuous story about a girl returning to a basketball court.
4. Result: The paper states that using a single model produces either "unclear content summarization or the generation of overly formatted and brief narrative sequences." The combination leverages GPT-4V's structured scene parsing and Gemini's creative generation capability.

Why the integrated-use methodology matters. The integrated-use section (11) is the paper's most forward-looking contribution. It demonstrates that the comparison study's findings—the identification of complementary strengths—can be directly operationalized. Rather than treating the models as competitors to be ranked, the paper shows they can function as components in a larger system. This is methodologically significant because it shifts the framing from "which model is better?" to "how can these models be composed to achieve capabilities neither possesses alone?" The paper does not quantify the performance gain of combination over single-model baselines, leaving this as a qualitative demonstration rather than a rigorous evaluation.

4. Key Insights and Innovations

Innovation 1: Architectural Interface Constraints as the Primary Explanatory Variable for Model Behavior

The paper's most significant intellectual move is not a quantitative ranking but a diagnostic reframing: the primary determinant of Gemini's relative performance versus GPT-4V across task types is not model "intelligence" in any general sense, but rather the specific architectural and interface design choices made by each model's developers—most critically, Gemini's single-image, non-memory input constraint.

This insight is distinctive because it inverts the typical interpretation framework for model comparisons. A naïve reading of the paper's findings—Gemini struggles on Raven's Progressive Matrices, temporal ordering, interleaved image-text reasoning, embodied agent navigation, and GUI interaction—would conclude that Gemini is simply a weaker model. The paper's contribution is to demonstrate that these failures are structurally uniform: they all trace to a single, identifiably architectural limitation (the inability to accept multiple images as sequential context or retain visual memory across turns) rather than to diffuse reasoning deficits. The evidence is convergent across sections: Gemini matches or approaches GPT-4V on single-image tasks (basic recognition in Section 2, scene text OCR in Section 3.1, multilingual description in Section 9.1, defect detection in Section 10.1) but systematically underperforms on any task requiring comparison across images (temporal ordering in Section 8.2, interleaved receipt calculation in Section 6.1, RPM pattern completion in Section 4.5) or sequential visual context accumulation (embodied navigation in Section 10.6, GUI navigation requiring tool-use memory in Section 10.7).

Prior comparison studies of multimodal models—including the "Dawn of LMMs" paper the authors build on—tended to report capability patterns without systematically attributing them to specific architectural constraints. The field's default assumption when Model A outperforms Model B on a task class is that Model A has "better reasoning" or "more knowledge." This paper provides a counterexample: Gemini's failures are not primarily reasoning failures but interface-access failures—the model may possess the capability to reason about multi-image relationships, but its interface design prevents it from accessing the inputs necessary to exercise that capability. The composite-image workaround, which converts multi-image tasks into single-image parsing problems, introduces additional cognitive burdens (sub-image boundary detection, resolution dilution, spatial-to-temporal inference) that confound capability assessment.

This reframing has practical significance beyond the specific comparison. It implies that future MLLM development should treat input interface design as a first-class capability concern, not a UX afterthought. A model's ability to process sequential visual inputs or retain visual context is not a convenience feature—it is a core reasoning affordance that determines which tasks the model can attempt at all. For model selection in practice, the insight means that choosing between Gemini and GPT-4V is less about generic "quality" and more about task-architecture fit: applications requiring multi-image reasoning (document comparison, video understanding, sequential troubleshooting) should prefer GPT-4V regardless of Gemini's single-image strengths.

The paper's failure to control for this asymmetry systematically (noted in the prior analysis) is itself informative—it reveals that the single-image constraint is so fundamental that fair comparison is impossible for a substantial class of tasks. This negative finding—that architectural incompatibility precludes meaningful head-to-head comparison on certain task families—is arguably more important than any individual performance ranking.

---

Innovation 2: Complementary Model Strengths as an Operationalizable System-Design Principle

The paper's second conceptual contribution is the demonstration that complementary model strengths can be operationalized through a chained pipeline to achieve capabilities neither model possesses alone, as shown in Section 11. This moves beyond comparison into a nascent design methodology for multimodal AI systems.

The background assumption in prior MLLM comparisons—both quantitative benchmark evaluations and qualitative studies like "Dawn of LMMs"—is that models are competitors to be ranked. The analysis framework asks: "Which model performs better on Task X?" The output is an ordinal ranking or a set of relative strengths and weaknesses. This paper retains that framework for most of its analysis but then, in Section 11, introduces a different question: "Given that these models have different strengths, can we compose them to achieve better results than either alone?"

The specific pipeline design—GPT-4V for fine-grained recognition and structured description, Gemini for retrieval, recommendation, and creative generation—is not theoretically deep. But the concept that qualitative comparison studies should culminate in composition experiments is a substantive methodological contribution. It transforms the output of a comparison study from a static report card into a dynamic system-building guide. The finding that GPT-4V "struggles to provide recommended links for products, while Gemini, in the initial phase, faces challenges in comprehensively identifying objects" (Section 11.1) but that the combination succeeds documents a genuine synergy: each model covers the other's failure mode.

The significance extends beyond the specific models. The paper's pipeline—one model parses, another generates—is a particular instance of a broader pattern: models with different training objectives, architectural constraints, and output capabilities may be composable into more capable systems. This is not a new idea in AI broadly (ensemble methods, mixture-of-experts, tool-use architectures), but its application to frontier closed-source MLLMs—where the component models are black boxes that cannot be fine-tuned or internally modified—is relatively unexplored. The paper demonstrates that effective composition is possible even with zero model-internal access, using only prompt-level integration.

The limitation is that this is a two-example qualitative demonstration, not a systematic evaluation with baselines. The paper does not quantify the performance delta between the combined system and each single-model baseline, nor does it explore failure modes of the pipeline (what happens when GPT-4V's description contains an error that propagates to Gemini's output?). But as a proof-of-concept for comparative-model-composition, it opens a research direction: systematic exploration of composition patterns (sequential, parallel, iterative, debate-style) for complementary MLLMs, and development of methods for automatically determining which model should handle which subtask based on the input characteristics.

---

Innovation 3: Response Characteristics—Verbosity, Precision, and Output Modality—as First-Class Evaluation Dimensions

The third conceptual contribution is the paper's implicit argument that response style is not a cosmetic variable but a functionally significant model property that should be evaluated alongside accuracy. Throughout the paper, a recurring behavioral pattern is documented: Gemini produces substantially longer, more detailed responses than GPT-4V, often including supplementary information (historical context, cooking instructions, Wikipedia links) and, crucially, non-text outputs (generated images, product links). GPT-4V produces concise, focused answers that directly address the prompt.

Prior MLLM evaluation work—both benchmarks and qualitative studies—has treated response verbosity as noise or as a confound to be controlled for (e.g., by length-normalizing metrics or explicitly prompting for concise answers). The paper does not control for verbosity; instead, it documents it as a primary behavioral characteristic and shows that its implications are task-dependent. In some contexts (landmark recognition in Section 2.2, logo description in Section 2.4, EQ test generation in Section 4.6), Gemini's verbosity and supplementary outputs are valuable—the model acts more like an informative tour guide or shopping assistant. In other contexts (object counting in Section 2.8, precise coordinate generation in Section 7, structured insurance reporting in Section 10.3), verbosity is costly—it introduces errors when detailed descriptions include fabricated details, and it can indicate that the model is generating plausible-sounding filler rather than precise answers.

The paper's decision to highlight (not suppress) this behavioral difference represents a methodological stance: evaluating a model solely on answer correctness misses functionally relevant properties of how it answers. A model that provides a correct answer embedded in a paragraph of partially hallucinated context is less useful for many applications than a model that provides only the correct answer—even if both achieve the same binary accuracy score. Conversely, a model that provides supplementary links and images enables applications (product recommendation, landmark education) that a text-only correct answer does not.

This insight has practical significance for model selection beyond benchmark rankings. An application developer choosing between Gemini and GPT-4V needs to consider not just which model is "more accurate" on average, but which model's response profile matches the application's requirements: is conciseness more valuable than comprehensiveness? Is the risk of verbose hallucination acceptable? Are supplementary outputs (links, images) beneficial or distracting? The paper provides qualitative evidence to inform these tradeoffs, even if it does not formalize them into a decision framework.

The cost, however, is that the verbosity difference confounds accuracy comparisons throughout the paper. When Gemini's detailed response contains both correct identifications and fabricated details, should the response be classified as satisfactory or unsatisfactory? The paper's case-by-case qualitative judgments make this decision implicitly, but the lack of a formal rubric means readers cannot assess whether the classification criteria are consistently applied or whether verbosity systematically biases one model's evaluation relative to the other.

---

Innovation 4: The Self-Consistency Diagnostic as a Probe of Model Reliability Independent of Accuracy

The paper's most methodologically innovative single experiment is the boat-counting self-consistency test in Section 6.2 (Fig. 50), which reveals a counterintuitive finding: Gemini is consistently wrong across four repeated queries (always outputting 5 boats), while GPT-4V is inconsistently sometimes-right (outputting 4, 4, 4, and 6 across four attempts). The paper frames this as demonstrating "Gemini exhibits stronger consistency in its results" despite both models failing to reliably produce the correct count.

This finding is significant because it uncouples reliability from accuracy—two properties that are often conflated in model evaluation. A model can be reliable (producing the same answer under repeated queries) without being accurate (that answer is wrong), and a model can be accurate on some queries without being reliable (the answer changes unpredictably). The practical implication is that model selection depends on which failure mode is more acceptable: a consistently-wrong model enables deterministic workarounds (if you know it always says 5 boats, you can adjust), while an inconsistently-sometimes-right model may be harder to integrate into reliable systems because its errors are unpredictable.

This experiment, while small (just one image, four repetitions), introduces a diagnostic concept that generalizes beyond this specific comparison. Self-consistency under repeated identical queries is a cheaply measurable property that provides information orthogonal to accuracy. A model that produces different answers to the same question across queries is either non-deterministic in its sampling (if temperature > 0) or fundamentally uncertain in its internal representation (if temperature = 0 but output still varies due to floating-point or scheduling non-determinism). Either way, self-consistency is a proxy for the model's confidence stability. The paper does not develop this concept formally—it is a single qualitative observation—but the idea that consistency should be evaluated as a separate dimension from accuracy is a contribution to MLLM evaluation methodology that could be operationalized in future work through systematic consistency-vs-accuracy tradeoff analyses across model families and task types.

The limitation is that the paper treats this as a single case study without controlling for temperature, prompt sensitivity, or sample size. Four queries on one image is anecdotal, not conclusive. The paper does not report whether Gemini's consistency generalizes to other counting tasks or whether it is an artifact of a particular image. Nevertheless, as a conceptual probe, the experiment raises a question that the broader MLLM evaluation literature had underemphasized: is a model that is consistently wrong preferable to one that is inconsistently right, and under what deployment conditions?

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use a curated set of 97 test cases (images with associated prompts) drawn from the "Dawn of LMMs" study by Yang et al. (2023), with the exception of images in Section 11 which are sourced from the internet. The test cases span ten capability dimensions (basic recognition, text recognition, image reasoning, textual reasoning, integrated image-text understanding, object localization, temporal understanding, multilingual capabilities, and seven industry application scenarios). The paper states in Section 1.4: "All our data is sourced from 'The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)' (except for the images in Section 11, which are sourced from the internet). We have utilized their images, GPT-4V's prompts, and corresponding results." This means the GPT-4V responses are not the current authors' own generations but rather reference data from a separately conducted study, providing a fixed baseline for comparison that avoids the risk of post-hoc prompt optimization or cherry-picking to favor one model.

Base models. The comparison is between two proprietary, closed-source multimodal large language models: Google's Gemini (specifically Gemini Pro, as indicated in Section 12) and OpenAI's GPT-4V(ision). The paper states in Section 12 that the findings represent a comparison with "Gemini Pro" and expresses anticipation for the release of "Gemini Ultra" which is expected "to bring more possibilities to the field of visual multimodal applications." Both models are accessed through their public chat interfaces—GPT-4V through its vision-enabled interface supporting multi-image sequential input, and Gemini through its multimodal interface with single-image input, image/link output capability, and YouTube-linked video understanding (as documented in Section 1.2). Neither model's internal architecture, parameter count, or training data is accessible to the researchers; the comparison is entirely behavior-based.

Metrics. The primary evaluation is qualitative rather than quantitative. Each response from both models is classified into one of two categories: "Satisfactory Answers" (indicated by blue highlighting in the figures) or "Unsatisfactory Answers" (indicated by red highlighting). The classification criteria, based on examination of the figures, include: (1) factual accuracy—is the described object, landmark, text, or quantity actually present in the image and correctly identified?; (2) task completion—does the response directly address the question asked in the prompt?; (3) absence of significant hallucination—are fabricated details, incorrect quantities, or impossible inferences absent from the response?; and (4) format compliance—does the output follow any specified format requirements (e.g., JSON for insurance reports, coordinate tuples for object localization)? The classification is performed by human researchers through consensus, without a formal inter-annotator agreement protocol reported. No aggregate numerical scores, accuracy percentages, or confusion matrices are computed across the test set. The paper's findings are expressed as qualitative behavioral patterns (e.g., "Gemini excels in providing detailed, expansive answers accompanied by relevant imagery and links" from Section 1.5) rather than as ranked scores.

Baselines. The primary comparison baseline is GPT-4V's performance as previously recorded in the "Dawn of LMMs" study. The paper does not introduce additional baselines beyond this head-to-head comparison. For the integrated-use experiments in Section 11, the implicit baselines are each model's individual performance on the same task (GPT-4V alone for product recommendation, Gemini alone for comprehensive recognition), though these single-model baselines are not formally evaluated—the paper states qualitatively that each model alone fails at the other's specialized function but does not present side-by-side evidence. The paper does not compare against any other multimodal models (open-source alternatives like LLaVA, MiniGPT-4, or InstructBLIP are mentioned in the introduction but not evaluated) or against text-only LLM baselines.

Generation budget / compute accounting. The paper does not employ a formal compute budget or control for inference cost across models. All responses are single-turn generations with no mention of sampling temperature, beam search parameters, or computational resource allocation. The paper does not report the number of tokens generated per response, the wall-clock time for inference, or any cost metric. This means differences in response verbosity (Gemini's responses are consistently longer) are not controlled for or factored into any efficiency comparison. The paper also does not account for the computational cost of the prompt adaptation strategies (composite image stitching, task decomposition into sequential queries) when comparing models.

Cross-validation / statistical protocol. No cross-validation, statistical significance testing, or confidence interval computation is performed. The evaluation is a single-pass qualitative assessment on the fixed test set of 97 figures. The paper does not repeat queries to assess response stability except for one deliberate exception: the boat-counting self-consistency test in Figure 50, where the same counting query is repeated four times to observe answer consistency. The paper does not report whether this single-instance finding generalizes across other tasks or images, and does not control for sampling temperature (if any) in this repetition.

Prompt adaptation protocol. A critical methodological element is that prompts originally designed for GPT-4V's multi-image sequential interface were adapted for Gemini's single-image constraint. Section 1.3 states: "The prompt design under consideration is tailored for GPT-4V, and direct input into Gemini may yield unsatisfactory responses. In such cases, adjustments to Gemini's prompt are made to align with the input requirements of its architecture." The paper does not specify which prompts were adjusted, what specific modifications were made, or what criteria triggered adjustment. This creates a methodological asymmetry: some Gemini responses were generated under adapted prompts while GPT-4V responses used the original prompts, meaning performance differences could reflect prompt quality rather than underlying capability differences. The paper's description of prompt adaptation is limited to structural changes (image compositing, task decomposition) rather than semantic rephrasing, but this is not explicitly verified across all test cases.

Input modality workarounds. For multi-image tasks, the paper employs two strategies for Gemini: (1) stitching multiple images into a single composite input, and (2) decomposing multi-step queries into separate single-image queries with manual aggregation (e.g., Figure 46 for receipt tax calculation, Figure 47 for beer-and-menu matching). These adaptations fundamentally change the task demands—the composite approach requires the model to parse sub-image boundaries, handle reduced effective resolution, and infer temporal or logical relationships from spatial arrangement rather than sequential presentation; the decomposition approach removes the multi-image integration burden entirely and tests only single-image recognition. The paper does not systematically compare the two adaptation strategies or assess their relative impact on performance, and does not report which strategy was used for which test case unless explicitly noted in a figure caption.

---

Main Qualitative Results

The paper reports its findings as qualitative behavioral patterns organized by capability dimension rather than as aggregate numerical comparisons. The results below are organized following the paper's section structure, with specific figure references for each observation.

Image Recognition and Understanding (Section 2)

Basic object recognition (Figure 1). Both models succeed at recognizing scene contents and complying with output format constraints (producing descriptions of specified word counts or with specified initial letters). The paper reports that "both GPT-4V and Gemini are both capable of accurately recognizing basic objects and providing textual descriptions as required." The figure caption notes that after prompt adjustment, "both GPT-4V and Gemini are able to provide satisfactory responses, indicating the ability to comprehend images and respond according to instructions." This represents the baseline finding that for simple single-image recognition with format constraints, the two models are qualitatively comparable.

Landmark recognition (Figures 2–3). Both models correctly identify all four landmarks tested (Kinkaku-ji Temple, Lombard Street, Manhattan Bridge, Trump Tower interior). The paper identifies a behavioral pattern: "GPT-4V generates concise descriptions, whereas Gemini tends to provide more detailed narratives" (Figure 2 caption) and "Gemini can display other images and links related to the landmark" (Section 2.2). Both models successfully identify the interior of Trump Tower, demonstrating recognition capability even for indoor tourist attractions where external architectural cues are absent. Gemini is observed to provide Wikipedia links and related imagery as supplementary outputs—a capability GPT-4V lacks.

Food recognition (Figures 4–5). Both models correctly identify a range of dishes spanning Chinese, Japanese, Western, and North American indigenous cuisines. The paper reports that "both models exhibit remarkable capacities in identifying a vast array of culinary dishes" and "further extending their recognition abilities to specific elements like ingredients, garnishes, and cooking techniques" (Figure 5 caption). Gemini again provides supplementary outputs: links to recipes (Figure 4).

Logo recognition (Figures 6–8). For both isolated logo identification and in-the-wild logo recognition (logos in natural scenes with occlusions, varied lighting, and orientations), both models perform accurately. The paper notes minor errors: in Figure 7, "GPT-4V's description shows minor instances of hallucination" (identifying a straw in a coffee cup where none exists, describing a can as a bottle), and in Figure 8, "GPT-4V's description in the first case exhibits a slight error." Gemini's responses are consistently more detailed, with structured breakdowns of logo elements and brand history. The paper's caption for Figure 8 notes that "both models manifest robust capabilities in recognizing logos in diverse situations."

Abstract image recognition (Figure 9). The task involves identifying which of ten tangram shapes best represents a flying goose and describing the other shapes. This is one of the first tasks where a clear performance gap emerges: "GPT-4V provides reasonable associations, the associations given by Gemini do not appear as logical" (Figure 9 caption). GPT-4V correctly identifies tangram 7 as the flying goose; Gemini selects tangram 1 with an implausible rationale. The paper attributes this to "Gemini's limited ability to recognize more abstract objects" and notes that "it's possible that combining multiple images into a single input image may have resulted in a decrease in Gemini's performance" (Section 2.5). This is significant because it represents the first clear case where the composite-image workaround may be degrading Gemini's performance on a task that requires comparing multiple sub-figures.

Scene understanding (Figure 10). Both models provide adequate descriptions of autonomous driving scenes, identifying road elements, traffic signs, and weather conditions. The paper reports that "GPT-4V's responses are more accurate, while Gemini's responses are more detailed." A specific error is noted: "Gemini has some discrepancies in understanding weather conditions" (Section 2.6)—Gemini describes clear blue sky when the image shows cloudy conditions, an error classified as unsatisfactory (red highlighting). The paper's overall assessment is that "the performance of both models is quite comparable" for this task class.

Counterfactual examples (Figure 11). Three images are presented with prompts asking about non-existent objects (a red ball near a cat, blue feathers on a bird's wing, a blue ceiling fixture). Both models correctly identify the absence in all three cases. The paper states: "Both models are capable of providing factual descriptions related to the scenes and objects depicted in the images" and "both models can identify that the fabricated object does not exist" (Section 2.7). This is a positive finding for both models: they resist hallucination when directly asked about absent objects.

Object counting (Figure 12). This task reveals a clear asymmetry. For counting apples in an image, GPT-4V correctly identifies 11 apples with step-by-step row-by-row counting guidance. Gemini initially miscounts: "there are 12 apples in total" with incorrect row counts (4, 5, 3 rather than 4, 4, 3). When corrected with explicit per-row counts, Gemini acknowledges the correction. For a second counting task (apples in a different arrangement), GPT-4V correctly identifies 7 apples, while Gemini counts 5. For counting oranges, GPT-4V correctly identifies 3, while Gemini outputs 5. The paper concludes: "Gemini's counting ability is relatively weaker compared to GPT-4V" and attributes this to its "relatively weaker multimodal memory performance" (Section 2.8). Notably, all of Gemini's counting responses are classified as unsatisfactory (red-highlighted).

Spot the difference (Figure 13). Given two similar stick-figure dinosaur drawings with three differences, neither model identifies all differences. GPT-4V identifies two differences (spikes, tail) but confuses an eye for a mouth. Gemini identifies only two differences but incorrectly infers that the right image contains a turtle (rather than a dinosaur), producing an unsatisfactory response. The paper states: "neither GPT-4V nor Gemini delivers completely satisfactory results in this task" (Section 2.9).

Text Recognition and Understanding in Images (Section 3)

Scene text recognition (Figures 14–15). Across a diverse set of scene text images (stock tickers, shop signs, informational displays), both models demonstrate strong OCR capability. The paper states in Section 3.1 that "both models exhibit good performance." Minor errors are noted: in Figure 14, Gemini misreads a large number as "3,642,039,031,955" (should be "3,642,039,031,055")—a single-digit error classified as unsatisfactory. In Figure 15, Gemini provides an incomplete reading of a COVID-19 testing sign, outputting "W Gor" for what should be "W @UW Medicine." The paper does not quantify error rates across the full set of scene text examples.

Zero-shot vs. few-shot dashboard reading (Figure 16). This experiment tests both models' ability to read speed from automobile dashboards, first in zero-shot and then in a 2-shot setting with worked examples. In the zero-shot setting, both models fail: GPT-4V reads 40 mph (incorrect), and Gemini reads 140 mph (incorrect) on the first dashboard. In the 2-shot setting, after seeing two examples of how to read the needle position, GPT-4V correctly identifies 60 mph on the test dashboard. Gemini incorrectly identifies 131 mph. The paper attributes Gemini's difficulty to "the lack of a memory mechanism" (Section 3.1) and notes that "both models fail to comprehend the dashboard" in zero-shot, but only GPT-4V benefits from the in-context examples. This is a significant finding: it suggests that Gemini's inability to retain and apply patterns from previous examples within a conversation—a direct consequence of its non-memory architecture—limits its few-shot learning capability for visual tasks.

Equation recognition (Figure 17). For converting mathematical formulas from images to LaTeX code, both models perform well on simple equations but struggle with complex ones. The paper identifies a specific deficiency in Gemini: "Gemini, in particular, fails to recognize the summation symbol" (Section 3.2). Both models produce incorrect LaTeX for complex equations involving summation notation and multiple nested expressions. GPT-4V's outputs are described as "slightly closer to the actual equation" (Figure 17 caption). Neither model's results are classified as fully satisfactory on the complex cases.

Chart text recognition (Figures 18–20). For generating Python code to reconstruct a line chart (Figure 18), GPT-4V produces a reconstruction that is "more similar to the original chart," with Gemini providing "a more chaotic shape." For table reading tasks (Figure 19), the results reverse: "Gemini outperforms GPT-4V in generating the correct table based on a given picture. GPT-4V exhibits errors in number recognition" (Figure 19 caption). For reading a trend line graph (Figure 20), both models perform poorly: "Both models show poor performance in recognizing charts, misjudging colors and values" (Figure 20 caption). Gemini is noted to be "slightly superior as it provides correct results on Steps 5 and 6, despite errors in the intermediate process details."

Image Reasoning Abilities (Section 4)

Humorous meme understanding (Figure 21). Both models correctly interpret two visual memes (a procrastination meme with Kermit the Frog, a "student handwriting deterioration" meme) and one humorous image (a rhinoceros painting). The paper states: "Both GPT-4V and Gemini can understand the question textually and visually, and gather necessary information to answer the question" and "both models display a correct understanding of the two joke images, demonstrating an ability to comprehend human humor" (Section 4.1). No performance gap is identified for humor understanding.

Multimodal knowledge and commonsense (Figures 22–24). Both models successfully answer science questions using visual inputs: identifying oceans intersected by the prime meridian (Figure 22), determining which gas sample has higher temperature from particle diagrams (Figure 22), identifying the northernmost U.S. state from a map (Figure 23), identifying producers in a food web (Figure 23), and explaining the distinction between evaporation and evapotranspiration from a water cycle diagram (Figure 24). Minor errors are noted: in Figure 22, "there are omissions in Gemini's description" (Gemini identifies only the Atlantic Ocean for the prime meridian question, omitting the Southern and Arctic Oceans that GPT-4V correctly lists). In Figure 23, both models make errors on the geography question. Overall, the paper assesses that "both models possess strong knowledge for answering general scientific queries, yet Gemini struggles with small text extraction" (Section 4.2), attributing Gemini's partial answers to difficulty extracting small text from composite images.

Detective reasoning (Figure 25). Given an image of a person's room and asked to infer characteristics, both models produce reasonable deductions about the occupant's likely age, interests, occupation, and lifestyle. GPT-4V provides a concise answer; Gemini provides "a more detailed thought process" (Section 4.3). The paper presents both responses as satisfactory.

Part-object association (Figure 26). The task is to combine three separate visual elements (cardboard shapes, jeans) into a coherent whole (a person). GPT-4V correctly describes how to arrange the parts to form a boy. Gemini provides a different interpretation: "The cardboard box is the body of a robot, and the robot's head is coming out of the box." The paper classifies GPT-4V's response as more satisfactory: "GPT-4V provides a more accurate and detailed answer" (Section 4.4) and notes that "GPT4-V and Gemini understand how to combine the three parts in a reasonable way. Of course, here GPT-4V provides a more satisfactory answer." This represents a case where Gemini provides a plausible but incorrect interpretation, while GPT-4V identifies the intended composition.

Intelligence tests—Raven's Progressive Matrices (Figures 27–30). This is the dimension where the most consistent performance gap is documented. Across five RPM-style pattern completion tasks:

• Figure 27 (WAIS-style classification): Gemini fails to correctly identify the odd-one-out among visual symbols. The paper states: "In response to example questions from the Wechsler Adult Intelligence Scale, Gemini performed poorly. This suggests that its ability to recognize and compare multiple images might be limited."

• Figure 28 (Simple pattern completion): Gemini fails on two tasks: for rotation-based pattern completion, Gemini selects an incorrect shape; for a more complex pattern, Gemini selects figure H with incorrect reasoning. The paper states: "Gemini performed poorly. This suggests that its ability to recognize, compare multiple images, and discern their inherent connections might be limited."

• Figure 29 (2×2 matrix completion): Both models select the correct answer (c), but Gemini provides incorrect reasoning—it describes a pattern of alternating shapes that does not match the actual task logic. The paper states: "both GPT-4V and Gemini generated the correct results. However, Gemini failed to provide a reasonable explanation."

• Figure 30 (Step-by-step guidance): When given explicit step-by-step guidance that decomposes the pattern-finding process, Gemini eventually produces the correct answer. The paper interprets this as evidence that "Gemini has the ability to compare differences between images, but it is not very familiar with the pattern of RPM tests" (Figure 30 caption).

The consistent pattern is that Gemini underperforms GPT-4V on abstract visual reasoning tasks requiring comparison across multiple sub-figures, with two potential explanations: (1) the composite-image workaround degrades the model's ability to parse the spatial arrangement of sub-figures, and (2) Gemini's multi-image comparison capability is genuinely weaker than GPT-4V's, even when presented in a single composite. The paper leans toward the first explanation but does not isolate the two.

Emotional intelligence tests (Figures 31–35). Both models demonstrate competence in understanding and generating emotional content:

• Emotion recognition (Figure 31): Both models correctly identify the emotional valence of images (a serene sunset evoking peace, a dark staircase evoking unease).

• Aesthetic judgment (Figures 32–33): Both models can evaluate visual aesthetics according to societal norms, identifying more aesthetically appealing images from pairs. Gemini provides more detailed aesthetic justifications.

• Emotion-conditioned generation (Figures 34–35): For generating descriptions designed to elicit specific emotional responses (lessening desire to visit a location, creating feelings of relaxation), "Gemini can generate better text based on the perceived or desired emotions than GPT4-V" (Figure 34 caption). Gemini produces longer, more evocative narratives with sensory details and atmospheric elaboration.

The paper's overall assessment in Section 4.6 is that "both models have a fairly accurate grasp of emotions and aesthetic judgment and are capable of producing different emotional outputs according to user requirements."

Textual Reasoning in Images (Section 5)

Visual math ability (Figure 36). Both models solve simple algebraic and geometric problems presented as images. For a quadratic equation, both correctly identify the two solutions. For a Pythagorean theorem problem, both correctly compute the hypotenuse length. The paper states: "For relatively simple problems, both models provided satisfactory solutions" (Section 5.1). No systematic difference in mathematical reasoning accuracy is documented.

Table and chart reasoning (Figures 37–41). Performance is mixed and task-dependent:

• Flowchart interpretation (Figure 37): Both models successfully describe and translate a flowchart into Python code. No performance gap is identified.

• Chart reading (Figure 38): Both models correctly interpret bar charts and line graphs, answering specific data extraction questions accurately. "Both GPT-4V and Gemini show promising results in understanding the details in the chart" (Figure 38 caption).

• Table reasoning (Figure 39): Both models correctly answer questions requiring data aggregation from tables (identifying families earning above a threshold, computing profit margins). No significant error is reported.

• Visual referring prompting (Figure 40): When user-drawn circles on a document need to be interpreted, "GPT-4V generate better result than Gemini" (Figure 40 caption). GPT-4V correctly identifies the anomalous comparison trend in a circled research results table; Gemini provides incorrect analysis.

• Paper experiment table (Figure 41): For understanding scientific conference slides with user markings, "GPT-4V can better understand user mark than Gemini to help user understand scientific conference slides" (Figure 41 caption). Gemini produces task descriptions that do not match the actual content of the slides.

• Complex experimental tables: The paper notes in Section 5.2 that "in the context of some experimental tables, the Gemini model struggles to provide accurate answers. This may also be attributed to image resolution issues, as the images are composed of multiple stitched pictures." This again implicates the composite-image workaround as a potential confound.

Document understanding (Figures 42–45). Both models demonstrate competence in interpreting diverse document types:

• Floor plan reading (Figure 42): Both models accurately identify rooms, dimensions, and layout elements. Gemini provides a more exhaustive room-by-room breakdown; GPT-4V provides a concise overview.

• Earth science diagram (Figure 43): Both models correctly label Earth's interior layers. GPT-4V's layer-to-number mapping differs from Gemini's—this appears to reflect ambiguity in the original diagram rather than model error.

• Poster interpretation (Figure 43): Both models correctly identify the dish (Wuhan hot dry noodles) and its city of origin. Gemini provides a more extensive cultural description.

• Academic paper understanding (Figure 44): Both models successfully summarize the first page of a technical paper ("GPT4Point"), identifying its main contributions. The paper does not audit factual accuracy of the summaries against the actual paper content.

• Web page GUI understanding (Figure 45): Both models correctly identify GUI elements (a microphone icon for voice search on Bing, the locations of interface buttons). The paper reports: "Both Gemini and GPT-4V success in helping user to understand the web page" (Figure 45 caption).

The paper's overall assessment in Section 5.3 is that "both models exhibit comparable efficacy in these tasks. Notably, Gemini tends to offer more elaborate responses, yet it falls short in terms of precision, underscoring an area for potential improvement."

Integrated Image and Text Understanding (Section 6)

Interleaved image-text inputs (Figures 46–47). This section provides the clearest evidence of how Gemini's single-image constraint creates task performance disparities:

• Receipt tax calculation (Figure 46): The task is to compute total tax from three separate receipts. GPT-4V (native multi-image mode) correctly processes all three sequentially and computes the total: $3.72 +$42.23 + $4.58 =$50.53. Gemini, given the three receipts as a composite image, produces an incorrect total: $4.58 (apparent misinterpretation of which receipt value corresponds to tax). When the task is decomposed—each receipt fed individually with separate queries—Gemini correctly extracts tax from each receipt:$3.72, $42.23, and$4.58. The paper's caption states: "GPT-4V can accurately provide output, while Gemini's responses to the concatenated image are not satisfactory. However, it performs well in providing individual answers for each image."

• Beer and menu matching (Figure 47): The task is to identify beer from a table image, then look up its price from a separate menu image. GPT-4V correctly identifies the beer as Magna, counts two bottles, and computes total cost of $12. Gemini, given the composite input, incorrectly identifies the beer and price. When decomposed into sequential queries, Gemini correctly identifies the Magna beer and computes$12 for two bottles. Again, the composite input produces unsatisfactory results; decomposed queries produce satisfactory results.

These findings are critical because they demonstrate that Gemini's failures on multi-image tasks are primarily interface-access failures rather than capability failures. When the multi-image integration burden is removed (by feeding images individually and having a human aggregate), Gemini performs comparably to GPT-4V. The paper's interpretation is that Gemini's underlying single-image recognition capabilities are sufficient; what is missing is the architectural support for multi-image reasoning.

Text-to-image generation guidance (Figures 48–50). For the task of iteratively improving image generation prompts:

• Self-reflection improvement (Figures 48–49): Both models can analyze generated images against target descriptions and suggest prompt improvements. In Figure 48, "GPT-4V reflects the error in the initial prompt that it does not mention the dog's breed, and makes the correct revision, while Gemini generates a better one with breed mentioned" (Figure 48 caption). In Figure 49, both models provide detailed suggestions for improving generated images.

• Self-consistency counting (Figure 50): In the methodologically distinctive boat-counting test, GPT-4V's four repeated queries produce three different answers (4, 4, 4, and 6 boats), while Gemini consistently outputs 5 boats across all four attempts—but 5 is incorrect. The paper states: "GPT-4V's four responses provided three different answers, even though two of them were correct; in contrast, all four of Gemini's responses were consistent, albeit incorrect. This suggests that Gemini exhibits stronger consistency in its results" (Figure 50 caption). This is presented as an interesting behavioral property rather than as a performance advantage.

Object Localization (Section 7)

Real-world object localization (Figures 51–52). The task requires models to output (x1, y1, x2, y2) normalized bounding box coordinates for specified objects. Both models demonstrate fundamental competence in producing plausible coordinate tuples. For localizing a blue Subaru SUV (Figure 51), both generate reasonable bounding boxes. For a black Audi car (Figure 52), both produce plausible coordinates. The paper's assessment is that "both models demonstrate commendable performance" (Section 7.1) and "both models are capable of achieving in localizing objects in real scene."

Abstract image localization (Figure 53). For localizing the head of a flying goose tangram figure, GPT-4V produces a plausible bounding box (0.42, 0.15, 0.58, 0.35). Gemini's coordinate output (0.37, 0.36, 0.47, 0.47) is assessed as incorrect—the paper states: "GPT-4V can localize the head of the abstract goose, while Gemini failed in this case" (Figure 53 caption). This mirrors the abstract image recognition finding (Figure 9) and suggests that Gemini's difficulty with abstract shapes extends to spatial localization in addition to recognition.

Temporal Video Understanding (Section 8)

Action recognition (Figure 54). Given a motion-capture sequence of a person performing push-ups, GPT-4V correctly describes the action and the frame-by-frame progression. Gemini provides a description that includes a hallucinated detail: "The person is wearing a virtual reality headset." This is unsatisfactory—the paper's caption states: "Gemini incorrectly infers that the subject is wearing a VR headset." The paper's summary in Section 8.1 notes: "GPT-4V's ability to sequentially process multiple images translates into more accurate results. This enhanced accuracy may be attributed to its capability to synthesize information across a series of frames, thereby offering a more coherent and precise interpretation of continuous actions."

Temporal ordering (Figure 55). The task requires ordering five scrambled keyframes of a sushi-making process. GPT-4V correctly sequences them: e (mixing rice) → c (placing rice on seaweed) → b (adding fillings) → a (cutting the roll) → d (finished sushi on plate). Gemini's ordering (A → D → B → E, with C omitted) is incorrect. The paper states: "Only GPT4-V is able to find the correct order while Gemini failed in this case" (Figure 55 caption). This is a particularly clear demonstration of the single-image constraint's impact: the scrambled keyframes, when composited into a single image, lose their sequential information, and Gemini cannot reconstruct the temporal order from the spatial arrangement.

Multilingual Capabilities (Section 9)

Multilingual image description (Figures 56–59). Both models successfully describe images in multiple languages (Chinese, French, Czech, and 20 additional languages) when prompted in the target language or in English. Minor differences are observed: Gemini occasionally includes supplementary links alongside descriptions (e.g., Figure 56 includes a link to a stock photo source). For the 20-language description task (Figures 58–59), both models produce coherent descriptions in all requested languages. The paper's assessment is that "both models demonstrate their versatility" (Section 9.1) and "both models provide reasonable results." No systematic errors or language-specific failure patterns are documented.

Multilingual scene text recognition (Figures 60–63). Both models successfully recognize scene text in diverse scripts: Arabic calligraphy (Figure 60), French museum signage (Figure 60), Chinese-English bilingual station signs (Figure 60), multilingual welcome signs (Figure 61), Portuguese posters (Figure 61), Greek postage stamps (Figure 61), Japanese text (Figure 62), Chinese dish labels (Figure 62), French wine labels (Figure 62), mixed-language Wikipedia screenshots (Figures 62–63), and Portuguese UI text (Figure 63). The paper consistently reports satisfactory performance for both models on all tested multilingual text examples. The overall assessment in Section 9.2 is that "both GPT-4V and Gemini perform well, demonstrating their ability to understand multiple languages."

Industry Applications (Section 10)

Defect detection (Figures 64–66). Across five defect types (hazelnut hole, fabric tear, stripped screw, car bumper dent, cracked nut, broken capsule, dirty/missing wheel hubcap), both models identify defects with reasonable accuracy. Gemini is noted to provide more detailed defect descriptions. In Figure 65, for broken capsule detection, "Gemini outshines by exhibiting a stronger ability to spot defects." In Figure 66, a comparative defect detection task, both models fail to detect a missing hubcap when shown a single tire in isolation, but both succeed when shown two tires side-by-side (one with hubcap, one without). The paper interprets this as evidence that "the ability of both GPT-4V and Gemini to detect anomalies can be reinforced through comparison or presence of multiple instances of similar defects" (Figure 66 caption).

Grocery checkout (Figure 67). GPT-4V identifies a more comprehensive list of items in a shopping basket (pink tulips, salmon fillets, strawberries, apple, Greek yogurt, sliced ham, raspberries) but includes some items not clearly visible. Gemini identifies only four items (apples, strawberries, yogurt, ham) but does so accurately and supplements with generated images of each item. The paper notes: "while GPT-4V offers breadth in its findings, Gemini offers accuracy, demonstrating varied strengths in performing such tasks" (Figure 67 caption). Gemini's image generation capability is presented as a practical advantage for this retail application.

Auto insurance (Figures 68–70). Both models provide damage assessments for car accident images, describing the severity and affected components. GPT-4V provides concise assessments; Gemini provides "more detailed responses and more comprehensive recommendations" (Section 10.3). For generating an incident report in JSON format (Figure 70), GPT-4V successfully produces the requested structured output: {"make": "Maruti Suzuki", "model": "Celerio", "license plate": "KL 52 N 5305", "damage description": "Front bumper, hood, and windshield damaged. Airbags deployed.", "estimated cost of repair": "N/A"}. Gemini refuses or fails to generate the structured format: "As a language model, I'm not able to assist you with that." The paper states: "Only GPT-4V is able to generate an insurance report according to the given format and content requirements, while Gemini does not exhibit the ability to produce outputs based on a specific format" (Figure 70 caption).

Customized captioner (Figure 71). The task is to describe individual objects (snail shell, snail, frog, turtle) with reference to their appearance in a composite scene. GPT-4V produces "more comprehensive, accurate, and orderly captions." Gemini "falls short, failing to recognize the presence of a snail shell in the left image and incorrectly describing the position of the snail" (Figure 71 caption). The paper attributes Gemini's difficulty to its inability to retain visual context from the scene image when describing individual crop-outs, again implicating the lack of multi-image memory.

Image generation evaluation (Figures 72–74). Both models successfully rate generated images on a 1–10 alignment scale against text prompts, providing consistent scores with explanations. Gemini tends to assign slightly higher scores and provides "more detailed rationales for its evaluations" (Figure 72 caption). In Figure 73, Gemini incorrectly identifies some aspects of an image but still provides detailed justification. Both models' evaluations are classified as satisfactory for all tested prompt-image pairs.

Embodied agent (Figures 75–78). For indoor navigation tasks where the model must reason about its position and plan movements through a home environment:

• Coffee machine operation (Figure 75): GPT-4V correctly maps menu buttons to physical buttons and identifies the correct button for an 8-ounce coffee. "Gemini provides entirely incorrect answers" (Figure 75 caption)—it misidentifies which button corresponds to which function.

• Indoor navigation (Figures 76–78): GPT-4V produces concise, correct navigation instructions (e.g., "turn right and move forward towards the hallway"). Gemini produces "prolonged and unnecessary directions" that include correct elements but also over-elaboration. For identifying the kitchen location (Figure 77), GPT-4V correctly identifies the kitchen on the right; "Gemini produces incorrect analysis and directions" (Figure 77 caption). For the final fridge approach (Figure 78), both models provide usable instructions, with GPT-4V being "notably more concise."

The paper's assessment in Section 10.6 is that "GPT-4V consistently provides more accurate responses. However, it is noteworthy that Gemini offers a more detailed reasoning process." This pattern—GPT-4V more accurate, Gemini more verbose—recurs across the embodied agent scenarios.

GUI navigation (Figures 79–95). This is the most extensive industry section, covering three scenarios: web browsing for a recipe (Figures 79–83), online shopping for a keyboard (Figures 84–92), and notification/video understanding (Figures 93–95).

• Web browsing—Mapo Tofu recipe (Figures 79–83): In the initial step, GPT-4V correctly identifies the Google Chrome icon and initiates a web search. Gemini fails to recognize the GUI entirely—"Gemini fails to recognize the information from the GUI entirely" (Figure 79 caption). In subsequent steps, GPT-4V provides accurate, succinct instructions for typing in the search bar, clicking search results, and navigating to the recipe. Gemini's responses are mixed: sometimes correct but verbose (Figure 80), sometimes providing "a vague hint only" (Figure 81), sometimes producing "redundant and unsatisfactory results" (Figure 88). When both models are asked to describe the final recipe page (Figure 83), both produce detailed descriptions with "each exhibiting some minor inaccuracies in the details."

• Online shopping—ergonomic keyboard (Figures 84–92): GPT-4V correctly navigates through the Amazon app: opening the app, searching for "ergonomic keyboard," applying the $50–$100 price filter, selecting a product, adding to cart, and proceeding to checkout. Gemini's performance is inconsistent: it correctly identifies some steps (Figure 85 search bar) but makes errors on others—it "mistakenly perceives the App Store as a shopping application" (Figure 84), "mistakenly perceives the search results as a product information page" (Figure 86), and in later steps, "advises consideration of whether the product fits needs, forgetting its role as a robot, and fails to provide operational directions" (Figure 91). GPT-4V maintains task focus throughout the multi-turn interaction; Gemini increasingly shifts to providing advice rather than executing actions.

• Notification understanding (Figures 93–94): Both models correctly interpret phone notification content and suggest appropriate responses. Gemini provides more comprehensive option enumeration. For computer desktop notifications (Figure 94), Gemini misinterprets the GUI state—"Gemini misinterprets the GUI, wrongly assuming that a web browser window and a terminal are open, leading to some incorrect directions" (Figure 94 caption)—while GPT-4V correctly identifies the notification and suggests the appropriate click action.

• Video watching (Figure 95): From two keyframes of a TikTok video, both models accurately describe the content (a corgi dog transitioning from sleeping to working). Gemini additionally extracts metadata (breed, account name, view count) from the GUI. "Both GPT-4V and Gemini accurately describe the information and content provided by two frames of the video. Their analyses and explanations are not identical, but each makes sense in their own way" (Figure 95 caption).

The overall pattern in GUI navigation mirrors the broader findings: GPT-4V provides more accurate, task-focused operational guidance, while Gemini provides more verbose responses that sometimes include helpful supplementary information but also exhibit higher rates of misinterpretation and task drift.

Integrated Use of Both Models (Section 11)

Product identification and recommendation (Figure 96). The two-stage pipeline (GPT-4V describes room contents → Gemini recommends product links) succeeds in producing specific, linked product recommendations for furniture and decor items. The paper reports that neither model alone achieves this: "GPT-4V struggles to provide recommended links for products, while Gemini, in the initial phase, faces challenges in comprehensively identifying objects" (Section 11.1). The combination produces working retailer links for specific items (TV, media console, mirror, pouf, armchair, rug, plant) with partial matching to actual products in the image.

Multi-image recognition and story generation (Figure 97). The pipeline (GPT-4V identifies seasons → Gemini generates Makoto Shinkai-style story) produces a coherent multi-paragraph narrative that weaves together the four seasonal scenes. The paper states: "GPT-4V accurately summarizes and identifies the content of the four sub-images, while Gemini generates a coherent narrative with extended length, aligning with both the content and stylistic requirements of the image" (Section 11.2). The single-model baselines are described qualitatively as producing "unclear content summarization or the generation of overly formatted and brief narrative sequences" but are not presented in the figure.

---

Ablation Studies and Robustness Checks

The paper does not conduct formal ablation studies in the traditional sense (systematically removing or varying components to measure their contribution). However, several experiments serve as implicit ablations or robustness probes:

Zero-shot vs. few-shot prompting (Figure 16, dashboard reading). By testing the same dashboard-reading task in both zero-shot and 2-shot settings, the paper implicitly ablates the effect of in-context examples. The finding: GPT-4V benefits substantially from examples (moving from incorrect zero-shot to correct 2-shot), while Gemini does not (incorrect in both settings). This isolates Gemini's difficulty with retaining and applying pattern information from examples—a direct consequence of its non-memory architecture.

Gradual guidance in RPM tasks (Figure 30 vs. Figures 27–29). The paper tests Gemini on RPM tasks both with direct "find the pattern" prompts (where it fails) and with step-by-step decomposed guidance (where it eventually succeeds). This implicit ablation suggests that Gemini possesses some multi-image comparison capability but requires explicit decomposition to access it—the composite-image presentation format may obscure the sequential reasoning steps that GPT-4V performs naturally with its sequential interface.

Composite vs. decomposed multi-image input (Figures 46–47). For both the receipt tax calculation and beer-menu matching tasks, the paper tests Gemini under two conditions: composite single-image input (where all sub-images are stitched together) and decomposed sequential single-image querying (where each sub-image is fed separately with its own prompt). The finding is consistent: Gemini fails on the composite input but succeeds when the task is decomposed into single-image queries with human aggregation. This is arguably the paper's most informative implicit ablation—it isolates the composite-image workaround itself as a performance bottleneck, demonstrating that Gemini's underlying single-image recognition capabilities are sufficient for the component subtasks.

Self-consistency under repeated queries (Figure 50, boat counting). The four repeated counting queries on the same image serve as a robustness check for response stability. The finding—GPT-4V is inconsistent (three different answers across four attempts) while Gemini is consistent (same answer all four times)—probes a behavioral property orthogonal to accuracy. This is a robustness check in the sense that it asks: if you query the model repeatedly on the same input, do you get the same answer?

Single vs. paired defect detection (Figure 66). The paper tests both models on detecting a missing hubcap first with a single tire image (where both fail) and then with a paired comparison image (where both succeed). This probes whether anomaly detection capability is enhanced by comparative context, and the finding is positive for both models—it is not specific to one.

Emotion-conditioned generation (Figures 34–35). By testing both models' ability to generate descriptions that elicit opposite emotional responses (lessening desire vs. creating relaxation for similar image types), the paper implicitly probes whether emotional output can be conditioned by prompt instruction. Both models demonstrate this capability, with Gemini generating more elaborate emotion-targeted text.

The paper does not ablate: the effect of prompt wording on single-model performance; the effect of temperature or sampling parameters (if any); the effect of composite image arrangement (layout, resolution, sub-image ordering) on Gemini's performance; the contribution of GPT-4V's description accuracy to the integrated-use pipeline's success; or the sensitivity of findings to the specific test cases selected from the "Dawn of LMMs" dataset.

---

Critical Assessment

Does the evidence support the paper's central qualitative claims?

The paper's overarching claim is that the two models exhibit systematic, characterizable behavioral differences with GPT-4V being "slightly stronger than Gemini Pro" overall (Section 12) but with Gemini possessing distinctive strengths in detailed description, supplementary output generation, and creative text production. The evidence for systematic behavioral differences is well-supported across most dimensions: the verbosity-precision tradeoff is documented consistently across recognition (Section 2), reasoning (Section 4), and application (Section 10) tasks; Gemini's supplementary output capability (links, images) appears repeatedly in landmark, food, logo, and product scenarios; and GPT-4V's conciseness advantage is evident in structured output tasks (Figure 70) and navigation tasks (Section 10.6).

However, the evidence for the claim that GPT-4V is "slightly stronger" overall rests on a qualitative aggregation of task-level judgments that is not formalized. The paper does not compute a win-loss-tie ratio or an aggregate accuracy score. The red/blue highlighting provides a visual impression of relative performance (GPT-4V has visibly fewer red-highlighted responses), but without a formal counting of satisfactory vs. unsatisfactory classifications per model, the "slightly stronger" assessment is an author impression rather than a measured result. The paper reports that this finding "aligns with the results reported by Gemini" (Section 12), referencing Google's own technical report which acknowledged performance parity or near-parity with GPT-4V on some benchmarks, but this external validation is limited: benchmark parity does not imply identical behavioral profiles on the qualitative tasks the paper examines.

The interface confound is the central unresolved methodological tension.

The paper's most important empirical finding is that Gemini's performance on multi-image tasks is substantially degraded, and the paper's own decomposition experiments (Figures 46–47) demonstrate that this degradation is at least partially attributable to the composite-image workaround rather than to underlying recognition capability. However, the paper does not systematically disentangle interface constraints from capability limitations. For any test case where Gemini underperforms on a composite image, we cannot determine from the paper's reported data whether the failure is due to:

1. The composite-image format itself (resolution degradation, sub-image parsing difficulty, spatial-arrangement-to-logical-relationship inference burden), or
2. A genuine limitation in Gemini's ability to perform the task even if presented optimally, or
3. Some combination of both.

The paper acknowledges this confound implicitly when it notes, in multiple figure captions and section summaries, that "Gemini's single-image input mode falls short compared to GPT-4V" (Section 1.5) or that failures "could also be attributed to Gemini's inability to recognize multiple images" (Section 4.5). But it does not report controlled experiments that would isolate these factors—for example, presenting GPT-4V with the same composite images given to Gemini to see if GPT-4V's performance also degrades, which would indicate that the composite format itself is the primary difficulty driver.

This is not a minor methodological limitation—it is a fundamental challenge to the interpretability of the comparison. If the composite-image format degrades any model's performance (including GPT-4V's) on multi-image tasks, then Gemini's underperformance relative to GPT-4V in those tasks overstates the capability gap. If, conversely, GPT-4V performs well on composite images (demonstrating that composite format is not inherently prohibitive), then Gemini's failures indicate a genuine comparative weakness. The paper does not run this control experiment, leaving the central interpretative question unresolved.

The GPT-4V "baseline" is not a controlled contemporaneous comparison.

The GPT-4V responses are drawn from a prior published study conducted months earlier (Yang et al., 2023). This introduces a temporal confound: if GPT-4V's API behavior changed between the original study's data collection and the current paper's Gemini testing (due to model updates, prompt processing changes, or safety filter adjustments), the comparison would not reflect the two models' behavior at the same point in time. The paper does not discuss whether GPT-4V was re-tested to verify that the prior responses remain reproducible, nor does it report the specific dates of Gemini testing. For closed-source, actively-updated models, this temporal confound could be significant—the paper effectively compares "GPT-4V as of mid-2023" with "Gemini as of late 2023."

The test set is small, non-random, and potentially unrepresentative.

The 97-figure test set, while diverse in task types, is selected from a prior qualitative study not designed to be a representative sample of multimodal capabilities. The selection criteria for these particular examples are not described in either the current paper or the prior "Dawn of LMMs" study. This means:

• No claim about relative frequency of error types is supportable. The paper documents that Gemini miscounts in Figure 12 and misidentifies landmarks in Figure 3, but we cannot infer how common these error types are across a broader sample of counting or landmark recognition tasks.

• Task difficulty is uncontrolled. Some test cases may be trivially easy for both models (basic object recognition, simple scene text) while others may be challengingly hard (complex RPM patterns, composite image parsing). The distribution of difficulty across task types is unknown.

• The taxonomic balance is author-determined. The paper devotes similar space to logo recognition (4 figures) and temporal understanding (2 figures), but these receive equal weight in the qualitative conclusions. The actual importance of these capabilities depends on application context, which the paper does not weight.

• Potential for unconscious selection bias. Although using pre-existing test cases from a prior study reduces the risk of deliberate cherry-picking, the original study's case selection may have been influenced by which examples produced interesting or illustrative GPT-4V behavior. If the original cases were chosen to showcase GPT-4V's capabilities (or limitations), they might not provide a neutral ground for comparing a new model.

The satisfactory/unsatisfactory classification is subjective and not reliability-tested.

The paper's evaluation rubric relies on author judgment to classify responses as red (unsatisfactory) or blue (satisfactory). There is no inter-annotator agreement reported, no formal classification criteria beyond the implicit ones described above, and no discussion of edge cases. In several figures, the classification is contestable:

• In Figure 13 (spot the difference), GPT-4V's response is classified as satisfactory despite confusing an eye for a mouth (a factual error). Gemini's response is classified as unsatisfactory despite also identifying two of three differences. The classification distinction between these two partially-correct responses is not clearly justified.

• In Figure 71 (customized captioner), Gemini's response is classified as unsatisfactory for failing to recognize the snail shell and misdescribing the snail's position, which are genuine errors—but the threshold for "satisfactory" vs. "unsatisfactory" when a partially-correct, partially-incorrect response is given is unclear.

• In Figure 83 (recipe page description), both models' responses are classified as satisfactory even though the paper notes "each exhibiting some minor inaccuracies in the details." This suggests that minor inaccuracies are tolerated in some contexts but not others, without an explicit rule.

Without inter-annotator reliability metrics or formalized criteria, the red/blue classification should be interpreted as illustrative of author judgment rather than as objective performance measurement.

Missing experiments that would have strengthened the paper.

Several experiments are conspicuously absent given the paper's stated goals:

1. GPT-4V-on-composite-images control. As discussed above, running GPT-4V on the same composite images given to Gemini would isolate whether composite format itself degrades performance or whether Gemini has a specific difficulty.

2. Gemini-on-video task using native YouTube input. Section 1.2 notes that Gemini has native video understanding via YouTube links. The temporal understanding evaluation (Section 8) tests Gemini on composite keyframes—a format it handles poorly—but never tests its native video mode. Comparing Gemini's native video understanding to GPT-4V's keyframe-based understanding would provide a more complete picture of temporal reasoning capabilities.

3. Systematic prompt sensitivity analysis. The paper adjusts prompts for Gemini but does not report whether prompt variations affect outcomes. Testing a few prompt phrasings per task would indicate whether reported failures are robust or prompt-sensitive.

4. Quantitative aggregation. Computing simple statistics—fraction of tasks where each model's response is satisfactory, broken down by capability dimension—would complement the qualitative description and enable dimension-level comparisons. The paper's qualitative approach makes it difficult to assess whether reported patterns (e.g., Gemini excelling at EQ but struggling at IQ) are consistent or selective.

5. Repeated queries across task types. The boat-counting self-consistency test (Figure 50) is interesting but isolated. Repeating this for other counting tasks, recognition tasks, and reasoning tasks would reveal whether Gemini's consistency and GPT-4V's inconsistency generalize or are task-specific.

6. Length-controlled comparison. Since Gemini's responses are consistently longer, controlling for verbosity—either by prompting Gemini for concise answers or by evaluating both models on information density rather than output volume—would distinguish the effects of answer style from answer accuracy.

The integrated-use results are a proof-of-concept, not a rigorous evaluation.

Section 11 demonstrates a two-example pipeline where GPT-4V output feeds Gemini input. While conceptually interesting, this falls short of a systematic evaluation:

• No single-model baselines are evaluated. The paper claims each model fails at the other's task but does not present evidence. For Figure 96, what happens when Gemini alone is asked to recommend products? What happens when GPT-4V is prompted to generate product links (even if it lacks that capability natively)? These baselines would quantify the pipeline's benefit.

• No pipeline failure analysis. What happens when GPT-4V's description contains an error? Does that error propagate to Gemini's output? Is the pipeline robust to recognition mistakes in the first stage? The paper does not test or discuss this.

• Only two pipeline directions are tested. GPT-4V → Gemini is explored; Gemini → GPT-4V is not. Could Gemini's detailed descriptions improve GPT-4V's reasoning? Could Gemini's supplementary links feed back into GPT-4V for verification? The design space is unexplored.

• No iterative or interactive composition. The pipeline is a single-pass feedforward: GPT-4V output is concatenated as Gemini input, with no feedback loop, error correction, or debate. More complex composition patterns (which the paper's own findings about complementary strengths would motivate) are not investigated.

These limitations mean the integrated-use results should be interpreted as existence proof—it is possible to chain the models to achieve complementary capabilities—rather than as evidence that such chaining is reliable, robust, or performance-enhancing relative to single-model baselines.

Where the claims hold and where they are conditional.

Claim: "Both models show comparable performance on basic image recognition tasks." This holds well for landmark, food, and logo recognition, and for scene text OCR in most cases. It does not hold for object counting (Gemini weaker) or abstract shape recognition (Gemini weaker). The claim is thus conditional on the specific recognition sub-task and on whether the image is presented individually or as part of a composite.

Claim: "Gemini excels in providing detailed, expansive answers accompanied by relevant imagery and links." Strongly supported across multiple task types (landmarks, foods, logos, grocery checkout). This is Gemini's most consistently documented advantage. However, the paper does not evaluate whether the generated links are actually relevant and correct—only that they are provided.

Claim: "GPT-4V distinguishes itself with its precision and succinctness in responses." Supported for tasks where conciseness is beneficial (GUI navigation, embodied agents, coordinate output) but the paper does not control for verbosity as a confound. GPT-4V may appear more precise partly because it says less; Gemini may appear less precise partly because it says more, creating more opportunities for observable error.

Claim: "GPT-4V is slightly stronger than Gemini Pro overall." This claim is impressionistic given the absence of aggregate metrics. It appears to be based on the visual preponderance of red (unsatisfactory) highlights for Gemini across the figures, but without formal counting, the magnitude and statistical reliability of the difference are unknowable from the paper's reported data. The claim is better interpreted as a qualitative author assessment than as a measured result.

Claim: "Combining two large models can leverage their respective strengths." Demonstrated for exactly two scenarios (product recommendation, story generation) with the caveats noted above. Whether the benefits generalize to other task types, whether they exceed single-model baselines with optimized prompting, and whether the pipeline is robust to first-stage errors are all open questions.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Larger Than the Test-Time Compute Budget Being Optimized

The assumption. The entire compute-optimal framework depends on estimating prompt difficulty before allocating the inference budget. The paper's method for doing so—generating 2048 samples per question and computing pass@1 (oracle) or averaging PRM final-answer scores (predicted)—is extraordinarily expensive. The authors explicitly acknowledge this in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

At 2048 samples per question, the difficulty estimation step alone consumes more compute than the largest test-time budgets studied (256–512 generations). The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it.

The consequence. In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter. The 4× figure should be understood as an upper bound on achievable efficiency rather than a realized deployment gain. For any application where users submit single questions (rather than batched evaluation of thousands of questions where the estimation cost can be shared), the difficulty estimation overhead makes the compute-optimal policy strictly more expensive than the best-of-N baseline it claims to improve upon. The paper's framing of "4× less compute" is therefore misleading for the most common deployment scenario—single-question inference—where the cost of generating 2048 preliminary samples to decide how to spend a budget of, say, 64 generations is absurd.

What evidence exists. Section 3.2 describes the estimation procedure explicitly, and the authors state: "we do not include this cost in our budget calculation." Figure 4 and Figure 8 show the 4× improvement figures calculated without estimation cost. The paper provides no experiment that measures the combined cost of estimation + strategy execution, and no analysis of how the required number of estimation samples trades off against estimation accuracy (e.g., can you estimate difficulty from 32 samples instead of 2048 and still achieve most of the gain?).

Mitigation status. The authors flag this as "a key avenue for future work" (Section 3.2) and mention the possibility of "pretraining or finetuning models to directly predict difficulty of a question" (Section 8). No such model is developed or evaluated. A potentially more practical direction—adaptive difficulty estimation where initial samples inform strategy mid-computation—is mentioned but not explored. The limitation is entirely unmitigated in the current work.

---

All Results Are on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*)

The assumption. The paper assumes that the difficulty-dependent scaling patterns observed on the MATH benchmark using PaLM 2-S* (Codey) generalize to other domains, other model families, and other reasoning tasks. The authors state in Section 4 that they "believe this model is representative of the capabilities of many contemporary LLMs," but this belief is empirically unverified.

The consequence. Several aspects of the findings could be model-specific or benchmark-specific:

• The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution—a model with different calibration or error patterns might exhibit qualitatively different difficulty-dependent scaling curves (e.g., beam search might not over-optimize on easy problems if the verifier is better calibrated).
• The MATH benchmark consists exclusively of competition-level symbolic math problems requiring multi-step deductive reasoning. It is unclear whether the finding that sequential revisions help on easy problems and beam search helps on medium problems generalizes to other reasoning domains (code generation, logical deduction, scientific QA) or to tasks requiring factual recall, creative generation, or open-ended reasoning.
• The test set of 500 questions, split into five difficulty quintiles of approximately 100 each and further split by two-fold cross-validation, means the compute-optimal policy is selected based on approximately 50 questions per fold per bin. The paper does not report confidence intervals on the compute-optimal scaling curves, making it impossible to assess whether observed differences between strategies at a given budget are statistically reliable or attributable to small-sample noise.
• The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families. A model with weaker in-context learning might fail to benefit from the edit-distance-based pairing strategy that works for PaLM 2-S*.

What evidence exists. All experiments in Sections 5–7 use MATH with PaLM 2-S*. The paper provides no cross-benchmark validation, no cross-model-family replication, and no analysis of how findings might vary with model architecture or training data. The authors acknowledge this scope limitation implicitly (Section 4 discusses why MATH was chosen) but do not systematically address it. The test-set size (500 questions) and cross-validation protocol are described in Section 3.2; the absence of confidence intervals or statistical testing is a gap in the reported analysis.

Mitigation status. The paper does not attempt to mitigate this limitation. The single-benchmark, single-model scope is a deliberate scoping choice (the paper is already heavy in experimental detail within this scope), but it means the claimed findings—4× efficiency gains, difficulty-dependent optimal strategies, the specific difficulty thresholds where strategies switch—cannot be assumed to transfer to other settings without replication. Future work explicitly identified in Section 8 includes extension to other domains and model families.

---

The 14× Larger Model Baseline Is Not Compute-Optimally Trained and Uses No Test-Time Compute

The assumption. The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately 14× more parameters, trained with the same data but with parameters scaled and data fixed. The larger model is evaluated with greedy decoding only—no majority voting, no best-of-N, no search, and no revision chain. The paper acknowledges departing from compute-optimal pretraining:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." (Section 7)

The consequence. This makes the pretraining baseline artificially weak in two ways:

1. Suboptimal pretraining allocation. A Chinchilla-optimal model (Hoffmann et al., 2022), where both parameters and data are scaled equally to consume the additional FLOPs, would likely outperform a parameter-only-scaled model at the same total pretraining compute. The reported advantages of test-time compute over pretraining (e.g., +27.8% relative improvement on easy questions at R << 1 for revisions, from the Figure 1 bar chart) could shrink or reverse against a properly compute-optimal larger model.

2. No test-time compute on the larger model. The comparison is between a smaller model with extensive test-time optimization and a larger model with zero test-time optimization (greedy decoding). A fairer comparison would give the larger model at least a modest test-time compute budget—for example, best-of-8 or best-of-16 sampling with majority voting—which could substantially improve its performance at relatively low additional inference cost. The paper's framing as "test-time compute can substitute for pretraining" would be more convincing if the larger model were also allowed some inference-time optimization.

The reported finding that a smaller model with test-time compute can outperform a 14× larger model is therefore conditional on the larger model being both non-optimally trained and evaluated without any inference-time augmentation. Against a stronger baseline (Chinchilla-optimal training + modest test-time compute), the claimed substitution effect would likely be smaller or restricted to a narrower range of difficulty levels and R values.

What evidence exists. Section 7 describes the FLOPs accounting and the parameter-only scaling approach. The Figure 9 results and the Figure 1 bar chart present the comparison numbers. The paper explicitly acknowledges the compute-optimal pretraining limitation (quoted above) but does not discuss the absence of test-time compute on the larger model. No ablation explores how the comparison changes if the larger model receives even a small inference budget.

Mitigation status. The non-compute-optimal pretraining is acknowledged as a limitation and deferred to future work. The absence of test-time compute on the larger model is not acknowledged as a limitation—the paper treats greedy decoding as the natural evaluation mode for the larger model, which is a strong implicit assumption that weakens the fairness of the FLOPs-matched comparison. Neither issue is mitigated experimentally.

---

Verifier Over-Optimization Is a Hard Ceiling That the Compute-Optimal Policy Mitigates But Does Not Solve

The assumption. The paper's compute-optimal framework assumes that the PRM verifier's scores are reliable enough to guide search toward correct solutions. The findings in Section 5.3 demonstrate that this assumption fails at high optimization budgets, particularly on easy problems where beam search degrades performance with increasing compute (Figure 3, right) due to verifier exploitation—search finds solutions that score highly under the PRM but are incorrect.

The consequence. The compute-optimal policy mitigates this by routing easy problems to best-of-N (weaker optimization) and medium-hard problems to beam search (stronger optimization). However, it does not eliminate the fundamental bottleneck. On medium-difficulty problems where beam search is deployed, over-optimization still limits the scaling ceiling—the beam search curves in Figure 3 flatten and sometimes decline well before the maximum budget is exhausted. This means:

• The 4× efficiency gain is bounded. The compute-optimal policy improves efficiency at low-to-medium budgets but cannot achieve unbounded improvements by scaling test-time compute further—the verifier quality ceiling limits the maximum achievable accuracy regardless of allocation strategy.
• The optimal strategy is verifier-dependent. If the PRM were better trained, more robustly calibrated, or trained on search-generated rather than i.i.d. samples, the difficulty thresholds where over-optimization kicks in would shift, and the optimal policy would change. The paper's specific findings (beam search degrades on bin 1–2, helps on bin 3–4) are therefore tied to the particular PRM quality achieved with the Monte Carlo rollout training procedure in Appendix D, not general properties of test-time compute.
• Lookahead search—the strongest optimizer—paradoxically performs worst overall (Figure 3, left), demonstrating that more sophisticated search amplifies the over-optimization problem rather than solving it. This is a negative result that the paper documents but does not resolve.

The qualitative examples in Appendix M (Figures 29, etc.) show search producing degenerate outputs (repetitive low-information steps, overly short 1–2 step solutions) that score highly under the PRM—confirming that over-optimization is not just a performance plateau but an active degradation mechanism.

What evidence exists. Figure 3 (right) provides direct evidence: on bin 1 (easiest), beam search accuracy decreases from approximately 78% to 77% as budget increases from 4 to 256 generations, while best-of-N improves from 68% to 88%. Figure 3 (left) shows lookahead search underperforming simpler methods at the same generation budget. Appendix M provides qualitative examples of degenerate PRM-exploiting outputs. Section 5.3 discusses over-optimization explicitly as the explanation for beam search's degradation at high budgets.

Mitigation status. The paper acknowledges over-optimization as a limiting factor (Section 5.3, Section 8) and the compute-optimal policy partly mitigates it by choosing weaker optimization where the verifier is unreliable. However, the underlying problem—the verifier is not robust to aggressive optimization—is not solved. The paper suggests future work on robust verifiers resistant to over-optimization (Section 8), including adversarial training and ensemble verification, but does not evaluate any such approaches. This limitation is a fundamental constraint on the scalability of the entire test-time compute paradigm, not merely an implementation detail.

---

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate and Revision Training Is Fragile

The assumption. The revision model is trained on sequences where all in-context answers are incorrect followed by a correct target (Section 6.1). The training data never includes cases where the current answer is already correct. At inference time, the model may encounter correct answers in its context (produced during earlier revision steps) and, having never been trained on this situation, will often incorrectly "revise" them into wrong answers.

The consequence. The paper reports that approximately 38% of correct answers get converted back to incorrect ones during revision chains (Section 6.1). This means:

• Revision chains are not monotonically improving. The pass@1 at each step improves on average (Figure 6, left), but the chain is non-monotonic for individual trajectories—a correct answer at step 3 may be corrupted at step 4. The within-chain selection mechanism (majority voting or verifier-based selection across the entire chain, rather than taking the final revision) is a patch that works around this problem but does not eliminate it.
• The optimal chain length is bounded by reversion risk. Longer chains produce more opportunities for correct answers to be generated but also more opportunities for them to be subsequently corrupted. The paper does not analyze this tradeoff explicitly, but the observed plateau in pass@1 improvement after steps 15–20 (Figure 6, left) likely reflects this tension.
• The revision training approach is fragile. The ReST^EM experiment in Appendix K (Figure 16) shows that attempting to further optimize the revision model with RL-style on-policy training substantially degrades performance—at 256 generations, fully sequential performance drops to approximately 33.5% compared to roughly 38.5% at the optimal ratio. This suggests that the positive revision results depend on specific training choices (offline data construction, edit-distance-based pairing) that do not survive more aggressive optimization, and that the model's revision capability is not robust to distribution shift in its training data.

What evidence exists. Section 6.1 reports the 38% reversion rate explicitly. Figure 6 (left) shows the pass@1 trajectory flattening after initial improvement. Figure 16 (Appendix K) demonstrates the ReST^EM degradation—performance drops substantially with sequential revisions for the RL-optimized model. The within-chain selection mitigation is described in Section 6.1.

Mitigation status. The paper mitigates the reversion problem at inference time using within-chain selection (majority voting or verifier-based selection across all steps of the chain), but this is a heuristic patch, not a solution to the underlying training data limitation. The paper does not explore more principled solutions (e.g., including "no revision needed" examples in training data, training a separate revision-necessity classifier, or using the PRM to dynamically decide when to stop revising). The ReST^EM negative result is reported but not deeply analyzed, leaving the fragility of revision training as an open problem. The authors acknowledge in Section 8 that combining PRM search with revisions—which might help guide when to revise versus restart—was not explored.

---

Hard Problems (Difficulty Bin 5) Remain Essentially Unsolved Regardless of Compute Budget

The assumption. The compute-optimal framework assumes that test-time compute can amplify the base model's existing capability—generating and selecting among candidate solutions using verifier guidance or iterative refinement. This assumption breaks down when the base model's pass@1 is approximately zero, meaning there are no correct solutions in the proposal distribution to find or refine.

The consequence. Across all methods studied—search against the PRM, iterative revisions, and their compute-optimal combinations—the hardest questions (difficulty bin 5) show near-zero improvement regardless of compute budget:

• In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all search methods and all budgets up to 256 generations.
• In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio, at a budget of 128 generations.
• In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%, well below the 14× larger model's performance (shown as stars) at all R values.

This means the compute-optimal framework offers no path forward for problems outside the base model's capability range. Test-time compute can improve efficiency on problems the model can sometimes solve, but it cannot enable the model to solve problems it fundamentally cannot approach. For applications where the problem distribution includes a non-trivial fraction of genuinely hard problems (out-of-distribution reasoning, novel problem structures, tasks requiring knowledge or reasoning patterns not present in pretraining), scaling test-time compute provides no benefit.

What evidence exists. The bin 5 (hardest quintile) results are consistently near-zero across all figures that break out performance by difficulty: Figure 3 (right) for search methods, Figure 7 (right) for revision ratios, and Figure 9 for the FLOPs-matched comparison. Section 7 explicitly states this finding in its takeaway box: test-time compute is not a substitute for pretraining on hard problems. The paper is transparent about this limitation.

Mitigation status. This is a fundamental, unmitigated limitation of the test-time compute paradigm as studied. The paper does not claim otherwise—it frames the finding as establishing a boundary condition for when test-time compute can substitute for pretraining (Section 7). However, for practitioners deciding whether to invest in test-time compute optimization versus model scaling, this limitation is critical: if the target problem distribution includes hard problems, test-time compute alone is insufficient and pretraining scale remains necessary. The paper identifies no approach that would extend test-time compute benefits to the hard-problem regime, and this is unlikely to be solvable without improving the base model's capability through pretraining—which the test-time compute framework explicitly treats as an alternative rather than a complement.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose a new model, training method, or benchmark. Its contribution to the field is a diagnostic reframing of how multimodal model comparisons should be conducted and interpreted. The landscape shift is not a paradigm revolution but a methodological corrective: the paper demonstrates that architectural interface constraints—input modality, memory architecture, output format—are first-class determinants of model behavior that must be analyzed as systematically as model accuracy. This reframing has several concrete consequences for how the field approaches MLLM evaluation and development.

From capability-ranking to interface-capability interaction analysis. Prior MLLM comparison work, including the "Dawn of LMMs" study this paper extends, operated primarily in a ranking paradigm: present both models with the same inputs, evaluate their outputs on the same criteria, and determine which model performs better on each task type. This paper subverts that paradigm by demonstrating that for a substantial class of tasks—multi-image reasoning, sequential visual memory, temporal ordering, interactive navigation—the comparison is not between two models performing the same task, but between two models performing fundamentally different tasks due to interface incommensurability. Gemini receives composited images where GPT-4V receives sequential images; Gemini's prompts are adapted while GPT-4V's are not; Gemini's video understanding mode (YouTube links) is untested while GPT-4V's keyframe mode is tested. The finding that Gemini's failures on multi-image tasks largely disappear when tasks are decomposed into single-image queries (Figures 46–47) demonstrates that the interface, not the underlying multimodal intelligence, is the binding constraint. This means future comparison studies must treat interface-task fit as a primary analysis dimension—reporting not just "Model A outperforms Model B on Task X" but "Model A's interface affords capability Y that Model B's interface does not, and Task X requires Y."

Reconciling contradictory surface-level findings. The paper's interface-centric analysis provides a unifying explanation for what would otherwise appear as contradictory capability patterns. Gemini matches or exceeds GPT-4V on some tasks (multilingual description, emotional intelligence generation, detailed landmark descriptions with supplementary links) while dramatically underperforming on others (temporal ordering, RPM pattern completion, interleaved receipt calculation). A capability-centric interpretation would struggle to reconcile these: the same model cannot simultaneously be "good at reasoning" (EQ tests, detective reasoning) and "bad at reasoning" (IQ tests, temporal ordering). The interface-centric interpretation resolves this: Gemini performs well on tasks that can be accomplished from a single image with rich descriptive output; it performs poorly on tasks requiring comparison across images or accumulation of visual context across turns. The apparent contradiction reflects not inconsistent reasoning capability but differential access to task-relevant information imposed by the interface. This resolution is methodologically significant because it provides a template for future comparison studies: when a model exhibits inconsistent performance across task types, the first diagnostic question should be whether interface constraints, rather than capability limitations, explain the pattern.

Redirecting research attention from model architecture to interface design. A naïve reading of this paper's findings might conclude that Gemini is simply a weaker model and that future MLLM development should focus on matching GPT-4V's multi-image sequential processing capability. The paper's more nuanced contribution is to demonstrate that interface design is not a UX afterthought but a core capability affordance—and that different interface designs enable different application niches. Gemini's image-and-link output capability (documented in landmark recognition, food recognition, logo recognition, and grocery checkout) enables a class of retrieval-and-recommendation applications that GPT-4V's text-only output cannot serve. GPT-4V's multi-image sequential memory enables embodied agent and GUI navigation applications that Gemini's single-image mode cannot handle. The implication for the field is not "multi-image input is better" but rather "interface capabilities should be designed to match application requirements, and model comparisons should evaluate interface-application fit rather than assuming a single optimal interface." This shifts the development conversation from "which architecture is superior?" to "which interfaces enable which applications, and how can models with complementary interfaces be composed?"

Establishing response characteristics as evaluation dimensions. The paper's consistent documentation of Gemini's verbosity and supplementary output generation alongside GPT-4V's conciseness and precision establishes response style as a functionally significant model property that should be evaluated alongside accuracy. This is not a new idea—the NLP community has long studied generation diversity, factuality-verbosity tradeoffs, and user preference for response length—but its systematic application to multimodal model comparison, where verbosity can mask or create errors in visually-grounded responses, represents a methodological contribution. The finding that Gemini provides more detailed descriptions but also more hallucinated details (e.g., the VR headset in Figure 54, the invented turtle in Figure 13) demonstrates that verbosity and precision can be inversely related in multimodal contexts. The boat-counting self-consistency test (Figure 50) adds a further dimension: response stability under repeated queries is measurable, informative, and orthogonal to accuracy. Future comparison studies should report not just "was the answer correct?" but "how long was the answer?", "did it contain hallucinated details?", and "is the answer stable under repeated queries?" as standard evaluation axes.

Validating model composition as a deployment strategy. The integrated-use experiments in Section 11, while preliminary, provide existence proof that complementary MLLMs can be composed via prompt-level chaining to achieve capabilities neither model possesses alone. This is not a new idea in AI broadly—ensemble methods, mixture-of-experts, and modular system design are well-established—but its demonstration with closed-source, black-box MLLMs where internal access is impossible is significant. It suggests that the "model wars" framing (Gemini vs. GPT-4V) may be less productive than a "model ecosystem" framing (Gemini and GPT-4V as composable components). For practitioners, this means that model selection need not be an exclusive choice; for researchers, it opens a design space of composition patterns (sequential, parallel, iterative, debate-style) that has barely been explored for frontier MLLMs. The paper's finding that GPT-4V's recognition precision and Gemini's retrieval capability are complementary strengths that can be chained into a product-recommendation pipeline (Figure 96) is a concrete template that can be extended to other complementary model pairs and task types.

Follow-Up Research This Work Enables

Controlled interface ablation: GPT-4V on composite images. The paper's central unresolved confound is whether Gemini's poor performance on multi-image tasks reflects genuine capability limitations or the composite-image workaround itself. A direct experiment would be: present GPT-4V with the same composite images given to Gemini for tasks like temporal ordering (Figure 55), interleaved receipt calculation (Figure 46), and RPM pattern completion (Figures 27–30). If GPT-4V's performance also degrades significantly on composite images relative to its native sequential mode, this would demonstrate that the composite format itself imposes a substantial cognitive burden—and that Gemini's apparent weakness on these tasks is partly an artifact of the comparison methodology rather than a capability gap. If GPT-4V performs well on composite images (maintaining accuracy comparable to its sequential mode), this would validate Gemini's composite-image failures as genuine capability limitations. Either outcome is informative: the first would motivate development of better multi-image-to-single-image encoding strategies; the second would validate the paper's implicit claim that Gemini's multi-image reasoning is genuinely weaker. A rigorous version of this experiment would systematically vary composite image parameters (layout, resolution per sub-image, number of sub-images, presence of borders between sub-images) to map the difficulty landscape of composite-image understanding for both models.

Gemini native video understanding vs. GPT-4V keyframe understanding. Section 1.2 documents that Gemini has a native video understanding capability via YouTube link input—a mode that the paper's temporal understanding evaluation (Section 8) does not test, instead using composite keyframes. A follow-up study would design a matched set of video understanding tasks (action recognition, temporal ordering, event prediction, causal reasoning about video sequences) and evaluate Gemini using its native YouTube-linked video input against GPT-4V using its native keyframe-sequential input. This would provide the first systematic comparison of two fundamentally different video understanding paradigms: end-to-end video processing (Gemini) versus keyframe-based sequential reasoning (GPT-4V). The research questions are: does Gemini's native video mode recover the temporal reasoning capability that appears absent in the keyframe-composite mode? Does native video processing confer advantages for tasks requiring fine-grained temporal dynamics (motion understanding, speed estimation, action transition detection) that keyframe sampling might miss? And does keyframe-based reasoning confer advantages for tasks requiring careful frame-by-frame comparison (spot-the-difference across time, precise temporal ordering) that end-to-end video processing might struggle with? A well-designed study would use a curated video dataset with both long-form narrative videos (where temporal context matters) and short action clips (where precise frame-level understanding matters), and would report both accuracy and response characteristics (detail level, hallucination rate) for each model in its native mode.

Systematic self-consistency analysis across task types and model families. The boat-counting self-consistency test (Figure 50) is the paper's most methodologically interesting single finding, but it is anecdotal—one image, four repetitions, one task type. A systematic follow-up would measure self-consistency (answer stability under repeated identical queries) across a diverse set of task types (recognition, counting, reasoning, generation), across both models, across multiple temperature settings, and across a statistically meaningful number of repetitions per query (e.g., 30–100 to estimate the distribution of responses). The research questions are: is Gemini consistently more self-consistent than GPT-4V across task types, or is the boat-counting result an outlier? Does self-consistency correlate with accuracy (do models tend to be consistent when correct and inconsistent when wrong, or are these independent dimensions)? Does self-consistency vary with task difficulty in a predictable way? And how does self-consistency interact with sampling temperature—can we make GPT-4V more consistent by lowering temperature, and if so, does that trade off against accuracy or diversity? This would operationalize self-consistency as a standard evaluation metric for MLLMs, complementing accuracy with a measure of response reliability that matters for deployment in deterministic or safety-critical systems. The finding that Gemini is consistently wrong while GPT-4V is inconsistently sometimes-right raises the provocative question: for which applications is a consistently wrong model preferable to an inconsistently right one? A follow-up study could answer this through user studies or application-specific cost-benefit analyses.

Prompt-level model composition patterns beyond feedforward pipelines. The integrated-use experiments (Section 11) demonstrate a simple feedforward pipeline (GPT-4V describes → Gemini generates/retrieves). This opens a much larger design space of composition patterns that the paper does not explore. Specific patterns worth investigating:

• Iterative refinement: GPT-4V describes a scene → Gemini generates a creative output (story, recommendation) → GPT-4V evaluates the output against the original image for factual consistency → Gemini revises based on GPT-4V's critique. This would test whether the complementary strengths can be combined in a feedback loop that improves output quality beyond single-pass composition.
• Debate-style verification: Both models are given the same visual reasoning task independently → their outputs are exchanged → each model critiques the other's answer → a final answer is selected by voting or by a third evaluation step. This would test whether disagreement between models can surface errors that neither model would catch on its own, and whether the models' different error profiles (GPT-4V: concise but sometimes inconsistent; Gemini: detailed but sometimes hallucinatory) enable effective cross-checking.
• Dynamic task routing: A lightweight classifier (or a few-shot prompt to either model) estimates task characteristics (single-image vs. multi-image, creative vs. precise, retrieval vs. reasoning) and routes the query to the model whose interface and behavioral profile best matches the task requirements. This would operationalize the paper's central insight—that interface-task fit determines performance—as an automated routing system.

Each of these patterns should be evaluated against single-model baselines with optimized prompting to ensure that composition provides genuine benefit beyond what either model could achieve alone with sufficient prompt engineering.

Controlled verbosity equalization to isolate accuracy from response style. Throughout the paper, Gemini's responses are substantially longer than GPT-4V's, and this verbosity difference confounds accuracy comparisons: longer responses provide more opportunities for observable error, and the paper's qualitative satisfactory/unsatisfactory classification may penalize verbosity. A follow-up study would systematically prompt both models for equivalent verbosity levels: prompt GPT-4V for detailed, expansive responses with supplementary information; prompt Gemini for concise, focused answers with minimal elaboration. Then re-evaluate a subset of the paper's test cases under these length-equalized conditions. The research question is: do the paper's observed performance patterns (GPT-4V more accurate on multi-image tasks, Gemini more detail-rich but occasionally hallucinatory) persist under controlled verbosity, or are they artifacts of default response style? If GPT-4V becomes more error-prone when forced to be verbose (suggesting that its conciseness is protective against hallucination), and Gemini becomes more accurate when forced to be concise (suggesting that its verbosity introduces errors that are not intrinsic to its understanding), this would have direct implications for how these models should be prompted in deployment. If the patterns persist regardless of verbosity, this would validate that the observed differences reflect underlying capability differences rather than stylistic defaults.

Difficulty-predictive meta-evaluation from qualitative case studies. The paper's evaluation is entirely qualitative and case-based, with no aggregate metrics. A meta-analysis follow-up could formalize the paper's findings by: (1) extracting all test cases from the paper (images, prompts, both models' responses, and the paper's satisfactory/unsatisfactory classification); (2) having multiple independent annotators re-classify the responses using formalized criteria (factual accuracy, task completion, hallucination presence, format compliance, verbosity); (3) computing inter-annotator agreement to assess the reliability of the paper's qualitative judgments; (4) computing aggregate win/loss/tie statistics per capability dimension to quantify the paper's "GPT-4V slightly outperforms Gemini Pro" claim; and (5) building a simple classifier that predicts which model will perform better on a given task based on task features (requires multi-image comparison? requires precise structured output? benefits from verbose description?). This would transform the paper's qualitative insights into testable, quantitative hypotheses about when each model should be preferred, and would establish a replicable methodology for meta-evaluating qualitative comparison studies.

Practical Applications and Downstream Use Cases

Application-specific model selection guided by interface-task fit analysis. The paper's central practical contribution is enabling evidence-based model selection for specific deployment contexts. Based on the documented behavioral patterns:

• E-commerce product identification and recommendation (motivated by Figures 67, 96): Gemini's image-and-link output capability makes it preferable for retail applications where identifying products and providing purchasable links is the primary task. However, the paper's grocery checkout result (Figure 67) shows Gemini identifying only 4 of 7+ basket items correctly, while GPT-4V identifies more items but includes some not clearly visible. The optimal deployment may be the two-stage pipeline demonstrated in Section 11: GPT-4V performs comprehensive recognition, Gemini generates links for recognized items. This architecture could reduce checkout errors while maintaining the link-generation capability that adds commercial value.

• Interactive troubleshooting and customer support (motivated by Figures 75–78, 79–95): GPT-4V's multi-image sequential memory and concise, task-focused operational guidance make it preferable for step-by-step navigation scenarios (embodied agent instruction, GUI guidance, technical support walkthroughs) where accurate, minimal instructions are critical and where the interaction requires accumulating visual context across steps. Gemini's tendency toward verbose advice and task drift (documented in Figures 77, 86, 91) makes it less suitable for these applications without careful prompt engineering to enforce conciseness and task adherence.

• Creative content generation grounded in visual inputs (motivated by Figures 34–35, 97): Gemini's strength in generating detailed, stylistically-controlled narratives and emotionally-targeted descriptions makes it preferable for applications where the output is creative text (story generation, marketing copy, emotional response generation) and where verbosity and elaboration are assets rather than liabilities. The integrated-use result (Figure 97) suggests that the optimal pipeline may again involve GPT-4V for structured visual parsing followed by Gemini for creative generation.

• Multilingual deployment requiring broad language coverage (motivated by Figures 56–63): Both models demonstrate strong multilingual capability across diverse scripts and languages, with no systematic performance gap documented. For applications where multilingual support is the primary requirement and task complexity is low (image description, text translation), model selection can be based on other factors (cost, latency, output format) without significant accuracy concerns, as both models perform comparably on the tested multilingual tasks.

Prompt engineering guidelines derived from behavioral patterns. The paper's documentation of response style differences provides actionable guidance for prompt design:

• When using Gemini: Explicitly prompt for conciseness when precision is more important than comprehensiveness. The paper shows that Gemini's default verbose style introduces errors (e.g., the VR headset hallucination in Figure 54, the turtle misidentification in Figure 13) that a shorter, more focused response might avoid. Prompts should include instructions like "answer in one sentence" or "provide only the requested information without additional context" for counting, localization, and structured-output tasks.

• When using GPT-4V: Explicitly prompt for detail and supplementary information when comprehensiveness is desired. The paper shows that GPT-4V defaults to concise responses that omit potentially useful context (e.g., landmark historical details in Figure 2, cooking instructions in Figure 4). Prompts should include "in detail" or "include relevant background information" when the application benefits from elaboration.

• For both models: Use step-by-step decomposition prompting for complex multi-step tasks. The dashboard reading experiment (Figure 16) and the RPM step-by-step guidance (Figure 30) demonstrate that explicit reasoning-step prompts can recover capability that is not evident from direct queries. The multi-step prompting strategy used for GPT-4V's apple counting (Figure 12)—"Count the number of apples in the image. Let's count the apples row-by-row"—is a template that can be applied to any counting, comparison, or sequential reasoning task.

Model composition as a cost-effective alternative to waiting for next-generation models. The integrated-use results (Section 11) suggest that organizations currently deciding between Gemini and GPT-4V for deployment need not wait for Gemini Ultra or GPT-4.5 to access capabilities neither current model possesses alone. A two-model system where GPT-4V handles recognition-heavy subtasks and Gemini handles generation-heavy subtasks can be implemented today using API-level orchestration. The paper's product recommendation pipeline (Figure 96) and story generation pipeline (Figure 97) provide concrete templates. The cost of orchestrating two API calls rather than one must be weighed against the cost and timeline of waiting for a single model that matches both models' complementary strengths—a calculation the paper does not perform but that its findings directly enable. For organizations where time-to-deployment matters more than per-query inference cost, the composition approach offers an immediately available path to capabilities that exceed either current model alone.