Capabilities of Gemini Models in Medicine

🎯 Pitch

Med-Gemini’s uncertainty‑guided web search drives USMLE accuracy to 91.1 %, yet re‑annotation reveals ~7 % of those benchmark questions carry errors or missing information—so the headline number already overstates real clinical progress. Across 14 benchmarks the model still opens a 44.5 % average margin over GPT‑4V, with the biggest leaps coming from fine‑tuned encoders for pathology and ECG waveforms.

1. Executive Summary

This paper introduces Med-Gemini, a family of multimodal medical models built on Gemini that are specialized for medicine through self-training with web search integration, multimodal fine-tuning with customized encoders, and long-context chain-of-reasoning prompting. Evaluated across 14 benchmarks spanning text-based reasoning, multimodal understanding, and long-context applications, Med-Gemini establishes new state-of-the-art performance on 10 of them, achieving 91.1% accuracy on MedQA (USMLE) using a novel uncertainty-guided search strategy (iteratively generating multiple reasoning paths, triggering web search when answer-entropy exceeds a threshold, and incorporating retrieved results) that surpasses Med-PaLM 2 by 4.5 percentage points and GPT-4 with MedPrompt by 0.9 percentage points. On multimodal benchmarks, Med-Gemini improves over GPT-4V by an average relative margin of 44.5%, with gains driven by fine-tuned models for specialized modalities like pathology and ECG waveforms as well as strong out-of-the-box performance on diagnostic challenges like NEJM Image Challenge. The long-context capabilities enable state-of-the-art retrieval from de-identified health records tens of thousands of words long and medical video question answering using only in-context learning, establishing that generalist multimodal foundation models can achieve expert-level medical performance across diverse tasks when augmented with domain-specific fine-tuning and inference-time reasoning strategies—though the paper demonstrates through expert re-annotation that approximately 7.4% of MedQA benchmark questions contain missing information, label errors, or ambiguity, indicating that further improvements on this popular benchmark may not directly reflect clinically meaningful progress.

2. Context and Motivation

The Core Problem: Medical AI Systems Lack Integrated Advanced Reasoning, Multimodal Understanding, and Long-Context Processing

The fundamental challenge this paper tackles is that real-world medical practice requires sophisticated AI capabilities that existing systems fail to deliver in combination. A clinician's daily work involves simultaneously reasoning under uncertainty about complex cases, synthesizing information across diverse data modalities (images, waveforms, text records, videos), staying current with rapidly evolving medical knowledge, and communicating findings clearly to patients and colleagues. While individual AI systems have demonstrated competence on isolated medical tasks—radiology report generation, medical exam question answering, dermatology image classification—the field lacks models that integrate these capabilities into a unified system robust enough for practical deployment.

This gap matters for several reasons the authors emphasize throughout the introduction and Section 5:

Clinical workflow integration is inherently multimodal and longitudinal. A patient encounter rarely involves just one data type. A physician interpreting a chest X-ray needs to consider the patient's history from the electronic health record (EHR), current symptoms, prior lab results, and possibly comparison with previous imaging—all while referencing up-to-date clinical guidelines. Systems that handle only text or only images cannot support this reality.
Medical knowledge is non-stationary and growing exponentially. The doubling time of medical information has been shrinking (Densen, 2011), with estimates suggesting it now doubles every 73 days or faster. A model trained on a static corpus cannot remain current with treatment guidelines, drug interactions, or emerging diseases. The COVID-19 pandemic illustrated how quickly public health understanding must update—and the dangers of medical misinformation when systems rely on outdated knowledge (Kouzy et al., 2020).
Information overload in clinical settings is a patient safety issue. Clinicians regularly face "needle-in-a-haystack" problems: finding a single crucial mention of a condition, medication, or procedure within thousands of pages of EHR records (Sbaffi et al., 2020; Klerings et al., 2015). Missing such information can lead to diagnostic errors, adverse drug events, or inappropriate treatment decisions. AI systems that can reliably process full patient histories without truncation could directly reduce this cognitive burden.
The regulatory and practical landscape favors non-diagnostic assistance first. While diagnostic AI attracts research attention, the more immediate deployment opportunities are in tasks like medical summarization, referral letter generation, and clinical information retrieval—where errors have lower risk profiles yet model outputs can significantly improve clinician efficiency and reduce burnout (Section 5). Current general-purpose LLMs, while impressive on benchmarks, have not been systematically evaluated for these lower-risk but high-utility tasks in a way that demonstrates real-world readiness.

Limitations of Prior Work Across Three Capability Dimensions

The paper identifies specific shortcomings in prior approaches across the three axes it tackles: clinical reasoning, multimodal understanding, and long-context processing.

Clinical Reasoning and Knowledge Grounding

The most prominent line of prior work on medical LLMs—models like Med-PaLM (Singhal et al., 2023a), Med-PaLM 2 (Singhal et al., 2023b), and prompting strategies like MedPrompt applied to GPT-4 (Nori et al., 2023)—achieved impressive performance on medical question-answering benchmarks, particularly MedQA (USMLE). However, the paper identifies several critical limitations:

Reliance on static, potentially outdated knowledge. These models are fine-tuned on fixed corpora of medical literature and exam questions. They cannot incorporate new information that emerged after training. As Section 2.1 notes, medical knowledge has high turnover—treatment guidelines change, new drugs are approved, and understanding of disease mechanisms evolves. A model answering questions based solely on its training data may provide factually outdated recommendations. The authors explicitly contrast this with physician practice, where clinicians "stay abreast of the latest medical information from a wide variety of authoritative sources" (Section 1).

Confabulation and premature conclusions without exploring alternative reasoning paths. The paper acknowledges that prior medical LLMs "exhibit significant reasoning shortcomings" (Section 2.1), specifically citing "factuality errors that compound in downstream reasoning steps or lead to models adopting premature conclusions without considering all possible reasoning pathways." This is a well-documented failure mode: LLMs can produce confident-sounding but incorrect reasoning chains that mislead rather than inform. Prior work on self-consistency (Wang et al., 2022b) and chain-of-thought prompting (Wei et al., 2022) improved reasoning quality but did not address the fundamental issue that the model has no mechanism to verify its intermediate steps against external ground truth or resolve uncertainty through targeted information retrieval.

Lack of tool use for knowledge retrieval. While the broader NLP community had developed methods for teaching LLMs to use tools (Schick et al., 2024; Hao et al., 2024; Nakano et al., 2021) and retrieval-augmented generation (Gao et al., 2024), medical LLMs had not integrated web search as a core capability. Zakka et al. (2024) demonstrated search tool-use for medical guideline and treatment recommendations, but this was not combined with a principled reasoning framework that decides when to search based on model uncertainty. The paper positions its self-training with search and uncertainty-guided inference as filling this gap: the model learns not just to use search results when provided, but to recognize when it is uncertain and generate targeted queries to resolve that uncertainty.

Limited reasoning demonstrations for fine-tuning. MedQA provides only multiple-choice ground truth labels, not expert demonstrations of the clinical reasoning process. Models fine-tuned on MedQA learn to predict the correct answer choice but not to produce the step-by-step reasoning that makes that answer defensible. The paper explicitly notes this: "MedQA only provides a multiple-choice ground truth, and lacks expert demonstrations of the reasoning process necessary to train LLMs for clinical reasoning across diverse settings." This leads to models that may get the right answer for the wrong reasons—a brittleness that becomes apparent when the model faces questions outside its training distribution.

Multimodal Medical Understanding

The paper situates its multimodal contributions against two categories of prior work:

Specialist multimodal models excel at specific medical imaging tasks but cannot generalize across modalities or integrate textual reasoning. Examples include models for radiology report generation (Tanno et al., 2024; Zambrano Chaves et al., 2024), pathology question answering (Lu et al., 2023), and ECG analysis (Wan et al., 2024). These models achieve high performance within their narrow domains but are architecturally incapable of, say, relating a dermatology image to a patient's text-described history or answering questions that span imaging modalities.

Generalist medical AI (GMAI) systems, exemplified by Med-PaLM M (Tu et al., 2024a) and LLaVA-Med (Li et al., 2024), attempt to handle multiple modalities within a single model. The paper acknowledges Med-PaLM M as a pioneering system that achieved competitive performance across radiology, pathology, dermatology, and genomics. However, the authors identify two key limitations of this prior GMAI work:

The generalist paradigm faces real-world trade-offs. While GMAI is "an meaningful research direction for medicine, real world considerations present trade-offs and requirements for task-specific optimizations which are at odds with each other" (Section 1). A single model optimized for all tasks may underperform specialized models on individual tasks while being too large for efficient deployment in resource-constrained settings. The paper explicitly departs from the GMAI philosophy, instead introducing "a family of models, each optimized for different capabilities and application-specific scenarios, considering factors such as training data, compute availability, and inference latency."
Novel modalities with limited pretraining exposure require customization. Generalist models like Med-PaLM M are trained primarily on natural images and medical images common on the public internet. Medical modalities like raw ECG waveforms, genomic sequences, or specialized surgical video have fundamentally different data distributions that may not be adequately represented in pretraining. The paper demonstrates this by building a specialized encoder for ECG data (Med-Gemini-S 1.0), showing that even a powerful foundation model benefits from modality-specific adaptation.
Medical dialogue capabilities remain unexplored. Prior GMAI systems demonstrate question-answering but not the multi-turn conversational abilities that would be needed for real clinical interactions—asking clarifying questions, requesting additional information (like images), explaining reasoning in accessible language, and appropriately deferring final decisions to human experts. The paper previews these capabilities qualitatively through dermatology and radiology dialogue examples (Figures 6 and 7), though it acknowledges these are demonstrations of "the art of the possible" rather than rigorously evaluated clinical tools.

Long-Context Medical Processing

The paper identifies long-context processing as perhaps the most underexplored capability in medical AI, despite its critical importance:

Prior models had severely limited context windows. Earlier medical LLMs and LMMs were constrained to processing relatively short texts—typically 512–4096 tokens (Li et al., 2023; Dai et al., 2022). This is grossly insufficient for real EHRs, where a single patient's records can span hundreds of thousands of words across multiple encounters, imaging reports, lab results, and clinical notes. Systems designed for these constraints either truncated documents (losing potentially critical information) or used hierarchical approaches that aggregate representations without preserving fine-grained cross-document relationships.

Medical video understanding was dominated by specialist systems. The computer vision community had developed specialized architectures for surgical phase recognition (Golany et al., 2022; Chen et al., 2018), tool detection (Nwoye et al., 2019), and skill assessment (Goodman et al., 2021, 2024). These systems are trained end-to-end for specific tasks and cannot handle natural language queries, generalize to new procedure types, or integrate video understanding with textual medical knowledge. The paper notes that "Limited model context windows have hindered the ability for vision-language models to capture long-range dependencies and complex relationships within videos" (Section 2.3). A model that can process entire surgical videos—not just sampled frames—and answer open-ended questions about the procedure represents a fundamentally different capability.

The "needle-in-a-haystack" EHR retrieval problem had no LLM-based solution. The paper's novel long EHR understanding task requires identifying whether a rarely-mentioned medical condition (sometimes mentioned only once in hundreds of thousands of words, often in ambiguous contexts like family history rather than confirmed diagnosis) is actually present. Prior approaches relied on rule-based annotation systems with extensive manual feature engineering (Feder et al., 2022). These systems are brittle, ontology-dependent, and cannot generalize to new conditions without additional engineering effort. The paper explicitly contrasts this with Med-Gemini's one-shot in-context learning approach that "can easily generalize to novel problem settings without the need of extensive manual engineering" (Section 4.3).

How This Paper Positions Itself

The paper's positioning is distinctive in several ways:

It rejects the "one model to rule them all" philosophy while still building on a unified foundation. Rather than attempting to create a single generalist medical AI system, the authors develop a family of models (Med-Gemini-L 1.0, Med-Gemini-M 1.0, Med-Gemini-M 1.5, Med-Gemini-S 1.0) optimized for different capability profiles. This acknowledges that "real world considerations present trade-offs" between model size, inference latency, training data requirements, and task-specific performance. The Gemini ecosystem provides the shared substrate—native multimodality, long-context capabilities, strong language understanding—but each variant is specialized through different fine-tuning strategies, encoder architectures, and inference-time techniques.

It frames web search not as an add-on but as a reasoning tool. Rather than treating retrieval-augmented generation as a separate pipeline component, the paper integrates search into the model's own decision-making about when it is uncertain. The uncertainty-guided search strategy (Section 2.1, Figure 2) explicitly models epistemic uncertainty through answer-entropy and uses that uncertainty to trigger targeted information retrieval. This is more sophisticated than always-on RAG approaches and better aligned with how clinicians actually work: consulting external resources when uncertain, not for every decision.

It takes benchmark limitations seriously. The paper's detailed re-annotation of MedQA (USMLE) with 18 board-certified physicians, identifying that approximately 7.4% of questions have missing information, label errors, or ambiguity, is notable for a capabilities paper. Rather than simply reporting a new state-of-the-art number and moving on, the authors characterize how much of the remaining error may be attributable to benchmark quality rather than model capability. This is a significant methodological contribution: it establishes that "further improvements in SoTA performance on the MedQA (USMLE) benchmark in isolation may not directly correlate to progress in the capabilities of medical LLMs for meaningful real-world tasks" (Section 5). The paper thus positions itself as advancing comprehensive, realistic evaluation rather than benchmark optimization.

It explicitly connects capabilities to clinical utility beyond diagnosis. While diagnostic tasks receive the most research attention, the paper evaluates on medical summarization, referral letter generation, and plain language simplification—tasks that "have a lower risk-profile yet model outputs can significantly improve the efficiency of care providers by alleviating administrative burdens" (Section 5). This pragmatic focus on augmenting clinician workflow rather than replacing clinical judgment is a deliberate positioning choice that reflects the realities of medical AI deployment, where regulatory approval for diagnostic systems is lengthy and fraught, while assistive documentation tools face lower barriers.

It acknowledges the safety-critical nature of the domain without overclaiming. Throughout the paper, the authors are careful to note that the demonstrated capabilities represent "the art of the possible" and that "such uses (particularly in safety-critical areas like diagnosis) would require considerable further research and development." This stands in contrast to some prior medical AI papers that made stronger claims about approaching or surpassing physician performance. The paper's positioning is: these capabilities are promising and represent genuine advances, but their translation to clinical practice requires "meticulous attention to the reliability and safety" including rigorous evaluation of fairness, privacy, equity, and bias across diverse populations and use cases (Section 6).

In summary, the paper addresses a three-dimensional gap in medical AI—integrated clinical reasoning with web search under uncertainty, multimodal understanding spanning text, images, and novel biosignal modalities, and long-context processing of EHRs and surgical video—and positions its family of specialized Gemini-derived models as demonstrating that these capabilities can be achieved simultaneously when foundation model strengths are augmented with domain-specific fine-tuning, modality-specific encoders, and inference-time reasoning strategies.

3. Technical Approach

3.1 Reader Orientation

The paper develops not a single model but a family of models (Med-Gemini), each built on a Gemini foundation model and then specialized for different medical capabilities through a combination of fine-tuning strategies, custom encoders, and inference-time reasoning procedures. The core problem solved is how to imbue a general-purpose multimodal foundation model with advanced clinical reasoning (including the ability to use web search when uncertain), specialized multimodal understanding (including novel biosignal modalities like ECG), and the ability to process extremely long medical contexts (hundreds of thousands of words of EHR notes, full surgical videos) — all while optimizing for deployment trade-offs like model size, inference latency, and training data requirements. The solution takes the form of a "family of specialists" that share a common Gemini backbone but diverge through targeted training: a large model fine-tuned with self-generated reasoning traces and web search for complex clinical reasoning (Med-Gemini-L 1.0), a mid-size model fine-tuned on multimodal medical datasets for image and text understanding (Med-Gemini-M 1.5), a mid-size model fine-tuned for long-form text generation tasks (Med-Gemini-M 1.0), and a small model augmented with a custom ECG encoder for biosignal question answering (Med-Gemini-S 1.0).

3.2 Big-Picture Architecture (Diagram in Words)

The system comprises four major component categories, each realized in one or more specific model variants:

Gemini Base Models — pretrained multimodal transformer decoders (Gemini 1.0 Ultra, Pro, and Nano; Gemini 1.5 Pro) with native support for interleaved text, image, audio, and video inputs. These serve as the initialization substrate providing foundational language understanding, multimodal reasoning, and (for Gemini 1.5 Pro) long-context capabilities up to millions of tokens.
Training Data Generators — pipelines that produce specialized fine-tuning data not available in the base pretraining corpus. For text reasoning, a self-training loop generates Chain-of-Thought (CoT) reasoning traces with and without web search results (MedQA-R and MedQA-RS datasets). For multimodal understanding, existing medical VQA and classification datasets are reformulated into instruction-following formats. For long-context tasks, no fine-tuning is performed; the model relies on in-context learning.
Med-Gemini Model Variants — the fine-tuned models optimized for different capability profiles:
- Med-Gemini-L 1.0: Gemini 1.0 Ultra fine-tuned on text-based reasoning datasets including MedQA-R, MedQA-RS, long-form QA pairs, and medical summaries.
- Med-Gemini-M 1.0: Gemini 1.0 Pro fine-tuned on medical summarization, referral letter generation, and simplification tasks.
- Med-Gemini-M 1.5: Gemini 1.5 Pro fine-tuned on eight multimodal medical tasks spanning radiology, pathology, dermatology, and CXR analysis.
- Med-Gemini-S 1.0: Gemini 1.0 Nano augmented with a novel ECG-specific encoder using cross-attention (based on Flamingo architecture), fine-tuned on ECG-QA.
Inference-Time Reasoning Procedures — algorithms that operate at deployment time to improve answer quality:
- Uncertainty-guided search (for Med-Gemini-L 1.0 on text-based reasoning benchmarks): an iterative process that generates multiple reasoning paths, computes answer-entropy, triggers web search when entropy exceeds a threshold, generates targeted search queries conditioned on conflicting reasoning paths, retrieves results, and re-generates reasoning with search context. Continues for up to four iterations.
- Chain-of-reasoning (for Med-Gemini-M 1.5 on long EHR tasks): a two-step in-context procedure that first retrieves all mentions of a target condition from the full EHR, then reasons across retrieved evidence to determine whether the condition is actually present.

Information flows as follows: an input (medical question, clinical image, ECG waveform, surgical video, or long EHR document) enters the appropriate Med-Gemini variant → the model generates an initial response (possibly with multiple reasoning paths) → if uncertainty-guided search is active, answer-entropy is computed and web search is conditionally triggered → search results augment the prompt for subsequent iterations → a final answer is produced through majority voting or direct generation.

3.3 Roadmap for the Deep Dive

First, the self-training with search framework for text-based reasoning (Section 2.1), because it is the most complex and novel component — understanding how MedQA-RS data is generated, and how the resulting model learns to use web search, establishes the foundation for the inference-time search strategy.
Second, the uncertainty-guided search algorithm at inference time (Section 2.1, right panel of Figure 2), which builds directly on the self-trained model's capabilities and is the mechanism that achieves the 91.1% MedQA accuracy.
Third, the multimodal fine-tuning procedure (Section 2.2), covering both standard image-to-text fine-tuning on eight medical tasks and the specialized ECG encoder architecture — this explains how Gemini's native multimodality is adapted to domain-specific medical modalities.
Fourth, the long-context processing approaches (Section 2.3), including the chain-of-reasoning for EHR understanding, the instruction prompting for medical video tasks, and the novel "needle-in-a-haystack" dataset curation procedure — these are methodologically simpler (no fine-tuning, only in-context learning) but important for understanding the evaluation results.
Fifth, the fine-tuning datasets and their construction procedures (scattered across Sections 2.1, 2.2, and Appendices C, D), since the data generation process — particularly for MedQA-RS — is itself a methodological contribution.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and capabilities paper whose core idea is that Gemini's foundational strengths in multimodality, long-context reasoning, and language understanding can be augmented with domain-specific fine-tuning, custom encoders, and inference-time search to create a family of specialized medical models that collectively achieve state-of-the-art performance across text, multimodal, and long-context benchmarks.

Self-Training with Search for Clinical Reasoning (Med-Gemini-L 1.0)

The central methodological innovation for text-based tasks is a framework that teaches the model to generate high-quality clinical reasoning chains and to use web search results when doing so. The process operates in two phases: (1) an iterative data generation and fine-tuning loop that creates training examples of reasoning with and without search (self-training phase), and (2) an inference-time algorithm that uses the fine-tuned model's own uncertainty to decide when to invoke search (deployment phase). We cover the training phase first.

Iterative Self-Training Data Generation

The goal is to create two new datasets — MedQA-R (Reasoning) and MedQA-RS (Reasoning and Search) — that extend the MedQA (USMLE) training set of 10,177 multiple-choice questions. MedQA-R adds synthetically generated Chain-of-Thought explanations. MedQA-RS adds both CoT explanations and instructions to use web search results as additional context when generating the reasoning.

The generation process operates in a loop with four key steps per iteration:

Step 1: Web search query generation and retrieval. For each question in the MedQA training set, the authors prompt the current version of Med-Gemini-L 1.0 to generate search queries whose results would help answer the medical question. They specifically set up the prompt so the model generates queries, not answers — this is a targeted information retrieval step, not a reasoning step. The generated queries are passed to a web search API (the paper does not specify which API, only that it is "a web search API"), and the retrieved results (snippets, links, and page content) are collected.

Step 2: In-context demonstration seeding. For each type of reasoning path (without search and with search), the authors hand-curate five expert demonstrations that serve as few-shot examples. Critically, the demonstrations differ in structure:

The without-search demonstrations contain accurate clinical reasoning explaining why the ground-truth answer is the best choice versus other potentially valid options. These are standard CoT examples showing multi-step differential diagnosis, weighing of evidence, and elimination of alternatives. They establish the desired reasoning style: thorough, balanced, and explicit about why incorrect answers are wrong.
The with-search demonstrations additionally show how to reference and quote search results within the reasoning chain. For example, a demonstration might include a segment like "According to the CDC guidelines retrieved from search, the first-line treatment for uncomplicated UTI is nitrofurantoin, which supports option B over option A." This teaches the model the format and integration pattern for search-augmented reasoning: search results are not just appended but are actively cited and used to resolve specific clinical decision points.

Step 3: Generating Chain-of-Thought reasoning. The current model is prompted with the five in-context seed demonstrations and asked to generate CoTs for all questions in the training set. For MedQA-RS, the prompt includes the search results retrieved in Step 1 as additional context — the model sees the question, the search results, and the expert demonstration showing how to use search results in reasoning, and then generates a CoT that follows this pattern. For MedQA-R, no search results are provided; the model generates reasoning purely from its own parametric knowledge.

Step 4: Filtering and fine-tuning. The generated CoTs are filtered: any CoT that leads to an incorrect final answer prediction is discarded. Only CoTs where the model's reasoning culminates in the correct answer choice are retained. The surviving CoTs become the training data for the next fine-tuning iteration. The model is fine-tuned on these filtered CoTs, which improves its ability to follow the demonstrated reasoning style and search integration pattern. This improved model then becomes the starting point for the next iteration — it re-generates the CoTs (Step 3 with the improved model), which are higher quality because the model now better emulates the expert demonstrations. The loop repeats until performance saturates (the paper does not specify the exact number of iterations, only that it continues "until the model's performance saturates").

What makes this self-training rather than standard supervised fine-tuning. The key property is that the training targets (the CoTs) are generated by the model itself, not by human experts. The five hand-curated demonstrations are only seeds to establish the format and style; the vast majority of training examples are model-generated. As the model improves through fine-tuning, it generates higher-quality CoTs in the next iteration, which further improve the model when fine-tuned upon. This is a form of bootstrap learning where the model's own outputs, after filtering for correctness, become the next round of training data. The correctness filter (Step 4) is what prevents the model from reinforcing its own errors — without this filter, the loop would amplify mistakes rather than improving reasoning.

The full fine-tuning data mixture. Med-Gemini-L 1.0 is not fine-tuned solely on MedQA-R and MedQA-RS. The complete mixture (Table C1) includes:

MedQA-R: 10,177 multiple-choice questions with synthetically generated reasoning (without search) — one CoT per question after filtering.
MedQA-RS: 10,177 multiple-choice questions with synthetically generated reasoning and search (with search) — one CoT per question after filtering. Note that MedQA-R and MedQA-RS use the same underlying questions; they differ only in whether the CoT incorporates search results.
Long-form QA dataset: 260 expert-crafted long-form responses to questions from HealthSearchQA, LiveQA, and MedicationQA (curated by Singhal et al., 2023a). These are detailed, multi-paragraph answers to open-ended consumer medical questions, teaching the model to produce thorough explanations rather than just answer choices.
Medical summarization dataset: 65 clinician-written summaries of medical notes from MIMIC-III. These teach the model to distill lengthy clinical documents into concise, structured summaries.

The total training set size is approximately 20,679 examples (10,177 MedQA-R + 10,177 MedQA-RS + 260 long-form QA + 65 summaries = 20,679), though Medical-R and MedQA-RS share underlying questions. The paper does not specify the exact mixture ratios, optimization hyperparameters, or number of training epochs for this fine-tuning stage.

The MedQA-RS Generation Prompt Structure

The paper provides a representative example of a MedQA-RS input prompt and generated CoT, which reveals the precise input-output format:

The input consists of:

A task instruction telling the model to answer a medical question using web search results.
The medical question with four answer options.
The retrieved search results (the paper shows one representative search result snippet in the example).

The output is a CoT that follows a specific pattern visible in the example:

The model first presents search results as context ("Based on the web search results, I found that...")
It then reasons through each answer option, using both clinical knowledge and search results to eliminate incorrect options
It explicitly addresses why certain options are wrong ("Option C is incorrect because...")
It culminates in selecting the correct answer with a clear justification grounded in both search results and clinical reasoning

The example in the paper (shown inline in Section 2.1) is abbreviated, but the structure makes clear that the CoT is designed to be a verifiable reasoning chain — each claim in the reasoning is either a standard clinical fact the model should know or backed by a specific search result, making the reasoning traceable and auditable.

Uncertainty-Guided Search at Inference Time

Once Med-Gemini-L 1.0 is fine-tuned with search-augmented reasoning data, the next innovation is when to invoke search during deployment. Rather than always searching (which would add latency and cost for every question) or never searching (which loses the benefit), the authors develop an adaptive strategy triggered by the model's own uncertainty. This is the right panel of Figure 2.

The procedure operates iteratively, with each iteration consisting of four steps. The authors specify that for MedQA, they run up to four iterations; for NEJM CPC and GeneTuring, they run one iteration.

Step 1: Multiple Reasoning Path Generation

Given a medical question (with its answer options if multiple-choice), Med-Gemini-L 1.0 generates multiple reasoning paths. The paper does not specify exactly how many paths are generated, only that it is "multiple" — the key is that enough paths are generated to estimate the distribution of answers. For the first iteration, the prompt consists only of the task instruction and the question. For subsequent iterations (if any), the prompt additionally includes the search results retrieved in Step 4 of the previous iteration.

Each reasoning path culminates in a predicted answer choice. The generation likely uses sampling (non-zero temperature) rather than greedy decoding to produce diversity in the reasoning paths, though the paper does not specify the sampling parameters.

Step 2: Uncertainty-Based Search Invocation

The model's uncertainty is measured by the Shannon entropy of the answer choice distribution across the multiple generated reasoning paths. Specifically:

Count the occurrences of each answer choice (A, B, C, D) among the generated paths.
Convert these counts to probabilities by dividing each count by the total number of paths.
Compute the Shannon entropy:

$H = -\sum_{i \in \{A,B,C,D\}} p_i \log(p_i)$

where $p_i$ is the empirical probability of answer choice $i$ across the generated reasoning paths.

What it computes: the Shannon entropy measures how concentrated or dispersed the answer distribution is. If all reasoning paths agree on the same answer, one $p_i \approx 1$ and all others $\approx 0$ , yielding $H \approx 0$ . If paths are evenly split among multiple answers, each $p_i \approx 1/4$ , yielding $H \approx \log(4) \approx 1.39$ (maximum entropy for four options).

Why this form: Shannon entropy is the standard information-theoretic measure of uncertainty for categorical distributions. High entropy indicates epistemic uncertainty — the model is unsure which answer is correct, potentially because it lacks relevant information. This is precisely the situation where retrieving external knowledge via web search could reduce uncertainty by providing missing information. The authors explicitly connect this to the diagnostic reasoning literature, citing Horvitz et al. (1984) on the Pathfinder system, which used entropy-based criteria to decide when to gather additional evidence in medical diagnosis.

The entropy is compared against a threshold (the paper does not specify the numerical value). If the entropy exceeds the threshold, the procedure moves to Steps 3 and 4 to invoke search. If the entropy is below the threshold, the majority vote answer is returned as the final answer — the model is sufficiently confident, and no search is needed.

Step 3: Uncertainty-Guided Search Query Generation

When the model is uncertain (high entropy), the next step is to generate search queries specifically targeted at resolving the conflict between the model's own generated reasoning paths. This is a crucial design choice: the model conditions on its own conflicting responses when generating queries.

The prompt for this step includes:

The original question
The multiple conflicting reasoning paths generated in Step 1
An instruction to generate three search queries whose results would help resolve the disagreement

The paper explicitly motivates this: "Our motivation of conditioning on previously generated but conflicting responses is to retrieve search results that are directly targeted at resolving the model's uncertainty to the question." This is more sophisticated than generating search queries from the question alone, because the model has already identified what it disagrees with itself about. For example, if two reasoning paths differ on whether a particular lab value is normal or abnormal, the model might generate a query specifically about the normal range for that lab value — directly addressing the source of uncertainty.

The model generates three search queries per invocation. The paper does not justify why three specifically, but it likely balances the trade-off between retrieving enough information to resolve uncertainty and not overwhelming the context window with excessive search results.

Step 4: Search Retrieval and Prompt Augmentation

The three generated queries are submitted to a web search engine (the same API used during training). The retrieved results are incorporated into a new input prompt alongside the original question. The augmented prompt now contains:

The original task instruction
The original medical question
The retrieved search results (presumably the top results for each query, though the paper does not specify how many results per query or how they are formatted)

This augmented prompt is fed back into Step 1 for the next iteration: the model generates new reasoning paths, now informed by the search results. The hypothesis is that with additional external information, the model's uncertainty will decrease — reasoning paths will converge toward a single answer as the missing information is filled in.

The maximum of four iterations for MedQA means the model can search up to four times, each time generating new queries based on its remaining uncertainty after incorporating previous search results. This creates a progressively refined understanding where each round of search targets increasingly specific knowledge gaps. For NEJM CPC and GeneTuring, only one iteration of search is performed, meaning the model makes a single round of search queries and then produces its final answer.

Why This Approach Over Always-On Retrieval

The paper explicitly contrasts uncertainty-guided search with always-on retrieval (searching for every question). The advantages claimed are:

Efficiency: Search is only invoked when needed, reducing latency and computational cost for questions the model can answer confidently from its parametric knowledge.
Targeted retrieval: Because queries are conditioned on identified conflicts in the model's own reasoning, they are more likely to retrieve relevant information than generic queries generated from the question alone.
Alignment with clinical reasoning: The process of recognizing uncertainty, seeking targeted information, and re-evaluating mirrors how clinicians actually work — they do not consult external references for every decision, only when they encounter ambiguity or knowledge gaps.

The paper does not provide an ablation comparing uncertainty-guided search to always-on search, so the empirical benefit of the uncertainty threshold (versus always searching, which might achieve even higher accuracy at higher cost) is not quantified.

Multimodal Fine-Tuning (Med-Gemini-M 1.5)

The multimodal specialization of Med-Gemini-M 1.5 is a more straightforward fine-tuning process compared to the text reasoning pipeline. The base model, Gemini 1.5 Pro, is already natively multimodal — it can process interleaved text, images, audio, and video — and has shown strong zero-shot performance on general multimodal benchmarks. However, medical imaging modalities (radiology, pathology, dermatology) have data distributions quite different from natural images, and the vocabulary and reasoning patterns for describing them are domain-specific. Fine-tuning addresses this distribution shift.

Fine-Tuning Datasets and Task Formulation

The model is fine-tuned on eight tasks spanning six datasets, all reformulated as instruction-following problems (Table D1, Figure D1):

Slake-VQA (9,849 training examples): Radiology visual question answering in both English and Chinese. Questions can be open-ended ("What organ is shown in this image?") or close-ended ("Is there cardiomegaly present?"). The task is formulated as: [Image] + [Question in English or Chinese] → [Answer].
Path-VQA (19,755 training examples): Pathology visual question answering from pathology textbook images. Questions cover color, location, appearance, shape, and other visual attributes of histopathology specimens. Approximately 50% open-ended, 50% yes/no. Task formulation: [Image] + [Question] → [Answer].
ROCO (29,907 training examples): Radiology and non-radiology image captioning from PubMed Central figures. The task is to generate a text description of the image content. Task formulation: [Image] + [Caption generation instruction] → [Caption].
PAD-UFES-20 (1,838 training examples): Dermatology image classification into six skin lesion types (basal cell carcinoma, melanoma, squamous cell carcinoma, actinic keratosis, melanocytic nevus, seborrheic keratosis). Each image includes up to 21 clinical features (age, gender, lesion location, size, etc.). The task is formulated as a close-ended multiple-choice question: [Image] + [Clinical features] + [Classification question with 6 options] → [Diagnosis]. The paper also creates two auxiliary classification tasks: one with RandAugment data augmentation (8 operations: autoContrast, equalize, invert, rotate, posterize, solarize, color, contrast) and one with upsampled minority classes to address class imbalance.
MIMIC-CXR (four sub-tasks, totaling approximately 493,442 training examples):
- Normal vs. abnormal binary classification: Classify frontal CXR images (AP/PA views) as normal or abnormal based on the CheXpert "no finding" label. Task: [Image] + [Binary classification question] → [Normal/Abnormal].
- 13-class abnormality classification: For images with abnormal findings (excluding normal images), identify which of 13 CheXpert conditions are present. Labels are derived from the CheXpert labeller, with uncertain labels grouped as positive. Task: [Image] + [Multiple-choice question with 13 options] → [Condition]. The 13 conditions are: atelectasis, cardiomegaly, consolidation, edema, enlarged cardiomediastinum, fracture, lung lesion, lung opacity, pleural effusion, pleural other, pneumonia, pneumothorax, support devices.
- Synthetic VQA: Automatically generated yes/no question-answer pairs extracted from radiology reports by querying Gemini base models. The authors specifically ensure balanced yes/no ratios to avoid spurious correlations. Task: [Image] + [Synthetic question] → [Yes/No].
- Report generation: Generate the FINDINGS and IMPRESSION sections of a radiology report given the CXR image and the INDICATION section as context. Following Tanno et al. (2024), training examples whose reports reference prior studies are filtered out to reduce hallucination of non-existent prior images. Task: [Image] + [Indication text] + [Report generation instruction] → [Findings and Impression].

Mixture ratios: The paper states that "the mixture ratio for each task is approximately proportional to the number of training samples in each dataset." This means that larger datasets like MIMIC-CXR (with hundreds of thousands of examples) dominate the training mixture, while smaller datasets like PAD-UFES-20 (with ~1,800 examples) receive proportionally less weight.

Evaluation of MIMIC-CXR fine-tuning is deferred. The paper notes that "the evaluation of MIMIC-CXR will be reported in a subsequent paper" (Appendix D.1). This means that the MIMIC-CXR tasks are used during training but their impact on downstream radiology report generation quality is not assessed in this paper. This is a notable gap: we cannot tell from the current results whether the extensive CXR fine-tuning improved, maintained, or degraded the model's radiology capabilities.

Multimodal Evaluation Benchmarks

The paper evaluates on seven benchmarks, divided into in-distribution (overlapping with fine-tuning data) and out-of-distribution:

In-distribution (fine-tuned on these datasets or very similar distributions):

PAD-UFES-20 (dermatology classification)
Slake-VQA (radiology VQA)
Path-VQA (pathology VQA)
ECG-QA (for Med-Gemini-S 1.0 only)

Out-of-distribution (not used in any training):

NEJM Image Challenge (clinical images with diagnostic questions)
USMLE-MM (multimodal USMLE sample questions)
MMMU-HM (health and medicine subset of MMMU)

The out-of-distribution benchmarks are evaluated using Med-Gemini-L 1.0 without any multimodal fine-tuning, testing the model's zero-shot multimodal medical knowledge acquired during Gemini's pretraining. This is a deliberate design choice: it demonstrates that Gemini's native multimodal capabilities already surpass GPT-4V on these benchmarks, and that the fine-tuned Med-Gemini-M 1.5 provides additional gains only where needed (specialized medical imaging).

ECG-Specific Encoder (Med-Gemini-S 1.0)

For ECG analysis, the authors take a fundamentally different architectural approach because ECG data is a raw time-series biosignal, not an image. Even though ECGs can be visualized as 12-lead plots (and humans interpret them visually), the native format is a multi-channel waveform sampled over time. The paper argues that processing raw waveforms rather than rendered images preserves temporal resolution and amplitude precision that might be lost in rasterization.

Architecture: Med-Gemini-S 1.0 augments Gemini 1.0 Nano (the smallest, on-device-capable variant) with a specialized ECG encoder that connects to the Gemini backbone through cross-attention layers based on the Flamingo architecture (Alayrac et al., 2022).

The Flamingo-style cross-attention works as follows: the ECG encoder processes the raw multi-channel waveform into a sequence of feature vectors (one per time step or time chunk). These feature vectors are not directly concatenated with the text input. Instead, at certain layers of the Gemini transformer, cross-attention layers attend from the text token representations to the ECG feature vectors. This allows every text token to dynamically query relevant information from the ECG signal — for example, when the model is about to generate text about "ST elevation," it can attend to the corresponding temporal window in the ECG features where ST segments are typically analyzed.

Why cross-attention over concatenation: Concatenating the ECG features directly into the input sequence would significantly increase sequence length (an ECG sampled at 500 Hz for 10 seconds produces 5,000 time points per lead × 12 leads = 60,000 raw samples, which even after compression would be a very long prefix). Cross-attention avoids this by making the ECG features a separate attended context, not part of the main sequence. Additionally, cross-attention allows the model to learn which portions of the ECG are relevant to each word it generates, rather than treating the entire ECG as a fixed prefix.

Training: The model is fine-tuned on a 159,306-example subset of ECG-QA (single-ECG questions only, excluding two-ECG comparison questions). The task is formulated as close-ended question answering: [Raw ECG waveform] + [Question about cardiac condition] → [Answer]. The paper experiments with both frozen Gemini parameters (only the ECG encoder and cross-attention layers are trained) and unfrozen Gemini parameters (the entire model is fine-tuned), achieving 57.7% and 58.4% macro-averaged accuracy respectively.

The paper does not specify the ECG encoder architecture (CNN? Transformer? Hybrid?), the training hyperparameters, or the cross-attention layer placement details beyond referencing Alayrac et al. (2022) as the basis.

Long-Context Processing via Instruction Prompting and Chain-of-Reasoning (Med-Gemini-M 1.5)

Unlike the text reasoning and multimodal pipelines, the long-context capabilities of Med-Gemini-M 1.5 require no fine-tuning. All long-context evaluations are performed using in-context learning (one-shot or zero-shot) with Gemini 1.5 Pro's native long-context window, which supports up to millions of tokens. This is a deliberate demonstration that the base model's architecture — specifically, the mixture-of-experts design enabling efficient processing of very long sequences — already supports sophisticated medical long-context reasoning without task-specific training.

Chain-of-Reasoning for Long EHR Understanding

The EHR "needle-in-a-haystack" task requires determining whether a specific medical condition (e.g., "hypothermia") is present in a patient's complete electronic health record, which may contain hundreds of clinical notes totaling 200,000–700,000 words. The challenge is that the condition may be mentioned only once, possibly in an ambiguous context (e.g., "concern for sepsis vs. hypothermia" in a differential diagnosis list rather than a confirmed diagnosis).

The solution is a two-step chain-of-reasoning process applied via one-shot prompting (Figure 8):

Step 1: Evidence retrieval. Med-Gemini-M 1.5 is prompted to retrieve all mentions of the target condition from the full EHR and produce direct quotes with note identifiers. The one-shot demonstration shows the model an example of this retrieval behavior: given a patient's records and a condition name, the model identifies all text snippets containing that condition, provides the exact quoted text, and records the note ID or source for each mention.

The prompt structure for Step 1 (Figure 8a) includes:

The full patient EHR (all notes concatenated, totaling 200K–700K words)
A one-shot example showing retrieval of mentions for a different condition in a different patient
An instruction to find all mentions of the specific target condition and provide direct quotes with note IDs

The model's output is a list of evidence snippets, each with a direct quote and source identification.

Step 2: Existence determination. The retrieved mentions from Step 1, along with the original task instruction about the condition, are fed back to Med-Gemini-M 1.5 with a second prompt (Figure 8b). This prompt instructs the model to evaluate each retrieved mention and determine whether the patient actually has a history of the condition. The one-shot demonstration shows the model categorizing mentions as "explicit confirmation," "strong indication," or "relevant mention" and then providing a binary yes/no conclusion with reasoning.

The crucial distinction made in this second step is between a mention of a condition and a confirmed diagnosis. The model must distinguish between:

A condition stated as a definitive diagnosis ("Patient has hypothermia")
A condition mentioned in a differential diagnosis or concern ("Rule out hypothermia")
A condition mentioned in family history or social history (which does not indicate the patient has it)
A condition referenced in a negated context ("No evidence of hypothermia")

This requires nuanced clinical language understanding that goes beyond keyword matching. The baseline heuristic method (Feder et al., 2022) required manually crafted rules for each condition to handle these distinctions. Med-Gemini-M 1.5 performs this reasoning via in-context learning with a single example, demonstrating that the model has internalized the clinical concept of "diagnosis versus mention" from its pretraining.

Why a two-step process rather than single-step. The paper designs the chain-of-reasoning to separate retrieval from reasoning because the full EHR context is too long for the model to simultaneously hold all evidence snippets in working memory while performing detailed reasoning about each one. By first extracting the relevant snippets (drastically reducing context length), the second step can operate on a focused, manageable set of evidence. This is analogous to how a human clinician would approach the task: first scan the record for mentions of the condition, then carefully evaluate each mention's clinical significance.

Dataset Curation for the Needle-in-a-Haystack Task

The evaluation dataset is constructed with deliberate difficulty:

Patient selection: 44 unique ICU patients from MIMIC-III are selected based on having more than 100 "high-value" clinical notes (defined as notes authored by physicians, PAs, NPs, or APRNs, including consult notes, progress notes, history and physical notes, discharge summaries, and operative notes). Structured EHR data (labs, vitals, medications) is excluded; the task uses only unstructured text.
Condition identification: For each patient, the heuristic annotation system from Feder et al. (2022) is run to identify all medical problems (conditions, symptoms, procedures) documented in the patient's records. The system first labels text spans as mentions of problems using machine learning annotators, then applies rule-based selection and aggregation to determine which problems are actually present.
Needle selection: Conditions with exactly one evidence snippet identified in the aggregation step are selected. These are "needle" conditions — mentioned rarely enough that missing the single mention would result in missing the condition entirely. From these, 100 positive examples (conditions confirmed present) and 100 negative examples (conditions mentioned but absent — e.g., "concern for sepsis" without actual sepsis diagnosis) are randomly sampled.
Ground truth establishment: Three human medical raters independently review each of the 200 examples. Raters see the condition name and the retrieved evidence snippets, then answer "Select ALL problems the patient HAS HAD based on the evidence." The majority vote determines the ground truth label. The final dataset contains 121 positive and 79 negative examples, with Krippendorff's alpha of 0.77 indicating substantial inter-rater agreement.

The negative examples are particularly interesting from a clinical perspective. A mention in an EHR does not equate to a diagnosis — phrases like "received IV Ceftriaxone for concern of UTI/sepsis" indicate consideration of sepsis without confirmation. The model must recognize this uncertainty and not over-claim conditions. The paper provides a specific example: a patient with one mention of "Sepsis" in the text "Received IV Ceftriaxone for concern of UTI/sepsis" — this is rated as negative (the patient does not have confirmed sepsis) because the context indicates it was a suspicion, not a diagnosis.

Instruction Prompting for Medical Video Understanding

For video tasks, Med-Gemini-M 1.5 is prompted zero-shot with task-specific instructions. Three video tasks are evaluated:

Medical Visual Answer Localization (MedVidQA). The task is to identify the temporal segment (start and end timestamps) in a medical instructional video that answers a natural language question. For example, given the question "How do you relieve calf strain?" and a physical therapy video, the model must output the timestamp range where the relevant technique is demonstrated.

The prompt (Figure 10) includes:

The video (processed natively by Gemini 1.5 Pro's video understanding pipeline)
Optionally, the video subtitles/captions (for the "with subtitles" variant)
The question about the medical procedure
Instructions to output the answer as a timestamp span

The paper evaluates two variants: one using both video and subtitle text, and one using only video. The subtitle variant achieves substantially higher performance (mIoU 65.8 vs. 43.4 without subtitles; Table 3), indicating that text captions significantly aid the model's understanding.

Critical View of Safety (CVS) Assessment. For laparoscopic cholecystectomy videos from the Cholec80-CVS dataset, the model must assess whether three criteria defining the Critical View of Safety are met, each scored 0, 1, or 2:

Clearance of the hepatocystic triangle: The triangular area between the cystic duct, common hepatic duct, and liver edge must be cleared of fat and fibrous tissue.
Two structures connected to the gallbladder: Only the cystic duct and cystic artery should be visible entering the gallbladder.
Two structures ONLY connected to the gallbladder: No other structures (like the common bile duct) should appear connected.

The prompt (Figure 9) includes the surgical video clip and instructions to evaluate each criterion separately, providing a score and explanation. The model's output includes detailed reasoning for each criterion, referencing specific visual features visible in the video (e.g., "visible hepatocystic triangle but not fully cleared of fat").

Surgical Action Recognition (qualitative only). Using the AVOS dataset (Annotated Videos of Open Surgery), the model is prompted (Figure 11) to segment an open surgical video into temporal segments and label each with the surgical action being performed (cutting, tying, suturing, or background). This is presented as a qualitative demonstration only, with no quantitative metrics.

Why zero-shot rather than fine-tuned for video. The paper's video results demonstrate that Gemini 1.5 Pro's native video understanding capabilities, combined with domain-appropriate prompting, can achieve competitive or state-of-the-art performance without any surgical video fine-tuning. This is significant because fine-tuning on surgical video would require large annotated datasets that are expensive to create and specific to surgical specialties. The zero-shot approach suggests that medical video understanding may be achievable through improved prompting strategies alone, though the paper acknowledges that "further investigations on prompting strategies or instruction fine-tuning may be required to improve the task performance" (Section 4.3), particularly since the supervised ResNet3D baseline outperforms Med-Gemini on CVS assessment (67.0% vs. 50.0% accuracy).

Summary of Design Choices and Their Justifications

Self-training with correctness filtering over human annotation. Generating expert clinical reasoning demonstrations at scale would require hiring clinicians to write detailed CoTs for thousands of questions — expensive, slow, and difficult to standardize. The self-training approach leverages the model's own capabilities, using only five hand-crafted seeds to establish style, and then bootstraps to high-quality training data through iterative refinement with correctness-based filtering. The correctness filter ensures the model learns from its successes, not its failures, making the process self-correcting.

Uncertainty-guided search over always-on retrieval. Always-on retrieval would add latency and cost for every query, including simple questions the model can answer confidently from parametric knowledge. The entropy-based threshold targets search to situations where the model explicitly recognizes its own uncertainty, mimicking clinical information-seeking behavior and potentially reducing over-reliance on search results for questions where the model's internal knowledge is sufficient.

Model family over single generalist system. A single model optimized for all tasks would face deployment trade-offs: the largest model needed for complex clinical reasoning (Med-Gemini-L 1.0) might be too expensive or slow for real-time multimodal dialogue; the smallest model needed for on-device ECG analysis (Med-Gemini-S 1.0) might lack sufficient capacity for long-context EHR reasoning. By offering a family of models, each can be deployed in its appropriate setting. The Gemini ecosystem's shared architecture makes this feasible — the same pretraining substrate supports specialization into different capability profiles without starting from scratch for each variant.

Two-step chain-of-reasoning over single-pass processing for long EHRs. Processing a 500,000-word EHR in a single pass while simultaneously retrieving, evaluating, and deciding on a condition's presence would challenge even a model with a million-token context window — the model would need to keep track of every mention across hundreds of pages while performing nuanced clinical reasoning. By separating retrieval (a scanning task) from determination (a reasoning task), the chain-of-reasoning reduces the cognitive load in each step and produces intermediate outputs (the retrieved evidence) that are themselves interpretable and auditable.

Cross-attention ECG encoder over signal-to-image conversion. Converting raw ECG waveforms to images (12-lead plot images) and then processing them with a vision encoder would lose temporal precision (the exact millisecond timing of waveform features) and add unnecessary rendering artifacts. The cross-attention approach preserves the raw signal and allows the text generation process to query specific temporal regions of the waveform dynamically, which is important for ECG features like PR interval duration or ST segment elevation that are defined by precise temporal relationships.

One-shot in-context learning for EHR retrieval over task-specific fine-tuning. The zero-shot and one-shot nature of the long-context evaluations demonstrates that Gemini 1.5 Pro's native capabilities are sufficient for these tasks. Fine-tuning would risk overfitting to the specific conditions and note formats in MIMIC-III, reducing generalization to different EHR systems, note styles, or medical conditions. The in-context approach means the model can be deployed on novel retrieval tasks (new conditions, new note formats) without retraining.

4. Key Insights and Innovations

Innovation 1: Uncertainty-Guided Search as a Reasoning Epistemology, Not Just a Retrieval Mechanism

The paper's most intellectually distinctive contribution is a conceptual reframing of when and why a medical AI system should consult external knowledge. Prior work on retrieval-augmented generation in medicine treated web search as a content pipeline: either always-on retrieval (augment every query with search results) or static retrieval (pre-indexed medical corpora queried by keyword matching). These approaches treat search as a data source — an alternative to parametric knowledge that is always available but may or may not be relevant. MedPrompt (Nori et al., 2023), for example, achieved strong MedQA results through sophisticated prompting, but did not model the model's own state of knowledge to decide when external information would help.

The key conceptual move in Med-Gemini-L 1.0's uncertainty-guided search is to treat the model's own epistemic state as the trigger for information seeking. The Shannon entropy over answer choices, computed across multiple sampled reasoning paths, serves as a quantitative measure of the model's uncertainty. High entropy means the model is internally conflicted — its own reasoning paths disagree about the answer. Rather than always retrieving or never retrieving, the system retrieves only when the model recognizes that it does not know, and crucially, it conditions the search queries on the specific nature of the disagreement among its own reasoning paths.

This represents a shift from retrieval as augmentation (adding more information to an already-formed reasoning process) to retrieval as uncertainty resolution (using external knowledge to adjudicate between the model's own competing hypotheses). It is explicitly analogized by the authors to the Pathfinder diagnostic system (Horvitz et al., 1984), which used entropy-based criteria to decide when to gather additional clinical evidence. By resurrecting this classic AI concept within the architecture of a modern LLM, the paper bridges decades-old work on diagnostic reasoning under uncertainty with contemporary foundation model capabilities.

Why this is fundamental rather than incremental. The uncertainty-guided search strategy is not simply a better search mechanism — it models a cognitive capability that distinguishes expert clinicians from knowledge bases. Experienced physicians do not consult UpToDate or PubMed for every patient encounter; they recognize when their own knowledge is sufficient and when a specific knowledge gap requires targeted investigation. The construction of search queries conditioned on identified conflicts between one's own competing hypotheses — rather than on the question alone — mirrors this expert behavior. The paper demonstrates that LLMs can be trained to emulate this meta-cognitive skill through the self-training with search data generation process, where the model learns not just to integrate search results but to generate queries that resolve the specific ambiguities it has identified.

Prior assumption this challenges. The dominant assumption in medical LLM research has been that better reasoning comes from better training data, more parameters, or cleverer prompting. This paper demonstrates that admission and diagnosis of one's own uncertainty, followed by targeted information retrieval, is an independent and complementary path to improved performance. The ablation in Figure 4a shows that self-training alone (without search) raises accuracy from approximately 84.3% to 87.5% on MedQA, while adding search at inference further raises it to 91.1%—a gain from reasoning improvement plus a gain from uncertainty-driven knowledge acquisition. These are separable effects, suggesting that uncertainty-guided retrieval is not just helping the model "know more" but helping it know what it needs to know.

Evidence anchor. Figure 4a shows the progressive improvement from each round of uncertainty-guided search (87.2% → 88.8% → 90.0% → 90.5% → 91.1% over four iterations plus the initial self-trained model without search). The continued gains across iterations suggest that each search round resolves some residual uncertainty, which then reveals new, more specific uncertainties that subsequent rounds can address. This stair-step pattern is consistent with the model iteratively refining its understanding, not simply retrieving one piece of missing information. The generalization of this strategy to NEJM CPC (Figure 3a: +4.0% top-10 accuracy from adding search) and GeneTuring (Table 1: SoTA on 7 of 12 modules) further demonstrates that uncertainty-guided search transfers across task formats without task-specific engineering.

Innovation 2: The Benchmark Saturation Argument — SoTA Gains on MedQA May No Longer Reflect Clinically Meaningful Progress

The paper makes a methodological contribution that will likely influence how the field evaluates medical AI systems: it demonstrates, through systematic expert re-annotation, that a substantial fraction of remaining errors on the MedQA (USMLE) benchmark are attributable to benchmark quality rather than model capability. This is not merely a data-cleaning effort — it is an argument that further improvements in SoTA performance on MedQA may not directly correlate to progress in the capabilities of medical LLMs for meaningful real-world tasks (Section 5).

The diagnostic move. The paper recruits 18 US board-certified primary care physicians to relabel all 1,273 MedQA test questions, collecting at least three independent ratings per question. Through bootstrapped committee voting, they identify that:

~3.8% of questions contain missing information (unanimous committee agreement). For example, a question referencing a figure, lab result, or clinical finding that is not provided in the text — making the question unanswerable regardless of medical expertise.
~2.9% likely have labeling errors (the ground truth answer from MedQA disagrees with the unanimous consensus of three physicians).
~0.7% are ambiguous (allow multiple correct answers even after revealing the ground truth to raters).

Filtering these questions raises Med-Gemini-L 1.0's accuracy from 91.1% to 91.8% ± 0.2% (using unanimous voting) or 92.9% ± 0.38% (using majority voting, which discards up to 20.9% of uncertain questions). More importantly, the analysis reveals that Med-Gemini-L 1.0's mistakes are disproportionately concentrated on questions with label errors and missing information, and that the model's own entropy-based uncertainty score tends to be higher on these questions (p = 0.033).

Why this transcends data cleaning. This finding changes how the field should interpret MedQA benchmark results. At 91.1% accuracy, the model is making errors on roughly 113 questions. If a meaningful fraction of those errors are on questions that expert clinicians themselves consider flawed, then the remaining "room for improvement" on MedQA is smaller than the raw accuracy number suggests. Chasing the next 1–2% gain may optimize for a mixture of genuine reasoning improvement and better fit to benchmark artifacts. The paper's own result — surpassing MedPrompt (90.2%) by 0.9% — would look quite different if the analysis revealed that most of that gain came from correctly answering questions with missing information or label errors (which would not represent genuine clinical reasoning improvement).

Comparison to prior benchmark assessment. While prior work (Stutz et al., 2023) had developed frameworks for evaluating AI systems under uncertain ground truth, and individual studies had noted quality issues in specific medical benchmarks (Xu et al., 2023), no prior paper of this scale had systematically relabeled an entire popular benchmark with multiple independent expert raters and used bootstrapped committee voting to quantify the impact of label uncertainty on model evaluation. This elevates the finding from anecdotal observation to reproducible methodology.

The broader implication for medical AI evaluation. The paper explicitly extends the argument beyond MedQA: "prior work has suggested similar issues with other popular benchmark datasets" (Section 5), and the authors demonstrate that re-splitting the PAD-UFES-20 dermatology dataset at the patient level (rather than image level) causes a 7.1% performance drop. This suggests that benchmark data quality and evaluation protocols — independent of model capability — can significantly affect reported results. The implication is that comprehensive benchmarking across diverse tasks, as this paper does with 14 benchmarks, is more informative than optimizing for a single benchmark number, and that the field needs better tools for characterizing when benchmarks have been "solved" in a way that makes further progress on them uninformative about real-world capability.

Evidence anchor. Figure 4b and Figure C1 provide the detailed breakdown of how accuracy changes as questions with missing information, label errors, and ambiguity are filtered. The bootstrapping experiment (1,000 repeated committee samples) provides confidence intervals on both the accuracy improvement and the fraction of questions discarded, ensuring the result is not dependent on a single rater's idiosyncratic judgments.

Innovation 3: The Model Family as Architecture — Specialization Within a Unified Foundation as an Alternative to Generalist Medical AI

The paper makes an architectural meta-contribution by explicitly rejecting the generalist medical AI paradigm that had become dominant in the field. Moor et al. (2023a) articulated the vision of GMAI — a single model handling all medical tasks across all modalities — and systems like Med-PaLM M (Tu et al., 2024a) demonstrated early feasibility. The default assumption in follow-up work was that increasing model scale and training data diversity would eventually produce a unified system that matched or exceeded specialist performance everywhere.

Med-Gemini takes the opposite position: shared foundations plus targeted specialization yields better real-world outcomes than a single generalist system, because deployment constraints (model size, inference latency, training data requirements, task-specific optimization needs) create trade-offs that a one-size-fits-all approach cannot satisfy. This is not presented as a theoretical argument but as an empirical design choice embodied in the model family structure.

The conceptual structure. The model family spans four capability profiles, each realized through a different combination of base model size, fine-tuning data, and architectural customization:

Med-Gemini-L 1.0 (Gemini 1.0 Ultra): Largest model, fine-tuned for complex clinical reasoning with web search. Optimized for accuracy on challenging diagnostic and genomic reasoning benchmarks where inference cost is secondary to correctness.
Med-Gemini-M 1.5 (Gemini 1.5 Pro): Mid-size model with native long-context capabilities, fine-tuned on multimodal medical data. Optimized for tasks requiring both visual understanding and language reasoning — radiology VQA, dermatology classification, medical video understanding, and long EHR retrieval — where the ability to process entire documents or videos is critical.
Med-Gemini-M 1.0 (Gemini 1.0 Pro): Mid-size model, fine-tuned for long-form medical text generation (summarization, referral letters, simplification). Optimized for assistive documentation tasks where output quality and fluency matter more than complex diagnostic reasoning.
Med-Gemini-S 1.0 (Gemini 1.0 Nano with custom ECG encoder): Smallest model, on-device capable, with a modality-specific encoder for biosignal analysis. Optimized for deployment in resource-constrained settings where privacy, latency, or offline operation is essential.

What distinguishes this from simply "we built multiple models." The intellectual contribution is not that models can be fine-tuned for different tasks — that is obvious. The contribution is framing specialization as a first-class architectural principle rather than a compromise. Each model variant makes deliberate capability trade-offs that are aligned with specific deployment scenarios. The largest model is not always the best choice; it would be inappropriate for on-device ECG analysis (too large, too slow) and unnecessary for referral letter generation (where the mid-size model already matches or exceeds human experts). The smallest model is not just a compressed version of the larger ones; it has a fundamentally different architecture (custom encoder) that enables capabilities the larger models lack (processing raw ECG waveforms without rasterization).

Evidence that the family structure is necessary, not just convenient. The paper demonstrates that Gemini's zero-shot multimodal performance (Med-Gemini-L 1.0 without fine-tuning) already achieves SoTA on out-of-distribution benchmarks like NEJM Image Challenge and USMLE-MM, surpassing GPT-4V by large margins. This suggests that for general medical images (dermatology photos, radiographic images, endoscopy stills), the base model's pretraining is sufficient — no specialization is needed. However, for specialized modalities like pathology images (Path-VQA) and ECG waveforms (ECG-QA), fine-tuning provides substantial gains that zero-shot performance cannot match, and for raw biosignals, a custom encoder is necessary because the base model has no native support for time-series waveform data. This differential need for specialization across modalities provides empirical support for the model family approach: some tasks benefit enormously from specialization, while others work well out-of-the-box. A single model optimized for all tasks would either be unnecessarily specialized for easy modalities or inadequately specialized for hard ones.

Evidence anchor. The multimodal results in Table 2 illustrate the differential specialization pattern: Med-Gemini-L 1.0 (zero-shot, no fine-tuning) achieves 69.7% on NEJM Image Challenge (outperforming GPT-4V at 61.0%), while Med-Gemini-M 1.5 (fine-tuned) achieves 64.7% token F1 on Path-VQA (outperforming the prior SoTA Med-PaLM M at 62.7%), and Med-Gemini-S 1.0 (custom encoder) achieves 57.7% on ECG-QA (outperforming GPT-4 with SE-WRN at 51.6%). These are three different models optimizing three different capability profiles — a single model could not simultaneously achieve all three results.

Innovation 4: Long-Context Medical Processing Demonstrates That In-Context Learning Can Replace Task-Specific System Engineering

The paper's long-context results demonstrate something more significant than the specific performance numbers: they show that general-purpose long-context foundation models can, through in-context learning alone, match or approach the performance of carefully engineered, task-specific systems that required extensive manual feature design and domain expertise to build. This represents a shift in how medical AI systems for complex data processing can be developed — from bespoke engineering per task to general capabilities deployed via prompting.

The contrast with prior engineering-heavy approaches. For the long EHR "needle-in-a-haystack" retrieval task, the baseline comparison is the heuristic annotation-aggregation system from Feder et al. (2022). That system required: (1) training or adapting machine learning annotators to label medical problem mentions in clinical notes, (2) designing rule-based selection criteria to decide which mentions indicate true conditions (vs. negated, historical, or family-history mentions), (3) engineering rule-based aggregation logic to combine evidence across notes, and (4) manually extending coverage to new medical conditions — a process that is ontology-dependent and requires clinical and engineering expertise for each new condition. The system is effective (F1 of 0.78 on this task) but brittle: it cannot generalize to conditions outside its engineered rule set, and adapting it to a new EHR system with different note formats or coding standards would require substantial re-engineering.

Med-Gemini-M 1.5 achieves an F1 of 0.77 on the same task using one-shot in-context learning — a single example demonstrating the retrieval and reasoning pattern, with no task-specific training, no feature engineering, no ontology construction, and no rule writing. The model is shown one example of how to retrieve mentions and determine existence, and it generalizes this pattern to 199 other conditions and patient records it has never seen.

The conceptual significance. This result suggests that the challenging aspects of medical information extraction — distinguishing confirmed diagnoses from differential diagnoses, recognizing negated mentions, understanding that "family history of X" does not mean the patient has X — are not specialized clinical reasoning skills that must be explicitly programmed. They are implicit in the language understanding capabilities that emerge from training a sufficiently large multimodal model on sufficiently diverse data. The model has internalized the pragmatic distinction between "Patient has pneumonia" (confirmation) and "Rule out pneumonia" (suspicion) and "Maternal grandmother had pneumonia" (family history) — distinctions that the prior heuristic system required hand-crafted rules to capture.

For video, the contrast is even sharper. The medical video understanding results achieve state-of-the-art on MedVidQA using zero-shot prompting — no surgical video training data at all. Prior state-of-the-art systems (Li et al., 2022; Weng and Li, 2023) were architectures specifically designed for the medical visual answer localization task, trained on the MedVidQA training set. Med-Gemini-M 1.5, prompted with task instructions and no training, achieves a mIoU of 65.8 with subtitles (vs. 58.3 for the prior SoTA) and 43.4 without subtitles (vs. 27.5). The fact that the model can watch an instructional medical video, understand the question, and localize the relevant segment — all without task-specific training — implies that Gemini's pretraining on general video understanding transfers to the medical domain.

The boundary of this capability — and why it matters. The paper does not claim that in-context learning solves everything. For the CVS assessment task, Med-Gemini-M 1.5 achieves 50.0% accuracy, substantially below the supervised ResNet3D baseline at 67.0%. This is attributed to the fact that both models can be treated as "the same" but the supervised model is trained specifically on CVS criteria in laparoscopic cholecystectomy videos — it has seen hundreds of examples of what a "cleared hepatocystic triangle" looks like. Med-Gemini, relying on zero-shot understanding of the CVS criteria description, cannot match this specialized visual discrimination. The fact that in-context learning works for some video tasks (instructional video QA) but not others (surgical safety assessment) is itself informative: it identifies where general video understanding is sufficient and where domain-specific visual features require specialized training.

Evidence anchor. Table 3 provides the quantitative comparison: Med-Gemini-M 1.5's one-shot F1 of 0.77 vs. the heuristic baseline's 0.78 on EHR retrieval; Med-Gemini-M 1.5's zero-shot mIoU of 65.8/43.4 vs. prior SoTA of 58.3/27.5 on MedVidQA (with/without subtitles); and the 50.0% vs. 67.0% gap on CVS assessment, highlighting the capability boundary. Table E2 provides confidence intervals for the EHR comparison, showing the performance is statistically indistinguishable from the engineered baseline.

5. Experimental Analysis

Evaluation Methodology

Datasets. The paper evaluates on 14 medical benchmarks spanning text-based reasoning, multimodal understanding, and long-context applications (Table 1, Table 2, Table 3, Table A1). The text-based reasoning benchmarks include: MedQA (USMLE) — 1,273 test questions in a 4-option multiple-choice format curated by Jin et al. (2021); NEJM CPC — complex diagnostic case challenges from the New England Journal of Medicine, evaluated at top-1 and top-10 accuracy following McDuff et al. (2023); GeneTuring — 600 open and close-ended QA pairs across 12 modules testing genomic knowledge (Hou and Ji, 2023). The multimodal benchmarks include: NEJM Image Challenge — 934 clinical images with 5-option diagnostic questions (Buckley et al., 2023); USMLE-MM — 46 multimodal USMLE sample questions; MMMU-HM — 150 questions from the health and medicine subset of MMMU (Yue et al., 2023); Path-VQA — pathology VQA with 6,761 test QA pairs (He et al., 2020); Slake-VQA — bilingual radiology VQA with 2,070 test QA pairs (Liu et al., 2021); PAD-UFES-20 — 6-class dermatology skin lesion classification (Pacheco et al., 2020); and ECG-QA — single-ECG close-ended question answering with 41,093 test samples (Oh et al., 2023). The long-context benchmarks include: MIMIC-III Needle-in-a-Haystack — a newly curated dataset of 200 examples (121 positive, 79 negative) where a medical condition is mentioned exactly once across 200K–700K words of de-identified EHR notes; MedVidQA — 148 test questions for medical visual answer localization in instructional videos (Gupta et al., 2023); and Cholec80-CVS — 572 annotated video clips for Critical View of Safety assessment in laparoscopic cholecystectomy (Ríos et al., 2023; Twinanda et al., 2016). Additionally, the paper evaluates on three real-world long-form text generation tasks using custom de-identified clinical datasets: medical summarization (31 H&P notes with expert after-visit summaries), referral letter generation (25 outpatient notes with expert referral letters), and medical simplification (25 Cochrane systematic review abstracts with plain language summaries from Devaraj et al., 2021).
Base models. The paper uses four Gemini variants as foundation models: Gemini 1.0 Ultra (the largest model, used for complex text-based reasoning in Med-Gemini-L 1.0), Gemini 1.5 Pro (used for multimodal fine-tuning and long-context processing in Med-Gemini-M 1.5, chosen for its native long-context window supporting millions of tokens and natively multimodal architecture), Gemini 1.0 Pro (used for long-form text generation tasks in Med-Gemini-M 1.0), and Gemini 1.0 Nano (the smallest model, augmented with a custom ECG encoder for Med-Gemini-S 1.0, chosen for on-device deployment feasibility). The choice to use multiple base model sizes is explicitly motivated by deployment trade-offs: complex diagnostic reasoning requires the largest model's capacity, while on-device ECG analysis requires a small model's efficiency, and long-context medical record processing requires Gemini 1.5 Pro's extended context window.
Metrics. For text-based multiple-choice benchmarks (MedQA, GeneTuring, USMLE-MM, MMMU-HM), the primary metric is prediction accuracy — the fraction of test questions where the model's final answer matches the ground truth. For NEJM CPC, top-1 and top-10 accuracy are reported: the fraction of cases where the correct diagnosis appears first or within the first 10 entries of the model's differential diagnosis list. For NEJM Image Challenge and PAD-UFES-20, classification accuracy is reported. For open-ended VQA tasks (Path-VQA, Slake-VQA), token-level F1 score is used following Tu et al. (2024a), which measures overlap between predicted and ground-truth answer tokens. For ECG-QA, macro-averaged accuracy across ECG question types is reported following Oh et al. (2023). For MedVidQA, Intersection over Union (IoU) at thresholds 0.3, 0.5, 0.7, and mean IoU (mIoU) are reported, measuring the temporal overlap between predicted and ground-truth video answer spans. For the long EHR task, precision, recall, and F1 are reported for binary condition presence classification. For long-form text generation tasks (summarization, referral letters, simplification), side-by-side blinded clinician preference is the evaluation method, with results reported as the fraction of evaluations where the model's output is rated as good or better than expert human output. For the MedQA relabeling study, inter-rater agreement is measured using average overlap between answer sets and Krippendorff's alpha.
Baselines. The paper compares against a comprehensive set of prior state-of-the-art systems. For text-based reasoning on MedQA: Med-PaLM 2 (Singhal et al., 2023b) and MedPrompt applied to GPT-4 (Nori et al., 2023), which achieved 90.2% accuracy using complex specialized prompting strategies. For NEJM CPC: AMIE (Tu et al., 2024b) and GPT-4 (McDuff et al., 2023). For GeneTuring: models reported in Hou and Ji (2023) including GPT-3.5 and GPT-4 variants with general web search. For multimodal benchmarks: GPT-4V evaluated on the same test sets using public APIs and the same prompts (for NEJM Image Challenge, USMLE-MM, MMMU-HM, PAD-UFES-20, and Slake-VQA); Med-PaLM M (Tu et al., 2024a) for Path-VQA, Slake-VQA, and PAD-UFES-20; GPT-4 with SE-WRN ECG features (Oh et al., 2023) for ECG-QA; and M³AE (Oh et al., 2023) for ECG-QA. For long-context tasks: the heuristic annotation-aggregation baseline from Feder et al. (2022) for long EHR understanding; prior SoTA models from Li et al. (2022) and Weng and Li (2023) for MedVidQA; and ResNet3D (trained on 5-fold cross-validation) and GPT-4V (using sampled frames at 1 fps, limited to 300 images) for CVS assessment. For long-form text generation, the baseline is human expert performance: clinician-written summaries, referral letters, and plain language summaries. The paper notes that direct GPT-4 comparisons were not possible for long-context tasks due to context window limitations in public APIs — the average context token length in the MIMIC-III Needle-in-a-Haystack dataset and most MedVidQA videos exceeds GPT-4's maximum context window.
Generation budget / compute accounting. For text-based reasoning tasks with uncertainty-guided search, the compute budget is measured in number of search iterations (up to 4 for MedQA, 1 for NEJM CPC and GeneTuring). The paper does not provide a token-level or FLOPs-level accounting of search cost. For multimodal tasks, generation is a single pass per input (no iterative refinement). For long-context EHR tasks, the chain-of-reasoning procedure uses two sequential generation steps. For video tasks, the full video is processed in a single pass through Gemini 1.5 Pro's native video understanding pipeline. Across all tasks, the paper does not standardize computational cost into a common unit (e.g., FLOPs, GPU-hours, or tokens generated), and no analysis is provided of the latency or monetary cost implications of different inference strategies. The difficulty estimation cost for the uncertainty-guided search (generating 2,048 samples per question for PRM-based difficulty binning in the prior compute-optimal scaling literature) is not applicable here since difficulty estimation is not used.
Cross-validation / statistical protocol. For the MedQA relabeling study, bootstrapping with 1,000 repeated committee samples is used to estimate confidence intervals around accuracy and fraction of remaining questions after filtering. Committees of 3 raters per question are sampled with replacement, and both majority and unanimous voting aggregation are evaluated. The paper reports mean ± standard deviation across bootstrap samples. For the long EHR task, three independent physician raters label each of the 200 examples, with majority vote serving as ground truth and Krippendorff's alpha (0.77) quantifying inter-rater agreement. For the long-form text generation evaluations, three independent U.S. board-certified physicians rate each example, and statistical significance of the preference rate (whether the model is preferred or tied with experts at a rate different from 0.5) is assessed using a two-sided t-test. For the CVS assessment baseline (ResNet3D), 5-fold cross-validation is used, with average accuracy across folds reported. A notable absence: the paper does not report confidence intervals or statistical significance tests for the main benchmark accuracy numbers (MedQA, NEJM CPC, GeneTuring, multimodal benchmarks), making it difficult to assess whether the reported improvements over prior SoTA (e.g., 91.1% vs. 90.2% on MedQA) are statistically reliable given test set sizes.

Main Quantitative Results

Text-Based Clinical Reasoning Benchmarks (Table 1, Figure 3, Figure 4)

MedQA (USMLE). Med-Gemini-L 1.0 achieves 91.1% accuracy on the MedQA test set using uncertainty-guided search with four iterations, establishing a new state-of-the-art (Table 1). This surpasses the prior SoTA of 90.2% from MedPrompt applied to GPT-4 (Nori et al., 2023) by 0.9 percentage points, and exceeds Med-PaLM 2 by 4.5 percentage points. The ablation in Figure 4a reveals the decomposition of this gain: the baseline Gemini 1.0 Ultra achieves approximately 84.3% accuracy before self-training (reading from the "No self-training" bar at 0 search iterations). Self-training alone (fine-tuning on MedQA-R and MedQA-RS without search at inference) raises accuracy to approximately 87.2%. Adding uncertainty-guided search then provides progressive gains: 88.8% after one iteration, 90.0% after two, 90.5% after three, and 91.1% after four. The total improvement from base model to final system is roughly 6.8 percentage points, with self-training contributing approximately 2.9 points and search contributing an additional 3.9 points across four iterations.

MedQA relabeling analysis (Figure 4b, Figure C1). The expert re-annotation of all 1,273 MedQA test questions by 18 U.S. board-certified primary care physicians reveals that approximately 3.8% of questions contain missing information (e.g., referenced figures or lab results not provided in the text) based on unanimous committee voting. An additional 2.9% likely contain label errors (the provided ground truth answer disagrees with unanimous physician consensus). Approximately 0.7% are ambiguous (allow multiple correct answers). Filtering questions with missing information using unanimous voting improves Med-Gemini-L 1.0's accuracy from 91.1% to 91.8% ± 0.2%. Using majority voting to filter all three categories (missing information, label errors, ambiguity) raises accuracy to 92.9% ± 0.38% by discarding up to 20.9% of questions. The inter-rater agreement is high: 94% for identifying missing information, 87.6% for identifying label errors, and 94.6% for identifying ambiguous questions. Importantly, Med-Gemini-L 1.0's mistakes are disproportionately concentrated on these problematic questions, and the model's own entropy-based uncertainty score is significantly higher on such questions (t-test, p = 0.033).

NEJM CPC. Med-Gemini-L 1.0 achieves 72.3% top-10 accuracy on the NEJM CPC complex diagnostic challenges (Figure 3a, Table 1), surpassing the prior SoTA AMIE model (59.1%) by 13.2 percentage points and GPT-4 (50.0%) by 22.3 percentage points. The addition of search at inference contributes a 4.0% absolute improvement to top-10 accuracy (from 68.3% without search to 72.3% with search). Performance stratified by medical specialty (Table C4) shows that Med-Gemini-L 1.0 with search achieves the best top-1 and top-10 accuracy in most specialties with at least 10 cases: Internal Medicine (top-1: 28.6%, top-10: 75.0%), Pediatrics (top-1: 50.0%, top-10: 83.3%), and Psychiatry (top-1: 45.5%, top-10: 72.7%). Pathology shows lower performance (top-1: 22.2%, top-10: 55.6%).

GeneTuring. Med-Gemini-L 1.0 outperforms prior SoTA models (Hou and Ji, 2023) on 7 of 12 GeneTuring modules (Table 1, Figure 3b), including Gene name extraction, Gene alias, Gene name conversion, Gene location, Protein-coding genes, Gene ontology, and TF regulation. The paper reports averaged accuracy of 53.3% for Med-Gemini-L 1.0 compared to 48.6% for the prior SoTA GPT-4-based models — a 4.7 percentage point improvement. The model's abstention rate varies across modules (Figure 3b); in modules where the model abstains more frequently, there is no penalty in the accuracy calculation since abstained responses are excluded from numerical evaluation. GeneGPT (Jin et al., 2024) achieves higher scores through specialized web APIs (not general web search), and is excluded from the primary comparison since Med-Gemini-L 1.0 uses only general web search.

Multimodal Understanding Benchmarks (Table 2)

Out-of-distribution benchmarks (zero-shot, Med-Gemini-L 1.0 without multimodal fine-tuning). Med-Gemini-L 1.0 achieves state-of-the-art performance on all three out-of-distribution multimodal benchmarks evaluated without any task-specific multimodal training:

NEJM Image Challenge: 69.7% accuracy, surpassing GPT-4V by 8.7 percentage points (from 61.0% to 69.7%) and the prior SoTA from Buckley et al. (2023) by an equal margin (Table 2). This benchmark spans diverse medical image types including radiographic images, dermatoscopic and clinical skin photographs, electrocardiograms, histopathology images, endoscopy images, and ophthalmoscopy images — the strong zero-shot performance demonstrates Gemini's broad pretraining coverage of medical imaging modalities.
USMLE-MM: 93.5% accuracy overall, surpassing GPT-4V by 13.1 percentage points (from 80.4% to 93.5%) (Table 2). Broken down by USMLE step: Step 1 achieves 89.5% (n=19), Step 2 achieves 92.9% (n=14), and Step 3 achieves 100.0% (n=13). The small sample sizes per step (13–19 questions) mean these step-level breakdowns should be interpreted cautiously.
MMMU-HM (health and medicine): 67.3% accuracy, surpassing GPT-4V by 2.6 percentage points (from 64.7% to 67.3%) (Table 2). This is the narrowest margin among the out-of-distribution benchmarks.

In-distribution benchmarks (fine-tuned models). The fine-tuned Med-Gemini variants achieve state-of-the-art on two of four in-distribution benchmarks, with competitive performance on the remaining two:

Path-VQA (Med-Gemini-M 1.5): 64.7% token F1 score, surpassing the prior SoTA Med-PaLM M (62.7%) by 2.0 percentage points and GPT-4V (measured by the authors using the same prompt, achieving approximately 36.0%) by a much larger margin (Table 2). This gain is attributed to fine-tuning on pathology-specific VQA data.
ECG-QA (Med-Gemini-S 1.0): 57.7% macro-averaged accuracy with frozen Gemini parameters, 58.4% with unfrozen parameters (Table 2). The frozen variant surpasses GPT-4 with SE-WRN ECG features (51.6%) by 6.1 percentage points, while the unfrozen variant surpasses the M³AE ECG foundation model (57.6%) by 0.8 percentage points. The small gap between frozen and unfrozen variants (0.7 percentage points) suggests that the custom ECG encoder captures most of the task-relevant signal, with fine-tuning the language model backbone providing marginal additional benefit.
Slake-VQA (Med-Gemini-M 1.5): 87.5% token F1, slightly below the prior SoTA Med-PaLM M at 89.3% (Table 2). This is the only in-distribution benchmark where Med-Gemini does not match or exceed prior SoTA, though the gap is modest (1.8 percentage points).
PAD-UFES-20 (Med-Gemini-M 1.5): 85.9% accuracy on the Med-PaLM M split, below Med-PaLM M's 88.0% (Table 2). The paper also evaluates on a new patient-level split (as opposed to the image-level split used by Med-PaLM M), which yields a lower accuracy (the exact value is not reported in the main text, but the paper notes a 7.1% drop, implying approximately 78.8% on the patient-level split). This performance discrepancy between image-level and patient-level splits highlights the sensitivity of benchmark results to evaluation protocol — images of the same patient's lesion appearing in both training and test sets inflates apparent performance when split is at the image level rather than patient level.

Aggregate multimodal comparison. Across the seven multimodal benchmarks, Med-Gemini models improve over GPT-4V by an average relative margin of 44.5%. This aggregate is computed from the five benchmarks where direct GPT-4V comparisons are available: NEJM Image Challenge (+8.7 percentage points absolute, relative gain calculated per benchmark), USMLE-MM (+13.1 pp), MMMU-HM (+2.6 pp), and the reproduced GPT-4V results on PAD-UFES-20 (+35.9 pp from 50.0% to 85.9%), Slake-VQA (+46.5 pp from 41.0% to 87.5%), and Path-VQA (+28.7 pp from 36.0% to 64.7%). The large relative gains on specialist medical imaging benchmarks reflect the substantial benefit of medical fine-tuning over GPT-4V's generalist visual capabilities.

Long-Context Processing Benchmarks (Table 3)

Long EHR "Needle-in-a-Haystack" retrieval. Med-Gemini-M 1.5 achieves an F1 of 0.77 on the MIMIC-III Needle-in-a-Haystack task with one-shot in-context learning (Table 3, Table E2 for confidence intervals). This is statistically indistinguishable from the heuristic annotation-aggregation baseline (Feder et al., 2022) which achieves an F1 of 0.78 — a carefully engineered system requiring manual feature engineering, rule-based selection, and rule-based aggregation, with ontology-dependent rules that must be extended for each new medical condition. The performance breakdown (Table E2) shows precision of 0.80 and recall of 0.75 for Med-Gemini-M 1.5, compared to precision of 0.81 and recall of 0.75 for the baseline. GPT-4 could not be benchmarked on this task because the average context token length significantly exceeds the maximum context window supported by public GPT-4 APIs.

Medical Video Question Answering (MedVidQA). Med-Gemini-M 1.5 achieves state-of-the-art on both MedVidQA variants using zero-shot prompting:

With subtitles: mIoU of 65.8, surpassing the prior SoTA from Weng and Li (2023) at 58.3 by 7.5 points (Table 3). IoU at threshold 0.3: 71.4; at 0.5: 68.4; at 0.7: 57.3. This substantial improvement over a trained specialist system suggests that the subtitle text provides strong linguistic cues that the model's language understanding capabilities can exploit, even without video-specific training.
Without subtitles (video only): mIoU of 43.4, surpassing the prior SoTA from Li et al. (2022) at 27.5 by 15.9 points (Table 3). IoU at threshold 0.3: 51.9; at 0.5: 45.3; at 0.7: 26.2. The large gap between the with-subtitles and without-subtitles performance (22.4 mIoU points) indicates that the embedded captions significantly aid the model's video understanding. Even without subtitles, however, the model substantially outperforms prior specialist systems, suggesting meaningful visual-only understanding.

Critical View of Safety (CVS) assessment. Med-Gemini-M 1.5 achieves 50.0% accuracy on predicting which CVS criteria are met in laparoscopic cholecystectomy video clips (Table 3). This outperforms GPT-4V (which achieves 29.0% accuracy when evaluated on the subset of clips ≤5 minutes, using frames sampled at 1 fps due to context length limitations) by 21 percentage points. However, the supervised ResNet3D baseline achieves 67.0% accuracy — a 17-percentage-point gap below the specialist model. This is the only long-context benchmark where Med-Gemini does not achieve state-of-the-art, and the paper explicitly acknowledges that "further investigations on prompting strategies or instruction fine-tuning may be required to improve the task performance."

Long-Form Medical Text Generation (Figure 5, Table C5)

Medical summarization (after-visit summaries). In side-by-side blinded comparisons by clinician raters, Med-Gemini-M 1.0's generated after-visit summaries are rated as good or better than expert human-written summaries in 67.7% of cases (p < 0.001 for the null hypothesis that the preference/tie rate equals 0.5; Table C5). On a 5-point Likert scale comparing overall quality, the model is preferred in 22.6% of cases, equivalent in 45.2%, and human experts are preferred in 32.3%.

Referral letter generation. Med-Gemini-M 1.0's generated referral letters are rated as good or better than expert human-written letters in 100% of cases evaluated (Figure 5, Table C5). Across 25 examples evaluated by three board-certified physicians each (75 total evaluations), the model is preferred in 58.7% of ratings, equivalent in 41.3%, and human experts are preferred in 0% of ratings. Aggregated by mapping Likert scales to [-2, 2] and taking the sign of the median value per example, the model is preferred in 20 out of 25 examples, tied in 5, with human experts preferred in none.

Medical simplification (plain language summaries). Med-Gemini-M 1.0's simplified summaries of Cochrane systematic review abstracts are rated as good or better than the original plain language summaries in 82.7% of cases (p < 0.001; Table C5). The model is preferred in 42.7% of ratings, equivalent in 40.0%, and human-written summaries are preferred in 17.3%.

Ablation Studies and Robustness Checks

Self-training contribution to MedQA accuracy (Figure 4a): Removing self-training (using the base Gemini 1.0 Ultra without fine-tuning on MedQA-R and MedQA-RS) reduces MedQA accuracy from approximately 87.2% to approximately 84.3% — a drop of 2.9 percentage points. This establishes that the self-training data generation process, which produces synthetic reasoning chains (CoTs) with and without web search, provides meaningful improvement beyond the base model's capabilities. The paper does not ablate the individual components of the self-training mixture (MedQA-R vs. MedQA-RS vs. long-form QA vs. summarization data), so the contribution of each dataset component cannot be assessed.

Number of uncertainty-guided search iterations on MedQA (Figure 4a): Accuracy improves with each additional search iteration: 87.2% (0 iterations, i.e., self-trained model without search), 88.8% (1 iteration), 90.0% (2 iterations), 90.5% (3 iterations), 91.1% (4 iterations). The marginal gains diminish with each iteration: +1.6 pp from iteration 0→1, +1.2 pp from 1→2, +0.5 pp from 2→3, +0.6 pp from 3→4. The diminishing returns are consistent with each iteration resolving progressively smaller fractions of residual uncertainty. The paper does not explore whether additional iterations beyond 4 would provide further gains or plateau.

Search contribution to NEJM CPC (Figure 3a): Adding search at inference improves Med-Gemini-L 1.0's top-10 accuracy from 68.3% to 72.3% (+4.0 percentage points). Top-1 accuracy also improves, though the exact value without search is not separately reported. The paper notes that only one iteration of uncertainty-guided search is used for NEJM CPC, and does not ablate whether additional iterations would help.

Model abstention on GeneTuring (Figure 3b): Med-Gemini-L 1.0's abstention rate varies substantially across GeneTuring modules. Cases where the model abstains are excluded from numerical evaluation, so the reported accuracy on each module is conditional on the model attempting to answer. The paper does not report accuracy including abstentions as incorrect, which would provide a more conservative (and potentially lower) performance estimate. The comparison with prior models (GPT-3.5, GPT-4) is somewhat confounded by different abstention behaviors across models — a model that abstains more aggressively on uncertain questions may achieve higher conditional accuracy by answering only the questions it is confident about.

Impact of MedQA label filtering on accuracy (Figure 4b, Figure C1): Using unanimous voting across bootstrap-sampled committees to filter questions with missing information or label errors: accuracy improves from 91.1% to 91.8% ± 0.2% while retaining approximately 93.3% of questions. Using majority voting: accuracy improves to 92.9% ± 0.38% by discarding up to 20.9% of questions. Filtering ambiguous questions (those allowing multiple correct answers) further reduces the test set size but does not significantly change accuracy. The key non-obvious finding is that label errors have a bigger impact on accuracy improvement than missing information or ambiguity — the model gets these questions wrong because the "correct" answer is disputed by experts, not because of a reasoning failure.

Patient-level vs. image-level split for PAD-UFES-20 (Table 2, Section 5): Moving from the Med-PaLM M image-level split to a new patient-level split causes a 7.1% absolute accuracy drop (from 85.9% to approximately 78.8%). This is a robustness check revealing that the image-level split overestimates generalization performance — when images of the same patient's lesion appear in both training and test sets (different images, same patient), the model can exploit patient-specific visual features rather than learning to recognize the disease category across patients. This is a significant caution for benchmark interpretation in dermatology AI.

ECG encoder: frozen vs. unfrozen Gemini backbone (Section D.3): Med-Gemini-S 1.0 achieves 57.7% accuracy with frozen Gemini 1.0 Nano parameters (only the ECG encoder and cross-attention layers trained) and 58.4% with unfrozen parameters (full model fine-tuned). The small 0.7 percentage point gap suggests that the custom encoder architecture captures most of the task-relevant signal, and fine-tuning the language model backbone is not critical for this modality. This is an important finding for deployment efficiency: training only the encoder and cross-attention layers is substantially cheaper than full model fine-tuning, yet achieves nearly equivalent performance.

Video understanding: with vs. without subtitles (Table 3): Med-Gemini-M 1.5's MedVidQA performance drops from mIoU of 65.8 (with subtitles) to 43.4 (without subtitles) — a 22.4 point gap. This large difference indicates that the textual information in captions provides strong cues that the model leverages heavily. The zero-shot performance without subtitles (43.4 mIoU) still substantially outperforms the prior trained specialist system (27.5 mIoU), suggesting genuine visual understanding, but the model's video comprehension is clearly augmented by linguistic information when available.

GPT-4V context window limitation for video tasks (Section 4.3, Table 3): The paper acknowledges that a fair comparison with GPT-4V on the CVS task was limited by GPT-4V's context window constraints — the model could only process up to 300 low-resolution frames, forcing filtering of video clips longer than 5 minutes. This makes the Med-Gemini vs. GPT-4V comparison on CVS (50.0% vs. 29.0%) fundamentally unfair in Med-Gemini's favor, since Med-Gemini processes the full video while GPT-4V processes only sampled frames from a restricted subset. The paper does not evaluate Med-Gemini on the same restricted subset of frames to enable a like-for-like comparison.

Critical Assessment

The experimental results provide broad support for the paper's central claim — that Gemini-derived models specialized for medicine achieve state-of-the-art performance across text, multimodal, and long-context benchmarks — but the nature and strength of this support varies considerably by task category, and several important claims are supported more weakly than the headline numbers suggest.

The 91.1% MedQA claim is well-supported, but its interpretation as a clinically meaningful advance is deliberately problematized by the authors themselves. The experiments demonstrate that Med-Gemini-L 1.0 achieves higher accuracy than prior systems (Figure 4a, Table 1). However, the relabeling analysis (Figure 4b) shows that approximately 7.4% of the test set has data quality issues, and Med-Gemini-L 1.0's errors are concentrated on these problematic questions. At 91.1% accuracy, the model makes errors on roughly 113 questions. If, say, 30–40 of those errors are on questions that clinicians themselves consider flawed (missing information, label errors, or ambiguously answerable), then the "true" accuracy on well-formed questions may be closer to 92–93%, and the remaining headroom for improvement on this benchmark — the signal about what better clinical reasoning would look like — is smaller than the raw 8.9% error rate implies. The paper's own conclusion that "further improvements in SoTA performance on the MedQA (USMLE) benchmark in isolation may not directly correlate to progress in the capabilities of medical LLMs for meaningful real-world tasks" (Section 5) is well-supported by the relabeling data. However, the paper does not perform a similar relabeling of the other benchmarks (NEJM CPC, GeneTuring, multimodal VQA datasets), leaving open the question of whether similar data quality issues exist elsewhere.

The uncertainty-guided search ablation is informative but incomplete. Figure 4a convincingly shows that self-training and each iteration of search contribute to performance. However, the paper does not include an ablation comparing uncertainty-guided search (searching only when entropy exceeds a threshold) with always-on search (performing web search for every question, regardless of uncertainty). Always-on search would likely achieve even higher accuracy at the cost of additional latency and compute — and without this comparison, we cannot quantify the accuracy cost of the entropy threshold. Similarly, the paper does not ablate the uncertainty-guided query generation against a baseline that generates search queries from the question alone (without conditioning on conflicting reasoning paths). The stated motivation — that conditioning on conflicts produces more targeted queries — is intuitive but empirically unverified within this paper.

The multimodal results establish strong zero-shot capabilities but the fine-tuning gains are modest on some benchmarks. Med-Gemini-L 1.0's zero-shot performance on NEJM Image Challenge (69.7%), USMLE-MM (93.5%), and MMMU-HM (67.3%) convincingly demonstrates that Gemini's pretraining covers medical imaging modalities well enough to surpass GPT-4V without any medical fine-tuning. This is a strong result. However, on the fine-tuned benchmarks where comparison with the same architecture's zero-shot performance would reveal the marginal benefit of fine-tuning, this comparison is absent: we do not know how Med-Gemini-L 1.0 (zero-shot) would perform on Path-VQA or Slake-VQA, so we cannot quantify how much the 64.7% token F1 on Path-VQA represents an improvement over the zero-shot baseline. The fact that Med-Gemini-M 1.5 (fine-tuned) slightly underperforms Med-PaLM M on Slake-VQA (87.5% vs. 89.3%) and PAD-UFES-20 (85.9% vs. 88.0%) suggests that fine-tuning on the MultiMedBench mixture does not universally improve performance — it may trade off performance on some tasks to improve others.

The referral letter generation result (100% preferred or tied with experts) is striking but should be interpreted with extreme caution. This is a small-scale evaluation (25 examples) on a dataset curated from a single de-identified EHR system, with letters written by clinicians following internal guidelines and evaluated by three physicians from the same pool. The finding that human experts were preferred in 0% of ratings raises the possibility that the evaluation design — the specific instructions given to raters, the criteria they were asked to use, the way the letters were presented — may have favored the model's outputs for reasons unrelated to clinical quality. For example, the model may produce more structured, more detailed, or more fluent letters without necessarily being more accurate in their clinical content. The paper does not provide granular evaluation axes for this task (unlike for summarization, where Table C5 breaks down axes like "completeness" and "accuracy"), making it impossible to assess whether the model's advantage lies in clinical substance or stylistic factors.

The long-context EHR result represents a genuine capability demonstration, but the evaluation has limitations. Achieving F1 of 0.77 — statistically matching a carefully engineered heuristic system — using one-shot in-context learning is a strong demonstration that long-context language understanding can substitute for task-specific engineering. However, the dataset of 200 examples from 44 unique patients has several constraints: (1) the ground truth is majority vote of three raters, which may not perfectly reflect clinical reality (Krippendorff's alpha of 0.77 is "good" but not "excellent"); (2) the examples are selected from the same hospital system (MIMIC-III) with consistent note formats and terminology, so the results may not generalize to other EHR systems; (3) only single-condition queries are tested — real clinical workflows require identifying all conditions present in a record, which is a substantially harder multi-label problem. The fact that GPT-4 could not be evaluated on this task due to context window limitations highlights a genuine capability advantage of Gemini 1.5 Pro, but also means we cannot assess whether this advantage is due to architectural innovation or simply longer context.

The video understanding results are impressive for zero-shot but reveal clear capability boundaries. The large gap between MedVidQA with subtitles (65.8 mIoU) and without (43.4 mIoU) shows that the model relies heavily on linguistic information. More importantly, the 17-point gap between Med-Gemini (50.0%) and a supervised ResNet3D (67.0%) on CVS assessment indicates that zero-shot prompting is insufficient for tasks requiring specialized visual discrimination. The paper acknowledges this limitation, but it challenges the implicit claim that Gemini's long-context video understanding is broadly applicable — it works well when the task can be solved through language-anchored visual search (finding "the segment where the physical therapist demonstrates the calf stretch exercise") but poorly when the task requires recognizing subtle surgical anatomy features that were likely underrepresented in pretraining.

Missing experiments that would strengthen the paper:

Ablation of search query generation strategy. The paper strongly motivates conditioning search queries on conflicting reasoning paths (rather than on the question alone), but does not empirically compare these approaches. This is a central methodological claim that remains untested.
Cross-dataset generalization of the revision/long-context approaches. All long-context evaluations use MIMIC-III, Cholec80, and MedVidQA — each a single dataset from a specific institution or source. Testing on EHR data from a different hospital system or surgical videos from a different procedure type would reveal whether the in-context learning approach generalizes or overfits to the specific note formats and visual features of the evaluation datasets.
Latency and cost analysis. The paper never quantifies the wall-clock time or computational cost of its inference strategies. For the uncertainty-guided search in particular, four iterations of generating multiple reasoning paths, computing entropy, generating search queries, performing web search, and re-generating responses likely incur substantial latency that may be prohibitive for real-time clinical applications. Without this analysis, "91.1% accuracy" is an incomplete specification of the system's practical utility.
Direct comparison between Med-Gemini-M 1.5 (fine-tuned) and Med-Gemini-L 1.0 (zero-shot) on multimodal benchmarks. Without knowing how much fine-tuning improves over the zero-shot baseline, the value of the multimodal fine-tuning procedure — a significant engineering effort spanning eight tasks — cannot be assessed relative to simply using the larger zero-shot model.
Evaluation of MIMIC-CXR fine-tuning impact. The paper uses MIMIC-CXR as training data (nearly 500,000 examples across four sub-tasks) but defers evaluation of its impact to a subsequent paper. This is a large fraction of the fine-tuning mixture whose effect is completely unmeasured in the current study.

The paper's claims, assessed against the evidence:

"Med-Gemini establishes new SoTA on 10 of 14 benchmarks": Supported by Tables 1, 2, 3, and A1. The 10 benchmarks where SoTA is claimed are MedQA, NEJM CPC, GeneTuring (7 of 12 modules, counted as one benchmark), NEJM Image Challenge, USMLE-MM, MMMU-HM, ECG-QA, Path-VQA, MedVidQA (two variants), and Long EHR. However, "SoTA on MedVidQA" requires noting that the prior SoTA was a trained specialist system, not another generalist model. "SoTA on Long EHR" requires noting that F1 of 0.77 is statistically tied with the baseline of 0.78 (not strictly higher).
"Surpass GPT-4 model family on every benchmark where direct comparison is viable, often by wide margins": Supported for text benchmarks (MedQA: 91.1% vs. 90.2% MedPrompt/GPT-4) and multimodal benchmarks (average 44.5% relative improvement over GPT-4V across five benchmarks). However, "every benchmark where direct comparison is viable" excludes the three long-context benchmarks where GPT-4 could not be evaluated due to context window limitations — which means the claim is true but narrower than it appears, since those benchmarks represent some of the most challenging and practically important tasks.
"Real-world utility suggested by surpassing human experts on medical text summarization and referral letter generation": The referral letter result (100% preferred/tied) is striking but based on 25 examples from a single dataset evaluated by three raters — an evaluation too small and narrow to support claims of "real-world utility." The summarization result (67.7% good or better) is more measured but similarly limited in scale and diversity. These results are better characterized as promising pilot evaluations that warrant larger-scale study rather than demonstrations of real-world utility.
"7.4% of MedQA questions are unfit for evaluation": Strongly supported by the rigorous relabeling study with 18 physicians and bootstrapped committee analysis (Figure 4b, Figure C1). This is the most methodologically robust contribution in the paper and has important implications for the field's interpretation of this benchmark going forward.

6. Limitations and Trade-offs

The Cost of Difficulty-Aware Allocation: Benchmark-Ceiling Uncertainty Is Not Resolved

The paper's uncertainty-guided search strategy represents a conceptual advance in when to retrieve information, but it does not address a deeper uncertainty that the MedQA relabeling analysis itself exposes: some proportion of "errors" may not be resolvable by any amount of search, because the benchmark questions themselves are flawed. The model's Shannon entropy over answer choices can detect that it is uncertain, but it cannot distinguish between uncertainty arising from a genuine knowledge gap (which web search might resolve) and uncertainty arising from an ambiguous, incomplete, or mislabeled question (which no amount of retrieval can fix). The paper shows that Med-Gemini-L 1.0's entropy-based uncertainty score is significantly higher on questions with missing information or label errors (p = 0.033; Section 4.1). This means the model correctly recognizes that these questions are problematic — but the search strategy will still attempt to resolve this uncertainty through retrieval, consuming compute and latency for questions that may be fundamentally unanswerable.

The consequence is a ceiling effect that the uncertainty-guided framework cannot overcome through additional search iterations. If approximately 7.4% of MedQA questions have data quality issues, and the model's errors are disproportionately concentrated on these questions, then further improvements in search query quality, retrieval corpus breadth, or iteration count cannot push accuracy meaningfully beyond ~92–93% on this benchmark — not because of any reasoning limitation, but because the ground truth labels are unreliable for the remaining questions. The paper acknowledges this implicitly by noting that filtering label errors has the biggest impact on measured accuracy (Section 4.1), but does not discuss how the uncertainty-guided search strategy should behave when the source of uncertainty is benchmark quality rather than model knowledge. In a real deployment, there is no ground-truth oracle to tell the system, "stop searching — this question is ill-posed." The search strategy has no built-in termination condition for irresolvable uncertainty, and will continue iterating until its budget is exhausted.

What evidence exists: The MedQA relabeling study (Figure 4b, Figure C1) provides direct evidence of the data quality ceiling. The finding that the model's uncertainty is elevated on problematic questions (p = 0.033) is reported in Section 4.1. However, the paper does not measure how many of the model's remaining errors after four search iterations fall into the "missing information" or "label error" categories versus genuine reasoning failures — this breakdown would quantify the fraction of residual error that no search strategy could fix. The paper also does not evaluate whether the model's search queries generated for problematic questions are coherent or productive (e.g., does the model hallucinate non-existent lab values to search for when the question references missing lab results?). The authors acknowledge the broader implication — "further improvements in SoTA performance on the MedQA (USMLE) benchmark in isolation may not directly correlate to progress in the capabilities of medical LLMs" (Section 5) — but this observation is about benchmark interpretation, not about a design limitation of the search strategy itself.

Mitigation status: Not addressed in the current architecture. The paper does not propose a mechanism for the model to recognize that a question is ill-posed and abstain or flag it, rather than iteratively searching for information that does not exist. Future work directions (Section 5) mention "rigorous evaluation beyond benchmarks" as important, but do not discuss architectural changes to handle benchmark-quality-induced uncertainty at inference time.

The Zero-Shot / Fine-Tuned Comparison Gap: We Cannot Measure the Marginal Value of Multimodal Fine-Tuning

The paper evaluates multimodal capabilities using two different models under two different conditions: Med-Gemini-L 1.0 (largest model, zero-shot, no multimodal fine-tuning) on out-of-distribution benchmarks, and Med-Gemini-M 1.5 (mid-size model, fine-tuned on eight multimodal tasks) on in-distribution benchmarks. This design establishes that Gemini has strong zero-shot medical multimodal capabilities and that fine-tuning produces models competitive with or surpassing prior SoTA on specialist tasks — but it prevents us from measuring how much the fine-tuning itself contributes to performance on the in-distribution benchmarks. We do not know how Med-Gemini-M 1.5 would perform on Path-VQA or Slake-VQA without the fine-tuning, because the zero-shot evaluation uses a different, larger base model (Gemini 1.0 Ultra vs. Gemini 1.5 Pro). We also do not know how Med-Gemini-L 1.0 (zero-shot) would perform on these in-distribution benchmarks.

The consequence is that the case for multimodal medical fine-tuning — which requires curating datasets, designing task prompts, managing training mixtures across eight tasks, and engineering auxiliary augmentation strategies — rests on an implicit assumption that fine-tuning improves over the zero-shot baseline, but that improvement is never isolated or quantified. If Gemini 1.5 Pro's zero-shot performance on Path-VQA were, say, 62% token F1 (close to the 64.7% achieved by the fine-tuned model), the marginal gain from fine-tuning would be small — raising the question of whether the engineering effort is justified versus simply using a larger zero-shot model. Conversely, if zero-shot performance were much lower (e.g., 40%), the fine-tuning gain would be substantial, strengthening the case for specialization. The paper also reports that Med-Gemini-M 1.5 slightly underperforms Med-PaLM M on Slake-VQA (87.5% vs. 89.3%) and PAD-UFES-20 (85.9% vs. 88.0%), which could indicate either that the fine-tuning mixture trades off performance across tasks, or that Med-PaLM M's fine-tuning strategy was better optimized for these specific benchmarks, or that Gemini 1.5 Pro's zero-shot performance was already close to Med-PaLM M and fine-tuning provided limited headroom. Without the zero-shot baseline, these hypotheses cannot be distinguished.

What evidence exists: The paper provides strong evidence for zero-shot multimodal capabilities — Med-Gemini-L 1.0 achieves SoTA on NEJM Image Challenge (69.7%), USMLE-MM (93.5%), and MMMU-HM (67.3%) without any fine-tuning (Table 2). The fine-tuned Med-Gemini-M 1.5 achieves strong results on Path-VQA (64.7% token F1) and competitive results on Slake-VQA (87.5%) and PAD-UFES-20 (85.9%). But these are disjoint evaluations: different benchmarks evaluated with different models. The paper does not report cross-model evaluation (e.g., Med-Gemini-L 1.0 on Path-VQA, or Med-Gemini-M 1.5 zero-shot on Slake-VQA), nor does it provide an ablation of the fine-tuning mixture (e.g., performance when MIMIC-CXR tasks are excluded, or when only a subset of the eight tasks is used). The MIMIC-CXR fine-tuning tasks — constituting nearly 500,000 training examples, the largest component of the fine-tuning mixture — have their evaluation entirely deferred to future work (Appendix D.1: "The evaluation of MIMIC-CXR will be reported in a subsequent paper"). This means a substantial fraction of the training compute is spent on tasks whose impact on model capabilities is completely unmeasured in the current study.

Mitigation status: Not addressed. The paper acknowledges the model family design as a deliberate choice — "real world considerations present trade-offs and requirements for task-specific optimizations which are at odds with each other" (Section 1) — but does not address the evaluation gap this design creates. The absence of cross-model, cross-condition evaluation is a structural feature of the experimental design, not an oversight that can be corrected by additional analysis of existing data. Future work would need to run the zero-shot model on the fine-tuned benchmarks and vice versa to fill this gap.

The 100% Referral Letter Result Raises Evaluation Validity Concerns That Are Not Interrogated

The paper reports that Med-Gemini-M 1.0's generated referral letters are rated as good or better than expert human-written letters in 100% of cases evaluated (Figure 5, Table C5), with human experts preferred in 0% of 75 total ratings. This is an extraordinary result — it claims not just superiority but complete dominance. Extraordinary results require extraordinary evidence about evaluation validity, and the paper provides insufficient information to assess whether the evaluation design, rather than model capability, produced this outcome. The specific concerns are:

Sample size and diversity: 25 examples drawn from a single de-identified EHR system. The clinicians who wrote the "expert" letters and the clinicians who evaluated them may share institutional norms, documentation styles, or expectations about what a referral letter should contain — making the evaluation potentially an in-distribution test of institutional formatting conventions rather than a test of clinical communication quality.

Evaluation criteria transparency: Unlike the medical summarization evaluation, where Table C5 provides breakdowns across axes including overall quality, completeness, accuracy, and readability, the referral letter evaluation provides only a single aggregate Likert scale. We do not know whether raters were instructed to evaluate clinical accuracy, appropriate level of detail, structural organization, adherence to referral guidelines, or some combination. If raters implicitly valued thoroughness and detail, a model that generates longer, more comprehensive letters could dominate even if it includes some clinically irrelevant information — the evaluation design cannot distinguish "better clinical communication" from "longer and more detailed."

Blinding concerns not addressed: The paper states raters were "blinded to the source of each referral letter" (Appendix C.4), but does not describe how blinding was implemented or whether it was tested. Model-generated text can have distinctive stylistic fingerprints (consistent formatting, absence of typos, specific phrase patterns) that may have made the source identifiable to experienced clinicians, potentially biasing ratings if raters held conscious or unconscious expectations about AI vs. human output quality.

The absence of negative ratings for the model is itself a signal that warrants investigation. In most human evaluation studies, even strong models receive some negative ratings due to rater disagreement, attention lapses, or genuine output flaws. A 0% human-preference rate across 75 independent ratings (3 raters × 25 examples) is an outlier that should trigger scrutiny of the evaluation protocol, not simply be reported as a success.

The consequence of these validity concerns is that the referral letter result cannot be used to support claims of real-world utility — a key component of the paper's narrative (Contribution 5: "Real-world utility of Med-Gemini," Section 1). The paper explicitly claims that "beyond performance on popular medical benchmarks, we preview the potential real-world utility of Med-Gemini through quantitative evaluations on tasks such as medical note summarization, clinical referral letter generation, and EHR question answering" (Section 1). If the strongest quantitative result supporting this claim (100% preference) is an artifact of evaluation design, the case for real-world utility rests entirely on the summarization result (67.7% good or better, a more plausible and measured claim) and qualitative demonstrations.

What evidence exists: The paper reports the aggregate preference rates (Figure 5) and Likert breakdowns (Table C5). For referral letters specifically, 58.7% of ratings prefer the model, 41.3% rate them as equivalent, and 0% prefer human experts. The p-value is reported as significant for the null hypothesis that the preference/tie rate equals 0.5. However, the paper provides no information about rater instructions for this specific task, no inter-rater agreement metrics, no analysis of whether the three raters showed consistent preference patterns, and no examples of model-generated vs. human-written letters that would allow readers to qualitatively assess the basis for the ratings.

Mitigation status: Not addressed. The paper does not acknowledge any validity concerns with the referral letter evaluation, does not discuss potential sources of bias in the evaluation design, and does not temper its interpretation of the result. This stands in contrast to the paper's careful treatment of MedQA benchmark limitations and its cautious framing of the multimodal dialogue demonstrations as "the art of the possible" requiring "considerable further research and development."

Long-Context Capability Is Evaluated on Single-Task, Single-Dataset Scenarios That Do Not Reflect Clinical Workflow Complexity

The long-context evaluations demonstrate impressive capabilities — retrieving single-condition mentions from 700K-word EHRs, localizing answers in instructional videos, assessing surgical safety criteria — but each evaluation isolates a single capability on a single carefully-constructed task. Real clinical long-context scenarios require integrated multi-capability reasoning that is not tested here. A clinician interacting with a patient's EHR is not just looking for whether a single condition is present; they are simultaneously tracking multiple active problems, reconciling medications across encounters, identifying temporal trends in lab values, and synthesizing this information into a coherent clinical picture. The MIMIC-III Needle-in-a-Haystack task tests whether the model can find one condition mentioned once — a retrieval task, not a synthesis task.

The consequence is that the long-context results may overstate readiness for clinical deployment by demonstrating competence on simplified proxies rather than integrated clinical reasoning. Specific capability gaps that are not tested include:

Multi-condition reasoning across notes: Can the model identify all active conditions across a full EHR and distinguish them from resolved historical conditions, family history mentions, and differential diagnoses? The needle-in-a-haystack task tests exactly one condition at a time, with ground truth labeling performed by human raters for that specific condition. Scaling this to a comprehensive problem list extraction — the actual clinical workflow — would require the model to generate and verify dozens of condition hypotheses simultaneously.
Temporal reasoning across encounters: Can the model track how a condition evolves across multiple encounters? For example, distinguishing "pneumonia that resolved after antibiotics in 2019" from "recurrent pneumonia in 2023" from "current pneumonia on admission" — all of which may be mentioned across different notes. The needle-in-a-haystack task collapses all mentions into a binary present/absent label, losing temporal structure.
Cross-document contradiction resolution: EHRs often contain conflicting information — one note may list a condition as active while a subsequent note lists it as resolved. Can the model identify and reconcile these contradictions? The current task design selects conditions with exactly one mention to avoid this complexity, but real records are messy.
Integration with structured data: The EHR task excludes structured data (labs, vitals, medications), using only unstructured clinical notes. Real clinical reasoning requires integrating numeric lab trends with textual descriptions — e.g., recognizing that a note mentioning "acute kidney injury" is supported by a rising creatinine trend in structured lab data.

For video understanding, similar gaps exist. The CVS assessment task is a single classification with defined criteria; a real surgical assistance system would need to simultaneously track instrument usage, recognize phase transitions, monitor for complications, and integrate preoperative imaging with intraoperative findings.

What evidence exists: The needle-in-a-haystack task is explicitly described as curated to select conditions with "only one evidence snippet identified in the aggregation step" (Section 2.3), with single-condition queries. The EHR records range from 200K to 700K words. The video tasks similarly isolate specific capabilities (answer localization, CVS criteria scoring) rather than testing integrated understanding. The paper does not claim that these tasks represent full clinical workflows — and indeed frames them as demonstrations of "the potential to significantly reduce cognitive load and augment clinicians' capabilities" (Section 5) — but the gap between demonstrated capability and clinical utility is larger than the text implies.

Mitigation status: Partially acknowledged. The paper notes that "significant further work would need to be performed to assess more nuanced and complex capabilities" (Section 4.3.1) for surgical video dialogue, and that "such uses (particularly in safety-critical areas like diagnosis) would require considerable further research and development" (Section 1). However, these caveats are general safety disclaimers rather than specific analyses of the capability gaps between the evaluated tasks and clinical reality. The paper proposes no framework for evaluating more integrated long-context clinical reasoning, nor does it identify which specific additional capabilities would need to be demonstrated before real-world deployment.

Inference-Time Cost and Latency Are Not Quantified Despite Being Central to the Approach's Practical Utility

The paper's headline results — 91.1% MedQA accuracy via four iterations of uncertainty-guided search, state-of-the-art multimodal performance via fine-tuning, long-context EHR retrieval via chain-of-reasoning — are reported without any quantification of the computational cost, wall-clock latency, or monetary expense required to achieve them. This is a critical omission for a paper whose central contribution is an inference-time strategy (uncertainty-guided search) and whose model family is explicitly motivated by deployment trade-offs ("considering factors such as training data, compute availability, and inference latency," Section 1).

The unquantified costs include:

Uncertainty-guided search on MedQA: Each iteration involves (1) generating multiple reasoning paths (the paper does not specify how many, but computing entropy requires enough paths to estimate a stable answer distribution — likely at least 5–10), (2) computing entropy and thresholding, (3) conditionally generating three search queries, (4) executing web search API calls and retrieving results, and (5) re-generating reasoning paths with search-augmented context. With up to four iterations, the total generation cost could be 20–40× that of a single greedy answer, plus search API latency. The paper does not report the average number of search iterations actually triggered (how many questions had entropy below threshold at each iteration and thus terminated early?), the average latency per MedQA question, the token cost of generated reasoning paths, or the API cost of web search queries. The progressive gains from each search iteration diminish (Figure 4a: +1.6, +1.2, +0.5, +0.6 percentage points), suggesting that later iterations provide marginal accuracy improvements at undiminished cost — the cost-benefit ratio worsens with each iteration, but this trade-off is not analyzed.

Long-context EHR processing: The chain-of-reasoning requires two sequential generation passes over the full context window of up to 700,000 words. The paper does not report the latency, memory footprint, or per-query cost of processing a near-million-token context through Gemini 1.5 Pro, nor whether the two-step approach could be collapsed into a single pass (and at what accuracy cost).

Multimodal fine-tuning: The paper fine-tunes on nearly 500,000 MIMIC-CXR examples plus five other datasets, but does not report training compute, GPU-hours, convergence behavior, or the number of model variants trained before selecting the reported configurations.

The consequence is that practitioners cannot assess whether the reported performance gains justify the computational cost for their deployment context. The paper explicitly motivates the model family as enabling deployment flexibility — "each optimized for different capabilities and application-specific scenarios, considering factors such as training data, compute availability, and inference latency" (Section 1) — but provides no data to support these deployment decisions. Would a real-time clinical decision support system tolerate the latency of four search iterations? Could Med-Gemini-M 1.5 process a 500K-word EHR quickly enough for an interactive clinician-facing tool? Is the accuracy gain from Med-Gemini-L 1.0 over Med-Gemini-M 1.0 worth the additional inference cost for referral letter generation? These are the questions the model family design was supposed to answer, but the paper provides no cost data to answer them.

What evidence exists: Section 2.1 specifies the uncertainty-guided search algorithm and states that up to four iterations are used for MedQA, one iteration for NEJM CPC and GeneTuring. Section 2.3 describes the two-step chain-of-reasoning for EHR understanding. The prompt structure and retrieval mechanism are described in detail. However, no latency, token count, memory usage, or cost data is reported anywhere in the paper or appendices. The paper also does not report the number of reasoning paths generated per iteration, the sampling temperature, or the entropy threshold value — parameters that directly affect both cost and accuracy. The ablation in Figure 4a shows accuracy per search iteration, but without cost data, this cannot be converted into a cost-accuracy trade-off curve.

Mitigation status: Not addressed. The paper does not acknowledge the absence of cost analysis as a limitation. The Discussion (Section 5) addresses data quality, benchmark limits, responsible AI, and future work directions (multimodal search retrieval, smaller model search integration), but does not mention the need for efficiency or cost analysis. This is a significant gap given that the model family approach is presented as a solution to deployment trade-offs — the paper argues for specialization based on deployment constraints but never quantifies those constraints.

Generalization Beyond the Evaluated Benchmarks, Models, and Medical Systems Is Completely Uncharacterized

All of the paper's quantitative results — across 14 benchmarks, three capability dimensions, and four model variants — are obtained using a single base model family (Gemini) evaluated on publicly available datasets predominantly from U.S. medical contexts (MedQA, MIMIC-III, NEJM, MedVidQA). The paper provides no evidence about how Med-Gemini's capabilities would transfer to different foundation model architectures, different medical systems (e.g., non-U.S. clinical guidelines, non-English medical records), or different deployment populations. This is not a minor external validity concern — it is a fundamental limitation for a paper that presents its contributions as demonstrating "the promise of Med-Gemini in many areas of medicine" (Abstract) and "compelling evidence for the promise of Gemini models in medicine" (Conclusion).

The specific generalization gaps include:

Foundation model dependence: All reported capabilities are built on Gemini. The uncertainty-guided search strategy, the self-training with search data generation pipeline, the multimodal fine-tuning procedure, and the long-context chain-of-reasoning are described in general terms, but their effectiveness may depend on Gemini-specific architectural features (mixture-of-experts for efficient long-context processing, native multimodal tokenization, specific pretraining data distributions). The paper does not replicate any results using alternative base models (e.g., GPT-4, PaLM 2, Claude), making it impossible to distinguish "contributions of the Med-Gemini training and inference strategies" from "contributions of Gemini's base capabilities." If another research group attempted to reproduce the uncertainty-guided search strategy using a different base model, they might obtain substantially different results — not because the strategy is ineffective, but because it depends on the base model's ability to generate coherent conflicting reasoning paths, produce targeted search queries, and integrate search results, capabilities that vary across model families.

Geographic and demographic scope: All text benchmarks are in English and based on U.S. medical standards (USMLE, NEJM, MIMIC-III from a Boston hospital). Clinical reasoning patterns, treatment guidelines, disease prevalence, and documentation practices vary substantially across countries and healthcare systems. The long EHR task uses MIMIC-III, which represents ICU patients at a single U.S. academic medical center — its note formats, clinical terminology, and patient demographics may not reflect EHR systems in community hospitals, other countries, or non-ICU settings. The paper does not evaluate on any non-English medical benchmarks, any non-U.S. clinical datasets, or any benchmark designed to test cross-population generalization.

Benchmark scope relative to medical AI landscape: The paper evaluates on 14 benchmarks, which the authors note is "the most comprehensive evaluation of medical LLMs and LMMs" (Section 5) to date. However, the authors also acknowledge that "over 350 medical benchmarks are available in the community" (Section 5, citing Meta, 2024). The 14 benchmarks are concentrated in medical question answering, visual diagnosis, and information retrieval — they do not cover important medical AI capabilities such as treatment planning, drug interaction prediction, patient outcome forecasting, clinical trial matching, or population health analytics. The strong results on the evaluated benchmarks may not predict performance on these unmeasured dimensions.

What evidence exists: All reported results (Tables 1, 2, 3, A1, C4, C5, E2) are on the specific benchmarks, datasets, and model configurations described in Sections 3 and 4 and the Appendices. The paper reports one cross-model comparison: Med-Gemini-M 1.5 on Path-VQA (64.7% token F1) vs. Med-PaLM M (62.7%), showing a 2.0 percentage point improvement. This is the only datapoint comparing Med-Gemini to a non-Gemini foundation model on the same benchmark using the same evaluation protocol. All other comparisons are to GPT-4/GPT-4V (different model family) or to prior published results (which may have used different prompts, evaluation scripts, or data preprocessing). The paper does not provide any analysis of how results vary across patient subgroups, English dialects, or medical specialties beyond the NEJM CPC specialty breakdown (Table C4) — a breakdown that reveals substantial variation (top-10 accuracy ranges from 55.6% in Pathology to 83.3% in Pediatrics, though sample sizes per specialty are small).

Mitigation status: Partially acknowledged. The paper notes that benchmarks "have limitations around dataset size and quality" (Section 5), that "retraining Med-Gemini-M 1.5 with a new split of the PAD-UFES-20 dermatology dataset leads to a drop of 7.1%" (Section 5), and that "further rigorous evaluation will be crucial before real-world deployment in this safety-critical domain" (Abstract). The Discussion (Section 5) devotes substantial space to responsible AI considerations including fairness, equity, and the need for "deep intersectional analysis of impact" — but these are presented as future work rather than as limitations of the current evaluation. The paper does not characterize its results as specific to the Gemini architecture, U.S. medical context, or English language, leaving readers to infer the scope of generalizability from the benchmark descriptions.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper makes two interventions that shift the conversation in medical AI, operating at different levels of abstraction. The first is a demonstration that generalist foundation models, when systematically specialized across text, multimodal, and long-context dimensions, can achieve state-of-the-art performance without the monolithic "one model for all tasks" architecture that had become aspirational in the field. The second, and potentially more consequential, is a methodological argument — backed by rigorous expert relabeling — that the most popular medical AI benchmark may be approaching a quality ceiling, making further optimization on it potentially misleading about genuine clinical reasoning progress.

The model-family architecture as a reframing of generalist medical AI. Prior to this work, the dominant vision for medical AI systems was articulated by Moor et al. (2023a) as "generalist medical AI" — a single model handling all medical tasks across all modalities. Systems like Med-PaLM M (Tu et al., 2024a) demonstrated early feasibility by achieving competitive performance across radiology, pathology, dermatology, and genomics within one model. The implicit assumption was that increasing model scale and training data diversity would eventually yield a unified system matching specialist performance everywhere. Med-Gemini departs from this vision by demonstrating that shared foundations plus targeted specialization produces better real-world outcomes than a single generalist, because deployment constraints create optimization trade-offs that a one-size-fits-all architecture cannot satisfy. This is not a theoretical claim — it is embodied in the design of four distinct model variants, each making different trade-offs between model size, inference cost, modality coverage, and task-specific performance.

The specific empirical evidence that supports this reframing: (1) Med-Gemini-L 1.0 achieves 91.1% on MedQA with uncertainty-guided search, a task requiring complex multi-step clinical reasoning best served by the largest model, while Med-Gemini-S 1.0 achieves 57.7% on ECG-QA with a custom biosignal encoder, a task requiring specialized waveform processing that the largest model cannot perform natively — a single model could not simultaneously achieve both results. (2) The out-of-distribution multimodal benchmarks (NEJM Image Challenge: 69.7%, USMLE-MM: 93.5%) are handled well by the zero-shot large model without fine-tuning, while specialized pathology (Path-VQA: 64.7% token F1) and ECG tasks require fine-tuning or custom encoders — demonstrating that the need for specialization varies by modality, and a model optimized for all tasks would be either unnecessarily specialized for easy modalities or inadequately specialized for hard ones. (3) The referral letter generation result (100% preferred or tied with experts using the mid-size Med-Gemini-M 1.0) demonstrates that complex diagnostic reasoning capability is unnecessary for documentation tasks where fluency, structure, and completeness matter more than differential diagnosis accuracy.

The conceptual shift: A single model attempting to be optimal on all axes simultaneously faces a tension between the capacity needed for complex reasoning (favoring larger models), the inference latency acceptable for real-time multimodal dialogue (favoring smaller models), the context window needed for full-EHR processing (requiring architectures like Gemini 1.5 Pro's mixture-of-experts), and the modality-specific encoders needed for raw biosignals (requiring architectural customization). By accepting that these tensions are fundamental rather than temporary — that they persist at any scale because they arise from deployment constraints, not model capacity limits — the model-family approach makes specialization a first-class architectural principle rather than a compromise. This reframes the research question from "how do we build a single model that does everything?" to "which capabilities should be bundled together for which deployment scenarios, and what is the minimum shared foundation needed to enable efficient specialization?"

The benchmark saturation argument as a diagnostic for the field. The MedQA relabeling study (Section 4.1, Figure 4b) — which recruited 18 board-certified primary care physicians to re-annotate all 1,273 test questions, used bootstrapped committee voting to quantify label uncertainty, and demonstrated that approximately 7.4% of questions have data quality issues — makes a methodological contribution that transcends this specific benchmark. It establishes that accuracy improvements on popular medical benchmarks can become decoupled from genuine clinical reasoning improvements when the remaining errors are concentrated on questions with flawed labels, missing information, or inherent ambiguity. This is not just a data-cleaning exercise; it changes how the field should interpret benchmark results.

The specific diagnostic signals: (1) Med-Gemini-L 1.0's model uncertainty (Shannon entropy over answer choices) is significantly elevated on questions with label errors or missing information (p = 0.033) — the model recognizes that these questions are problematic, yet benchmark evaluation treats the model's uncertainty as an error. (2) Filtering questions with label errors using unanimous physician voting improves measured accuracy from 91.1% to 91.8%; using majority voting to filter all problematic categories raises it to 92.9% by discarding up to 20.9% of questions — showing that the "room for improvement" on this benchmark is smaller than the raw accuracy number suggests. (3) The fact that Med-Gemini surpasses MedPrompt/GPT-4 (90.2%) by only 0.9 percentage points, but the label-error rate alone is ~2.9%, means that the margin of improvement over prior SoTA is within the noise of benchmark quality issues — a sobering observation for a field that has treated each percentage point on MedQA as meaningful progress.

This finding changes which research directions become more versus less attractive:

Directions that become more attractive:

Comprehensive multi-benchmark evaluation over single-benchmark optimization. The paper evaluates on 14 benchmarks; its strongest contributions are the breadth of demonstrated capabilities, not the 0.9% MedQA margin. Future work that demonstrates capability across diverse, clinically meaningful tasks — even without setting a new SoTA on any single one — becomes more valuable than work that achieves +1% on MedQA through complex prompting. The paper's own long-context evaluations (matching an engineered heuristic system with one-shot in-context learning) and long-form text generation evaluations (matching or exceeding human experts on summarization and simplification) exemplify this shift toward demonstrating useful capabilities rather than optimizing benchmark scores.
Benchmark quality assessment as a standard component of model evaluation. The bootstrapped committee relabeling methodology (Figure C1) provides a template that other researchers can apply to other benchmarks. If similar analyses of NEJM CPC, GeneTuring, or the multimodal VQA benchmarks revealed comparable data quality issues, the field would develop a more nuanced understanding of what benchmark improvements actually measure. The paper explicitly notes that "prior work has suggested similar issues with other popular benchmark datasets" (Section 5) and demonstrates that re-splitting PAD-UFES-20 at the patient level causes a 7.1% accuracy drop — indicating that evaluation protocol, not just data quality, can substantially affect reported results.
Research on model epistemics (knowing when it doesn't know). The finding that Med-Gemini-L 1.0's entropy-based uncertainty correlates with benchmark quality issues (p = 0.033) suggests that model uncertainty estimation is not just a safety mechanism but a diagnostic tool for identifying flawed evaluation data. This opens a research direction where models actively flag potentially problematic test examples during evaluation, complementing human relabeling efforts and enabling more efficient identification of benchmark quality issues.

Directions that become less attractive:

Prompt engineering arms races on MedQA. If a meaningful fraction of remaining errors are on questions with label errors or missing information — as the paper demonstrates — then sophisticated prompting strategies that extract the last few percentage points from this benchmark may be optimizing for a mixture of genuine reasoning improvement and better fit to benchmark artifacts. The paper's own MedPrompt comparison (90.2% vs. 91.1%) represents a 0.9 percentage point gain, while the label error rate alone is ~2.9% — suggesting that the signal-to-noise ratio for incremental MedQA improvements is deteriorating. Future work that reports only MedQA accuracy, without characterizing which errors are on high-quality versus problematic questions, will be less informative than work that provides this breakdown.
"Single number beats all" paper narratives. A paper whose primary contribution is "we got X% on benchmark Y, beating prior SoTA by Z%" now faces a higher burden of proof: the improvement must be demonstrated to occur on well-formed questions, not on questions with missing information or label errors. The MedQA relabeling study provides a concrete methodology for meeting this burden, and papers that do not perform any label quality analysis will be increasingly vulnerable to the critique that their reported gains may reflect benchmark noise rather than capability improvement.

Reconciliation of prior contradictory findings. The paper partially reconciles a tension in the medical AI literature between papers reporting that LLMs can match or exceed physician performance on medical exams (Singhal et al., 2023b; Nori et al., 2023) and skeptical voices noting that benchmark performance does not translate to clinical utility (Fleming et al., 2023; Xu et al., 2023). The reconciliation is not through new capability results — Med-Gemini achieving 91.1% continues the trend of rising benchmark scores — but through the relabeling analysis, which demonstrates that benchmark scores can rise while the remaining errors concentrate on questions that clinicians themselves consider flawed. This means that high benchmark scores and limited clinical utility are not contradictory: the model may genuinely excel at answering well-formed questions while still failing on the types of ambiguous, incomplete, or multi-answer questions that characterize real clinical reasoning. The paper also provides direct evidence for the benchmark-utility gap through the PAD-UFES-20 re-split result: a 7.1% accuracy drop from image-level to patient-level splitting shows that protocol differences alone can create performance differences comparable to or larger than genuine model improvements.

Follow-Up Research This Work Enables

Quantifying the accuracy cost of the uncertainty threshold versus always-on search. The paper's uncertainty-guided search strategy (Section 2.1, Figure 4a) demonstrates that conditioning search on model uncertainty improves performance from 87.2% to 91.1% across four iterations, but it never answers the counterfactual: what accuracy would be achieved by performing web search for every question, regardless of uncertainty? Always-on search would add latency and cost but might achieve higher accuracy — potentially 92% or more — by resolving uncertainty even for questions where the model is initially confident but wrong. This experiment is straightforward to conduct: disable the entropy threshold, perform one to four iterations of search on all questions, and measure the accuracy-latency trade-off curve. A strong follow-up would report: (1) accuracy vs. fraction of questions searched vs. latency for uncertainty-guided and always-on search, (2) whether the questions where always-on search improves over uncertainty-guided search are the same questions where the model is confidently wrong (low entropy but incorrect answer), and (3) whether the model can be trained to recognize this "unknown unknown" state (low entropy but incorrect) as distinct from genuinely known answers. The paper's finding that model uncertainty correlates with benchmark quality issues (p = 0.033) but does not perfectly identify all erroneous answers suggests that some errors occur under low uncertainty — quantifying this fraction and determining whether search can rescue these errors would define the ceiling for uncertainty-guided retrieval strategies.

Cross-foundation-model replication of the uncertainty-guided search and self-training pipeline. All of the paper's text reasoning results use Gemini 1.0 Ultra as the base model. The uncertainty-guided search strategy — generating multiple reasoning paths, computing answer-entropy, conditionally generating search queries from conflicting paths, and iteratively refining — is described in general terms, but its effectiveness may depend on Gemini-specific properties: the model's ability to generate diverse but coherent reasoning paths, its calibration (how well entropy reflects actual uncertainty), and its capacity to integrate search results without being derailed by irrelevant or contradictory information. A critical follow-up would replicate the pipeline using a different base model family (e.g., GPT-4, Claude 3, Llama 3) on the same benchmarks, ideally with the same self-training data generation procedure. A negative result — e.g., GPT-4 showing smaller or nonexistent gains from uncertainty-guided search, or requiring different entropy thresholds, or generating less useful search queries from conflicting reasoning paths — would establish that the strategy is architecture-dependent, not a general inference-time reasoning improvement. A positive result — consistent gains across model families — would establish uncertainty-guided search as a broadly applicable technique, analogous to chain-of-thought prompting, that can be layered onto any sufficiently capable base model. The paper's self-training data generation pipeline (MedQA-R and MedQA-RS) is also described in sufficient detail to replicate; measuring whether the quality of self-generated CoTs varies across base models would reveal whether the self-training approach depends on the base model's initial reasoning quality.

Evaluating whether uncertainty-guided search queries actually resolve the identified conflicts. The paper's key design claim for uncertainty-guided search is that "conditioning on previously generated but conflicting responses is to retrieve search results that are directly targeted at resolving the model's uncertainty to the question" (Section 2.1). This claim remains empirically untested within the paper. A targeted follow-up would analyze the generated search queries, the retrieved results, and the subsequent reasoning paths to measure: (1) whether queries generated from conflicting reasoning paths are more specific or clinically relevant than queries generated from the question alone (a baseline that is trivial to implement by removing the conflicting paths from the query-generation prompt), (2) whether the search results actually address the identified points of disagreement (e.g., if two reasoning paths disagree about whether a lab value is normal, does the search result provide the normal range?), and (3) whether the model's post-search reasoning explicitly resolves the pre-search conflict using the retrieved information or simply adopts the search-result answer without addressing why the alternative reasoning path was wrong. This would require annotating a sample of search iterations with these meta-evaluations, likely using clinician raters to assess whether the search results are relevant and whether the conflict resolution is clinically sound. If the analysis reveals that queries generated from conflicting paths are not significantly better than question-only queries, or that search results are often tangential to the actual disagreement, the core motivation for the uncertainty-guided approach would be weakened, and simpler always-on retrieval with question-based queries might be equally effective.

Comprehensive difficulty-stratified evaluation of the long-context EHR capabilities. The MIMIC-III Needle-in-a-Haystack task (Section 4.3, Table 3) demonstrates that Med-Gemini-M 1.5 can match an engineered heuristic system using one-shot in-context learning, but the evaluation is restricted to conditions mentioned exactly once across all notes — the simplest possible retrieval scenario once the condition is found. A systematic follow-up would construct a graded difficulty benchmark by varying: (1) mention frequency — conditions mentioned 1, 2–5, 6–20, or >20 times, testing whether the model's performance degrades as it must aggregate evidence across multiple mentions with potentially contradictory implications, (2) mention context — conditions mentioned in diagnostic certainty ("patient has X"), differential diagnosis ("rule out X"), family history ("mother had X"), negated contexts ("no evidence of X"), or historical/resolved contexts ("history of X, resolved in 2019"), testing whether the model correctly distinguishes these pragmatic contexts as the heuristic baseline does through rule-based selection, (3) condition type — acute conditions (sepsis, pneumonia) versus chronic conditions (hypertension, diabetes) versus rare conditions (Miller Fisher syndrome), testing whether the model's performance varies with condition prevalence in pretraining data, and (4) EHR length — records of 50K, 200K, 500K, and 1M+ words, testing whether the "lost in the middle" phenomenon (Liu et al., 2024) affects retrieval of conditions mentioned in the middle of very long contexts. The paper's current dataset (200 examples, 44 patients, single-mention conditions only) provides a strong proof-of-concept but cannot characterize the capability boundaries. A graded benchmark with 500–1,000 examples across these dimensions would transform the result from "the model can do this task" to "here is the difficulty ceiling below which the model can replace the engineered system, and above which the engineered system still outperforms."

Patient-level and cross-institution generalization of the dermatology fine-tuning. The paper's finding that re-splitting PAD-UFES-20 at the patient level causes a 7.1% accuracy drop (Section 5) is presented as a caution about benchmark evaluation but points to a deeper question about the fine-tuning procedure: does the model learn patient-specific visual features (lighting conditions, camera characteristics, skin tone patterns unique to individual patients) rather than disease-specific features? A direct follow-up would: (1) fine-tune Med-Gemini-M 1.5 exclusively on the patient-level split of PAD-UFES-20 and compare performance to the image-level-split model, measuring whether the performance gap is due to data leakage (images of the same lesion in train and test) or genuine patient-specific feature learning, (2) evaluate both models on an entirely external dermatology dataset (e.g., the SCIN dataset from Ward et al., 2024, which the paper uses qualitatively in Figure 6 but not quantitatively) to measure cross-dataset generalization, and (3) stratify performance by skin tone using the Fitzpatrick skin type annotations available in PAD-UFES-20, testing whether the performance drop from image-level to patient-level splitting is uniform across skin tones or concentrated in underrepresented groups — a critical fairness question that the paper identifies as important future work (Section 5, "Responsible AI") but does not address in current evaluations. This experiment would distinguish between two hypotheses: (H1) the 7.1% drop is purely a data leakage artifact, and the model generalizes well across patients and datasets once leakage is removed, versus (H2) the model overfits to patient-specific and dataset-specific features, and cross-dataset generalization is substantially worse than within-dataset performance even with proper splitting.

Cost-accuracy trade-off characterization for deployment-relevant inference strategies. The paper reports accuracy across search iterations (Figure 4a: 87.2% → 88.8% → 90.0% → 90.5% → 91.1%) but provides no latency, token count, or monetary cost data, making it impossible for practitioners to determine which operating point on this curve is optimal for their deployment context. A practically essential follow-up would measure: (1) wall-clock latency per MedQA question at each iteration count (with breakdown into generation time, search API time, and entropy computation overhead), (2) total output tokens generated per question (reasoning paths + search queries + final answer), (3) search API cost per iteration (assuming a commercial web search API), and (4) the actual distribution of search iterations triggered — what fraction of questions terminate at iteration 0 (below threshold), iteration 1, iteration 2, iteration 3, and iteration 4, and what is the accuracy conditional on termination iteration? This last measurement would reveal, for example, that 60% of questions terminate at iteration 0 with 95% accuracy (fast, cheap, accurate), 25% require 1–2 iterations, and 15% require 3–4 iterations with diminishing accuracy gains — enabling a dynamic budget allocation strategy that spends compute where it helps most. Without these measurements, the uncertainty-guided search strategy is an academic result (91.1% accuracy under unspecified cost) rather than an engineering specification (e.g., "achieves 90.5% accuracy at median latency of 4.2 seconds per question, with 80th percentile latency of 8.7 seconds"). The paper's model-family design philosophy — optimizing for deployment trade-offs — makes this omission particularly notable: the design claims to consider "training data, compute availability, and inference latency" (Section 1) but provides no latency or compute data to guide those decisions.

Practical Applications and Downstream Use Cases

Automated clinical documentation in high-volume outpatient settings. The paper's long-form text generation evaluations (Figure 5, Table C5) provide preliminary evidence that Med-Gemini-M 1.0 can produce after-visit summaries and referral letters rated as comparable to or better than human expert output. This is directly applicable to outpatient clinic workflows where documentation burden is a leading cause of clinician burnout. In a typical primary care practice, a physician may spend 1–2 hours per day on documentation outside of patient-facing time. The after-visit summary result — 67.7% of model-generated summaries rated as good or better than expert human summaries — suggests that deployment as a first-draft generation tool could meaningfully reduce documentation time, with clinicians reviewing and editing rather than composing from scratch. The referral letter result (100% preferred or tied, though subject to the evaluation validity concerns noted in Section 6) suggests an even stronger case for automation, given that referral letters are relatively formulaic and the model can populate them from structured EHR data. The practical implementation would integrate Med-Gemini-M 1.0 into an EHR system where it generates draft summaries and letters that clinicians review in-line, with the key metric being time saved per clinician per day. The paper's 25-example evaluation is a pilot; deployment would require larger-scale validation with 500–1,000 examples across multiple clinic types, specialties, and documentation templates, plus measurement of clinician acceptance and editing behavior.

EHR "needle-in-a-haystack" retrieval for complex case review and clinical trial matching. The long EHR retrieval result (F1 0.77, matching an engineered heuristic system with one-shot in-context learning; Table 3) demonstrates that Med-Gemini-M 1.5 can identify rarely-documented conditions within records spanning 200K–700K words — a task that is time-consuming and error-prone for human clinicians. Two immediate deployment scenarios: (1) Complex patient case review: When a patient with an extensive medical history is admitted, the admitting physician must rapidly identify all active conditions, prior procedures, relevant medications, and allergies — information that may be scattered across hundreds of notes from multiple specialties. A system that can answer targeted natural-language queries ("Has this patient ever had an adverse reaction to contrast dye?" "Is there any documented history of malignant hyperthermia?") with direct quote evidence and source note identification, as Med-Gemini-M 1.5 demonstrates in Figure 8, could reduce the time required for chart review from tens of minutes to seconds, while providing auditable evidence trails. (2) Clinical trial eligibility screening: Determining whether a patient meets inclusion/exclusion criteria for a clinical trial often requires verifying the presence or absence of specific conditions, lab values, prior treatments, and comorbidities — a manual review process that is a major bottleneck in trial recruitment. A long-context model that can process the complete patient record against a structured set of eligibility criteria, flagging matches and mismatches with cited evidence, could substantially accelerate screening throughput. The paper's current task tests single-condition queries; extending to multi-condition screening with structured criteria templates is a natural next step, with the key metrics being recall (not missing eligible patients) and precision (not flagging ineligible patients due to misinterpreted mentions in family history or differential diagnosis contexts).

Medical education and surgical training augmentation with video understanding. The paper's qualitative demonstrations of Med-Gemini-M 1.5's video understanding capabilities — identifying surgical actions in AVOS videos (Figure 11), conducting educational dialogue about laparoscopic cholecystectomy (Figure 12), and localizing instructional content in physical therapy videos (Figure 10, with SoTA quantitative results in Table 3) — suggest immediate applications in medical education. For surgical training specifically: a system that can process full-length procedure videos and answer trainee questions in natural language ("At what point did the surgeon achieve the critical view of safety?" "Show me the segment where the cystic duct was clipped") could augment traditional video review, which requires trainees to watch hours of footage to find relevant teaching moments. The MedVidQA quantitative results (mIoU 65.8 with subtitles, 43.4 without) demonstrate that the model can localize relevant segments with reasonable accuracy even without task-specific training, and the subtitle-based performance is state-of-the-art. For deployment in surgical education, the system would index a library of annotated procedure videos and support natural-language querying with timestamp-accurate retrieval, with the key metrics being retrieval accuracy (does the returned segment actually show what was asked for?) and educational utility (do trainees learn faster or perform better on skills assessments after using the system?). The paper's zero-shot results on the CVS task (50.0% accuracy, below the supervised ResNet3D at 67.0%) indicate that specialized surgical anatomy assessment requires fine-tuning, not just prompting — a practical deployment would likely combine zero-shot retrieval (which works well for language-anchored queries about actions and phases) with fine-tuned classifiers for safety-critical visual assessments (which require domain-specific training data).