ArXiv: 2510.09201

🎯 Pitch

Current prompt tuning is stuck in a text-only box, even for models that see. MPO breaks out by jointly evolving image and text prompts, using parent evaluations as priors to slash the search budget by 42% while achieving a 7.4-point average gain over text-only methods.

1. Executive Summary

This paper introduces the novel problem of multimodal prompt optimization, which expands automatic prompt optimization beyond text to jointly optimize the pair of textual and non-textual prompts for Multimodal Large Language Models (MLLMs), and proposes the Multimodal Prompt Optimizer (MPO) — a unified framework that uses alignment-preserving exploration to jointly refine both modalities through cohesive backpropagation (generating unified feedback that guides simultaneous updates) and three complementary operators — generation, edit, and mix (creating entirely new non-textual prompts, incrementally refining existing ones, and blending multiple candidates, respectively) — and employs a prior-inherited Bayesian UCB selection strategy that warm-starts the evaluation of child prompts using their parent's performance as an informative prior. Across 10 datasets spanning images (PlantVillage, CUB), videos (Drive&Act, VANE-Bench), and molecules (Absorption, BBBP, CYP Inhibition), MPO consistently outperforms leading text-only optimization methods — achieving substantial gains such as a 7.4 percentage point average improvement over ProTeGi — and reduces the evaluation budget by 42% compared to a prior-free baseline, establishing that expanding the prompt search space into the multimodal domain unlocks capabilities of MLLMs that text-only prompts cannot access while demonstrating that effective multimodal prompt optimization requires both cross-modally coherent exploration and efficient candidate selection to navigate the enlarged combinatorial space.

2. Context and Motivation

The Core Gap: Prompt Optimization Remains Blind to Non-Textual Modalities

The fundamental problem this paper addresses is straightforward yet consequential: automatic prompt optimization, despite its demonstrated success for text-only LLMs, has been entirely confined to the textual modality, even as the models themselves have expanded to process images, videos, molecules, and more. This creates a structural mismatch — MLLMs are capable of leveraging rich multimodal signals, but the tools we use to elicit their best behavior are restricted to text.

To appreciate the severity of this gap, consider what a "prompt" means for an MLLM. When a user queries GPT-4V, Qwen2.5-VL, or Gemini about an image, the prompt is not merely the text instruction ("classify this bird"). The full prompt includes the image itself, any reference images provided as exemplars, and the textual framing. Yet existing automatic prompt optimization (APO) methods — including the most sophisticated approaches like ProTeGi (Pryzant et al., 2023) and SEE (Cui et al., 2025b) — treat only the textual portion as optimizable. The non-textual components are either fixed (a single user-uploaded query image) or absent (no reference images, no visual demonstrations).

This constraint is not a minor inconvenience. As the authors illustrate in Figure 1, textual descriptions of visual concepts carry inherent ambiguity. Describing "a Black-footed Albatross with a pale bill" in words requires the reader to interpret "pale" and "bill shape" through language alone — a lossy, subjective process. A reference image of the bird conveys the same information directly, with the precision that text struggles to match. By restricting optimization to text, existing methods forfeit the very expressive capacity that makes MLLMs powerful.

Why This Matters: Practical and Conceptual Stakes

The paper motivates this problem along several dimensions that carry both immediate practical weight and broader conceptual significance for the field.

Underutilization of MLLM capabilities. MLLMs are trained at enormous expense to process multimodal inputs — vision encoders, video processing pipelines, molecular structure tokenizers are all integrated with LLM backbones through large-scale pre-training and instruction tuning (Liu et al., 2023; Chu et al., 2023; Gemini, 2025; Bai et al., 2025). Yet if prompts remain text-only, these multimodal processing pathways are only engaged for the query itself, not for the instructional context that shapes how the model reasons. The consequence is a systematic underinvestment: organizations spend heavily on multimodal training but then deploy these models with text-only prompting strategies that cannot leverage the full architecture. The paper's experiments quantify this gap — across 10 datasets, text-only optimization methods consistently underperform MPO by substantial margins (Table 1), demonstrating that the text-only limitation is not merely theoretical but carries measurable performance costs.

Real-world task demands. The paper evaluates on domains where multimodal context is genuinely essential, not optional. Fine-grained bird species classification (CUB-200-2011) requires distinguishing visually similar species — a task where "the bird has dark plumage and a curved crest" is an intrinsically impoverished description compared to a reference image showing the Crested Auklet alongside similar species. Plant disease identification (PlantVillage) requires recognizing subtle visual symptoms (yellowing edges, dark spots, brown patches) that are laborious to describe exhaustively. Medical visual question answering (SLAKE) demands spatial reasoning about radiological images that textual descriptions cannot substitute. Drug property prediction (Absorption, BBBP, CYP Inhibition) requires understanding molecular substructures whose textual SMILES representations are opaque to all but expert chemists. In each of these domains, the paper argues, text-only prompts are not just suboptimal — they are fundamentally limited in ways that multimodal signals can overcome.

The burden of manual prompt engineering is amplified for modalities. The original motivation for APO was to reduce the human effort required to craft effective text prompts. When prompts expand to include non-textual components, this burden intensifies: not only must the human designer write clear instructions, but they must also select, create, or edit reference images, video clips, or molecular structures that support the task. For a video action recognition task, what 5-second clip best illustrates "adjusting the sun visor" versus "checking the rearview mirror"? For a molecular property prediction task, which reference molecule best exemplifies blood-brain barrier penetration? These are non-trivial decisions requiring domain expertise, and the combinatorial space of possible multimodal prompts (text × image/video/molecule) is vastly larger than text alone. Automating this process — the central contribution of MPO — addresses a bottleneck that manual approaches cannot feasibly solve at scale.

A gap in the research landscape. The paper positions itself within a well-developed APO literature that has made rapid progress in text-only settings. Methods like APE (Zhou et al., 2023) use LLMs to generate candidate prompts by paraphrasing or reverse-engineering from examples. OPRO (Yang et al., 2024) treats the LLM as an optimizer guided by prompt-performance pairs. EvoPrompt (Guo et al., 2024) applies evolutionary operators (mutation, crossover) to a population of prompts. ProTeGi (Pryzant et al., 2023) simulates gradient descent by generating textual "gradients" (failure critiques) and editing prompts in the opposite semantic direction. SEE (Cui et al., 2025b) jointly optimizes instructions and in-context examples through alternating explore-exploit phases. These methods have demonstrated impressive results — but their optimization vocabulary is exclusively linguistic. None of them can generate, edit, or select non-textual prompt components. The paper's observation is that this limitation is not inherent to the optimization problem but rather to the design of existing methods, which were conceived in an era when LMs processed only text. The field's progression to MLLMs has created a capability that the optimization tools have not caught up to.

Where Prior Approaches Fall Short

The paper identifies specific limitations in existing work along several axes that collectively motivate the need for a new framework.

Text-only APO methods cannot express multimodal information. The most direct limitation is architectural: text-only optimizers, by design, operate in the space of token sequences. When they generate or refine prompts, the output is always a string. Even if that string contains elaborate visual descriptions ("the bird has a white wing patch bordered by black primaries and a yellow nape"), these descriptions must be interpreted by the MLLM through language understanding, introducing an unnecessary compression step. The model must reconstruct visual features from text rather than perceiving them directly. This is analogous to describing a photograph over the phone rather than showing it — possible but needlessly lossy. The paper demonstrates empirically that this matters: in the ablation study on modality contributions (Table 3), using MPO's optimized text prompt alone (without the optimized image) achieves only 55.6% on PlantVillage and 64.2% on CUB, while the full multimodal prompt achieves 76.4% and 78.6% respectively. The gap between text-only and multimodal prompts is substantial and consistent.

Independent optimization risks cross-modal misalignment. A naive extension to multimodal optimization might update text and non-textual prompts independently — first optimize the text, then generate a matching image, or vice versa. The paper shows this "Sequential" approach (Section 4.2, Figure 4) achieves lower performance and worse cross-modal alignment (measured by DSG score) than MPO's joint optimization. The reason is intuitive: feedback from task failures often implicates both modalities. If the model misclassifies a bird because the text instruction emphasized beak shape while the reference image was poorly lit, an independent update might fix the text but leave the suboptimal image unchanged, or generate a new image that contradicts the refined text. MPO's cohesive backpropagation addresses this by generating a single unified feedback that encodes weaknesses across both modalities, ensuring that subsequent updates to text and non-textual components are guided by the same diagnostic signal.

Existing multimodal prompting work is instance-specific, not task-level. The paper distinguishes its contribution from a separate line of research on instance-specific multimodal prompting. Methods like MM-CoT (Zhang et al., 2024b) generate intermediate textual rationales for individual queries. Visual prompting techniques add bounding boxes, points, or segmentation masks to guide attention on a per-image basis (Zhou et al., 2024; Jiang et al., 2024; Lin et al., 2024). In text-to-image generation, prompt optimization refines the input to produce outputs more faithful to user intent (Mañas et al., 2024; Mo et al., 2024; Gao et al., 2025). While effective, these techniques optimize per query — each new input requires fresh optimization. MPO, in contrast, pursues the APO paradigm of discovering a single reusable prompt that improves performance across an entire task. This distinction matters for deployment: a per-query optimizer incurs inference-time overhead on every user request, while a task-level optimizer amortizes its cost over all future uses. The paper argues that the task-level multimodal optimization problem has been entirely unaddressed, and that solving it requires fundamentally different machinery (exploration strategies that balance diversity and alignment across a task distribution, selection strategies that efficiently identify high-quality prompts from a large candidate pool).

The enlarged search space demands efficient selection. Prior APO methods typically handle candidate selection through uniform evaluation (each candidate gets equal budget) or standard bandit algorithms (UCB). But these strategies were designed for text-only search spaces where prompt quality varies relatively smoothly and good prompts are not extremely sparse. When the search space expands to include non-textual modalities, the paper argues, high-quality prompts become sparser — there are many more ways for a multimodal prompt to be bad (misaligned text and image, irrelevant reference, contradictory signals) than to be good. Consequently, uniform allocation wastes substantial budget evaluating low-potential candidates. The paper provides both theoretical and empirical motivation for addressing this: Proposition 3.1 guarantees that informative parent priors reduce the best-arm identification cost, and Figure 5 shows that MPO's prior-inherited Bayesian UCB achieves the same performance as uniform allocation with only 30% of the budget, a 70% reduction in resource cost. This efficiency gain is not merely convenient — it is necessary for the approach to scale, since evaluating multimodal prompts requires running the full MLLM on a batch of task examples, which is computationally expensive.

How This Paper Positions Itself

The paper frames its contribution through a clear problem definition that expands the APO formulation to the multimodal domain. The objective function (Section 3.1) formalizes this: instead of finding an optimal text prompt t∗t^*, the goal is to find an optimal pair (t∗,m∗)(t^*, m^*) where tt is a textual prompt and mm is a non-textual prompt (image, video, molecule), maximizing expected task performance over the dataset. This is not a minor notational change — it doubles the degrees of freedom and introduces the cross-modal consistency constraint that the two components must work together rather than at cross-purposes.

The paper positions MPO as addressing this expanded objective through two technical innovations that correspond to the two challenges identified above:

  1. Alignment-preserving exploration (Section 3.2) tackles the cross-modal consistency challenge. Rather than treating text and non-textual optimization as independent processes, MPO couples them through a shared feedback signal (cohesive backpropagation) and derives modality-specific conditions for non-textual generation from the same refinement process that updates the text. The three operators — generation (exploring entirely new non-textual prompts), edit (incrementally refining existing ones), and mix (blending strengths of multiple candidates) — are designed to provide complementary exploration behaviors that collectively cover the multimodal search space more effectively than any single operation. The paper's ablation (Table 4) confirms that combining all three operators outperforms any individual one.

  2. Prior-inherited Bayesian UCB (Section 3.3) tackles the candidate selection challenge. The key insight is empirical: parent and child prompt performance correlates strongly (Pearson's r=0.88r = 0.88, Figure 3). This correlation is not guaranteed — it could be that optimization steps are essentially random perturbations — but the paper demonstrates it holds robustly across the optimization trajectory, suggesting that good parent prompts tend to produce good children, and vice versa. The selection strategy exploits this by initializing each child prompt's Beta-distributed score estimate with pseudo-observations proportional to the parent's posterior mean, effectively warm-starting the evaluation. The paper provides a formal proposition (Proposition 3.1) guaranteeing that this informative prior reduces best-arm identification cost relative to a uniform prior, under the assumption that the parent prior is KL-closer to the true child performance than the uninformative prior (which Figure 3 supports).

The paper also positions itself at the intersection of multiple research threads — MLLM development, APO, bandit-based algorithm selection, and multimodal representation learning — but does so judiciously. The related work section (Section 2) situates MPO relative to each thread without overclaiming. The paper does not claim to advance MLLM architectures or training; it takes these as given and focuses on the orthogonal problem of how to use them effectively. Similarly, it does not claim to invent Bayesian UCB but to apply it with a novel prior-inheritance mechanism that is specifically motivated by the structure of prompt optimization (where parent-child relationships provide natural informative priors). This focused positioning — addressing a clearly defined gap with technically motivated solutions — gives the paper a coherent narrative arc.

In summary, the paper's motivating argument is: MLLMs can process multimodal inputs, but the methods we use to prompt them cannot optimize multimodal prompts; this limitation is both practically costly and technically unnecessary; solving it requires (a) exploration strategies that maintain cross-modal coherence while probing a larger search space and (b) selection strategies that efficiently identify high-quality candidates in a sparser reward landscape. MPO is presented as a first framework to satisfy both requirements, with extensive empirical validation across diverse modalities as evidence.

3. Technical Approach

This is primarily a systems and algorithms paper whose core idea is that multimodal prompt optimization requires two complementary mechanisms: exploration strategies that jointly refine text and non-textual prompts while maintaining cross-modal coherence, and selection strategies that leverage parent-child performance correlations to efficiently identify high-quality multimodal prompts in an enlarged combinatorial search space.

3.1 Reader Orientation

The paper builds a system that automatically discovers the best multimodal prompt — a pair of a textual instruction and a non-textual reference (image, video, or molecule) — that maximizes an MLLM's task performance, replacing the manual, text-only process of prompt engineering with an automated optimization loop that simultaneously refines both modalities and intelligently allocates evaluation resources.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components operating in a closed optimization loop:

  1. Base MLLM (Qwen2.5-VL or Qwen3) — the model being prompted. It takes the multimodal prompt and task queries as input, generates predictions, and its errors constitute the failure set that drives optimization. This model is treated as a black box — its parameters are frozen.
  2. Optimizer LLM (GPT-4o mini) — an LLM that performs the meta-cognitive work: analyzing failures, generating improvement feedback, refining text prompts, and producing textual conditions that guide non-textual prompt generation or editing. It is the "brain" of the optimization process.
  3. Modality-Specific Generator (GPT-Image, Wan2.1, GPT-4o mini) — an external tool (text-to-image, text-to-video, or text-to-molecule) that converts textual conditions into actual non-textual prompts. This decouples prompt design (handled by the optimizer) from prompt synthesis (handled by specialized generators).
  4. Candidate Selection Module (Prior-Inherited Bayesian UCB) — a bandit-based evaluator that allocates a fixed evaluation budget across candidate multimodal prompts. It inherits prior information from parent prompts to warm-start child evaluations and uses UCB scoring to balance exploration and exploitation during the limited evaluation rounds.
  5. Beam Search Controller — orchestrates the iterative loop: maintains a beam of top-b multimodal prompts, generates b² children per iteration via the exploration operators, evaluates children through the selection module, and selects the top-b prompts to carry forward.

Information flows as follows: an initial textual prompt enters → the optimizer generates b² initial multimodal prompts using only the generation operator → the selection module evaluates all candidates by running the base MLLM on mini-batches of task data → the top-b candidates (by posterior mean score) are retained → for each of T iterations: each parent prompt's failures are analyzed to produce unified feedback → the optimizer jointly refines the text and generates a modality-specific condition → the condition is passed to the generator to create/revise/blend a non-textual prompt → all child prompts enter the selection module with priors inherited from their parents → the top-b prompts become parents for the next iteration → after T iterations, the prompt with the highest posterior mean is returned.

3.3 Roadmap for the Deep Dive

  • First, the formal problem definition (Section 3.1), which establishes the multimodal prompt optimization objective and clarifies why the expanded search space introduces two specific challenges — cross-modal consistency and candidate sparsity — that the subsequent mechanisms address.
  • Second, the alignment-preserving exploration framework (Section 3.2), which is the core mechanism for generating candidate multimodal prompts. This covers: cohesive backpropagation (how unified feedback is generated), joint multimodal update (how text and non-textual components are refined together), and the three exploration operators (generation, edit, mix) that provide complementary search behaviors.
  • Third, the prior-inherited Bayesian UCB selection strategy (Section 3.3), which addresses candidate evaluation efficiency. This covers: the empirical motivation (parent-child performance correlation), the Bayesian modeling with informative priors, the UCB selection mechanism, and the theoretical guarantee that informative priors reduce best-arm identification cost.
  • Fourth, the iterative optimization algorithm that integrates exploration and selection within a beam search framework, specifying how operators are applied, how many candidates are generated per iteration, and how the overall budget is managed.

3.4 Detailed, Sentence-Based Technical Breakdown

This paper formalizes a new optimization problem and proposes a framework with two tightly integrated components: alignment-preserving exploration for generating coherent multimodal candidates, and prior-inherited Bayesian selection for efficiently identifying the best among them.

Problem Definition: Expanding Prompt Optimization to the Multimodal Space

The paper begins by formally defining MLLMs and then redefining the prompt optimization objective to include non-textual modalities. This formalization serves as the conceptual foundation for everything that follows, because it makes explicit what "optimal" means in the multimodal setting and why the expanded space is qualitatively harder to search.

MLLM definition. An MLLM is represented as a function MLLM: $(T \cup M)^* \rightarrow T$, where $T$ is the textual input space, $M$ is the non-textual input space (images, videos, molecules, etc.), and $*$ denotes the Kleene star (representing a finite sequence over the combined spaces). Given a multimodal query $q$ and a prompt $p$ (each potentially containing both textual and non-textual components), the model generates a textual output $y = \text{MLLM}(p, q)$. The critical observation in the definition is the parenthetical: prior work has implicitly restricted $p$ to a purely textual form $p = t \in T$, leaving the non-textual dimensions of $M$ unused for prompt optimization.

Multimodal prompt definition. A multimodal prompt is defined as a pair $p = (t, m) \in T \times M$, where $t$ is the textual prompt and $m$ is the non-textual prompt (for example, a reference image of bird species for fine-grained classification, a short video clip illustrating a driver action, or a molecular structure exemplifying blood-brain barrier penetration).

Optimization objective. Given a task dataset $D$ consisting of query-answer pairs $(q, a)$ and an evaluation metric function $f$ (such as accuracy or F1 score), the objective of multimodal prompt optimization is to find:

(t∗,m∗)=arg⁡max⁡(t,m)∈T×ME(q,a)∼D[f(MLLM(t,m,q),a)](t^*, m^*) = \arg\max_{(t,m) \in T \times M} \mathbb{E}_{(q,a) \sim D} \left[ f\left( \text{MLLM}(t, m, q), a \right) \right]

where $t^*$ and $m^*$ are the optimal textual and non-textual prompts respectively, $T \times M$ is the joint search space of all possible text and non-text prompt pairs, $D$ is the task dataset, $f$ is a task-specific evaluation metric (e.g., accuracy for classification, F1 for imbalanced molecular tasks), and the expectation is taken over query-answer pairs sampled from the task distribution.

What it computes: the function finds the single multimodal prompt pair $(t, m)$ that, when used as context for the MLLM across all queries in the task distribution, maximizes the expected score on the evaluation metric. The optimization is over the Cartesian product of the textual and non-textual spaces, meaning both components are free variables.

Why this form: the product space $T \times M$ explicitly captures the combinatorial nature of multimodal prompt design. If text and non-textual prompts were optimized independently ($\arg\max_t$ followed by $\arg\max_m$), the resulting pair might be suboptimal because the two modalities interact — a good text prompt paired with a misaligned image could be worse than a mediocre text prompt paired with a well-matched image. The joint optimization over the product space ensures that the selected pair is globally optimal with respect to their combined effect on the MLLM's output. The paper notes that this joint search space "introduces an entirely new axis of non-textual information, which in turn raises two fundamental challenges": maintaining cross-modal consistency (text and non-textual components should provide complementary, not conflicting signals) and managing candidate sparsity (high-quality prompts become rarer in the enlarged space).

Alignment-Preserving Exploration: Joint Optimization with Complementary Operators

The first major component of MPO addresses the cross-modal consistency challenge: how to explore the enlarged multimodal prompt space while ensuring that textual and non-textual components evolve coherently rather than diverging into contradictory signals. The solution has two sub-mechanisms — cohesive backpropagation and joint multimodal update — plus three exploration operators that provide complementary search behaviors.

Cohesive Backpropagation

Failure set identification. For a given multimodal prompt $p = (t, m)$, the system first identifies the set of task instances where the MLLM makes errors when using this prompt: $F = \{(q, a, y) \mid y \neq a\}$, where $y$ is the model's predicted answer and $a$ is the ground truth. This failure set is a concrete diagnostic — it contains the specific examples where the current prompt fails, providing grounded evidence for what needs to improve.

Unified feedback generation. Instead of generating separate feedback for the text and non-textual components (which would risk producing contradictory improvement signals), MPO generates a single unified feedback that encodes cross-modal weaknesses:

∇p=(∇t,∇m)=MLLM(t,m;F)\nabla p = (\nabla t, \nabla m) = \text{MLLM}(t, m; F)

where $\nabla p$ is the unified feedback (also called a "semantic gradient" by analogy to ProTeGi), $\nabla t$ is the textual component of the feedback describing weaknesses in the text prompt, $\nabla m$ is the textual component describing weaknesses in the non-textual prompt, and $F$ is the failure set of wrong examples.

What it computes: the optimizer MLLM (GPT-4o mini) receives the current multimodal prompt along with the failed examples and generates a natural language analysis that diagnoses what went wrong in both modalities simultaneously. The output is a single piece of text that covers both the textual instruction's deficiencies and the non-textual prompt's shortcomings.

Why this form: generating a single feedback covering both modalities ("cohesive backpropagation") ensures that the improvement signals for text and non-text are derived from the same diagnostic reasoning. If feedback were generated independently — first analyzing text failures, then analyzing image failures — the two analyses might identify different root causes, propose conflicting fixes, or miss cross-modal interactions (e.g., the text instruction was correct but the reference image was misleading regarding a specific feature). The unified feedback "mitigates the risk of overfitting updates to one modality" by keeping both improvement trajectories anchored to a shared understanding of what went wrong.

The meta-prompt for this step (provided in Appendix A.4, Figure 10) instructs the optimizer to "analyze the failure case" and produce two sections: a "Text Prompt Analysis" that identifies "missing information, vague instructions, or ambiguous wording" and an "Image Prompt Analysis" that identifies "lack of clarity, poor composition, irrelevant details, or missing key features." The failure set size is fixed at 3 examples in the implementation.

Joint Multimodal Update

Using the unified feedback, MPO simultaneously refines the textual prompt and generates a modality-specific condition that directs non-textual revision:

(t′,c)=MLLM(t,m;F,∇p)(t', c) = \text{MLLM}(t, m; F, \nabla p)

where $t'$ is the updated textual prompt, $c$ is a modality-specific condition (in textual form) describing how the non-textual prompt should be modified, and the optimizer MLLM conditions on the current prompt, the failure set, and the unified feedback.

The condition $c$ is then fed to a modality-specific generator $g$ to produce the updated non-textual prompt:

m′=g(c)m' = g(c)

where $g$ is a task-appropriate generator (GPT-Image for images, Wan2.1 for videos, GPT-4o mini for molecule SMILES strings).

What it computes: the optimizer produces two outputs from a single forward pass: the revised text instruction and a description (in natural language) of what the non-textual prompt should look like after modification. This description is then materialized by an external generator into an actual image, video, or molecule. The two components are generated in a coupled fashion — the text revision and the condition $c$ emerge from the same reasoning step, ensuring they are semantically aligned.

Why this form: the key design choice is that $c$ is generated by the optimizer, not by the generator. The generator's role is purely executional — it takes a textual specification and produces a corresponding non-textual output. The optimizer remains in control of what to generate, maintaining the semantic coherence between $t'$ and $m'$. An alternative design where a separate vision-language model independently generates image suggestions would risk the text and image drifting apart. By deriving $c$ from the same reasoning that produces $t'$, MPO ensures that "updates to $m$ remain consistent with the revised textual prompt $t'$, rather than being optimized in isolation."

The meta-prompts for this step (Figures 11, 12, 13 in Appendix A.4) show that the optimizer is instructed to "write a detailed prompt for an image generator" (or editor, or mixer) that "enhances or redesigns the reference image to resolve issues found in the analysis" and, in the same call, produce an "improved text prompt" that "resolves ambiguities found in the failure analysis" and "elaborates on how the reference image should be interpreted." This coupling within a single LLM call is the mechanism for alignment preservation.

The Three Exploration Operators

The joint multimodal update produces $t'$ and $c$, but the specific type of non-textual modification varies depending on which operator is invoked. MPO defines three operators that provide complementary exploration behaviors, ensuring that the search covers the multimodal space broadly (generating entirely new prompts), deeply (refining existing ones), and combinatorially (blending strengths of multiple prompts). The choice of operator is made randomly for each child prompt generation: at each step, one of the three operators is uniformly sampled.

Generation operator. This operator creates an entirely new non-textual prompt from scratch, without referencing any prior non-textual candidate:

m′=g(cgen,∅)m' = g(c_{\text{gen}}, \emptyset)

where $c_{\text{gen}}$ is the generation condition produced by the optimizer (describing what the new image/video/molecule should contain), and $\emptyset$ indicates that no prior non-textual prompt is provided as input to the generator.

The condition $c_{\text{gen}}$ is produced alongside the updated text: $(c_{\text{gen}}, t') = \text{MLLM}(t, m; \nabla p, F)$.

What it computes: the optimizer analyzes the failure set and unified feedback, then writes a detailed description of what a new reference image/video/molecule should look like to address the identified weaknesses. This description is fed to the generator, which produces a novel non-textual prompt from scratch. The text prompt is simultaneously updated.

Why this operator: generation "explores entirely new non-textual prompts, e.g., novel spatial arrangements in visual inputs or unique substructures in molecules." By decoupling from past candidates, it avoids local optima and explores unexplored regions of the multimodal space. This is particularly important in early iterations when no non-textual prompts exist yet ("initial prompts are unavailable") or when the current candidate pool has converged to a narrow region. In the first iteration, only the generation operator is used to initialize the pool of multimodal prompts (since no non-textual prompts exist to edit or mix).

Edit operator. This operator performs fine-grained refinements of an existing non-textual prompt while preserving its useful structure:

m′=g(cedit,{m})m' = g(c_{\text{edit}}, \{m\})

where $c_{\text{edit}}$ is the edit condition describing specific modifications (e.g., "increase resolution, add labels to bird species, remove distracting background"), and $\{m\}$ indicates that the current non-textual prompt is provided as the starting point for editing.

The condition and updated text are produced as: $(c_{\text{edit}}, t') = \text{MLLM}(t, m; \nabla p, F)$.

What it computes: the optimizer identifies specific visual or structural attributes of the current non-textual prompt that need adjustment (e.g., a reference image shows birds but they are too small to see beak details) and generates an editing instruction that preserves the overall composition while fixing the problematic elements. The generator treats the existing non-textual prompt as input and applies the specified edits.

Why this operator: edit "enables targeted, incremental refinements, making it particularly effective when a prompt is already strong but requires adjustment on specific attributes rather than a complete redesign." This is analogous to local search in optimization — when a prompt is near-optimal but has a specific flaw (e.g., the image shows the right bird species but with confusing lighting), it is more efficient to fix the flaw than to generate an entirely new prompt from scratch. The generator's editing capability (e.g., GPT-Image's inpainting-style modifications) determines what refinements are possible.

Mix operator. This operator blends complementary strengths from multiple parent prompts to create a new candidate that synthesizes the best of each:

m′=g(cmix,{mi}i=1K)m' = g(c_{\text{mix}}, \{m_i\}_{i=1}^K)

where $c_{\text{mix}}$ is the mixing condition describing how to combine the input non-textual prompts, $\{m_i\}_{i=1}^K$ is a set of $K$ non-textual prompts from different parent candidates, and the mixed output $m'$ is expected to inherit beneficial features from each.

The condition and updated text are produced from multiple parents: $(c_{\text{mix}}, t') = \text{MLLM}(\{t_i, m_i; \nabla p_i, F_i\}_{i=1}^K)$, where the optimizer sees the text prompts, non-textual prompts, feedback, and failure sets from all $K$ parents simultaneously.

In the implementation, $K = 2$: one parent is the current prompt being mutated, and a second parent $\tilde{p}$ is randomly selected from the other prompts in the beam (specifically, from $P \setminus \{p\}$, the beam minus the current parent). This is analogous to crossover in evolutionary algorithms.

What it computes: the optimizer analyzes the failures and strengths of two different multimodal prompts, identifies which features of each contribute to good performance, and generates a description of a new non-textual prompt that combines those features. For example, if one reference image excels at showing beak shapes but has poor lighting, and another excels at color representation but lacks structural clarity, the mix condition might specify "retain the anatomical labeling and clarity from Image A, but adopt the color palette and lighting conditions from Image B." The generator then produces a new image that synthesizes these specifications.

Why this operator: mix "blends the complementary strengths of multiple multimodal prompts" and "avoids over-reliance on a single candidate, enabling exploration of intermediate solutions better than individual ones." This operator is particularly important when the optimization has converged to a set of prompts that each excel at different aspects of the task — one prompt handles edge cases well but is weak on common cases, another is strong on common cases but fails on rare ones. Mix can synthesize a new prompt that inherits both strengths.

Implementation detail on the mix operator parent selection: for the generation and edit operators, one parent prompt is randomly selected from the top-b prompts of the previous iteration to produce one child prompt ($p \rightarrow p'$). For the mix operator, $K$ parent prompts are selected to produce one child ($\{p_1, ..., p_K\} \rightarrow p'$). At each iteration (except the first), $b^2$ child prompts are produced by "evenly applying the generation, edit, and mix operators." With beam size $b = 3$, this means $b^2 = 9$ child prompts per iteration, with roughly 3 from generation, 3 from edit, and 3 from mix.

Prior-Inherited Bayesian UCB: Efficient Candidate Selection

The second major component of MPO addresses the candidate selection challenge: given a pool of newly generated child prompts, how to efficiently allocate a limited evaluation budget to identify the highest-performing ones for the next iteration. The solution is a Bayesian bandit strategy that warm-starts child evaluations using parent performance as an informative prior.

The Cold-Start Problem in Multimodal Prompt Selection

Existing prompt optimization methods typically use one of two selection strategies: (a) uniform allocation, where each candidate receives an equal share of the evaluation budget (e.g., evaluate all 9 child prompts on 11 examples each if the budget is 100), or (b) standard bandit algorithms like UCB, which adaptively allocate budget based on observed performance but initialize each new candidate with an uninformative prior (e.g., Beta(1,1), representing no prior knowledge).

The paper argues that both approaches become inefficient in the multimodal setting because "high-quality prompts become relatively sparse, and a large portion of the evaluation budget risks being wasted on low-potential candidates." This is a statement about the geometry of the multimodal search space: most multimodal prompts (random text + random image) are bad — they provide conflicting or irrelevant signals. Consequently, uniform allocation evaluates many inevitably-poor candidates, and standard UCB must spend budget discovering that each new candidate is poor before abandoning it (the cold-start problem: every new arm starts from scratch).

Empirical Motivation: Parent-Child Performance Correlation

The paper's key empirical insight is that parent prompt performance strongly predicts child prompt performance. They "analyze the optimization trajectory, measuring the correlation between the performance of parent prompts and the average performance of their children" (Figure 3). The reported statistics are:

  • Pearson correlation coefficient: $r = 0.88$
  • R-squared: $R^2 = 0.78$
  • P-value: $3.33 \times 10^{-170}$

What this means: good parent prompts tend to produce good children; poor parent prompts tend to produce poor children. This is not a trivial observation — it could have been that prompt optimization steps are essentially random perturbations, in which case parent performance would carry no information about child performance. The strong correlation validates the beam search structure (keeping the best parents and mutating them is better than random restarts) and, more importantly, suggests that parent performance can serve as a prior for estimating child performance before any child-specific evaluations are conducted.

Why this matters for selection: if we know a child was generated from a parent that performed poorly (e.g., 30% accuracy), we should be skeptical that this child will perform well, and we can allocate less evaluation budget to it. Conversely, if a child was generated from a parent that performed well (e.g., 80% accuracy), we should be optimistic and invest more budget to confirm and refine our estimate. This is the intuition that Prior-Inherited Bayesian UCB operationalizes.

Bayesian Modeling with Informative Priors

MPO models the expected score of each multimodal prompt $p_i$ as a Beta distribution:

Beta(αi,βi)\text{Beta}(\alpha_i, \beta_i)

where $\alpha_i$ and $\beta_i$ are the pseudo-counts of successful and failed evaluations, respectively.

What this distribution represents: the Beta distribution is the conjugate prior for the Bernoulli distribution, which is the appropriate model when the outcome of each evaluation is binary (correct/incorrect for classification, or more generally success/failure). The parameters $\alpha_i$ and $\beta_i$ can be interpreted as: after observing $\alpha_i + \beta_i$ evaluations, the model believes the prompt's true success probability is distributed according to Beta($\alpha_i, \beta_i$). The posterior mean $\hat{\mu}_i = \alpha_i / (\alpha_i + \beta_i)$ is the expected success rate given current evidence.

Why Beta over other distributions: Beta is the standard choice for modeling probabilities because (a) it is the conjugate prior for Bernoulli and binomial likelihoods, making Bayesian updates trivial (just add successes to $\alpha$ and failures to $\beta$), and (b) it is defined on $[0,1]$, matching the domain of accuracy/F1 scores. Alternatives like Gaussian distributions would have support outside $[0,1]$ and require more complex updates.

Prior inheritance mechanism. When a child prompt $p_i$ is generated from a parent prompt $p_{\text{par}(i)}$, its Beta prior is initialized proportionally to the parent's posterior mean performance:

αi=μ^par(i)⋅S+1,βi=(1−μ^par(i))⋅S+1\alpha_i = \hat{\mu}_{\text{par}(i)} \cdot S + 1, \quad \beta_i = (1 - \hat{\mu}_{\text{par}(i)}) \cdot S + 1

where $\hat{\mu}_{\text{par}(i)} = \alpha_{\text{par}(i)} / (\alpha_{\text{par}(i)} + \beta_{\text{par}(i)})$ is the parent's posterior mean (its estimated success probability based on all evaluations the parent has received), and $S > 0$ is the prior strength hyperparameter controlling how many pseudo-observations are inherited.

What it computes: before any child-specific evaluations occur, the child's Beta distribution is already centered around the parent's posterior mean, with $S$ determining the weight of this prior information. Specifically, the child starts with $S + 2$ pseudo-observations: $S$ of them are distributed between successes and failures according to the parent's mean (e.g., if the parent has $\hat{\mu} = 0.8$ and $S = 10$, the child starts with $0.8 \times 10 = 8$ pseudo-successes and $0.2 \times 10 = 2$ pseudo-failures), plus 1 additional count in each of $\alpha$ and $\beta$ from the standard $+1$ Laplace smoothing (which ensures the prior is well-defined even if $\hat{\mu} = 0$ or $\hat{\mu} = 1$).

Why this form: the $+1$ terms in both $\alpha_i$ and $\beta_i$ correspond to a Beta(1,1) uniform prior as the base. The $\hat{\mu}_{\text{par}(i)} \cdot S$ term shifts the prior mean away from 0.5 (uniform) toward the parent's estimated performance, while $S$ controls how strongly the prior is anchored there. This is a principled Bayesian approach: the prior represents our belief about the child's performance before seeing any data, and that belief is informed by the parent's performance (which Figure 3 shows is predictive).

For the mix operator (multiple parents): if a child is generated by the mix operator (combining $K$ parent prompts), the prior is computed using the average posterior mean of the multiple parent prompts: $\hat{\mu}_{\text{par}(i)} = \frac{1}{K} \sum_{k=1}^K \hat{\mu}_{\text{par}_k(i)}$. This is a conservative choice — the mixed child inherits a prior that is the average of its parents' performances, reflecting uncertainty about which parent's traits will dominate.

Hyperparameter S: the prior strength $S$ is set to 10% of the evaluation budget per prompt. With a default evaluation budget of 100 per candidate, $S = 10$. This means each child starts with 10 pseudo-observations inherited from its parent. The paper analyzes sensitivity to $S$ (Figure 9), showing that small $S$ under-utilizes the parent prior (leading to suboptimal performance similar to standard UCB), while excessively large $S$ causes over-reliance on the parent prior (the model cannot adapt when a child genuinely outperforms its parent). Optimal performance is achieved at intermediate $S$.

UCB Selection and Iterative Evaluation

With priors initialized, MPO uses a Bayesian Upper Confidence Bound (UCB) strategy to iteratively select which candidate to evaluate next. The procedure runs for a fixed number of rounds until the evaluation budget is exhausted.

UCB score computation. At each round $t$, the prompt with the highest upper quantile of its Beta posterior is selected:

j=arg⁡max⁡i∈{1,...,k}BetaQuantile(qt;αi,βi)j = \arg\max_{i \in \{1, ..., k\}} \text{BetaQuantile}(q_t; \alpha_i, \beta_i)

where $q_t = 1 - \frac{1}{t(\log N)^c}$ is the quantile level that decreases over time (shrinking from exploration toward exploitation), $N$ is the total evaluation budget, and $c$ is an exploration parameter (set to $c = 1$ in the implementation based on the algorithm in Appendix A.5).

What it computes: the function BetaQuantile($q_t; \alpha_i, \beta_i$) returns the $q_t$-th quantile of the Beta distribution with parameters $\alpha_i, \beta_i$. This is an optimistic estimate of the prompt's true performance — it assumes the true mean is at the upper end of what the current evidence supports. The quantile level $q_t$ starts high (e.g., 0.95), encouraging exploration by being optimistic about uncertain prompts, and gradually decreases, shifting toward exploitation of prompts with established high performance.

Why UCB over alternatives: UCB naturally balances exploration and exploitation through the quantile mechanism. A prompt with few evaluations (small $\alpha_i + \beta_i$) has a wide Beta distribution, so its upper quantile can be very high even if its posterior mean is moderate — this encourages exploring uncertain prompts. A prompt with many evaluations and a high posterior mean has a narrow distribution, so its upper quantile is close to its mean — this exploits known good prompts. Uniform allocation (evaluating all candidates equally) wastes budget on certainly-poor prompts. Greedy selection (always picking the highest posterior mean) would prematurely converge to a prompt that happened to perform well on early evaluations due to noise.

Evaluation and update. After selecting prompt $j$, the system evaluates it on a small batch of data:

st=E(q,a)∼Dmini[f(MLLM(t,m,q),a)]s_t = \mathbb{E}_{(q,a) \sim D_{\text{mini}}} \left[ f(\text{MLLM}(t, m, q), a) \right]

where $D_{\text{mini}}$ is a mini-batch of $B$ examples sampled from the training dataset, and $s_t$ is the average task score on this mini-batch.

The Beta posterior is then updated:

αj←αj+st⋅B,βj←βj+(1−st)⋅B\alpha_j \leftarrow \alpha_j + s_t \cdot B, \quad \beta_j \leftarrow \beta_j + (1 - s_t) \cdot B

What it computes: the $s_t \cdot B$ term represents the effective number of successes in the mini-batch (e.g., if accuracy on 10 examples is 0.7, we add 7 pseudo-successes), and $(1 - s_t) \cdot B$ is the effective number of failures (3 pseudo-failures in this example). This is the standard conjugate update for a Beta-Bernoulli model.

Why batched evaluation: evaluating on a single example per round would require many rounds to accumulate evidence, increasing the overhead of model calls. Batched evaluation ($B$ examples per round) amortizes the overhead while still allowing adaptive allocation across rounds. The batch size $B$ is not explicitly specified in the paper but is implied by the total budget $N$ and the number of rounds $N/B$.

Selection after budget exhaustion. Once the total evaluation budget $N$ is exhausted, the prompts are ranked by their posterior means $\hat{\mu}_i = \alpha_i / (\alpha_i + \beta_i)$, and the top-$b$ prompts are selected as parents for the next optimization iteration. The entire procedure is detailed in Algorithm 2 of Appendix A.5.

Theoretical Guarantee: Proposition 3.1

The paper provides a formal proposition that the prior-inherited mechanism reduces best-arm identification cost under an average KL-closeness assumption.

Assumption (Appendix B, Equation 3). Let $I$ be the population of child arms produced during optimization. The parent estimate is, on average, KL-closer to the true child performance than the mean of the uninformative prior:

Ei∼I[d(μi,μ^par(i))−d(μi,12)]≤−γ\mathbb{E}_{i \sim I} \left[ d(\mu_i, \hat{\mu}_{\text{par}(i)}) - d(\mu_i, \tfrac{1}{2}) \right] \leq -\gamma

for some $\gamma > 0$, where $d(p, q) = p\log(p/q) + (1-p)\log((1-p)/(1-q))$ is the Bernoulli KL divergence, $\mu_i$ is the true (unknown) mean performance of child $i$, $\hat{\mu}_{\text{par}(i)}$ is the parent's posterior mean, and $1/2$ is the mean of the uniform Beta(1,1) prior.

What this means in plain terms: on average, the parent's estimated performance is closer to the child's true performance (in KL divergence) than the uninformed guess of 0.5. This is a formalized version of the empirical observation (Figure 3) that parent performance correlates with child performance.

Proposition 3.1 (paraphrased): with the prior defined in Equation 1 (prior inheritance) and under the assumption that the parent prior is more informative than uniform, the best-arm identification cost of Bayesian UCB is non-increasing — meaning it takes no more evaluations (and potentially fewer) to identify the highest-performing child compared to using an uninformative prior.

Proof sketch (from Appendix B). The proof identifies two synergistic mechanisms: (1) tighter credible intervals at fixed sample counts (the prior strength $S$ acts as additive effective sample size, meaning confidence intervals are narrower for the same amount of real data), and (2) more efficient sample allocation (the informative prior biases the UCB toward exploring promising arms and away from certainly-poor arms, leading to more pulls of the optimal arm and fewer wasted on suboptimal ones). Both mechanisms follow from the KL-closeness assumption and standard Beta-Bernoulli concentration bounds.

Why this matters: the guarantee provides theoretical justification for why prior inheritance should help — it is not merely a heuristic but a principled Bayesian mechanism that reduces evaluation waste. In the multimodal setting where evaluation is expensive (each batch requires running the MLLM), this efficiency gain is critical for the framework to be practical. The empirical validation (Figure 5) confirms the theoretical prediction: MPO achieves the same performance as uniform allocation with 70% less budget.

Iterative Optimization Algorithm

The complete MPO algorithm integrates alignment-preserving exploration and prior-inherited selection within a beam search framework. The paper provides Algorithm 1 in Appendix A.5, which specifies the exact procedure.

Initialization (iteration 0). The algorithm starts with a single textual prompt $(t_0, \emptyset)$ — no non-textual prompt exists yet. The initial prompt is evaluated on the training dataset to obtain its performance $\hat{\mu}$. Then, $b^2$ multimodal prompts are generated using only the generation operator (since no non-textual prompts exist to edit or mix). Specifically, for $i = 1$ to $b^2$: the optimizer analyzes failures of the current prompt, generates unified feedback, produces a generation condition $c_{\text{gen}}$ and an updated text $t'$, and the generator creates $m' = g(c_{\text{gen}}, \emptyset)$. All $b^2$ child prompts are added to the candidate pool $C$.

The Bayesian UCB selection (Algorithm 2) is then invoked to select the top-b prompts from the pool, which become the initial beam $P$ for the main optimization loop.

Why $b^2$ initial candidates: with beam size $b = 3$, this generates $b^2 = 9$ diverse initial multimodal prompts. Since all are generated from scratch (no editing or mixing), they provide broad coverage of the multimodal space to start the beam with diverse exemplars. The subsequent beam search then refines from these starting points.

Main optimization loop (iterations 1 to T). For each of $T = 13$ iterations (as specified in Section 4.1: "the number of iterations of $T = 13$"):

  1. Candidate generation: for each parent prompt $p = (t, m)$ in the beam $P$ (size $b$), generate $b$ child prompts. For each child:

    • Identify the failure set $F_p$ on the training data.
    • Generate unified feedback $\nabla p$ via cohesive backpropagation.
    • Randomly sample an operator from $\{\text{generation, edit, mix}\}$.
    • Apply the selected operator to produce $(t', m')$ via joint multimodal update.
    • Add the child to the candidate pool $C$.

    This produces $b \times b = b^2 = 9$ child prompts per iteration (3 parents $\times$ 3 children per parent).

  2. Candidate selection: invoke Prior-Inherited Bayesian UCB (Algorithm 2) on the combined pool of parent and child prompts $P \cup C$ (size $b + b^2$). Each child receives a prior inherited from its parent(s). The selection algorithm allocates the evaluation budget $N$ (default: 100 evaluations per prompt) across rounds, adaptively choosing which prompt to evaluate next. After budget exhaustion, the top-b prompts (by posterior mean) become the new beam $P$.

  3. Iteration increment: proceed to the next iteration with the updated beam.

Why beam search: the beam search structure (maintaining top-b prompts and generating $b^2$ candidates per iteration) balances exploration breadth with computational cost. A full breadth-first search over the operator space would be exponentially expensive. Random search (generate many independent prompts) would fail to exploit the parent-child correlation. Beam search provides a middle ground: it greedily keeps the best candidates while generating enough diversity ($b^2$ new candidates per iteration) to escape local optima. The paper's training dynamics analysis (Figure 7) shows that MPO continues improving well beyond the point where ProTeGi plateaus, suggesting the multimodal exploration with beam search indeed helps escape local optima that trap text-only methods.

Final selection. After $T = 13$ iterations, the prompt with the highest posterior mean among the final beam is returned as the optimized multimodal prompt $p^* = (t^*, m^*)$.

Key hyperparameters:

  • Beam size $b = 3$
  • Iterations $T = 13$
  • Evaluation budget per prompt $N = 100$ (reduced to one-third of available training instances for CUB subtasks with fewer than 100 examples)
  • Prior strength $S = 10$ (10% of evaluation budget)
  • Failure set size: 3 examples
  • Batch size $B$: implicitly $N / (\text{number of rounds})$
  • Exploration parameter $c = 1$
  • Optimizer model temperature: 0.7 (to encourage diversity in prompt generation)
  • Base model temperature: 0 (for consistent evaluation)

Modality-Specific Considerations

The paper tailors the MPO framework to each modality with specific design choices.

Image modality (PlantVillage, CUB, SLAKE, DrivingVQA, RSVQA). The modality-specific generator is GPT-Image (OpenAI, 2025). The full meta-prompt suite for the image modality is provided in Appendix A.4, showing how the optimizer is instructed to analyze image-specific weaknesses (clarity, composition, labeling of visual features) and generate corresponding image generation, editing, and mixing instructions. All three operators (generation, edit, mix) are used.

Video modality (Drive&Act, VANE-Bench). The modality-specific generator is Wan2.1 (1.3B parameters), a lightweight open-source text-to-video model. Three implementation adaptations are made:

  • When a video query is part of the failure set, three representative frames (first, middle, last) are sampled from the query for analysis, since processing full videos in the optimizer context would be expensive.
  • Due to "the high complexity of video editing and mixing," only the generation operator is used for video prompt optimization. This is a practical constraint: current video generation and editing tools are less mature than image tools, making fine-grained video editing and mixing unreliable.
  • Generated videos are 5 seconds at 16 fps, then downsampled to 5 frames at 1 fps to construct the video prompt. This temporal subsampling reduces the computational cost of processing the video prompt while retaining key temporal information.

Molecular modality (Absorption, BBBP, CYP Inhibition). Molecules are represented using the 1D SMILES (Simplified Molecular Input Line Entry System) notation, which is a text string encoding molecular structure (e.g., "CCO" for ethanol). The modality-specific generator is GPT-4o mini itself — since SMILES is a textual format, the optimizer can directly generate and manipulate molecular structures without an external image/video generator. All three operators (generation, edit, mix) are used, with the "edit" and "mix" operations applied to SMILES strings. For molecular tasks, F1 score is used as the optimization objective instead of accuracy to handle class imbalance.

Evaluation metric by modality. For image and video tasks, accuracy is used. For molecular tasks, F1 score is used due to class imbalance in the datasets. For answer correctness determination, task-specific criteria are applied: the final predefined label is extracted for standard classification, strict formatting rules are applied for binary and closed-ended QA tasks, and exact match is used for open-ended QA tasks.

Cross-validation protocol. The best-performing prompt on the training set (selected by performance during the optimization process) is evaluated on the test set. Results are averaged over three independent runs to account for stochasticity in the optimization process (different random seeds affect operator sampling, prompt generation, and evaluation batch sampling).

4. Key Insights and Innovations

Innovation 1: Reframing Prompt Optimization as an Inherently Multimodal Search Problem

The paper's most fundamental conceptual move is not a specific algorithm but a redefinition of what "prompt optimization" means in the era of MLLMs. Prior to this work, the field of Automatic Prompt Optimization (APO) had implicitly accepted that prompts are textual — the search space, the optimization operators, and the evaluation all operated in the space of token sequences. This was not an active limitation that researchers chose; it was a boundary inherited from the LLM era that persisted even as models gained multimodal capabilities. The paper's framing innovation is to argue that this boundary is artificial and costly: if the model can process images, videos, and molecules as part of its input, then the prompt itself can and should include these modalities, and the optimization process should discover the best multimodal combination.

This reframing matters because it changes the optimization objective from a single-modality problem to a cross-modal coordination problem. In text-only APO, the optimizer asks "what words should I prepend to the query?" In multimodal APO, the optimizer asks "what combination of text and non-textual reference simultaneously provides the clearest instruction, the most informative demonstrations, and the least ambiguity?" This is not merely "text optimization plus image selection" — it is a fundamentally different search problem because the value of a textual instruction depends on which non-textual prompt accompanies it, and vice versa. A text prompt that says "classify this bird by comparing beak shapes" is excellent with a reference image showing four species with labeled beaks, but nearly useless with a reference image of a landscape. The paper formalizes this interdependence through the joint optimization over the product space T × M (Section 3.1), which distinguishes multimodal prompt optimization from the sequential, independent optimization of text and non-text that a naive extension would attempt.

Prior work that touched on multimodal prompting — instance-specific methods like MM-CoT (Zhang et al., 2024b) or visual prompting with bounding boxes (Zhou et al., 2024; Jiang et al., 2024) — optimized per query, not per task. The paper's framing establishes task-level multimodal prompt optimization as a distinct problem with different requirements: the discovered prompt must generalize across a distribution of queries, which demands robustness that per-query methods do not need. This distinction is conceptually analogous to the difference between few-shot learning (optimizing per-instance adaptation) and meta-learning (optimizing a single set of initial parameters that work across a task distribution).

The empirical evidence that validates this reframing is the consistent and substantial gap between text-only optimized prompts and MPO's multimodal prompts — for example, the 11.4 percentage point improvement over the best text-only baseline on the average across all benchmarks (MPO achieves 65.1% vs. ProTeGi's 60.0% in Table 1). More tellingly, the ablation on modality contributions (Table 3) shows that the optimized text prompt alone achieves only 55.6% on PlantVillage while the full multimodal prompt achieves 76.4%, demonstrating that the text and image components are "not merely additive but mutually reinforcing" — a direct consequence of the cross-modal coordination that the joint optimization enables.

This reframing is fundamental rather than incremental because it expands the definition of the problem space rather than improving performance within an existing space. It opens an entirely new axis of optimization that the APO literature had not previously considered, and it establishes the vocabulary — multimodal prompt, cross-modal consistency, alignment-preserving exploration — for a new subfield.

Innovation 2: Cross-Modal Alignment as an Optimization Constraint (Not Just an Evaluation Criterion)

The paper's second conceptual contribution is identifying cross-modal alignment as the central challenge in multimodal prompt optimization and designing the exploration mechanism around maintaining it. Prior work in multimodal learning has studied cross-modal alignment extensively — in vision-language pre-training (CLIP, BLIP), in text-to-image generation evaluation (DSG, VQA-based alignment metrics), and in multimodal representation learning. However, alignment has been treated primarily as a property to be measured or a training objective for model parameters. This paper is the first to treat alignment as an active constraint during the prompt optimization process itself — the optimizer must not only find a high-performing prompt but must do so without allowing the text and non-textual components to drift into contradiction.

The diagnostic insight is that a naive extension of text-only APO to multimodal prompts — optimizing text first, then generating a matching image, or updating both independently — is not just suboptimal but actively harmful because it produces prompts where one modality undermines the other. The paper demonstrates this through the cross-modal alignment analysis (Figure 4), which compares MPO's joint optimization against sequential optimization, random image substitution, in-distribution image queries, and out-of-distribution image queries. The result is a clear gradient: MPO achieves the highest alignment (measured by DSG score) and the highest performance gain; sequential optimization falls in the middle; and independent or mismatched image-text pairs perform worst. This pattern isolates alignment as the causal mechanism — the better the text and non-textual components cohere, the larger the performance improvement.

What makes this an innovation rather than an obvious observation is that the field's default assumption was that prompt quality decomposes additively: better text + better image = better prompt. The paper shows this is false. A good text prompt paired with the "wrong" good image (one that was optimized for a different prompt or task) performs worse than a mediocre text prompt with a properly aligned image. This implies that the cross-modal interaction term dominates the additive components, which is a non-trivial empirical finding with direct consequences for how optimization must proceed. The cohesive backpropagation mechanism — generating a single unified feedback that covers both modalities — is the algorithmic embodiment of this insight: by forcing text and non-textual improvements to be derived from the same diagnostic reasoning, MPO prevents the independent drift that causes misalignment.

This contribution is fundamental as a concept but incremental as an implementation: the mechanism (cohesive backpropagation) is a specific instance of generating joint feedback, but the conceptual move — treating cross-modal alignment as the primary constraint in multimodal prompt search — changes how one thinks about designing optimization algorithms for this space. It implies that future multimodal prompt optimizers will need some form of alignment guarantee, whether through joint feedback, contrastive objectives, or explicit alignment metrics integrated into the search process.

Innovation 3: Inheritance as an Alternative to Cold-Start Evaluation in Sparse Search Spaces

The paper's third key insight concerns the biology of the optimization process itself: parent-child performance correlation as a reliable structural property that can be exploited to drastically reduce evaluation costs. This is not the first work to use Bayesian bandit methods for prompt selection — prior work has applied UCB (Ashizawa et al., 2025) and pure-exploration bandits (Shi et al., 2024; Li et al., 2026) to prompt optimization. However, these prior applications treat each new candidate prompt as an independent arm with no prior information, initializing from a uniform Beta(1,1) distribution. The paper's diagnostic move is to recognize that this independence assumption is both wasteful and empirically false in the multimodal setting: children inherit properties from their parents, and parent performance carries substantial information about child potential.

The evidence for this correlation (Figure 3: Pearson's r = 0.88, R² = 0.78) is central to the innovation. The authors do not merely observe this correlation and exploit it — they argue that it is a structural feature of beam search-based prompt optimization that arises because the optimization operators (mutation, crossover) produce gradual rather than random changes. This is important because it means the correlation is not an accident of the specific implementation but a general property that any beam search or evolutionary prompt optimizer can expect to observe. The correlation magnitude (0.88) is high enough that prior inheritance provides substantial efficiency gains, but not so high that it would make child evaluation unnecessary (which would be the case if r ≈ 1.0).

The theoretical proposition (Proposition 3.1) formalizes when and why this inheritance helps. The key condition — that the parent prior is KL-closer to the true child performance than the uniform prior — is both intuitive (good parents tend to produce good children) and empirically verifiable (Figure 3). The proposition's guarantee that best-arm identification cost is non-increasing under this condition provides formal justification for what would otherwise be a heuristic. This is a meaningful theoretical contribution because it connects the structural properties of the optimization process (parent-child correlation) to the efficiency properties of the selection algorithm (bandit regret), establishing a bridge between prompt optimization and bandit theory that prior work had not drawn.

The practical significance is substantial: MPO achieves the same performance with 70% less evaluation budget than uniform allocation and 42% less than a prior-free Bayesian UCB baseline (Figure 5). In absolute terms, this means the optimizer can explore more candidates within the same total compute budget, or achieve the same performance with less compute. For multimodal prompt optimization — where each evaluation requires running the full MLLM on a batch of task examples — this efficiency gain is what makes the approach practical at scale. Without it, the enlarged multimodal search space would be prohibitively expensive to explore.

This contribution is incremental in mechanism but fundamental in framing: Bayesian UCB with an informative prior is not a new algorithm, but the insight that the optimization trajectory itself provides the informative prior (through parent-child relationships) transforms prompt selection from an independent-arm problem to a transfer-learning problem. The paper's emphasis on measuring and validating the correlation before building a mechanism around it — rather than assuming it — is methodologically instructive for future work.

Innovation 4: Verifier-Free Multimodal Optimization Through Generators as Executional Backends

The fourth innovation is architectural rather than algorithmic: the paper's decoupling of prompt design from prompt synthesis through an external modality-specific generator. In principle, one could imagine multimodal prompt optimization where the optimizer MLLM directly outputs images (if it has image generation capabilities) or where a single unified model handles both optimization and non-textual generation. The paper instead introduces a three-party architecture: the optimizer LLM (GPT-4o mini) designs prompts in a textual meta-language, the modality-specific generator (GPT-Image, Wan2.1, GPT-4o mini for molecules) materializes these specifications into actual non-textual prompts, and the base MLLM evaluates the resulting prompts. This architecture is conceptually significant because it separates three concerns that would otherwise be conflated: understanding what makes a good prompt (the optimizer's role), creating valid non-textual content (the generator's role), and evaluating task performance (the base model's role).

What makes this an innovation rather than an engineering convenience is that it removes the need for a multimodal verifier — a component that prior work on verifier-guided test-time compute (e.g., process reward models in the LLM scaling literature) treats as essential. In text-only APO, the optimizer generates candidate text prompts, which are evaluated by running the base model and checking outputs against ground truth. The candidate generation and the evaluation use the same modality (text) and the same model family (LLMs). In multimodal APO, a naive approach might require a verifier that can assess the quality of images or videos as prompts — is this reference image good for bird classification? — which would be a much harder learned function. The paper sidesteps this entirely by treating the generator as a black-box translator from textual specifications to non-textual outputs and using only the base model's task performance as the fitness signal. The optimizer never needs to "see" or "judge" the non-textual prompt directly; it only needs to specify what should be generated and observe whether the resulting multimodal prompt improves task performance.

This architecture also enables generator flexibility — the paper demonstrates that MPO works with GPT-Image (a large proprietary model), SANA1.5 (a lightweight 1.6B open-source model), and even GPT-4o mini itself (for molecular SMILES generation). The performance gap between these generators (Table 2, Bottom Right: 76.4% for GPT-Image vs. 71.8% for SANA1.5 on PlantVillage) shows that better generators improve results, but the framework degrades gracefully with weaker generators — SANA1.5 still outperforms the text-only baseline (69.0%). This modularity is practically important because it means the framework can improve as generation technology improves, without requiring changes to the optimizer or selection components.

The contribution is architecturally fundamental, establishing a pattern — "optimizer designs, generator executes, base model evaluates" — that generalizes across modalities and generator implementations. It is analogous to the proposer-verifier decomposition in test-time compute scaling, but with the verifier replaced by the base model's task performance and the proposer split into a designer (optimizer) and an executor (generator). This tripartite architecture is likely to be reusable for future multimodal optimization problems beyond prompt discovery.

Innovation 5: The Operators as a Taxonomy of Multimodal Exploration Behaviors

The fifth innovation is the paper's systematization of multimodal prompt exploration into a small set of complementary operators (generation, edit, mix) that collectively provide broad, deep, and combinatorial coverage of the search space. This is not the first work to use operator-based exploration in prompt optimization — EvoPrompt (Guo et al., 2024) used mutation and crossover operators, and ProTeGi (Pryzant et al., 2023) used paraphrase and gradient-based editing. However, these prior operators were defined over the textual space (paraphrase = semantic edit of text, crossover = recombining text fragments). The paper's contribution is to redefine what these operators mean for non-textual modalities and to show that the same three operations — create from scratch, incrementally refine, blend multiple sources — map naturally across images, videos, and molecules, each with modality-specific semantics.

The innovation is the abstraction, not the operators themselves. By defining generation, edit, and mix as abstract operations on multimodal prompts — each taking textual conditions and (optionally) existing non-textual prompts as input, and producing new non-textual prompts as output — the paper creates a common vocabulary for multimodal search that is independent of the specific modality. A "generation" operation on images (creating a new reference image from scratch) has the same structural role as a "generation" operation on molecules (creating a new SMILES string from scratch): both explore entirely new regions of their respective spaces. An "edit" operation on videos (modifying a specific scene or frame) has the same structural role as an "edit" operation on images (adjusting lighting or adding labels): both perform local refinement. This abstraction means that the beam search controller — which orchestrates the overall optimization loop, decides how many candidates of each type to generate, and manages the parent-child relationships — does not need to know anything about the modality. It operates entirely in terms of the three abstract operators.

The evidence that this taxonomy is sound comes from the ablation study (Table 4), which shows that each operator individually outperforms the text-only baseline, but their combination achieves the best performance. This demonstrates that the operators are indeed complementary — generation provides diversity that edit cannot (since edit is constrained to stay near the parent), edit provides precision that generation cannot (since generation starts from scratch), and mix provides recombination that neither generation nor edit can achieve alone. The qualitative analysis (Figure 6 for images, Figure 14 and Table 8 for molecules) further illustrates this complementarity with concrete examples: generation introduces novel visual compositions (new grid layouts of bird species), edit fine-tunes local features (adding labels, improving resolution), and mix blends broader attributes (combining the anatomical clarity of one reference with the color accuracy of another).

This contribution is taxonomic rather than algorithmic: it organizes the space of possible exploration behaviors into a minimal set of operations that are simultaneously expressive enough to cover the search space and abstract enough to apply across modalities. It provides a template for future multimodal optimization frameworks — define how to generate, edit, and mix for the target modality, and the same overall search strategy applies. The paper does not claim these three operators are exhaustive or optimal, but the empirical evidence that they work across three qualitatively different modalities (images, videos, molecules) suggests they capture something fundamental about how to explore structured multimodal spaces.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper evaluates MPO across 10 datasets spanning three modalities — images, videos, and molecules. For the image modality, five datasets are used: PlantVillage (Mohanty et al., 2016) for diseased leaf identification (38 disease categories across 14 crop species, with custom subtasks constructed by selecting four crop species each having at least three distinct classes, split 50/50 for training/testing), CUB-200-2011 (Wah et al., 2011) for fine-grained bird classification (grouped by family name into 12 subtasks of 3–4 visually similar species, split 50/50 with at least 80 instances per split), SLAKE (Liu et al., 2021) for medical visual question answering across CT, MRI, and X-Ray modalities, DrivingVQA (Corbière et al., 2025) with 2,039 training and 521 test instances of closed-ended driving scenario questions (filtered to retain only instances with a single correct answer), and RSVQA (Lobry et al., 2020) for open-ended remote sensing VQA using the low-resolution image set. For the video modality, two datasets are used: Drive&Act (Martin et al., 2019) for driver action recognition with 6,642 training and 2,222 test instances (videos preprocessed by sampling frames at 1 fps), and VANE-Bench (Gani et al., 2025) for video anomaly detection with 293 training and 263 test instances created using a 60/40 custom split from 10-frame video clips. For the molecular modality, three tasks from the Therapeutics Data Commons (TDC) benchmark (Huang et al., 2021) are used: Absorption (four subtasks: PAMPA, HIA, Pgp, Bioavailability), BBBP (Martins et al., 2012) for blood-brain barrier penetration prediction with 1,453 train and 382 test examples, and CYP Inhibition (Veith et al., 2009) spanning five cytochrome P450 enzyme inhibition subtasks, all using official random splits. For image and video tasks, 300 test examples are sampled; for molecular tasks, the entire test set is used.

  • Base model(s). For image and video tasks, the base MLLM is Qwen2.5-VL (7B) (Bai et al., 2025). For molecular tasks, the base model is Qwen3 (8B) (Yang et al., 2025), a text-only LLM since molecular inputs are represented as text strings (SMILES). The choice of Qwen2.5-VL (7B) is motivated by it being "representative of the capabilities of many contemporary LLMs" in the multimodal space, sitting in a regime where base performance is non-trivial but far from saturation — for example, human-designed prompts achieve only 42.2% on PlantVillage and 47.9% on CUB (Table 1), leaving substantial room for prompt optimization to improve performance. The paper also validates generalizability by testing with larger variants (Qwen2.5-VL 72B) and alternative architectures (Gemma3 12B, InternVL-3.5 14B) in robustness experiments (Table 2, Top), showing that MPO's gains hold across model scales and families. The base model is always used at temperature 0 for consistent evaluation, while the optimizer model (GPT-4o mini) operates at temperature 0.7 to encourage diverse candidate generation.

  • Metrics. The primary optimization metric varies by modality: accuracy is used for image and video tasks (classification and question answering), while F1 score is used for molecular tasks to handle class imbalance inherent in drug property prediction datasets. The metric $f$ in the optimization objective (Equation in Section 3.1) is therefore task-specific. For answer correctness determination, task-specific evaluation criteria are applied: the final predefined label is extracted for standard classification tasks, strict formatting rules are applied for binary and closed-ended QA tasks, and exact match is used for open-ended QA tasks. All main results are reported as the average accuracy (or F1 for molecular tasks) over three independent optimization runs, with performance measured on the held-out test set after selecting the best-performing prompt on the training set.

  • Baselines. The paper benchmarks MPO against three categories of methods: Manual prompting baselines include Human (a simple handcrafted prompt designed by the researchers for each task), Chain-of-Thought (CoT) (Wei et al., 2022) which appends "Let's think step by step" to the human prompt, and Few-Shot (1-Shot, 3-Shot, 5-Shot) which prepends in-context examples drawn from the training data to the human prompt. Text-only automatic prompt optimization baselines include six representative methods: APE (Zhou et al., 2023), which generates candidate prompts by reverse-engineering instructions from examples and paraphrasing; OPRO (Yang et al., 2024), which treats the LLM as an optimizer guided by prompt-performance pairs; EvoPrompt (Guo et al., 2024), which uses evolutionary operators (mutation, crossover) on a population of text prompts; PE2 (Ye et al., 2024), which optimizes the meta-prompt used to steer the LLM optimizer through structured templates; ProTeGi (Pryzant et al., 2023), which simulates gradient descent by generating textual feedback ("gradients") from failures and editing prompts in the opposite semantic direction; and SEE (Cui et al., 2025b), which jointly optimizes text instructions and in-context examples through alternating explore-exploit phases. All baselines operate purely in the textual modality — they can only generate, refine, and select text strings as prompts.

  • Generation budget / compute accounting. The paper controls for computational cost by equalizing the number of prompts explored across all methods rather than directly measuring FLOPs. Specifically, all methods (APE, OPRO, EvoPrompt, PE2, ProTeGi, SEE, and MPO) explore the same total number of candidate prompts during optimization. For MPO, each candidate multimodal prompt is allocated an evaluation budget of 100 examples from the training set (reduced to one-third of available instances for CUB subtasks with fewer than 100 training samples). The prior strength hyperparameter $S = 10$ represents 10% of this evaluation budget, meaning each child prompt inherits 10 pseudo-observations from its parent before any real evaluations occur. For computational cost comparison, the paper reports model request counts (Table 5): all methods make the same number of base model calls (11.7k), while optimizer model calls vary — APE uses 117 calls (one-step exploration), ProTeGi uses 234 calls (two-step: feedback generation and refinement), SEE uses 153 calls (combining both approaches), and MPO uses 234 calls (two-step) plus 117 modality-specific generator calls. The paper notes that these additional generator calls are manageable because lightweight open-source generators like SANA1.5 (1.6B) can be used without sacrificing the performance advantage over text-only methods (validated in Table 2, Bottom Right).

  • Cross-validation / statistical protocol. The optimization loop operates on the training set to discover the best multimodal prompt, and the performance of this discovered prompt is evaluated on the held-out test set. This is the standard train/test split protocol for prompt optimization, and it avoids the circularity of selecting prompts based on test performance. All main results are averaged over three independent optimization runs with different random seeds, which affects the stochastic components of the optimization: operator sampling (generation, edit, or mix are chosen randomly for each child prompt), prompt generation by the optimizer LLM (at temperature 0.7), evaluation mini-batch sampling in the Bayesian UCB selection, and the initialization of the first iteration's candidate pool. The paper does not report confidence intervals or standard errors, reporting only the mean accuracy/F1 over the three runs. The test set sizes are: 300 examples for each image and video task (sampled from the full test set), and the full official test set for molecular tasks (382 examples for BBBP, variable sizes for Absorption and CYP Inhibition subtasks as defined by the TDC splits).

Main Quantitative Results

Overall Performance Across Modalities (Table 1)

The headline result is that MPO achieves an average accuracy of 65.1% across all 10 datasets, outperforming the best text-only automatic optimization method ProTeGi (60.0%) by 5.1 percentage points, and surpassing the strongest manual baseline 3-Shot (49.5%) by 15.6 percentage points. These numbers are reported as averages over three independent runs, and the per-dataset breakdown reveals several consistent patterns:

  • On image tasks, MPO achieves the largest absolute gains over text-only methods. On PlantVillage (38-class diseased leaf identification), MPO reaches 76.4% compared to SEE's 69.0% (a gain of 7.4 points) and ProTeGi's 64.4% (a gain of 12.0 points). On CUB-200-2011 (fine-grained bird classification across 12 subtasks), MPO reaches 78.6% versus SEE's 71.6% (7.0 points higher) and ProTeGi's 70.0% (8.6 points higher). On SLAKE (medical VQA across CT, MRI, X-Ray), the gains are more modest: MPO reaches 38.2% versus PE2's 35.8% (2.4 points higher), with all methods clustered in a relatively narrow band (34.3% to 38.2%), suggesting that medical image understanding is bottlenecked by the base model's capability rather than prompt quality. On DrivingVQA and RSVQA, MPO achieves 56.0% and 55.9% respectively, representing gains of 1.6–2.3 points over the best text-only baselines.

  • On video tasks, MPO shows particularly strong performance on Drive&Act (driver action recognition), reaching 58.3% versus ProTeGi's 53.0% (5.3 points higher). This is notable because MPO uses only the generation operator for video prompts (edit and mix are excluded due to the complexity of video manipulation), yet still achieves substantial gains, suggesting that even the ability to create task-relevant reference video clips from scratch provides significant value. On VANE-Bench (video anomaly detection VQA), MPO reaches 71.2% versus ProTeGi's 65.5% (5.7 points higher). The video results collectively show that MPO's gains are not restricted to static images but extend to dynamic temporal content.

  • On molecular tasks, MPO achieves the largest relative gains over text-only methods. On Absorption (HIA, PAMPA, Pgp, Bioavailability), MPO reaches an average F1 of 64.5% versus ProTeGi's 58.2% (6.3 points higher). On BBBP (blood-brain barrier penetration), MPO reaches an F1 of 67.6% versus ProTeGi's 65.7% (1.9 points higher). On CYP Inhibition (five cytochrome P450 enzymes), MPO reaches an average F1 of 60.2% versus ProTeGi's 57.0% (3.2 points higher). These gains are particularly meaningful because molecular property prediction is a high-stakes domain where small improvements can translate to significant resource savings in drug discovery pipelines. The performance on molecular tasks also validates that MPO's framework generalizes beyond visual modalities to structured scientific data.

Comparison to manual baselines. MPO's advantage over manual prompting is dramatic: Human prompts (handcrafted by researchers) achieve only 44.1% average accuracy; Chain-of-Thought ("Let's think step by step") achieves 40.8% (actually lower than Human on some tasks, such as SLAKE where CoT drops to 30.8% vs. Human's 35.2%); and few-shot prompting shows diminishing or inconsistent returns (1-Shot: 47.2%, 3-Shot: 49.5%, 5-Shot: 49.3%). The fact that 5-Shot underperforms 3-Shot on average (and performs worse than 3-Shot on 6 out of 10 datasets) suggests that additional in-context examples can introduce noise or conflict rather than providing consistent benefit — this is a known phenomenon in multimodal prompting that the paper's optimization-based approach circumvents entirely.

Comparison to text-only APO methods. Among text-only automatic methods, ProTeGi (60.0%) and SEE (59.1%) lead the pack, followed by PE2 (58.2%), with APE (51.3%), EvoPrompt (49.5%), and OPRO (46.9%) trailing significantly. The gap between ProTeGi/SEE and the weaker APO methods (~8–10 points) underscores that not all text-only optimization strategies are equally effective — methods that incorporate failure analysis and structured exploration (ProTeGi, SEE) substantially outperform those that rely on simple generation and scoring (APE, OPRO). MPO builds on the strongest baseline (ProTeGi's gradient-based approach) and extends it into the multimodal domain, achieving an additional 5.1 points of average improvement.

A fine-grained look at individual subtasks (Table 6 for images, Table 7 for molecules) reveals that MPO's gains are not uniform across all subsets — on some CUB subtasks (e.g., "cormorant" classification) MPO achieves only 58.4% versus ProTeGi's 53.5% (a 4.9-point gain), while on others (e.g., "auklet" classification) MPO reaches 73.7% versus ProTeGi's 42.4% (a staggering 31.3-point gain). This heterogeneity suggests that the value of multimodal prompts is task-dependent: for subtasks where visual features are easily describable in text (cormorants have distinctive shapes), text-only prompts suffice, but for subtasks where species differences are subtle and text descriptions are inherently ambiguous (auklets have nuanced crest and bill variations), reference images provide information that text fundamentally cannot convey.

Generalizability to Diverse Backbone Models (Table 2)

The paper validates that MPO's effectiveness is not tied to a specific base model, optimizer model, or modality-specific generator.

Base model generalizability (Table 2, Top). MPO is tested with four different base MLLMs on CUB: Qwen2.5-VL (72B), Gemma3 (12B), InternVL-3.5 (14B), and GPT-4.1 nano. Across all four models, MPO consistently outperforms both the best text-only baseline (ProTeGi or SEE) and manual prompting baselines. On Qwen2.5-VL (72B) — the largest model tested — MPO achieves 80.4% versus ProTeGi's 74.1% (6.3 points higher). On Gemma3 (12B), MPO achieves 73.1% versus SEE's 68.1% (5.0 points higher). On InternVL-3.5 (14B), MPO achieves 73.2% versus ProTeGi's 71.9% (1.3 points higher). On GPT-4.1 nano, MPO achieves 65.9% versus SEE's 61.6% (4.3 points higher). The gains are largest on the strongest model (Qwen2.5-VL 72B: +6.3 points), suggesting that more capable models can better exploit the richer multimodal signals provided by MPO's prompts — an important finding because it implies that multimodal prompt optimization becomes more valuable as base model capabilities improve, rather than being made obsolete by stronger models.

Optimizer model generalizability (Table 2, Bottom Left). MPO uses GPT-4o mini as its default optimizer, but the paper tests with two alternative optimizers on the PlantVillage dataset: Qwen2.5-VL (7B) and Gemini 2.5 Flash, as well as the stronger GPT-4o. The results show consistent improvement over the text-only baseline SEE across all optimizer choices: with Qwen2.5-VL (7B) as optimizer, MPO achieves 69.1% versus SEE's 65.2% (+3.9 points); with Gemini 2.5 Flash, MPO achieves 74.8% versus SEE's 68.2% (+6.6 points); with GPT-4o mini (default), MPO achieves 76.4% versus SEE's 69.0% (+7.4 points); with the stronger GPT-4o, MPO achieves 78.0% versus SEE's 69.2% (+8.8 points). The monotonic improvement as optimizer capability increases (from Qwen2.5-VL's 69.1% to GPT-4o's 78.0%) suggests that better optimizer models produce better multimodal prompts by generating more insightful failure analyses and more effective generation/edit/mix conditions. This is an encouraging scaling trend — as optimizer LLMs continue to improve, the prompts they discover should improve correspondingly.

Modality-specific generator generalizability (Table 2, Bottom Right). MPO uses GPT-Image as its default image generator, but the paper tests two alternatives on PlantVillage: SANA1.5 (1.6B parameters, an open-source lightweight model) and Nano Banana. All three generators outperform the text-only SEE baseline (69.0%): SANA1.5 achieves 71.8% (+2.8 points), Nano Banana achieves 72.9% (+3.9 points), and GPT-Image achieves 76.4% (+7.4 points). A stronger variant of GPT-Image ("GPT-Image-Medium") achieves 76.6%, essentially identical to the default, suggesting diminishing returns beyond a certain generator quality threshold. The fact that even a 1.6B-parameter open-source generator enables MPO to outperform text-only optimization is practically significant — it means the framework can be deployed without reliance on proprietary generation APIs, and the cost of the generator can be scaled down substantially while retaining benefits.

Analysis on Cross-Modal Alignment (Figure 4)

The paper quantifies the importance of cross-modal alignment between text and non-textual prompt components by comparing MPO against four degraded variants that progressively weaken the alignment, and measuring both alignment quality (using the DSG score, a standard metric that decomposes textual descriptions into atomic queries and verifies each against visual content using an MLLM) and task performance improvement over the Human baseline.

  • MPO (joint optimization): achieves the highest DSG alignment score (~0.28, read from Figure 4) and the highest median performance improvement (~53%). This is the upper-right point on the scatter plot, establishing the ceiling for both alignment and performance.

  • Sequential optimization: where the textual prompt is optimized first (using text-only ProTeGi-style optimization), and then the non-textual prompt is refined afterward conditioned on the already-optimized text. This variant achieves a lower alignment score (~0.20) and a lower median improvement (~40%). The gap between Sequential and MPO demonstrates that optimizing text and non-text sequentially — even if both are eventually paired — produces worse alignment than joint optimization, because the text optimization step has no awareness of what visual information will later be available, potentially overfitting to text-only cues.

  • Random Image Prompt: where the image component is replaced with another optimized image prompt from a different text prompt (i.e., an image that was optimized but not jointly with this specific text). This drops alignment to approximately 0.14 and improvement to approximately 32%, illustrating that even well-optimized images are not interchangeable — the pairing matters.

  • In-Distribution Image Query: where the image component is replaced with a random image sampled from the same task (e.g., a random bird image from CUB). Alignment drops further to approximately 0.08, improvement to approximately 28%.

  • OOD Image Query: where the image is replaced with an image from a completely different task. Alignment is near zero (~0.02), and improvement collapses to approximately 24%.

The paper reports median values with Q1 and Q3 error bars on both axes. The clear positive correlation between DSG alignment score and performance improvement across these five variants establishes that cross-modal alignment is causally linked to task performance — the better the text and image cohere, the more the multimodal prompt helps the MLLM. This analysis isolates alignment as the mechanism behind MPO's gains: it is not merely that adding any image helps (random or mismatched images provide minimal benefit), but that adding a semantically aligned image helps, and joint optimization is the way to achieve that alignment.

Modality Contribution Ablation (Table 3)

To understand whether the gains come from the textual or non-textual component (or their interaction), the paper ablates the optimized multimodal prompt on two datasets:

  • Human Text + Human Image (baseline): the starting point before optimization. PlantVillage: 42.2%, CUB: 47.9%.

  • Human Text + MPO Image: using the human-written text prompt paired with MPO's optimized reference image. PlantVillage: 50.4% (+8.2 points over baseline), CUB: 58.2% (+10.3 points). This shows that the optimized image alone provides substantial benefit, even when paired with a suboptimal text prompt.

  • MPO Text + Human Image: using MPO's optimized text prompt but without the optimized image (image component is the original human-provided image or absent). PlantVillage: 55.6% (+13.4 points over baseline), CUB: 64.2% (+16.3 points). This shows that the optimized text also provides substantial independent benefit.

  • MPO Text + MPO Image (full multimodal): PlantVillage: 76.4% (+34.2 points over baseline), CUB: 78.6% (+30.7 points). The full multimodal prompt provides gains substantially larger than the sum of the individual modality gains — on PlantVillage, the individual gains are 8.2 + 13.4 = 21.6, but the combined gain is 34.2; on CUB, the individual gains are 10.3 + 16.3 = 26.6, but the combined gain is 30.7. This super-additive effect confirms the paper's claim that the two modalities are "not merely additive but mutually reinforcing" — the optimized text instructs the model on how to use the optimized image, and the optimized image provides visual grounding that makes the text instructions more effective.

Exploration Operator Analysis (Table 4 and Figure 6)

Quantitative ablation (Table 4). The contribution of each exploration operator is evaluated on four PlantVillage subtasks (Apple, Corn, Grape, Potato) by running MPO with only a single operator type (all child prompts generated using only generation, only edit, or only mix) versus the full MPO with all three. The text-only baseline SEE achieves 69.0% average across the four subtasks. Each individual operator outperforms SEE: Generation achieves 73.3% (+4.3 points), Edit achieves 72.5% (+3.5 points), Mix achieves 74.8% (+5.8 points). The full MPO with all three operators achieves 76.4% (+7.4 points over SEE). All three operators outperform the text-only baseline individually, but their combination achieves the best performance, confirming they are complementary rather than redundant. The pattern varies by subtask: on Grape (the hardest subtask with the lowest baseline performance), Mix shows a particularly large advantage (65.1% vs. Generation's 53.7% and Edit's 56.2%), suggesting that blending information from multiple parent prompts is especially valuable when the search space is challenging.

Qualitative analysis (Figure 6). The paper visualizes the optimization trajectory of the best-performing multimodal prompt on a CUB subtask (grosbeak species classification). The optimization process alternates between operators across iterations: Edit → Mix → Mix → Mix → Gen → Gen → Edit. The "Task Classes" box shows example images of four grosbeak species (Rose Breasted, Pine, Blue, Evening), and the optimized image prompt (shown at the right of the figure) is a composite reference image that displays all four species with visible distinguishing features. The trajectory illustrates how operators serve different roles at different stages — early iterations use Edit to refine initial prompts, middle iterations use Mix to combine strengths from multiple candidates, and later iterations alternate between Generation (to introduce novelty and avoid local optima) and Edit (to fine-tune the best candidates).

Selection Strategy Efficiency (Figure 5)

The paper compares four selection strategies on PlantVillage by measuring the average test score achieved at different levels of evaluation budget per prompt (from 25 to 100):

  • Uniform: allocates the budget equally across all candidate prompts. Reaches approximately 73.8% at budget 100.

  • Standard UCB (Auer, 2002): a bandit algorithm with uniform Beta(1,1) priors. Reaches approximately 74.2% at budget 100, slightly outperforming Uniform, but requires more budget to reach the same performance as MPO — at budget 50, UCB achieves approximately 73.5%, while MPO achieves this at budget ~25.

  • MPO w/o Prior: an ablated variant that uses Bayesian UCB but without prior inheritance (all children start from Beta(1,1)). Reaches approximately 74.5% at budget 100. The gap between this variant and full MPO isolates the contribution of prior inheritance.

  • MPO (full): with prior-inherited Bayesian UCB. Reaches approximately 75.5% at budget 100.

The key efficiency comparisons (read from Figure 5 by drawing horizontal lines): MPO achieves the same performance as Uniform at budget 100 using only approximately 30% of that budget, yielding a 70% reduction in evaluation cost. Compared to standard UCB, MPO achieves its budget-100 performance at approximately 48% of the budget (52% savings). Compared to MPO w/o Prior, MPO achieves its budget-100 performance at approximately 58% of the budget (42% savings). These efficiency gains are attributed to the warm start provided by prior inheritance — children from good parents start with optimistic priors and receive more evaluation budget early, while children from poor parents are identified as low-potential more quickly and abandoned.

All four strategies converge to similar performance at budget 100, which is expected since the total evidence accumulated is sufficient to identify the best prompt regardless of allocation strategy. The advantage of MPO's strategy is in the low-to-medium budget regime where allocation decisions matter most — precisely the regime that matters for scaling to large candidate pools where evaluating every candidate exhaustively is infeasible.

Training Dynamics (Figure 7)

The paper tracks the test performance of the top-1 prompt over 13 optimization iterations on CUB, comparing MPO against ProTeGi (the strongest text-only baseline). Both methods start at similar performance levels (approximately 67–68% at iteration 0). In the first three iterations, both methods improve at similar rates, reaching approximately 72–73%. After iteration 3, the trajectories diverge sharply: ProTeGi plateaus, gaining only 1.1 additional points (from ~73% to ~74.1% at iteration 12), while MPO continues to improve steadily, gaining 6.4 additional points (from ~73% to ~79.4% at iteration 12). The final gap at iteration 13 is approximately 5.3 points (MPO: ~79.4%, ProTeGi: ~74.1%).

This analysis demonstrates that text-only optimization hits a performance ceiling imposed by the expressiveness of the textual space — after a few iterations, text-only methods converge to the best text prompt they can find, and further iterations yield diminishing returns because all "good" textual variations have been explored. MPO, by expanding the search space into the non-textual modality, escapes this ceiling — the additional degrees of freedom provided by optimizing reference images, videos, or molecules mean there is always a new dimension to explore (different visual compositions, different molecular substructures, different video clips), and the optimizer can continue improving beyond the point where text-only methods saturate.

Hidden State Visualization (Figure 8)

To understand why multimodal prompts change model behavior, the paper visualizes the hidden states of the base MLLM when processing different types of prompts. Specifically, intermediate-layer embeddings are averaged following Zhang et al. (2024a) and projected to 2D using PCA. Four types of prompts are compared:

  • Text-only optimization methods (APE, OPRO, EvoPrompt, PE2, ProTeGi, SEE): hidden states from these methods all cluster together in the PCA space, indicating that despite their different optimization strategies, the text prompts they produce guide the MLLM's internal representations into a similar semantic region. This explains why their performance is bounded — they all activate similar reasoning pathways in the model.

  • MPO text-only component (T): the textual portion of MPO's optimized prompt, evaluated without the optimized image. Its hidden states fall within the same cluster as the text-only methods, confirming that MPO's text prompt, in isolation, induces similar representations to other text-only methods.

  • MPO full multimodal (T+I): the complete multimodal prompt (optimized text + optimized image). Its hidden states project to a distinct, well-separated region of the PCA space, far from the text-only cluster. This demonstrates that the non-textual component fundamentally alters the model's internal representations — it shifts the MLLM into a different reasoning regime that text alone cannot access.

This analysis provides mechanistic evidence for the paper's central claim: multimodal prompts are not merely "better text prompts with pictures attached." They actively change how the model represents and processes the task, enabling reasoning pathways that are inaccessible through text-only prompting. The separation in PCA space is consistent with the performance gap — prompts that induce representations outside the text-only cluster (MPO full) achieve higher accuracy than those within it (all text-only methods).

Sensitivity to Prior Strength (Figure 9)

The prior strength hyperparameter $S$ (which controls how many pseudo-observations are inherited from parent to child) is varied from 0 to 25 on PlantVillage, and the average score of the best prompt at convergence is reported. At $S = 0$ (no prior inheritance, equivalent to standard UCB), the average score is approximately 74.2%. Performance rises as $S$ increases, reaching a peak of approximately 76.5% at $S = 10$ — this is the default value used in all experiments. Beyond $S = 10$, performance declines, falling to approximately 74.8% at $S = 25$.

The interpretation is straightforward: too little prior strength ($S$ small) under-utilizes the informative parent signal, forcing the selector to re-learn child quality from scratch and wasting evaluation budget. Too much prior strength ($S$ large) causes over-reliance on the parent prior, making it difficult for a genuinely improved child to overcome an mediocre parent's pessimistic initialization — the selector is "anchored" to the parent's performance and slow to update. The optimal $S = 10$ represents a balance where inherited knowledge provides a warm start but does not dominate the actual evaluation evidence. The paper's choice to set $S$ as a fixed fraction (10%) of the evaluation budget rather than as an absolute number is practically motivated — it ensures the prior strength scales with the total evidence available, maintaining the same warm-start-to-evidence ratio across different budget settings.

Personalization on Auklet SubTask (Tables 15 and 16, Appendix C.3)

The paper provides a detailed qualitative comparison between SEE (the best text-only baseline on CUB) and MPO on the Auklet classification subtask of CUB. SEE achieves only 50.0% accuracy on this subtask (Table 15), while MPO achieves 80.3% (Table 16) — a 30.3-point improvement on a single subtask. The qualitative examples in Tables 15 and 16 show why this gap is so large:

  • SEE's optimized text prompt (Table 15) is extremely verbose — over 500 words — containing detailed physical descriptions of all four auklet species (Parakeet Auklet: "striking blue bill and a notable yellow eyebrow stripe"; Rhinoceros Auklet: "horn-like projection on its bill"; Crested Auklet: "long, curved crest of feathers"; Least Auklet: "stubby bill"). It also includes a 10-step classification checklist, confidence level instructions, and hypothetical example scenarios. Despite this exhaustive text, the model misclassifies a Crested Auklet as a Rhinoceros Auklet, reasoning that "the bird has a robust body" and "a horn-like projection on the bill" — features that are present in the text description but incorrectly mapped to the wrong species when the model must translate text into visual feature detection.

  • MPO's optimized multimodal prompt (Table 16) is concise — approximately 150 words — and is paired with an optimized reference image showing auklet species. The text explicitly instructs the model to "compare it directly with the examples in the hybrid image" and to "prioritize bill shape in cases of ambiguity." The model's response correctly identifies the bird as a Crested Auklet, reasoning that "the bill is relatively short and thick, with a slight curve at the tip" and "the presence of a crest on its head is clearly visible in the target image" — features that are directly grounded in the reference image rather than reconstructed from text.

Crucially, on examples where SEE's optimized prompt fails, MPO's multimodal approach successfully fixes 66.6% of the misclassifications. This statistic — reported in Appendix C.3 — is among the strongest evidence that multimodal prompts resolve errors that are inherently difficult for text-only optimization to address, because the errors stem from the ambiguity of translating textual descriptions to visual features rather than from poor text prompt design.

Ablation Studies and Robustness Checks

  • Optimizer model robustness (Table 2, Bottom Left): MPO maintains its advantage over SEE across four different optimizer models (Qwen2.5-VL 7B, Gemini 2.5 Flash, GPT-4o mini, GPT-4o), with the performance gap growing as optimizer capability increases (from +3.9 points with Qwen2.5-VL to +8.8 points with GPT-4o). This suggests that MPO's architecture benefits from stronger optimizers — better failure analysis and condition generation lead to better multimodal prompts — but the framework does not depend on a specific optimizer model to function.

  • Generator robustness (Table 2, Bottom Right): MPO works with three different image generators of varying capability (GPT-Image, SANA1.5 1.6B, Nano Banana), all outperforming the text-only baseline SEE. The lightweight open-source SANA1.5 (1.6B parameters) achieves 71.8% versus SEE's 69.0%, demonstrating that the framework does not require expensive proprietary generation APIs. A stronger variant of GPT-Image ("Medium") achieves 76.6%, only marginally higher than the default GPT-Image (76.4%), suggesting that beyond a certain quality threshold, generator improvements yield diminishing returns — the bottleneck shifts from generation quality to the optimizer's ability to specify good conditions.

  • Base model robustness (Table 2, Top): MPO outperforms text-only baselines across four base MLLMs (Qwen2.5-VL 72B, Gemma3 12B, InternVL-3.5 14B, GPT-4.1 nano), with gains ranging from +1.3 points (InternVL-3.5) to +6.3 points (Qwen2.5-VL 72B). The variation in gain magnitude across models is notable — the strongest model benefits the most from multimodal prompts, suggesting that more capable models have greater capacity to exploit the additional information provided by non-textual prompts. This is a non-trivial finding: it could have been that stronger models already "know" everything and prompt optimization provides diminishing returns, but the data shows the opposite — stronger models extract more value from better prompts.

  • Operator complementarity (Table 4): Each of the three exploration operators (Generation, Edit, Mix) individually outperforms the text-only baseline SEE (73.3%, 72.5%, and 74.8% respectively vs. 69.0%), and their combination (76.4%) exceeds any individual operator. The Mix operator shows the strongest single-operator performance on the hardest subtasks (Grape: 65.1% vs. Generation's 53.7% and Edit's 56.2%), confirming its role in recombining complementary strengths when no single parent is adequate.

  • Prior strength sensitivity (Figure 9): Performance is maximized at $S = 10$ (the default), drops when $S$ is too small (under-utilizes parent prior) or too large (over-relies on parent prior), with the overall range from ~74.2% ($S=0$) to ~76.5% ($S=10$) representing a 2.3-point swing. This is a moderate sensitivity — getting $S$ wrong by a factor of 2.5 (using $S=25$ instead of $S=10$) costs about 1.7 points, which is noticeable but not catastrophic.

  • Computational cost comparison (Table 5): All methods make the same number of base model calls (11.7k) since the number of explored prompts and evaluation budget are equalized. MPO makes 234 optimizer model calls (same as ProTeGi's two-step process) plus 117 generator calls (for the modality-specific generation, editing, and mixing). The paper notes these additional calls are manageable because lightweight open-source generators can be used without sacrificing the performance advantage (as validated by the SANA1.5 result in Table 2), and the performance gain (5.1 points over ProTeGi) justifies the marginal cost increase. This is an explicit acknowledgment that multimodal optimization incurs additional computational overhead, but the paper argues the overhead is justified by the performance gains.

  • Cross-modal alignment isolation (Figure 4): The five variants (MPO, Sequential, Random Image, In-Distribution Query, OOD Query) demonstrate that performance improvement correlates with DSG alignment score. MPO achieves both the highest alignment and highest improvement. The Sequential variant — where text is optimized first, then image is generated conditioned on the optimized text — achieves lower alignment and lower improvement, directly supporting the paper's claim that joint optimization (cohesive backpropagation) is necessary to maintain cross-modal consistency. If independent optimization were sufficient, Sequential would perform closer to MPO; the observed gap validates the alignment-preserving mechanism.

Critical Assessment

Do the Experiments Support the Paper's Central Claims?

Claim 1: Multimodal prompt optimization consistently outperforms text-only prompt optimization. The evidence in Table 1 strongly supports this claim across 10 datasets and three modalities. MPO outperforms the best text-only baseline (ProTeGi) on every single dataset, with gains ranging from 1.6 points (DrivingVQA) to 12.0 points (PlantVillage) and an average improvement of 5.1 points. The robustness checks in Table 2 demonstrate this advantage persists across different base models, optimizer models, and generators. However, the claim would be stronger if confidence intervals were reported — with only 3 independent runs, the statistical significance of the per-dataset gains is uncertain, particularly for smaller gains like the 2.4-point improvement on SLAKE (38.2% vs. 35.8%). Additionally, all experiments use only one MLLM family (Qwen) as the base model; the generalizability to other prominent MLLMs (GPT-4V, Gemini, Claude) is not tested, though the four-model validation in Table 2 (Top) partially addresses this concern by including Gemma3 and InternVL-3.5.

Claim 2: Cross-modal alignment is necessary for realizing gains from multimodal prompts. The cross-modal alignment analysis (Figure 4) provides good evidence for this claim. The gradient of performance improvement across five alignment conditions — from MPO (highest alignment, highest improvement) down to OOD image query (lowest alignment, lowest improvement) — establishes a clear relationship. The sequential optimization variant (which breaks the joint alignment constraint) performs worse than MPO, directly supporting the specific mechanism (cohesive backpropagation) that MPO uses to maintain alignment. However, the analysis is limited to the DSG metric as the alignment measure, and DSG was originally designed for evaluating text-to-image generation consistency, not for measuring prompt-level cross-modal alignment. Whether DSG captures all relevant aspects of alignment for the prompt optimization context is not validated. Additionally, this analysis is only shown in aggregate; per-dataset or per-difficulty-level alignment-performance correlations would provide a richer picture of when alignment matters most.

Claim 3: Prior-inherited Bayesian UCB reduces evaluation budget compared to prior-free alternatives. Figure 5 provides strong evidence: MPO achieves the same performance as Uniform allocation with 70% less budget, and 42% less budget than the prior-free UCB variant. The parent-child correlation that motivates this mechanism is well-documented (Figure 3: Pearson's r = 0.88, P-value = 3.33 × 10⁻¹⁷⁰), leaving no doubt that the correlation is real. However, the budget reduction is measured on PlantVillage only — it is not reported whether similar efficiency gains hold across datasets or modalities. Since the parent-child correlation magnitude likely varies by task difficulty and modality, the reported 42–70% savings may not generalize uniformly. Additionally, the proposition's assumption — that the parent prior is KL-closer to the true child performance than the uniform prior — is only verified empirically through the correlation analysis, not through a direct KL-divergence comparison.

Claim 4: MPO escapes the performance ceiling of text-only optimization. The training dynamics analysis (Figure 7) supports this claim on CUB: ProTeGi plateaus after iteration 3 (+1.1 points thereafter), while MPO continues improving (+6.4 points after iteration 3). This is a single dataset, however, and the claim would be strengthened by showing similar dynamics on other datasets. The hidden state visualization (Figure 8) provides complementary mechanistic evidence — multimodal prompts shift model representations to a region inaccessible to text-only prompts, suggesting a qualitative difference in how the model processes the task.

Genuine Weaknesses and Missing Analyses

Small test sets and lack of statistical rigor. The test set size is 300 examples for all image and video tasks, and variable for molecular tasks (382 for BBBP, similar for others). With only 3 independent runs and no reported confidence intervals, standard errors, or significance tests, it is impossible to assess whether the observed performance differences (particularly small ones like MPO vs. ProTeGi on SLAKE: 38.2% vs. 35.4%) are statistically reliable or within noise. Given the stochasticity in the optimization process (random operator selection, temperature 0.7 generation, random mini-batch sampling), three runs may not adequately capture the variance.

No ablation on the number of operators or their mixing ratio. The paper uses all three operators with even allocation (each operator generates roughly $b^2 / 3$ children per iteration). The operator ablation (Table 4) only tests each operator individually versus all three combined, but does not explore whether two operators suffice (e.g., Generation + Edit without Mix), whether the 1:1:1 ratio is optimal, or whether the operator mix should change over iterations (more Generation early, more Edit late — a natural annealing schedule). The qualitative example in Figure 6 shows operators alternating across iterations, but whether this observed pattern is optimal or merely an artifact of random sampling is not investigated.

No combination of MPO with stronger text-only baselines. The paper compares MPO against text-only methods and shows MPO wins, but does not test whether the text-only components of MPO (the optimized text prompt without the image) are already stronger than baseline text-only methods. The modality contribution ablation (Table 3) partially addresses this — MPO Text without the optimized image achieves 55.6% on PlantVillage and 64.2% on CUB, while the best text-only baseline ProTeGi achieves 64.4% and 70.0% respectively. So on PlantVillage, MPO's text-only component is worse than ProTeGi's full text prompt (55.6% vs. 64.4%), while on CUB, it is competitive (64.2% vs. 70.0%). This suggests that MPO's advantage comes primarily from the multimodal combination, not from a better text optimizer. However, a direct comparison of MPO's text optimization against ProTeGi's text optimization (both evaluated without images) would isolate the contribution of MPO's optimizer design from the contribution of multimodality.

Difficulty estimation or task-level performance prediction is absent. The paper does not characterize when multimodal prompts help most. Are the gains concentrated on "hard" examples (where text descriptions are insufficient and visual reference is crucial) or uniformly distributed? For CUB subtasks, Table 6 shows massive variance — MPO improves auklet classification by 31.3 points over ProTeGi but cormorant classification by only 4.9 points. This heterogeneity is noted but not systematically analyzed. A difficulty-stratified analysis (analogous to the difficulty-bin analysis in the LLM test-time compute paper) would provide actionable guidance on when to invest in multimodal prompt optimization versus when text-only suffices.

The video modality results use only the generation operator. Due to the complexity of video editing and mixing, the video experiments (Drive&Act, VANE-Bench) use only the generation operator. This means the video results evaluate a reduced version of MPO (single-operator exploration), not the full three-operator framework. The strong performance on video tasks (5.3 and 5.7 points over ProTeGi) demonstrates that even generation-only MPO is effective, but the contribution of edit and mix operators — which are central to MPO's design — is entirely untested for videos. This is explicitly acknowledged as a limitation (Section 4.2: "only the generation operator" for videos), but the implications for MPO's generality are not fully discussed.

No comparison to instance-specific multimodal prompting. The paper distinguishes MPO from instance-specific methods (MM-CoT, visual prompting with bounding boxes) and argues they solve different problems (task-level vs. per-query optimization). However, a comparison against these methods would contextualize MPO's performance — if a per-query optimized prompt achieves 85% on a task where MPO achieves 78%, the gap represents the cost of task-level generalization. No such comparison is provided.

Evaluation budget fairness. All methods explore the same number of prompts and use the same evaluation budget per prompt (100 examples). However, MPO requires additional computation for the generator calls (117 calls per run, Table 5). The paper argues this cost is manageable (especially with lightweight generators), but it is an additional cost not incurred by text-only methods. A FLOPs-matched comparison (where text-only methods are given proportionally more evaluation budget to compensate for their lower optimizer cost) is not conducted, similar to how the LLM test-time compute paper matched FLOPs between pretraining and inference. This means the reported gains are partly attributable to MPO using more total computation, not just better allocation of equal computation.

Single optimizer model family for main results. The main results (Table 1) use GPT-4o mini as the optimizer across all experiments. The optimizer generalizability study (Table 2, Bottom Left) tests three alternatives on PlantVillage, but not on the full 10-dataset benchmark. It is possible that GPT-4o mini has specific strengths (e.g., strong image description capabilities) that other optimizers lack, and MPO's gains are partly attributable to the optimizer model choice rather than the MPO framework itself. The paper shows MPO works with other optimizers (Qwen2.5-VL, Gemini), which partially addresses this, but the magnitude of the gain varies (from +3.9 to +8.8 points), suggesting optimizer quality does matter.

No analysis of failure modes or error types. The paper provides qualitative examples of MPO's successes (Tables 14, 16) and a single text-only failure case (Table 15), but does not systematically categorize what types of errors multimodal prompts fix versus what types remain. A taxonomy of residual errors after MPO optimization would reveal whether the remaining errors are due to fundamental model limitations (which multimodal prompts cannot address), or whether they represent opportunities for further prompt optimization (e.g., with additional modalities or more sophisticated operators).

Where the Conditions of the Claims Matter

  • The claim that MPO "consistently outperforms" text-only methods holds across all tested datasets and modalities, but the magnitude of the gain varies substantially: from 1.6 points on DrivingVQA to 12.0 points on PlantVillage. The claim should be understood as "outperforms, with variable margins depending on how much the task benefits from non-textual information." Tasks where visual features are easily described in text (e.g., DrivingVQA's lane-marking and traffic-sign questions) benefit less from multimodal prompts than tasks where visual features are subtle and ambiguous in text (e.g., fine-grained bird classification).

  • The claim of "70% evaluation budget reduction" via prior-inherited Bayesian UCB is demonstrated on PlantVillage only (Figure 5). The generalizability of this efficiency gain to other datasets and modalities is not established, though the mechanism (parent-child correlation) is demonstrated in aggregate (Figure 3), suggesting it should generalize.

  • The claim that MPO "escapes the performance ceiling of text-only methods" is demonstrated on CUB only (Figure 7). The training dynamics on other datasets are not shown, so it is unknown whether ProTeGi plateaus on other tasks or continues to improve.

  • The claim that multimodal prompts are "mutually reinforcing" (super-additive gains) is demonstrated on PlantVillage and CUB only (Table 3). The modality contribution ablation is not replicated for video or molecular tasks, where the text-image dynamics may differ — for molecules, the SMILES string is the non-textual prompt, and its interaction with text instructions is qualitatively different from image-text interaction.

6. Limitations and Trade-offs

6.1 Difficulty Estimation and Cost of Exploration Are Unaccounted For in Headline Efficiency

The assumption or constraint. The prior-inherited Bayesian UCB selection strategy (Section 3.3) is designed to reduce the evaluation budget wasted on low-potential candidate prompts. The paper reports a 70% reduction in evaluation cost compared to uniform allocation (Figure 5) and a 42% reduction compared to a prior-free UCB baseline. However, these savings are measured solely within the evaluation phase — the cost of generating the candidate prompts in the first place, including the optimizer LLM calls for failure analysis, feedback generation, and condition writing, as well as the modality-specific generator calls for producing non-textual prompts, is not included in the efficiency calculation. The paper explicitly acknowledges the additional computation:

"MPO incurs an additional computational cost by calling a modality-specific generator to explore non-textual prompts" (Appendix C.1, Table 5 discussion)

and quantifies this overhead: 117 additional generator calls per optimization run, with optimizer model calls matching ProTeGi's 234 (Table 5). The paper also implicitly acknowledges that the expanded multimodal search space requires substantial exploration before evaluation even begins — the $b^2 = 9$ initial candidates in iteration 0 are all generated from scratch using the generation operator, and each subsequent iteration generates $b^2 = 9$ more candidates via generation, edit, and mix operators, requiring 9 optimizer LLM calls per iteration for cohesive backpropagation and joint multimodal update, plus the corresponding generator calls.

The consequence. The headline efficiency claim — that MPO reduces evaluation budget by 42–70% — is an incomplete picture of total computational cost. A practitioner deploying MPO must pay for: (1) the base MLLM calls for evaluation (the cost that the efficiency analysis optimizes over), (2) the optimizer LLM calls for generating failure analyses, feedback, and conditions (234 calls per run), and (3) the modality-specific generator calls for producing non-textual prompts (117 calls per run). These additional costs are not amortized into the efficiency savings. Depending on the relative cost of these operations — which varies by model choice (GPT-4o mini vs. GPT-4o for the optimizer, GPT-Image vs. SANA1.5 for generation) — the total wall-clock time and API cost could be substantially higher than text-only methods, even if MPO requires fewer base model evaluations. The paper acknowledges this qualitatively ("this cost is manageable, as this process can utilize lightweight, open-source generators") but does not provide a FLOPs-matched or cost-matched comparison that would allow a practitioner to assess whether the performance gains justify the additional overhead.

What evidence exists in the paper. Table 5 in Appendix C.1 provides model request counts: all methods make 11.7k base model calls, text-only methods make 117–234 optimizer calls with zero generator calls, and MPO makes 234 optimizer calls plus 117 generator calls. The paper demonstrates in Table 2 (Bottom Right) that using lightweight generators like SANA1.5 (1.6B parameters) still outperforms text-only baselines (71.8% vs. SEE's 69.0%), suggesting the generator cost can be reduced without sacrificing the performance advantage. However, no ablation measures the tradeoff between generator quality and total cost. The end-to-end optimization time or total FLOPs is not reported.

Mitigation status. Partially addressed. The paper explicitly acknowledges the additional cost and demonstrates that it can be mitigated through generator choice (SANA1.5). The paper frames this as a manageable increase rather than a fundamental limitation, noting that "the performance gain (5.1 points over ProTeGi) justifies the marginal cost increase" (Appendix C.1). However, this justification is qualitative and not rigorously measured. Future work on direct difficulty prediction (mentioned in Section 8 of the paper's conclusion, though not quoted here) could potentially reduce the exploration cost, but no such mechanism is developed or evaluated in the current work.

6.2 The Method Cannot Help on Tasks Where the Base Model Fundamentally Lacks Capability

The assumption or constraint. MPO optimizes the prompt that an MLLM receives, but it does not change the model's underlying parameters, knowledge, or reasoning capabilities. The base model — Qwen2.5-VL (7B) for image/video tasks, Qwen3 (8B) for molecular tasks — has fixed capabilities determined by its pretraining and instruction tuning. A multimodal prompt can surface these capabilities more effectively by providing richer contextual signals, but it cannot create capabilities that do not exist. The paper implicitly assumes that the base model possesses sufficient task-relevant knowledge and reasoning ability such that better prompts unlock material improvements — i.e., that the performance bottleneck is at least partly attributable to prompt quality rather than fundamental model limitations.

This is most clearly visible in the SLAKE medical VQA results (Table 1), where MPO achieves only 38.2% accuracy, marginally above ProTeGi's 35.4% and PE2's 35.8%. All methods — manual, few-shot, and optimized — are compressed into a narrow band (30.8% to 38.2%), and the absolute performance ceiling appears low regardless of prompt quality. Similarly, on the hardest CYP Inhibition subtasks (Table 7), MPO achieves F1 scores in the 50–65% range, with gains over text-only methods that are modest (2–5 points on several subtasks). The paper also notes this phenomenon directly in the context of the training dynamics analysis (Section 4.2, Figure 7): text-only methods "hit a performance ceiling imposed by the expressiveness of the textual space," and MPO escapes this ceiling by adding non-textual modalities, but there is no guarantee that the multimodal ceiling is anywhere near 100% — the model may saturate at 40%, 60%, or 80% depending on its inherent capability.

The consequence. For tasks where the base MLLM's fundamental capability is low — complex medical image interpretation, challenging molecular property prediction, domains where the model has insufficient pretraining exposure — multimodal prompt optimization will provide diminishing or negligible returns. A practitioner who observes that human-designed prompts achieve only 30% accuracy on their target task should not expect MPO to dramatically close the gap to 80% or 90%. The maximum achievable performance is bounded by the base model's knowledge and reasoning capacity, not by prompt quality. This is analogous to the finding in the LLM test-time compute scaling literature that "hard problems remain essentially unsolved" regardless of inference budget allocation — test-time compute (or, in this case, test-time prompt optimization) amplifies existing capability but does not create it. The paper does not provide guidance on how to determine, a priori, whether a given task is in the "prompt-bottlenecked" regime (where MPO will provide large gains) or the "capability-bottlenecked" regime (where gains will be marginal).

What evidence exists in the paper. The cross-dataset variation in MPO's gains provides indirect evidence. On PlantVillage and CUB — tasks where visual features are discriminative and well-represented in the model's training distribution — MPO achieves 76.4% and 78.6%, with gains of 7.4–12.0 points over text-only baselines. On SLAKE — a specialized medical domain — MPO achieves only 38.2% with a gain of 2.4–2.8 points over text-only methods. On DrivingVQA — general driving scenarios — the gain is 1.6 points (56.0% vs. 54.4%). This pattern is consistent with the capability-bottleneck hypothesis (gains are larger when the base model already performs reasonably well), but the paper does not explicitly analyze this relationship or attempt to predict gain magnitude from base model performance.

Mitigation status. Not addressed. The paper does not provide a method for estimating whether a given task is prompt-bottlenecked or capability-bottlenecked before investing in multimodal prompt optimization. This is an important practical concern: a practitioner considering deploying MPO would benefit from a diagnostic that uses cheap initial evaluations to estimate the potential ceiling, analogous to the difficulty estimation used in the LLM test-time compute paper. The paper's training dynamics analysis (Figure 7) shows that text-only optimization plateaus quickly, but does not establish whether the multimodal optimization similarly plateaus at a level that is or is not close to the maximum achievable given the base model's capabilities.

6.3 The Framework Has Not Been Validated on Open-Ended or Subjective Tasks Without Clean Correctness Signals

The assumption or constraint. MPO's entire optimization loop depends on a task-specific evaluation metric $f$ that can judge whether the MLLM's output is correct: the optimizer generates prompts, evaluates them by comparing the MLLM's predictions against ground-truth answers using $f$, and uses the resulting scores to select high-performing prompts and generate improvement feedback. The paper evaluates on tasks where correctness is well-defined: classification accuracy (PlantVillage, CUB, Drive&Act, CYP Inhibition, BBBP), binary or multiple-choice VQA (SLAKE, DrivingVQA, VANE-Bench), and molecular property prediction with binary labels. The evaluation function $f$ is exact match (for open-ended QA), label extraction (for classification), or F1 score (for imbalanced molecular tasks). In all cases, there is a ground-truth reference against which the model's output can be automatically and unambiguously scored.

The paper does not evaluate on tasks where correctness is ambiguous, subjective, or multi-dimensional: open-ended dialogue, creative writing, summarization quality, instructional following where multiple valid outputs exist, or tasks where evaluation requires human judgment rather than automated metrics. The cohesive backpropagation mechanism (Section 3.2) relies on a failure set $F = \{(q, a, y) | y \neq a\}$ — this set is well-defined only because correctness ($y = a$ vs. $y \neq a$) is binary. On tasks where "good" output spans a range of quality, constructing a failure set and generating diagnostic feedback becomes substantially more complex.

The consequence. The paper's claims about MPO's effectiveness apply specifically to tasks with clean, automatable correctness signals — classification, closed-form QA, binary prediction. Whether the framework generalizes to the broader landscape of MLLM applications (open-ended generation, summarization, creative tasks, multi-turn dialogue) is unknown and untested. For such tasks, the optimization loop would require either: (a) a learned reward model that approximates human judgment, introducing its own biases and optimization vulnerabilities (analogous to the reward hacking problem in RLHF), or (b) expensive human evaluation in the loop, which would make MPO's iterative beam search (13 iterations × 9 candidates per iteration = 117+ candidates requiring evaluation) prohibitively expensive. Additionally, the prior-inherited Bayesian UCB assumes a scalar reward signal for each evaluation batch; for tasks where quality is multi-dimensional (e.g., summarization involves accuracy, conciseness, and fluency), reducing to a single scalar would lose information.

The paper's primary evaluation domain — MATH-aligned questions with ground-truth answers — is actually the easiest setting for automatic prompt optimization because the correctness signal is clean and unambiguous. This limits the practical applicability of the current MPO framework to tasks with similar properties, which exclude many of the most common MLLM use cases.

What evidence exists in the paper. All 10 evaluation datasets have well-defined ground-truth answers. The paper reports the use of task-specific evaluation criteria for determining answer correctness (Section 4.1, Implementation Details): "the final predefined label is extracted for standard classification, strict formatting rules are applied for binary and closed-ended QA tasks, and exact match is used for open-ended QA tasks." The optimization objective (Section 3.1) is defined as maximizing $\mathbb{E}_{(q,a) \sim D}[f(\text{MLLM}(t, m, q), a)]$, which presumes $f$ is well-defined. No experiments test MPO on open-ended generation tasks, and no ablation examines sensitivity to noisy or approximate reward signals.

Mitigation status. Not addressed. The paper does not discuss this limitation or propose extensions for subjective evaluation. The finding is not presented as a limitation but rather as the natural scope of the experiments. A practitioner considering applying MPO to tasks without clean correctness signals (e.g., dialogue quality, content moderation, creative generation) would need to independently determine how to adapt the optimization loop, with no guidance from the paper on whether the core mechanisms (cohesive backpropagation, prior-inherited UCB, the three operators) would remain effective with learned or human-provided reward signals.

6.4 The Method Has Only Been Validated on a Single MLLM Family (Qwen) with One Optimizer Model as Default

The assumption or constraint. All main experiments in Table 1 use Qwen2.5-VL (7B) as the base MLLM for image and video tasks, Qwen3 (8B) for molecular tasks, and GPT-4o mini as the optimizer model. While the paper validates generalizability across some alternative configurations — four base models on CUB (Table 2, Top), four optimizer models on PlantVillage (Table 2, Bottom Left), and three image generators on PlantVillage (Table 2, Bottom Right) — these robustness checks are limited in scope. The base model generalizability test is conducted on a single dataset (CUB, 12 bird classification subtasks). The optimizer generalizability test and generator generalizability test are conducted on a single dataset (PlantVillage, a plant disease classification task). The interaction effects between these components — whether MPO's advantage persists with, for example, a weaker base model paired with a stronger optimizer, or a stronger base model paired with a weaker generator — are not systematically explored.

Additionally, all tested models belong to a particular class of MLLMs: autoregressive transformer-based models with vision encoders, pretrained on large-scale vision-language data, and instruction-tuned. The paper's claims do not extend to MLLMs with fundamentally different architectures (e.g., early-fusion models, non-autoregressive models, models with drastically different visual processing pipelines) or to models trained on different data distributions (e.g., domain-specific medical MLLMs that might have different sensitivity to prompt variations).

The consequence. A practitioner using a different MLLM family (e.g., GPT-4V, Gemini 2.5 Flash, Claude, LLaVA, InternVL) cannot assume that MPO will provide the same magnitude of gains. The benefit of multimodal prompts depends on how well the base MLLM can leverage visual reference information — models with weaker vision-language alignment might benefit less from reference images, while models with stronger alignment might benefit more. The paper's own data supports this concern: in Table 2 (Top), MPO's gain over ProTeGi varies from +1.3 points on InternVL-3.5 (14B) to +6.3 points on Qwen2.5-VL (72B). This 5.0-point spread in gain magnitude across four models — all tested on the same dataset (CUB) — indicates that model choice substantially influences MPO's effectiveness. If the gain on a practitioner's chosen model is only 1–2 points, the additional cost of multimodal optimization (generator calls, more complex optimization pipeline) may not be justified.

The optimizer model generalizability test similarly shows substantial variation: MPO's gain over SEE ranges from +3.9 points (Qwen2.5-VL as optimizer) to +8.8 points (GPT-4o as optimizer) on PlantVillage. A practitioner constrained to use a weaker optimizer (e.g., for cost or privacy reasons) may see substantially smaller benefits.

What evidence exists in the paper. Table 2 provides the relevant robustness data, but as noted, each component is tested on only one dataset. The paper does not report a factorial design where base model, optimizer model, and generator are varied simultaneously, making it impossible to disentangle interaction effects. The paper also does not report failure cases — are there model-dataset combinations where MPO performs worse than text-only baselines? A negative result of this type would be practically important (a practitioner could then know to avoid MPO for specific model-task pairs) but is not explored.

Mitigation status. Partially addressed. The robustness checks in Table 2 demonstrate that MPO's advantage is not an artifact of a single lucky configuration. However, the limited scope of these tests (single dataset per component) means the evidence for broad generalizability is suggestive rather than conclusive. A practitioner would be making an evidence-based bet, not relying on a proven guarantee, when applying MPO to a new model family.

6.5 The Sequential Nature of Iterative Beam Search Creates High Latency for Deployment-Style Optimization

The assumption or constraint. MPO's optimization loop (Algorithm 1) is fundamentally sequential: each iteration requires generating failure sets, running cohesive backpropagation, producing joint multimodal updates for $b^2$ child candidates, evaluating these candidates through the Bayesian UCB procedure, and selecting the top-$b$ to carry forward. The Bayesian UCB evaluation itself is sequential within each iteration — prompts are evaluated one batch at a time, with each evaluation result informing which prompt to evaluate next (the UCB decision rule). With $T = 13$ iterations, a beam size of $b = 3$, and a per-candidate evaluation budget of 100 examples, the optimization process requires: 13 × 9 = 117 candidate generations (each involving an optimizer LLM call plus a generator call), plus $T \times (N/B) \times b$ rounds of batched base MLLM evaluation, each dependent on the previous round's UCB scores. The paper optimizes for total compute (number of model calls, reported in Table 5) but does not measure or optimize for wall-clock latency.

The consequence. For a practitioner who needs to run prompt optimization once and then deploy the optimized prompt for many inference requests, the optimization latency may be acceptable — it is a one-time cost amortized over future use. However, for scenarios where prompt optimization must be run frequently (e.g., per-task customization in a multi-tenant system, periodic re-optimization as the underlying MLLM is updated, or optimization over a frequently changing task distribution), the wall-clock time becomes a practical constraint. The sequential UCB evaluation — which cannot be parallelized because each round depends on the previous — is the primary bottleneck. If a single round of evaluation (running the base MLLM on a batch of $B$ examples) takes 5 seconds, 50 rounds per iteration × 13 iterations = 650 rounds × 5 seconds = approximately 54 minutes just for the evaluation phase, plus the time for 117 optimizer calls and generator calls. Actual wall-clock time would depend on API latency, batch sizes, and hardware, but the sequential structure ensures it cannot be reduced below a certain floor.

The paper's prior-inherited UCB is designed to reduce the number of evaluations, not to enable parallelism. A practitioner facing latency constraints might prefer a strategy that can evaluate many candidates in parallel, even if less efficient in total compute, because parallelism trades compute for wall-clock time. MPO provides no mechanism for such a tradeoff.

What evidence exists in the paper. The paper reports the number of model calls (Table 5) as a proxy for computational cost, but does not report wall-clock time, end-to-end optimization duration, or any latency measurements. The Algorithm 2 specification in Appendix A.5 shows the sequential UCB evaluation loop: at each round $t$, prompt $j$ is selected based on current UCB scores, evaluated on $D_{\text{mini}}$, and its posterior is updated before the next round. This loop is inherently sequential. No ablation compares this against a fully parallel evaluation strategy (evaluate all candidates equally, then select) in terms of latency-performance tradeoff.

Mitigation status. Not addressed. The paper does not discuss latency as a design consideration, and the evaluation framework is optimized solely for sample efficiency (number of evaluations) rather than wall-clock time. This is a reasonable choice given the paper's focus on a different problem — the cold-start efficiency of candidate selection — but practitioners with latency constraints should be aware that MPO's efficiency gains are in the number of evaluations, not in end-to-end runtime, and that the sequential UCB evaluation prevents full parallelism.

6.6 The Test Sets Are Small and the Paper Does Not Report Statistical Significance

The assumption or constraint. The primary results in Table 1 are computed on test sets of: 300 examples for all image and video tasks (sampled from the full test sets), variable sizes for molecular tasks (382 for BBBP, similar for individual CYP and Absorption subtasks as defined by the official TDC splits). The results are averaged over only three independent optimization runs, and the paper does not report standard deviations, standard errors, confidence intervals, or any formal statistical test (e.g., paired t-test, bootstrap confidence intervals) for the differences between MPO and baselines.

This matters because the optimization process is stochastic at multiple levels: operator selection is random (uniform sampling from {generation, edit, mix}), prompt generation by the optimizer LLM is at temperature 0.7 (introducing variance in the generated text prompts and conditions), evaluation mini-batches are randomly sampled from the training set during Bayesian UCB selection, and the test set evaluation itself is subject to sampling variance. With only 3 runs, the sample mean is an unreliable estimator of the true expected performance, and the standard error of the mean could be large enough that some of the smaller reported gains (e.g., MPO vs. ProTeGi on DrivingVQA: 56.0% vs. 54.4%, a difference of 1.6 points; on RSVQA: 55.9% vs. 54.2%, a difference of 1.7 points; on SLAKE: 38.2% vs. 35.4%, a difference of 2.8 points) are not statistically distinguishable from noise.

The consequence. The paper's central claim — that MPO "consistently outperforms all baselines across image, video, and molecular domains" (Section 4.2) — is supported directionally (MPO is higher than every baseline on every dataset) but the magnitude and reliability of the gains, particularly for the smaller improvements (1–3 percentage points), are not statistically established. A practitioner deciding whether to adopt MPO based on the reported gains cannot assess whether a 1.6-point improvement on their target domain is a reliable effect they can expect to replicate, or a within-noise fluctuation that might disappear with a different random seed or test set split. This is especially relevant for high-stakes applications (medical VQA, drug property prediction) where decisions based on small performance differences carry material consequences.

The small test set size (300 examples) further compounds this: a 1.6-point difference on a 300-example test set corresponds to roughly 5 additional correct answers. The variance of a binomial proportion with 300 trials is approximately $p(1-p)/300$; at $p \approx 0.55$, the standard error is approximately $\sqrt{0.55 \times 0.45 / 300} \approx 0.029$, or 2.9 percentage points. This means the 95% confidence interval for a single test set evaluation is approximately ±5.7 points — wider than many of the reported gains. While the average over 3 runs reduces this variance somewhat (by a factor of $\sqrt{3}$ for independent runs), the paper does not provide the per-run scores, standard deviations, or confidence intervals needed to assess whether the differences are statistically significant.

What evidence exists in the paper. The paper states "Results are averaged over three independent runs" in the Table 1 caption, but no measure of variance is reported anywhere in the main results. Appendix C.2 provides full per-subtask results for image and molecule modalities (Tables 6 and 7), but these are also reported as averages over three runs without error bars. The scatter plot in Figure 4 shows median values with Q1 and Q3 bars for the cross-modal alignment analysis, but this is for an aggregated analysis rather than for the primary performance comparisons. The parent-child correlation analysis (Figure 3) reports a P-value ($3.33 \times 10^{-170}$), confirming that the paper can compute and report statistical measures when they are central to a specific claim, making their absence from the main performance comparisons more notable.

Mitigation status. Not addressed. The paper does not acknowledge the small test set or the lack of statistical reporting as limitations. This is a standard weakness in prompt optimization research (many papers report averages over 3 runs without error bars), but it limits the strength of the conclusions, particularly for the smaller gains. A practitioner should interpret the per-dataset gains with appropriate caution — the large gains (7–12 points on PlantVillage and CUB) are likely robust, while the small gains (1–3 points on DrivingVQA, RSVQA, SLAKE) may not be statistically significant given the test set sizes and number of runs.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose a new model architecture, a new training objective, or a new decoding strategy. Its contribution is more foundational: it redefines what a "prompt" is for the era of multimodal models and, in doing so, opens an entirely new axis of optimization that the field had overlooked. The shift is conceptual before it is technical — the paper argues convincingly that the text-only boundary on automatic prompt optimization was an inherited constraint from the LLM era, not a principled design choice, and that lifting this constraint yields consistent, substantial performance gains across three qualitatively different modalities (images, videos, molecules).

The magnitude of this shift is best characterized as opening a new subfield rather than solving an existing one. Multimodal prompt optimization was not a recognized problem before this work. The paper not only defines it formally — with a clean objective function over the product space T × M (Section 3.1) — but also provides the first complete framework for solving it, establishes evaluation benchmarks, and demonstrates empirical advantages over the strongest text-only methods. This is analogous to how the LLM test-time compute scaling paper defined "compute-optimal test-time scaling" as a distinct problem from pretraining scaling, rather than merely proposing a better search algorithm within an existing paradigm.

Several shifts in how researchers should think about prompt optimization follow directly from this work:

Prompt optimization is no longer a text-only problem. The most immediate consequence is that future work on APO for MLLMs cannot credibly ignore the non-textual modality. The paper demonstrates not just that multimodal prompts can help (which might be obvious — an image is worth a thousand words) but that they help in a way that is inaccessible to text-only optimization regardless of budget. The text-only methods compared in Table 1 include sophisticated approaches (ProTeGi, SEE) that use failure analysis, evolutionary operators, and in-context example optimization — yet MPO outperforms all of them by an average of 5.1 points, and by up to 12 points on individual datasets. The hidden state visualization (Figure 8) provides mechanistic evidence for why: multimodal prompts shift the MLLM's internal representations to a region that text-only prompts cannot reach, regardless of how well those text prompts are crafted. This implies that text-only APO faces a hard ceiling — not a soft limitation that can be overcome with better text optimization algorithms, but a fundamental expressiveness bound imposed by the modality itself.

Cross-modal alignment becomes a first-class optimization constraint. Prior work in multimodal learning has studied alignment extensively — in vision-language pretraining, in text-to-image evaluation, in multimodal representation learning — but always as a property of the model (how well the model aligns representations across modalities) or of the output (how well a generated image matches a text description). This paper treats alignment as a constraint on the optimization process itself: the prompt optimizer must not only find a high-performing prompt but must do so while ensuring that the text and non-textual components evolve coherently. The cross-modal alignment analysis (Figure 4) demonstrates that alignment quality and task performance are causally linked — breaking alignment (through sequential optimization, random image substitution, or mismatched image-text pairs) degrades performance in proportion to the alignment drop. This finding implies that future multimodal optimization methods must include explicit mechanisms for maintaining cross-modal coherence, whether through joint feedback (as in MPO's cohesive backpropagation), contrastive objectives, or alignment-aware selection criteria.

Parent-child inheritance is a viable strategy for efficient exploration in structured search spaces. The paper's third conceptual shift concerns how we think about candidate evaluation in prompt optimization. The standard paradigm — inherited from the hyperparameter optimization and neural architecture search literatures — treats each candidate as an independent arm. The paper demonstrates that this independence assumption is both wasteful and empirically false in prompt optimization: parent and child prompts are strongly correlated (Pearson's r = 0.88, Figure 3), and exploiting this correlation through prior-inherited Bayesian UCB reduces evaluation cost by 42–70% (Figure 5). This finding generalizes beyond multimodal prompt optimization: any iterative optimization process that uses beam search, evolutionary algorithms, or other parent-child generation structures can potentially benefit from prior inheritance, provided the parent-child correlation is positive. The paper provides both the empirical methodology for measuring this correlation and the theoretical justification (Proposition 3.1) for when the mechanism is guaranteed to help.

The optimization process becomes a first-class object of study. A subtler but equally important shift is the paper's treatment of the optimization trajectory itself as a source of information. The parent-child correlation analysis (Figure 3) and the training dynamics comparison against ProTeGi (Figure 7) represent a move toward studying not just what the optimizer finds but how it finds it — the dynamics of exploration, plateauing, and escape from local optima. This is methodologically significant because it suggests that the structural properties of the optimization process (correlation structure, convergence rate, plateau behavior) can be measured, analyzed, and exploited, rather than being treated as black-box noise.

Reconciliation of a latent contradiction. The paper implicitly resolves a tension that was emerging in the multimodal prompting literature: on one hand, instance-specific methods (MM-CoT, visual prompting with bounding boxes) demonstrated that multimodal signals can substantially improve per-query performance; on the other hand, task-level prompt optimization methods remained stubbornly text-only, with no clear path to incorporating non-textual information. The tension was between the demonstrated value of multimodal signals and the apparent difficulty of optimizing them at the task level. MPO resolves this by showing that task-level multimodal optimization is not only possible but can be made efficient through the right combination of exploration operators and selection strategies. The instance-specific and task-level approaches are not competitors — they operate at different granularities — and MPO establishes that both are viable and complementary.

Research directions that become more attractive. The paper makes the case that improving modality-specific generators is a high-leverage investment for multimodal prompt optimization — better generators (GPT-Image vs. SANA1.5) produce better multimodal prompts, and this relationship appears monotonic (Table 2, Bottom Right). It also directs attention toward developing better difficulty estimation or task-level diagnostics that can predict, a priori, whether a given task will benefit from multimodal prompt optimization. And it opens the door to combining text-only APO methods with multimodal exploration — since MPO's optimizer is decoupled from its generator, any improvement in text prompt optimization (better failure analysis, more sophisticated evolutionary operators) can be plugged into MPO's framework to generate better textual conditions for multimodal generation.

Research directions that become less attractive. The paper's results suggest that purely text-only prompt optimization for MLLMs has hit diminishing returns — the best text-only methods (ProTeGi, SEE) achieve similar performance on most tasks, and further text-only optimization beyond a few iterations yields marginal gains (Figure 7, ProTeGi's plateau). Investing effort in more sophisticated text-only optimization algorithms without addressing the modality gap is likely to yield small improvements at best. Similarly, methods that treat text and non-textual prompt components as independently optimizable (sequential optimization) are less attractive given the evidence that cross-modal alignment requires joint optimization — Figure 4 shows that even a simple sequential approach underperforms MPO's joint optimization, despite using the same underlying models and budget. Finally, candidate selection strategies that assume independence between arms (uniform allocation, standard UCB without informative priors) become less attractive given the demonstrated efficiency gains from prior inheritance — the 42% budget reduction relative to prior-free UCB (Figure 5) is large enough that ignoring parent-child structure is simply wasteful for any beam search or evolutionary optimizer where such structure exists.

Follow-Up Research This Work Enables

1. Cheap difficulty estimation for multimodal tasks before full optimization. The paper observes — but does not systematically exploit — that the benefit of multimodal prompt optimization varies dramatically across subtasks. On CUB, MPO's gain over ProTeGi ranges from +4.9 points (cormorant classification) to +31.3 points (auklet classification). This heterogeneity implies that some tasks are "text-saturating" (text prompts already capture most of the achievable performance, and multimodal prompts add little) while others are "text-starved" (visual features are inherently hard to describe in words, and multimodal prompts provide large gains). A practical user needs to know which category their task falls into before investing in multimodal optimization, which requires generator calls and the full MPO pipeline. A strong follow-up would train a lightweight classifier that takes as input only the task dataset (query-answer pairs, or even just a few dozen examples with the base model's text-only predictions) and predicts the expected gain from multimodal prompt optimization — effectively, a difficulty estimator for the multimodal prompt space analogous to the pre-optimization difficulty estimation in the LLM test-time compute paper. The training signal would be the observed gain (MPO performance minus ProTeGi performance) across many tasks, and the classifier could use features like: the variance of base model confidence on text-only prompts, the semantic diversity of the label space, or the performance gap between 1-shot and 5-shot text-only prompting (a cheaper proxy for whether more information helps). The paper provides the raw data needed to build such a classifier — the per-subtask results in Tables 6 and 7, with matched text-only baseline scores available for direct gain computation on dozens of subtasks across images and molecules.

2. Dynamic operator scheduling with learned or adaptive mixing ratios. The paper allocates the three exploration operators (generation, edit, mix) uniformly — each produces roughly one-third of child candidates per iteration — and the operator choice is random. This is a reasonable starting point, but the optimization trajectory in Figure 6 (CUB grosbeak classification) shows that operators serve qualitatively different roles at different stages: early iterations use Edit for refinement, middle iterations use Mix heavily for recombination, and later iterations alternate between Generation (escape local optima) and Edit (fine-tune). This pattern suggests that a dynamic operator schedule could substantially improve efficiency — using more Edit early to refine from initial candidates, more Mix in the middle to recombine diverse strengths, and more Generation late to prevent stagnation. A concrete experiment: train a small meta-controller (perhaps a lightweight LLM or a simple RL policy) that observes the diversity of the current beam (e.g., pairwise performance differences between beam members) and the rate of improvement over recent iterations, and selects which operator to apply next. Compare against the uniform-random baseline on the full 10-dataset benchmark. The null result (no improvement from dynamic scheduling) would be informative — it would suggest that the operators are sufficiently complementary that even random mixing explores the space effectively, or that the optimization landscape is smooth enough that local refinement (Edit) is almost always the right move. A positive result would quantify how much efficiency is left on the table by uniform allocation.

3. Stress-testing MPO on tasks without clean correctness signals (learned verifiers). The paper evaluates exclusively on tasks with well-defined ground-truth answers (classification accuracy, exact-match VQA, binary property prediction). This leaves open the question of whether MPO's mechanisms — particularly cohesive backpropagation and prior-inherited UCB — remain effective when the reward signal is noisy, approximate, or multi-dimensional. A strong follow-up would replicate MPO on open-ended generation tasks using a learned verifier as the reward signal: for example, summarization quality evaluated by a fine-tuned reward model (e.g., a Llama-based judge), or dialogue quality evaluated by an ensemble of automated metrics (ROUGE, BLEURT, G-Eval). The key question is whether the parent-child correlation (Figure 3) degrades under noisy reward — if parent and child performance are less correlated when measured by a learned verifier rather than exact-match accuracy, the prior-inherited UCB mechanism loses its empirical foundation, and the selection strategy would need to be adjusted (e.g., by reducing the prior strength S as a function of estimated reward noise). This experiment would also test whether cohesive backpropagation can generate useful feedback when the failure set is defined by a learned verifier's judgment rather than binary correctness — can the optimizer LLM produce meaningful diagnostics from errors like "the summary scored 0.6/1.0 on conciseness" rather than "the answer was wrong"? The null result (MPO provides no benefit over text-only APO on open-ended tasks) would establish an important boundary condition: multimodal prompt optimization is effective only when the optimization signal is clean, and alternative approaches (human-in-the-loop, learned verifiers with uncertainty quantification) are needed for subjective tasks. A positive result (MPO retains gains) would dramatically expand the practical applicability of the framework.

4. Combining MPO with instance-specific multimodal prompting for compound gains. The paper distinguishes MPO from per-query multimodal methods (MM-CoT, visual prompting with bounding boxes) and evaluates only against task-level baselines. A natural extension is to stack MPO's task-level multimodal prompt with instance-specific prompting and measure whether the gains are additive, sub-additive, or super-additive. For example: on CUB, first use MPO to discover an optimized multimodal prompt (text + reference image), then apply MM-CoT (generating intermediate textual rationales per query) on top of MPO's prompt, and measure whether the combined system outperforms either method alone. The key hypothesis is that MPO provides better global context (what features to attend to, how to compare across species) while instance-specific methods provide better local adaptation (where the specific bird's beak is, what lighting conditions affect this particular image). If the gains are super-additive, it suggests task-level and instance-level optimization are complementary rather than competing approaches. The experiment requires careful control: compare (a) Human text prompt + MM-CoT, (b) MPO multimodal prompt only, (c) MPO multimodal prompt + MM-CoT, and (d) an upper bound using both MPO and per-query oracle information (e.g., ground-truth bounding boxes). The gap between (c) and (d) quantifies how much of the remaining error is due to imperfect instance-level adaptation versus fundamental model limitations.

5. Extension to additional modalities: audio, 3D point clouds, and structured data. The paper validates MPO on images, videos, and molecules — three modalities that differ in their temporal structure (static vs. dynamic) and representation (continuous pixel space vs. discrete SMILES strings). The framework is architecturally agnostic to modality — the optimizer generates textual conditions, and the generator produces modality-specific outputs — so extending to new modalities requires only: (a) a modality-specific generator (text-to-audio, text-to-3D, text-to-graph), and (b) task datasets with ground-truth correctness signals. A strong follow-up would validate on audio classification (e.g., ESC-50 for environmental sound classification, where a reference audio clip of "dog barking" could be more informative than a text description of barking), 3D object recognition (e.g., ModelNet40, where a reference 3D point cloud could show canonical poses of object categories), or tabular data interpretation (where a reference table or chart could demonstrate the expected schema). The key question is whether the three-operator taxonomy (generation, edit, mix) remains sufficient for these modalities — for audio, "edit" might mean adding background noise or changing pitch; for 3D, "edit" might mean rotating or scaling; for tabular data, "edit" might mean adding or removing columns. If the operators map naturally, MPO's architecture transfers with minimal modification. If not, the follow-up would need to define modality-specific operators that provide comparable breadth, depth, and combinatorial exploration. The null result — MPO provides no benefit for a particular modality — would help characterize where multimodal prompts are useful (when the non-textual modality provides complementary information to text, as images do for visual classification) versus where they are redundant (when the non-textual modality is just an alternative representation of the same information already available in text, as SMILES strings might be for some molecular properties).

6. Systematic analysis of when and why multimodal prompts fail (error taxonomy). The paper provides qualitative examples of MPO's successes (Tables 14, 16) and a single text-only failure case (Table 15), but does not categorize the residual errors that remain after MPO optimization. A diagnostic follow-up would take the MPO-optimized prompts on each dataset and manually analyze the remaining errors, building a taxonomy like: (a) model capability errors — the base MLLM lacks the knowledge or reasoning capacity to answer correctly regardless of prompt (e.g., SLAKE medical VQA questions requiring specialist radiology knowledge not present in the model's training data); (b) prompt ambiguity errors — the multimodal prompt is good but not perfect, and further optimization could fix specific remaining ambiguities (e.g., the reference image shows four bird species but the lighting makes two of them indistinguishable); (c) cross-modal conflict errors — the text and image prompts provide subtly conflicting signals (e.g., the text says "prioritize beak shape" but the reference image emphasizes body size, causing the model to weigh features incorrectly); (d) generator limitation errors — the text-to-image generator cannot produce a reference with sufficient fidelity (e.g., molecular structures with specific stereochemistry that GPT-4o mini cannot accurately generate). This taxonomy would directly inform where to invest effort: if most errors are type (a), better prompts cannot help and investment should go into better base models or domain-specific fine-tuning; if most are type (b) or (c), improved optimization algorithms (better operators, more iterations, stronger optimizer models) could yield further gains; if most are type (d), investment should go into better modality-specific generators. The paper provides the raw outputs needed for this analysis — MPO's optimized prompts and the model's predictions on test examples — and the analysis could be done on the existing datasets without additional experiments.

Practical Applications and Downstream Use Cases

1. Automated prompt engineering for domain-specific multimodal applications. The most direct application is using MPO to automatically generate optimized prompts for specialized MLLM deployments where manual prompt engineering is expensive or where domain experts are scarce. Consider a healthcare provider deploying an MLLM for dermatology triage — patients upload images of skin conditions, and the model must classify them into categories requiring different levels of urgency. A dermatologist could manually craft a text prompt and select reference images, but this is time-consuming and the expert's time is expensive. MPO could take a dataset of labeled dermatology images, a base MLLM (e.g., a medically fine-tuned Qwen variant), and a text-to-image generator, and automatically discover a multimodal prompt that maximizes triage accuracy. The paper's results on PlantVillage (plant disease classification, from 42.2% human-designed to 76.4% MPO-optimized) provide a rough magnitude estimate — in a domain where visual features are discriminative and the base model has relevant training, MPO can roughly double the accuracy of a handcrafted prompt. The optimization cost (hundreds of model calls, Table 5) is trivial compared to the cost of expert time for manual prompt engineering, and the optimized prompt can be deployed to serve thousands of patients.

2. Cost-efficient deployment of smaller MLLMs with optimized multimodal prompts. The paper's generalizability results (Table 2, Top) show that MPO's gains are largest on the strongest base model (Qwen2.5-VL 72B: +6.3 points over ProTeGi) but still substantial on smaller models (Gemma3 12B: +5.0 points; GPT-4.1 nano: +4.3 points). This creates an opportunity for model compression through prompt quality: rather than deploying a large, expensive MLLM with a basic prompt, deploy a smaller, cheaper MLLM with an MPO-optimized multimodal prompt. The paper's data suggests this could be competitive or superior in some regimes — the optimized prompt on a 7B model might match or exceed the unoptimized prompt on a 14B model. A concrete deployment scenario: a mobile app that identifies bird species from user photos. Running a 72B-parameter model on-device is infeasible; running a 7B model with an MPO-optimized multimodal prompt (78.6% on CUB, Table 1) could provide accuracy comparable to a much larger model with a basic prompt. The optimization is a one-time cost amortized over millions of user queries. The paper does not directly measure the model-size-vs-prompt-quality tradeoff curve, but the generalizability results provide the key data points needed to estimate it: take the performance of model A with human prompt, model B with human prompt, and model A with MPO prompt, and identify where MPO on the smaller model crosses the larger model's human-prompt baseline.

3. Bootstrapping training data for domain-specific MLLM fine-tuning. A common workflow for adapting MLLMs to specialized domains involves: (1) collect labeled data, (2) manually engineer prompts, (3) use the prompted model to generate additional training data (e.g., through rejection sampling or self-training), and (4) fine-tune on the expanded dataset. Step (2) is a bottleneck — if the initial prompt is poor, the generated training data will be noisy, and the fine-tuned model inherits this noise. MPO can automate step (2) and, critically, can operate on relatively small labeled datasets — the paper uses training sets as small as 80 examples for some CUB subtasks, and MPO's evaluation budget per prompt scales with available training data (reduced to one-third of instances for small CUB subtasks). Once MPO discovers a high-quality multimodal prompt, that prompt can be used to generate additional training data with higher quality, which in turn improves the fine-tuned model. The paper's results on CUB — where MPO achieves 78.6% on a 7B model — suggest that MPO-generated prompts could produce training data of substantially higher quality than human-designed prompts, creating a virtuous cycle where better prompts enable better training data, which enables better fine-tuned models, which could then be used with even better prompts. This application is directly suggested by the paper's demonstration that MPO works with small training sets and provides consistent gains across tasks.

4. Accelerated molecular property screening in drug discovery pipelines. The molecular modality results (Table 7) are particularly relevant for pharmaceutical applications. In early-stage drug discovery, computational screening of candidate molecules for properties like oral absorption (HIA), blood-brain barrier penetration (BBBP), and cytochrome P450 inhibition (CYP) is used to prioritize which molecules to synthesize and test experimentally. Traditional approaches use specialized cheminformatics models (random forests, graph neural networks) trained on molecular fingerprints. The paper demonstrates that MLLMs (specifically Qwen3 8B) prompted with MPO-optimized multimodal prompts (text + reference SMILES molecules) achieve competitive performance — e.g., 67.6% F1 on BBBP, 64.5% average F1 on Absorption. A pharmaceutical company could deploy MPO to: (a) optimize the prompt for their specific screening task and molecule library, using their existing labeled data as the training set; (b) screen new candidate molecules with the optimized prompt, achieving higher accuracy than with a generic prompt; and (c) iterate — as new experimental data becomes available, re-run MPO to update the prompt. The cost of prompt optimization (hundreds of model calls, Table 5) is negligible compared to the cost of synthesizing and testing even a single drug candidate, and a 2–5 percentage point improvement in screening accuracy (which MPO provides over text-only methods on molecular tasks, Table 1) could translate to millions of dollars in saved experimental costs by better prioritizing which molecules to advance. The paper's demonstration that MPO works with GPT-4o mini as the molecular generator (no external chemistry-specific generator needed) makes this application particularly practical, as the entire pipeline can run with off-the-shelf LLM APIs.

When to Prefer This Method

The paper does not explicitly position MPO against a specific alternative multimodal prompt optimization framework (no such framework existed prior to this work). Rather, it defines the problem and provides the first solution. The practical decision facing a practitioner is not "MPO vs. Alternative Multimodal Optimizer" but rather "MPO vs. text-only optimization" and "MPO vs. manual multimodal prompt design." The paper's results support the following decision guidance, grounded in the empirical findings:

  • Prefer MPO over text-only optimization when: (1) the task involves visual, temporal, or structural information that is difficult to describe exhaustively in text — fine-grained classification (CUB: +8.6 points over ProTeGi), disease identification from images (PlantVillage: +12.0 points), molecular property prediction where SMILES strings are opaque to non-experts (Absorption: +5.6 points over ProTeGi); (2) the base MLLM has reasonable zero-shot or few-shot performance on the task (suggesting it possesses relevant knowledge that better prompts can surface), but text-only prompts leave a substantial gap to ceiling — if the human-designed text prompt achieves 40–60%, MPO has headroom to provide 10+ point gains; (3) the deployment scenario allows a one-time optimization cost to be amortized over many future queries, making the additional generator calls (Table 5) economically justified; and (4) a modality-specific generator of sufficient quality is available — even a lightweight open-source generator (SANA1.5, 1.6B) suffices (Table 2, Bottom Right), so cost is not prohibitive.

  • Prefer text-only optimization (or manual prompting) over MPO when: (1) the task can be fully specified in text without ambiguity — DrivingVQA lane-marking questions show only +1.6 points over ProTeGi, suggesting the additional visual reference provides minimal marginal value; (2) the base MLLM's performance is very low regardless of prompt quality (e.g., below 30% with a human-designed prompt), suggesting the bottleneck is model capability rather than prompt design — MPO cannot create capabilities that do not exist in the base model; (3) the deployment requires frequent re-optimization with tight latency constraints, making MPO's sequential beam search loop (13 iterations × sequential UCB evaluation rounds) impractical in wall-clock time; or (4) the task lacks clean, automatable correctness signals (open-ended generation, subjective evaluation) and no reliable learned verifier is available — MPO's optimization loop depends on binary or scalar reward signals, and the behavior with noisy or learned reward models is untested.

  • When deploying MPO, the practical configuration choices are: use beam size b = 3 and iterations T = 13 (established as effective defaults), set prior strength S to 10% of the per-prompt evaluation budget (optimal in the sensitivity analysis, Figure 9), use the lightest generator that meets quality requirements (SANA1.5 for images if proprietary APIs are unavailable), and allocate evaluation budget based on available training data — as few as 30–50 examples per candidate suffice (the paper uses budgets of 100 where data permits, and one-third of available instances for small CUB subtasks). Expect the optimization to require roughly as many base model calls as text-only optimization (same number of explored prompts), plus a manageable additional cost for generator calls (~117 per run with the default settings). The one-time cost should be evaluated against the cumulative performance gain over all future inference requests.