Scaling Catalog Attribute Extraction with Multi-modal LLMs

🎯 Pitch

LLMs can act as black-box optimizers purely through iterative prompting, discovering that the strange instruction "Take a deep breath and work on this problem step-by-step" boosts GSM8K math accuracy to 80.2%, far above the human-designed "Let's think step by step" (71.8%). The method simply shows an LLM previous solutions and their scores, then asks it to generate a better one.

1. Executive Summary

This paper introduces Optimization by PROmpting (OPRO), a method that uses large language models as black-box optimizers by describing the optimization task in natural language and iteratively prompting the LLM to generate new candidate solutions based on previously evaluated solutions and their scores in a meta-prompt. The approach is first demonstrated on small-scale linear regression and Traveling Salesman Problem instances—where LLMs sometimes match hand-designed heuristics—and then applied to prompt optimization on GSM8K and Big-Bench Hard with models including PaLM 2-L and text-bison, where the optimizer LLM generates instruction strings that are evaluated by a separate scorer LLM for task accuracy. Starting from low-accuracy initial prompts, OPRO discovers instructions that outperform human-designed prompts by up to 8% on GSM8K with zero-shot prompting (e.g., "Take a deep breath and work on this problem step-by-step" reaching 80.2% accuracy versus the 71.8% of "Let's think step by step") and by up to 50% on Big-Bench Hard tasks, establishing that an LLM can serve as an effective optimizer solely through iterative prompting that leverages the full optimization trajectory—without requiring gradient access, formal problem specification, or model fine-tuning—though the optimizer is sensitive to the starting point and struggles when the loss landscape is too bumpy or the problem scale exceeds the context window.

2. Context and Motivation

The Core Problem: Optimizing Without Gradients, Formally Specified Objectives, or Model Access

The fundamental problem this paper addresses is how to perform optimization when the objective function is a black box — no gradients are available, the function may be non-differentiable, and in many practical settings, you cannot even formally write down what you're optimizing. Consider the task of "find a prompt string that maximizes a language model's accuracy on math word problems." The search space is the discrete, combinatorial space of all possible natural language strings. The objective function — running that prompt on a held-out set and measuring accuracy — is expensive to evaluate, provides no gradient signal, and has a landscape that is notoriously sensitive: semantically similar prompts can yield drastically different accuracies (the paper notes that "Let's think step by step" achieves 71.8% while the semantically combined "Let's work together to solve this problem step by step" scores only 49.4%). This is a genuine optimization problem, but none of the standard optimization toolkits apply.

This gap matters for several reasons the paper establishes:

Prompt engineering is high-stakes and currently ad-hoc. LLMs are known to be extremely sensitive to prompt format (Zhao et al., 2021; Lu et al., 2021; Wei et al., 2023). The difference between a well-crafted and a mediocre prompt can be 20+ percentage points of accuracy on the same task. Yet the process of finding good prompts is largely manual — researchers and engineers rely on intuition, trial-and-error, and folklore ("add 'step by step'"). This is expensive, time-consuming, and unlikely to find optimal prompts for every model-task combination, especially given that "the optimal prompt formats can be model-specific and task-specific" (Section 1).

API-only access to LLMs is increasingly common. When the LLM is available only through an API (as with GPT-4 or text-bison), methods that require gradient access to the model — soft prompt tuning (Lester et al., 2021; Li & Liang, 2021), gradient-guided discrete search (Shin et al., 2020; Wen et al., 2023), or reinforcement learning that backpropagates through the model (Deng et al., 2022) — are simply inapplicable. An optimization method that works purely through the API's text-in/text-out interface is needed.

Beyond prompts: many real-world optimization problems lack formal specification. The paper's motivating examples of linear regression and the Traveling Salesman Problem illustrate a broader point: in many practical settings, even when we can formally define the problem (we know what TSP is), we may want an optimizer that works from a natural language description alone, without requiring a programmer to implement a specialized solver. The vision is that a general-purpose LLM could serve as a universal optimizer — describe your objective in English, provide some initial attempts with their scores, and let the LLM iteratively propose improvements.

Where Existing Approaches Fall Short

The paper identifies specific limitations across several categories of prior work:

Gradient-based prompt optimization requires model internals. Soft prompt-tuning methods (Lester et al., 2021; Li & Liang, 2021; Liu et al., 2021; Qin & Eisner, 2021) learn continuous vector embeddings prepended to the input, optimizing them via gradient descent on the training objective. While effective, these methods require full access to model weights and gradients — impossible with API-only models. Gradient-guided discrete prompt search (Shin et al., 2020; Wen et al., 2023; Gao et al., 2020) similarly requires gradient signals from the model. Reinforcement learning approaches for prompt optimization (Deng et al., 2022; Zhang et al., 2023) also typically require model-internal access or are designed around training explicit policy networks.

Edit-based and paraphrase-based prompt optimization is too constrained. Some prior work operates by making local edits to an existing prompt. APE (Zhou et al., 2022b) generates initial instructions via the LLM, then prompts the LLM to produce semantically similar variants of the best instructions — effectively constraining the search to a small neighborhood around good prompts. APO (Pryzant et al., 2023) uses the LLM to generate natural language feedback on how to improve an instruction, then edits it accordingly. Other edit-based methods use human-defined operations like swapping phrases (Prasad et al., 2022) or automated paraphrasing via back-translation (Xu et al., 2022). The key limitation across all of these is that they operate on a single prompt at a time, attempting to incrementally improve it. They do not leverage the full optimization trajectory — the history of many previous attempts with their scores — to identify patterns of what makes prompts work. As the paper states:

"Different from edit-based approaches, the optimizer LLM in our work directly generates new instructions at each optimization step, and the optimizer LLM is merely asked to improve the task accuracy without being required to imitate past instructions."

Concurrent evolutionary approaches (EvoPrompt) miss task context. Some concurrent work proposes meta-prompts that explicitly instruct the LLM to perform genetic algorithm operations — mutation and crossover of existing prompts (Fernando et al., 2023; Guo et al., 2023). The paper's experimental comparison with EvoPrompt (Section 5.5, Figure 12) reveals a critical weakness: because EvoPrompt does not include task exemplars in its meta-prompt, it lacks understanding of what the task actually is. When starting from generic initial prompts ("Let's solve the problem" and "Here is the answer") on GSM8K, EvoPrompt actually degrades performance — it mutates and crosses over prompts without understanding whether the mutations are relevant to the task. The paper demonstrates that providing exemplars fixes this (EvoPrompt works better on BBH sports_understanding when given task-specific initial instructions), but the deeper point is that optimization without task understanding is blind.

Mathematical optimization typically requires formal problem specification. Standard optimization algorithms — gradient descent (Amari, 1993; Qian, 1999), evolutionary algorithms (Bäck & Schwefel, 1993), derivative-free methods (Rios & Sahinidis, 2013) — all require the problem to be formally specified: decision variables defined, constraints encoded, objective function implemented in code. The paper's vision is that natural language description of the problem, combined with the LLM's general reasoning capabilities, could replace this formal specification step entirely for some problem classes.

Prior work on LLMs for optimization uses them as operators, not as the optimizer itself. Some work has used language models as mutation and crossover operators within evolutionary algorithms (Meyerson et al., 2023; Lehman et al., 2022; Chen et al., 2023a), but these approaches still rely on an external optimization framework — the LLM is a tool within a larger algorithm, not the decision-making core. OptFormer (Chen et al., 2022) trains a transformer model on large collections of hyperparameter optimization data to serve as a learned hyperparameter optimizer. In contrast, OPRO performs optimization solely through prompting, without any additional training, external algorithmic framework, or formal problem encoding.

How This Paper Positions Itself

The paper positions OPRO as a new paradigm for optimization where the LLM is the optimizer itself — not a component within an optimization algorithm, but the entity that observes previous solutions and their scores, reasons about what makes them good or bad, and proposes new candidates. This is fundamentally different from using LLMs as mutation operators or edit-based improvers because:

The full optimization trajectory informs each decision. The meta-prompt contains a sorted list of many past solutions with their scores, enabling the LLM to identify patterns — "solutions containing the phrase 'step by step' tend to score higher," or "shorter instructions are performing better" — without these patterns needing to be explicitly programmed. The paper states this explicitly: "Including optimization trajectory in the meta-prompt allows the LLM to identify similarities of solutions with high scores, encouraging the LLM to build upon existing good solutions to construct potentially better ones without the need of explicitly defining how the solution should be updated" (Section 2.2).
Task understanding comes from exemplars, not formal specification. Rather than requiring the user to define the optimization problem in mathematical terms, the meta-prompt includes a few input-output examples from the task. This is a key design choice validated by the ablation studies (Section 5.3, Figure 7e-f): removing exemplars causes performance to collapse, while adding more beyond 3 does not help. The exemplars give the optimizer LLM a concrete sense of what the task requires, which in turn shapes what kinds of instructions will be effective.
The optimization process is iterative and self-correcting. Each step generates multiple candidate solutions (typically 8), evaluates them against the objective function, and incorporates the results into the next step's meta-prompt. This creates a feedback loop where the LLM can see which of its proposals succeeded and which failed, gradually steering toward higher-scoring regions of the solution space. The paper shows this process works even starting from very weak initial instructions: starting from an empty string with 34.0% accuracy on GSM8K, OPRO eventually finds prompts exceeding 80%.
Different LLMs serve as different "optimization algorithms" with different characteristics. The paper demonstrates that pre-trained PaLM 2-L, instruction-tuned PaLM 2-L, text-bison, gpt-3.5-turbo, and gpt-4 all successfully optimize prompts, but produce instructions of different styles and with different convergence speeds. This is analogous to how different gradient-based optimizers (SGD, Adam, RMSprop) have different behaviors on the same problem.

The paper also explicitly positions itself relative to the exploration-exploitation tradeoff, a fundamental challenge in optimization. The meta-prompt design and temperature parameter provide knobs to balance these: including high-scoring past solutions encourages exploitation (building on what works), while the sampling temperature controls exploration (higher temperature produces more diverse, potentially novel solutions). The ablation study on temperature (Figure 10) shows that temperature 1.0 achieves the best balance — lower temperatures get stuck exploiting the same solution, while higher temperatures ignore the trajectory and fail to exploit.

Finally, the paper acknowledges clear boundaries for the approach. It is not designed to "outperform state-of-the-art gradient-based optimization algorithms for continuous mathematical optimization, nor surpass the performance of specialized solvers for classical combinatorial optimization problems" (Section 3.2, Limitations). The goal is to demonstrate that LLMs can optimize through prompting — that the capability exists and has practical utility for the important application of prompt optimization — while being transparent about where the approach fails (bumpy loss landscapes like the Rosenbrock function, large-scale problems exceeding context windows, and difficulty navigating from poor starting regions).

3. Technical Approach

3.1 Reader Orientation

OPRO is a meta-algorithm — a procedure that uses an LLM as the decision-making core of an optimization loop, where the LLM proposes candidate solutions, observes their quality (via an external evaluator), and uses that feedback to propose improved solutions in subsequent iterations. It solves the problem of black-box optimization over discrete, unstructured solution spaces described in natural language, where gradient-based methods cannot apply and where the solution space is too large and poorly structured for exhaustive search, all without requiring any model training or access to model internals.

3.2 Big-Picture Architecture (Diagram in Words)

The OPRO system has four components connected in a loop:

The meta-prompt — a text template that combines the optimization problem description, a set of task exemplars, and the historical trajectory of previously evaluated solutions with their scores. This is the input to the optimizer LLM.
The optimizer LLM — any instruction-following LLM (e.g., PaLM 2-L-IT, gpt-3.5-turbo, gpt-4) that reads the meta-prompt and generates new candidate solutions (e.g., new instruction strings for prompt optimization, new (w, b) pairs for linear regression, new TSP routes).
The objective function evaluator — an external process that computes the quality score of each candidate solution. For prompt optimization, this is a separate "scorer LLM" (can be the same or different from the optimizer) that uses each candidate instruction to answer questions from a training set, with accuracy as the score. For mathematical optimization, this is a direct computation (e.g., mean squared error for linear regression, total route length for TSP).
The optimization trajectory — a sorted list maintained across iterations that records each generated solution and its score. This list is fed back into the meta-prompt for the next iteration, creating a feedback loop.

Information flows cyclically: the meta-prompt is constructed → the optimizer LLM generates 8 candidate solutions → the evaluator scores each candidate → the trajectory is updated with new (solution, score) pairs → the meta-prompt for the next iteration is rebuilt with the updated trajectory → repeat for up to 200 steps or until convergence.

3.3 Roadmap for the Deep Dive

This section proceeds in four layers, building from the foundational principles to the complete system:

First, the formal desiderata for LLM-based optimization — what properties an LLM must exhibit to serve as an effective optimizer, and why these properties matter. This establishes why the meta-prompt and generation procedure are designed the way they are.
Second, the meta-prompt design — the structure of the input that the optimizer LLM receives, including the two essential components (problem description and optimization trajectory), their formatting conventions, and the design rationale behind each choice. This is the "program" that tells the LLM what to optimize and how.
Third, the solution generation mechanism — how the optimizer LLM produces candidates, including the key design decisions around batch generation (8 per step), temperature control for exploration-exploitation balance, and the handling of the meta-instructions that regularize outputs.
Fourth, the complete optimization loop for both mathematical optimization (linear regression and TSP, Sections 3.1–3.2) and prompt optimization (Section 4) — how the meta-prompt, optimizer, evaluator, and trajectory interact across iterations, with all hyperparameters, dataset splits, and evaluation protocols specified.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical methods paper whose core idea is that LLMs can serve as black-box optimizers when given a meta-prompt containing (a) a natural language description of the optimization problem with exemplars and (b) a sorted history of previously evaluated solutions with their scores, enabling the LLM to identify patterns in high-scoring solutions and iteratively propose improvements without any gradient signal, formal problem encoding, or model fine-tuning.

Desiderata for LLM-Based Optimization

The paper establishes two fundamental requirements that any LLM-based optimizer must satisfy (Section 2.1). These are not implementation details — they are constraints that shape every design choice in the meta-prompt and generation procedure.

Desideratum 1: Making Use of Natural Language Descriptions. The optimizer must be able to understand the optimization task from a natural language description alone, without requiring the user to formally specify decision variables, constraints, or the objective function in mathematical notation. The paper frames this as the primary advantage over traditional optimization algorithms: "people [can] describe their optimization tasks without formal specifications" (Section 2.1). For prompt optimization specifically, the task description takes the form of a high-level text summary plus several input-output exemplars that implicitly define what success looks like. For mathematical optimization, the description includes the problem statement (e.g., "minimize a function with two input variables w, b") and the format of candidate solutions.

Desideratum 2: Trading Off Exploration and Exploitation. The optimizer must balance two competing goals that are fundamental to all optimization (Section 2.1). Exploitation means focusing on regions of the solution space near already-discovered good solutions — refining and locally improving known winners. Exploration means venturing into entirely new regions of the solution space to avoid missing potentially better solutions that are qualitatively different from anything seen so far. The paper explicitly states that "it is important for LLMs serving as optimizers to balance these two competing goals" and that the optimizer "should be able to exploit promising areas of the search space where good solutions are already found, while also exploring new regions of the search space so as to not miss potentially better solutions." The primary mechanism for controlling this tradeoff is the sampling temperature applied during solution generation (Section 2.3), but the meta-prompt design also contributes: including a sorted trajectory of past solutions implicitly shows the optimizer which regions are promising (encouraging exploitation), while generating multiple candidates per step and using exemplar randomization provides diversity (enabling exploration).

Meta-Prompt Design: The Two Essential Components

The meta-prompt is the sole interface between the user and the optimizer LLM — it encodes everything the optimizer needs to know about the problem and the optimization history. The paper identifies two essential parts (Section 2.2), each serving a distinct role.

Component 1: Optimization Problem Description. This is the text that tells the optimizer LLM what it is optimizing. The description includes:

The objective function specification. For prompt optimization, the meta-instruction tells the optimizer to "generate a new instruction that achieves a higher accuracy" (Section 2.2). For linear regression, the meta-instruction says "Give me a new (w, b) pair that is different from all pairs above, and has a function value lower than any of the above" (Figure 19, Appendix C.1). For TSP, the instruction is "Give me a new trace that is different from all traces above, and has a length lower than any of the above" (Figure 20, Appendix C.1).
Solution constraints and format specifications. The meta-prompt specifies the desired output format (e.g., "The output must end with a pair [w, b], where w and b are numerical values" for linear regression; "Write the text in square brackets" for prompt optimization with PaLM 2-L-IT, Figure 3). These are essentially parsing instructions that make the generated solutions machine-readable.
Customized meta-instructions as informal regularization. The paper notes that users "can also provide customized meta-instructions as an informal regularization of the generated solutions, such as 'the instruction should be concise and generally applicable'" (Section 2.2). This is a subtle but important point: unlike traditional optimization where constraints must be formally encoded, OPRO allows users to express preferences and soft constraints in natural language, which the LLM interprets and (imperfectly) follows.
Task exemplars. For prompt optimization, the problem description includes "several exemplars randomly selected from the training set to exemplify the task of interest" (Section 1). These are complete input-output pairs from the task, with the generated instruction's insertion point marked by <INS>. For example, Figure 3 shows a GSM8K exemplar: the input is a math word problem, the output is the numerical answer, and <INS> marks where the generated instruction will be inserted (at A_begin, meaning at the start of the model's answer). The exemplars serve three functions simultaneously: they demonstrate the task format, they show the instruction insertion position, and they give the optimizer LLM a concrete sense of what kinds of reasoning the task requires.

Component 2: Optimization Trajectory. This is the historical record that tells the optimizer LLM how previous attempts have performed. The trajectory includes:

Past solutions and their optimization scores, sorted in ascending order. The paper emphasizes the sorting direction: "the optimization trajectory includes past solutions and their optimization scores, sorted in the ascending order" (Section 2.2). This means the worst solutions appear first in the meta-prompt and the best solutions appear last. The ablation study (Section 5.3, Figure 7a-b) confirms this ordering matters: ascending order (worst-to-best) outperforms both descending (best-to-worst) and random ordering. The paper hypothesizes that "the optimizer LLM output is affected more by the past instructions closer to the end of the meta-prompt," consistent with the recency bias documented in Zhao et al. (2021) — LLMs are more likely to generate tokens similar to those appearing near the end of the prompt. By placing the best solutions last, the optimizer is more strongly influenced by high-quality examples when generating new candidates.
Score representation. The default approach is to round accuracy scores to integers, which the paper describes as "equivalent to bucketizing the accuracy scores to 100 buckets" (Section 5.3). The ablation study compares this to bucketizing into 20 buckets (coarser granularity) and to omitting scores entirely (keeping only the solutions in ascending order but without their scores). Figure 7c-d shows that having scores at all is crucial — without scores, the optimizer loses the ability to distinguish quality differences among past solutions — but that 100 buckets (integer rounding) provides sufficient granularity.
Trajectory length management. Because LLM context windows are finite, the meta-prompt cannot include all historical solutions. The paper defaults to keeping "the best 20 instructions so far" (Section 5.1) in the meta-prompt. For mathematical optimization with smaller solution representations, the same principle applies: "the best 20 (w, b) pairs in history" are retained for linear regression (Section 3.1).

The paper explicitly states the rationale for including the trajectory: it "allows the LLM to identify similarities of solutions with high scores, encouraging the LLM to build upon existing good solutions to construct potentially better ones without the need of explicitly defining how the solution should be updated" (Section 2.2). In other words, the trajectory encodes the direction of improvement implicitly — the LLM can observe that solutions with certain properties (e.g., containing "step by step," being concise, using specific vocabulary) tend to cluster at the high end of the sorted list, and can generate new solutions that share those properties.

Solution Generation: Batch Sampling, Temperature, and Meta-Instructions

At each optimization step, the optimizer LLM reads the meta-prompt and generates new candidate solutions. The paper identifies two key challenges in this generation phase and addresses them with specific design choices (Section 2.3).

Challenge 1: Optimization Stability. The optimization process is inherently noisy — not every generated solution improves over prior ones, and early in the optimization when the solution space has been only sparsely explored, the trajectory may contain many low-quality solutions. The paper notes that "LLM output can be drastically affected by low-quality solutions in the input optimization trajectory, especially at the beginning when the solution space has not been adequately explored. This sometimes results in optimization instability and large variance" (Section 2.3).

The solution is batch generation: at each optimization step, the optimizer LLM generates multiple candidate solutions (the default is 8) rather than a single one. This serves a function analogous to mini-batch gradient descent — computing gradients over a batch of examples reduces variance compared to stochastic gradient descent with a single example. With LLM-based optimization, generating 8 candidates per step means the optimizer is "simultaneously exploring multiple possibilities and quickly discovering promising directions to move forward" (Section 2.3). Even if some generated solutions are poor, the batch is likely to contain at least one that maintains or improves performance, providing continuity for the optimization trajectory.

The ablation study on batch size (Section 5.3, Figure 8) compares generating 1, 2, 4, 8 (default), and 16 instructions per step. The x-axis in these plots is the total number of evaluated instructions, not the number of steps — this equalizes the evaluation budget across conditions. For example, with batch size 1, the optimization runs for 1600 steps (evaluating 1600 total instructions); with batch size 8, it runs for 200 steps (also evaluating 1600 total instructions). The results show that batch size 8 achieves the best overall performance. Smaller batch sizes (1, 2) suffer from instability — with only one or two candidates per step, the optimization is more vulnerable to individual low-quality proposals derailing the trajectory. Larger batch size (16) underperforms because it reduces the number of optimization steps for a fixed evaluation budget, meaning the optimizer has fewer opportunities to incorporate feedback from new evaluations into subsequent proposals. The paper frames this tradeoff explicitly: "to achieve better performance with a fixed budget for the number of instructions to evaluate, the number of per-step instructions should not be too large, so as to allow more optimization steps to incorporate richer information of past instructions with their accuracies" (Section 5.3).

Challenge 2: Exploration-Exploitation Trade-off. The balance between exploring new regions of the solution space and exploiting regions near already-discovered good solutions is controlled primarily through the sampling temperature of the optimizer LLM (Section 2.3). The paper's description is precise: "a lower temperature encourages the LLM to exploit the solution space around the previously found solutions and make small adaptations, while a high temperature allows the LLM to more aggressively explore solutions that can be notably different."

The default temperature is 1.0. The ablation study (Section 5.3, Figure 10) evaluates temperatures of 0.0, 0.5, 1.0, 1.5, and 2.0. The results reveal:

Temperature 0.0 (greedy decoding): The optimizer "often gets stuck at the same instruction for tens of steps, resulting in flat optimization curves" (Section 5.3). With no randomness, the model deterministically produces the same output given the same input — but since the meta-prompt changes slightly each step (new candidates and scores are added, exemplars may be re-sampled), the deterministic output can still drift. However, the lack of diversity severely limits exploration.
Temperature 0.5: Similarly "lacks exploration and thus creativity," producing optimization curves that are flatter than temperature 1.0.
Temperature 1.0 (default): Achieves the best performance, striking an effective balance where the optimizer both exploits the trajectory (building on patterns from high-scoring solutions) and explores new variations (through the randomness in sampling).
Temperatures 1.5 and 2.0: The optimizer "more often ignores the trajectory of previous instructions presented in the meta-prompt and thus lacks exploitation, therefore the optimization curve does not have a steady upward trend." At very high temperatures, the generated solutions become essentially random with respect to the optimization history, breaking the feedback loop that enables iterative improvement.

This temperature-dependent behavior demonstrates that OPRO's effectiveness depends on the optimizer LLM attending to the trajectory — it must produce solutions that are informed by the history, not random independent samples. The optimal temperature of 1.0 suggests that the LLM needs some stochasticity to avoid mode collapse (generating the same solution repeatedly) but not so much that it ignores the conditioning signal from the meta-prompt.

Meta-instructions for output formatting. Beyond the core optimization instructions, the meta-prompt includes formatting instructions that ensure generated solutions are parseable. For prompt optimization with PaLM 2-L-IT, the instruction is "Write your new text that is different from the old ones and has a score as high as possible. Write the text in square brackets" (Figure 3). For GPT models, the format specification is more structured: "Generate an instruction that is different from all the instructions <INS> above, and has a higher score than all the instructions <INS> above. The instruction should begin with <INS> and end with </INS>" (Figure 22, Appendix C.2). These format specifications are practical necessities — without them, parsing the generated solution from free-form LLM output becomes unreliable.

An important detail: the meta-instruction explicitly asks for solutions that are "different from the old ones." This is a soft constraint intended to prevent the optimizer from simply regurgitating previously seen solutions. However, the paper notes in Appendix A that optimizer LLMs "do not 100% reliably follow this instruction even if its own outputs often include sentences like 'I will provide a new pair that is different', making the output self-contradictory." The observation that outputs are "almost guaranteed to be different from in-context old solutions when the model output contains a comparison of the new pair and all old pairs" suggests a potential improvement — explicitly triggering the LLM to compare its proposal against all prior solutions — but this is left as future work.

The Complete Optimization Loop: Mathematical Optimization (Linear Regression and TSP)

For the motivating mathematical optimization examples (Section 3), the optimization loop follows a consistent pattern with problem-specific variations in the meta-prompt content and solution representation.

Initialization. Each optimization starts from a small set of randomly generated solutions. For linear regression, this is 5 randomly sampled (w, b) pairs. For TSP, this is 5 randomly generated routes. These initial solutions establish the first entries in the optimization trajectory — they provide the optimizer LLM with an initial sense of the solution space and typical score magnitudes, even though they are unlikely to be optimal.

Per-step generation. At each step, the optimizer LLM receives a meta-prompt containing:

The best 20 solution-score pairs from history, sorted in ascending order by score (worst first, best last).
Meta-instructions specifying the optimization goal and output format.
For linear regression (Figure 19, Appendix C.1): "Now you will help me minimize a function with two input variables w, b. I have some (w, b) pairs and the function values at those points. The pairs are arranged in descending order based on their function values, where lower values are better." (Note: this meta-prompt uses descending order for the objective values — contrary to the general description of ascending order — because lower objective values are better for minimization, so the "best" solutions have the lowest values and appear last.)
For TSP (Figure 20, Appendix C.1): "You are given a list of points with coordinates below: ... Below are some previous traces and their lengths. The traces are arranged in descending order based on their lengths, where lower values are better. ... Give me a new trace that is different from all traces above, and has a length lower than any of the above."

The optimizer LLM is prompted 8 times per step (the default batch size for mathematical optimization, matching the prompt optimization setting). Each prompt produces one candidate solution. These candidates are evaluated against the true objective function (computing MSE for linear regression, computing route length for TSP), and the results are added to the history.

Termination. The optimization terminates either when the LLM "is unable to propose new solutions with better optimization scores" or when a maximum number of steps is reached (Section 2). For the linear regression experiments, Table 2 reports the actual number of steps taken before reaching the global optimum — this varies by model and problem difficulty, ranging from 3.8 steps (text-bison on a simple within-region problem with wtrue=16, btrue=10) to 50.4 steps (gpt-4 on a far-outside problem with wtrue=36, btrue=-1).

Key parameter: maximum trajectory length. The paper keeps the best 20 (w, b) pairs for linear regression and the best 5 traces (of varying length, but approximately 5) for TSP. The choice of 20 for linear regression and TSP is a practical constraint driven by the LLM context window — the meta-prompt must fit within the model's maximum input length, and each solution-score pair consumes tokens. For TSP with n=50 nodes, the trace representation alone is long (a sequence of 50 node indices), making it "hard to fit large-scale optimization problem descriptions in the prompt" (Section 3.2, Limitations).

Black-box nature. The paper emphasizes that the optimization is black-box: "the analytic form does not appear in the meta-prompt text. This is because the LLM can often calculate the solution directly from the analytic form" (Section 3.1). For linear regression, the LLM is not told that the objective is mean squared error or given the formula y = wx + b. It only sees input points and their function values. This tests whether the LLM can infer the optimization direction purely from the numerical pattern of scores — a capability the results confirm for small-scale problems.

The Complete Optimization Loop: Prompt Optimization

For prompt optimization (Section 4), the loop has additional complexity because the "objective function evaluator" is itself an LLM, and the solution space (natural language instructions) requires more elaborate meta-prompt design.

Problem setup. The task is formalized as: given a training set of input-output pairs for some natural language task, find an instruction string that, when inserted at a specified position in the prompt, maximizes the accuracy of a separate scorer LLM on those pairs. The paper defines three instruction insertion positions (Section 4.1):

Q_begin: The instruction is prepended before the question. Used with instruction-tuned scorer LLMs (like text-bison) where the prompt does not follow a strict QA template. Example: "{instruction} Janet's ducks lay 16 eggs per day..." (Figure 17, Appendix B).
Q_end: The instruction is appended after the question but before the answer. Example: "Janet's ducks lay 16 eggs per day... {instruction}" (Figure 18).
A_begin: The instruction is prepended to the beginning of the scorer LLM's output. This is used with pre-trained (non-instruction-tuned) scorer LLMs (like PaLM 2-L) where the prompt is formatted as a QA sequence: "Q: [question] A: {instruction}" (Figure 16). The paper notes this position is "applicable to pretrained LLMs without instruction tuning, where the prompt is formatted as a sequence of QA pairs" (Section 4.1).

The choice of insertion position is determined by the scorer LLM type and is held fixed throughout optimization. The optimizer LLM generates the instruction string; the insertion position determines how that string is combined with task inputs before being fed to the scorer.

Training data for optimization. The paper uses a subset of the task's training data to compute the objective function (training accuracy) during optimization. The key finding is that "a small number or fraction of training samples (e.g., 3.5% of the training set for GSM8K, 20% for Big-Bench Hard) is sufficient" (Section 4.1). Specifically:

GSM8K: 3.5% of the 7,473 training examples ≈ 262 examples. These are randomly sampled once and used throughout the entire optimization process, so the training accuracy computed at each step is an approximation of the accuracy on the full training set.
BBH: 20% of each task's examples (each BBH task has up to 250 total examples, so approximately 50 training examples). The remaining 80% are held out for final test evaluation.

This split is described as a 20-80 train-test split within each BBH task: "For each task, we utilize a subset of 20% examples for prompt optimization, and the rest examples are for testing" (Section 5.2.2). The paper does not set aside a separate validation set in the default configuration, a choice discussed in the overfitting analysis (Section 5.4).

Meta-prompt construction for prompt optimization. The meta-prompt for prompt optimization (illustrated in Figure 3 and detailed in Section 4.2) contains the same two core components as the mathematical optimization case, but with task-specific elaborations:

Optimization trajectory: The best 20 instructions from history, sorted by training accuracy in ascending order. The paper notes that only the highest-scoring instructions are retained "in consideration of the LLM context length limit" (Section 4.2). The scores are displayed as integers (the default 100-bucket representation).
Optimization problem examples (exemplars): Three input-output pairs randomly sampled from the training set. The default is 3 exemplars; the ablation study (Figure 7e-f) shows that 3 exemplars outperform both 0 and 10. The exemplars serve as the task description — from the input-output pair, the optimizer LLM can infer the task type (e.g., math word problem). Crucially, each exemplar shows the instruction insertion point via the <INS> marker, "and this is essential for the optimizer LLM to generate instructions of the same style" (Section 4.2). The exemplars are re-sampled at each optimization step — "In each optimization step, we add several (three for example) training examples to the meta-prompt by random sampling the training set or choose the ones the previous instructions fall short of" (Section 4.2). This re-sampling provides a form of data augmentation, exposing the optimizer to different examples over time and potentially helping it generate instructions that generalize across the training distribution.
Meta-instructions: Instructions to the optimizer LLM that explain the optimization goal and output format. For PaLM 2-L-IT (Figure 3), the meta-instructions include statements like "Write your new text that is different from the old ones and has a score as high as possible. Write the text in square brackets." For GPT models (Figure 22), the meta-instructions are more elaborate: "Generate an instruction that is different from all the instructions <INS> above, and has a higher score than all the instructions <INS> above. The instruction should be concise, effective, and generally applicable to all problems above."

A key design choice: the meta-prompt does not explicitly instruct the LLM to edit, mutate, or cross over existing prompts. Unlike EvoPrompt (Guo et al., 2023) which explicitly instructs the LLM to "cross over the two prompts and generate a new one, then mutate the newly generated prompt" (Section 5.5), or APO (Pryzant et al., 2023) which "instructs the LLM to produce text feedback on how to update an old instruction" (Section 6), OPRO simply presents the trajectory and asks for a new instruction with a higher score. The LLM must implicitly figure out how to use the trajectory to guide improvement — it might notice patterns, combine elements from multiple high-scoring instructions, avoid patterns common in low-scoring instructions, or generate entirely novel directions. This design choice is deliberate: the paper states that "the optimizer LLM in our work directly generates new instructions at each optimization step, and the optimizer LLM is merely asked to improve the task accuracy without being required to imitate past instructions" (Section 6).

Per-step generation and evaluation. At each optimization step:

The meta-prompt is constructed with the current trajectory (best 20 instructions), 3 randomly sampled exemplars, and the meta-instructions.
The optimizer LLM is prompted 8 times (default batch size) with this meta-prompt, using temperature 1.0 (default). Each prompt call produces one new instruction.
Each of the 8 instructions is evaluated by the scorer LLM on the training subset to compute its training accuracy. For the scorer LLM, temperature is set to 0 (greedy decoding) to ensure deterministic and reproducible evaluations.
The 8 new (instruction, accuracy) pairs are added to the trajectory. The trajectory is re-sorted in ascending order, and only the best 20 are retained.
The process repeats for the next step (up to 200 steps by default, or until convergence).

Scorer LLM evaluation protocol. The paper specifies that when evaluating generated instructions, "the scorer LLM greedily decodes" (temperature = 0, Section 5.1). This ensures that the accuracy measurement is deterministic and reproducible — a given instruction always produces the same accuracy on the same training subset, eliminating noise from the evaluation process. The accuracy metric is exact-match: the scorer LLM's output is compared to the ground-truth answer, and the fraction of correct matches across the training subset is the score.

Optimization curves and convergence. The paper tracks training accuracy (on the subset) across steps and plots it as an optimization curve. Figure 1(a) shows the curve for GSM8K with PaLM 2-L-IT as optimizer: starting from "Let's solve the problem" at 60.5% training accuracy, the curve shows an overall upward trend reaching ~80% by step 150, with several "leaps" — step transitions where the average accuracy jumps substantially (e.g., step 5 to step 6, from ~74% to ~78%). The paper explains these leaps (Section 5.2.1): "a leap in our optimization curve does not always correspond to a much better instruction being discovered; instead, it can be due to a large qualitative improvement of all 8 generated instructions in this step." The mechanism is that after a particularly good instruction is discovered, subsequent steps generate similar instructions, gradually displacing the worse ones from the trajectory. When the trajectory's composition shifts qualitatively — most of the best 20 instructions now share a common high-quality pattern — the LLM produces an entire batch of good instructions, causing the average to jump.

Test evaluation after optimization. After the optimization loop terminates, the final instructions are evaluated on the held-out test set. The paper reports test accuracy for the single instruction with the highest training accuracy (the "top instruction"), as shown in Table 4 for GSM8K and Tables 8-15 for BBH tasks. The test evaluation uses the same scorer LLM and insertion position as the training evaluation, but applies the instruction to the full test set rather than the training subset.

Hyperparameter defaults (Section 5.1). The complete set of default hyperparameters for prompt optimization:

Optimizer LLM temperature: 1.0 (for generation diversity).
Scorer LLM temperature: 0 (greedy decoding for deterministic evaluation).
Batch size (instructions per step): 8.
Maximum trajectory length: 20 best instructions retained in the meta-prompt.
Exemplars per step: 3, randomly sampled from the training subset.
Maximum optimization steps: 200 (the paper notes that far fewer steps are often sufficient to find excellent instructions).
Training subset size: 3.5% of GSM8K training set (~262 examples); 20% of each BBH task's total examples.
Score representation: Integer-rounded accuracy (100-bucket granularity).
Trajectory ordering: Ascending by score (worst first, best last).

Position-specific meta-prompt variations. The meta-prompt format differs depending on the optimizer LLM type and the instruction insertion position. The paper provides three distinct meta-prompt templates:

PaLM 2-L-IT (Figure 3): Uses a conversational instruction format. The meta-prompt begins with "I have some texts along with their corresponding scores. The texts are arranged in ascending order based on their scores, where higher scores indicate better quality." Exemplars are shown in a QA format with <INS> marking the insertion point.
Pre-trained PaLM 2-L (Figure 21): Uses a few-shot format since the model is not instruction-tuned. The meta-prompt says "Create a piece of text at the beginning of the answer to enhance the precision in solving diverse grade school math problems." Below this, it shows examples in the format Precision: 4 <TEXT>A dime</TEXT>, where "Precision" is a score proxy (presumably a transformed accuracy) and <TEXT> encloses the instruction. Because this model is not instruction-tuned, it performs better with few-shot formatting — the meta-prompt includes two initial instructions to seed the trajectory (the empty string with precision 32.2 and "The answer is" with precision 33.3).
GPT models (gpt-3.5-turbo, gpt-4; Figure 22): Uses a more structured directive format. The meta-prompt begins with "Your task is to generate the instruction <INS>." It presents previous instructions with scores, exemplars as problems with ground-truth answers, and ends with a detailed specification: "Generate an instruction that is different from all the instructions <INS> above, and has a higher score than all the instructions <INS> above. The instruction should begin with <INS> and end with </INS>. The instruction should be concise, effective, and generally applicable to all problems above."

The paper does not systematically evaluate which meta-prompt format is optimal for which optimizer LLM — the templates are chosen based on each model's known prompting preferences (instruction-tuned models work well with instructions; pre-trained models work better with few-shot formatting; GPT models benefit from structured, explicit directives).

Comparison with One-Step Instruction Generation

To validate that the iterative optimization process (leveraging the trajectory across multiple steps) is necessary — rather than simply the LLM's ability to generate instructions when prompted — the paper includes a comparison with a one-step baseline (Section 5.3). The baseline generates 50 instructions in a single step from a meta-prompt that includes task exemplars, the initial instruction with its accuracy, and the same meta-instructions, but no trajectory of intermediate solutions (since there are no intermediate steps). All other hyperparameters are identical.

The results show that one-step generation performs substantially worse:

On GSM8K, the best instruction among all 50 one-step generations was still "Let's solve the problem" (the initial seed), with 64.4% training accuracy and 60.8% test accuracy. In contrast, OPRO found "Let's do the math!" at step 5 with 78.2% training accuracy — a 13.8 percentage point improvement on training and 15.5 points on test.
On BBH sports_understanding, the best one-step instruction achieved 84.0% training and 80.0% test accuracy. OPRO at step 4 found an instruction with 88.0% training and 84.5% test accuracy.

This comparison demonstrates that the trajectory is not merely a convenience — it is essential for the LLM to iteratively refine its understanding of what makes an instruction effective. Without seeing the progression of (instruction, score) pairs, the LLM's first batch of proposals is essentially random with respect to what will work well, and it cannot improve upon its initial guesses. The trajectory provides the feedback signal that enables learning across steps.

Handling of Overfitting in Prompt Optimization

The paper's default configuration does not set aside a separate validation set during optimization — all available training data is used to compute the objective function score. The paper addresses this design choice in Section 5.4, explaining the rationale and providing empirical justification.

The argument for no validation set. The paper acknowledges that overfitting can cause training accuracy to be higher than test accuracy, but argues that "overfitting is less harmful when each candidate solution (natural language instruction in the prompt optimization context) overfits to a similar extent. In this case, a higher training accuracy solution still achieves a higher validation/test accuracy, and one can adopt solutions with the highest training accuracies as the final result" (Section 5.4). In other words, if all instructions overfit by roughly the same amount (e.g., 10 percentage points), then the relative ranking of instructions by training accuracy still reflects their relative ranking by test accuracy, even though the absolute numbers are inflated.

Empirical validation. The paper runs experiments where the training data is split into 1/3 training, 1/3 validation, and 1/3 test. Figure 11 shows that "the validation accuracy curves trend up and down alongside the training curves in both prompt optimization settings" (Section 5.4). This parallel movement confirms that training and validation accuracy are correlated — when training accuracy improves, validation accuracy also tends to improve, and vice versa. The paper acknowledges that overfitting still occurs ("training accuracies are often 5%-20% higher than our test accuracies"), but since the optimization selects instructions based on relative training accuracy ranking, the absolute gap does not affect the selection.

Mitigation strategies for production use. The paper suggests that "setting aside a larger training set and optimizing for fewer steps (early stopping) may help reduce overfitting" (Section 5.4), but these are presented as suggestions for practitioners rather than evaluated recommendations. The default configuration prioritizes simplicity (no validation split) over rigor, justified by the empirical observation that the training-validation correlation is sufficient for optimizer guidance.

4. Key Insights and Innovations

Innovation 1: The LLM as the Optimizer — A Paradigm Shift from "Optimization with LLMs" to "LLMs as Optimizers"

The paper's most fundamental conceptual move is not a specific technique but a reframing of the relationship between LLMs and optimization. Prior work that combined language models with optimization fell into two categories: (1) optimization for LLMs — gradient-based prompt tuning (Lester et al., 2021; Li & Liang, 2021) and RL-based prompt search (Deng et al., 2022; Zhang et al., 2023) that optimize prompts using model gradients or learned policies, and (2) LLMs as operators within optimization algorithms — using language models as mutation and crossover operators inside evolutionary algorithms (Meyerson et al., 2023; Lehman et al., 2022; Chen et al., 2023a; Guo et al., 2023). In both paradigms, the LLM is a component — either the thing being optimized or a tool used by an external optimization framework.

OPRO inverts this relationship: the LLM is the optimizer. There is no external algorithm orchestrating the search, no formal encoding of the optimization problem, no programmed update rule. The optimization loop is nothing more than a conversation — "here's what we've tried so far and how well each attempt worked; now propose something better." The LLM handles all the functions that a traditional optimization algorithm would: interpreting the objective from natural language, inferring promising search directions from the trajectory of past attempts, managing the exploration-exploitation tradeoff through its own generation process, and formulating new candidates.

What makes this genuinely novel — rather than an obvious application of LLM capabilities — is that it works without the LLM being told how to optimize. The meta-prompt does not say "identify patterns in high-scoring solutions and combine them," "perform a local search around the best solution," or "use the scores to estimate a gradient direction." It simply presents the history and asks for improvement. The LLM must implicitly discover an optimization strategy from the pattern of (solution, score) pairs. The fact that different LLMs (PaLM 2-L, text-bison, gpt-3.5-turbo, gpt-4) all succeed at this — yet produce instructions of recognizably different styles (Table 1: PaLM 2-L-IT produces "Take a deep breath and work on this problem step-by-step" while gpt-3.5-turbo produces "A little bit of arithmetic and a logical approach will help us quickly arrive at the solution to this problem") — demonstrates that the meta-prompt is not hard-coding a specific optimization strategy but rather creating a scaffold within which each model's reasoning capabilities can operate as an optimizer.

This is a fundamental shift rather than an incremental refinement. It recasts optimization from something you program to something you describe, with implications that extend beyond the paper's empirical demonstrations. If the capability generalizes, it means that a single general-purpose language model can serve as an optimizer for any problem that can be described in natural language and evaluated — without requiring the user to understand optimization algorithms, define decision variables, or even recognize that they are solving an optimization problem. The paper's mathematical optimization examples (Section 3) gesture toward this vision: the same optimizer LLM that finds prompts for GSM8K can also find routes for TSP and coefficients for linear regression, simply by changing the problem description in the meta-prompt.

Innovation 2: The Optimization Trajectory as Implicit Gradient — Enabling Iterative Improvement Without Explicit Update Rules

The second conceptual contribution is the discovery that a sorted list of (solution, score) pairs functions as an implicit optimization signal that enables iterative improvement without any programmed update rule, explicit gradient, or edit operation specification. This distinguishes OPRO from all prior LLM-based prompt optimization methods and explains why it can improve from very weak starting points where edit-based approaches would struggle.

To understand why this is distinctive, consider what prior approaches required:

APE (Zhou et al., 2022b) first generates initial instructions, selects the best ones, then prompts the LLM to generate "semantically similar" variants. The optimization signal is explicit: "make something like this good instruction." But this constrains search to a local neighborhood around already-good solutions — it cannot make large leaps across the solution space because it is anchored to semantic similarity with the top candidates.
APO (Pryzant et al., 2023) instructs the LLM to produce natural language feedback on how to improve a specific instruction, then edits it accordingly. The optimization signal requires the LLM to first diagnose why an instruction is suboptimal before proposing improvements — a metacognitive step that may fail when the LLM does not understand the task well enough to critique instructions meaningfully.
EvoPrompt (Guo et al., 2023) explicitly instructs the LLM to perform crossover and mutation operations on pairs of prompts. The optimization signal is the genetic algorithm metaphor — the LLM is told how to combine and modify prompts, but it lacks task exemplars to understand what makes a prompt effective. The paper's experimental comparison (Section 5.5, Figure 12) shows this fails catastrophically when starting from generic prompts: EvoPrompt degrades performance because it mutates prompts blindly, without task understanding.

OPRO's trajectory-based approach is fundamentally different: the LLM is never told how to improve — it is shown the evidence of what has worked and trusted to infer the pattern. The sorted list of 20 (instruction, score) pairs implicitly encodes:

Direction of improvement: Because scores are in ascending order, the best instructions appear at the end. The LLM's documented recency bias (Zhao et al., 2021) — the tendency to generate tokens similar to those at the end of the prompt — means it naturally produces solutions resembling the high-scoring ones. The ablation study (Figure 7a-b) confirms that ascending order (worst first, best last) outperforms descending and random orders, likely because it places the best examples where they exert the most influence.
Patterns of quality: By observing that multiple high-scoring instructions share certain features (e.g., containing "step by step," being concise and declarative, specifying a problem-solving methodology), the LLM can abstract what makes instructions effective without anyone labeling those features explicitly. The paper does not instruct the LLM to identify these patterns — the meta-prompt simply presents the data and asks for improvement.
Gradient of the solution space: The progression from low-scoring to high-scoring solutions encodes the "direction" of improvement in the discrete solution space — analogous to how a gradient points in the direction of steepest ascent in continuous optimization. The LLM, by comparing solutions at different quality levels, can infer what kinds of changes tend to increase scores.

The crucial empirical evidence for this implicit-gradient interpretation is the one-step baseline comparison (Section 5.3). When the LLM generates 50 instructions in a single step — seeing only the initial instruction with its score but no trajectory of intermediate improvements — it cannot find anything better than the starting point on GSM8K. This is not because the LLM cannot generate good instructions (it clearly can, given the trajectory). It is because without the trajectory, the LLM lacks the signal to distinguish good from bad directions in the solution space. The trajectory provides the "gradient" — the series of small improvements that shows what kinds of changes are productive — and without it, the LLM's proposals are essentially random with respect to quality.

This insight is fundamental because it identifies the mechanism by which LLM-based optimization works: not through explicit reasoning about optimization strategies (though the LLM may do some of that), but through the LLM's pattern-matching and in-context learning capabilities applied to the optimization history. It also explains why temperature matters (Figure 10): too low and the LLM cannot explore beyond the trajectory's patterns; too high and it ignores the trajectory entirely. The optimal temperature of 1.0 is the point where the LLM is sufficiently influenced by the trajectory to exploit its patterns but sufficiently stochastic to explore variations.

Innovation 3: Task Exemplars as Optimization Context — Solving the "Blind Optimization" Problem

A third conceptual contribution is the identification and resolution of what might be called the blind optimization problem: when an optimizer does not understand what the task is, it cannot distinguish between changes that improve quality and changes that are merely arbitrary mutations. The paper's comparison with EvoPrompt (Section 5.5) provides the clearest diagnostic evidence.

EvoPrompt (Guo et al., 2023) uses meta-prompts that instruct the LLM to perform genetic algorithm operations — crossover and mutation — on pairs of prompts. The meta-prompt includes the prompts to be manipulated and instructions for how to combine and modify them. What it does not include is any information about what task these prompts are for. When starting from generic initial prompts ("Let's solve the problem" and "Here is the answer") on GSM8K, EvoPrompt not only fails to improve — it degrades performance (Figure 12a). The LLM is performing crossover and mutation operations faithfully (it follows the instructions), but because it does not know it is optimizing prompts for grade-school math word problems, its mutations are semantically arbitrary — they change the prompts in ways that have no systematic relationship to task accuracy.

OPRO solves this by including task exemplars in the meta-prompt. These exemplars are not optional — the ablation study (Figure 7e-f) shows that removing them causes performance collapse, with the no-exemplar condition failing to improve over the starting point on both GSM8K and BBH sports_understanding. But their function is more subtle than simply "showing the LLM what the task looks like." The exemplars serve three distinct roles that together enable effective optimization:

Task identification: From 3 input-output pairs on GSM8K, the LLM can infer that this is a math word problem task requiring arithmetic reasoning. This shapes what kinds of instructions will be effective — instructions about "step-by-step reasoning" are relevant; instructions about "choosing the funniest pun" are not.
Instruction format calibration: The exemplars show the exact insertion point of the generated instruction via the <INS> marker. This is essential for the optimizer LLM to "generate instructions of the same style" (Section 4.2). The paper notes that when gpt-3.5-turbo optimizes A_begin instructions without seeing the insertion context, it often generates imperative or interrogative sentences more suitable for Q_begin — a format mismatch that the exemplars prevent.
Difficulty calibration: The exemplars communicate the complexity level of the task. GSM8K exemplars involve multi-step arithmetic; BBH ruin_names exemplars involve pun-based wordplay; BBH temporal_sequences exemplars involve timeline reasoning. This calibration helps the optimizer LLM generate instructions of appropriate specificity — broad enough to apply to all instances of the task, but specific enough to provide useful guidance.

The finding that 3 exemplars are sufficient and 10 do not help further (Figure 7e-f) is practically significant: it means the optimization does not require fitting large amounts of task data into the already-crowded meta-prompt. Three exemplars provide enough task context without dominating the prompt to the point where the trajectory (the optimization signal) gets diluted.

This insight is incremental but practically crucial because it identifies a failure mode in concurrent LLM-based optimization work (EvoPrompt) and provides a simple fix. It also clarifies why OPRO's meta-prompt design has two essential components rather than one: the trajectory provides the optimization signal (what to improve and in what direction), while the exemplars provide the task signal (what the optimization is about). Both are necessary; neither is sufficient alone.

Innovation 4: LLM Optimization Capability as an Emergent Property — Not All LLMs Are Equal Optimizers

The paper's fourth contribution is an empirical characterization of LLM optimization capability as an emergent property that varies systematically across models, problem types, and difficulty levels. This is not merely a benchmark comparison — it is a diagnostic that reveals what kind of reasoning makes an LLM a good optimizer and where the capability breaks down.

Consider the linear regression and TSP results (Tables 2 and 3). On small-scale problems, all tested LLMs function as optimizers — they observe solution-score pairs and propose new solutions that improve the objective. But the quality of their optimization varies dramatically:

Convergence speed: On linear regression with the ground truth within the starting region, gpt-4 reaches the global optimum in 4.0 ± 1.5 steps while gpt-3.5-turbo takes 7.6 ± 4.5 steps (Table 2). This is an almost 2× difference in optimization efficiency between models applied to the same problem with the same meta-prompt.
Exploration efficiency: gpt-4 finds the optimum after exploring only 17.2 unique (w, b) pairs on average, while gpt-3.5-turbo explores 36.0 and text-bison explores 40.0. This means gpt-4 is not just faster — it makes better proposals per attempt, requiring fewer evaluations to locate the optimum.
Scaling behavior: On TSP with n=50, gpt-4 achieves an 11.0% optimality gap (comparable to heuristic algorithms), while text-bison and gpt-3.5-turbo get stuck at 219.8% and 133.0% respectively — 20× and 12× worse (Table 3). All three models see the same meta-prompt format and trajectory information. The difference is in their ability to reason about the optimization landscape at scale.

These patterns suggest that optimization capability is not binary (can optimize vs. cannot) but graded — it correlates with the model's general reasoning capabilities, but in ways that are not simply explained by "better models do better at everything." The paper provides a specific diagnostic observation about gpt-4's advantage: "when the history shows the objective values of (w, b) = (8, 7), (w, b) = (8, 6), and (w, b) = (8, 5) are decreasing, it has a highest chance to propose (w, b) = (8, 4) for evaluation" (Section 3.1). In other words, gpt-4 is better at extrapolating trends from the trajectory — recognizing that if decreasing b while holding w constant improves the objective, it should continue decreasing b. This is the kind of pattern that a human optimizer would recognize ("the objective seems monotonic in b in this region; keep moving in that direction"), and the fact that it emerges more strongly in more capable models suggests it is not explicitly programmed but arises from general reasoning capability.

The prompt optimization results reinforce this graded-capability picture. Different optimizer LLMs produce instructions of qualitatively different styles (Table 1): PaLM 2-L-IT produces concise, imperative instructions ("Take a deep breath and work on this problem step-by-step"); gpt-3.5-turbo produces longer, more elaborately reasoned instructions ("A little bit of arithmetic and a logical approach will help us quickly arrive at the solution to this problem"); gpt-4 produces instructions that combine command and explanation ("Let's combine our numerical command and clear thinking to quickly and accurately decipher the answer"). These stylistic differences are not random — they reflect each model's "optimization personality," the characteristic way it navigates the solution space. Some styles are more effective than others on particular tasks, but the key point is that the optimization algorithm is not separable from the optimizer — there is no "OPRO algorithm" independent of the LLM that executes it. Different LLMs running the same meta-prompt procedure produce different optimization trajectories with different convergence characteristics.

This insight is fundamental because it reframes the question from "can LLMs optimize?" (which the paper answers affirmatively) to "what makes an LLM a good optimizer, and how can we predict or improve optimization capability?" The failure cases in Appendix A provide additional diagnostic value: the optimizer sometimes hallucinates function values, gets stuck at non-optimal points, or fails to navigate bumpy loss landscapes. These failures are not random — they reveal the specific cognitive limitations that cap optimization performance. For instance, the Rosenbrock function failure (Figure 13) shows that when the optimizer reaches the flat region near (0, 0), it cannot "see" the narrow valley leading to (20, 400) because the local gradient information in the trajectory is misleading — all nearby points have higher objective values, so the optimizer incorrectly concludes it has reached the minimum. This is the same kind of failure that gradient-based optimizers experience in the Rosenbrock valley, but the LLM's failure mode is reasoning-based rather than gradient-based — it concludes it is at the optimum rather than being trapped by vanishing gradients. Understanding these failure modes is a necessary step toward building better LLM optimizers, and the paper provides the first systematic catalog of them.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation benchmarks are GSM8K (Cobbe et al., 2021), a dataset of 7,473 training and 1,319 test grade-school math word problems, and Big-Bench Hard (BBH) (Suzgun et al., 2022), a suite of 23 challenging BIG-Bench tasks covering arithmetic, symbolic manipulation, and commonsense reasoning, with each task containing up to 250 total examples. For mathematical optimization case studies, synthetic linear regression data is generated with 50 points per problem, and TSP instances are randomly generated by sampling node coordinates in [-100, 100].
Base model(s). The optimizer LLMs evaluated are pre-trained PaLM 2-L, instruction-tuned PaLM 2-L (PaLM 2-L-IT), text-bison (all from the PaLM 2 model family; Anil et al., 2023), gpt-3.5-turbo (gpt-3.5-turbo-0613), and gpt-4 (gpt-4-0613). The scorer LLMs for prompt optimization are pre-trained PaLM 2-L and text-bison. The models span both pre-trained and instruction-tuned variants, and both API-accessible (GPT) and internally-accessible (PaLM) families, chosen to demonstrate the generality of OPRO across model types and to test whether the optimization capability varies with model scale and training methodology.
Metrics. For prompt optimization, the metric is task accuracy — the fraction of test examples for which the scorer LLM's output matches the ground-truth answer exactly, using the grading function from the respective benchmark. Training accuracy (computed on the training subset) serves as the objective function during optimization. For linear regression, the metric is the squared error objective value (minimized). For TSP, the metric is the optimality gap, defined as (solution distance - oracle distance) / oracle distance, where oracle solutions are computed by the Gurobi solver.
Baselines. For prompt optimization on GSM8K, the baselines are the zero-shot instructions "Let's think step by step." (Kojima et al., 2022), "Let's work this out in a step by step way to be sure we have the right answer." (Zhou et al., 2022b), "Let's solve the problem.", and the empty string. On BBH, the same baselines apply per task, with the empty string serving as the optimization starting point. For TSP, the baselines are the Nearest Neighbor (NN) and Farthest Insertion (FI) heuristic algorithms. For the meta-prompt design comparison, the baseline is EvoPrompt (Guo et al., 2023) in both its Genetic Algorithm (GA) and Differential Evolution (DE) variants.
Generation budget / compute accounting. The primary budget metric for prompt optimization is the number of evaluated instructions — each generated instruction is evaluated once on the training subset, so the total compute cost is proportional to the number of optimizer LLM calls (for generation) plus scorer LLM calls (for evaluation). The default settings use 8 instructions per step for up to 200 steps = 1,600 total evaluated instructions. Ablation studies on batch size (Figure 8) equalize the total evaluation budget by running more steps with smaller batch sizes (e.g., 1,600 steps at batch size 1, 400 steps at batch size 4). For mathematical optimization, the budget is measured in number of unique (w, b) pairs explored (linear regression) or number of optimization steps (TSP).
Cross-validation / statistical protocol. For prompt optimization, the default configuration uses a simple train-test split without a separate validation set: 3.5% of GSM8K training data (~262 examples) or 20% of each BBH task's examples for optimization, with the remainder held out for final test evaluation. The paper justifies this by showing in Section 5.4 (Figure 11) that training and validation accuracy curves trend together, so relative ranking of instructions by training accuracy is preserved for test accuracy. Each ablation study is averaged over 3 optimization repetitions with shaded regions showing standard deviation (e.g., Figures 7–10). For mathematical optimization, each setting is run 5 times with different random starting points, and means ± standard deviations are reported (Tables 2–3).

Main Quantitative Results

Mathematical Optimization: Linear Regression

The linear regression experiments (Section 3.1, Table 2) establish the basic feasibility of LLM-based optimization and reveal systematic differences across optimizer LLMs and problem difficulties. With the ground truth within the starting region [10, 20] × [10, 20] (wtrue=15, btrue=14), all models converge to the global optimum, but with markedly different efficiency: gpt-4 requires only 4.0 ± 1.5 steps and explores 17.2 ± 5.1 unique (w, b) pairs, compared to text-bison's 5.8 ± 2.6 steps and 40.0 ± 12.4 unique pairs (Table 2, first row). This approximately 2.3× difference in exploration efficiency means gpt-4 makes substantially better proposals per attempt — it more accurately infers the descent direction from the trajectory.

When the ground truth moves outside the starting region, performance degrades for all models. At wtrue=36, btrue=-1 (far outside, last row of Table 2), text-bison takes 35.8 ± 6.4 steps and explores 174.0 ± 28.2 unique pairs, while gpt-4 takes 50.4 ± 18.8 steps and 116.4 ± 32.7 unique pairs. Interestingly, gpt-4 now requires more steps but fewer unique evaluations — it converges slowly but efficiently per evaluation, while text-bison evaluates more rapidly but less discriminately. The paper provides a diagnostic observation about gpt-4's advantage: "when the history shows the objective values of (w, b) = (8, 7), (w, b) = (8, 6), and (w, b) = (8, 5) are decreasing, it has a highest chance to propose (w, b) = (8, 4) for evaluation" (Section 3.1). This pattern-extrapolation behavior — recognizing monotonic trends and continuing them — is more pronounced in gpt-4 and appears to be a key mechanism for efficient optimization.

All models struggle more as the ground truth moves farther from the starting region, confirming that distance in the solution space matters — the LLM must navigate a longer path using only local trajectory information, and the chance of getting stuck at a non-optimal point increases with distance traveled.

Mathematical Optimization: Traveling Salesman Problem

The TSP results (Section 3.2, Table 3) demonstrate that LLM optimization capability is surprisingly strong on small instances but degrades dramatically with problem scale. On n=10 problems, all three LLMs find the optimal solution for all 5 instances. However, convergence speed differs by approximately 4×: gpt-4 reaches optimality in 9.6 ± 3.0 steps, compared to 40.4 ± 5.6 for text-bison and 46.8 ± 9.3 for gpt-3.5-turbo. This is a substantial practical difference — gpt-4 requires roughly one-quarter the number of expensive LLM calls to solve the same problems.

At n=15, the gap widens: gpt-4 achieves a 0.2 ± 0.2% optimality gap (essentially optimal) on 4 of 5 problems (58.5 ± 29.0 steps to converge on those successes), while gpt-3.5-turbo achieves 1.2 ± 1.1% (4 successes at 202.0 ± 41.1 steps) and text-bison achieves 4.4 ± 1.3% with 0 successes — meaning text-bison never found the optimal solution on any of the 5 n=15 problems. The 4.4% gap is comparable to or worse than the FI heuristic (1.2 ± 0.6%), which runs in milliseconds without LLM calls.

At n=20, the degradation continues: gpt-4 achieves 1.4 ± 0.6% optimality gap (2 successes), gpt-3.5-turbo reaches 4.4 ± 2.5% (1 success), and text-bison collapses to 30.4 ± 10.6% (0 successes). The FI heuristic achieves 0.2 ± 0.1% — far better than all LLMs. At n=50, only gpt-4 remains competitive with heuristics (11.0 ± 2.6% vs. NN's 19.7 ± 3.1% and FI's 9.8 ± 1.5%), while both text-bison (219.8%) and gpt-3.5-turbo (133.0%) produce solutions with approximately 20× and 12× worse optimality gaps respectively.

The key pattern is that LLM optimization capability degrades superlinearly with problem size — doubling from n=10 to n=20 causes text-bison to go from 0% to 30% optimality gap. The paper attributes this primarily to context window constraints: "the length limit of the LLM context window makes it hard to fit large-scale optimization problem descriptions in the prompt" (Section 3.2), though the degradation pattern suggests this is not simply a binary limit (can fit vs. cannot fit) but rather a graceful degradation as the trajectory information becomes increasingly compressed relative to the problem complexity.

Prompt Optimization: GSM8K

The GSM8K results (Section 5.2.1, Figures 1a and 4, Table 4) are the paper's central demonstration of practical utility. The headline numbers: starting from "Let's solve the problem." at 60.5% training accuracy, OPRO with PaLM 2-L-IT as optimizer and pre-trained PaLM 2-L as scorer discovers the instruction "Take a deep breath and work on this problem step-by-step" which achieves 80.2% test accuracy — an improvement of 8.4 percentage points over "Let's think step by step." at 71.8%, and 46.2 points over the empty string baseline at 34.0% (Table 4, first row under "Ours").

The optimization trajectory in Figure 1a shows the characteristic pattern: an upward trend with several "leaps" where the average accuracy across all 8 generated instructions jumps substantially. The paper documents specific intermediate discoveries: "Let's break it down!" at Step 4 with 71.3% training accuracy, "Let's calculate our way to the solution!" at Step 5 with 73.9%, and "Let's do the math!" at Step 6 with 78.2%. Notably, "Let's do the math!" at Step 6 has only 1.5 percentage points lower test accuracy than the final "Take a deep breath..." found at Step 107 (80.2% vs. 78.7% approximate test accuracy, comparing training scores 78.2 vs. 80.2), indicating that most of the optimization gain occurs within the first ~10 steps, with subsequent steps providing diminishing returns.

Different optimizer-scorer combinations produce qualitatively different top instructions (Table 4):

PaLM 2-L-IT → PaLM 2-L: "Take a deep breath and work on this problem step-by-step." (80.2% test) — concise, imperative, and containing the "step-by-step" phrase discovered by Kojima et al. (2022).
Pre-trained PaLM 2-L → PaLM 2-L: "Break this down." (79.9%) — notably shorter (3 words vs. 11), achieving comparable accuracy through extreme concision.
gpt-3.5-turbo → PaLM 2-L: "A little bit of arithmetic and a logical approach will help us quickly arrive at the solution to this problem." (78.5%) — longer, more elaborately reasoned, and semantically different from "step-by-step."
gpt-4 → PaLM 2-L: "Let's combine our numerical command and clear thinking to quickly and accurately decipher the answer." (74.5%) — also long-form, lower accuracy than PaLM 2-L-based optimizers.

The finding that "Break this down." (pre-trained PaLM 2-L optimizer) achieves 79.9% — nearly matching the 80.2% of "Take a deep breath..." from PaLM 2-L-IT — is particularly interesting because pre-trained PaLM 2-L was not instruction-tuned. The meta-prompt for pre-trained PaLM 2-L uses a few-shot format (Figure 21) rather than instruction format, yet it still functions as an effective optimizer, producing highly concise instructions (most are short phrases like "Here you go:" at 61.3% and "Let's do it:" at 75.1%) that are stylistically appropriate as A_begin prefixes.

With text-bison as scorer (Table 4, bottom section), the accuracies are generally lower but show similar patterns: PaLM 2-L-IT optimizer finds "Let's work together to solve math word problems! ..." at 64.4% test, text-bison optimizer finds "Let's work through this problem step-by-step:" at 68.5%, and gpt-3.5-turbo finds "Analyze the given information, break down the problem into manageable steps..." at 66.5%. The best text-bison-scored instruction (68.5%) modestly outperforms the "Let's think step by step" baseline at 64.4% (+4.1 points), a smaller absolute gain than with PaLM 2-L as scorer, likely because text-bison's base performance is lower and less sensitive to prompt variations.

Figure 4 shows optimization curves with different scorer-optimizer pairs: pre-trained PaLM 2-L as both scorer and optimizer (Figure 4b) demonstrates that a model can optimize its own prompting — an important capability for self-improvement loops. Starting from the empty instruction (32.2 training accuracy) and "The answer is" (33.3), the optimization reaches approximately 80% training accuracy by step 80, with generated instructions following consistent stylistic patterns (prefix-like phrases suitable for A_begin formatting).

Prompt Optimization: Big-Bench Hard

The BBH results (Section 5.2.2, Figures 5 and 6, Tables 5 and 7–9) extend the demonstration across 23 diverse reasoning tasks. Figure 5 presents per-task accuracy differences compared to "Let's think step by step." (Kojima et al., 2022) and the empty string baseline. The key aggregate finding: with PaLM 2-L as scorer and PaLM 2-L-IT as optimizer (Figures 5a,b), OPRO-discovered instructions outperform "Let's think step by step." by over 5% on 19/23 tasks and outperform the empty string starting point by over 5% on 20/23 tasks.

The magnitude of improvement varies dramatically across tasks. On some tasks, the gains are enormous: for example, on word_sorting with PaLM 2-L scorer, OPRO achieves 54.4% overall accuracy vs. 4.0% for "Let's think step by step." — a 50.4 percentage point improvement (Table 7). On sports_understanding with PaLM 2-L scorer, OPRO reaches 90.0% vs. 47.2% for the baseline — a 42.8 point improvement. These tasks are cases where the generic "step by step" instruction is actively harmful (word_sorting requires alphabetical ordering, not sequential reasoning), and OPRO discovers task-appropriate instructions (e.g., "Alphabetical order of given words:" for word_sorting in Table 8).

On other tasks, the gains are modest or even negative. On tracking_shuffled_objects_seven_objects with PaLM 2-L scorer, OPRO achieves only 19.6% vs. 60.8% for "Let's think step by step." — a 41.2 point decrease. This is the worst failure case and represents a task where the optimization trajectory fails to identify effective instruction patterns, possibly because the task requires tracking object swaps across multiple steps — a type of reasoning that is not well-served by any simple prefix instruction.

The optimization curves (Figures 6, 23, 24) show that upward trends are present across almost all BBH tasks, though the starting points, convergence rates, and final plateaus vary substantially. On ruin_names (Figure 6a), starting from the empty string at 64.0% training accuracy, the optimization climbs steadily to ~88% by step 50, with semantically meaningful intermediate discoveries: from the empty instruction to "Consider the following when editing artist or movie names humorously:" (72.0% at Step 1), to an elaborated version with examples (82.0% at Step 38). On temporal_sequences (Figure 6b), the optimization shows a more gradual climb from 64.0% to ~80% over 150 steps, with the instructions becoming progressively more detailed and specific about the solution methodology.

Table 5 presents the top instructions for three representative BBH tasks across different optimizer-scorer pairs. On movie_recommendation with text-bison scorer and PaLM 2-L-IT optimizer, the top instruction at 91.6% is "What is the highest-rated movie similar to the given movies, with a similar IMDb rating and released in the same year?" — a task-specific instruction that encodes domain heuristics. On ruin_names with PaLM 2-L scorer and PaLM 2-L-IT optimizer, the top instruction is the interrogative "Which is the funniest pun on the artist or movie name?" at 88.0% — notably different in style from the declarative instructions found for GSM8K. The stylistic diversity across tasks confirms that the optimizer LLM tailors instructions to task content, not merely generating generic "step by step" variants.

Semantically Similar Instructions Can Have Drastically Different Accuracies

Section 5.2.3 provides a striking diagnostic example of the prompt sensitivity that motivates automated prompt optimization. With PaLM 2-L as scorer on GSM8K test: "Let's think step by step." achieves 71.8%, "Let's solve the problem together." achieves 60.5%, and "Let's work together to solve this problem step by step." — which is the semantic combination of the two higher-performing instructions — achieves only 49.4%. This is not a monotonic relationship where adding useful phrases monotonically increases performance. The paper notes this behavior "increases both the variance across single-step instructions and the oscillation during optimization, and motivates us to generate multiple instructions at each step to improve the optimization stability."

Transferability of Found Instructions

Section 5.2.4 (Table 6) evaluates whether instructions optimized for GSM8K transfer to two other math reasoning benchmarks, MultiArith (Roy & Roth, 2016) and AQuA (Ling et al., 2017). The top GSM8K-optimized instruction "Take a deep breath and work on this problem step-by-step." achieves 95.3% on MultiArith and 54.3% on AQuA with PaLM 2-L scorer, outperforming "Let's think step by step." at 85.7% and 44.9% respectively — a +9.6 point improvement on MultiArith and +9.4 on AQuA. With text-bison scorer, the transfer gains are more modest and mixed: the top GSM8K instruction achieves 96.8% on MultiArith vs. 92.5% for the baseline (+4.3 points), but on AQuA it achieves 37.8% vs. 31.9% (+5.9 points). These results indicate that instructions optimized on one math reasoning benchmark transfer positively to others in the same domain, though the magnitude of transfer depends on the scorer LLM.

Comparison with EvoPrompt

Section 5.5 (Figure 12) provides a direct comparison with EvoPrompt (Guo et al., 2023), a concurrent LLM-based prompt optimization method that uses genetic algorithm and differential evolution meta-prompts. On GSM8K with gpt-3.5-turbo optimizer and PaLM 2-L scorer (Figure 12a), starting from two simple generic instructions "Let's solve the problem." and "Here is the answer.":

OPRO steadily improves training accuracy from ~60% to ~78% over 150 steps.
EvoPrompt (GA) degrades performance from ~60% to ~45%.
EvoPrompt (DE) also degrades to ~50%.

The paper's diagnosis: "EvoPrompt does not utilize exemplars for prompt optimization, thus it lacks the understanding of the task to optimize for" and "relies on good-quality and task-specific initial prompts to optimize from." When given task-specific initial prompts on BBH sports_understanding ("Solve the sports understanding problem." and "Give me the answer to sports understanding."), EvoPrompt (DE) improves from ~68% to ~80% over 200 steps (Figure 12b), but "the optimization curve is less stable than OPRO." OPRO, in contrast, reaches ~92% on the same task. This comparison validates the paper's claim that the optimization trajectory plus exemplars is more effective than explicit genetic algorithm instructions without exemplars.

Ablation Studies and Robustness Checks

All ablation studies in Section 5.3 use text-bison as scorer and PaLM 2-L as optimizer on GSM8K and BBH sports_understanding, with 3 repetitions and standard deviation shading.

Instruction ordering in the meta-prompt (Figure 7a-b): Ascending order by score (worst first, best last — the default) achieves both higher final accuracy and faster convergence than descending (best first, worst last) or random ordering. On GSM8K, ascending order reaches approximately 68% training accuracy by step 50, compared to ~63% for descending and ~61% for random. The paper hypothesizes this is due to recency bias (Zhao et al., 2021): "the optimizer LLM output is affected more by the past instructions closer to the end of the meta-prompt." By placing the best instructions last, the optimizer is most strongly influenced by high-quality examples when generating new candidates.

Effect of instruction scores (Figure 7c-d): Three conditions are compared: 100-bucket score representation (integer-rounded accuracy, default), 20-bucket representation (coarser granularity), and no scores at all (only instructions in ascending order without their accuracy values). On GSM8K, the no-scores condition plateaus around 58% and shows almost no improvement over steps — without score information, the optimizer cannot distinguish high-quality from low-quality past instructions and thus cannot steer toward better solutions. The 20-bucket condition achieves intermediate performance (~64%), suggesting that score granularity matters but 100 buckets (integer percentages) is sufficient and further granularity would not help.

Effect of exemplars (Figure 7e-f): Three conditions: 3 exemplars (default), 10 exemplars, and no exemplars. On GSM8K, the no-exemplar condition plateaus at approximately 57% — comparable to the no-scores condition — demonstrating that exemplars are equally essential. The 10-exemplar condition initially performs similarly to 3 exemplars but shows a shallower upward trend, with the paper hypothesizing that "including more exemplars results in a longer meta-prompt with a dominating exemplar part, which may distract the optimizer LLM from other important components like the optimization trajectory." This is a practical insight: the meta-prompt has limited "attention budget" from the LLM, and too many exemplars dilute the optimization signal from the trajectory.

Number of generated instructions per step (Figure 8): Compared at equal total evaluation budgets (x-axis = total number of evaluated instructions): 1 instruction/step × 1600 steps, 2/step × 800 steps, 4/step × 400 steps, 8/step × 200 steps (default), 16/step × 100 steps. The default of 8 achieves the best performance. Batch size 1 (pure sequential) shows high variance and slower improvement — consistent with the paper's claim that generating multiple solutions per step reduces variance analogous to mini-batch gradient descent. Batch size 16 shows slower improvement because "more optimization steps to incorporate richer information of past instructions with their accuracies" is sacrificed — with only 100 steps of feedback, the optimizer has fewer opportunities to learn from evaluations.

Starting point (Figure 9): On GSM8K with text-bison scorer and Q_begin instructions (Figure 9a), three starting configurations are compared: empty string alone, "Solve the following problem." alone, and all three of empty string + "Solve the following problem." + "Let's solve the problem." The final accuracies after 200 steps are similar (~66–68%) regardless of starting point, and the generated instructions show similar styles (all containing phrases like "solve this problem"). On GSM8K with PaLM 2-L scorer and A_begin instructions (Figure 9b), starting points matter more: "Let's solve the problem." (default) outperforms the empty string in the first ~30 steps, while "Let's think step by step." (already a strong instruction) outperforms both throughout, plateauing near 72% while the others reach ~68% and ~62% respectively. The paper notes that "it takes the optimizer LLM more steps to get rid of worse instructions presented in the meta-prompt when starting from instructions with lower accuracies" — a finding that motivates future work on accelerating convergence from weak starting points.

Optimizer temperature (Figure 10): Temperatures of 0.0, 0.5, 1.0 (default), 1.5, and 2.0 are compared. Temperature 0.0 produces flat optimization curves — the optimizer "often gets stuck at the same instruction for tens of steps." Temperature 0.5 shows some improvement but plateaus lower than 1.0. Temperature 1.0 achieves the best final performance and steadiest upward trend. Temperatures 1.5 and 2.0 produce erratic curves — at 2.0, the optimizer "more often ignores the trajectory of previous instructions" and generates essentially random proposals, breaking the feedback loop. The finding that 1.0 is optimal for both GSM8K and sports_understanding (Figures 10a and 10b) suggests this may be a robust default across tasks.

One-step instruction generation vs. iterative optimization (Section 5.3): The comparison between generating 50 instructions in a single step (no trajectory, only initial instruction + exemplars + meta-instructions) and OPRO's iterative 200-step process reveals: on GSM8K, the one-step baseline's best instruction is still "Let's solve the problem." (the initial seed) with 64.4% training and 60.8% test accuracy — no improvement at all over the starting point. OPRO found "Let's do the math!" at Step 5 with 78.2% training accuracy. On BBH sports_understanding, the one-step best achieves 84.0% training and 80.0% test, while OPRO finds an instruction at Step 4 with 88.0% training. This ablation confirms that the iterative trajectory — seeing how scores evolve across attempts — is essential for the optimizer to learn what constitutes improvement, not merely a nice-to-have.

Critical Assessment

Claim: "Optimized prompts outperform human-designed prompts by up to 8% on GSM8K"

This claim is well-supported by the specific experimental evidence but represents an upper-bound estimate under favorable conditions. The 8% figure comes from comparing 80.2% (OPRO's best with PaLM 2-L-IT optimizer, PaLM 2-L scorer) against 71.8% (the "Let's think step by step" baseline), yielding an 8.4 percentage point absolute improvement (Table 4). However, several contextual factors limit the generality of this claim:

The 8% figure is model-specific. When text-bison is the scorer, the best OPRO instruction achieves only 68.5% vs. 64.4% for "Let's think step by step" — a 4.1 point improvement, roughly half of the 8% figure. The magnitude of improvement depends strongly on the scorer LLM and its baseline sensitivity to prompting.

The comparison is zero-shot only. The "Let's think step by step" baseline is evaluated in the zero-shot setting. Few-shot chain-of-thought prompting (Wei et al., 2022) with 8 exemplars achieves higher accuracy on GSM8K than any zero-shot instruction, including OPRO-optimized ones. The paper acknowledges this implicitly by evaluating only zero-shot instructions (no few-shot exemplars in the scorer prompt), but the claim of "outperforming human-designed prompts" should be understood as "outperforming zero-shot human-designed prompts."

The training accuracy metric used during optimization correlates with but does not perfectly predict test accuracy. Section 5.4 shows that training accuracies are "often 5%-20% higher than our test accuracies" (e.g., on GSM8K, "Take a deep breath..." has 80.2% training but the test accuracy is also 80.2% — a case with no gap — but many BBH tasks show larger gaps). The optimization selects instructions based on training accuracy, and the assumption that training ranking equals test ranking is empirically validated (Figure 11) but not guaranteed.

The optimization required approximately 200 steps × 8 instructions = 1,600 total instruction evaluations, each requiring scorer LLM calls on the training subset. This is a substantial compute cost that is not factored into the "performance improvement" metric — the 8% accuracy gain comes with a significant one-time optimization cost.

Claim: "Optimized prompts outperform human-designed prompts by up to 50% on Big-Bench Hard tasks"

This claim is supported but requires careful interpretation of "50%." The paper states the improvement can be "over 50%" (Section 1), and the largest absolute gain in Table 7 is on word_sorting with PaLM 2-L scorer: OPRO achieves 54.4% vs. 4.0% for "Let's think step by step" — a 50.4 percentage point absolute improvement, which is a ~13.6× relative improvement (or ~1,260% relative). On sports_understanding, the improvement is 90.0% vs. 47.2% — a 42.8 point absolute gain (91% relative). These are genuinely large improvements. However:

These represent the upper tail of a distribution. On 19/23 tasks, the improvement is over 5 percentage points (Section 5.2.2), but on some tasks, OPRO underperforms the baseline — most notably on tracking_shuffled_objects_seven_objects (19.6% vs. 60.8%, a 41.2 point decrease). The 50% figure cherry-picks the best case; the average improvement across all 23 tasks is considerably smaller.

The baselines are not optimized for each task. "Let's think step by step" is a generic reasoning prompt. A human expert designing prompts specifically for word_sorting would likely produce something closer to OPRO's "Alphabetical order of given words:" rather than using "Let's think step by step." The comparison is between an automatically optimized task-specific prompt and a generic one-size-fits-all prompt — a comparison that favors OPRO by construction.

The improvement metric conflates two effects: (1) the "step by step" instruction being actively harmful for some tasks (like word_sorting, where sequential reasoning is irrelevant) and (2) OPRO finding genuinely better instructions. For tasks where "step by step" is harmful, most of the "improvement" comes from removing the harmful instruction rather than discovering a uniquely good one. This is evidenced by the fact that on many tasks, the empty string baseline (no instruction at all) already outperforms "Let's think step by step" (Table 7: word_sorting with PaLM 2-L scorer shows 22.0% for empty string vs. 4.0% for "step by step"), so OPRO's 54.4% represents improvement over both removing the harmful instruction and adding useful guidance.

Claim: "LLMs are able to optimize different kinds of objective functions simply through prompting"

This claim is supported for small-scale problems but bounded by clear failure modes. The linear regression and TSP results (Tables 2 and 3) demonstrate that LLMs can perform black-box optimization on problems with up to ~50 decision variables (TSP nodes), finding solutions competitive with heuristic algorithms at small scales. However, the boundaries are sharp:

Scale dependence is severe. On TSP, performance degrades from optimal (0% gap at n=10 for all models) to unusable (30.4% gap for text-bison, 219.8% at n=50) — a superlinear degradation that makes the approach impractical beyond small instances. The context window limitation is a hard constraint that no amount of prompt engineering can fully circumvent.
The Rosenbrock function failure (Appendix A, Figure 13) demonstrates that LLMs struggle with optimization landscapes that are challenging for traditional optimizers as well. Getting stuck at (0, 0) rather than finding the global optimum at (20, 400) indicates that the LLM's optimization capability is not independent of the objective function's geometry — it shares failure modes with gradient-based methods (getting trapped by misleading local information) while adding LLM-specific failure modes (hallucinating function values, failing to follow formatting instructions).
The paper does not compare against traditional optimization algorithms (gradient descent for linear regression, specialized TSP solvers beyond Gurobi for optimality gap computation, evolutionary algorithms, Bayesian optimization). The claim is that LLMs "are able to optimize" — a capability demonstration — not that they are competitive with state-of-the-art methods. This is an appropriate scope limitation, but it means the practical value of LLM-based mathematical optimization (as opposed to prompt optimization) remains unproven.

Claim: "LLMs serve as effective prompt optimizers that discover instructions outperforming human-designed baselines"

This claim is the paper's most robust finding, supported across multiple optimizer-scorer pairs, two benchmark families, and 23 diverse BBH tasks. The ablation studies systematically validate the importance of each meta-prompt component. However, several experiments that would strengthen this claim are absent:

Missing: Comparison against a random search baseline with the same evaluation budget. The paper compares against one-step generation (50 instructions in one step) but does not compare against generating 1,600 random instructions (or 1,600 instructions from a meta-prompt with only exemplars but no trajectory). This would distinguish whether the optimization trajectory is truly guiding improvement or whether simply evaluating many instructions (some of which will be good by chance) is sufficient. The one-step baseline with 50 instructions hints at this (no improvement over the starting point on GSM8K), but 50 is far fewer than the 1,600 evaluated in the full optimization.

Missing: Evaluation of whether optimized instructions overfit to the specific scorer LLM. The paper evaluates transfer across datasets (GSM8K → MultiArith, AQuA) but not transfer across scorer LLMs. An instruction optimized for PaLM 2-L might not transfer to text-bison or GPT models — this is critical for practical use where the scorer LLM may be updated or changed.

Missing: Analysis of instruction diversity and mode collapse. The trajectory retains only the best 20 instructions. If these 20 become very similar (mode collapse), the optimizer may stop exploring. The paper's temperature ablation (Figure 10) shows that temperature 0.0 causes getting stuck, but there is no quantitative analysis of instruction diversity (e.g., semantic similarity metrics, n-gram overlap) across optimization steps to diagnose whether and when mode collapse occurs.

Missing: Human evaluation of instruction quality. All evaluation is automated (scorer LLM accuracy). There is no assessment of whether the optimized instructions are interpretable, sensible, or would be considered good by human prompt engineers. Some instructions in Table 8 and 9 are quite elaborate (e.g., multistep_arithmetic_two: "The order of operations in mathematics is PEMDAS..."), while others are cryptic (dyck_languages: "{ }"). Whether these instructions are robust or merely exploit quirks of the scorer LLM is untested.

Missing: Computational cost analysis. The paper does not report the total API cost or wall-clock time for a full 200-step optimization. With 8 instructions per step × 200 steps = 1,600 optimizer LLM calls + 1,600 × (training set size) scorer LLM calls, this is a substantial expense. For GSM8K with 262 training examples, this is 1,600 × 262 ≈ 419,200 scorer LLM inferences. A cost-benefit analysis comparing this one-time optimization cost against the ongoing inference savings from using a better prompt would help practitioners decide whether OPRO is worth running.

Claim: "The optimization trajectory enables the LLM to discover patterns of high-quality solutions"

This claim, central to the paper's conceptual contribution, is supported by the ablation studies but not directly demonstrated. The evidence is indirect: removing the trajectory (one-step generation) or removing scores from the trajectory (no-scores condition, Figure 7c-d) hurts performance, implying the trajectory is useful. But the mechanism by which the LLM uses the trajectory — whether it identifies patterns, extrapolates trends, or simply generates paraphrases of the best instructions — is not empirically isolated.

A direct test would be to provide the trajectory but with scrambled scores (so the pattern of what constitutes improvement is corrupted while the set of instructions remains the same). If the LLM is genuinely identifying patterns of quality, scrambled scores should eliminate improvement. If the LLM is simply generating variations of whatever instructions appear in the prompt (regardless of their scores), performance would be similar. This experiment is not run.

Similarly, the claim about recency bias (that ascending order is better because the LLM attends more to the end of the prompt) is plausible but not directly tested. If recency bias is the mechanism, then placing the best instructions at the beginning and explicitly instructing the LLM to "pay special attention to the first instructions" should produce different results — but this is not evaluated.

Overall Experimental Strengths

Broad model coverage: The evaluation spans 5 optimizer LLMs across 2 model families, demonstrating that OPRO is not model-specific.
Diverse task coverage: 23 BBH tasks test generalization across reasoning types beyond arithmetic.
Thorough ablation design: Each meta-prompt component is systematically varied with controlled budgets and multiple repetitions.
Honest failure reporting: Appendix A catalogs specific failure modes rather than hiding them.
Practical hyperparameter defaults: The paper converges on reasonable defaults (temperature 1.0, 8 per step, 3 exemplars, 20 trajectory items) validated by the ablations.

Overall Experimental Weaknesses

Single benchmark family for prompt optimization: All prompt optimization experiments use GSM8K and BBH. No code generation, dialogue, summarization, or other NLP tasks are tested.
No confidence intervals on final test accuracies: Test accuracies are reported as point estimates without error bars (Tables 4, 7, 10, 14), making it impossible to assess whether differences between optimizer LLMs or between OPRO and baselines are statistically significant.
The 20-instruction trajectory limit is not ablated: The paper keeps the best 20 instructions but never varies this number (e.g., 5, 10, 20, 50). The optimal trajectory length likely interacts with context window size and problem complexity.
Optimization hyperparameters (temperature, batch size, exemplar count) are tuned on the same tasks used for evaluation, raising mild overfitting concerns, though the cross-task consistency of optimal hyperparameters (Figure 10 shows temperature 1.0 works for both GSM8K and sports_understanding) partially mitigates this.
The mathematical optimization experiments are small-scale demonstrations (n≤50 for TSP, 2D for linear regression) that do not establish competitiveness with traditional methods at practical scales.

6. Limitations and Trade-offs

The Difficulty Estimation and Trajectory Construction Cost Is a Hidden Overhead

The constraint. OPRO requires a training set to compute the objective function (accuracy) that guides optimization — "prompt optimization requires a training set to compute the accuracy that guides the optimization process. Currently the training set at least contains tens of samples, so that the optimized prompt does not severely overfit to the training samples" (Section 7, Conclusion). For prompt optimization, each candidate instruction must be evaluated by the scorer LLM on this training subset. At the default settings — 8 instructions per step × up to 200 steps = up to 1,600 candidate evaluations — each evaluation requires running the scorer LLM on the entire training subset (262 examples for GSM8K, ~50 for each BBH task). This yields approximately 1,600 × 262 ≈ 419,200 scorer LLM inferences for GSM8K. The meta-prompt itself also consumes substantial context length, as it contains up to 20 full-length instructions, each with its integer score, plus 3 complete task exemplars (often multi-sentence math word problems), plus meta-instructions — the full input can run to thousands of tokens per optimizer LLM call.

The consequence. The headline accuracy improvements (e.g., +8% on GSM8K zero-shot) come with a substantial one-time optimization cost that is never quantified in the paper. A practitioner deciding whether to run OPRO versus manually writing a few candidate prompts has no cost-benefit framework. If each scorer LLM call costs $0.01 and each optimizer LLM call costs$ 0.01 (roughly API pricing at the time), the full optimization could cost thousands of dollars — easily exceeding the cumulative savings from a 5% accuracy improvement in most production scenarios. This cost is front-loaded (paid once upfront) while the benefit accrues over future inferences, but the paper provides no break-even analysis. Moreover, the optimization cost scales with training set size — the paper uses 3.5% of GSM8K (~262 examples) and claims this is "sufficient," but the minimum viable training set size is not ablated. A practitioner with a very small task (e.g., 10 examples) cannot know whether OPRO would still work or would catastrophically overfit.

Evidence in the paper. The paper never reports the total FLOPs, API cost, or wall-clock time for a complete optimization run. Section 5.4 acknowledges overfitting: "training accuracies are often 5%-20% higher than our test accuracies" (Section 5.4) — a gap that grows with optimization steps — but there is no systematic study of how training set size affects either optimization success or overfitting severity. The one-step baseline (50 instructions in a single step without trajectory) serves as a weak proxy for a cost-controlled comparison, but 50 evaluations is far fewer than the 1,600 in the full optimization, making it a straw-man baseline. The paper does not compare OPRO's 1,600-evaluation optimization against, say, simply evaluating 1,600 randomly generated instructions and picking the best — a comparison that would reveal whether the trajectory-guided optimization is more cost-effective than brute-force search.

Mitigation status. Not addressed. The paper flags the general challenge of training set size ("A promising direction is to incorporate richer feedback about the error cases besides the aggregated accuracy... and potentially further reduce the example set size needed for prompt optimization," Section 7) but does not study the cost-benefit tradeoff, provide cost estimates, or compare against cost-equivalent baselines.

The Optimizer Is Fundamentally Limited to Problems Where the Base Model Already Has Non-Trivial Capability

The constraint. OPRO can only optimize solutions that the optimizer LLM can, in principle, generate. If the optimizer LLM has no understanding of the task domain or consistently produces irrelevant solutions, no amount of trajectory feedback can correct this — the optimization is bounded by the LLM's prior knowledge and generation capabilities. For prompt optimization, this manifests as a dependency on the starting point quality: "it takes the optimizer LLM more steps to get rid of worse instructions presented in the meta-prompt when starting from instructions with lower accuracies" (Section 5.3, Starting Point ablation). For mathematical optimization, the limitation is starker: on TSP with n=50, gpt-3.5-turbo achieves a 133.0% optimality gap and text-bison reaches 219.8% — their generated solutions are so poor that the trajectory contains no useful signal for improvement (Table 3). The optimizer is stuck in a region of the solution space where all proposals are bad, and it cannot "see" the path to better solutions because the trajectory provides no gradient — a cold-start problem.

The consequence. OPRO cannot bootstrap itself from zero task knowledge. For prompt optimization on a completely novel task type — imagine optimizing prompts for a new programming language, a specialized scientific domain, or a task requiring capabilities the optimizer LLM lacks — the optimizer would generate instructions that miss the point entirely, score poorly, and provide no gradient for improvement. The optimization would flatline at the starting point quality. This is a fundamental capability boundary: test-time optimization (OPRO) amplifies existing capability but does not create capability that is not already latent in the optimizer LLM. The paper's TSP results demonstrate this boundary empirically: at small scales where the LLM can reason about route structure, optimization succeeds; at large scales where the problem exceeds the LLM's spatial reasoning capacity, optimization fails catastrophically. For prompt optimization, this means OPRO is most useful for tasks where the user can already write a mediocre prompt (the LLM understands the domain well enough to produce task-relevant instructions when shown exemplars) — it cannot help with tasks the LLM fundamentally misunderstands.

Evidence in the paper. The TSP scaling results (Table 3) show that performance degrades superlinearly: text-bison goes from 0.0% optimality gap at n=10 to 30.4% at n=20 to 219.8% at n=50. The paper describes this explicitly: "The performance of OPRO degrades dramatically on problems with larger sizes" (Section 3.2). The starting point ablation (Figure 9b) shows that when starting from the empty string on GSM8K with PaLM 2-L scorer, OPRO takes many more steps to reach the same performance as starting from "Let's solve the problem" — and never catches up to starting from "Let's think step by step" (which is already a strong instruction). The Rosenbrock function failure (Appendix A, Figure 13) shows the optimizer getting permanently stuck at (0, 0) because "it is hard to further navigate x and y along the narrow valley in the loss landscape towards (20, 400)" — the LLM infers from local trajectory information that (0, 0) is the optimum, lacking the global perspective to recognize the narrow valley.

Mitigation status. Partially addressed. The paper acknowledges this limitation ("how to reduce the sensitivity to initialization and better balance exploitation with exploration remains a challenge," Section 7) and provides the starting point ablation showing that better initial prompts accelerate convergence. But no solution is proposed — the paper does not explore curriculum approaches (starting with easier sub-tasks), multi-start optimization (running OPRO from multiple random starting points and selecting the best), or hybrid approaches that combine OPRO with traditional search methods for cold-start scenarios.

The Approach Is Single-Benchmark and Does Not Establish Generalization Across Task Types or Model Families

The constraint. All prompt optimization experiments use exactly two benchmarks: GSM8K (math reasoning) and Big-Bench Hard (23 diverse but individually small reasoning tasks). All experiments use models from the PaLM 2 and GPT families. The paper does not evaluate on code generation benchmarks (HumanEval, MBPP), dialogue tasks, summarization, translation, knowledge-intensive QA, safety-critical tasks, or any task where the output is not a short exact-match answer. The mathematical optimization case studies are limited to 2D linear regression and TSP — no higher-dimensional continuous optimization, no constrained optimization beyond TSP's implicit constraints, no multi-objective optimization.

The consequence. The paper's central claim — "LLMs serve as optimizers" — is demonstrated only for a specific slice of the optimization landscape: discrete optimization over short text strings where the objective function is exact-match accuracy on reasoning tasks. A practitioner working on optimizing prompts for a customer-support chatbot (where quality is measured by user satisfaction scores, not exact match), a code generation system (where prompts must produce syntactically valid code), or a creative writing application (where "good" is subjective and multi-dimensional) cannot assume OPRO will work. The transferability experiment (GSM8K → MultiArith and AQuA, Table 6) suggests intra-domain transfer is possible, but this is still math → math — there is no inter-domain transfer test. The prompt optimization results could be specific to the interactive pattern of "insert instruction → LLM reasons step-by-step → output answer," and may not extend to prompts that, for instance, specify output formatting, control model persona, or constrain response length. The mathematical optimization results could be specific to problems with smooth, monotonic objective landscapes — the Rosenbrock failure hints at this — and may not extend to problems with many local optima or deceptive gradients.

Moreover, the model specificity of optimized prompts is demonstrated (different optimizer-scorer pairs produce different top instructions with different accuracies in Table 4) but not systematized. An instruction optimized for PaLM 2-L-IT as optimizer and PaLM 2-L as scorer might work poorly with text-bison as scorer, but the paper never tests scorer → scorer transfer. If optimized prompts are fragile to model choice, the practical value of OPRO is reduced — organizations that frequently update or A/B test different scorer LLMs would need to re-optimize prompts each time.

Evidence in the paper. The benchmark selection is deliberate: "GSM8K and Big-Bench Hard... are reasoning benchmarks where prompting techniques have achieved remarkable performance breakthrough" (Section 1). The paper reports results on 23 BBH tasks, but these are all structured as short-answer reasoning problems (multiple choice, classification, short text generation). Table 7 shows per-task results where OPRO underperforms baselines on tasks like tracking_shuffled_objects_seven_objects (19.6% OPRO vs. 60.8% baseline) and sometimes barely improves over the empty string (date_understanding: 52.0% OPRO vs. 44.8% empty string with text-bison scorer). This per-task variance suggests that OPRO's effectiveness is task-dependent in ways not fully characterized. The paper does not report any experiments on non-reasoning tasks. All prompt optimization uses exact-match accuracy as the objective — the paper does not discuss whether OPRO would work with BLEU, ROUGE, human preference scores, or other evaluation metrics.

Mitigation status. Not addressed. The paper makes no claims about generalization beyond the evaluated benchmarks and does not discuss task-type limitations. The transferability experiment (GSM8K → MultiArith/AQuA) is intra-domain and does not address the broader generalization question.

The Meta-Prompt Does Not Incorporate Error Case Information, Limiting the Efficiency of Improvement

The constraint. The optimizer LLM sees only aggregate accuracy scores for each candidate instruction — not which training examples each instruction got wrong or why it got them wrong. The meta-prompt includes exemplars randomly sampled from the training set (or chosen as ones "the previous instructions fall short of," Section 4.2), but these are used only to demonstrate the task format, not to provide diagnostic feedback about specific failure modes. The paper explicitly acknowledges this: "one limitation of our current implementation is that the optimizer LLM does not effectively utilize error cases in the training set to infer promising directions to improve the generated instructions. In our experiments, we tried including error cases in the meta-prompt rather than randomly sampling from the training set at each optimization step, but the results are similar, indicating that the error cases alone are not informative enough for the optimizer LLM to grasp the cause of the wrong prediction" (Section 7).

The consequence. The optimizer operates with an extremely impoverished feedback signal — a single scalar accuracy score per instruction. This is analogous to optimizing a function using only zeroth-order information (function values) without any gradient or diagnostic feedback. The consequence is slow convergence and inefficient exploration: the optimizer must propose many candidate instructions to empirically discover what works, because it cannot reason about why certain approaches fail. For example, on GSM8K, if the optimizer generates an instruction that produces correct arithmetic but consistently fails on problems requiring unit conversion, the training accuracy drops, but the optimizer has no way to know that unit conversion is the specific weakness — it only sees the aggregate score decrease. It must stumble upon a unit-conversion-aware instruction through trial and error rather than being able to target the deficiency.

This limitation is particularly severe for tasks with heterogeneous examples: if the training set contains a mixture of problem types (e.g., arithmetic, algebra, geometry), an instruction that improves accuracy on arithmetic but worsens it on geometry might show no net accuracy change, and the optimizer would have no signal to disentangle these effects. The one-step baseline comparison (Section 5.3) dramatically illustrates the consequence: without seeing how scores change across attempts (the trajectory provides the optimization signal), the LLM cannot improve at all — but even with the trajectory, the improvement is guided only by aggregate scores, not by diagnosis of specific failures.

The paper's attempted fix — including error cases in the meta-prompt — "results are similar, indicating that the error cases alone are not informative enough for the optimizer LLM to grasp the cause of the wrong prediction" (Section 7). This suggests that simply showing failed examples is insufficient; the optimizer needs more structured feedback (e.g., "this instruction fails on problems requiring multi-step unit conversion" or "this instruction produces answers that are off by a factor of 10 on large-number arithmetic") to efficiently target improvements.

Evidence in the paper. The explicit acknowledgment in Section 7 is the primary evidence. The optimization curves (Figures 1a, 4, 6, 23, 24) show gradual improvement over hundreds of steps, consistent with inefficient zeroth-order optimization. The finding that fine-grained score bucketing (100 buckets vs. 20 buckets, Figure 7c-d) matters suggests the optimizer benefits from more precise score information, but this is still aggregate — it does not provide per-example diagnostics. The paper's observation that semantically similar instructions can have drastically different accuracies (Section 5.2.3: "Let's think step by step" at 71.8% vs. "Let's work together to solve this problem step by step" at 49.4%) suggests that the objective landscape is highly sensitive to subtle phrasing differences that aggregate scores alone cannot explain — the optimizer would need per-example feedback to understand why the combination instruction underperforms despite combining elements of two high-scoring instructions.

Mitigation status. Acknowledged and left as future work. The paper states: "A promising direction is to incorporate richer feedback about the error cases besides the aggregated accuracy, and summarize the key features that distinguish between high-quality and low-quality generated prompts in the optimization trajectory. Such information may inform the optimizer LLM of how to more efficiently improve over the past generated instructions" (Section 7). No implementation or evaluation of such richer feedback is provided.

Hard Problems at the Margin of the Base Model's Capability See Negligible Improvement, and the Difficulty Boundary Is Not Characterized

The constraint. OPRO improves solutions by generating new candidates that resemble high-scoring past solutions. This mechanism fundamentally requires that some high-scoring solutions exist in the trajectory to provide the gradient signal. For problems where the optimizer LLM's initial proposals are all poor — because the task is too hard, the domain is unfamiliar, or the base model simply cannot generate effective solutions — the trajectory contains only low-quality examples with similar (low) scores, providing no directional signal. The paper observes this directly in several places: on the hardest TSP problems (n=50), text-bison and gpt-3.5-turbo produce solutions with >100% optimality gaps and never improve (Table 3); on the hardest BBH tasks, some show near-zero improvement over the starting point (e.g., geometric_shapes with text-bison scorer: from 15.6% empty string to 23.6% OPRO — a modest 8-point gain but still very low absolute performance, Table 7); and the Rosenbrock function gets permanently stuck at (0, 0) (Appendix A, Figure 13).

The consequence. OPRO has an implicit difficulty threshold below which it is ineffective, but this threshold is not characterized, predicted, or detectable a priori. A practitioner cannot know, before running the expensive optimization, whether their problem is "easy enough" for OPRO to help. The paper's GSM8K results are strong (from 60.5% to 80.2% training accuracy) because the starting point is already reasonably good — the base model can solve these math problems with some prompting, and OPRO finds a better prompt. But if the practitioner's task is one where the base model's best zero-shot accuracy is 5%, OPRO would likely provide no improvement — the trajectory would contain mostly 5% scores with no variance, giving the optimizer no signal about what constitutes a better instruction. The paper does not provide any diagnostic to estimate whether a given task-model combination falls above or below this threshold.

This limitation is practically important because it means OPRO cannot be used to "solve" hard tasks — it can only optimize prompts for tasks the base model already handles moderately well. The paper's finding that EvoPrompt degrades performance when starting from generic prompts (Figure 12a) and that the optimizer gets stuck at temperature 0.0 (Figure 10) are both manifestations of this same fundamental dynamic: if the quality signal in the trajectory is too weak or too uniform, optimization fails. The difficulty boundary is most visible in the per-BBH-task results (Table 7): on tasks where the empty string already achieves high accuracy (dyck_languages: 94.8% with PaLM 2-L scorer), OPRO reaches 100.0% — the model already knew how to do the task and OPRO found a marginally better prompt. On tasks where the empty string is very low (multistep_arithmetic_two: 3.6% with PaLM 2-L scorer), OPRO reaches 58.8% — a large relative gain from a weak starting point. But on some tasks where the empty string is low, OPRO provides almost no gain (geometric_shapes: 33.2% → 60.8% is decent, but 60.8% is still poor absolute performance). The paper does not analyze what distinguishes these cases.

Evidence in the paper. The TSP optimality gap degradation (Table 3) is the clearest quantitative evidence: gpt-4 achieves 0.0% gap at n=10, 0.2% at n=15, 1.4% at n=20, and 11.0% at n=50 — a smooth degradation. But gpt-3.5-turbo degrades from 0.0% to 1.2% to 4.4% to 133.0% — a catastrophic collapse between n=20 and n=50, suggesting a threshold effect where the problem complexity exceeds the model's reasoning capacity and optimization collapses entirely. The per-BBH-task variance (Table 7) shows a wide range of absolute improvements from OPRO, but no analysis clusters tasks by difficulty or predicts which will benefit. The starting point ablation (Figure 9b) shows that starting from "Let's think step by step" (already 71.8% test accuracy) saturates faster and reaches a higher plateau than starting from weaker prompts — the stronger the starting point, the more benefit OPRO provides, which is the opposite of what one would hope (that optimization helps most when the starting point is poor).

Mitigation status. Not addressed. The paper does not propose a method for estimating a priori whether OPRO will help on a given task, does not analyze the relationship between starting accuracy and optimization gain, and does not provide guidance for practitioners on when to use OPRO versus alternative methods. The difficulty-dependent effectiveness is observed as an empirical pattern but not formalized or predicted.

The Approach Is Not Compared Against Cost-Equivalent Baselines, Making Efficiency Claims Unverifiable

The constraint. OPRO is an iterative optimization algorithm that consumes a substantial evaluation budget: up to 200 steps × 8 instructions = 1,600 total instruction evaluations, each requiring scorer LLM calls on the full training subset. The paper's main efficiency claim — that OPRO finds better prompts than human-designed ones — compares OPRO's 1,600-evaluation output against static human-designed baselines that required zero automated evaluations. This is not a cost-equivalent comparison: the human prompts cost nothing to produce (beyond the human's time), while OPRO costs 1,600 automated evaluations. A fair cost-equivalent baseline would be: (a) manually writing N candidate prompts (where N is the human-time equivalent of running OPRO computationally), or (b) generating 1,600 random instructions from the LLM (without the trajectory mechanism) and selecting the best, or (c) running a simpler optimization algorithm (random search, grid search over prompt templates, Bayesian optimization with text embeddings) with the same evaluation budget.

The consequence. The paper's headline claim that OPRO "outperforms human-designed prompts by up to 8% on GSM8K" is true under the experimental conditions but does not establish that OPRO is efficient — that it finds better prompts per unit of computation than simpler alternatives. The one-step ablation (50 instructions generated without trajectory, achieving no improvement over the starting point) partially addresses this by showing that the trajectory is necessary, but 50 evaluations is far fewer than 1,600 — it is not a cost-equivalent comparison. The EvoPrompt comparison (Figure 12) uses the same per-step evaluation budget but relies on different meta-prompts, making it a comparison of meta-prompt designs rather than a cost-effectiveness comparison.

Without a cost-equivalent random search baseline, we cannot rule out the possibility that OPRO's 1,600 trajectory-guided evaluations produce roughly the same best instruction as simply generating 1,600 instructions from a meta-prompt containing only exemplars (no trajectory) and taking the best. If the trajectory mechanism primarily serves to diversify the search (preventing mode collapse) rather than to guide it toward better solutions, random search with diversity constraints might match OPRO's performance at the same cost. The paper's finding that temperature 1.0 is optimal (Figure 10) and that lower temperatures cause mode collapse is consistent with the interpretation that diversity maintenance is a primary function of the trajectory mechanism.

For practitioners, the missing cost-equivalent baselines mean the paper does not provide enough information to decide whether running OPRO is worth its computational cost compared to: (a) spending the same compute budget on other optimization methods, (b) spending the same engineer-hours on manual prompt engineering, or (c) simply evaluating a large number of LLM-generated candidate prompts and picking the best. The paper's own data hints that simple approaches might be competitive: on GSM8K, the instruction "Break this down." (3 words, found by pre-trained PaLM 2-L optimizer) achieves 79.9% test accuracy, nearly matching the 80.2% of "Take a deep breath and work on this problem step-by-step." (11 words, found after 107 steps by PaLM 2-L-IT). If a 3-word instruction achieves comparable performance to an 11-word one, and both are found by OPRO, it raises the question of whether much simpler generation + selection strategies could find comparable instructions at lower cost.

Evidence in the paper. The paper compares against one-step generation (50 instructions, not cost-equivalent) and EvoPrompt (different meta-prompt, same evaluation budget condition). It does not compare against random search with the full 1,600-evaluation budget, grid search over instruction templates, Bayesian optimization with text embeddings, or any non-LLM-based prompt optimization method. The ablation studies (Figures 7–10) vary OPRO's hyperparameters but do not include non-OPRO baselines at equal cost. The mathematical optimization experiments do not compare against traditional optimization algorithms (gradient descent for linear regression, evolutionary algorithms or simulated annealing for TSP) running with equivalent computational budgets — the comparisons are only against heuristics (NN, FI) that run in milliseconds without iterative search. The paper explicitly acknowledges: "OPRO is designed for neither outperforming the state-of-the-art gradient-based optimization algorithms for continuous mathematical optimization, nor surpassing the performance of specialized solvers for classical combinatorial optimization problems" (Section 3.2, Limitations) — so the lack of comparison is declared, but this scope limitation also means the paper cannot claim OPRO is efficient relative to traditional methods for these problems.

Mitigation status. Not addressed. The paper does not propose or run cost-equivalent baselines, does not normalize accuracy improvements by computational cost, and does not discuss what comparison would be necessary to establish efficiency. The one-step baseline and EvoPrompt comparison are steps in the right direction but do not close the gap.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not merely propose a new prompt optimization method — it establishes a new class of optimization procedure where the optimizer is a language model operating purely through the text-input/text-output interface, requiring no gradients, no formal problem encoding, and no model training. The conceptual shift moves from "we optimize prompts for LLMs" (gradient-based soft prompt tuning, RL-based prompt search) and "we use LLMs as operators within optimization algorithms" (mutation and crossover in evolutionary algorithms) to "the LLM is the optimizer" — a self-contained optimization agent that observes solution-score pairs in natural language and generates improved candidates. This is less a refinement of existing optimization techniques and more an expansion of what counts as an optimization algorithm: the optimization "algorithm" is not programmed but rather is whatever optimization strategy the LLM implicitly implements when shown a trajectory of past attempts.

The methodological impact extends beyond the specific prompt optimization application. By demonstrating that the same meta-prompt scaffolding works across mathematical optimization (linear regression, TSP) and prompt optimization — with the same optimizer LLM, the same trajectory format, and the same iterative generation procedure — the paper provides an existence proof for a new category of universal black-box optimizer. This optimizer requires only that the solution space be expressible in natural language and that the objective function be evaluable (by a separate LLM, by computation, or by any external oracle). The formalism is so minimal — "here are solutions and their scores, sorted from worst to best; generate a new solution with a better score" — that it can be applied to any problem where the solution and its quality can be expressed as text. This generality, while bounded by the LLM's capabilities and the context window, is qualitatively different from traditional optimization algorithms that must be programmed with explicit update rules, decision variable encodings, and constraint specifications.

The paper also provides a diagnostic for why prior LLM-based prompt optimization methods succeeded or failed, reconciling apparently contradictory results. APE (Zhou et al., 2022b) and APO (Pryzant et al., 2023) succeeded because they at least implicitly used some form of improvement signal (semantic similarity to top prompts or natural language feedback on a specific prompt). EvoPrompt (Guo et al., 2023) failed on GSM8K when starting from generic prompts because it lacked task exemplars — the meta-prompt instructed the LLM to mutate and cross over prompts, but the LLM did not know what task those prompts were for, making its mutations semantically arbitrary. The paper's experimental demonstration (Section 5.5, Figure 12) clarifies that optimization without task understanding is blind — the trajectory provides the optimization signal, but the exemplars provide the task signal, and both are necessary. This resolution is practically valuable because it tells future researchers: if you want to use LLMs to optimize something, (1) show the LLM examples of the task so it understands what it is optimizing, and (2) show the LLM the history of previous attempts with their scores so it has a gradient to follow. Missing either component causes collapse.

The paper also shifts the research attention from search algorithm design to verifier (scorer) quality and feedback richness. In traditional optimization, the algorithm's update rule — gradient computation, step size scheduling, momentum — is the primary locus of innovation. In OPRO, the "update rule" is whatever the optimizer LLM implicitly learns from the trajectory, and the paper demonstrates that this implicit learning works across multiple LLMs without modification. The bottleneck is not the search algorithm but the quality and granularity of the feedback signal. The ablation showing that removing scores from the trajectory collapses performance (Figure 7c-d) and that finer-grained score bucketing helps (100 buckets vs. 20 buckets) demonstrates that the optimizer needs precise quality differentiation. The failure to utilize error cases — showing the optimizer which examples each instruction got wrong, not just the aggregate accuracy — is identified as a key limitation (Section 7). This reframes the research agenda: instead of asking "how should the LLM generate new candidates?" (the search algorithm question), ask "what feedback signal enables the LLM to identify the most promising directions for improvement?" (the verifier/scorer question). This aligns with the trajectory's implicit-gradient mechanism — the trajectory is only as informative as the scores that distinguish good from bad solutions, and richer feedback (per-example diagnostics, error-type categorization) could provide a much stronger gradient signal.

Finally, the paper establishes that optimization capability is an emergent graded property of LLMs, not a binary capability that either exists or does not. The systematic differences across optimizer LLMs — gpt-4 converges ~4× faster than text-bison on TSP, produces ~2.3× more efficient proposals on linear regression, but also fails more expensively on some tasks — suggest that optimization ability correlates with general reasoning capability but follows distinct scaling patterns. This opens a new evaluation axis for LLMs: not just "how well does this model answer questions?" but "how well does this model observe a trajectory of attempts and propose improvements?" The failure cases in Appendix A (hallucinated function values, getting stuck at non-optimal points, format non-compliance) provide a diagnostic taxonomy that future work can use to benchmark and improve LLM optimization capability specifically, potentially through targeted fine-tuning on optimization trajectories.

Follow-Up Research This Work Enables

Richer feedback signals beyond aggregate accuracy. The paper explicitly identifies the key bottleneck: "the optimizer LLM does not effectively utilize error cases in the training set to infer promising directions to improve the generated instructions" and that "error cases alone are not informative enough for the optimizer LLM to grasp the cause of the wrong prediction" (Section 7). A direct follow-up would replace the scalar accuracy score with structured diagnostic feedback. For GSM8K, this could mean reporting per-problem-type accuracy (addition, subtraction, multiplication, division, multi-step) or categorizing errors (arithmetic error, misunderstanding the problem, unit conversion error). For TSP, this could mean visualizing the route or reporting subpath inefficiencies. The hypothesis is that the optimizer LLM, given richer feedback, would generate more targeted improvements — for instance, if it sees that a prompt achieves 90% on addition but only 30% on division problems, it might generate an instruction specifically addressing division. A strong experiment would compare optimization convergence rates when the meta-prompt includes: (a) aggregate accuracy only (current default), (b) per-example correctness (a list of which training examples were solved correctly vs. incorrectly), (c) error-type categories with counts, and (d) natural language summaries of common failure modes (generated by another LLM analyzing the error cases). The paper's finding that the naive inclusion of error cases did not help suggests the feedback needs to be structured — raw error cases are noise; abstracted error patterns might be signal.

Adaptive difficulty estimation and budget allocation during optimization. The paper observes that optimization is much faster from better starting points (Figure 9b: starting from "Let's think step by step" at 71.8% converges faster than from the empty string at 34.0%) and that some tasks see rapid improvement while others plateau near the starting point. A natural extension is online difficulty estimation: after an initial burst of evaluations (e.g., 5 steps × 8 instructions = 40 evaluations), estimate whether the optimization trajectory shows improvement (a positive slope in the training accuracy curve). If it does, continue with OPRO. If it does not — suggesting either that the task is too easy (already at ceiling) or too hard (no improvement signal) — fall back to an alternative strategy: for easy tasks, stop early and use the best found instruction; for hard tasks, switch to a different initialization or a different optimizer LLM. This would address the cold-start problem identified in the starting point ablation (Section 5.3, Figure 9) by detecting unpromising trajectories early and reallocating compute. A concrete experiment: run OPRO for 200 steps on the full set of 23 BBH tasks, and at step 10, predict the final test accuracy (or whether the optimization will beat the baseline) from the trajectory so far. If the prediction is reliable, early stopping could reduce the average optimization cost by 5–10× without sacrificing final performance. The paper's finding that "Let's do the math!" was found at Step 6 with 78.2% training accuracy — nearly matching the Step 107 best of 80.2% — on GSM8K provides a case study: the majority of the gain occurred in the first 6 steps, and the remaining 101 steps contributed less than 2 percentage points.

Verifier (scorer) robustness and over-optimization detection. The paper's prompt optimization procedure uses the same scorer LLM (e.g., pre-trained PaLM 2-L) for both evaluating instructions during optimization and for final test evaluation. This creates an over-optimization risk: the optimizer LLM might discover instructions that exploit quirks of the scorer LLM rather than genuinely improving task performance. Section 5.4 shows that overfitting does occur (training accuracies 5–20% higher than test accuracies) and that validation curves track training curves, but a more systematic study is needed. A follow-up would train a separate "adversarial" scorer — perhaps a different model or a differently-prompted version of the same model — and check whether instructions optimized against one scorer transfer to the other. If an instruction achieves 80% on PaLM 2-L but 50% on text-bison (both as scorers), it is likely exploiting PaLM 2-L-specific quirks. The paper's transferability experiment (GSM8K → MultiArith, AQuA, Table 6) tests cross-dataset transfer but not cross-scorer transfer. A strong experiment would: (a) optimize instructions against scorer A, (b) evaluate against scorer A (standard), (c) evaluate against scorer B (cross-scorer transfer), and (d) optimize separately against scorer B and evaluate against scorer A. The correlation between scorer-A-optimized and scorer-B-optimized instruction rankings would quantify how much of OPRO's gain is scorer-specific exploitation versus genuine task-relevant improvement. This connects to the broader verifier over-optimization problem identified in the test-time compute scaling literature — the paper's finding that the scorer LLM's evaluation is the sole feedback signal means that any bias or noise in the scorer directly shapes the optimization trajectory.

Combining OPRO with traditional optimization techniques for hybrid search. OPRO is a zeroth-order optimizer — it uses only function evaluations, no gradient estimates. Traditional derivative-free optimization methods (evolutionary strategies, Bayesian optimization, simulated annealing) have well-understood theoretical properties and convergence guarantees that OPRO lacks. A natural hybrid would use OPRO to propose candidate solutions (leveraging the LLM's semantic understanding of the solution space) and a traditional optimizer to manage population diversity, mutation rates, and selection pressure. For prompt optimization, this could mean: (a) maintaining a population of 100 instructions, (b) at each generation, using OPRO's meta-prompt (with trajectory of the top 20) to generate 8 new instructions, (c) using a genetic algorithm's selection and crossover operations on the broader population (including lower-ranked instructions that OPRO would discard) to maintain diversity, and (d) using a Bayesian optimization acquisition function to decide which instructions to evaluate next (balancing exploration of uncertain regions against exploitation of high-scoring regions). The paper's EvoPrompt comparison (Section 5.5) shows that pure genetic algorithms (without task exemplars) fail catastrophically, suggesting the hybrid should retain OPRO's exemplar + trajectory scaffold but add the diversity maintenance and structured exploration that traditional optimizers provide. The finding that temperature 1.0 balances exploration and exploitation (Figure 10) — and that temperatures 1.5+ cause the optimizer to ignore the trajectory — suggests that the LLM alone cannot maintain adequate exploration diversity; an external diversity mechanism might allow higher effective temperatures without trajectory-ignoring collapse.

Scaling laws for LLM-based optimization: optimizer model size vs. optimization performance. The paper evaluates 5 optimizer LLMs of varying sizes and capabilities (PaLM 2-L, PaLM 2-L-IT, text-bison, gpt-3.5-turbo, gpt-4) but does not systematically characterize how optimization capability scales with model size, training data, or instruction-tuning status. A scaling law study would answer: does optimization performance (convergence speed, final solution quality, robustness to poor initialization) improve predictably with optimizer model scale? If gpt-4 is ~4× more efficient than gpt-3.5-turbo on TSP (Table 3) and ~2× more efficient on linear regression (Table 2), is this ratio constant across tasks? Does a 10× larger optimizer LLM produce 10× better optimization trajectories, or does the benefit saturate? The paper's findings hint at non-monotonic scaling: gpt-4 produces markedly better TSP solutions at n=50 (11.0% optimality gap vs. 133.0% for gpt-3.5-turbo, Table 3), but for prompt optimization on GSM8K, gpt-4 actually produced the lowest accuracy (74.5% vs. 80.2% for PaLM 2-L-IT, Table 4). This suggests that optimization capability is not simply a function of model scale — instruction-tuning, domain familiarity, and stylistic tendencies all matter. A systematic study would evaluate optimizer LLMs spanning 3–4 orders of magnitude in parameter count on a standardized suite of optimization tasks (linear regression with varying dimensionality, TSP with varying n, prompt optimization on 3–4 benchmark types), measuring convergence speed, final solution quality, and failure rate as a function of model size. The paper's taxonomy of failure modes (Appendix A) provides a starting framework for what to measure: hallucination rate, format compliance rate, trend extrapolation accuracy, and susceptibility to getting stuck at local optima.

Cross-domain optimization transfer and meta-learned optimization strategies. The paper shows that OPRO can optimize prompts for GSM8K, TSP routes, and linear regression coefficients — three qualitatively different problem types — using the same meta-prompt scaffolding. This raises the question: does the optimizer LLM learn a general "optimization skill" that transfers across domains, or does it treat each problem independently? If a model has been exposed to many optimization trajectories across diverse tasks (perhaps through meta-learning or fine-tuning on synthetic optimization data), does it become a better optimizer on new tasks? The OptFormer work (Chen et al., 2022) trained a transformer on hyperparameter optimization trajectories and found transferable optimization capability. An analogous experiment would fine-tune an LLM on a corpus of synthetic optimization trajectories — linear regression, TSP, hyperparameter tuning, prompt optimization — each formatted in OPRO's meta-prompt style (solution-score pairs + problem description + meta-instructions), then test whether the fine-tuned model converges faster or to better solutions on held-out optimization tasks. The paper's observation that pre-trained PaLM 2-L (not instruction-tuned) can serve as an optimizer when given a few-shot meta-prompt format (Figure 21, Appendix C.2) suggests that optimization capability is latent in base models and might be amplifiable through training. A negative result — that fine-tuning on optimization trajectories provides no benefit over the base model — would be equally informative, suggesting that optimization capability is a zero-shot emergent property not improvable through current fine-tuning methods.

Practical Applications and Downstream Use Cases

Automated prompt engineering for production LLM deployments. Organizations deploying LLMs for specific tasks — customer support classification, document summarization, SQL query generation, content moderation — currently rely on manual prompt engineering, an ad-hoc process where developers iteratively tweak prompt wording and observe accuracy changes. OPRO provides a drop-in replacement: the developer provides a small set of labeled examples (the paper shows ~50–250 examples suffice) and specifies the scorer LLM, and OPRO automatically discovers prompts that outperform human-written ones. The paper's GSM8K result — 80.2% test accuracy for "Take a deep breath and work on this problem step-by-step" vs. 71.8% for the widely-used "Let's think step by step" (Table 4) — represents an 8.4 percentage point improvement on a benchmark where performance has been aggressively optimized by the community. For a production system handling millions of queries, an 8% accuracy improvement directly translates to reduced error rates, fewer escalations to human agents, and improved user satisfaction. The transferability result (Table 6: GSM8K-optimized prompts transfer to MultiArith and AQuA with gains of 9–10 points on MultiArith) suggests that prompts optimized on a representative training set generalize to related tasks, reducing the need for per-task optimization. The practical workflow would be: (1) collect 50–250 labeled examples for the task, (2) run OPRO once (hours to a day of compute, depending on scorer LLM cost and training set size), (3) deploy the best discovered prompt, (4) periodically re-run OPRO if the task distribution shifts or the scorer LLM is updated. The paper's overfitting analysis (Section 5.4, Figure 11) provides confidence that training accuracy ranking correlates with test accuracy ranking, meaning the optimization procedure can be trusted without a held-out validation set if labeled data is scarce.

LLM self-improvement loops via automated instruction discovery. The finding that pre-trained PaLM 2-L can serve as both optimizer and scorer (Figure 4b: the model optimizes prompts for its own task performance) enables a self-improvement architecture: a single LLM generates candidate instructions, evaluates them on its own outputs, and iteratively improves its own prompting strategy. This removes the need for a separate scorer LLM (which may have different failure modes) and enables a model to "prompt-engineer itself." For an LLM deployed in an interactive setting (chatbot, coding assistant), this could manifest as an offline process that periodically optimizes the system prompt based on user feedback, or as a runtime process that maintains a population of candidate prompts and selects among them per-query based on estimated difficulty (though the paper does not explore per-query prompt selection). The paper's finding that pre-trained PaLM 2-L generates stylistically consistent A_begin instructions (short prefix phrases like "Here you go:" and "Let's do it:", Section 5.2.1) indicates that the optimizer LLM understands the insertion position and generates format-appropriate instructions even without explicit format instructions in the meta-prompt — a property essential for automated self-improvement systems where the prompt format may be complex.

Data annotation and synthetic data generation with optimized prompts. Many data annotation pipelines use LLMs with carefully crafted prompts to generate labels, extract structured information, or produce synthetic training data. Prompt quality directly affects annotation accuracy and downstream model quality. OPRO enables systematic prompt optimization for these pipelines: given a small set of human-annotated gold labels (50–250 examples), the annotation prompt can be optimized to maximize agreement with human labels. The optimized prompt can then be used to label a much larger unlabeled corpus, with confidence that the labeling accuracy is higher than it would be with a hand-written prompt. The paper's BBH results — e.g., 90.0% on sports_understanding with PaLM 2-L scorer vs. 47.2% for "Let's think step by step" (Table 7, a 42.8 point gain) — demonstrate that OPRO can find task-specific instructions that dramatically outperform generic reasoning prompts. For annotation pipelines where the cost of human labeling is high ( $0.10–$ 1.00 per example), the one-time cost of running OPRO (a few hundred dollars in API costs) is negligible compared to the ongoing savings from higher automated labeling accuracy, which reduces the number of examples requiring human review. The paper's observation that semantically similar instructions can have drastically different accuracies (Section 5.2.3: 71.8% for "Let's think step by step" vs. 49.4% for the semantically combined "Let's work together to solve this problem step by step") underscores why systematic optimization is necessary for annotation pipelines — a subtle wording change can degrade accuracy by 20+ points, and manual trial-and-error is unlikely to navigate this sensitive landscape effectively.