Large Language Models as Analogical Reasoners

ArXiv: 2310.01714

🎯 Pitch

LLMs can ditch hand-crafted reasoning examples entirely—by simply prompting them to recall and generate their own relevant problems and solutions on the fly, a method that outperforms both zero-shot and manual few-shot chain-of-thought. This 'analogical prompting' even boosts GPT-3.5 to match GPT-4 on competitive programming tasks when it first self-generates a tutorial.

1. Executive Summary

This paper introduces analogical prompting, a new prompting approach that guides large language models to solve problems by first self-generating relevant exemplars—and optionally high-level knowledge—in context before tackling the given problem, inspired by how humans recall related past experiences during analogical reasoning. Evaluated on GPT-3.5-turbo, GPT-4, and PaLM 2-L across mathematical reasoning (GSM8K and MATH), code generation (Codeforces), and BIG-Bench reasoning tasks, the method outperforms both 0-shot CoT and manual few-shot CoT, achieving an average accuracy gain of +4% and enabling GPT-3.5-turbo to match GPT-4's 16% Acc@1 on Codeforces when generating both knowledge and exemplars. The approach establishes that self-generated exemplars can substitute for manually labeled ones, but only when the base LLM is sufficiently strong (text-davinci-002 and larger)—with smaller models, retrieval-based CoT remains superior because weaker LLMs fail to produce valid or useful exemplars.

2. Context and Motivation

The Core Problem: Guiding LLM Reasoning Requires Either Generic Advice or Costly Labels

Chain-of-thought (CoT) prompting has emerged as one of the most effective techniques for eliciting complex reasoning from large language models. The core idea—prompting the model to produce intermediate reasoning steps before giving a final answer—significantly improves performance on tasks ranging from mathematical problem solving to logical deduction. However, the paper identifies a fundamental tension in how CoT is deployed in practice, a tension that sits at the intersection of two competing goals: providing specific, relevant guidance for each problem and minimizing the human effort required to prepare that guidance.

The two dominant CoT paradigms embody opposite sides of this tradeoff:

0-shot CoT (Kojima et al., 2022) offers a generic instruction—typically "think step by step"—that applies uniformly to every problem in a task. This requires zero manual labeling and zero per-problem customization. The cost is specificity: a geometry problem about coordinate squares and a probability problem about airplane meals both receive the identical "think step by step" prompt. As the paper demonstrates (Section 6, Tables 1–3), this generic guidance, while useful, can be insufficient for complex tasks like competitive programming on Codeforces, where simply thinking step by step fails to surface the algorithmic patterns (e.g., prefix product, dynamic programming) needed for correct solutions.

Few-shot CoT (Wei et al., 2022b) addresses the specificity gap by prepending several hand-labeled exemplars—complete question-rationale-answer triplets—to the prompt. These exemplars provide concrete demonstrations of the desired reasoning style and structure, enabling the LLM to leverage its in-context learning abilities. But this comes at a steep price: someone must craft or curate these exemplars for every task, and, in the standard formulation, the same fixed set of exemplars is used for all test problems. If the MATH dataset contains problems spanning algebra, geometry, probability, and number theory, a fixed set of 3–5 exemplars drawn from a few of those categories provides reasoning guidance that may be mismatched to the specific problem at hand—an algebra exemplar does little to help a model reason about coordinate geometry (Figure 1, middle panel).

This creates the central research question the paper poses (Section 1):

"Can we achieve the best of both worlds and automate the generation of relevant exemplars to guide LLMs' reasoning process?"

In other words: can we obtain the specificity of few-shot CoT (exemplars tailored to each problem) without the labeling burden, and without the complexity of external retrieval systems?

Why This Problem Matters: The Practical and Conceptual Stakes

The practical importance is straightforward: labeling reasoning exemplars is expensive and task-specific. For every new reasoning benchmark, every new domain of deployment, and every new reasoning style, a practitioner must either settle for generic 0-shot prompting or invest manual effort in curating exemplars. This is a bottleneck for scaling LLM applications to diverse reasoning tasks, particularly in specialized domains (legal reasoning, medical diagnosis, scientific research) where exemplar creation requires domain expertise. A method that automatically generates high-quality, per-problem exemplars removes this bottleneck entirely—it works out-of-the-box on any reasoning task without per-task preparation.

Beyond the labeling cost, there is a deeper conceptual issue: fixed exemplars cannot adapt to problem diversity within a dataset. The MATH benchmark, for instance, contains problems from seven distinct mathematical subjects (algebra, counting and probability, geometry, intermediate algebra, number theory, pre-algebra, and precalculus). A fixed set of five exemplars can cover at most five of these, and even then only at a surface level. The probability that any given test problem sees a truly relevant exemplar is low. The paper's key insight—and the motivation for analogical prompting—is that this mismatch partially explains why few-shot CoT, despite its expense, leaves substantial performance on the table relative to what might be possible with problem-specific guidance.

There is also a cognitive inspiration at play. The paper draws explicitly on the psychological literature of analogical reasoning (Vosniadou & Ortony, 1989; Gentner & Holyoak, 1997; Polya, 2004), where humans, when faced with a novel problem, spontaneously recall related problems they have solved in the past and use the structural similarities between the past and present problems to derive solution strategies. This is not a superficial analogy—it is a deep claim about how to structure LLM prompting. Rather than treating the LLM as a passive reasoner that just needs to "think harder," analogical prompting treats the LLM as a knowledge repository that has encountered vast numbers of problems during pretraining and can be prompted to explicitly surface and repurpose that latent knowledge in context. The paper is, in part, testing whether this cognitive model of reasoning translates to improved LLM performance.

Where Prior Approaches Fall Short

The paper situates its contribution against a landscape of existing solutions, each of which addresses some aspect of the specificity-vs-cost tradeoff but leaves gaps.

Fixed few-shot CoT (Wei et al., 2022b) requires manual labeling and applies the same exemplars to all problems. As argued above, this is both expensive and suboptimal for multi-domain datasets. The paper's results (Tables 1–3) show that analogical prompting consistently matches or exceeds fixed few-shot CoT, confirming empirically that tailored exemplars provide a genuine advantage over generic ones.

Retrieval-based CoT (Zhang et al., 2022b; Shum et al., 2023) attempts to solve the specificity problem by dynamically retrieving the most similar labeled exemplars from a training set for each test problem, using sentence embeddings (e.g., Sentence-BERT) to measure similarity. This improves relevance but introduces new limitations:

• It requires a labeled training set of reasoning exemplars to exist in the first place—this is unavailable for many tasks, especially niche or emerging ones.
• It adds infrastructure complexity: a retrieval pipeline must be built, maintained, and run at inference time, increasing latency and operational cost.
• The relevance of retrieved exemplars is bounded by the training set's coverage. If the training set contains no problems structurally similar to a novel test problem, retrieval cannot help.

The paper explicitly compares retrieval-based CoT to self-generation in Section 6.3, finding an important scaling interaction: retrieval outperforms generation with smaller (weaker) base LLMs, while generation outperforms retrieval with larger (stronger) base LLMs (Table 4). This is because weak LLMs fail to generate valid exemplars, but strong LLMs can draw on their pretraining knowledge to produce exemplars that are not only relevant but potentially more diverse than what a finite training set can offer.

Recitation-based approaches (Sun et al., 2022) prompt LLMs to recite relevant facts before answering open-domain questions. This is closely related in spirit—self-generating context before solving—but focuses on factual recall rather than reasoning process recall. The paper explicitly distinguishes analogical prompting as focusing on "recalling problem-solving and reasoning processes rather than factual knowledge" (Section 2.1). This distinction matters because reasoning exemplars are more complex than facts: they must include not just declarative knowledge (a formula, a definition) but procedural knowledge (the sequence of steps that produces a solution).

Self-consistency, least-to-most, and other CoT enhancements (Wang et al., 2022; Zhou et al., 2022) improve reasoning performance through mechanisms orthogonal to exemplar quality—self-consistency samples multiple reasoning paths and takes a majority vote; least-to-most decomposes problems into subproblems. These techniques are complementary to analogical prompting (the paper shows that combining them yields further gains; Table 8), but they do not address the core challenge of providing relevant, per-problem reasoning guidance without manual labeling.

Prior work on analogical reasoning in LLMs (Webb et al., 2023; Hu et al., 2023) evaluates whether LLMs can recognize analogies (i.e., identify that two problems are structurally similar). The paper positions itself differently: rather than evaluating analogical recognition as an end task, it uses analogical generation as a means to improve downstream problem-solving performance. This is a novel application of the analogical reasoning concept to the prompting methodology itself.

How This Paper Positions Itself

The paper frames analogical prompting not as an incremental improvement over CoT but as a new paradigm that fundamentally reframes the relationship between the LLM's pretraining knowledge and its inference-time behavior. The key conceptual move is this: instead of viewing prompting as a way to instruct the model what to do (think step by step, follow these examples), analogical prompting views prompting as a way to activate and repurpose the model's own stored problem-solving experiences. The LLM has already seen (during pretraining) thousands of worked examples of probability problems, geometry problems, and dynamic programming problems. The prompt's job is to make the model surface those examples in context, where they can serve as in-context learning demonstrations for the current problem.

This framing leads to a specific technical approach with clear advantages that the paper claims over existing methods:

1. No labeling required: The method works on any task without per-task exemplar curation, since exemplars come from the model's own generation.
2. Per-problem adaptation: Each problem gets exemplars tailored to its specific domain (e.g., probability exemplars for probability problems, geometry for geometry), unlike fixed few-shot CoT.
3. Simplicity: The method operates in a single forward pass—no retrieval infrastructure, no multi-step prompting, no external data dependencies.
4. Composability: The method can be augmented with self-generated high-level knowledge ("tutorials") that complement the specific exemplars, and can be combined with orthogonal improvements like self-consistency (Table 8) for further gains.

The paper also positions its contribution relative to model scale. It explicitly acknowledges that analogical prompting's effectiveness is contingent on base LLM strength (Section 6.4, Table 4). With weak models (text-curie-001, text-davinci-001), the method underperforms few-shot CoT with labeled exemplars because the self-generated exemplars are unreliable or irrelevant. With strong models (text-davinci-002 and -003, GPT-3.5-turbo, GPT-4), self-generation matches or exceeds labeled exemplars. This scaling behavior is a central empirical finding and distinguishes analogical prompting from retrieval-based methods, which show the opposite trend (better with small models, relatively worse with large ones). The paper thus situates analogical prompting as a technique for the era of increasingly capable LLMs—a method that becomes more valuable as underlying model quality improves.

3. Technical Approach

3.1 Reader Orientation

Analogical prompting is a prompting method—a specific way of formatting text that you feed into a large language model—that makes the model automatically generate useful, problem-specific examples of how to reason, and then use those examples to guide its own solution to a new problem. The problem it solves is the tradeoff between the cost of manual labeling (few-shot CoT is expensive and generic) and the weakness of no guidance at all (0-shot CoT is too vague); the solution takes the shape of a single prompt that instructs the model to first recall (generate) relevant past problems and their solutions, and then apply the patterns from those recollections to the question at hand.

3.2 Big-Picture Architecture (Diagram in Words)

The system has three major components, all operating within a single prompt template fed once to the LLM:

1. Problem Statement: the raw question or task description to be solved (e.g., a math word problem, a code generation specification, a logical reasoning puzzle). This is the fixed input $x$.

2. Self-Generation Instructions: textual directives that tell the LLM what to generate and in what order. These instructions are hand-designed per task type (math, code, logic) and direct the model to produce two optional intermediate artifacts:

   • Knowledge/Tutorial (optional, used primarily for complex tasks like Codeforces): high-level explanations of the core concepts, algorithms, or theorems relevant to the problem.
   • Exemplars: $K$ self-generated question-rationale-answer triplets that are structurally analogous to the target problem.

3. Solution Generation: the final instruction block that tells the LLM to solve the original problem, now with the benefit of the self-generated knowledge and exemplars in its context window.

Information flow: The prompt template concatenates [Problem] → [Knowledge generation instruction] → [Exemplar generation instruction] → [Solve the original problem] into a single text block. The LLM processes this in one continuous forward pass, autoregressively generating first the knowledge (if prompted), then the exemplars, then the final solution. All generated intermediate content exists as context tokens for the final solution, enabling in-context learning from the model's own outputs.

3.3 Roadmap for the Deep Dive

• First, the prompt template design and the instruction sequencing, because the order of generation (knowledge before exemplars, exemplars before solution) is architecturally load-bearing and explains how self-generated context influences downstream reasoning.
• Second, the self-generated exemplars mechanism in detail—the instruction phrasing, the diversity constraint, the single-pass design choice, and the number of exemplars to generate—since exemplar generation is the core technical novelty.
• Third, the self-generated knowledge plus exemplars extension, which adds a hierarchical structure (abstract tutorial then concrete examples) for complex tasks, and the rationale for why knowledge should precede exemplars rather than follow them.
• Fourth, the technical design decisions that differentiate analogical prompting from alternative approaches (independent generation, retrieval, fixed few-shot), including the empirical scaling behavior that determines when each approach is appropriate.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a prompting methodology paper whose core idea is that LLMs can serve as their own exemplar generators: by carefully structuring instructions within a single prompt, we can make a model recall and re-express its pretraining knowledge of related problems, then use that self-generated content as in-context demonstrations for solving a new problem.

Analogy as a Prompting Principle

The paper draws on the psychological concept of analogical reasoning, where humans "draw from relevant past experiences to tackle new problems" (Section 1). When a human encounters a novel math problem—say, finding the area of a square given four coordinate points—they spontaneously think: "Do I know a related problem?" and recall how they solved similar problems (finding the area of a square with a known side length, computing distances between points). They might also recall abstract principles: "To find a square's area, I need the side length; to find the side length from coordinates, I use the distance formula."

The paper's technical hypothesis is that LLMs contain an analogous capability—they have been exposed to millions of problem-solution pairs during pretraining—and that explicit prompting can surface these latent memories as structured text, which then serves as in-context learning demonstrations. The innovation is not in the cognitive analogy itself, but in the prompt engineering that operationalizes it: a specific instruction template and generation order that reliably produces useful exemplars and knowledge, then leverages them downstream.

Critically, this is not a training method or a model architecture change. No fine-tuning occurs. The model weights are frozen. The entire technique operates purely through prompt design.

The Prompt Template Structure

The core prompt template follows a consistent architecture across tasks, with task-specific variations in the instruction wording. The general structure is (Figure 2 for math, Figure 3 for code):

[Task framing sentence]

# Problem:
[The target problem statement x]

# Instructions:
## [Optional: Algorithms/Tutorial identification]
## Relevant Problems / Example Problems:
Generate K relevant and distinct problems. For each problem:
  - After "Q: ", describe the problem
  - After "A: ", explain the solution and enclose the answer in \boxed{}.
## Solve the Initial Problem:
Q: Copy and paste the initial problem here.
A: Explain the solution and enclose the ultimate answer in \boxed{} here.

The prompt uses # symbols (Markdown header notation) to structure the response format. The paper notes this explicitly: "Using # symbols in the prompt (e.g., # Relevant Problems) helps LLMs structure the response better" (Section 4.1).

What this template does: The instructions are designed to make the LLM follow a specific generation order: first produce the intermediate artifacts (exemplars, optionally preceded by knowledge), then produce the final answer. Because the LLM generates autoregressively left-to-right, when it reaches the "Solve the Initial Problem" section, the previously generated exemplars and knowledge are already in its context window. The model can attend to these self-generated tokens when producing the final solution—this is in-context learning from its own output.

Why this single-pass design? The paper explicitly considered an alternative: "independently generate exemplars by separately sampling them from the LLM and then re-prompt the LLM with all the exemplars" (Section 4.1). This two-step approach (generate exemplars in pass 1, concatenate with problem and solve in pass 2) would work but "our current single-pass prompt approach achieves comparable performance and offers greater convenience, eliminating the need for multiple prompts" (Section 4.1). The single-pass design has practical advantages—lower latency (one API call instead of two), no need to manage intermediate storage—and is conceptually elegant: the model generates its own in-context demonstrations and uses them in the same forward pass.

The "Copy and paste" instruction: The prompt explicitly tells the model to "Copy and paste the initial problem here" before solving. This ensures the problem statement appears immediately before the solution in the model's output, providing clean separation between the exemplar generation phase and the solution phase. Without this instruction, the model might skip restating the problem and jump directly to solving, which could confuse the output structure.

Self-Generated Exemplars: The Core Mechanism

The heart of analogical prompting is the instruction block that makes the LLM generate $K$ relevant problem-solution pairs before solving the target problem. The exact instruction varies by task but follows a consistent pattern.

The generation instruction for MATH (Appendix D.1):

## Relevant Problems:
Recall three examples of math problems that are relevant to the initial problem. 
Note that your problems should be distinct from each other and from the initial 
problem (e.g., involving different numbers and names). For each problem:
- After "Q: ", describe the problem 
- After "A: ", explain the solution and enclose the ultimate answer in \boxed{}.

What the model generates: The LLM outputs $K$ complete exemplars, each consisting of a problem statement (labeled Q:) and a full solution with reasoning steps and a boxed final answer (labeled A:). For instance, when given a probability problem about pilots getting fish meals, GPT-3.5-turbo generated three exemplars about drawing colored balls from bags—all probability problems involving sampling without replacement (Figure 2). These exemplars are not random; they are domain-congruent with the target problem (probability, not geometry or algebra) because the model, having read the target problem, infers the relevant domain and samples from its learned distribution of problems in that domain.

The diversity constraint—why it matters: The instruction explicitly says "your problems should be distinct from each other and from the initial problem (e.g., involving different numbers and names)." The paper identifies this as crucial: "This step is crucial as some LLMs have a tendency to repetitively generate identical problems, which can be misleading when solving the target problem" (Section 4.1). Without the diversity instruction, an LLM might generate three nearly identical exemplars (same problem with slight numeric variations), which provides little additional information beyond the first exemplar and could bias the model toward a too-narrow solution strategy. The paper reports an ablation (Appendix C, Table 6) showing that with diversity instructions on MATH, GPT-3.5-turbo achieves 37.3% accuracy, while without diversity instructions it achieves 35.2%—a 2.1 percentage point drop, confirming that diverse exemplars are measurably better than repetitive ones.

Why distinct exemplars help: Diverse exemplars expose the model to a broader range of problem-solving patterns within the same domain. For a probability problem, one exemplar might involve sampling without replacement from a bag, another might involve independent events, and a third might involve conditional probability. This variety gives the model multiple structural templates to draw from when solving the target problem, rather than overfitting to a single narrow pattern.

The number of exemplars $K$: Through experimentation (Section 6.5, Table 5), the paper found that $K = 3$ to $5$ works best. The table shows:

• $K = 1$: 76.1% on GSM8K, 34.8% on MATH (GPT-3.5-turbo)
• $K = 2$: 77.0% on GSM8K, 36.7% on MATH
• $K = 3$: 77.5% on GSM8K, 37.3% on MATH
• $K = 4$: 77.3% on GSM8K, 37.0% on MATH
• $K = 5$: 77.8% on GSM8K, 37.1% on MATH

The performance stabilizes at $K \geq 3$, with $K = 3$ or $5$ being the best choices depending on the dataset. A single exemplar ($K = 1$) underperforms because the model "overly relies on a single exemplar" and lacks the diversity of patterns needed for robust in-context learning. The paper notes this "aligns with the findings in the standard few-shot in-context learning in LLMs (Brown et al., 2020)"—the established few-shot learning literature also finds that 3–5 demonstrations are typically sufficient.

Per-task $K$ choices: For GSM8K, the paper uses $K = 5$ exemplars; for MATH, $K = 3$; for BIG-Bench tasks, $K = 3$; for Codeforces, $K = 3$. The variation reflects empirical tuning and task complexity—simpler tasks (GSM8K) benefit slightly from more exemplars, while more complex tasks (Codeforces) are already demanding enough in context length that 3 exemplars suffice alongside the knowledge generation.

Self-Generated Knowledge + Exemplars: Adding a Hierarchical Structure

For the Codeforces code generation task, the paper introduces an additional generation step: self-generated knowledge in the form of a tutorial on core algorithmic concepts, generated before the exemplars. The instruction structure (Figure 3, Appendix D.3) is:

## Algorithms:
Identify the core concepts or algorithms used to solve the problem.
## Tutorial:
Write a useful tutorial about these algorithms.
## Example Problems:
Provide three examples of relevant competitive programming problems that involve 
these algorithms. For each problem, describe the problem, explain the solution 
in detail, and then write the correct Python3 code.
## Python3 code to solve the original problem:
- Explanation of the solution:
- Python3 code to solve the problem:

Why knowledge plus exemplars? The paper identifies a failure mode in pure exemplar-based prompting for complex tasks: "LLMs may overly rely on the low-level exemplars and fail to generalize when solving the target problems" (Section 4.2). Exemplars are concrete—they show specific solutions to specific problems. Knowledge ("tutorials") is abstract—it explains the general principles and algorithms that underlie those concrete examples. By generating both, the model builds a two-level representation: the tutorial provides the algorithmic framework (e.g., "the prefix product algorithm works by..."), and the exemplars show concrete instantiations of that framework (e.g., "here is a problem that uses prefix products to find products of subarrays").

This mirrors how expert humans teach: first explain the concept, then show examples. The paper explicitly frames this as a complementarity: "knowledge act[s] as high-level takeaways that complement low-level exemplars, which prevents LLMs from overly relying on specific exemplars and helps to generalize to new problems" (Section 6.2).

Why knowledge BEFORE exemplars? The ordering decision is deliberate and empirically validated. The paper reports that "generating knowledge before exemplars yields superior results" (Section 4.2) compared to generating knowledge after exemplars (Table 7: 15% Acc@1 vs. 14% Acc@1 on Codeforces with GPT-3.5-turbo). The mechanism: when knowledge is generated first, the model first identifies the core concepts, and this identification "in turn, helps LLMs generate exemplars that align more closely in terms of the fundamental problem-solving approaches rather than surface-level lexical similarities" (Section 4.2). In other words, generating a tutorial about the prefix product algorithm before generating exemplars makes the subsequent exemplars more likely to feature the prefix product algorithm, rather than surface-level matches (e.g., problems that mention "arrays" or "products" but use different algorithms).

A concrete example of the benefit: without the knowledge-first instruction, when given a Codeforces problem about finding a split point in an array where prefix product equals suffix product, the model might generate exemplars about palindrome sequences (surface-level text similarity to "sequence"). With the knowledge-first instruction, it generates a tutorial on the prefix product algorithm, and then the exemplars are about prefix product applications—a much better structural match (Section 6.2).

Where knowledge matters most: The paper notes that "the performance gains achieved by generating knowledge are less significant in other tasks like GSM8K and BIG-Bench, however, likely because these tasks are less complex" (Section 6.2). The knowledge generation adds tokens and computational cost, so it is reserved for the most demanding tasks (competitive programming) where the additional abstraction provides genuine benefit.

Design Decision: Single-Pass Generation vs. Independent Generation vs. Retrieval

Why single-pass over independent generation? The paper explicitly considered an alternative: generate exemplars in a separate prompt, then concatenate them with the problem in a second prompt. This two-step approach is viable and "does work" (Section 4.1), but the single-pass approach was chosen for two reasons: (1) it "achieves comparable performance" to the two-step method, and (2) it "offers greater convenience, eliminating the need for multiple prompts." The convenience argument is practical but significant—a single API call has lower latency, lower cost (no intermediate token generation that gets discarded and re-sent), and simpler implementation.

Why self-generation over retrieval? Section 6.3 provides a detailed comparison. Retrieval from a labeled training set has the advantage of reliability: "Exemplars retrieved from a labeled dataset are inherently valid and correct, unlike generated exemplars, which lack this guarantee." Generation has the advantage of being "more self-contained and convenient, as it does not rely on external labeled data or retrieval steps," and can potentially produce exemplars that are better tailored because "it can draw upon the entire (pre-)training data the LLM has been exposed to" rather than being limited to a finite training set.

The crucial empirical finding (Table 4) is that the best approach depends on base LLM strength:

For weak models (curie, davinci-001), retrieval is slightly better because generation fails to produce valid exemplars—the model doesn't have enough stored knowledge to recall useful problems. For strong models (davinci-002, -003), generation outperforms retrieval because the model can produce relevant, correct exemplars that are potentially more diverse than what a finite training set contains. This establishes a clear decision rule: use retrieval for smaller/weaker LLMs, use self-generation for larger/stronger LLMs.

Reasoning About the Mechanism: Why Does This Work?

The paper does not provide a mechanistic analysis of why self-generated exemplars improve downstream performance, but the design choices imply a theory. When the LLM generates exemplars before solving:

1. Domain priming: Generating probability exemplars activates the probability-related knowledge, vocabulary, and reasoning patterns in the model's internal representations via the autoregressive generation process. These activations persist in the model's key-value cache and influence the subsequent solution generation.

2. In-context learning: The generated exemplars serve as explicit demonstrations of the desired reasoning format and solution structure. The model can attend to these demonstrations when generating the final solution, effectively performing few-shot in-context learning where the shots were generated by the model itself.

3. Structural alignment: By generating exemplars that are structurally similar to the target problem (same mathematical domain, same algorithmic technique), the model explicitly surfaces the solution template that applies, making it more likely to follow that template when solving.

The qualitative analysis in Section 6.6 supports this interpretation: in 35/50 correctly solved problems, the generated exemplars were both relevant and correct, suggesting that the model successfully retrieved appropriate problem-solving patterns and applied them. The primary failure mode (28/50 incorrectly solved problems) was when the exemplars were correct and relevant but the model still couldn't solve the new problem—due to "a generalization gap between the exemplars and the new problem" or "overreliance on specific exemplars, leading to misdirection." This indicates that the method works best when the target problem is within the difficulty envelope of the generated exemplars; it breaks down when the target requires a deeper generalization that the exemplars don't adequately capture.

The "Why This Form" for the Prompt Structure

The prompt template has several design properties that matter:

Explicit section markers (# headers): These create a parseable output structure. Without them, the model might blend exemplars and solution text in ways that are hard to disambiguate. The header markers serve as boundary delimiters that the model learns to respect during pretraining (Markdown-formatted documents are common in web training data), making it more likely to produce cleanly separated sections.

"Copy and paste" the initial problem: This instruction ensures the problem appears in the output immediately before the solution, which maximizes the attention the model can pay to the problem statement during solution generation. Without it, there might be a long gap (the exemplars) between the problem statement in the prompt and the solution in the output, which could weaken the model's ability to stay focused on the specific details of the target problem.

Enclosing answers in \boxed{}: This LaTeX boxing convention is commonly used in MATH dataset solutions and math competition problems. By instructing the model to use this format, the prompt aligns with the model's existing knowledge of how mathematical answers should be presented, increasing the likelihood of clean, parseable final answers that can be automatically extracted for accuracy evaluation.

Instruction to be "distinct": As discussed, this is a negative constraint—it tells the model what not to do (generate repetitive problems) rather than only what to do. This is important because LLMs have a default tendency toward repetition (they are trained to maximize likelihood, and repeating a pattern is a locally high-likelihood strategy). The explicit counter-instruction overrides this tendency.

Summary of Design Choices and Their Justifications

• Single-pass generation in one prompt over multi-step independent generation: simpler, comparable performance, lower latency.
• Knowledge before exemplars (for Codeforces) over exemplars before knowledge: primes the model to generate structurally aligned exemplars rather than surface-level matches; empirically 1% better Acc@1.
• $K = 3$–$5$ exemplars over $K = 1$: avoids over-reliance on a single exemplar; consistent with few-shot learning norms.
• Diversity constraint over unconstrained generation: prevents repetitive exemplars that provide redundant information; empirically +2.1% on MATH.
• \boxed{} answer formatting: aligns with model's training distribution for math answers, enabling clean evaluation.
• Self-generation over retrieval for strong LLMs: draws on the full pretraining corpus rather than a finite training set; empirically 4–7% better than retrieval for text-davinci-003 on GSM8K.
• No fine-tuning, no external data: the method is purely a prompting technique, making it immediately applicable to any sufficiently strong off-the-shelf LLM without per-task preparation.

4. Key Insights and Innovations

Innovation 1: Reframing LLM Reasoning as a Self-Contained Analogical Retrieval Process

The paper's most fundamental conceptual move is not a prompting technique—it is a reframing of what happens inside an LLM during reasoning. Prior to this work, the dominant paradigm treated CoT prompting as a way to instruct the model: "think step by step" tells the model what reasoning format to adopt; few-shot exemplars tell the model what reasoning patterns to imitate. In both cases, the model is positioned as a passive executor of externally provided reasoning strategies. The guidance comes from outside the model.

Analogical prompting inverts this relationship. The guiding hypothesis is that LLMs already contain—encoded in their weights from pretraining—a vast repository of problem-solving experiences, and the prompt's role is not to provide reasoning guidance but to activate recall of that stored knowledge. The model is repositioned from an executor of reasoning instructions to an active participant that surfaces its own relevant past experiences and uses them to guide current problem-solving. This is a fundamental shift in how we conceptualize the LLM's role in the reasoning process: from a tabula rasa that must be told how to think, to a knowledge-rich agent that needs only the right cue to retrieve and repurpose what it already knows.

This reframing is significant beyond the specific technique of self-generating exemplars. It suggests a general design principle for LLM prompting: when the model already possesses relevant knowledge from pretraining, the most effective prompts are those that elicit self-expression of that knowledge rather than those that impose external structure. This principle has implications far beyond math problem solving—it applies to any domain where pretraining corpora contain relevant problem-solution pairs (code, legal reasoning, medical diagnosis, scientific reasoning). The paper does not fully explore this generalization, but the conceptual framework is the main intellectual contribution.

The distinction from prior work is sharp. Kojima et al. (2022) showed that generic instructions (0-shot CoT) improve reasoning, but the mechanism was an instruction-following one: the model learns during instruction tuning to produce reasoning steps when told to. Wei et al. (2022b) showed that providing exemplars (few-shot CoT) improves reasoning, but the mechanism was in-context learning from externally provided demonstrations. Retrieval-based CoT (Zhang et al., 2022b) extends this by selecting which external exemplars to show, but still treats exemplars as something the model receives from outside. Sun et al. (2022)'s recitation approach is the closest precursor—it makes LLMs recall facts before answering questions—but operates on factual knowledge, not procedural reasoning patterns. Analogical prompting is the first work to argue that reasoning procedures themselves can be self-recalled and repurposed, and to build a prompting methodology around that claim.

The evidence that this reframing is not just philosophical but practically consequential comes from Table 4: with strong models, self-generated exemplars outperform retrieved exemplars. This means the model's stored knowledge—when properly prompted—provides better guidance than a hand-curated training set. The model is not a blank slate that needs external instruction; it contains reasoning resources that can be activated through the right meta-cognitive prompt.

Significance level: Fundamental shift in framing. This is not an incremental improvement to CoT; it changes what we believe prompting does—from instructing to activating.

Innovation 2: The Scale-Dependence of Self-Generation as a Diagnostic for Model Capability

The paper makes a sharp empirical observation that carries diagnostic weight beyond the specific technique: self-generation of useful exemplars works only when the base LLM crosses a capability threshold. This is not a trivial "bigger models are better" finding. It is a specific claim about what kind of capability is required: the model must have sufficiently internalized the problem domain during pretraining to produce valid, relevant, and diverse exemplars on demand.

Table 4 is the central evidence, and its structure is revealing. At the weak end (text-curie-001, text-davinci-001), self-generated exemplars are essentially worthless—the model cannot produce valid problem-solution pairs, so the method performs no better than 0-shot baselines (2% and 9% on GSM8K, respectively). At these scales, labeled exemplars from few-shot CoT (2% and 10%) and retrieved exemplars (3% and 11%) modestly outperform generation. At the strong end (text-davinci-002, text-davinci-003), the relationship flips: self-generation matches or exceeds both fixed and retrieved exemplars (48% vs. 43–47% for davinci-002; 61% vs. 54–57% for davinci-003).

This finding establishes self-generation quality as a diagnostic signal for model readiness. If a model can generate useful exemplars for a domain, it has internalized that domain's problem-solving patterns to a degree that enables not just problem-solving but problem-formulating and solution-explaining. This is a higher bar than mere accuracy. It also suggests that analogical prompting serves dual purpose: it is both a technique for improving performance and an implicit test of whether the model genuinely "understands" a domain.

The comparison to retrieval-based CoT deepens this insight. Retrieval works better with weak models because it offloads the exemplar-generation burden to an external dataset—the model doesn't need to know how to create good exemplars, only how to use them. Generation works better with strong models because the model's internal knowledge is broader and more flexible than any finite training set. This implies a transition point in model capability where the optimal strategy shifts from external augmentation to internal recall. The paper does not attempt to characterize this transition point theoretically, but its empirical identification is a useful contribution for practitioners deciding between prompting strategies as models evolve.

This stands in contrast to the standard narrative around in-context learning, which typically shows that providing exemplars helps across all model scales (Brown et al., 2020)—bigger models just benefit more. The analogical prompting result is qualitatively different: for small models, self-generation doesn't work at all; it's not that bigger models benefit more, it's that there is a capability floor below which the technique is worse than doing nothing. This non-linear threshold behavior is an important corrective to the assumption that all prompting techniques scale monotonically with model size.

Significance level: Fundamental for model evaluation methodology; practically important for deployment decisions.

Innovation 3: Knowledge Generation as a Structural Priming Mechanism, Not Just Additional Content

The extension from exemplars-only to knowledge-plus-exemplars (Section 4.2) might appear to be a simple additive improvement—more generated content, better performance. But the paper's analysis reveals a subtler and more interesting mechanism: generating abstract knowledge before concrete exemplars changes the kind of exemplars the model generates, not just the volume of useful content in the context window.

The evidence is the comparison in Section 6.2 and Table 7. The performance difference between "knowledge after exemplars" (14% Acc@1 on Codeforces) and "knowledge before exemplars" (15% Acc@1) is modest in absolute terms, but the qualitative mechanism the paper describes is conceptually significant. Without the knowledge-first instruction, the model generates exemplars based on surface-level lexical similarity to the problem statement—a problem about array products might trigger exemplars about palindrome sequences because both mention "sequences" and "finding a point." With the knowledge-first instruction, the model first identifies the core algorithm (prefix product), writes a tutorial about it, and then generates exemplars that structurally match the algorithmic pattern rather than the surface text. The exemplars become algorithmically congruent rather than just lexically similar.

This is a form of structured reasoning about reasoning. The model is not just recalling similar problems; it is analyzing the target problem to identify its core concepts, then using that analysis to guide its recall. This meta-cognitive structure—understand the problem type first, then find examples of that type—is precisely what expert human problem-solvers do (Polya, 2004, is explicitly cited in the paper). The innovation is demonstrating that this meta-cognitive structure can be induced in an LLM through prompt ordering alone, without any architectural changes or training.

The implication extends beyond the specific technique. It suggests that generation order in multi-stage prompting is a first-class design parameter with effects that go beyond simply providing more context. Ordering generation to proceed from abstract to concrete (knowledge → exemplars → solution) is not just about information flow; it's about guiding the model's internal retrieval process so that subsequent generations are more structurally aligned with the problem's deep requirements. This is a generalizable design principle for any prompting task that involves multiple stages of self-generated content.

The finding that knowledge generation helps significantly on Codeforces but less so on GSM8K and MATH (Section 6.2) is also diagnostic. It suggests that knowledge generation is most useful when the task requires identifying and applying specific named algorithms or techniques (prefix product, dynamic programming, graph traversal) that benefit from explicit articulation, rather than general mathematical reasoning that can be captured by examples alone. This boundary condition helps practitioners decide when to invest the additional tokens in knowledge generation.

Significance level: Incremental to the main technique but conceptually significant—it reframes prompt ordering as a retrieval-guiding mechanism, not just an information-presentation choice.

Innovation 4: Reconciling Self-Generation and Retrieval Through a Capability Threshold Model

The paper does not just compare self-generation and retrieval—it establishes a unified picture of when each approach is appropriate, resolving a potential tension in the prompting literature. Prior to this work, one might reasonably ask: should I retrieve exemplars from a dataset, or should I let the model generate its own? The answer might appear domain-dependent or model-dependent, but without a systematic comparison, practitioners would be left guessing.

Table 4 provides a clear, empirical answer that takes the form of a threshold model: below a certain model capability level, retrieval dominates; above it, generation dominates. The threshold in the paper's experiments sits somewhere between text-davinci-001 and text-davinci-002 on the GSM8K task. Below this threshold, the model lacks the stored problem-solving knowledge to generate valid exemplars, so external retrieval is superior. Above it, the model's internal knowledge is richer and more flexible than a finite training set, so self-generation wins.

This reconciliation has practical significance because it turns an either-or debate into a contingent decision rule based on an observable quantity (model scale/capability). It also has theoretical significance because it suggests a natural progression for prompting strategies as models improve: early in a model family's development, retrieval augmentation is necessary; as models become more capable, self-contained methods become not just viable but superior. This progression mirrors broader trends in AI where internalization of knowledge eventually supersedes external knowledge bases (e.g., the shift from retrieval-augmented QA to large-model closed-book QA).

The finding also implicitly addresses a limitation of retrieval-based methods that the paper does not dwell on: retrieval is capped by the coverage of the training set. A training set of 7,500 GSM8K problems contains a finite variety of problem types, and retrieval can only find the closest match. A strong LLM, in contrast, has seen orders of magnitude more mathematical reasoning during pretraining—it can generate exemplars that are not just closer matches but potentially in entirely different domains that are structurally analogous in ways a retrieval system based on sentence embeddings would miss. The paper's 61% vs. 57% advantage for generation over retrieval (text-davinci-003, Table 4) is modest but suggests this flexibility advantage is real.

A nuance worth noting: the paper's retrieval baseline uses Sentence-BERT embeddings for similarity, which capture semantic similarity but not structural or analogical similarity. A retrieval system specifically designed for analogical matching might close or reverse the gap with generation. The paper does not explore this, making the generation-vs-retrieval comparison specific to the retrieval method used. This doesn't invalidate the threshold model, but it does mean the exact threshold location is methodology-dependent.

Significance level: Practically significant as a decision framework; a useful empirical contribution that resolves a methodological question.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on three categories of reasoning tasks: (1) GSM8K (Cobbe et al., 2021), consisting of elementary math word problems; (2) MATH (Hendrycks et al., 2021b), consisting of advanced math problems from high school competitions covering seven subjects (algebra, counting/probability, geometry, etc.); and (3) BIG-Bench (Srivastava et al., 2022; Suzgun et al., 2022), specifically five reasoning tasks: word sorting, logical deduction (five objects), temporal sequences, reasoning about colored objects, and formal fallacies. For code generation, the authors collected 50 Codeforces Level-A problems published between January and August 2023, filtered to have problem descriptions within 2000 tokens for GPT-3.5-turbo (Appendix B). Each problem includes the full problem statement and test cases (both public and hidden). The Codeforces dataset is evaluated twice with results averaged due to its small size. The authors use the specific MATH split from Lightman et al. (2022) with 12,000 training and 500 test questions.

• Base model(s). Three model families are tested: GPT-3.5-turbo and GPT-4 (OpenAI, 2023; Ouyang et al., 2022), accessed between June and September 2023, and PaLM 2-L (Anil et al., 2023). For the scale analysis in Section 6.4, the paper additionally tests the OpenAI API models text-curie-001, text-davinci-001, text-davinci-002, and text-davinci-003, where "scale broadly indicates the amount of training data and parameter count used by the LLM" (Section 6.3). The choice to span multiple model families and scales is deliberate: it tests whether analogical prompting's benefits generalize beyond a single model architecture and whether they depend on model capability.

• Metrics. For mathematical problem solving (GSM8K, MATH), the paper reports accuracy—the fraction of problems for which the model's final answer matches the ground truth. Answers are extracted using the \boxed{} format. For code generation (Codeforces), the paper reports Acc@1 and Acc@10: Acc@k measures whether at least one of k sampled model outputs is correct (passes all test cases). This follows the standard code generation evaluation protocol from Li et al. (2022b) and Chen et al. (2023). For BIG-Bench tasks, the paper reports accuracy per task. For mathematical tasks and BIG-Bench, output is generated with temperature 0 (greedy decoding, one sample per problem). For Codeforces, temperature 0.7 is used with 10 samples per problem to compute Acc@k. The paper does not report confidence intervals or statistical significance tests for any results.

• Baselines. The paper compares against four prompting methods. (1) 0-shot: the raw problem statement with no additional instruction. (2) 0-shot CoT (Kojima et al., 2022): the problem statement augmented with a generic instruction like "think step by step." (3) Few-shot CoT (Wei et al., 2022b): the problem statement prepended with a fixed set of K labeled question-rationale-answer exemplars. For GSM8K and MATH, K = 5 exemplars are drawn from the respective training sets. For BIG-Bench and Codeforces, K = 3 manually annotated exemplars are used. These exemplars are fixed—the same set is used for all test problems. (4) Few-shot retrieved CoT (Zhang et al., 2022b; Shum et al., 2023): for each test problem, the K = 5 most similar labeled exemplars are dynamically retrieved from the training set using Sentence-BERT (Reimers & Gurevych, 2019) cosine similarity. This baseline is only applicable when a labeled training set exists (GSM8K, MATH). The paper also reports results for self-consistency (Wang et al., 2022) as a complementary technique applied on top of analogical prompting (Table 8), but this is not a primary baseline.

• Generation budget / compute accounting. The paper measures cost loosely in terms of number of generated tokens rather than FLOPs or wall-clock time. The key comparison is: analogical prompting self-generates exemplars (output tokens) while few-shot CoT prepends labeled exemplars (input tokens). The paper acknowledges this asymmetry in Section 8: "Compared to few-shot CoT, we use fewer input tokens and more output tokens, as exemplars are counted as input in few-shot CoT and as output in our approach." For Codeforces, the knowledge + exemplars variant generates substantially more tokens than baselines. No systematic FLOP-equivalence analysis is performed—the comparison is primarily accuracy-at-face-value, not accuracy-per-token or accuracy-per-dollar. For Codeforces Acc@10, 10 samples are generated per problem for all methods, making the sampling budget equal.

• Cross-validation / statistical protocol. The Codeforces evaluation is run twice with results averaged ("Because this dataset is relatively small, we conduct the evaluation twice and then report the average results," Appendix B). For other tasks, single-run evaluation is reported. The paper does not describe any cross-validation, bootstrap, or significance testing. The 50-problem Codeforces test set, 500-problem MATH test set, and standard GSM8K test set (1,319 problems) provide varying degrees of statistical reliability, but the paper treats all results as point estimates without uncertainty quantification.

Main Quantitative Results

Mathematical Problem Solving (GSM8K and MATH)

Headline results (Table 1). Analogical prompting with self-generated exemplars outperforms all baselines on both GSM8K and MATH across all three model families tested. The gains are most pronounced on MATH, where problem diversity makes fixed exemplars particularly poorly matched.

On GSM8K with GPT-3.5-turbo: analogical prompting achieves 77.8%, compared to 76.7% for 5-shot CoT (+1.1 points), 75.8% for 0-shot CoT (+2.0 points), and 75.0% for 0-shot (+2.8 points). The absolute gains over few-shot CoT are modest—1.1 percentage points—which is expected given that GSM8K problems are relatively homogeneous (elementary word problems), making fixed exemplars reasonably representative of the task distribution.

On GSM8K with text-davinci-003: analogical prompting achieves 61.0%, substantially outperforming 5-shot CoT at 54.0% (+7.0 points) and 0-shot CoT at 50.3% (+10.7 points). The paper notes that for text-davinci models, "we use an in-context demonstration of how to generate exemplars" (Table 1 footnote), meaning the prompt itself includes an example of exemplar generation—a meta-demonstration—which the other models do not require. This detail is important: the 61.0% result for text-davinci-003 is not comparable to the GPT-3.5-turbo and PaLM2 results in terms of prompt purity, since it receives additional in-context guidance on the format of self-generation.

On GSM8K with PaLM 2-L: analogical prompting achieves 81.7%, versus 80.7% for 5-shot CoT (+1.0 point), 78.2% for 0-shot CoT (+3.5 points), and 60.8% for 0-shot (+20.9 points). The gain over few-shot CoT is again modest, but the gain over 0-shot CoT is substantial—suggesting that PaLM 2-L benefits more from the structural guidance of exemplars (whether self-generated or manually provided) than from the general "think step by step" instruction.

On MATH with GPT-3.5-turbo: analogical prompting achieves 37.3%, compared to 34.9% for 5-shot CoT (+2.4 points), 33.9% for 0-shot CoT (+3.4 points), and 33.0% for 0-shot (+4.3 points). The larger absolute gain over few-shot CoT on MATH (+2.4 points) compared to GSM8K (+1.1 points) is consistent with the paper's core argument: MATH contains diverse problem types (algebra, geometry, probability, etc.), so fixed exemplars are frequently mismatched to the test problem's domain. Self-generated exemplars adapt to each problem's domain, yielding a larger advantage.

On MATH with PaLM 2-L: analogical prompting achieves 34.8%, versus 34.3% for 5-shot CoT (+0.5 points) and 29.8% for 0-shot CoT (+5.0 points). The near-identical performance to few-shot CoT on PaLM 2-L is notable—it suggests that for this model on this task, self-generated exemplars provide roughly equivalent quality to hand-labeled ones, but with zero labeling cost.

Interpretation of the MATH results. The paper claims that MATH involves "a range of reasoning types, including algebra, probability, and geometry," and that this "aligns with our approach of crafting tailored exemplars for each problem" (Section 6.1). The results support this: the gain over few-shot CoT is larger on MATH (+2.4 points) than on the more homogeneous GSM8K (+1.1 points with GPT-3.5-turbo). However, the paper does not break down MATH accuracy by sub-topic, so we cannot verify that the gain comes specifically from mismatched exemplars in fixed few-shot CoT rather than from some other mechanism.

Code Generation (Codeforces)

Headline results (Table 2). On the challenging Codeforces competitive programming task, analogical prompting with knowledge + exemplars achieves the strongest performance across both GPT-3.5-turbo-16k and GPT-4, with the knowledge component providing additional gains beyond exemplars alone.

On GPT-3.5-turbo-16k: knowledge + exemplars achieves 15% Acc@1 and 29% Acc@10, versus 13% Acc@1 and 25% Acc@10 for exemplars-only (+2 points Acc@1, +4 points Acc@10). The baselines are substantially lower: 3-shot CoT achieves 11% Acc@1 and 27% Acc@10; 0-shot CoT achieves 9% Acc@1 and 27% Acc@10; 0-shot achieves 8% Acc@1 and 24% Acc@10. The gain of knowledge + exemplars over 3-shot CoT is 4 points Acc@1 (15% vs. 11%, a 36% relative improvement), which is practically meaningful for competitive programming where even single-digit improvements are hard-won.

On GPT-4: knowledge + exemplars achieves 19% Acc@1 and 37% Acc@10, versus 17% Acc@1 and 32% Acc@10 for exemplars-only (+2 points Acc@1, +5 points Acc@10). Baselines: 3-shot CoT at 17% Acc@1 and 31% Acc@10; 0-shot CoT at 16% Acc@1 and 29% Acc@10; 0-shot at 16% Acc@1 and 30% Acc@10.

The interesting comparison across model tiers. With analogical prompting (knowledge + exemplars), GPT-3.5-turbo-16k achieves 15% Acc@1—only 1 point behind GPT-4's 0-shot performance of 16% Acc@1. The paper highlights this: "With our prompting method, GPT3.5-turbo achieves competitive performance with GPT4, with a 15% Acc@1 compared to GPT4's 16% Acc@1" (Section 6.1). This is a striking demonstration of test-time prompting partially closing the capability gap between model tiers—the smaller model with a smarter prompt nearly matches the larger model with a naive prompt.

The role of knowledge generation. The exemplars-only variant already outperforms all baselines on GPT-3.5-turbo (13% Acc@1 vs. 11% for 3-shot CoT), but adding knowledge provides a further 2-point boost (15% Acc@1). On GPT-4, the pattern holds: exemplars-only at 17% Acc@1 vs. knowledge + exemplars at 19% Acc@1. This confirms that the tutorial/algorithm identification step provides genuine value beyond what exemplars alone offer, particularly on Acc@10 (29% vs. 25% for GPT-3.5-turbo, a 4-point gap). The Acc@10 improvement suggests that knowledge generation helps the model produce more consistently correct solutions across multiple samples, not just improve the single best attempt.

BIG-Bench Reasoning Tasks

Headline results (Table 3). Across five diverse BIG-Bench reasoning tasks, analogical prompting with GPT-3.5-turbo outperforms 0-shot CoT on all tasks, with particularly large gains on word sorting (+7.6 points: 75.2% vs. 67.6%) and reasoning about colored objects (+6.4 points: 68.0% vs. 61.6%). It is competitive with 3-shot CoT, slightly outperforming it on word sorting (75.2% vs. 68.4%, +6.8 points), logical deduction (41.6% vs. 36.4%, +5.2 points), and formal fallacies (58.8% vs. 55.6%, +3.2 points), while slightly underperforming on temporal sequences (57.6% vs. 58.0%, -0.4 points) and essentially tying on reasoning about colored objects (68.0% vs. 62.0%—but this is +6.0, not a tie; I will re-read). Let me verify: Table 3 shows ours 68.0% vs. 3-shot 62.0% for colored objects, so ours is +6.0 better. For temporal sequences, ours is 57.6% vs. 3-shot 58.0%, a negligible 0.4-point gap.

The key pattern: analogical prompting consistently beats 0-shot CoT, often by substantial margins (word sorting +7.6, colored objects +6.4, logical deduction +6.4), and is generally competitive with or slightly better than 3-shot CoT that uses hand-labeled exemplars. The fact that self-generated exemplars can match manually crafted ones on these tasks is notable because BIG-Bench tasks are diverse and "may not have dedicated training data, so they align well with our approach of self-generating custom exemplars" (Section 5.1). Unlike GSM8K and MATH, there is no large training set of reasoning exemplars for these specific BIG-Bench tasks—making retrieval-based approaches inapplicable and making the zero-label nature of analogical prompting particularly valuable.

Scale Analysis: Self-Generation vs. Retrieval vs. Fixed Exemplars

Headline results (Table 4). This experiment on GSM8K using the OpenAI API model family (text-curie-001 through text-davinci-003) reveals a clear interaction between model scale and the optimal exemplar strategy.

For text-curie-001 (weakest model): all methods perform near chance (2–3% accuracy). Self-generation (2%) is no better than 0-shot CoT (2%) or 0-shot (2%). Fixed few-shot CoT (2%) and retrieved CoT (3%) also barely help. The model is simply too weak to solve GSM8K problems regardless of prompting.

For text-davinci-001: a gap emerges between 0-shot (6%) and few-shot methods. Fixed few-shot CoT (10%) and retrieved CoT (11%) substantially outperform 0-shot CoT (6%). Self-generated exemplars (9%) trail retrieved CoT (11%) by 2 points—the model is strong enough to benefit from exemplars, but not strong enough to generate useful ones reliably.

For text-davinci-002: the crossover occurs. Self-generated exemplars (48%) now slightly exceed fixed few-shot CoT (43%, +5 points) and retrieved CoT (47%, +1 point). 0-shot CoT trails at 22%.

For text-davinci-003 (strongest model in this family): self-generated exemplars (61%) clearly outperform fixed few-shot CoT (54%, +7 points) and retrieved CoT (57%, +4 points). This is the headline number for the scale analysis: at the high end of model capability, self-generation provides a 4–7 point advantage over retrieval from a labeled training set.

Why this matters. This establishes a capability threshold for analogical prompting. With weak models, the technique fails because generated exemplars are invalid or irrelevant—the model lacks the stored knowledge to produce useful problem-solution pairs. With strong models, self-generation surpasses external retrieval because the model's internal knowledge (from pretraining on a vast corpus) is richer and more flexible than any finite labeled training set. The paper states: "Our method outperforms the retrieved CoT with larger-scale LLMs... This is likely because the LLM has effectively learned related tasks during training and can generate useful exemplars. Conversely, with smaller-scale LLMs, the retrieved CoT performs better, and self-generation fails to produce useful or valid exemplars" (Section 6.3).

Combining Analogical Prompting with Self-Consistency

Headline results (Table 8). Self-consistency (Wang et al., 2022)—sampling multiple reasoning paths and taking a majority vote—is an orthogonal technique that can be applied on top of any prompting method. The paper demonstrates this combination briefly.

On GSM8K with GPT-3.5-turbo: analogical prompting alone achieves 77.8%; with self-consistency it reaches 85.3% (+7.5 points). On MATH: analogical prompting alone achieves 37.3%; with self-consistency it reaches 46.0% (+8.7 points). These results are reported without details on the number of samples used for self-consistency, making them difficult to interpret precisely. The paper frames this as evidence that analogical prompting "can complement and integrate with" existing CoT enhancements (Section 2.2), but the lack of experimental detail limits the informativeness of this result.

Ablation Studies and Robustness Checks

Number of self-generated exemplars (K): Table 5 ablates K on GSM8K and MATH with GPT-3.5-turbo. For GSM8K, accuracy rises from 76.1% (K=1) to 77.5% (K=3) to 77.8% (K=5). For MATH, the pattern is similar: 34.8% (K=1), 37.3% (K=3), 37.1% (K=5). The gains saturate at K=3, with minimal further improvement from more exemplars. K=1 underperforms due to "excessive reliance on a single exemplar" (Section 6.5). This result is consistent with standard few-shot in-context learning findings where 3–5 demonstrations are typically optimal, and it confirms that the self-generation mechanism follows similar scaling patterns to externally-provided exemplars.

Diversity constraint in exemplar generation: Table 6 compares prompt variants with and without explicit instructions to generate "diverse" and "distinct" exemplars on GSM8K and MATH (GPT-3.5-turbo). Without the diversity instruction, GSM8K accuracy drops from 77.8% to 75.9% (-1.9 points). MATH drops from 37.3% to 35.2% (-2.1 points). The paper explains: "some LLMs have a tendency to repetitively generate identical problems, which can be misleading when solving the target problem" (Section 4.1). This ablation confirms that the diversity instruction is a load-bearing component—without it, the model generates redundant exemplars that provide no additional information beyond the first one and may even bias the model toward an overly narrow solution approach.

Knowledge before vs. after exemplars (Codeforces): Table 7 compares ordering for the knowledge generation step on Codeforces with GPT-3.5-turbo. Generating knowledge before exemplars achieves 15% Acc@1 and 29% Acc@10. Generating knowledge after exemplars achieves 14% Acc@1 and 27% Acc@10. The 1-point Acc@1 difference is small but consistent (confirmed across two evaluation runs due to the Codeforces averaging protocol). The paper attributes this to a structural priming effect: generating knowledge first helps the model "identify the core concepts of the problem," which then guides it to "generate exemplars that align more closely in terms of the fundamental problem-solving approaches rather than surface-level lexical similarities" (Section 4.2). The authors give a concrete example: without knowledge-first, the model might generate exemplars about palindromic sequences (surface similarity to "finding a split point"); with knowledge-first, it generates exemplars about the prefix product algorithm (structural similarity).

Qualitative analysis of generated exemplar quality (Section 6.6): The paper manually analyzed 100 problems (50 correctly solved, 50 incorrectly solved) from GSM8K and MATH, categorizing the generated exemplars:

• For the 50 correctly solved problems: 35/50 (70%) had exemplars that were both relevant and correct; 9/50 (18%) had relevant but incorrect exemplars (the model generated a flawed solution in the exemplar but still solved the target correctly); 6/50 (12%) had irrelevant exemplars.
• For the 50 incorrectly solved problems: 28/50 (56%) had relevant and correct exemplars but the model still failed to solve the new problem. The breakdown of these failures: 12/50 were due to a "generalization gap" (the target problem was harder than the exemplars), 8/50 due to "overreliance on specific exemplars, leading to misdirection," and 8/50 due to "other issues, such as calculation errors." An additional 12/50 (24%) had relevant but incorrect exemplars, and 10/50 (20%) had irrelevant exemplars.

The most informative finding in this analysis is the failure mode: even when the model generates perfect exemplars, it still fails 28% of the time (28/100 total), and the dominant reason is a generalization gap—the target problem requires reasoning that goes beyond the patterns shown in the exemplars. This establishes a clear boundary condition: analogical prompting helps when the target problem is within the difficulty envelope of the self-generated exemplars, but breaks down when the target requires capabilities beyond what the exemplars demonstrate.

Self-consistency combination (Table 8): As discussed above, this is more a demonstration of composability than a true ablation, reported with minimal detail.

Critical Assessment

The experimental results provide solid support for analogical prompting's core value proposition—self-generated exemplars can match or exceed hand-labeled ones without labeling cost—but several aspects of the evaluation limit the strength and scope of the conclusions that can be drawn.

Claim: "Our approach outperforms 0-shot CoT and manual few-shot CoT in a variety of reasoning tasks." This claim is supported, but the margin over few-shot CoT is often small. On GSM8K with GPT-3.5-turbo, the gain is +1.1 points (77.8% vs. 76.7%); on MATH with PaLM 2-L, it is +0.5 points (34.8% vs. 34.3%). These differences are within what could reasonably be attributed to prompt engineering variance rather than a fundamental mechanism advantage. The paper does not report confidence intervals, standard deviations, or significance tests, so we cannot assess whether a 1.1-point difference on a 1,319-question test set is statistically meaningful. The gains are larger on MATH with GPT-3.5-turbo (+2.4 points) and on GSM8K with text-davinci-003 (+7.0 points), suggesting that the advantage depends on the interaction between model, task, and baseline exemplar quality—but the paper does not systematically characterize this interaction. The average +4% accuracy gain cited in the abstract comes from averaging across all tasks and models, which obscures the heterogeneity: on some configurations, the gain is near zero.

A fairer characterization of the evidence: analogical prompting reliably matches few-shot CoT (it never substantially underperforms it) while sometimes exceeding it by meaningful margins, particularly on diverse-domain tasks (MATH, BIG-Bench) and with stronger base models. This makes it a practical replacement for few-shot CoT—you lose no accuracy and eliminate labeling cost—rather than a universal accuracy improvement. The paper could have strengthened this claim by reporting per-subject MATH breakdowns to directly show that the gain comes from better exemplar matching on diverse problems, but such a breakdown is not provided.

Claim: "It can tailor the generated exemplars and knowledge to each problem, offering adaptability." This is qualitatively supported but not quantitatively isolated. The paper shows qualitative examples where generated exemplars match the problem domain (probability exemplars for probability problems, geometry exemplars for geometry problems), and the qualitative analysis in Section 6.6 shows that 70% of correctly solved problems had relevant and correct exemplars. However, there is no controlled experiment that measures how much of the performance gain comes from per-problem tailoring versus simply having any exemplars at all. A key missing experiment: compare analogical prompting to a baseline that receives the self-generated exemplars but from a different problem (i.e., mismatched exemplars), to isolate the value of tailoring. Without this, the "adaptability" claim is inferred rather than tested.

Claim: "This method presents several advantages: it obviates the need for labeling or retrieving exemplars, offering generality and convenience." This claim is strongly supported in principle but the paper does not quantify the cost savings. The self-generation approach indeed requires no labeled training data (unlike few-shot CoT) and no retrieval infrastructure (unlike retrieval-based CoT). This is a genuine practical advantage. However, the paper does not compare total computational cost: analogical prompting generates more output tokens (the exemplars), while few-shot CoT consumes more input tokens (the labeled exemplars). The tradeoff depends on API pricing (output tokens typically cost more than input tokens) and on task characteristics. A cost-normalized comparison (accuracy per dollar) would have been informative but is absent. Additionally, the prompt templates for analogical prompting are longer and more complex than 0-shot or few-shot CoT prompts—the prompts in Appendix D span dozens of lines with carefully structured instructions. This complexity is a form of "labeling cost" (prompt engineering effort) that the paper does not account for in its convenience claim. While designing one prompt template per task type is less work than labeling hundreds of exemplars, it is not zero effort, and the prompts are task-specific.

Weakness: Single model family for scale analysis. The scale analysis (Table 4) uses only the OpenAI API model family (text-curie-001 through text-davinci-003). These models are from a specific training paradigm and architecture. The threshold behavior (crossover from retrieval > generation to generation > retrieval at davinci-002) might be specific to this model family's scaling properties. The paper does not test whether PaLM 2 models or open-source models (LLaMA, Mistral) show similar thresholds. This limits the generalizability of the "use retrieval for small models, generation for large models" decision rule.

Weakness: Small and narrow code generation benchmark. The Codeforces dataset contains only 50 problems, all at Level-A (the easiest difficulty tier on Codeforces). With two evaluation runs, we're looking at 100 total evaluations. A 1–2 point Acc@1 difference on a 50-problem set is within sampling noise. The paper acknowledges the small size and averages over two runs, but does not report variance across runs. Additionally, testing only Level-A problems means we cannot assess whether the knowledge generation mechanism scales to harder problems that require more complex algorithmic reasoning (Level-B, C, D). It is possible that knowledge generation becomes more valuable on harder problems (where algorithm identification is more critical) or less valuable (where the model cannot reliably identify the correct algorithm from the problem description).

Weakness: No domain beyond math, code, and logic. All evaluated tasks fall into the broad categories of mathematical reasoning, algorithmic problem solving, and formal logic. The paper does not test analogical prompting on natural language reasoning tasks (reading comprehension, natural language inference, common-sense QA), creative tasks (story generation, argumentation), or knowledge-intensive tasks (factual QA). The analogical reasoning mechanism is theoretically domain-general, but the empirical evidence is confined to domains with clear right/wrong answers and structured reasoning paths. Whether the method works for less structured reasoning tasks is untested.

Missing experiment: "Mismatched exemplars" control. To isolate whether the benefit comes from per-problem tailoring or simply from having more exemplars in context, the paper should have compared: (a) analogical prompting with self-generated exemplars tailored to the problem, vs. (b) analogical prompting applied to a different problem (so the generated exemplars are for a different domain) but with the same number of exemplars provided in context. If (a) substantially outperforms (b), the tailoring claim is directly validated. This experiment is not run.

Missing experiment: Human evaluation of generated exemplar quality. The qualitative analysis in Section 6.6 is performed by the authors ("We manually analyzed...") with no inter-annotator agreement reported. The categorization into "relevant and correct," "relevant but incorrect," and "irrelevant" is subjective—especially for edge cases where an exemplar is partially relevant. A more rigorous evaluation with multiple annotators and agreement metrics would strengthen the claim that "generated exemplars were often relevant or correct" (Section 6.6).

Missing baseline: Prompt-length-matched 0-shot CoT. Analogical prompting generates more tokens than 0-shot CoT—the exemplars and knowledge add substantial content to the model's output. A fairer baseline would give 0-shot CoT the same total token budget (e.g., by allowing the model to "think" for more tokens) to control for the possibility that the benefit comes from additional computation rather than the analogical structure itself. This is partially addressed by the few-shot CoT comparison (which also provides exemplars, just externally), but for the 0-shot CoT comparison specifically, the compute budget is not matched.

6. Limitations and Trade-offs

The Capability Ceiling: Zero Improvement on Hardest Problems

The assumption or constraint. Analogical prompting rests entirely on the base LLM's ability to recall and generate useful exemplars for a given problem. If the model has not internalized relevant problem-solving patterns during pretraining—either because the domain is too difficult or too novel—self-generation produces nothing useful. The paper is refreshingly transparent about this boundary, though it does not frame it as a formal limitation in Section 8. Instead, the evidence appears in the results.

The consequence. On problems fundamentally outside the model's capability range, analogical prompting provides essentially no benefit, and may even distract the model with irrelevant generated content. This is not a "diminishing returns" scenario—it is a hard ceiling. No amount of additional exemplars, knowledge generation, or self-consistency post-processing can compensate for the base model's inability to produce correct solutions. This matters enormously for deployment: if your problem distribution contains a non-trivial fraction of genuinely hard problems, analogical prompting will not help on that subset, and you are effectively paying the generation cost of the exemplars for zero accuracy gain.

What evidence exists in the paper. The evidence is indirect but conclusive. The qualitative analysis (Section 6.6) finds that among 50 incorrectly solved problems, 28 (56%) had generated exemplars that were relevant and correct—yet the model still failed. Of these 28, 12 (43%) failed due to "a generalization gap between the exemplars and the new problem" where "the new problem is harder than the exemplars." This is the paper's own diagnosis: the model can recall related problems, but cannot bridge the difficulty gap to solve a harder variant. The paper also notes that "8/50" failures were due to "overreliance on specific exemplars, leading to misdirection"—a case where self-generated content actively harms performance by anchoring the model to an inappropriate solution template.

Further supporting the ceiling hypothesis: the paper's MATH results (Table 1) show that the absolute accuracy improvement from analogical prompting over few-shot CoT is +2.4 points (37.3% vs. 34.9% with GPT-3.5-turbo)—meaningful but far from solving the remaining ~63% of problems. The hardest MATH problems (competition-level questions requiring non-trivial theorem application or multi-step proofs) are unlikely to benefit from self-generated exemplars because the model cannot produce correct exemplars at that difficulty tier in the first place.

Mitigation status. The paper acknowledges this only obliquely in Section 8: "self-generation can fail if the LLM lacks sufficient strength or has not learned relevant knowledge to the new problems to solve. Conversely, with a stronger LLM, it can draw upon relevant prior knowledge to tackle slightly more complex problems." The phrasing "slightly more complex" is telling—it admits that the method extends capability only marginally beyond what the model already knows. No solution is proposed other than using stronger base models. This is a fundamental, not incidental, limitation: analogical prompting amplifies existing knowledge but does not create new capability. For problems where the model's pass@1 is near zero, the method is ineffective and may be counterproductive.

The Hidden Cost of Difficulty Estimation: Tokens Spent on Useless Exemplars

The assumption or constraint. Analogical prompting always generates K exemplars (and optionally knowledge) for every problem, regardless of whether those exemplars will actually help solve it. The generation is unconditional—there is no gating mechanism that decides "this problem is too hard, skip the exemplars and just try a direct solution," or "this problem is trivially easy, one exemplar is enough."

The consequence. On problems where self-generation cannot help (the hardest tier, as discussed above, or trivially easy problems where the model already knows the answer), the method wastes inference compute on generating exemplars that provide zero or negative value. This is a direct efficiency loss: the model produces hundreds of extra output tokens that do not improve accuracy. In cost-sensitive deployments (API-based systems where users pay per token), this overhead is paid on every query regardless of benefit. The paper does not measure how often self-generated exemplars are irrelevant or harmful, making it impossible to estimate the waste rate. But the qualitative analysis provides a lower bound: across 100 problems, 16/100 (16%) had exemplars that were "irrelevant" (Section 6.6). On those 16 problems, the model paid the generation cost for no benefit.

This becomes more acute when analogical prompting is combined with self-consistency (Table 8). Self-consistency requires sampling multiple independent reasoning paths. If analogical prompting is applied to each sample, the cost multiplies: each of N samples generates its own set of exemplars and knowledge, even though the exemplars across samples are likely substantially overlapping or redundant. The paper demonstrates this combination (+7.5 points on GSM8K, +8.7 on MATH) but provides no information about the number of samples used or the total token cost, making it impossible to assess whether the accuracy gain justifies the multiplicative compute increase.

What evidence exists in the paper. The qualitative analysis (Section 6.6) shows that 12% of correctly solved problems had irrelevant exemplars (the model solved correctly despite useless self-generation) and 20% of incorrectly solved problems had irrelevant exemplars. This means that on at least 16% of all problems, exemplar generation was pure overhead. The paper provides no token-count analysis comparing analogical prompting to baselines—a significant gap given that the method's primary selling point is convenience and label-elimination, not compute efficiency. The Section 8 acknowledgment is partial: "One limitation of our approach is increased inference computation, as our approach generates more tokens than vanilla 0-shot and 0-shot CoT prompting." It compares input/output token tradeoffs with few-shot CoT but does not quantify the overhead or discuss its implications for deployment costs.

Mitigation status. Not addressed. The paper does not propose any adaptive mechanism to skip or reduce exemplar generation on problems where it is unlikely to help. A natural extension—using a lightweight classifier or the model's own confidence to decide whether to generate exemplars—is not explored. The limitation is simply acknowledged in Section 8 without remediation.

Prompt Engineering Complexity Replaces Labeling Cost

The assumption or constraint. The paper positions analogical prompting as eliminating the need for "manually labeling reasoning exemplars for each task" (Section 1) and as a method that is "more self-contained and convenient, as it does not rely on external labeled data or retrieval steps" (Section 6.3). The implicit claim is that the method reduces overall human effort compared to few-shot CoT.

The consequence. This framing overlooks a different form of human effort: the prompt templates for analogical prompting are substantially more complex than standard few-shot CoT prompts, and they require per-task design and tuning. The MATH prompt (Appendix D.1) spans 15+ lines with nested instructions, explicit formatting requirements (\boxed{}, Q:/A: labeling, "Copy and paste the initial problem here"), and domain-specific directives. The Codeforces prompt (Appendix D.3) is longer still, with separate sections for Algorithms, Tutorial, Example Problems, and solution generation, plus explicit Python3 code formatting instructions. Designing these prompts requires understanding both the task structure and the model's behavior—this is expert labor, not zero-cost automation.

Furthermore, the method introduces new hyperparameters that must be tuned per task: the number of exemplars K (3 vs. 5, with different optimal values for GSM8K vs. MATH per Table 5), whether to include knowledge generation (useful for Codeforces, less so for GSM8K per Section 6.2), the ordering of knowledge and exemplars (knowledge-first performs better per Table 7), and the diversity constraint phrasing (without it, performance drops per Table 6). Each of these choices requires ablation experiments on a validation set. For a practitioner deploying analogical prompting on a new task, they must either replicate this tuning process (which requires labeled validation data—partially defeating the "no labels" claim) or guess the hyperparameters and accept potentially suboptimal performance.

The paper does not report how much prompt engineering effort went into arriving at the final templates. The prompts in Appendix D are presented as finished artifacts, not as the result of an iterative design process, but the specific design choices (diversity constraint, section ordering, \boxed{} formatting, knowledge-first ordering) strongly suggest substantial iteration.

What evidence exists in the paper. The paper itself demonstrates the sensitivity of the method to prompt design. Table 6 shows a 2.1-point MATH accuracy drop when the diversity instruction is removed. Table 7 shows a 1-point Acc@1 difference on Codeforces depending on whether knowledge is generated before or after exemplars. Section 6.5 shows that K must be tuned (K=1 underperforms significantly). These are all prompt engineering choices that a practitioner would need to discover and validate for each new task. The paper acknowledges prompt sensitivity briefly in Section 8: "it is known that LLM performance can be influenced by specific prompt phrases used to query the model (Jiang et al., 2020), and our work is also subject to this prompt sensitivity." But this is framed as a general property of LLMs, not as a specific limitation of the analogical prompting approach—which, because it involves multi-stage self-generation with interdependent sections, may be more sensitive to prompt design than simpler methods.

Mitigation status. Not addressed. The paper provides the prompt templates for reproduction, which reduces the burden for the specific tasks studied, but offers no methodology for designing prompts for new tasks, no automated prompt optimization, and no analysis of how robust the method is to minor prompt variations. The "convenience" claim should be qualified: analogical prompting eliminates exemplar labeling but introduces prompt engineering as a substitute cost.

Single-Model-Family Scale Analysis Limits the Threshold Claim

The assumption or constraint. The paper's key finding about scale dependence—that self-generation outperforms retrieval only above a certain model capability threshold—is based on a single model family (OpenAI API: text-curie-001 through text-davinci-003) evaluated on a single task (GSM8K). The paper generalizes this to a claim: "Our method outperforms the retrieved CoT with larger-scale LLMs... Conversely, with smaller-scale LLMs, the retrieved CoT performs better" (Section 6.3). The implicit assumption is that this threshold behavior is universal across model families and tasks.

The consequence. We cannot know whether the threshold observed in Table 4 (crossover at text-davinci-002) generalizes to other model families (PaLM, LLaMA, Claude, Mistral) or other tasks (MATH, Codeforces, BIG-Bench). Different model families have different pretraining data mixtures, different architectural properties, and different scaling trajectories. A model might be strong enough to self-generate useful GSM8K exemplars but not MATH exemplars, or vice versa. The "use retrieval for small models, generation for large models" decision rule, while a useful heuristic, may not transfer.

The paper does test analogical prompting on PaLM 2-L (Tables 1) and GPT-4 (Table 2), showing that it works on those strong models. But it does not test the failure mode on weaker PaLM models or on weaker open-source models. Nor does it test retrieval-based CoT on PaLM 2 or GPT-4 to confirm that the crossover pattern replicates. The scale analysis (Table 4) is isolated to the OpenAI API family on GSM8K only.

This matters for practitioners using non-OpenAI models or newer model families. A team deploying LLaMA-3-70B or Claude-3 cannot confidently apply the threshold rule from Table 4—they would need to replicate the generation-vs-retrieval comparison for their specific model and task.

What evidence exists in the paper. Table 4 is the only experiment comparing generation and retrieval across scales, and it is restricted to 4 OpenAI models on 1 task. The broader main results (Tables 1–3) show that analogical prompting works on GPT-3.5-turbo, GPT-4, and PaLM 2-L—but these are all large, capable models near the "strong" end of the spectrum. There is no evidence of the failure mode on weaker models outside the OpenAI family. The paper's claim that "our prompting method surpasses vanilla 0-shot and 0-shot CoT across all scales" (Section 6.4) is supported by Table 4 (it does surpass 0-shot and 0-shot CoT at all scales), but the more specific claim about generation vs. retrieval crossover is supported by a single model family.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation—the scale analysis is presented as definitive. Extending the comparison to other model families (particularly open-source models where pretraining data and architecture are known) would substantially strengthen the generalizability of the threshold claim.

No Evidence on Latency or Serial Dependency Costs

The assumption or constraint. The paper measures cost in terms of "generations" and token counts, not wall-clock time or latency. The single-pass design (generate exemplars, then generate solution in the same forward pass) is presented as a convenience advantage over multi-step approaches (Section 4.1). But the single-pass design has a latency implication that the paper does not discuss: the model must generate all exemplars (and optionally knowledge) before it can begin generating the solution. These are serial dependencies—the solution cannot start until the exemplar generation is complete.

The consequence. For a user waiting for an answer, analogical prompting imposes a latency penalty proportional to the length of the self-generated exemplars. If the model generates 3 exemplars averaging 100 tokens each, the user waits an extra ~300 tokens of generation time before the solution even begins. This is fundamentally different from few-shot CoT, where the exemplars are pre-computed (they are input tokens, processed in parallel during the prefilling phase) and the model can begin generating the solution immediately after processing the prompt. The paper acknowledges the input/output token asymmetry but frames it as a cost issue, not a latency issue: "Compared to few-shot CoT, we use fewer input tokens and more output tokens, as exemplars are counted as input in few-shot CoT and as output in our approach" (Section 8).

In latency-sensitive applications (interactive assistants, real-time code completion, live tutoring), the serial generation of exemplars may be unacceptable regardless of accuracy gains. A user asking a math question expects an answer in seconds, not tens of seconds. The method's latency cost depends on the length of generated exemplars, which varies by task and problem complexity—the paper provides no statistics on exemplar lengths.

This limitation also interacts with the self-consistency combination (Table 8). Self-consistency requires generating N independent solution paths. If each path uses analogical prompting and generates its own exemplars, the latency multiplies by N (for sequential sampling) or the throughput cost multiplies by N (for parallel sampling). Neither cost is quantified.

What evidence exists in the paper. The paper provides no latency measurements, no token-length statistics for generated exemplars, and no discussion of the latency vs. accuracy tradeoff. The prompt templates are shown in Appendix D, and from these we can estimate that generated exemplars for MATH are roughly 100–200 tokens each, meaning 3 exemplars add 300–600 output tokens before the solution begins. For GPT-3.5-turbo with typical generation speeds of ~50–100 tokens/second, this translates to an additional 3–12 seconds of latency. This is a rough estimate, not a measurement, and the paper provides no basis for refining it.

Mitigation status. Not addressed at all. The paper's discussion of limitations (Section 8) focuses on "increased inference computation" in terms of token counts and FLOPs, not latency. For a prompting method, latency is a first-class practical concern that should be measured and reported alongside accuracy.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a reframing of the LLM prompting paradigm that shifts the field's understanding of what prompting accomplishes. Prior to this work, the dominant mental model treated prompting as instruction-delivery: you tell the model what to do (0-shot CoT: "think step by step"), or you show it examples of what to do (few-shot CoT: here are three worked problems, now imitate them). The model is a passive recipient of externally provided reasoning guidance. Analogical prompting inverts this relationship by treating the model as an active knowledge repository that can be prompted to surface and repurpose its own stored problem-solving experiences. The prompt's job is not to provide reasoning strategies but to trigger recall of relevant strategies the model already possesses.

This reframing matters because it changes what practitioners believe about LLM capabilities—and therefore what they attempt to build. If you believe the model needs external exemplars to reason well, you invest in curating labeled datasets, building retrieval pipelines, and maintaining exemplar banks. If you believe the model can self-generate useful guidance, you invest in prompt engineering that elicits this self-generation, and you deploy on new tasks immediately without per-task data preparation. The paper provides the first systematic evidence that, for sufficiently strong models, the self-generation approach not only works but matches or exceeds the performance of externally provided exemplars (Tables 1–4)—while eliminating the labeling and retrieval costs entirely.

The work also resolves a latent tension in the prompting literature that the paper itself does not explicitly frame as a conflict, but which becomes clear in retrospect. The two dominant CoT paradigms—0-shot (cheap, generic, weak) and few-shot (expensive, specific, strong)—have been treated as an unavoidable tradeoff: you pay for specificity with manual labor. Retrieval-based CoT attempted to automate the specificity side but introduced infrastructure complexity and remained bounded by training set coverage. Analogical prompting demonstrates that this tradeoff is not fundamental—it is an artifact of treating exemplars as something that must come from outside the model. When the model is strong enough, specificity can be achieved through self-generation at zero labeling cost, with no retrieval infrastructure, and with the potential for exemplars that are more diverse and better tailored than what any finite training set contains (Table 4: 61% for self-generation vs. 57% for retrieval on text-davinci-003).

The paper's most significant conceptual contribution is the capability threshold model implicit in Table 4: self-generation works only above a certain model strength, below which external retrieval remains superior. This provides a unified framework for understanding when different prompting strategies are appropriate—a framework that scales with model improvement. As base models become more capable (GPT-3.5 → GPT-4, PaLM 2 → future models), the threshold shifts downward, making self-contained methods viable for an expanding range of tasks and deployment scenarios. This is not an incremental improvement to CoT; it changes what we believe is happening inside the model during prompting—from instruction-following to knowledge-activation—and establishes a new design principle: when the model already knows something, the best prompt is the one that helps it remember.

Less dramatically but still significantly, the paper provides the first clear evidence that self-generated analogies can serve as a functional substitute for human-labeled reasoning exemplars across diverse reasoning tasks. This is not obvious a priori—one might expect that LLM-generated exemplars would contain errors, be irrelevant, or fail to provide the structural guidance that hand-crafted exemplars offer. The paper's qualitative analysis (Section 6.6) partially validates this concern: 18% of correctly solved problems used exemplars that were relevant but incorrect, meaning the model generated flawed examples and still solved the target problem correctly. This suggests that even imperfect self-generation provides useful structural priming, a finding that has implications for how we think about the robustness of in-context learning.

The paper also makes retrieval-based CoT less attractive as a default strategy for state-of-the-art models. Prior to this work, retrieval was the natural answer to "how do we get relevant exemplars without manual labeling?" The paper shows that for strong models, self-generation is both simpler (no retrieval infrastructure) and more effective (+4–7 points on GSM8K with text-davinci-003). This doesn't eliminate the case for retrieval—it remains superior for weaker models, and for domains where the model lacks pretraining exposure—but it shifts the burden of proof: for any new task with a strong base model, self-generation should be the first approach tried, with retrieval reserved for cases where generation quality is demonstrably inadequate.

Follow-Up Research This Work Enables

Adaptive exemplar budgeting—skip generation when it won't help. The paper's qualitative analysis (Section 6.6) reveals that 16% of all problems had irrelevant generated exemplars, and 28% of incorrectly solved problems had correct exemplars but the model still failed due to a generalization gap. This means the method wastes tokens on exemplar generation for a non-trivial fraction of problems. A natural follow-up is to train a lightweight classifier—perhaps a small model or a linear probe on top of the base LLM's embeddings—that predicts, from the problem text alone, whether self-generated exemplars are likely to help. The classifier could use signals from the paper: problem domain (analogical prompting helps more on diverse-domain tasks like MATH), estimated difficulty (the method fails on problems harder than the model's exemplar-generation capability), and the base model's confidence. The experiment: compare analogical prompting with and without this adaptive gating on MATH and Codeforces, measuring both accuracy and total token cost. The hypothesis is that adaptive gating recovers most of the accuracy gain (by generating exemplars on the ~70% of problems where they help) while substantially reducing wasted computation on the remaining ~30%.

Cross-model-family threshold characterization. Table 4's finding—self-generation overtakes retrieval at text-davinci-002-level capability—is based on a single model family (OpenAI API) on a single task (GSM8K). We do not know whether this threshold generalizes across architectures (dense vs. mixture-of-experts), training paradigms (RLHF vs. base models), or model families (PaLM, LLaMA, Claude, Mistral). A systematic replication would test analogical prompting vs. retrieval-based CoT on GSM8K and MATH across a range of model sizes within at least three families: LLaMA-3 (8B, 70B), Mistral (7B, 8×7B MoE), and the GPT family (if API access permits). The key measurement: for each family, identify the smallest model size where self-generation matches or exceeds retrieval, and determine whether this threshold—expressed in terms of estimated pretraining compute or benchmark performance—is consistent across families. A negative result (threshold varies wildly by family) would suggest the crossover depends on pretraining data composition or architectural inductive biases, not just raw capability. A positive result (consistent threshold) would establish a robust engineering guideline for prompting strategy selection.

Generalization gap diagnosis—why do correct exemplars fail to help? The paper identifies that 28% of incorrectly solved problems had generated exemplars that were both relevant and correct, yet the model still failed (Section 6.6). The paper attributes 12 of these 28 failures to a "generalization gap" where the target problem is harder than the exemplars. This observation is currently post-hoc and qualitative. A rigorous follow-up would construct a controlled experiment: for a set of MATH problems, pair each test problem with self-generated exemplars that vary systematically in difficulty (measured by the base model's pass@1 on the exemplar problems) and in structural similarity (measured by embedding distance or sub-topic match). Then measure how solution accuracy varies as a function of exemplar difficulty relative to target difficulty. The concrete hypothesis: accuracy drops sharply when the target problem's difficulty exceeds the maximum difficulty of the generated exemplars by some threshold. This would operationalize the "generalization gap" concept and provide predictive guidance: if the model cannot generate exemplars at or above the target's difficulty level, analogical prompting is unlikely to help, and resources should be spent elsewhere (e.g., more parallel samples, a stronger base model).

Structured retrieval for analogical matching vs. self-generation. The paper's retrieval baseline uses Sentence-BERT cosine similarity (Section 5.3), which captures semantic similarity but not structural or analogical similarity. Two problems about "finding a split point" might have high semantic similarity while being algorithmically unrelated (one requires prefix products, the other requires palindrome checking). The paper's own qualitative analysis suggests this is a real issue: without knowledge-first generation, the model produces exemplars based on surface-level lexical matches (Section 6.2). A direct follow-up would build a retrieval system that uses structural matching—for code problems, matching based on the required algorithm (extracted via LLM-generated tags or static analysis); for math problems, matching based on the solution template or equation types. The experiment: compare this structured retrieval baseline against self-generation on Codeforces and MATH, testing the paper's implicit claim that self-generation's advantage comes from better structural alignment that retrieval misses. If structured retrieval closes the gap or surpasses self-generation, it would refine our understanding of why self-generation works—not because generation is inherently superior, but because current retrieval methods use the wrong similarity metric.

Self-generated exemplar ensembling for robustness. The qualitative analysis reveals that generated exemplars are sometimes incorrect (18% of correctly solved problems used incorrect exemplars; Section 6.6). Yet the model often still solves the target correctly, suggesting some robustness to exemplar noise. A natural extension is to generate multiple independent exemplar sets (by varying the diversity prompt, the temperature, or the number of exemplars K) and ensemble the resulting solutions. This is distinct from self-consistency (which ensembles solutions from the same prompt): exemplar ensembling would test whether diversity in the generated context leads to diversity in the solution strategy, potentially surfacing different valid approaches to the same problem. The experiment: on MATH and Codeforces, generate 3 independent exemplar sets per problem (using different random seeds or prompt variations), produce a solution from each set, and apply majority voting or verifier-based selection. Compare accuracy against single-set analogical prompting and against standard self-consistency. The hypothesis is that exemplar ensembling provides orthogonal benefits to solution-path ensembling—the former varies the reasoning context, the latter varies the reasoning execution.

Zero-shot analogical prompting via instruction tuning. The paper's analogical prompting requires hand-designed prompt templates with task-specific instructions (diversity constraints, K exemplars, section ordering). This prompt engineering is itself a form of labeling cost. A follow-up would eliminate this by instruction-tuning a model to perform analogical reasoning without explicit per-task templates. The training data: for a diverse set of reasoning tasks (GSM8K, MATH, Codeforces, BIG-Bench tasks, plus additional synthetic or web-scraped problem sets), use a strong model (GPT-4) to generate (problem, self-generated-exemplars, solution) triplets using analogical prompting. Then fine-tune a smaller model on these triplets with a meta-instruction like "Before solving, recall and solve related problems that will help you." The experiment: test whether this fine-tuned model, given only a problem statement and the meta-instruction, spontaneously generates relevant exemplars and solves problems at accuracy levels approaching the prompted strong model. This would convert analogical prompting from a prompt-engineering technique into a model capability—a more robust and deployment-friendly form of the method. The paper's finding that ReST^EM-based optimization degraded revision model performance (referenced in the prior sections' discussion of the examined paper) provides a cautionary parallel: on-policy training can amplify spurious patterns, so careful data filtering and curriculum design would be essential.

Practical Applications and Downstream Use Cases

Zero-label deployment on niche reasoning tasks. For organizations deploying LLMs on specialized reasoning tasks where labeled exemplars don't exist—legal document analysis, medical case reasoning, engineering problem-solving, financial modeling—analogical prompting eliminates the need to hire domain experts to create reasoning exemplars. The method works out-of-the-box: provide the problem and the prompt template, and the model self-generates guidance. The paper's BIG-Bench results (Table 3) directly demonstrate this value: on tasks like formal fallacies and logical deduction, which "may not have dedicated training data" (Section 5.1), analogical prompting matches or exceeds 3-shot CoT with manually annotated exemplars. For a legal tech company deploying an LLM to analyze contract clauses, the alternative to analogical prompting is either (a) hiring lawyers to write dozens of annotated reasoning exemplars per clause type, or (b) accepting the weaker performance of 0-shot CoT. The paper's evidence suggests analogical prompting provides few-shot-level accuracy at 0-shot labeling cost—a compelling value proposition for long-tail specialized domains. The key deployment consideration: the base model must be strong enough in the target domain to generate useful exemplars. The paper's threshold analysis (Table 4) suggests GPT-3.5-turbo-level models are sufficient for well-represented domains, but for highly specialized domains, model capability must be verified first.

Cost-efficient self-improvement data generation. When using LLMs to generate training data for fine-tuning smaller models (a common distillation or self-improvement pipeline), the quality of generated solutions directly determines downstream model quality. Analogical prompting offers a way to improve generation quality without increasing human annotation cost: for each training problem, use analogical prompting (rather than 0-shot or few-shot CoT) to generate the solution, producing higher-accuracy training targets. The paper's MATH results are the relevant evidence: analogical prompting achieves 37.3% vs. 34.9% for 5-shot CoT on GPT-3.5-turbo (Table 1)—a +2.4 point improvement on the solutions that would become training data. For a self-improvement loop generating 100,000 training solutions, this translates to 2,400 additional correct solutions in the training set, which can meaningfully improve the fine-tuned model's performance. The cost tradeoff depends on the relative expense of generation vs. training: analogical prompting generates more output tokens per problem (the exemplars), but if the improved solution quality reduces the number of required training iterations or improves final model accuracy, the net cost may be favorable. The paper's finding that self-consistency further boosts analogical prompting (Table 8: 85.3% on GSM8K, 46.0% on MATH) suggests a high-quality—but expensive—data generation pipeline: for each training problem, generate N solutions with analogical prompting and select the majority-vote answer as the training target.

Prompting-as-a-service for model-tier gap closure. The paper's Codeforces result—GPT-3.5-turbo with analogical prompting (15% Acc@1) nearly matching GPT-4 with 0-shot prompting (16% Acc@1)—has direct commercial implications for API-based LLM services. Organizations using GPT-4 for complex reasoning tasks (at approximately $30–60 per million output tokens) could potentially switch to GPT-3.5-turbo (at approximately$0.50–1.50 per million output tokens) with analogical prompting and achieve comparable accuracy at a fraction of the cost—a 20–60× cost reduction per query by trading a slightly higher token count (exemplar generation) for a dramatically lower per-token price. The paper's 15% vs. 16% Acc@1 comparison is on a 50-problem Codeforces set, so the evidence is preliminary—this use case requires validation on larger benchmarks and on the specific task distribution of interest. But the pattern is consistent with the paper's broader finding that prompting improvements can partially substitute for model scale, and it provides a concrete optimization target for cost-conscious deployments: identify tasks where your expensive model's accuracy is matched by a cheaper model with smarter prompting, and route those queries accordingly.

When to Prefer This Method

The paper articulates an explicit tradeoff between self-generation and retrieval-based exemplar provision (Section 6.3, Table 4), establishing the following decision rule:

• Prefer analogical prompting (self-generated exemplars) when:

  • The base LLM is sufficiently strong to generate useful exemplars—operationalized as text-davinci-002-level capability or above (Table 4: self-generation overtakes retrieval at this scale on GSM8K). For current models, this includes GPT-3.5-turbo, GPT-4, PaLM 2-L, and comparably capable models.
  • No labeled training set of reasoning exemplars exists for the task, making few-shot CoT and retrieval-based CoT inapplicable (the paper's BIG-Bench scenario; Section 5.1).
  • The task involves diverse problem types where fixed exemplars are frequently mismatched—MATH (spanning 7 subjects) benefits more (+2.4 points over few-shot CoT) than the more homogeneous GSM8K (+1.1 points; Table 1).
  • Simplicity of deployment is valued over optimal accuracy: self-generation requires no retrieval infrastructure, no labeled data, and no per-task exemplar curation.

• Prefer retrieval-based CoT when:

  • The base LLM is relatively weak (text-davinci-001-level or below; Table 4), and self-generated exemplars are likely to be invalid or irrelevant—the model fails to produce useful problem-solution pairs.
  • A high-quality labeled training set exists with good coverage of the test distribution (e.g., GSM8K's 7,500 training problems with reasoning steps), and the retrieval infrastructure cost is acceptable.
  • Reliability of exemplar correctness is paramount: retrieved exemplars from a labeled dataset are guaranteed correct, while self-generated exemplars can contain errors (18% of correctly solved problems in the paper's analysis used incorrect exemplars; Section 6.6).

• Add self-generated knowledge (tutorials) when:

  • The task requires identifying and applying specific named algorithms or techniques (Codeforces: prefix product, dynamic programming), rather than general mathematical reasoning. The knowledge component improves Acc@1 by 2 points on Codeforces (Table 2) but provides negligible gains on GSM8K and MATH (Section 6.2).
  • Always generate knowledge before exemplars: this ordering yields +1 point Acc@1 on Codeforces (Table 7) by priming the model to generate structurally aligned rather than surface-level-similar exemplars.