Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks

ArXiv: 2211.12588

🎯 Pitch

Chain-of-thought makes language models do their own arithmetic, but LLMs are terrible calculators—errors compound across steps and they can’t solve equations at all. This paper shows that simply having the model write a Python program instead of natural-language math and handing computation to an interpreter boosts accuracy by roughly 12% across math and finance benchmarks, with the biggest gains on complex algebra and iteration problems.

1. Executive Summary

This paper introduces Program of Thoughts (PoT) prompting, a method that disentangles numerical reasoning from computation by having language models generate Python programs—rather than natural language arithmetic steps—as intermediate reasoning, then delegating actual calculation to an external interpreter (e.g., solving cubic equations via SymPy or expressing iteration as a for-loop). Evaluated on five math word problem datasets (GSM8K, AQuA, SVAMP, TabMWP, MultiArith) and three financial-QA datasets (FinQA, ConvFinQA, TATQA) using Codex (code-davinci-002), PoT achieves an average few-shot gain of roughly 12% over Chain-of-Thought prompting, with the largest improvements on problems requiring complex symbolic manipulation—linear/polynomial equations, iteration, and combinatorics—while performing comparably to CoT on simpler arithmetic and probability questions, establishing that an LLM’s reasoning and its numerical computation can be productively separated only when the reasoning itself can be faithfully expressed as executable code.

2. Context and Motivation

The Core Problem: LLMs Are Bad at Arithmetic, and CoT Makes It Worse

This paper addresses a fundamental mismatch in how language models are asked to solve numerical reasoning problems. When prompted with Chain-of-Thought (CoT), an LLM is expected to not only work out what steps to take—the reasoning—but also to actually execute the arithmetic in those steps—the computation. For example, CoT might generate "First, multiply 147 by 38 = 5586. Then divide 5586 by 3 = 1862." The model must produce both the reasoning and the correct numerical result.

The central claim of this paper is that this conflation is a design mistake. The authors identify three specific failure modes that arise from asking an LLM to do its own arithmetic (Section 1 and throughout):

1. Arithmetic calculation errors. Language models are probabilistic next-token predictors, not deterministic calculators. When dealing with large numbers (e.g., millions in financial datasets like FinQA) or multi-digit operations, they are highly prone to mistakes. The paper gives a concrete illustration: if each addition in a 50-step iterative calculation has a 90% chance of being correct, the probability of getting the final answer right is less than 1% — compounding error across steps makes long chains of arithmetic unreliable.

2. Inability to solve complex mathematical expressions. LLMs cannot reliably solve polynomial equations, systems of equations, or other symbolic manipulation tasks purely through text generation. The paper's Figure 1 (lower example) shows CoT failing on a cubic equation that requires solving for an interest rate — the model produces a wrong answer because it cannot perform the algebraic manipulation needed.

3. Inefficiency at expressing iteration. When a problem requires repeating an operation many times (e.g., "every second glass costs 60% of the price" for 16 glasses), CoT must unroll the loop into a long sequence of repeated reasoning steps, which is verbose, error-prone, and consumes context window space. As the number of iterations grows, accuracy degrades rapidly.

These are not minor edge cases. They systematically affect any problem requiring precise calculation — math word problems, financial analysis, engineering computations — which are precisely the domains where accuracy matters most.

Why This Problem Matters: Beyond Benchmark Scores

The practical significance extends well beyond improving accuracy on MATH or GSM8K. The paper's framing implies a broader principle about how AI systems should be architected: language models should handle language, and specialized tools should handle computation. This division of labor matters for several reasons:

Financial applications demand precision. The paper evaluates on FinQA and ConvFinQA, which involve real financial statements with numbers in the millions. A 1% calculation error on a financial analysis task is unacceptable — yet that is exactly the kind of error LLMs produce when trusted with arithmetic. PoT's external interpreter eliminates this class of error entirely.

Complex real-world math requires symbolic manipulation. Many problems in science, engineering, and finance involve solving equations that cannot be reduced to simple arithmetic. CoT has no mechanism for this. PoT, by generating Python code that can call SymPy's solve function, can handle polynomial equations, systems of equations, and other symbolic math that would be impossible for a text-only approach.

Iterative or conditional logic is natural in code, awkward in text. Problems involving "for each item," cumulative sums over variable-length lists, or conditional branching are expressed naturally in a few lines of Python but require dozens of lines of fragile text in CoT. This is not just an efficiency concern — the longer the CoT reasoning chain, the more opportunities for hallucination or reasoning drift.

Theoretical significance. The paper challenges the implicit assumption behind CoT — that reasoning and computation are the same activity and should be performed by the same model. By showing that separating them yields consistent, large improvements, PoT argues that LLMs are better understood as reasoning engines that orchestrate specialized tools, rather than as monolithic systems that must internalize all capabilities. This anticipates the later wave of tool-use research (Toolformer, ART, etc.) that the paper explicitly connects to in Section 4.4.

Where Prior Approaches Fall Short

The paper identifies three categories of prior work and explains their limitations relative to the problem PoT solves.

Chain-of-Thought prompting (Wei et al., 2022). CoT was the state-of-the-art prompting method for numerical reasoning when this paper was written. It works by providing few-shot exemplars that include natural language reasoning steps leading to the answer, prompting the LLM to imitate this step-by-step style. The paper acknowledges CoT's success — it achieves strong results on GSM8K, SVAMP, and other benchmarks — but argues that its design is fundamentally limited for computation-heavy problems. CoT treats the LLM as both reasoner and calculator, which means it inherits all three failure modes described above (arithmetic errors, inability to solve complex equations, inefficiency with iteration).

The paper is careful not to dismiss CoT entirely. It explicitly notes (Section 5) that CoT remains the better choice for semantic reasoning tasks like commonsense QA (StrategyQA), where the reasoning cannot be easily expressed as executable code. PoT is positioned as a complement to CoT for the specific class of problems where computation can be cleanly separated from reasoning.

CoT with an external calculator (CoT+calc). A natural fix is to keep CoT's reasoning style but post-process the generated text to extract arithmetic expressions and compute them with an external calculator. This was proposed in Wei et al. (2022) and is evaluated as a baseline in this paper's Table 2. The results show that CoT+calc provides only mild improvements over vanilla CoT — for example, on GSM8K, Codex CoT achieves 63.1%, while CoT+calc achieves 65.4%, far behind PoT's 71.6%.

The paper attributes this weak performance to the rigidity of post-processing: extracting equations from free-form text is brittle, with low recall. The model might express arithmetic in ways the parser doesn't recognize, or it might make errors in the reasoning itself that can't be fixed by correcting the final calculation. PoT avoids this entirely by having the model generate structured code from the start, making the boundary between reasoning and computation unambiguous.

Direct equation generation without intermediate steps. The paper compares PoT to an approach where the model directly generates a single mathematical equation to solve the problem (e.g., solve(20000*(1+x)**3 - 2000 - x*20000*3 - 1000, x)). The authors cite Wei et al. (2022)'s observation that directly generating such equations is "challenging for LLMs." Their own ablation in Table 6 confirms this: removing the multi-step reasoning from PoT (the "PoT - MultiStep" variant, which asks the model to output the final equation directly) causes performance to drop dramatically — on GSM8K, from 71.6% to 45.8%.

The key insight is that breaking down the equation into intermediate reasoning steps — even in code — matters. PoT is not just "generate code instead of text." It is "generate code that expresses the reasoning process step by step, with semantically meaningful variable names, so that the LLM can leverage the same multi-step reasoning capabilities that make CoT effective, while still benefiting from precise external computation."

Fine-tuning approaches. Prior to CoT prompting, the dominant paradigm was to fine-tune models on datasets with expert-annotated reasoning steps (Ling et al., 2017; Cobbe et al., 2021). The paper notes (Section 1) that these methods are "data-intensive, requiring a significant number of training examples with expert-annotated steps." PoT, as a prompting method, requires no fine-tuning — only a handful of exemplars (4–8 shots) written by the authors. This makes it dramatically cheaper to deploy and more flexible across domains.

How This Paper Positions Itself

The paper positions PoT as an evolution of CoT that preserves its core insight — step-by-step reasoning elicits better performance from LLMs — while fixing its fundamental weakness — LLMs should not be trusted with numerical computation. The framing in Section 2.2 is explicit about this lineage:

"Unlike CoT, PoT relegates some computation to an external process (a Python interpreter). The LLMs are only responsible for expressing the 'reasoning process' in the programming language."

Three design choices define how PoT differs from and improves upon prior work:

1. Programs, not equations. PoT generates multi-step Python programs with semantically meaningful variable names (interest_rate, total_eggs, cost_per_glass), not opaque mathematical expressions. The paper's ablation (Table 6, "PoT - Binding") shows that replacing descriptive variable names with generic a, b, c hurts performance — on GSM8K, accuracy drops from 71.6% to 60.2%. The semantic binding helps the LLM ground its reasoning in the problem's entities, much like CoT's natural language steps help ground numeric reasoning in real-world context.

2. Complementary to CoT, not a replacement. The paper explicitly designs PoT to work with CoT when needed (Section 2.3, Figure 8). For problems requiring both symbolic computation and additional textual reasoning (like the AQuA dataset, where computed answers must be matched to multiple-choice options), PoT generates the program to compute the intermediate result, then CoT takes over to handle the final textual reasoning step. This composability is a deliberate design choice that acknowledges PoT's scope boundaries.

3. A general principle, not a task-specific hack. While the paper evaluates on math and finance datasets, it frames PoT as applicable to any domain where reasoning can be expressed as executable code and computation can be offloaded to a reliable interpreter. Section 5 explicitly extends this to "problems which require highly symbolic reasoning skills" and connects to the emerging tool-use paradigm where LLMs orchestrate external APIs.

Relationship to contemporary work. The paper acknowledges PaL (Gao et al., 2022) as a nearly simultaneous proposal with similar ideas (Section 3.3, Table 5). PoT and PaL both use hybrid text/code reasoning, but PoT achieves higher accuracy — particularly on SVAMP (85.2% vs. 79.4%) and ASDIV (85.2% vs. 79.6%). The paper attributes this to PoT's emphasis on multi-step reasoning within the program (semantic binding, stepwise decomposition) rather than generating monolithic code blocks.

The paper also positions itself as a precursor to the broader tool-use paradigm that emerged shortly after (Section 4.4), citing Toolformer (Schick et al., 2023) and ART (Paranjape et al., 2023) as generalizations of the Python interpreter concept to other tools (search engines, calculators, string extractors). This framing is important: PoT is not presented as a one-off prompting trick but as an early demonstration of a principle — language models as reasoning coordinators that delegate specialized computation to external tools — that would become a major research direction.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

What the system is: Program of Thoughts (PoT) is a prompting strategy that teaches a large language model to express its step-by-step reasoning as executable Python code rather than as natural language arithmetic, then runs that code through a real Python interpreter to obtain precise numerical answers. What problem it solves and the shape of the solution: The core problem is that LLMs make arithmetic errors, cannot solve complex equations, and are inefficient at expressing iteration when doing Chain-of-Thought reasoning — PoT solves this by splitting the job in two: the LLM handles the reasoning (deciding what operations to perform, in what order, on which quantities) and a Python interpreter handles the computation (executing those operations with perfect precision). The solution's shape is therefore a pipeline: (1) the LLM receives the question and few-shot exemplars showing how to write reasoning as code, (2) the LLM generates a Python program with semantically meaningful variable names and step-by-step logic, (3) the program is executed in a sandboxed Python environment, and (4) the final value of the ans variable is returned as the answer.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components connected in a linear pipeline with one conditional branching point:

1. Prompt Constructor — takes the input question, formats it according to a template (including 4–8 few-shot exemplars or a zero-shot instruction), and linearizes any structured inputs (tables, conversations) into plain text. This component handles the heterogeneity of input formats across the eight evaluated datasets.

2. Language Model (Codex code-davinci-002) — receives the constructed prompt and generates a multi-step Python program that expresses reasoning about the problem. Under few-shot prompting, the model imitates the exemplar programs; under zero-shot prompting, it follows a natural language instruction. The output is a string containing Python code that ends with an ans variable assignment.

3. Python Interpreter (Python 3.8 with SymPy) — executes the generated program in a sandboxed environment that blocks importing additional modules beyond a pre-approved set (math, sympy, and solve_it, a custom wrapper around SymPy's solve). The execution extracts the value of the ans variable after the program completes.

4. Answer Post-Processor — takes the interpreter's output and, depending on the dataset, either returns it directly (for GSM8K, SVAMP, MultiArith, where answers are plain numbers), rounds it to a specified precision for numeric matching, or feeds it back into a second LLM call as an intermediate result (for AQuA multiple-choice, where the computed value must be mapped to the closest option).

Information flow narrative (what happens first, second, third): First, the Prompt Constructor takes the raw question along with any tables or conversation history and flattens everything into a single text string — tables become |-separated columns with \n-separated rows, conversation turns are concatenated with \n. Second, this text is prefixed with either a few-shot prompt (containing 4–8 exemplar question-code-answer triples) or a zero-shot instruction ("Write Python Code to solve the following questions. Store your result as a variable named 'ans'"). Third, Codex receives this full prompt and autoregressively generates a Python program, stopping when it outputs a complete code block. Under zero-shot prompting, the # token logit is suppressed by a bias of -2 to prevent the model from generating reasoning as comments rather than executable code. Fourth, the generated code string is extracted and executed in a Python 3.8 subprocess with sympy available but no other external module imports permitted (for security). Fifth, the interpreter returns the value bound to the variable named ans. Sixth, for datasets requiring exact-match numeric answers, the value is rounded to the dataset-appropriate precision and compared to the ground truth; for multiple-choice datasets (AQuA), the value is injected into a follow-up prompt where the LLM selects the closest option — this is the "PoT as intermediate step" path (Section 2.3, Figure 8). For the financial datasets FinQA and ConvFinQA, the answer is compared using math.isclose with a relative tolerance of 0.001 (to accommodate floating-point imprecision in financial calculations). For TabMWP and TATQA, the official dataset-provided evaluation scripts are used.

3.3 Roadmap for the Deep Dive

• First, the prompt construction and input linearization mechanism, because this determines what information the LLM actually sees and how the system handles the eight datasets' heterogeneous input formats (text, tables, conversation, and their combinations).
• Second, the few-shot and zero-shot prompting templates, including the exemplar design philosophy, the zero-shot instruction, and the # token suppression trick, since these are the primary design levers that make PoT work without any model fine-tuning.
• Third, the structure of the generated Python programs, examining the multi-step reasoning pattern, the semantic variable binding, the use of SymPy and custom solve_it for symbolic math, and the ans output convention — this is the heart of what makes PoT different from both CoT and direct equation generation.
• Fourth, the Python execution environment and security model, including module whitelisting, the solve_it wrapper, and why sandboxing matters for safe deployment.
• Fifth, the PoT-as-intermediate-step extension (Section 2.3, Figure 8), which combines PoT with CoT for problems requiring both symbolic computation and additional textual reasoning — this is the architectural pattern that shows PoT is composable with other methods rather than a standalone replacement.
• Sixth, design rationales, synthesizing why specific choices (programs over equations, semantic bindings, multi-step code, zero-shot # suppression) were made in light of the ablation evidence in Table 6 and the breakdown analysis in Figure 6.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a prompting-method design paper whose core idea is that generating executable code — with semantically meaningful variable names, multi-step decomposition, and an external interpreter for computation — is a more reliable way to elicit numerical reasoning from LLMs than generating natural-language arithmetic chains. The method requires no fine-tuning: all behavior is controlled by the structure of the few-shot exemplars or zero-shot instruction provided in the prompt.

Prompt Construction and Input Linearization

The eight evaluated datasets span three fundamentally different input formats: plain text questions (GSM8K, AQuA, SVAMP, MultiArith), table-plus-text questions (TabMWP, FinQA, TATQA), and conversation-plus-table-plus-text questions (ConvFinQA). The prompt constructor has a single responsibility: convert any of these into a flat text string that Codex can process, since Codex's API accepts only a single text prompt.

Table linearization. When a dataset includes tabular data, the table is converted to a text representation modeled after the format introduced in Chen (2022). Each row of the table becomes one line of text, columns within a row are separated by the | character, and empty cells are filled with the placeholder -. So a financial table with columns (Year, Revenue, Profit) and two data rows becomes:

Year|Revenue|Profit
2020|1000000|200000
2021|1200000|250000

This linearization preserves the two-dimensional structure in a one-dimensional form that the LLM can process sequentially. The choice of | as column separator is motivated by its rarity in natural text (reducing confusion with actual content) and its use in prior table-reasoning work.

Text and table combination. When a problem includes both free-text context and tables (as in FinQA, ConvFinQA, TATQA, and TabMWP), the text and linearized tables are concatenated with \n newline separators between them. There is no special delimiter marking where text ends and tables begin — the LLM learns to distinguish them from context, since the |-separated lines are visually distinct from prose paragraphs.

Conversation history. For ConvFinQA, which involves multi-turn conversational financial questions, all dialog turns are concatenated in chronological order, separated by \n. Each turn preserves speaker identification (e.g., "User: ..." and "Assistant: ...") as it appears in the original dataset. There is no special tokenization or role-based formatting beyond what the dataset provides.

Final prompt assembly. The complete prompt is assembled as the concatenation of:

1. A task instruction (either the few-shot instruction introducing the exemplars or the zero-shot instruction described below)
2. For few-shot: 4–8 exemplar blocks, each containing a question and its corresponding Python program
3. The target question (with its linearized tables and conversation context if applicable)
4. A final line prompting the model to generate: # Python code, return ans (few-shot) or implicit continuation (zero-shot)

The total prompt length must fit within Codex's context window (which at the time was 8000 tokens for code-davinci-002). The authors do not report maximum prompt lengths, but with 8 exemplars of 100–300 tokens each plus the target question, the prompts likely occupy 2000–4000 tokens — well within limits.

Why this approach over alternatives. The alternative would be to train separate model architectures for each input format (table encoders, conversation encoders, etc.) or to fine-tune a model on each dataset. The linearization approach is zero-cost — it requires no model modification, no retraining, and no dataset-specific engineering beyond writing the few-shot exemplars. It works because LLMs at the scale of Codex (175B parameters) have seen sufficient tabular data and dialog transcripts during pretraining to parse these linearized formats without special handling. This is an empirical claim validated by the strong results across all eight datasets in Table 2, not a theoretical guarantee.

Few-Shot and Zero-Shot Prompting Templates

The paper uses two distinct prompting regimes, each with its own template, instruction, and failure modes.

Few-shot prompting template. The prompt begins with an implicit instruction conveyed through the exemplars themselves — the model learns the desired behavior by observing the pattern. The exemplar structure shown in Figure 3 (left) and the appendix prompts follows a consistent pattern:

Question: [natural language problem description]
# Python code, return ans
[multi-step Python program ending with assignment to 'ans']

Each exemplar is a complete (question, program) pair. The comment # Python code, return ans is not an instruction to the model — it is part of the exemplar pattern that the model learns to replicate. After seeing 4–8 such exemplars, the model receives the target question and generates the comment followed by the program.

The exemplars are hand-written by the authors for 10–20 candidate questions per dataset, then a subset of 4–8 is selected by tuning on a small validation set. This is important: the authors do not claim that any randomly chosen exemplars work equally well. The sensitivity analysis in Figure 5 shows that with only 2 exemplars, performance can vary by up to 7% depending on which exemplars are chosen, and that this variance shrinks as the number of exemplars increases to 6–8.

The number of exemplars is dataset-dependent: simpler datasets like FinQA use fewer shots (around 4), while more diverse datasets like AQuA and TATQA use 8 shots "to cover more diverse problems" (Section 3.1). This reflects an intuitive principle: the more varied the problem types in a dataset, the more exemplars are needed to adequately cover the space.

Zero-shot prompting template. The zero-shot prompt, shown in Figure 3 (right), replaces exemplars with a natural language instruction:

# Write Python Code to solve the following questions. Store your result as a variable named 'ans'.
from sympy import Symbol
from sympy import simplify
import math
from sympy import solve_it
# solve_it(equations, variable): solving the equations and return the variable value.

# Question: [target problem]

There are several design decisions embedded in this template. First, the from sympy import ... lines pre-load common imports into the model's context, guiding it toward generating code that uses these libraries — the model sees these imports as part of the prompt and is more likely to generate code that relies on them. Second, the comment defining solve_it teaches the model the interface of the custom wrapper function without needing an exemplar. Third, the instruction explicitly names the output variable ('ans'), standardizing extraction across all generated programs.

The # token suppression trick (zero-shot only). The paper identifies a specific failure mode in zero-shot PoT: the LLM can "fall back to generating a reasoning chain in comments rather than in the program" (Section 2.2). That is, the model might write:

# First, calculate the total distance
# total_distance = speed * time
# total_distance = 60 * 2.5 = 150

where the reasoning is in comments and the computation is still being done (incorrectly) by the model itself, defeating the purpose of PoT. To prevent this, the authors suppress the logit for the # token by adding a bias of -2 during decoding. This makes the model less likely to start lines with #, pushing it toward writing actual executable code rather than comment-based reasoning. The bias value of -2 was chosen through preliminary study: "we found that -2 as the bias can achieve the best result" (Section 3.1). Values too large would prevent necessary comments (like the # Python code, return ans header) while values too small would not adequately suppress the comment-fallback behavior.

Why few-shot and zero-shot are both evaluated. Few-shot prompting requires dataset-specific exemplars, which is cheap (writing 20 candidate exemplars takes human effort) but not zero-cost. Zero-shot prompting requires no dataset-specific preparation at all — the same template works across all datasets. The paper evaluates both to establish PoT's effectiveness across the full spectrum of deployment scenarios, from "we have a handful of training examples and can write prompts" to "we have a new dataset and need immediate results with no preparation."

Structure of the Generated Python Programs

The programs that PoT generates are not arbitrary Python scripts. They follow a specific multi-step reasoning pattern with semantic variable binding that is central to PoT's effectiveness. This pattern is what distinguishes PoT from both CoT (which uses natural language for reasoning) and direct equation generation (which compresses all reasoning into a single opaque expression).

Multi-step decomposition. Rather than producing a single equation that directly computes the answer, PoT programs break the reasoning into sequential steps, each line building on previous lines. For example, from the GSM8K exemplars (Appendix 7.2):

total_eggs = 16
eaten_eggs = 3
baked_eggs = 4
sold_eggs = total_eggs - eaten_eggs - baked_eggs
dollars_per_egg = 2
ans = sold_eggs * dollars_per_egg

Each line does one conceptually simple thing: extract a quantity from the problem, combine quantities with an operation, or produce the final answer. The model is not asked to jump directly from problem text to a formula like ans = (16 - 3 - 4) * 2 — instead, it works through the reasoning incrementally, mirroring how a human would solve the problem on paper.

The ablation in Table 6 quantifies how much this matters. The variant "PoT - MultiStep" prompts the model to directly generate the final equation rather than step-by-step code. On GSM8K, accuracy drops from 71.6% to 45.8% — a 25.8 percentage point decline. This is the single largest ablation effect in the paper, demonstrating that the multi-step structure is responsible for a large fraction of PoT's advantage. The model needs the intermediate steps to organize its reasoning; compressing everything into one equation is too difficult, consistent with the CoT literature's finding that step-by-step reasoning elicits better performance.

Semantic variable binding. Variable names in PoT programs carry semantic meaning: total_eggs rather than x, dollars_per_egg rather than a, interest_rate rather than r. This is not cosmetic — it serves a cognitive function for the language model. By binding problem entities to named variables, the model explicitly declares what each quantity represents, which helps it maintain consistency across steps and reduces confusion between similar quantities (e.g., cost_of_original_house vs. cost_of_repair vs. value_of_house in the house-flipping exemplar).

The "PoT - Binding" ablation in Table 6 tests this by replacing semantic variable names with generic a, b, c. The results show consistent degradation: GSM8K drops from 71.6% to 60.2% (−11.4 points), SVAMP drops from 85.2% to 83.8% (−1.4 points), FinQA drops from 64.5% to 61.6% (−2.9 points). The larger drop on GSM8K (which has more complex multi-variable problems) compared to SVAMP (which has simpler problems) suggests that semantic binding matters more as the number of quantities in a problem grows — when there are many variables to track, descriptive names prevent the model from confusing them.

The mechanism is likely related to how Codex was trained: it was trained on a corpus that includes many Python programs with descriptive variable names (from GitHub repositories, tutorials, and documentation). The pretraining distribution conditions it to expect that variables have meaningful names, and this prior may help it generate more coherent programs when it follows that convention.

Iteration with for-loops and while-loops. One of the three failure modes the paper attributes to CoT is inefficiency at expressing iteration. PoT solves this by generating loops. For example, from the Kylar glasses exemplar:

num_glasses = 16
first_glass_cost = 5
second_glass_cost = 5 * 0.6
ans = 0
for i in range(num_glasses):
    if i % 2 == 0:
        ans += first_glass_cost
    else:
        ans += second_glass_cost

A CoT solution would need to unroll this loop into 16 lines of arithmetic (one per glass), each with a chance of calculation error. The PoT version compresses 16 iterations into 5 lines of code, and the Python interpreter executes the loop with perfect accuracy. For problems with larger iteration counts — the paper's example of 50 additions in the introduction, where CoT's per-step error compounds to a <1% success rate — this capability is critical.

For problems requiring an unknown number of iterations (convergence, payback periods), PoT uses while-loops. The Carlos lemon tree exemplar demonstrates this:

total_cost = 90
cost_of_watering_and_feeding = 3
cost_of_each_lemon = 1.5
num_of_lemon_per_year = 7
ans = 0
while total_cost > 0:
    total_cost += cost_of_watering_and_feeding
    total_cost -= num_of_lemon_per_year * cost_of_each_lemon
    ans += 1

The while-loop accumulates years until the initial investment is recovered — something CoT cannot do at all unless the number of iterations is known in advance and small enough to unroll.

Symbolic mathematics with SymPy and solve_it. For problems requiring equation solving (polynomial equations, systems of linear equations), PoT programs use the SymPy symbolic mathematics library. The paper wraps SymPy's solve function in a custom solve_it helper to simplify the interface:

# solve_it(equations, variable): solving the equations and return the variable value.

In the generated programs, solve_it is called with either a single equation or a list of equations, plus the variable(s) to solve for. For example, from the AQuA flight duration exemplar:

duration = Symbol('duration', positive=True)
delay = 30 / 60
total_distance = 600
original_speed = total_distance / duration
reduced_speed = total_distance / (duration + delay)
solution = solve_it(original_speed - reduced_speed - 200, duration)
ans = solution[duration]

This approach handles equation types (cubic, quadratic, transcendental) that CoT cannot solve at all — the lower example in Figure 1 explicitly shows CoT failing on a cubic equation while PoT solves it via SymPy. The positive=True constraint on the Symbol declaration restricts the solver to physically meaningful solutions (e.g., negative durations are excluded), mirroring how a human would set domain constraints.

The solve_it wrapper exists for two reasons. First, it simplifies the interface — the model only needs to learn one function signature rather than SymPy's full solve API. Second, it provides a security boundary: solve_it is a controlled function that the execution environment exposes, allowing only equation-solving operations without exposing SymPy's full capabilities (which could include file I/O or other dangerous operations in theory).

The ans output convention. Every PoT program ends by assigning the final answer to a variable named ans. This convention serves as a contract between the LLM's code generation and the answer extraction step: the system knows exactly which variable to read after execution. The convention is taught through the exemplars (every exemplar ends with ans = ...) and in the zero-shot instruction ("Store your result as a variable named 'ans'").

For problems with structured answers (like the rectangle dimensions question where the answer is a pair), ans can be a tuple: ans = (solution[width], solution[height]). The post-processing step handles these cases by extracting the tuple elements as needed for the dataset's evaluation format.

Datatype handling. For multiple-choice datasets like AQuA, the program does not output the final letter option directly — it computes the numeric answer, and then an optional second LLM call (the "PoT as intermediate step" path) maps that numeric answer to the closest option. For numeric-answer datasets like GSM8K, the program outputs a number directly. For TabMWP and TATQA, which have mixed numeric and text answers, the program outputs whatever the reasoning determines (number or string), and the official evaluation scripts handle comparison.

Python Execution Environment and Security Model

The paper acknowledges a practical risk: "PoT would require execution of 'generated code' from LLMs, which could contain certain dangerous or risky code snippets like import os; os.rmdir()" (Section 6, Limitations). Since the generated programs come from a language model that can produce arbitrary text, they cannot be trusted to be safe by default.

Module whitelisting. The execution environment addresses this by blocking all imports of additional modules beyond a predetermined safe set. The whitelist includes:

• math — standard mathematical functions (sqrt, sin, cos, log, etc.)
• sympy — symbolic mathematics (symbols, equations, solving, simplification)
• solve_it — the custom wrapper around sympy.solve

Any import statement for a module not on this list (e.g., import os, import subprocess, import requests) would fail at execution time. The paper notes that this "brutal-force blocking works reasonable for math QA, however, for other unknown symbolic tasks, it might hurt PoT's generalization" (Section 6). This is a deliberate tradeoff: security over flexibility. For the math and finance domains evaluated, the whitelist is adequate; for general-purpose programming tasks, it would be too restrictive.

Python version and libraries. The execution uses Python 3.8 with the SymPy library (version unspecified but likely recent at the time of writing, late 2022). SymPy is a pure-Python symbolic mathematics library that provides algebraic simplification, equation solving, calculus, and other mathematical operations without requiring external compiled dependencies. This makes it suitable for sandboxed execution environments.

The solve_it wrapper implementation. While the paper does not provide the full source code of solve_it, its behavior is clear from the usage in exemplars:

• Input: one or more equations (expressions assumed equal to zero) and one or more Symbol objects to solve for
• Output: a dictionary mapping solved symbols to their numeric values
• Wraps sympy.solve to handle the common case where the model sets up equations as expressions (e.g., original_speed - reduced_speed - 200) rather than explicit Eq objects

Why sandboxing matters beyond security. There is a second reason for the restricted environment: determinism. Math and finance problems have unambiguous correct answers, and the Python interpreter provides deterministic computation — given the same program, it always produces the same result. This is not true of LLM-based computation, where the same reasoning chain might produce different arithmetic results on different samples due to the probabilistic nature of token generation. PoT's external interpreter eliminates this source of variance, which is important for the self-consistency experiments (where 40 samples are drawn and the majority answer is taken) — the variance comes only from different reasoning paths in the generated code, not from arithmetic errors within those paths.

PoT as an Intermediate Step (Combining PoT with CoT)

Not all problems can be solved purely through computation. Some require additional textual reasoning after the computation is done — for instance, matching a computed numeric answer to a multiple-choice option, or reasoning about the semantic implications of a numerical result. Section 2.3 and Figure 8 introduce a compositional extension: PoT handles the computation, then CoT handles the residual textual reasoning.

The two-stage pipeline. The process works in two sequential LLM calls:

1. Stage 1 (PoT): The question is prompted with PoT exemplars to generate a Python program. The program is executed, producing an intermediate answer (the value of ans). The program may explicitly indicate that further reasoning is needed by ending with a comment like # keep prompting.

2. Stage 2 (CoT): If the program indicates continuation (or if the dataset requires it, as with AQuA's multiple-choice format), the intermediate answer is formatted as a natural language statement (e.g., "according to the program: duration = 2.05") and appended to the original question. This augmented question is then fed to the LLM with CoT exemplars to produce the final answer.

Concrete example from AQuA (Figure 3, left exemplar). The original question asks: "In a flight of 600 km, an aircraft was slowed down due to bad weather. Its average speed for the trip was reduced by 200 km/hr and the time of flight increased by 30 minutes. The duration of the flight is:" with options A)1 hour, B)2 hours, C)3 hours, D)4 hours, E)5 hours.

Stage 1 (PoT) generates a program that solves the equation system and produces ans = 1 (the duration is 1 hour). But the program could also produce a numeric value that doesn't directly match an option — for instance, if the computed answer were 2.05 hours. Stage 2 (CoT) would take this 2.05, reason that none of the options are 2.05 hours, and select the closest option or identify that the answer format needs conversion.

The paper notes that this two-stage prompting "is only needed for the AQuA dataset because the other datasets can all be solved by PoT-only prompting" (Section 2.3). This is an important scoping claim: for most numerical reasoning benchmarks, pure PoT suffices; the two-stage architecture is a fallback for problems where the answer format requires additional reasoning that cannot be expressed as Python code.

Implementation detail. The decision to invoke Stage 2 can be either explicit (the program outputs a flag) or implicit (the dataset format requires it). In the AQuA case, it is implicit: the dataset expects a letter option (A–E), not a numeric value, so the numeric output of PoT must be mapped to an option letter. This mapping is done by a second LLM call with CoT exemplars that demonstrate selecting the closest option to a computed value.

Why not do everything in CoT or everything in PoT. The hybrid approach acknowledges a boundary condition: PoT works when the reasoning can be fully expressed as imperative code with a deterministic output; CoT works when the reasoning requires natural language understanding, common sense, or flexible text generation that cannot be easily encoded in Python. By composing them, the system gets the best of both — precise computation from PoT, flexible reasoning from CoT — without forcing either method to handle tasks it is poorly suited for. This compositional design anticipates the later tool-use paradigm where LLMs orchestrate multiple specialized components rather than trying to be monolithic generalists.

Design Rationales: Why These Specific Choices

The architecture embodies several deliberate choices, each justified by empirical evidence in the paper's experiments and ablations.

Programs over natural-language arithmetic (the core bet). The fundamental design decision is to express reasoning as code rather than text. The justification is the three failure modes in Section 1: arithmetic errors (eliminated by the interpreter), inability to solve complex equations (enabled by SymPy), and inefficient iteration (enabled by loops). The evidence for this bet is the consistent double-digit improvement over CoT across all eight datasets (Table 2), with the largest gains on problems requiring the capabilities CoT lacks — the breakdown analysis in Figure 6 shows PoT outperforming CoT most dramatically on linear/polynomial equations (PoT: 72%, CoT: 62%), iterative problems (PoT: 38%, CoT: 20%), and combinatorics (PoT: 40%, CoT: 40%), while performing similarly on arithmetic (PoT: 88%, CoT: 86%) and probability (PoT: 40%, CoT: 40%).

Multi-step code over single equations (the decomposition principle). The decision to generate step-by-step programs rather than direct equations is justified by the "PoT - MultiStep" ablation, where compressing the reasoning into a single equation causes a 25.8-point drop on GSM8K. This mirrors the CoT finding that step-by-step reasoning elicits better performance, suggesting a general principle: LLMs reason better when they can externalize intermediate states, whether those states are written in English or Python.

Semantic variable names over generic names (the grounding principle). The "PoT - Binding" ablation justifies naming variables descriptively. The 11.4-point drop on GSM8K when switching to a, b, c suggests that variable names serve as a grounding mechanism — they link the abstract quantities in the code to the concrete entities in the problem text, helping the model maintain a consistent mapping between the problem's world and the program's state.

External interpreter over internal computation (the separation principle). This is the defining architectural choice, and it has a clear justification: LLMs are probabilistic text generators, not deterministic computers. The paper's error analysis supports this — a substantial fraction of CoT errors are arithmetic mistakes that an interpreter would not make. The external interpreter transforms PoT from an approximate system (where the LLM might calculate correctly 90% of the time) to an exact one (where, given correct reasoning code, the computation is 100% accurate). The residual errors come from incorrect reasoning in the generated code, not from computational mistakes.

Codex (code-davinci-002) over GPT-3 (text-davinci-002). The choice of a code-trained model over a text-trained model is justified by Table 4: Codex achieves 71.6% on GSM8K and 85.2% on SVAMP, while GPT-3 (which shares the same 175B parameter count but was trained primarily on text) achieves only 60.4% and 80.1% respectively. The paper attributes this gap to "the following text-based instruction tuning undermines the models' capabilities to generate code" — that is, the RLHF fine-tuning applied to text-davinci-002 to make it better at following natural language instructions may have degraded its code generation abilities, while code-davinci-002 was specifically optimized for code. The 16B open-source CodeGen models perform dramatically worse (8.2% and 12.7% on GSM8K for the multi-language and Python-only variants respectively), suggesting that model scale and training data quality are critical for PoT to work — the capability to generate correct reasoning programs from natural language questions emerges only at large scale.

Few-shot over fine-tuning. The decision to use prompting rather than fine-tuning is justified by cost and flexibility: fine-tuning requires "a significant number of training examples with expert-annotated steps" (Section 1), while PoT requires only 4–8 hand-written exemplars per dataset. This makes PoT applicable to new datasets with minimal human effort. The tradeoff is that PoT is limited by the model's few-shot learning capability, which is weaker than what fine-tuning can achieve on large training sets — but the paper shows that few-shot PoT already achieves state-of-the-art results on most evaluated datasets (excluding GPT-4), suggesting that the prompting approach is sufficient for these benchmarks.

4. Key Insights and Innovations

Innovation 1: Reasoning and Computation Are Separable Competences, and Conflating Them Is the Root Cause of CoT's Numerical Failures

The paper's most fundamental conceptual move is diagnosing why Chain-of-Thought prompting fails on numerically complex problems, then using that diagnosis to motivate a clean architectural separation. This is not a metric-driven observation — it is a redefinition of the problem itself.

Prior to PoT, the dominant framing treated step-by-step reasoning as a unified competence: an LLM that could "show its work" was expected to handle both the logical decomposition and the arithmetic execution within the same generation. CoT (Wei et al., 2022) demonstrated that providing natural-language rationales in few-shot exemplars elicits this behavior, and the field largely accepted the conflation as inevitable — if the model is doing the reasoning, it might as well do the math too. The "CoT + calculator" variant (Wei et al., 2022) half-acknowledged the problem by post-processing arithmetic expressions, but it kept the model's generation format unchanged (natural language with embedded math) and relied on brittle regex-based extraction to fix errors after the fact.

PoT's diagnostic reframing identifies the conflation itself as the error. The three failure modes articulated in Section 1 — arithmetic calculation errors, inability to solve complex equations, and inefficiency at expressing iteration — are not presented as accidental limitations that might be fixed with better training or larger models. They are presented as fundamental category errors: asking a probabilistic next-token predictor to behave like a deterministic calculator. This is a more precise claim than "LLMs are bad at math." It says: LLMs are bad at a specific role (computation executor) that CoT implicitly assigns to them, and they are bad at it for structural reasons that scale alone will not fix — no amount of parameter scaling makes a language model into a reliable floating-point arithmetic unit.

What makes this framing distinctive is that it converts an empirical weakness into a design principle. The paper does not say "we found a better prompting template that achieves higher accuracy." It says: "CoT's architecture — reasoning and computation interleaved in natural language — is incorrectly designed for numerical tasks. The correct architecture separates them." This is visible in the paper's deliberate language: "disentangling computation from reasoning" in the title, "delegate computation steps to an external language interpreter" in the abstract, "relegate some computation to an external process" in Section 2.2. The word choice emphasizes transfer of responsibility, not improvement within the same paradigm.

The evidence that this reframing is causal rather than correlational comes from the breakdown analysis in Figure 6. If PoT were simply a better prompting template in general, we would expect uniform improvement across all problem types. Instead, the improvement is sharply concentrated: PoT outperforms CoT dramatically on linear/polynomial equations (72% vs. 62%), iterative problems (38% vs. 20%), and combinatorics (40% vs. 40% — a tie), but performs comparably on arithmetic (88% vs. 86%) and probability (40% vs. 40%). The pattern matches the diagnosis precisely: problems requiring capabilities CoT structurally lacks (equation solving, iteration, symbolic manipulation) show large gains; problems requiring only simple arithmetic that CoT handles adequately show minimal gains. This differential improvement pattern is what distinguishes a genuine diagnostic insight from a generic accuracy boost.

The significance extends beyond the immediate benchmark results. By establishing that reasoning and computation are separable — and that separating them yields large, consistent improvements — the paper provides conceptual grounding for the entire subsequent tool-use paradigm (Toolformer, ART, and related work that the paper anticipates in Section 4.4). It is one thing to say "LLMs can call APIs"; it is another to argue that they should, because computation and reasoning are different cognitive operations that different architectures handle best. PoT makes the latter, stronger argument.

Innovation 2: Programs as the Intermediate Reasoning Language — Not Just Computation Carriers but Cognitive Scaffolding

A common misinterpretation of PoT is that its contribution is "use code instead of text" — a surface-level format change. The paper's actual insight is subtler and more significant: programs serve as cognitive scaffolding that preserves the multi-step reasoning benefits of CoT while fixing CoT's computational unreliability. The code is not merely a computation specification; it is a reasoning trace expressed in a formal language.

To see why this distinction matters, consider what PoT is not. It is not "generate a single equation and solve it" — the "PoT - MultiStep" ablation in Table 6 shows that compressing the reasoning into one equation causes a catastrophic 25.8-point drop on GSM8K (71.6% → 45.8%). It is not "generate code with generic variable names" — the "PoT - Binding" ablation shows an 11.4-point drop when semantic names are replaced with a, b, c. It is not even "generate code that happens to be multi-step" — the programs must be deliberately structured as reasoning chains where each line corresponds to one conceptual step in the problem-solving process.

The innovation is recognizing that the benefits of CoT come from the act of externalizing intermediate reasoning states, not from the natural-language format in which those states are expressed. CoT's key finding was that when LLMs write down their reasoning step by step, they reason better — the externalization process itself improves the internal computation. PoT's contribution is showing that this externalization benefit transfers to a different medium (code) provided the medium preserves the step-by-step structure and the semantic grounding of variables to problem entities.

This is a conceptual advance over the natural-language-centric view of reasoning that CoT implicitly endorsed. Before PoT, the default assumption — visible in the CoT and zero-shot CoT papers — was that reasoning traces must be in natural language because reasoning is linguistic. PoT challenges this: reasoning traces can be in any symbol system that supports (a) sequential decomposition of complex thoughts, (b) named references to intermediate results, and (b) compositional assembly of those results into final conclusions. Python code satisfies all three criteria while adding a fourth that natural language lacks: unambiguous operational semantics (the interpreter defines exactly what each line means).

The evidence for this transfer claim comes from the comparison of PoT against CoT on problem types that require no complex computation. On arithmetic questions in the AQuA breakdown (Figure 6), PoT achieves 88% while CoT achieves 86% — essentially identical. If PoT's advantage came entirely from the external interpreter's arithmetic precision, we would expect it to match CoT on problems where CoT's arithmetic is reliable (which it is for simple operations). The fact that PoT matches or slightly exceeds CoT on these problems — while dramatically exceeding it on computationally complex ones — supports the claim that code-based reasoning traces are at least as effective as natural-language traces for eliciting logical reasoning, independent of the computation-accuracy benefit.

The semantic binding finding adds another layer. In natural language CoT, variables are implicitly bound through natural language descriptions ("let the original price be $x$"). In PoT, they are explicitly bound through named Python variables (original_price = ...). The ablation shows that this explicit binding matters — when removed, performance degrades, particularly on problems with many quantities (GSM8K drops more than SVAMP). This suggests that the simple act of naming intermediate quantities in a formal symbol system provides a reasoning benefit beyond what informal natural-language naming provides, possibly because the formal system enforces consistency (a variable's value cannot silently change between references) in a way that natural language does not.

This insight reframes what "intermediate reasoning" means for LLMs. It is not about English per se; it is about creating a persistent, structured representation that the model can refer back to as it generates subsequent tokens. Code provides a more reliable version of this representation than prose because its semantics are fixed by the interpreter rather than by the model's fallible linguistic understanding.

Innovation 3: The Difficulty-Dependent Benefit Structure — PoT Is Not Uniformly Better, It Is Better Where Computation Is the Bottleneck

The paper contains an implicit but important finding that goes beyond the headline accuracy numbers: PoT's advantage over CoT is not a uniform improvement across all problem types, but rather a targeted fix for the specific failure modes it was designed to address. This is a significant departure from the typical prompting-paper narrative of "our method beats the baseline on aggregate."

The breakdown analysis in Figure 6 is the key evidence. On AQuA, the authors manually classify test questions into categories (geometry, polynomial, symbolic, arithmetic, combinatorics, linear equation, iterative, probability) and report accuracy for PoT and CoT within each category. The results show a clear pattern: PoT's advantage is largest on linear/polynomial equations (+10 points: 72% vs. 62%), iterative problems (+18 points: 38% vs. 20%), and symbolic problems (+36 points: 56% vs. 20% — the single largest category-level gap). On arithmetic, probability, and geometry, the gap is small or zero.

What makes this an innovation rather than an obvious consequence of the method design is the diagnostic specificity it enables. The paper's three failure modes (arithmetic errors, inability to solve complex equations, inefficient iteration) were hypothesized in the introduction. The breakdown analysis confirms that these are precisely the problem types where PoT helps, and that PoT provides no special benefit on problem types where these failure modes are absent. This closes the loop from diagnosis → design → validation in a way that strengthens the paper's central argument: the problem really was the conflation of reasoning with computation, because fixing that conflation helps exactly on computation-heavy problems and not on reasoning-only problems.

This difficulty-dependent pattern has practical implications that the paper does not fully explore but that are important for deployment decisions. It means PoT is not a universal replacement for CoT — it is the right tool for problems where computation is the bottleneck, while CoT remains competitive (and perhaps preferable, given its greater flexibility for non-computational reasoning) on problems where reasoning structure is the bottleneck. The paper acknowledges this boundary in Section 5: "For semantic reasoning tasks like commonsense reasoning (StrategyQA), we conjecture that PoT is not the best option. In contrast, CoT can solve more broader reasoning tasks." The breakdown analysis provides empirical grounding for this conjecture within the numerical reasoning domain itself — even among math problems, PoT's value varies systematically with the computational demands of the question.

The broader significance is methodological. The paper demonstrates a pattern for how to evaluate prompting methods: not just aggregate accuracy, but breakdown by problem characteristics that the method was designed to address. This is a higher standard than the field typically applies, and it allows a more nuanced understanding of when and why a method works rather than just whether it works.

Innovation 4: Compositional Prompting — PoT Is Not a Replacement for CoT but a Module That Can Be Combined with It

The "PoT as intermediate step" mechanism (Section 2.3, Figure 8) introduces a design pattern that was relatively novel at the time: prompting strategies can be composed as modules, with one method's output feeding into another method's input, to handle problems that neither method can solve alone. This anticipates the multi-step, tool-combining architectures that became prominent in subsequent work (ReAct, ART, Toolformer) but does so within the simpler constraints of pure prompting — no fine-tuning, no architectural modifications, no external tool registration beyond the Python interpreter.

The specific composition is PoT → CoT: PoT generates and executes a program to compute an intermediate numeric result, then that result is formatted as natural language context and fed into a CoT prompt to perform additional textual reasoning (e.g., matching the numeric answer to a multiple-choice option). This addresses a genuine boundary condition: some problems require both precise computation (which PoT provides) and flexible natural-language reasoning (which CoT provides), and neither method alone suffices.

What makes this distinctive is that it treats prompting methods not as monolithic competitors but as composable reasoning primitives with different strengths. The paper does not frame PoT as "better than CoT" but as "complementary to CoT" — PoT handles the computation, CoT handles the residual reasoning, and the pipeline architecture allows both to operate within their competence zones. This is a more sophisticated view of LLM capabilities than the "one prompt to rule them all" approach that characterized most contemporaneous prompting work.

The evidence that this composition is necessary rather than optional comes from the AQuA results. AQuA is a multiple-choice dataset where final answers are letter options (A–E), not numbers. Pure PoT cannot output a letter — it outputs numeric computation results. CoT can output letters but struggles with the computation. The compositional architecture (PoT computes the number, CoT maps it to the closest option) solves both problems. The paper reports 54.1% on AQuA with few-shot PoT (Table 2) and notes that this includes the compositional approach for multi-choice questions — without composition, pure PoT would be inapplicable to AQuA's output format.

The significance extends beyond the specific AQuA use case. The compositional pattern implies a general design philosophy: LLMs should be thought of as orchestrators that can chain together different reasoning strategies, tools, and output formats within a single inference pipeline, with each component selected based on its fitness for a specific sub-task. This is the architecture that the tool-use and agent literature would develop more fully in subsequent years, but PoT presents an early, clean demonstration of the principle without requiring the infrastructure (tool registration, API definitions, action spaces) that later work would add.

The limitation — acknowledged in the paper — is that the composition is currently manual: the human designing the prompt must decide which problems need two-stage processing and write exemplars accordingly. The paper does not propose an automated mechanism for deciding when to invoke the second stage. This leaves open the question that later work on adaptive tool use would address: can the LLM itself decide when to delegate to an external tool versus when to reason internally?

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on eight datasets spanning math word problems (MWP) and financial question answering. The MWP datasets are: GSM8K (Cobbe et al., 2021) using the 1,318-question test set; AQuA (Ling et al., 2017) using the 253-question test set; SVAMP (Patel et al., 2021) using the 1,000-question test set; TabMWP (Lu et al., 2022) using the 7,861-question test set; and MultiArith (Roy & Roth, 2015) using the 600-question test set. The financial QA datasets are: FinQA (Chen et al., 2021b) using the 1,147-question test set; ConvFinQA (Chen et al., 2022) using the 421-question test set; and TATQA (Zhu et al., 2021) using the 1,668-question dev set (since the test set labels are not public). These datasets span diverse input formats — plain text questions, table-plus-text questions, and conversation-plus-table-plus-text questions — making the evaluation a stress test of PoT's generalizability across input modalities.

• Base model(s). The primary model is OpenAI Codex (code-davinci-002, 175B parameters), a version of GPT-3 fine-tuned on code. The choice is motivated by PoT's requirement to generate executable Python programs: a code-trained model is expected to produce more reliable and syntactically correct code than a text-only model. Ablation experiments also test text-davinci-002 (175B, text-trained), gpt-3.5-turbo (ChatGPT), codegen-16B-multi (Nijkamp et al., 2022), codegen-16B-mono, CodeT5+ (Wang et al., 2023b, 16B), and Xgen (7B) to assess how PoT's effectiveness depends on model scale and training domain. The paper also reports published results for PaLM 540B (Chowdhery et al., 2022) and LaMDA 137B (Thoppilan et al., 2022) as baselines when available from prior work.

• Metrics. The primary metric is accuracy computed via exact match, though the exact matching procedure varies by dataset to accommodate different answer formats. For GSM8K, SVAMP, and MultiArith, the predicted number is rounded to a specified precision and compared to the reference number. For AQuA, PoT computes an intermediate numeric answer, then a second LLM call (CoT) maps it to the closest multiple-choice option; accuracy is measured on the final option selection. For TabMWP, ConvFinQA, and TATQA, the official evaluation scripts provided with each dataset are used. For FinQA, the evaluation is relaxed for CoT baselines specifically: because "LLMs cannot perform the computation precisely (especially with high-precision floats and large numbers)," math.isclose with a relative tolerance of 0.001 is used to compare answers — this concession to CoT's arithmetic weakness makes the PoT–CoT comparison more conservative (PoT's advantage would be larger under strict matching).

• Baselines. The paper compares against several categories of baselines:

  • Direct prompting: the LLM outputs the answer directly without intermediate reasoning steps. Evaluated for Codex, GPT-3, and PaLM (Table 2).
  • Chain-of-Thought (CoT) (Wei et al., 2022): few-shot prompting with natural language reasoning steps. Evaluated for Codex, GPT-3, PaLM, and LaMDA. This is the primary baseline.
  • CoT with calculator (CoT+calc) (Wei et al., 2022): CoT generation followed by post-processing that extracts arithmetic expressions and computes them with an external calculator. Evaluated for Codex, GPT-3, and PaLM.
  • CoT with self-consistency (CoT-SC) (Wang et al., 2022b): majority voting over 40 CoT completions sampled at temperature 0.4.
  • Published state-of-the-art (SoTA): the best known results on each dataset from prior work, which vary by dataset — on GSM8K/AQuA/SVAMP it is CoT+SC; on FinQA it is Wang et al. (2022a); on ConvFinQA it is FinQANet (Chen et al., 2022); on TabMWP it is Dynamic Prompt Learning (Lu et al., 2022); on TATQA it is RegHNT (Lei et al., 2022).
  • Zero-shot CoT (Kojima et al., 2022): used as the baseline for zero-shot experiments in Table 3.
  • PaL (Gao et al., 2022): a contemporary method that also uses hybrid text/code reasoning, compared in Table 5.

• Generation budget / compute accounting. The paper does not formalize compute in FLOPs or tokens. Instead, the few-shot regime uses a single generation (greedy decoding) per question for the main PoT and CoT comparisons. For self-consistency experiments, 40 completions are sampled per question at temperature 0.4, making the compute cost 40× higher. The zero-shot experiments also use single greedy generations. The paper does not discuss or compare inference latency, token costs, or total FLOPs between methods — the implicit comparison is at equal number of LLM calls (one per question for greedy, 40 for SC). The Python interpreter execution cost is negligible compared to LLM inference and is not accounted for.

• Cross-validation / statistical protocol. There is no formal cross-validation. For few-shot experiments, the authors follow a manual prompt engineering protocol: they write 10–20 candidate exemplars per dataset, then "tune the exemplar selection on a small validation set to choose the best 4–8 shots for the full set evaluation" (Section 3.1). The validation set is not further specified. For the sensitivity analysis in Figure 5, the authors randomly sample k = (2, 4, 6, 8) exemplars from a pool of 20 three times (v1, v2, v3) and report the performance of each sample to measure variance. This reveals that at K=2, performance varies by up to 7%, and that variance shrinks as K increases to 6–8. There is no reporting of confidence intervals, statistical significance tests, or multiple-run averaging — all reported numbers appear to be from single evaluation runs.

Main Quantitative Results

Few-Shot PoT vs. CoT (Greedy Decoding)

The headline result is Table 2 (upper section, "Few-shot prompt (Greedy Decoding)"). Across all eight datasets, PoT with Codex greedy decoding outperforms Codex CoT by margins ranging from 4.0 to 24.1 percentage points. The specific comparisons:

• GSM8K: PoT 71.6% vs. CoT 63.1% (+8.5 points). This is a 13.5% relative improvement.
• AQuA: PoT 54.1% vs. CoT 45.3% (+8.8 points). This is a 19.4% relative improvement.
• SVAMP: PoT 85.2% vs. CoT 76.4% (+8.8 points, but only 4.0 points over the CoT+calc variant at 77.0%). The smaller absolute gain on SVAMP is attributed to its relative simplicity — the paper notes "the improvement is 4% mainly due to its simplicity" (Section 3.2).
• TabMWP: PoT 73.2% vs. CoT 65.2% (+8.0 points).
• FinQA: PoT 64.5% vs. CoT 40.4% (+24.1 points). This is the largest absolute gain across all datasets — a 59.7% relative improvement.
• ConvFinQA: PoT 64.6% vs. CoT 45.6% (+19.0 points). Note: the table reports CoT on ConvFinQA as 45.6% but the text in Section 3.2 says 45.5% — this 0.1-point discrepancy is likely a rounding artifact in the text.
• TATQA: PoT 69.0% vs. CoT 61.4% (+7.6 points).

The average gain across all datasets is approximately 12% (matching the abstract's claim). The larger improvements on FinQA and ConvFinQA are explicitly attributed to "miscalculations on LLMs for large numbers (e.g. in the millions)" — these financial datasets involve precise arithmetic with large values that CoT handles unreliably, while PoT's Python interpreter computes them exactly.

CoT+calc as an ablation of the "external computation" axis. The CoT+calc baseline (Table 2) tests whether simply adding a calculator to CoT's output can close the gap with PoT. The results show it cannot: on GSM8K, CoT+calc achieves 65.4% vs. PoT's 71.6% (a 6.2-point gap remaining); on SVAMP, 77.0% vs. 85.2% (8.2-point gap). The paper attributes this to "the rigid post-processing step, which can lead to low recall in terms of calibrating the calculation results" — extracting arithmetic expressions from free-form CoT text is brittle, and errors in the reasoning itself (not just the final calculation) cannot be fixed by a calculator.

GPT-3 vs. Codex. Table 2 reports GPT-3 CoT at 46.9% on GSM8K vs. Codex PoT at 71.6% — a 24.7-point gap. GPT-3 PoT is not reported (the paper only evaluates PoT on Codex and other code-trained models), which limits the ability to attribute PoT's gains specifically to the prompting method vs. the code-trained backbone. However, Table 4 addresses this partially: text-davinci-002 PoT achieves 60.4% on GSM8K, which is 11.2 points below code-davinci-002 PoT's 71.6%, suggesting that the code-trained model is substantially better at generating correct reasoning programs even when using the same PoT prompting method.

Few-Shot PoT + Self-Consistency vs. CoT + Self-Consistency

Table 2 (middle section, "Few-shot prompt (Self-Consistency Decoding)") shows that PoT-SC maintains its advantage over CoT-SC, though the relative gap narrows on some datasets. With 40 samples at temperature 0.4:

• GSM8K: PoT-SC 80.0% vs. CoT-SC 78.0% (+2.0 points). This 2-point gap is smaller than the 8.5-point greedy gap, suggesting that self-consistency helps CoT more than it helps PoT — likely because CoT's arithmetic errors are partially mitigated by majority voting (different samples make different errors, and the correct answer can emerge from the consensus).
• AQuA: PoT-SC 58.6% vs. CoT-SC 52.0% (+6.6 points). The gap remains substantial.
• SVAMP: PoT-SC 89.1% vs. CoT-SC 86.8% (+2.3 points).
• TabMWP: PoT-SC 81.8% vs. CoT-SC 75.4% (+6.4 points).
• FinQA: PoT-SC 68.1% vs. CoT-SC 44.4% (+23.7 points). The massive gap persists because CoT's errors on large-number arithmetic are systematic — the model consistently miscalculates, so majority voting cannot recover the correct answer.
• ConvFinQA: PoT-SC 67.3% vs. CoT-SC 47.9% (+19.4 points).
• TATQA: PoT-SC 70.2% vs. CoT-SC 63.2% (+7.0 points, but note the table reports CoT-SC on TATQA as 63.2% — the text mentions 63.9% in Section 3.2, another rounding inconsistency).

The paper notes that "self-consistency decoding is less impactful for both PoT and CoT" on the financial datasets — the CoT-SC numbers on FinQA (44.4%) are only marginally above the greedy CoT numbers (40.4%), suggesting that when the underlying errors are systematic (miscalculating large numbers), sampling diversity does not help because all samples make similar mistakes.

Comparison to published SoTA. PoT-SC achieves the best known results on all MWP datasets among non-GPT-4 methods. On GSM8K, PoT-SC's 80.0% edges out the prior SoTA (CoT-SC) at 78.0%. On SVAMP and AQuA, PoT-SC similarly establishes new highs. On the financial datasets, PoT-SC is near the best known results — the published SoTA on FinQA is 68.0% (Wang et al., 2022a), and PoT-SC achieves 68.1%, essentially matching it; on ConvFinQA, PoT-SC's 67.3% is close to FinQANet's 68.9%. The paper also reports GPT-4 results separately in the bottom section of Table 2 (not part of the main comparison), where GPT-4 PoT achieves 97.2% on GSM8K and 97.4% on SVAMP — dramatically higher than the Codex results and included to show the method scales to more capable models.

Zero-Shot PoT vs. Zero-Shot CoT

Table 3 reports zero-shot results (no exemplars, only the instruction shown in Figure 3, right) on five MWP datasets, compared against zero-shot CoT results taken from Kojima et al. (2022). The zero-shot PoT prompt uses only a natural language instruction ("Write Python Code to solve the following questions...") plus pre-loaded imports, with no dataset-specific exemplars.

• GSM8K: Zero-shot PoT 57.0% vs. zero-shot CoT 40.5% (+16.5 points). This gap is nearly twice the few-shot gap (+8.5 points), suggesting that the exemplar-free setting amplifies PoT's relative advantage — without exemplars to guide CoT's reasoning format, CoT degrades more than PoT does.
• AQuA: Zero-shot PoT 43.9% vs. zero-shot CoT 31.9% (+12.0 points).
• SVAMP: Zero-shot PoT 70.8% vs. zero-shot CoT 63.7% (+7.1 points).
• TabMWP: Zero-shot PoT 66.5% vs. zero-shot CoT 53.5% (+13.0 points). Notably, zero-shot PoT on TabMWP (66.5%) actually exceeds few-shot CoT (65.2%), meaning the code-based reasoning approach without any exemplars outperforms text-based reasoning with hand-written exemplars.
• MultiArith: Zero-shot PoT 92.2% vs. zero-shot CoT 79.3% (+12.9 points).
• Average: Zero-shot PoT achieves 66.1% average accuracy vs. 53.7% for zero-shot CoT, a 12.4-point gain. The paper notes that "zero-shot PoT outperforms zero-shot CoT by an even larger margin" than in the few-shot setting (Section 3.2).

The zero-shot results also compare against GPT-3 Direct (no reasoning, answer only) at 31.0% average, establishing that the gains come from the reasoning process, not just from the model reading the question.

Comparison with PaL (Contemporary Work)

Table 5 compares PoT against PaL (Gao et al., 2022), a simultaneous proposal that also uses hybrid code/text reasoning for math problems. On six datasets:

• GSM8K: PoT 71.6% vs. PaL 72.0% (PoT is 0.4 points lower — essentially tied).
• GSM8K-Hard: PoT 61.8% vs. PaL 61.2% (+0.6 points).
• SVAMP: PoT 85.2% vs. PaL 79.4% (+5.8 points).
• ASDIV: PoT 85.2% vs. PaL 79.6% (+5.6 points).
• ADDSUB: PoT 92.2% vs. PaL 92.5% (0.3 points lower — tied).
• MultiArith: PoT 99.5% vs. PaL 99.2% (+0.3 points — tied).

The pattern is clear: on simpler datasets (GSM8K, ADDSUB, MultiArith), PoT and PaL perform similarly; on more complex datasets requiring multi-step reasoning (SVAMP, ASDIV), PoT holds a substantial advantage (~6 points). This is consistent with the paper's emphasis on multi-step decomposition and semantic binding as PoT's distinguishing features — on complex problems where these matter, PoT outperforms a method that uses code but may not emphasize step-by-step structuring to the same degree.

Breakdown Analysis: Where PoT Helps Most

Figure 6 provides the most granular performance analysis, breaking down AQuA test questions into eight manually classified categories and reporting PoT and CoT accuracy within each. The results (read approximately from the bar chart):

The categories where PoT dominates — polynomial equations (+40), symbolic (+36), iterative (+18), linear equations (+10) — are exactly those requiring complex equation-solving, symbolic manipulation, or repeated computation. These are the three failure modes the paper identified in its introduction. The categories where PoT and CoT perform similarly — arithmetic (+2), combinatorics (0), probability (0) — are those where CoT's arithmetic is reliable enough and the reasoning structure, not the computation, is the bottleneck.

The geometry result (PoT 20% vs. CoT 50%) is striking: PoT actually underperforms CoT by 30 points on geometry problems. The paper does not discuss this finding in the main text (it is visible only in the bar chart), but it is consistent with the method's design — geometry problems require spatial reasoning and diagram interpretation that cannot be easily expressed as Python code. This negative result reinforces the paper's scope claim: PoT is not a universal replacement for CoT; it is specifically effective when computation is the bottleneck and ineffective (or harmful) when non-computational reasoning dominates.

Model Backend Ablation

Table 4 compares PoT performance across different language models on GSM8K, SVAMP, and FinQA:

The key patterns: (1) Code-trained models (code-davinci-002) substantially outperform text-trained models of the same scale (text-davinci-002), with a 11.2-point gap on GSM8K — this validates using Codex as the primary model. (2) ChatGPT (gpt-3.5-turbo) outperforms Codex on both datasets (76.3% vs. 71.6% on GSM8K), suggesting that more recent instruction-tuned models can generate effective PoT programs even without code-specific training. (3) Open-source 16B models perform dramatically worse than 175B proprietary models — the best 16B model (CodeGen-mono) achieves only 12.7% on GSM8K vs. Codex's 71.6%, a catastrophic gap. The paper attributes this to "non-sufficient pre-training and model size," suggesting that the capability to generate correct multi-step reasoning programs from natural language questions exhibits a sharp emergence threshold above 16B parameters.

Sensitivity to Exemplar Selection

Figure 5 visualizes PoT's sensitivity to which specific exemplars are chosen for few-shot prompting on GSM8K and FinQA. Three versions (v1, v2, v3) of K-shot demonstrations are randomly sampled from a pool of 20 hand-written exemplars, for K = 2, 4, 6, 8.

The GSM8K results show:

• At K=2, performance ranges from ~0.58 to ~0.65 across the three versions — a 7-point spread.
• At K=4, the range narrows to ~0.67–0.69 (2-point spread).
• At K=6, the range is ~0.67–0.70.
• At K=8, the range is ~0.70–0.73.
• There is a clear upward trend with more shots: K=8 consistently outperforms K=2 by 5–8 points.

For FinQA:

• At K=2, performance ranges from ~0.55 to ~0.62 (7-point spread, similar to GSM8K).
• At K=4, the range is ~0.58–0.68 (broader spread than GSM8K).
• At K=6, the range is ~0.64–0.69.
• At K=8, the range narrows to ~0.62–0.72 (surprisingly, v1 at K=8 drops to 0.62, worse than K=6 for that version).
• The upward trend with more shots is less consistent for FinQA than for GSM8K.

The key finding is that exemplar selection matters substantially, especially at low shot counts, and that increasing shots both improves average performance and reduces variance. The paper's choice of 4–8 shots (depending on dataset complexity) is informed by this analysis — 8 shots provides the most stable and highest performance for diverse datasets.

Ablation Studies and Robustness Checks

Semantic binding (PoT vs. PoT - Binding): Table 6 compares standard PoT against a variant where all variable names are replaced with generic a, b, c (removing the semantic grounding to problem entities). On GSM8K, accuracy drops from 71.6% to 60.2% (−11.4 points). On SVAMP, the drop is smaller: 85.2% to 83.8% (−1.4 points). On FinQA, the drop is intermediate: 64.5% to 61.6% (−2.9 points). The differential impact suggests that semantic binding matters more for complex problems with many quantities to track — GSM8K has diverse multi-step problems, while SVAMP's simpler structure means the model can track quantities with generic names. This is a non-obvious finding: the benefit of descriptive variable names is not cosmetic but functional, scaling with problem complexity.

Multi-step decomposition (PoT vs. PoT - MultiStep): The "PoT - MultiStep" variant prompts the LLM to directly generate the final equation rather than breaking it down into sequential steps. This causes the largest performance drop of any ablation. On GSM8K, accuracy collapses from 71.6% to 45.8% (−25.8 points). On SVAMP, from 85.2% to 81.9% (−3.3 points — smaller because SVAMP problems are inherently simpler). On FinQA, from 64.5% to 58.9% (−5.6 points). The 25.8-point GSM8K drop is the single largest effect size in the paper, demonstrating that the step-by-step structure of the generated programs is responsible for roughly one-third of PoT's total accuracy on challenging problems. This corroborates the CoT literature's finding that externalizing intermediate reasoning states improves LLM performance, and extends it to the code domain.

Backend model comparison (Table 4): As discussed in the quantitative results, the code-trained vs. text-trained comparison (code-davinci-002 at 71.6% vs. text-davinci-002 at 60.4% on GSM8K) validates the choice of Codex. The 16B open-source model results (8.2–12.7% on GSM8K) demonstrate that PoT's effectiveness is strongly scale-dependent — the method does not work with smaller models, suggesting an emergent capability threshold.

Few-shot vs. zero-shot: The difference between few-shot PoT (71.6% on GSM8K) and zero-shot PoT (57.0% on GSM8K, Table 3) represents a 14.6-point gap attributable to the exemplars alone. This is larger than the typical few-shot vs. zero-shot gap for CoT (few-shot CoT at 63.1% vs. zero-shot CoT at 40.5% = 22.6 points), suggesting that PoT is somewhat more robust to the absence of exemplars than CoT — the code-generation instruction plus pre-loaded imports provides enough guidance for the model to produce reasonable programs even without demonstrations.

Self-consistency (SC) as a robustness method: The SC results (Table 2, middle section) serve as an implicit robustness check. The consistent improvement from greedy to SC for both PoT and CoT (e.g., GSM8K PoT: 71.6% → 80.0%; CoT: 63.1% → 78.0%) demonstrates that both methods benefit from sampling diversity and majority voting. The narrowing of the PoT–CoT gap under SC (from 8.5 points greedy to 2.0 points with SC on GSM8K) suggests that part of PoT's greedy advantage comes from reduced variance — PoT's programs, when correct, compute the right answer deterministically, while CoT's arithmetic introduces variance that SC can partially mitigate. However, on financial datasets where CoT's errors are systematic (large-number miscalculations), SC provides minimal help (FinQA CoT-SC: 44.4% vs. CoT greedy: 40.4%, only +4.0 points), while PoT-SC maintains a 23.7-point lead.

PoT as intermediate step (AQuA-specific): The compositional prompting approach (Section 2.3, Figure 8) is itself an ablation of pure PoT for multiple-choice datasets. Without the second CoT stage, PoT cannot output letter options — the program returns a numeric value, and there is no mechanism to map it to A/B/C/D/E. The fact that PoT achieves 54.1% on AQuA (Table 2) with this two-stage approach, compared to CoT's 45.3%, demonstrates that the composition is functional. The paper does not report an ablation of removing the second stage, presumably because pure PoT would score 0% on a multiple-choice dataset (since the output format would not match).

Number of shots: Figure 5 serves as an ablation of shot count. The finding that variance shrinks and average performance increases with more shots (up to 8) validates the paper's choice of 4–8 shots for the main experiments. The diminishing returns at higher shot counts (e.g., GSM8K v1: K=6 at 0.70 vs. K=8 at 0.73 — only a 3-point gain) suggest that 6–8 shots is near the saturation point for this method on these datasets.

The # token suppression in zero-shot: The paper describes suppressing the # token logit by a bias of -2 in the zero-shot setting to prevent the model from generating comment-based reasoning rather than executable code. The bias value of -2 is reported as the result of "preliminary study" (Section 3.1). However, no ablation table is provided showing performance at different bias values or without suppression. This is a missing ablation — the reader cannot assess how critical this trick is or whether the -2 value generalizes beyond the datasets tested.

Linearization format: The paper does not ablate the table linearization strategy (using | as column separator, - for empty cells, \n for row separation). It is possible that alternative linearization strategies would produce different results, but this is not explored.

Critical Assessment

Claim 1: "PoT has an average performance gain over CoT of around 12% across all datasets." This claim (from the abstract) is well-supported by the data in Table 2 and Table 3, but requires careful interpretation of what "around 12%" means. Across the seven few-shot datasets in Table 2, the average gap between Codex PoT and Codex CoT is approximately 12.4 percentage points (calculating from the reported numbers: 8.5 + 8.8 + 8.8 + 8.0 + 24.1 + 19.0 + 7.6 = 84.8 / 7 ≈ 12.1). However, this average is substantially skewed by the two financial datasets with very large gaps (FinQA +24.1, ConvFinQA +19.0). Excluding those, the MWP-only average gap is approximately 8.5 points — still substantial but lower. The paper does not break this down or acknowledge the skew. A more precise statement would be: "PoT improves over CoT by 4–9 points on MWP datasets and 8–24 points on financial QA datasets, with an average across all eight datasets of roughly 12 points."

Claim 2: "By combining PoT with self-consistency decoding, we can achieve extremely strong performance on all the math datasets and financial datasets." Supported. PoT-SC achieves the best known non-GPT-4 results on all MWP datasets (Table 2) and approaches or matches SoTA on financial datasets. However, "extremely strong" is vague — on AQuA, PoT-SC achieves only 58.6%, which is strong relative to contemporaneous methods but leaves 41.4% of questions unsolved. The paper is transparent about this limitation, noting that "PoT still struggles with AQuA dataset with complex algebraic questions with only 58% accuracy" (Section 6).

Claim 3: The three failure modes (arithmetic errors, complex equations, inefficient iteration) are the cause of CoT's underperformance, and PoT specifically addresses these. The breakdown analysis in Figure 6 provides strong evidence for this claim. The problem categories where PoT shows the largest gains (polynomial +40, symbolic +36, iterative +18 points) map directly onto the hypothesized failure modes. The categories where PoT shows minimal or no gain (arithmetic +2, combinatorics 0, probability 0) map onto problems where those failure modes are absent. This differential improvement pattern is precisely what we would expect if the diagnosis is correct. A stronger test would require a larger-scale breakdown analysis across multiple datasets (not just AQuA) and with formal statistical testing, but the single-dataset analysis is nevertheless persuasive because the pattern is so stark.

Claim 4: The separation of reasoning from computation is what matters, not just "using code." The ablation evidence in Table 6 supports this. If the benefit came simply from generating Python code, the "PoT - Binding" and "PoT - MultiStep" variants would perform similarly to full PoT — they still generate executable Python. Instead, both ablations show significant degradation, with MultiStep causing a catastrophic 25.8-point drop on GSM8K. This demonstrates that the structure of the code — multi-step, semantically grounded — is responsible for a large fraction of PoT's gains, consistent with the paper's framing that "thoughtful" coding (not just coding) is the active ingredient.

Genuine weaknesses in the experimental design:

• Single model family for the main claim. The primary results all use Codex (code-davinci-002). While Table 4 tests other models, the numbers are only reported for two datasets (GSM8K, SVAMP) and are not compared against CoT baselines for those models. We cannot assess whether PoT's advantage over CoT is consistent across model families, or whether it is specific to Codex's code-training. The paper would be stronger with a full Table 2 replicated for at least one additional model (e.g., GPT-3.5-turbo, which Table 4 shows performs well).

• No comparison against fine-tuned baselines on the same model. The "Published SoTA" row in Table 2 includes fine-tuned models (e.g., FinQANet for ConvFinQA, Dynamic Prompt Learning for TabMWP, RegHNT for TATQA). These models were specifically trained on these datasets, while PoT uses no training data. The comparison is therefore between a zero-training prompting method and fully supervised methods — impressive that PoT-SC matches or exceeds them, but the comparison is not controlled (different base models, different training regimes). A fair comparison would fine-tune Codex on these datasets and compare against PoT.

• Small test sets for some datasets. AQuA has only 253 test questions. When broken into 8 categories for Figure 6, some categories may contain very few questions (the paper does not report per-category sample sizes). The geometry category where PoT scores 20% vs. CoT's 50% could be based on as few as 5–10 questions — too small to draw reliable conclusions about PoT's geometry weakness. The paper does not report confidence intervals anywhere, which makes it difficult to assess whether apparent differences (especially small ones, like PoT vs. CoT on arithmetic at 88% vs. 86%) are statistically reliable.

• Exemplar tuning on a "small validation set" is underspecified. The paper mentions tuning exemplar selection on a validation set (Section 3.1) but does not describe this set — its size, whether it is a held-out portion of the training data, or how many exemplar configurations were tried. Without this information, it is impossible to assess whether the reported results reflect overfitting to the validation set. The sensitivity analysis in Figure 5 partially addresses this concern by showing that performance is reasonably stable when K ≥ 6, but the analysis uses random sampling from a fixed pool, not the tuned selection process used for the main results.

• No latency or cost analysis. PoT requires executing generated Python code, which adds latency beyond the LLM inference time. For problems using SymPy's symbolic solver, the execution could be non-trivial (though likely still small relative to LLM inference). The paper does not measure or report this overhead. Additionally, PoT requires a Python execution environment, which may not be available in all deployment contexts (browser-based inference, API-only access without code execution capabilities).

• Missing ablations: The paper does not ablate the # token suppression trick (how much does it matter? what is the optimal bias?). It does not ablate the table linearization strategy. It does not ablate the solve_it wrapper vs. using raw SymPy. It does not test whether the improvements hold at lower temperatures for greedy decoding (the main results use temperature 0, but no sweep is reported). It does not evaluate on the MATH dataset (Hendrycks et al., 2021), which would provide a harder test of symbolic reasoning capabilities.

The geometry result as an unexamined negative finding. Figure 6 shows PoT scoring 20% vs. CoT's 50% on geometry problems — a 30-point disadvantage. This is the only category where PoT underperforms CoT, and the paper does not discuss it in the main text. The finding is consistent with PoT's design (geometry requires spatial reasoning that is hard to encode in Python), but the paper misses an opportunity to characterize PoT's failure modes more precisely. Does PoT fail because it cannot parse geometry from text? Because it generates incorrect programs? Because the programs execute correctly but produce wrong answers due to incorrect geometric reasoning expressed in code? An error analysis of the geometry failures would strengthen the paper's claims about PoT's scope boundaries.

Comparison with PaL is incomplete. Table 5 compares PoT and PaL on six datasets, but only for greedy decoding (no self-consistency). PaL-SC results would provide a stronger comparison. Additionally, the paper does not report PaL's performance on the financial datasets, which is where PoT shows its largest gains over CoT — comparing against PaL on FinQA and ConvFinQA would test whether PoT's advantage in those domains comes from the specific prompting design or simply from using code at all.

The self-consistency narrowing-of-gap phenomenon is observed but not explained. On GSM8K, the PoT–CoT gap shrinks from 8.5 points (greedy) to 2.0 points (SC). The paper notes this (Section 3.2: "self-consistency decoding is less impactful for both PoT and CoT" — though this statement is about financial datasets, not the MWP narrowing) but does not analyze why. One hypothesis: SC helps CoT by allowing different arithmetic errors to cancel out through majority voting, partially closing the computation-accuracy gap that PoT addresses architecturally. If this interpretation is correct, it suggests that SC is a partial substitute for PoT's external computation, but only when errors are uncorrelated across samples (which they are not on financial datasets with systematic large-number errors). This would be a useful insight, but the paper does not develop it.

The zero-shot results use a different CoT baseline than the few-shot results. Zero-shot PoT is compared against zero-shot CoT from Kojima et al. (2022), which uses text-davinci-002 (GPT-3). But zero-shot PoT uses code-davinci-002 (Codex) — the models are different. This makes the comparison partially confounded: part of the 16.5-point gap on GSM8K could be due to Codex being a stronger model than GPT-3, not due to PoT being a better method than CoT. The paper does not report zero-shot CoT on Codex, which would be the fair comparison. (The zero-shot PoT prompt includes pre-loaded imports and a code-generation instruction that would be nonsensical for CoT, but a comparable zero-shot CoT prompt for Codex could be constructed.)

Summary of experimental strength and limitations. The experiments convincingly demonstrate that PoT outperforms CoT on numerical reasoning tasks, with the largest gains on problems requiring complex computation. The ablation studies provide good evidence that both multi-step decomposition and semantic binding contribute to PoT's effectiveness. The breakdown analysis confirms that PoT helps precisely where its design says it should. However, the experimental scope is narrower than it appears: all main results are on a single model family (Codex), several baselines use different models, the test sets for some datasets are small, statistical significance is never assessed, and several important ablations and analyses are missing. The paper's claims are generally well-supported by the data presented, but the degree of support would be stronger with multi-model replication, formal significance testing, and a more thorough analysis of failure modes (especially the geometry result that contradicts the main narrative).

6. Limitations and Trade-offs

6.1 The Method Cannot Handle Problems Outside the Base Model's Computational Expressiveness

The assumption or constraint. PoT assumes that the reasoning required to solve a problem can be faithfully expressed as an executable Python program using only the whitelisted modules (math, sympy, and the custom solve_it wrapper). This assumption breaks down for problem types requiring reasoning modalities that do not translate naturally into imperative code. The paper explicitly acknowledges this scope boundary in Section 5:

"For semantic reasoning tasks like commonsense reasoning (StrategyQA), we conjecture that PoT is not the best option. In contrast, CoT can solve more broader reasoning tasks."

The consequence. When the reasoning required cannot be encoded in Python — spatial reasoning, geometric intuition, commonsense inference, or natural language understanding — PoT will either fail to generate a correct program or, worse, generate a syntactically valid program that executes successfully but encodes wrong reasoning, producing a confident and precise but incorrect answer. This failure mode is particularly dangerous because the external interpreter provides no safeguard against reasoning errors: the computation will be executed perfectly, but on a flawed logical foundation.

What evidence exists in the paper. The breakdown analysis in Figure 6 provides direct evidence of this limitation. On geometry problems in AQuA, PoT achieves only ~20% accuracy compared to CoT's ~50% — a 30-point disadvantage and the only category where PoT underperforms CoT. The paper does not analyze these geometry failures further, but the result is consistent with the hypothesis that spatial and geometric reasoning resists expression as Python code in ways that algebraic and arithmetic reasoning do not. Similarly, on probability and combinatorics problems, PoT and CoT tie at 40% — the reasoning bottleneck is not computational but logical, and switching to code provides no benefit because the model struggles equally with the underlying reasoning in both media.

The paper does not provide error analysis on the geometry failures, leaving it unclear whether PoT fails because (a) the model cannot parse geometric constraints from text into code, (b) the generated programs encode incorrect geometric reasoning, or (c) the generated programs are syntactically valid but conceptually wrong (e.g., applying an algebraic formula to a problem that requires spatial visualization). All three failure modes are plausible and have different implications for how PoT should be extended or when it should be avoided.

Mitigation status. The paper proposes partial mitigation through the compositional "PoT as intermediate step" architecture (Section 2.3, Figure 8), where PoT handles the computational sub-problem and CoT handles the residual textual reasoning. However, this only helps when the problem can be decomposed into a computational part and a reasoning part. For geometry problems where the core reasoning itself is spatial, no amount of compositional chaining will help — the computation is not the bottleneck. The paper acknowledges this limitation but does not propose a solution, leaving it as a fundamental scope boundary: PoT is the right tool when computation is the bottleneck, but it offers no advantage — and can be actively harmful — when reasoning structure is the bottleneck. Practitioners need a way to determine which problem type they face before choosing PoT vs. CoT, and the paper provides no automated mechanism for making this determination.

6.2 The Code Execution Requirement Introduces Security, Infrastructure, and Latency Costs That Are Not Accounted For

The assumption or constraint. PoT requires executing generated Python code in a live interpreter. The paper assumes this execution environment is available, sandboxed, and fast enough to be negligible relative to LLM inference. The authors acknowledge the security concern explicitly in Section 6 (Limitations):

"PoT would require execution of 'generated code' from LLMs, which could contain certain dangerous or risky code snippets like 'import os; os.rmdir()', etc. We have blocked the LLM from importing any additional modules and restrict it to using the pre-defined modules."

They further note that "such brutal-force blocking works reasonable for math QA, however, for other unknown symbolic tasks, it might hurt PoT's generalization."

The consequence. The deployment overhead is multi-dimensional and unquantified in the paper:

• Security risk. Module whitelisting prevents known dangerous imports (os, subprocess, requests, etc.), but this is a blacklist-based defense against an adversary (the LLM) that can generate arbitrary text. The paper does not discuss whether Codex can generate code that exploits vulnerabilities in SymPy itself, or whether infinite loops (while True), memory exhaustion (large data structures), or excessive compute (combinatorial explosion in symbolic solving) are handled. A single while loop with a non-terminating condition in a generated program could hang the execution environment indefinitely.
• Infrastructure requirement. PoT cannot run in contexts where only LLM inference is available — browser-based demos, API-only deployments without server-side code execution, or environments where Python is not installed. This limits PoT's applicability compared to CoT, which requires only text generation.
• Latency overhead. Every PoT inference requires an additional step beyond LLM generation: the generated code must be parsed, executed in a Python subprocess, and its output captured. For problems using SymPy's symbolic solver (e.g., solving cubic equations or systems of equations), execution time could be non-trivial — SymPy's solve can be slow for complex expressions. The paper does not measure this overhead. For the self-consistency experiments where 40 samples are generated per question, the execution cost is multiplied by 40.
• Module whitelisting limits generalization. The paper's security model restricts imports to math, sympy, and solve_it. This works for the math and finance domains evaluated, but for problems requiring domain-specific libraries (statistical modeling with scipy, unit conversion with pint, date/time manipulation with datetime), the whitelist would need to be expanded — increasing the attack surface.

What evidence exists in the paper. None. The paper provides no measurements of execution time, no analysis of program runtime distributions, no classification of execution failures (syntax errors, runtime exceptions, infinite loops, wrong answers from correct-looking but incorrect programs), and no evaluation of how often the generated code is actually executable. The security discussion is confined to a brief acknowledgment in the Limitations section with no empirical data. This is a significant gap for a method that adds an entirely new failure mode (runtime errors) that CoT does not have.

Mitigation status. The security mitigation (module whitelisting) is described but not evaluated — no adversarial testing is reported, no analysis of what fraction of generated programs attempt to import disallowed modules, no discussion of timeout mechanisms or resource limits. The latency and infrastructure costs are not mitigated or even measured. The paper does not discuss whether the overhead is negligible relative to LLM inference (which, for Codex at 175B parameters, is already substantial), or whether it could become the dominant cost for problems involving heavy symbolic computation. Future work should quantify these costs and develop safety mechanisms (timeouts, memory limits, static analysis of generated code before execution) that go beyond simple import blocking.

6.3 All Primary Results Are on a Single Model Family with No Replication Across Architectures

The assumption or constraint. The paper's headline few-shot and few-shot+SC results (Table 2) use a single model: OpenAI Codex (code-davinci-002, 175B parameters). The paper treats this model as representative, stating in Section 3.1 that "we mainly use the OpenAI Codex (code-davinci-002) API for our experiments." All comparisons between PoT and CoT in Table 2 — the evidence for the paper's central claim of ~12% average improvement — use this single model.

The consequence. The paper cannot distinguish between three possibilities: (1) PoT is a generally effective prompting strategy that works across model families, (2) PoT's effectiveness is specific to Codex's code-training — a text-trained model of equal scale might show much smaller gains or none at all, or (3) PoT's effectiveness is specific to the 175B scale — smaller models might not benefit. The distinction matters enormously for practitioners deciding whether to adopt PoT. If PoT only works with code-trained models, organizations using GPT-3, Claude, or open-source text models cannot use it. If PoT only works at 175B+ scale, it is inaccessible to all but the largest compute budgets.

What evidence exists in the paper. The backend ablation in Table 4 provides partial evidence on a narrower scope. It reports PoT performance on GSM8K and SVAMP for seven models, but only for PoT — no CoT baselines are reported for these models, making it impossible to compute the PoT–CoT gap. We learn that text-davinci-002 PoT achieves 60.4% on GSM8K vs. Codex PoT's 71.6% — an 11.2-point gap attributable to model differences. But we do not know whether text-davinci-002 CoT achieves 63.1% (like Codex CoT) or 46.9% (like GPT-3 CoT) or something else entirely. Without this, we cannot determine whether PoT's advantage is robust to the choice of backbone.

For the 16B open-source models (CodeGen-multi, CodeGen-mono, CodeT5+, Xgen), PoT performance collapses to 8.2–12.7% on GSM8K — far below either Codex PoT (71.6%) or Codex CoT (63.1%). This strongly suggests that PoT does not work with smaller models, but again, without CoT baselines for these models, we cannot determine whether the failure is PoT-specific (these models cannot generate correct reasoning code) or reflects a general reasoning deficit (these models also fail at CoT). The paper conjectures the gap is "attributed to non-sufficient pre-training and model size," but this is not tested.

Mitigation status. The paper provides no mitigation. It does not report CoT baselines for the additional models in Table 4. It does not replicate the full Table 2 on any model other than Codex. The closest thing to replication is the PaL comparison in Table 5, which uses different models (no model specified — presumably the same Codex) and only for greedy decoding on six datasets. The paper acknowledges that the capability to generate code is model-dependent (Section 5: "code-trained models perform substantially better"), but does not extend this to the core PoT vs. CoT comparison. A proper replication would require at minimum: reporting PoT and CoT on text-davinci-002 and gpt-3.5-turbo for all eight datasets, and ideally on at least one open-source model large enough to have non-trivial reasoning capability (e.g., a 70B+ model).

6.4 The Financial Dataset Gains Are Partially Attributable to an Evaluation Asymmetry That Favors PoT

The assumption or constraint. For the FinQA dataset, the paper applies different evaluation standards to PoT and CoT. The metrics section (Section 3.1) states:

"For FinQA, we relax the evaluation for CoT because LLMs cannot perform the computation precisely (especially with high-precision floats and large numbers), so we adopt 'math.isclose' with relative tolerance of 0.001 to compare answers."

This means CoT's answers are accepted as correct if they fall within 0.1% of the ground truth — a tolerance designed to forgive CoT's floating-point imprecision and large-number miscalculations. PoT, by contrast, produces answers from a Python interpreter with exact precision in most cases (integer arithmetic, SymPy symbolic results) or standard IEEE 754 floating-point precision (which is more precise than the tolerance grants but not exact for all decimal values).

The consequence. The evaluation asymmetry makes the PoT–CoT comparison on FinQA conservative — PoT's reported advantage (+24.1 points: 64.5% vs. 40.4%) would be even larger under strict exact match, because some CoT answers that are marked correct under the relaxed tolerance would be marked incorrect under strict matching, while PoT's answers (being interpreter-computed) would be largely unaffected. However, the asymmetry also creates a measurement validity concern: the FinQA results for CoT are not comparable to FinQA results reported in other papers that use strict exact match, and the 40.4% number does not represent CoT's true accuracy under the standard evaluation protocol for that dataset.

Furthermore, the tolerance of 0.001 is not justified by the problem semantics. In financial contexts, an error of 0.1% on a calculation involving millions of dollars is an error of thousands of dollars — potentially unacceptable in practice. By using a relative tolerance, the paper implicitly accepts that CoT's answers are "close enough," which may not align with the requirements of actual financial applications.

What evidence exists in the paper. The evaluation asymmetry is disclosed in Section 3.1, which is good — the paper is transparent about the decision. However, no sensitivity analysis is provided: what would CoT's accuracy be under strict exact match? Under a tighter tolerance (0.0001)? Under an absolute tolerance rather than relative? The ~24-point gap on FinQA is the single largest PoT–CoT difference across all datasets, and it is the primary driver of the "average 12% gain" claim in the abstract. Understanding how much of this gap reflects genuine reasoning improvement versus measurement artifact is important for interpreting the headline result.

The ConvFinQA and TATQA datasets use their official evaluation scripts (which presumably apply standard exact-match or near-exact-match criteria), so the evaluation asymmetry is limited to FinQA. But FinQA is also the dataset with the largest absolute gap — without it, the average PoT–CoT gain across the remaining six datasets is substantially lower (~8.5 points for MWP plus ~7–19 points for the other financial datasets).

Mitigation status. The paper does not mitigate this issue beyond disclosure. It does not report CoT accuracy under strict matching, does not conduct an ablation to determine what fraction of CoT's errors on FinQA are due to arithmetic imprecision (which PoT fixes) versus reasoning failures (which PoT does not fix), and does not discuss whether the 0.001 tolerance was chosen through validation-set tuning (which would constitute data leakage) or was a fixed a priori choice. A more rigorous evaluation would report both strict and relaxed accuracy for both methods, allowing readers to assess the impact of the tolerance choice on the reported gap.

6.5 PoT Performance Is Highly Sensitive to Model Scale, with a Sharp Drop Below 175B Parameters That Limits Practical Deployability

The assumption or constraint. The paper implicitly assumes access to a 175B-parameter code-trained model. Codex (code-davinci-002) is a proprietary model accessed via API — it cannot be run locally, fine-tuned, or deployed in offline/secure environments. For practitioners who need on-device inference, private deployment, or cost-effective serving at scale, smaller open-source models would be necessary.

The consequence. Table 4 reveals that PoT performance degrades catastrophically at smaller model scales. On GSM8K:

• Codex (175B): 71.6%
• CodeGen-mono (16B): 12.7%
• CodeGen-multi (16B): 8.2%
• CodeT5+ (16B): 12.5%
• Xgen (7B): 11.0%

The drop from 175B to 16B is approximately 60 percentage points — PoT essentially stops working entirely. The 16B models perform worse than even the "Direct" (no reasoning) baseline for Codex on GSM8K (19.7%, from Table 2), meaning that asking these models to generate reasoning programs produces worse results than simply asking them to output the answer directly. This is a complete method failure, not a gradual degradation.

The paper does not report CoT baselines for these smaller models, so we cannot determine whether the failure is PoT-specific (small models cannot generate correct code) or reflects a general reasoning collapse below 175B. But regardless of the cause, the practical implication is clear: PoT as described in this paper cannot be used with models smaller than ~175B parameters. This eliminates the possibility of using PoT with the vast majority of open-source models available at the time of writing (LLaMA-2 70B, Mistral 7B, CodeLlama 34B, etc.) and makes PoT dependent on proprietary API access to very large models.

What evidence exists in the paper. Table 4 provides the evidence for the scale sensitivity. The paper interprets these results by suggesting "a huge gap could be attributed to non-sufficient pre-training and model size" (Section 3.3). However, the paper does not systematically investigate where the capability threshold lies — is the threshold at 50B? 100B? 150B? — because no intermediate-scale models are tested. The paper also does not test whether fine-tuning a smaller model specifically for program-of-thought generation (using the exemplar-writing approach described in Section 3.1, but as training data rather than prompts) could recover some of the lost performance.

The zero-shot results in Table 3 also hint at a model dependency: zero-shot CoT baselines are from Kojima et al. (2022) using text-davinci-002 (GPT-3), while zero-shot PoT uses code-davinci-002 (Codex). The models differ, so part of the 16.5-point gap on GSM8K could be a model effect rather than a method effect — but again, without CoT baselines on Codex, this cannot be disentangled.

Mitigation status. The paper does not attempt to mitigate the scale dependency. It does not propose model distillation, fine-tuning recipes, or prompt simplifications that might make PoT viable at smaller scales. The discussion of open-source models (Section 3.3) simply notes their poor performance without suggesting paths forward. For practitioners who cannot access 175B-scale code-trained models, the paper offers no actionable guidance. This is a significant omission, given that much of the practical interest in prompting methods comes from their applicability to a wide range of models — if PoT only works with one specific proprietary model at one specific scale, its practical impact is substantially narrower than the paper suggests.

6.6 The Prompt Engineering Protocol Is Underspecified and Likely Overfit to the Specific Datasets and Exemplars Tested

The assumption or constraint. PoT's performance depends critically on the quality and selection of few-shot exemplars. The paper's protocol for creating these exemplars is labor-intensive and underspecified in ways that affect both reproducibility and the strength of the claims. The authors describe their process as:

"We generally write prompts for 10-20 examples and then tune the exemplar selection on a small validation set to choose the best 4-8 shots for the full set evaluation." (Section 3.1)

Several aspects of this protocol are unclear: How large is the "small validation set"? Is it a held-out subset of the training data, or is it drawn from the test distribution? How many exemplar configurations were tried before selecting the best? Were exemplars iteratively refined based on validation performance (which would constitute a form of training on the validation set)?

The consequence. The reported results may not reflect what a new user would achieve when applying PoT to a different dataset or domain. The exemplar writing and selection process involves human judgment, domain knowledge, and iterative refinement that is difficult to standardize or automate. This creates two related problems:

1. Reproducibility gap. A practitioner attempting to replicate the paper's GSM8K results must write their own exemplars, which may differ from the authors' (the Appendix provides exemplars for GSM8K and AQuA, but not for the other six datasets). The sensitivity analysis in Figure 5 shows that at K=2 shots, performance can vary by 7 percentage points depending on which exemplars are chosen — and this is variation within the authors' own exemplar pool. Variation across different authors writing different exemplars for the same dataset is likely larger.

2. Overfitting risk. If the authors iteratively refined exemplar selection based on validation set performance, the selected exemplars may be implicitly tuned to the specific quirks of the validation set (and, by extension, the test set if the validation set is not fully independent). The two-fold cross-validation mentioned for the compute-optimal experiments in the referenced paper summary is not applied here — PoT's exemplar selection uses a simpler "tune on validation, evaluate on test" protocol that is more vulnerable to overfitting, especially given the small test sets for some datasets (AQuA: 253 questions, ConvFinQA: 421 questions).

What evidence exists in the paper. Figure 5 provides direct evidence of exemplar sensitivity. The three random draws (v1, v2, v3) from the same 20-exemplar pool produce different performance at the same shot count, with variance as large as 7 points at K=2. The variance shrinks with more shots (to ~3 points at K=8), confirming that the paper's choice of 4–8 shots is in the regime where exemplar sensitivity is reduced but not eliminated. The paper does not provide a systematic analysis of which exemplar properties drive performance — is it diversity of problem types covered? Similarity to test questions? Code style? Length? — leaving practitioners with no guidance on how to construct good exemplars for new domains.

The paper also does not report how much performance changed during the "tuning" phase — were the initial exemplar sets substantially worse than the final selected sets? Did tuning on the validation set produce large gains that might not transfer to the test set? Without these details, the reader cannot assess whether the exemplar selection protocol introduces a material overfitting risk.

Mitigation status. The paper partially mitigates this limitation through the sensitivity analysis (Figure 5), which demonstrates that PoT performance is reasonably stable when K ≥ 6. The zero-shot results provide a stronger mitigation: zero-shot PoT requires no exemplars at all and still substantially outperforms zero-shot CoT (Table 3), showing that the method's effectiveness is not entirely dependent on careful exemplar engineering. However, the zero-shot results still trail few-shot results by a wide margin (57.0% vs. 71.6% on GSM8K), meaning that exemplar engineering remains necessary to achieve the headline numbers. The paper does not propose methods for automating exemplar selection (e.g., embedding-based retrieval of similar training examples, or using the LLM itself to generate exemplars), which would reduce the human effort and subjectivity involved.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper is best understood not as a paradigm shift but as a sharp diagnostic reframing that draws a clean boundary through a previously muddled problem — numerical reasoning with LLMs — and, in doing so, opens a specific, productive design space that the field has since converged on. The core move is deceptively simple: identify which component of CoT's reasoning chain is failing (computation, not reasoning structure), then replace only that component with a reliable external tool, preserving the multi-step reasoning pattern that makes CoT effective.

Before PoT, the dominant framing for numerical reasoning was monolithic: the LLM does everything — reads the question, plans the steps, executes the arithmetic, and outputs the answer — within a single autoregressive generation. CoT (Wei et al., 2022) improved this by eliciting step-by-step reasoning, but it preserved the architecture where the same model performs the computation. The "CoT + calculator" variant (Wei et al., 2022) half-acknowledged the problem by post-processing CoT's generated arithmetic, but it kept the model's generation format unchanged and relied on brittle regex extraction — a patch, not a redesign.

PoT's reframing identifies the conflation of reasoning with computation as the architectural error, not an incidental weakness. The three failure modes in Section 1 — arithmetic calculation errors, inability to solve complex equations, inefficient iteration — are not presented as limitations that better models or more training data will fix. They are presented as structural mismatches between what language models are (probabilistic next-token predictors) and what computation requires (deterministic algorithmic execution). This distinction implies that scaling alone — more parameters, more training FLOPs — will not make LLMs into reliable calculators, because the failure is categorical, not a matter of degree.

The evidence that this reframing is causal rather than merely descriptive comes from the differential improvement pattern the paper documents. If PoT were simply a better prompting template in general, we would expect near-uniform gains across all problem types. Instead, the breakdown analysis in Figure 6 shows gains are sharply concentrated: +40 points on polynomial equations, +36 on symbolic problems, +18 on iterative problems — exactly the categories where the three failure modes apply — versus +2 points on arithmetic, 0 on combinatorics, 0 on probability, and −30 on geometry. The method helps precisely where its diagnostic says the bottleneck lies and does not help (or hurts) elsewhere. This specificity is what distinguishes a genuine insight about mechanism from a generic accuracy improvement.

The paper's broader impact is that it anticipates and empirically validates the tool-use paradigm that has since become a major research direction. When Section 4.4 connects PoT to Toolformer (Schick et al., 2023) and ART (Paranjape et al., 2023), it is not merely citing related work — it is positioning PoT as an early demonstration of a principle: LLMs should be reasoning coordinators that delegate specialized computation to external tools, not monolithic systems that internalize all capabilities. The compositional "PoT as intermediate step" architecture (Section 2.3, Figure 8) is a concrete instantiation of this principle: PoT handles the computational sub-problem, CoT handles residual textual reasoning, and the pipeline architecture allows each component to operate within its competence zone. This modular, delegating design is precisely the pattern that later agent-based systems would adopt, and PoT provides early empirical evidence that it works — not in theory, but on real benchmarks with measurable accuracy gains.

The paper also resolves a latent tension in the CoT literature. CoT was known to be effective for reasoning, but its arithmetic errors were an acknowledged weakness. The "CoT + calculator" fix was known to provide only marginal gains (Table 2: +2.3 points on GSM8K for Codex). PoT explains why the calculator fix was insufficient: extracting arithmetic from free-form text is brittle, and errors in the reasoning itself (not just the final calculation) cannot be fixed post-hoc. By generating structured code from the start, PoT makes the reasoning–computation boundary unambiguous and machine-readable, eliminating the extraction problem entirely. This insight — that the interface between reasoning and tools matters as much as the tools themselves — has implications beyond numerical reasoning, extending to any setting where LLMs generate structured outputs meant for downstream processing (SQL queries, API calls, planning formalisms).

Finally, the paper shifts the research gravity of numerical reasoning with LLMs. Before PoT, the primary axis of improvement was "better prompting" — more elaborate chains, better exemplars, self-consistency sampling. PoT demonstrates that the more productive axis is architectural: redesigning the interface between the LLM and its computational environment. The 25.8-point gain from multi-step code over single-equation generation (Table 6, "PoT - MultiStep" ablation on GSM8K) is larger than the entire gain from self-consistency sampling (8.5 points) or from switching from GPT-3 to Codex (~17 points on GSM8K CoT). The implication is that how you structure the LLM's interaction with external computation matters more than which model you use or how you sample from it — a finding that redirects research effort toward interface design rather than pure model scaling.

Follow-Up Research This Work Enables

Automated difficulty estimation for routing between PoT and CoT. The breakdown analysis in Figure 6 reveals that PoT is actively harmful on geometry problems (20% vs. CoT's 50%) while dramatically helpful on polynomial and symbolic problems. This creates a practical routing problem: given a new question, should the system use PoT or CoT? The paper proposes no automated mechanism for this decision — the exemplars are hand-written per dataset, and the method choice is fixed for all questions. A concrete follow-up would train a lightweight classifier (fine-tuned on problem text embeddings from a model like Sentence-BERT) to predict which method (PoT or CoT) will succeed on a given question, using the AQuA breakdown categories as training labels. The classifier could be evaluated by comparing a routing system (PoT for predicted PoT-favorable problems, CoT otherwise) against pure PoT and pure CoT on held-out math datasets. If routing succeeds, the system could outperform either method alone — the geometry failures would be caught and routed to CoT, preserving PoT's gains on symbolic problems. The experiment would need at minimum the MATH dataset (Hendrycks et al., 2021), which has both problem-type labels and a wide difficulty range, plus a multi-model replication to test whether the routing boundary is model-specific or general.

Systematic error analysis of PoT program failures. The paper reports aggregate accuracy but provides no breakdown of how PoT fails when it fails. A generated program can fail in at least four distinct ways: (a) syntax errors (the code does not parse), (b) runtime exceptions (division by zero, wrong variable type, SymPy timeout), (c) execution succeeds but produces a wrong answer due to incorrect reasoning encoded in the code (value grounding errors or logic errors, per the paper's TAT-QA error analysis in Section 3.3), or (d) execution succeeds and produces the right answer but the extraction of the ans variable fails. The TAT-QA error analysis hand-classifies 198 failures into 47% value grounding errors, 33% logic errors, 15% both, and 5% false positives — but this is for one dataset only and does not distinguish between code-that-doesn't-run and code-that-runs-but-is-wrong. A thorough error taxonomy across all eight datasets, distinguishing these failure modes and measuring their relative frequency, would provide a roadmap for targeted improvements: if syntax errors dominate, better code-trained models or syntax-checking before execution would help; if grounding errors dominate, better table linearization or entity linking is needed; if logic errors dominate, the reasoning capability itself is the bottleneck and PoT cannot help. The paper's Table 6 ablation quantifying the role of semantic binding and multi-step structure provides a template — similar ablation-style analysis of failure modes would shift PoT from a "works on average" method to a "here's exactly where and why it breaks" method.

PoT with a verifier for self-consistency at the program level. The paper uses self-consistency (SC) at the output level: sample 40 programs, execute them, and take the majority answer. But this discards information: if 38 of 40 programs are syntactically valid but produce 15 different answers, and 2 programs are syntactically valid and produce the ground-truth answer, majority voting picks the wrong result. A stronger approach would apply a verifier (analogous to the process reward models used in the companion paper on test-time compute) at the program level: score each generated program for correctness before execution, possibly by checking intermediate variable values against problem constraints, or by training a learned verifier on (program, correctness) pairs. Since the Python interpreter provides deterministic execution traces, a verifier could be trained on features like: presence of certain operations, consistency of variable assignments, agreement with numerical heuristics (e.g., total price should be positive), and structural properties of the code (loop bounds, function calls). This is a natural combination with the PRM search framework from the test-time compute paper: treat PoT program generation as a search problem where the verifier scores partial programs, enabling beam search over program prefixes rather than blind parallel sampling. The feasibility is demonstrated by the fact that PoT programs are structured and constrained (whitelisted modules, fixed output variable ans) — a much more tractable search space than natural language reasoning chains.

Scaling laws for program-of-thought generation. The paper's backend ablation (Table 4) reveals a sharp capability threshold: Codex (175B) achieves 71.6% on GSM8K PoT, while 16B code-trained models achieve 8–12% — a catastrophic collapse. But the paper tests only two scale points (175B and 16B) with only two datasets. A scaling law study — measuring PoT accuracy for models at, say, 1B, 7B, 13B, 34B, 70B, and 175B parameters, with both code-trained and text-trained variants, on a range of math datasets — would characterize where the capability emerges and whether it follows a predictable functional form. The key question is whether the emergence is smooth (log-linear in parameters) or sharp (step-function at some threshold), and whether that threshold differs by problem complexity. This matters because if PoT's capability follows a smooth scaling law, we can predict when smaller open-source models will become viable; if it is a sharp threshold, we know the capability requires a specific scale that smaller models may never reach. The study would need the kind of FLOPs-matched comparison framework introduced in the test-time compute paper (Section 7 of that work) to distinguish model-scale effects from the inference-compute budget.

Cross-domain generalization: PoT for code generation and formal theorem proving. The paper restricts evaluation to math word problems and financial QA — domains where the answer is a number or short phrase. But the core mechanism — express reasoning as executable code, delegate computation to an interpreter — generalizes to any domain where reasoning can be expressed in a formal language with deterministic execution. Two natural extensions are: (1) Code generation itself: given a natural language specification, have the LLM first generate a "program of thoughts" that reasons about the algorithmic approach (pseudocode, invariants, edge cases) and then generates the final implementation, with a test-suite executor playing the role of the Python interpreter. This is a recursive application of PoT: the reasoning about code is expressed as structured comments or assertions, and the actual code generation benefits from the intermediate reasoning. (2) Formal theorem proving: given a mathematical statement, have the LLM generate a proof sketch as structured reasoning steps, with an SMT solver or interactive theorem prover (Lean, Coq) playing the role of SymPy — verifying each reasoning step rather than executing arithmetic. The paper's compositional PoT-as-intermediate-step architecture (Figure 8) is directly applicable: PoT handles the formal verification step, CoT handles the natural-language reasoning about which lemma to apply next. The experiment would require a dataset like MiniF2F or LeanDojo, with evaluation metrics being proof success rate, and the comparison baseline would be pure CoT-style proof generation without formal verification of intermediate steps.

Toward zero-shot PoT without # suppression brittleness. The zero-shot PoT results (Table 3) demonstrate that PoT works without exemplars, but the method relies on a fragile trick: suppressing the # token logit by a bias of −2 to prevent the model from generating comment-based reasoning rather than executable code. The paper does not ablate this trick. A concrete study would sweep bias values (−1, −2, −3, −5, −10) on multiple datasets to determine whether −2 is optimal, how sensitive the method is to this hyperparameter, and whether the optimal value transfers across models and datasets. More importantly, the study would explore alternatives to logit suppression: (a) post-processing: detect whether the generated output is comments-only and re-prompt with stronger instructions; (b) instruction engineering: modifying the zero-shot prompt to more strongly discourage comment-only output without logit manipulation (e.g., "Write executable Python code, not comments"); (c) few-shot fallback: if the zero-shot output is non-executable, automatically append one exemplar and retry. The evaluation metric would be not only accuracy but the executability rate — what fraction of generated programs parse and run without errors, independent of answer correctness. A high executability rate with low correctness indicates reasoning failures; a high correctness rate with low executability (the current zero-shot failure mode) indicates a format-compliance problem that should be fixable through better prompting rather than fragile logit manipulation.

Practical Applications and Downstream Use Cases

Financial document QA with guaranteed arithmetic precision. The paper's strongest results are on the financial datasets: PoT achieves 64.5% on FinQA vs. CoT's 40.4% (+24.1 points) and 64.6% on ConvFinQA vs. 45.6% (+19.0 points). These gains are attributed to PoT's elimination of large-number arithmetic errors — in financial documents, revenues and costs are often in the millions or billions, and CoT's probabilistic arithmetic produces errors that are both common and severe (a 1% error on a $100M figure is a$1M mistake). A deployment scenario with immediate practical value is automated financial report analysis: an LLM reads SEC filings, earnings reports, or financial statements expressed as tables and text, and answers numerical questions with guaranteed arithmetic precision. The benefit is not just accuracy but trustworthiness: when the Python interpreter computes the answer, a human auditor can verify the program logic (possibly even executing it themselves) rather than having to trust the LLM's arithmetic. This shifts the verification burden from "did the model hallucinate this number?" to "does the reasoning in this program correctly model the financial relationship?" — a much more tractable question. FinQA's 1,147-question test set provides a concrete benchmark for this use case, and the paper's 64.5% accuracy (without self-consistency) represents a lower bound on what a deployed system could achieve, since additional improvements from better exemplars, larger models (GPT-4 PoT: 74.0%), or self-consistency (PoT-SC: 68.1%) are available.

Educational math tutoring with step-level feedback. The PoT paradigm naturally produces auditable reasoning traces: a generated program is not just a black-box answer generator but a human-readable sequence of reasoning steps with explicit variable assignments. This makes PoT suitable for educational applications where the system needs to explain its reasoning to a student or where a teacher needs to verify the system's problem-solving approach. The step-by-step structure (e.g., total_eggs = 16; eaten_eggs = 3; ...) provides natural breakpoints for feedback: if a student's answer differs from the system's, the intermediate variable values can be compared to identify which step diverges. This is a significant advantage over CoT, where the reasoning is embedded in prose and comparisons require natural language understanding. The paper's GSM8K results (71.6% greedy, 80.0% with SC) demonstrate that PoT can solve the majority of grade-school math problems, and the ablation showing that multi-step structure is responsible for ~26 points of accuracy (Table 6) confirms that the programs are genuinely step-by-step rather than shortcuts to the answer. A tutoring application could use PoT to generate a model solution, then ask the student to identify where their own reasoning differed — turning math problem-solving from an answer-checking exercise into a reasoning-comparison exercise.

Low-latency batch inference for math-heavy document processing. The paper's zero-shot PoT results (Table 3) are particularly relevant for batch processing scenarios where per-dataset exemplar engineering is impractical. Zero-shot PoT achieves 57.0% on GSM8K and 70.8% on SVAMP without any human-written exemplars — using only a fixed instruction and pre-loaded imports. For an organization processing thousands of math problems from diverse sources (textbooks, online forums, standardized tests), writing few-shot exemplars for each problem type is infeasible. Zero-shot PoT provides a single, reusable prompt that works across problem types with reasonable accuracy, and the Python interpreter guarantees that arithmetic in correctly-reasoned solutions is exact. The batch-processing benefit comes from the interpreter: unlike CoT, where arithmetic errors require re-sampling or self-consistency to mitigate (multiplying inference cost by 40× for the SC results), PoT's computation is deterministic once the program is generated, so a single greedy decoding pass is sufficient for problems where the reasoning in the generated code is correct. This means that at equal accuracy (e.g., zero-shot PoT at 57.0% vs. zero-shot CoT at 40.5% on GSM8K), PoT requires fewer samples and less total inference compute — a cost advantage that compounds in high-volume batch settings.

Tool-augmented LLM APIs with sandboxed code execution. The paper's architectural choice — Python interpreter as an external tool, whitelisted modules for security — provides a template for LLM API providers (OpenAI, Anthropic, Google) that want to offer code-execution capabilities as part of their inference stack. The paper demonstrates that: (a) LLMs can reliably generate executable reasoning programs when prompted appropriately, (b) the generated programs are simple enough that a whitelist of math and sympy covers the vast majority of mathematical use cases, and (c) module-import blocking prevents the most obvious security vulnerabilities. An API offering could expose a /execute endpoint that takes an LLM-generated program, runs it in a sandboxed environment, and returns the result — effectively productizing the PoT pipeline. The financial dataset results (24-point and 19-point gains) provide the business case: for numerical QA, the code-execution capability is not a marginal improvement but a requirement for acceptable accuracy. The limitation analysis (Section 6) points to what the sandbox needs: timeouts for infinite loops, memory limits, and possibly static analysis to reject programs with while True or unbounded recursion before execution. The paper does not provide these guardrails, but it defines the minimum viable product clearly enough that an engineering team could implement it directly from the paper's description (Python 3.8, SymPy, blacklist certain imports, capture ans variable, return result).