MATH: Measuring Mathematical Problem Solving with the MATH Dataset

🎯 Pitch

Scaling up Transformers has revolutionized language tasks, but this paper reveals that even a 175B-parameter model scores only 6.9% on high-school competition math, with extrapolated trends demanding an absurd 10³⁵ parameters to hit 40%—a clear boundary where brute-force scaling fundamentally fails and algorithmic innovation becomes non-negotiable.

1. Executive Summary

This paper introduces MATH, a dataset of 12,500 challenging competition mathematics problems designed to benchmark the mathematical problem-solving capabilities of machine learning models, along with AMPS, a large auxiliary pretraining corpus. The authors evaluate large Transformer models—including GPT-2 variants from 0.1B to 1.5B parameters and GPT-3 up to 175B parameters—on MATH and find that accuracy remains strikingly low, reaching only 6.9% for the best fine-tuned model, establishing that simply scaling model size yields diminishing returns: a 15× increase in parameters provides only a 28% relative improvement (from 5.4% to 6.9% accuracy). The paper further shows that generating step-by-step solutions as scratch space actually degrades performance relative to directly outputting the final answer, yet training on full solutions improves accuracy by approximately 10% relative, and conditioning models on partial ground-truth solutions during inference improves performance substantially—with models reaching roughly 40% accuracy when given 99% of the solution—indicating that models can leverage provided reasoning traces but cannot yet productively use their own generated intermediate steps. Extrapolating from observed scaling trends, the authors estimate that achieving 40% accuracy through parameter scaling alone would require models with approximately 10³⁵ parameters, establishing the core boundary condition that algorithmic breakthroughs—not merely larger Transformers—are necessary for meaningful progress on mathematical problem-solving tasks of this difficulty.

2. Context and Motivation

The Problem: Mathematical Reasoning Remains Beyond Current Models

By late 2020 and early 2021, when this paper was written, enormous Transformer models had demonstrated remarkable, sometimes startling, proficiency across a wide range of text-based tasks. Models like GPT-3 (Brown et al., 2020) could generate coherent essays, translate languages, answer trivia questions, and even produce functional code from natural language descriptions, all with minimal task-specific training. The prevailing narrative in the field—supported by scaling laws research from Kaplan et al. (2020) and Henighan et al. (2020)—suggested that many limitations of language models could be overcome simply by increasing model size and dataset size, with performance improving along predictable log-linear trends.

The MATH paper challenges this narrative head-on by identifying a specific capability that appears fundamentally resistant to scaling: mathematical problem solving, defined not as straightforward calculation or plug-and-chug arithmetic, but as the kind of multi-step, heuristic-driven reasoning required to solve competition-level mathematics problems. The authors frame this as a test of genuine problem-solving ability:

"the ability to analyze a problem, pick out good heuristics from a large set of possibilities, and chain them together to produce an answer."

This matters because mathematical reasoning represents a qualitatively different kind of intelligence than the pattern-matching and statistical association that large language models excel at. Unlike tasks such as sentiment analysis, summarization, or even code completion—where surface-level statistical regularities in the training data provide strong signal—competition mathematics requires selecting the right approach from a combinatorially large space of possible operations, applying it precisely, and verifying that each step logically follows from the previous one. A single arithmetic error or misapplied theorem anywhere in a chain of reasoning can render the entire solution incorrect.

The Gap: Existing Benchmarks Don't Test This Capability

The authors argue that prior mathematical reasoning benchmarks fall into two categories, neither of which adequately captures the problem-solving skill that MATH targets.

Formal Theorem Proving Benchmarks. Work on benchmarks like HOList (Bansal et al., 2019), Coq-based environments (Huang et al., 2019), and MetaMath (Polu and Sutskever, 2020) focuses on automated theorem proving in formal languages. These are rigorous and automatically verifiable, but they suffer from a critical practical limitation: they require problems to be expressed in formal proof languages that are far removed from how humans actually write and reason about mathematics. As Szegedy (2020) argues—and the MATH paper explicitly cites—manually converting advanced mathematics into formal proof languages is "extremely time-consuming," making this an "impractical path towards world-class mathematical reasoning" if the goal is to work with mathematics as humans actually practice it. The MATH paper positions itself as addressing this concern directly by creating a dataset where mathematics is expressed in natural language and standard LaTeX notation—the same format used in textbooks, papers, and competitions.

Plug-and-Chug Calculation Benchmarks. The other category includes datasets like the DeepMind Mathematics dataset (Saxton et al., 2019), which consists of algorithmically generated problems covering operations like addition, function evaluation, and list sorting. Figure 2 in the paper shows that state-of-the-art accuracy on such datasets is dramatically higher than on MATH—approaching 100% on some subtasks—precisely because these problems test calculation ability rather than problem-solving. A model can learn the pattern "when you see $f(x) = x^2 + x - 4$ and are asked for $f(0)$ , substitute and compute" without developing any capacity for the kind of strategic reasoning that distinguishes mathematical problem solving from arithmetic. Similarly, Lample and Charton (2020) achieved >95% accuracy on symbolic integration by framing it as a sequence-to-sequence translation task, but their problems were algorithmically generated from a known grammar—making them fundamentally pattern-matching exercises rather than open-ended reasoning challenges.

The paper summarizes this distinction clearly in Figure 1, which contrasts "plug-and-chug" problems like "Divide 1136975704 by -142121963" with MATH problems like "The equation $x^2 + 2x = i$ has two complex solutions. Determine the product of their real parts." The latter requires recognizing that completing the square simplifies the equation, manipulating complex exponentials, factoring out terms, and applying trigonometric identities—a chain of strategic decisions that cannot be reduced to a single memorized operation.

Why This Gap Is Critical

The importance of mathematical problem solving as a benchmark capability extends beyond mathematics itself. The authors argue that mathematical reasoning serves as a proxy for domain-general problem solving—the kind of flexible, multi-step reasoning that would be necessary for any AI system that needs to operate autonomously in complex environments. Mathematics is appealing as a testbed for this capability because:

Correctness is objective and verifiable. Unlike open-ended generation tasks where quality is subjective, mathematical answers are uniquely defined and can be checked with exact match after normalization. This enables rigorous, automated evaluation without the ambiguities inherent in metrics like BLEU or human preference judgments. The paper's careful answer normalization scheme—handling fraction equivalence ( $\frac{1}{2} \equiv 0.5$ ), polynomial factor ordering ( $4(x+1)(x-1) \equiv 4(x-1)(x+1)$ ), and spacing/parenthesis variations—makes this possible at scale.
Problems span a wide difficulty spectrum and multiple subdomains. MATH includes problems tagged with difficulty levels from 1 to 5 (following Art of Problem Solving conventions) across seven subjects: Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus. This enables fine-grained analysis of where and why models fail, rather than a single aggregate number.
Step-by-step solutions provide rich training and evaluation signal. Every MATH problem includes a full solution derivation, enabling research on whether models can learn to generate their own reasoning traces—a capability that would be transformative for interpretability, verification, and iterative self-improvement.
Scaling trends can be empirically measured. Because MATH is a fixed benchmark with clear accuracy metrics, researchers can plot how performance changes with model size, training data, and algorithmic innovations—exactly the kind of empirical science that drove progress in other areas of deep learning.

Where Prior Approaches Fall Short

The paper identifies several specific limitations in existing work:

Enormous Transformers are rapidly solving most text-based tasks, creating complacency. The authors explicitly cite benchmarks like SuperGLUE (Wang et al., 2019), which were "solved by simply training enormous Transformers" (He et al., 2020). They also show—in Appendix A.2 and Figure 6 of the paper—that on logical reasoning tasks like LogiQA (Liu et al., 2020), Transformer performance is improving so rapidly with scale that human-level performance appears imminent. Their analysis of the C-Test, a pattern completion test that correlates 77% with human IQ, shows GPT-3 achieving approximately 40% accuracy on the hardest examples where average humans score around 20%. These results collectively suggest that scaling alone is sufficient for many reasoning-like tasks—which makes MATH's resistance to scaling all the more significant and informative.

No existing dataset combines natural language mathematics, diverse subjects, step-by-step solutions, and genuine difficulty. The paper positions MATH as filling a specific void. Prior datasets either:

Used formal languages inaccessible to standard language models (theorem proving benchmarks).
Consisted of simple, algorithmically-generated problems that models already solve (DeepMind Math, symbolic integration).
Were small, multiple-choice, or lacked step-by-step solutions (MMLU mathematics subset from Hendrycks et al., 2021, which the paper notes "includes only a few hundred questions about mathematics").
Had quality issues with annotation errors (MathQA/AQuA-RAT, where Miao et al., 2020 found "27% of the problems do not match their labeled answers").

The field lacked a benchmark that withstands scaling. This is perhaps the paper's most pointed motivation. By 2021, the NLP community had developed a pattern: propose a benchmark, achieve near-random performance initially, then watch as larger models and more training data steadily erode the benchmark until it is "solved." The authors explicitly want to break this pattern by creating a benchmark where scaling alone won't work, forcing the community to develop genuinely new algorithmic approaches. Their extrapolation—that achieving 40% accuracy through scaling would require ~10³⁵ parameters, far beyond any practical compute budget—is designed to be provocative and to redirect research effort toward innovation rather than scale.

How This Paper Positions Itself

The paper's self-positioning is unusually clear and intentional. It is not a methods paper proposing a new model architecture or training technique. It is a dataset and benchmark paper in the tradition of datasets that have catalyzed entire research directions (ImageNet for vision, SQuAD for reading comprehension, GLUE for natural language understanding). The authors explicitly frame MATH as:

A stress test for the "scaling solves everything" hypothesis. If Transformers can master competition mathematics simply by becoming larger, then the hypothesis holds and MATH will eventually be solved. If not—and the paper's initial results strongly suggest not—then MATH serves as a clear signal that new ideas are needed.
A platform for research on mathematical reasoning. By providing step-by-step solutions, the dataset enables work on generating reasoning traces, verifying intermediate steps, using scratch space, and self-improvement through solution generation. The paper's own experiments on these fronts—showing that models can benefit from solutions during training (+10% relative improvement) but cannot yet use their own generated solutions at test time (accuracy decreases)—demonstrate the kind of research questions the dataset enables.
A complement to, not a replacement for, formal theorem proving. The paper acknowledges the value of formal verification but argues that understanding mathematics as humans write it is a distinct and important capability. The two approaches—formal verification for rigor, natural language mathematics for accessibility and scale—are portrayed as complementary.
A call to action for algorithmic innovation. The abstract and conclusion explicitly state that "solving the MATH dataset with machine learning would be of profound practical and intellectual significance" and that doing so will require "new algorithmic advancements from the broader research community." This is not a paper that claims to have solved the problem; it is a paper that claims to have defined a problem worth solving, with carefully chosen properties that make it both challenging and tractable for research.

The paper also introduces AMPS (the Auxiliary Mathematics Problems and Solutions pretraining corpus) as a practical contribution to bootstrap progress. AMPS serves two purposes: it demonstrates that mathematics-specific pretraining can partially compensate for model scale (a 0.1B parameter model pretrained on AMPS matches a fine-tuned 13B parameter GPT-3 without AMPS pretraining), and it provides the community with a large, clean resource for future work on mathematical language models. The inclusion of AMPS reflects the authors' practical orientation: while MATH is designed to be hard, they also want to enable immediate progress by providing high-quality pretraining data that didn't previously exist at this scale or quality.

3. Technical Approach

3.1 Reader Orientation

This paper introduces a benchmark dataset and evaluation framework—not a novel model or algorithm—designed to measure the mathematical problem-solving capabilities of machine learning models, particularly large language models. The core problem it addresses is that existing mathematical benchmarks either test only calculation ability (through algorithmically-generated plug-and-chug problems) or require formal theorem-proving languages inaccessible to standard text-based models, leaving no rigorous way to assess whether models can perform the kind of heuristic-driven, multi-step reasoning that characterizes genuine mathematical problem solving. The solution takes the form of a carefully curated dataset of 12,500 competition mathematics problems with step-by-step solutions and a unified answer normalization scheme, paired with a large auxiliary pretraining corpus (AMPS) to bootstrap progress, together enabling automated evaluation through exact-match accuracy while providing rich training signal through full solution derivations.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components:

MATH Dataset (12,500 problems) — The core benchmark, consisting of competition mathematics problems spanning seven subjects and five difficulty levels, each with a ground-truth final answer (wrapped in \boxed{}) and a full step-by-step solution. Problems are sourced from AMC 10, AMC 12, AIME, and other competitions, and figures are encoded in the Asymptote vector graphics language rather than as raster images.
Answer Normalization and Grading Pipeline — A rule-based system that extracts text between \boxed{} delimiters from model outputs and tests equivalence against the ground-truth answer, accounting for equivalent fraction representations ( $\frac{1}{2} \equiv 0.5$ ), polynomial factor orderings ( $4(x+1)(x-1) \equiv 4(x-1)(x+1)$ ), coefficient formatting ( $5x \equiv 5 \cdot x$ ), and other notational variations.
AMPS Pretraining Corpus (23 GB) — An auxiliary dataset of over 100,000 Khan Academy problems with step-by-step solutions and approximately 5 million algorithmically-generated problems from 100 hand-designed Mathematica scripts (37 of which include full solutions). This provides mathematics-specific pretraining data that the paper shows can partially compensate for model scale.
Autoregressive Language Models (GPT-2, GPT-3, BART) — The models evaluated on MATH. These are decoder-only or encoder-decoder Transformers pretrained on general text, then optionally pretrained on AMPS, and finally fine-tuned on the MATH training set (7,500 problems) using a mixture of final-answer and full-solution generation formats.
Evaluation Protocol — The procedure for measuring model performance: models are prompted to generate either a final answer directly (using the prefix "⟨P⟩ Final Answer:") or a full step-by-step solution followed by a boxed answer (using "⟨P⟩ Full Solution:"). Beam search with beam sizes between 10 and 20 is used for decoding, and accuracy is computed as the fraction of test-set problems where the extracted and normalized answer matches the ground truth exactly.

Information flows as follows: a problem from the MATH test set → the model generates tokens autoregressively (either directly producing an answer or generating a full solution with a boxed answer) → the grading pipeline extracts and normalizes the answer → the normalized answer is compared against the ground-truth answer → accuracy is aggregated across the 5,000 test problems, optionally broken down by subject, difficulty level, or both.

3.3 Roadmap for the Deep Dive

First, the MATH dataset construction—problem sources, formatting rules, the answer normalization scheme, and the difficulty/subject categorization—since understanding what the benchmark is is prerequisite to understanding how models are evaluated on it.
Second, the AMPS pretraining corpus—its composition (Khan Academy + Mathematica), how it differs from prior pretraining datasets, and why it was constructed—since AMPS pretraining is a key experimental condition that dramatically affects results.
Third, the model architectures and training procedures—pretraining on AMPS, fine-tuning on MATH, and the mixed-format training objective—since these define the exact experimental conditions under which accuracy numbers are obtained.
Fourth, the evaluation protocol—prompting formats, beam search configuration, answer extraction, and normalization rules—since this determines what "accuracy" means and whether comparisons across models are meaningful.
Fifth, the experimental design choices—why GPT-2/GPT-3/BART, why beam search, why the specific hyperparameters, and what ablations were performed to validate these choices.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a dataset and benchmarking paper whose core technical contribution is the careful construction of MATH, a corpus that enables automated evaluation of mathematical problem-solving through exact-match accuracy while simultaneously providing rich training signal through step-by-step solution derivations, and the introduction of AMPS, a mathematics-specific pretraining corpus that demonstrates the value of domain-adaptive pretraining for mathematical reasoning.

MATH Dataset Construction and Formatting

Problem Sources and Selection. The MATH dataset draws exclusively from high school mathematics competitions, including the AMC 10, AMC 12, AIME, and other contests whose problems are publicly available through the Art of Problem Solving (AoPS) website at aops.com/community/c3158_usa_contests. These competitions span decades and collectively represent some of the most carefully designed mathematical problems available—each problem is crafted by expert mathematicians to test deep understanding rather than rote calculation, typically requiring the solver to recognize which mathematical tools apply, apply them correctly, and chain multiple steps together without arithmetic errors. The paper states that 12,500 problems are included, split into 7,500 training problems and 5,000 test problems, providing sufficient training data "for models to learn many useful heuristics for problem solving" while maintaining a large enough test set for reliable evaluation.

The seven subjects are Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus. These were chosen to span the full range of pre-college competition mathematics, from relatively accessible topics like Prealgebra (which might involve basic equation solving or fraction manipulation) to advanced topics like Precalculus (which can involve complex numbers, trigonometric identities, and analytic geometry). The paper notes that "while subjects like Prealgebra are generally easier than Precalculus, within a subject problems can take on different difficulty levels," which motivates the dual categorization by both subject and difficulty.

Difficulty levels are encoded as integers from 1 to 5, following the AoPS convention where:

Level 1 corresponds to the easiest problems in a given subject, such as the first few problems of an AMC 8 exam
Level 5 corresponds to the hardest problems, such as AIME-level problems which require sophisticated problem-solving techniques

This is explicitly a human-centric difficulty scale, not derived from model performance. The paper uses this labeling to analyze whether "human difficulty and machine difficulty track each other"—a question answered affirmatively in Figure 9, where higher difficulty levels correspond to lower model accuracy.

LaTeX Formatting and Figure Encoding. Every problem and solution is consistently formatted using LaTeX, the standard typesetting language for mathematical documents. This is a deliberate design choice: LaTeX can encode virtually all mathematical notation compactly and unambiguously, and it is already the format used by AoPS and most mathematical writing. Unlike formal theorem-proving languages (Coq, MetaMath, HOL Light) which require re-expressing mathematics in an artificial logical syntax, LaTeX preserves the natural language flow of mathematical reasoning while providing machine-parseable structure for equations, fractions, matrices, and other mathematical objects.

For geometric figures and diagrams, the paper makes a novel technical choice: rather than storing figures as raster images (PNG, JPEG) which would require multi-modal models to process, figures are encoded in the Asymptote vector graphics language. Asymptote is a programming language for creating technical drawings, and the paper shows in Figure 7 that Asymptote code can concisely specify diagrams—a cylinder with labeled dimensions, geometric constructions, coordinate systems—using drawing primitives like draw, Circle, and label. This means that purely text-based language models can (in principle) process figure-containing problems without any visual processing module, because the figure specification is part of the text input stream. The paper shows in Figure 3 that models can even generate Asymptote code themselves, though the quality of these generated figures is variable.

Answer Formatting Rules. To enable exact-match evaluation—checking whether a generated answer matches the ground truth—the authors impose a set of formatting conventions that eliminate representational ambiguity. These rules are enforced during dataset construction on the ground-truth answers and form the basis of the normalization pipeline applied to model outputs during evaluation. The specific rules are:

Probabilities are expressed as simplified fractions — so $2/4$ would be written as $\frac{1}{2}$ , not $2/4$ or $0.5$ . This ensures that the same probability always has exactly one correct string representation.
Matrix entry fractions use x/y notation, all other fractions use \frac{x}{y} — this distinguishes fractions that are matrix elements (which are conventionally written inline) from fractions that represent standalone numbers (which are conventionally written with a fraction bar). Without this distinction, a model could produce $\frac{1}{2}$ or $1/2$ for the same answer, and both would be mathematically correct but textually different.
Coefficients are encoded without a multiplication symbol — so $5x$ not $5*x$ or $5 \cdot x$ . This matches standard mathematical notation where multiplication of a constant and a variable is implicit.
Expressions with multiple variables are entered in alphabetical order — so $xy + yz$ not $yx + zy$ . This is crucial because unlike arithmetic where order never matters, polynomial expressions with symbolic variables conventionally follow alphabetical ordering to avoid the combinatorics of equivalent permutations.
Polynomials are expressed in decreasing degree order — so $x^2 + 3x + 2$ not $2 + 3x + x^2$ . Again, this eliminates representational variation that doesn't reflect mathematical difference.

Answer Normalization During Evaluation. When evaluating a model, the system does not require the model to output the answer in exactly the canonical format. Instead, a normalization function maps both the ground-truth answer and the model's generated answer onto a common representation before comparison. The paper describes the following equivalence classes:

Different fraction encodings are treated as equivalent: \frac{x}{y}, \dfrac{x}{y}, and x/y all map to the same normalized form. This is necessary because models may unpredictably choose any of these LaTeX commands.
Different parenthesis encodings are treated as equivalent: \left( and ( produce the same rendering in LaTeX but are different tokens. The normalization strips these distinctions.
Units may be included or omitted from an answer, and spaces are ignored. This means 12\text{ cm} and 12 are equivalent when the answer is a length.
Common equivalent number representations are treated as the same: $0.5$ and $1/2$ , or $0.1$ and $.1$ . This is crucial because MATH problems often have answers that can be expressed either as simplified fractions or decimal approximations, and the model shouldn't be penalized for choosing one over the other.
Factorized polynomial orderings are treated as equivalent: $4(x+1)(x-1)$ is equivalent to $4(x-1)(x+1)$ . This is enforced by parsing the factorized form and comparing the multiset of factors, not the string itself.
Variable ordering in expressions is normalized as described above.

The paper states that "these rules cover nearly all ways that different generated or actual solutions can be equivalent in practice," meaning that the normalization pipeline is effectively complete—there is likely very little residual ambiguity where two mathematically equivalent answers would fail the exact-match check due to formatting differences alone.

Answer Extraction from Model Outputs. A critical feature of MATH is that the ground-truth final answer is always wrapped in the \boxed{} command—a LaTeX convention that puts a visible box around the answer in rendered documents. This serves a dual purpose. First, it unambiguously delineates where the answer begins and ends, even when the answer is embedded in a longer solution. Second, it provides a natural target for models: during training, models learn that the final answer appears inside \boxed{}, and during evaluation, the grading pipeline extracts whatever text appears between the \boxed{} delimiters in the model's output. The paper states that "automatic assessment starts by determining the beginning and end of the answer" by parsing for these delimiters, then applying the normalization rules to compare against the ground truth. This design means that the evaluation is robust to models that generate varying amounts of preamble, commentary, or even incorrect derivations—as long as the final boxed answer is correct and properly delimited, the model gets credit.

Human-Level Performance. To calibrate expectations and provide a rough sense of the benchmark's difficulty for humans, the authors conducted a small human evaluation. The protocol: 20 randomly sampled problems from the MATH test set, 1 hour time limit, no calculators, performed by university students with varying mathematical backgrounds. A computer science PhD student who "does not especially like mathematics" scored 40% (8/20). Students who "like mathematics" scored 70–75% (14–15/20). A participant with a perfect AMC 10 score and multiple USAMO qualifications scored 90% (18/20). A three-time IMO gold medalist also scored 90% (18/20), with missed questions "exclusively due to small errors of arithmetic." The paper notes that "expert-level performance is theoretically 100% given enough time," but the time constraint and no-calculator rule make even expert-human performance imperfect on a small sample. These numbers provide important context: 40% accuracy on MATH is roughly the level of a mathematically disinclined PhD student, 90% is IMO gold medalist level, and 100% would require superhuman consistency.

AMPS Pretraining Corpus Construction

Motivation. The authors argue that mathematics is underrepresented in standard web-crawled pretraining corpora. Most internet text is natural language prose, code, or informal discussion—mathematical notation, formal derivations, and step-by-step solutions constitute a tiny fraction of the training data for models like GPT-2 and GPT-3. This domain mismatch means that even very large language models may lack the fundamental mathematical vocabulary and basic skills that would be necessary to approach competition-level problems. The AMPS corpus is designed to address this gap by providing a dense, high-quality source of mathematical content that teaches models "the fundamentals of mathematics" before they are asked to tackle MATH problems.

Khan Academy Subset. The first component of AMPS consists of "over 100,000 problems pulled from Khan Academy," covering 693 distinct exercise types. These exercises range from elementary arithmetic (addition, subtraction) through algebra, geometry, trigonometry, and calculus (Stokes' theorem, surface integrals). The paper shows the full list of 693 modules in Figures 12–15, demonstrating coverage of essentially the entire K-12 mathematics curriculum plus early college-level topics. Critically, every Khan Academy exercise includes a full step-by-step solution typeset in LaTeX, meaning that models pretrained on this data learn not just what the answer is but how to derive it—the intermediate reasoning steps that are absent from most existing mathematics datasets.

The Khan Academy exercises are structured for human learning: they start with simple one-step problems (e.g., "add within 10") and progressively build to multi-step problems requiring integration, differentiation, and vector calculus. The paper notes that these exercises "can be regenerated using code from github.com/Khan/khan-exercises/," meaning the dataset is reproducible and could be regenerated with different random parameters to prevent memorization.

The paper also includes an example in Figure 8 that illustrates the style of Khan Academy problems:

Problem: In history class, the girl to boy ratio is 9 to 6. If there are a total of 60 students, how many boys are there?
Solution: A ratio of 9 girls to 6 boys means that a set of 15 students will have 9 girls and 6 boys. A class of 60 students has 4 sets of 15 students. Because we know that there are 6 boys in each set of 15 students, the class must have 4 groups of 6 boys each. There is a total of 24 boys in history class.

This exemplifies the AMPS style: problems are expressed in natural language with embedded mathematics, and solutions walk through reasoning step-by-step in prose, not just providing a final answer.

Mathematica Subset. The second component is substantially larger: "approximately 5 million problems generated from manually designed Mathematica scripts," based on 100 distinct modules covering topics in algebra, calculus, counting and statistics, geometry, linear algebra, and number theory. Table 1 in the paper lists a subset of these topics: conic sections, polynomial GCD, De Moivre's theorem, function inverses (algebra); arclength, Jacobian, Laplacian, divergence, curl, gradients, integrals (calculus); expectation, geometric/harmonic mean, KL divergence, variance (statistics); triangle area, inradius, polygon angles, polyhedron diameter (geometry); characteristic polynomials, eigenvalues, reduced row echelon form (linear algebra); modular inverse, Euler's totient function, Chinese remainder theorem (number theory).

The paper shows a concrete Mathematica script for polynomial GCD problems in Figure 8. The script uses:

Random parameters for polynomial degree (d1, d2)
Random coefficients rounded to a configurable precision (roundbasis)
The PolynomialGCD function to compute the answer algorithmically
LaTeX formatting via TeXForm for output

A key design choice: 37 of the 100 Mathematica scripts generate not just problems and answers but full step-by-step solutions in LaTeX. This means that for these modules, the pretraining data provides the same kind of reasoning traces that MATH provides, teaching models to produce intermediate derivations rather than just answers. The paper does not provide details on how these step-by-step solutions are generated algorithmically—presumably by exploiting Mathematica's ability to show intermediate steps for certain operations—but the presence of solutions for 37% of the Mathematica data is an important feature.

The Mathematica scripts use the computer algebra system to "manipulate fractions, transcendental numbers, and analytic functions," which distinguishes AMPS from prior algorithmically-generated mathematics datasets like DeepMind Mathematics (which mainly used simple arithmetic and algebra with standard number types). The ability to generate problems involving $\pi$ , $e$ , logarithms, and trigonometric functions means that models pretrained on AMPS are exposed to the same kinds of mathematical objects that appear in MATH competition problems.

Upsampling and Downsampling. Because Khan Academy and Mathematica differ dramatically in size (100K vs. 5M problems), the two sources would be severely imbalanced if pretrained on in proportion to their sizes. The paper addresses this: "During pretraining, we upsample Khan Academy data by a factor of 5 and we downsample Mathematica by a factor of 2." This means that in each training epoch, the effective ratio of Khan Academy to Mathematica examples is roughly $5:1$ rather than $1:50$ , ensuring that the more carefully curated, pedagogically-designed Khan Academy problems are not drowned out by the algorithmically-generated Mathematica problems. The paper does not provide an ablation on this ratio, but the choice reflects a judgment that Khan Academy's human-designed solutions are higher-quality signal than algorithmically-generated problems, even if the latter are more numerous.

Contrast with Prior Pretraining Data. The paper explicitly contrasts AMPS with two alternative sources of mathematical pretraining data:

General web text (implicit in standard GPT-2/GPT-3 pretraining). Mathematics represents a small fraction of internet text, so standard pretraining provides sparse coverage of mathematical notation and reasoning patterns.
Math StackExchange questions and answers (~3 GB). The paper reports an ablation where pretraining on AMPS plus StackExchange data yielded 6.0% accuracy, which is worse than the 6.2% accuracy achieved by pretraining on AMPS alone. The authors interpret this as evidence that AMPS is "more useful for pretraining even than diverse real-world mathematics data." The implication is that StackExchange data—while mathematically rich—is noisy, contains many incorrect or incomplete answers, and mixes mathematical content with non-mathematical discussion, diluting the learning signal.

The total size of AMPS is "over 23 GB of mathematics problems, making it larger than the 16 GB of natural language used to train BERT (Devlin et al., 2019)." This comparison anchors AMPS's scale: it's not merely a small auxiliary dataset but a substantial corpus that could serve as the primary pretraining data for a mathematics-focused language model.

Model Architectures and Training Procedures

Model Selection. The paper evaluates primarily autoregressive language models—specifically, decoder-only Transformers—which is the standard architecture for text generation tasks. The specific models used are:

GPT-2 at four sizes: 0.1B, 0.3B, 0.7B, and 1.5B parameters. These are used with HuggingFace's default generation parameters except where noted.
GPT-3 at two sizes: the 'Curie' model (approximately 13B parameters) and the 'Davinci' model (approximately 175B parameters), accessed via the OpenAI API. Due to API limitations, GPT-3 models could not be pretrained on AMPS—the API only supports fine-tuning on user-provided data, not full pretraining.
BART at one size: BART-Large (0.4B parameters), evaluated in Appendix A.5 as an alternative architecture.

The paper documents a negative result with T5 (Raffel et al., 2020): "T5's tokenizer removes many LaTeX symbols, so after a broad hyperparameter sweep lasting two weeks, its performance was not competitive." This is an important practical finding—the choice of tokenizer can effectively disqualify a model from mathematical reasoning tasks if it strips out or fragments the LaTeX commands that encode mathematical structure. GPT-2's tokenizer, by contrast, processes numbers one digit at a time (following Henighan et al., 2020), which provides fine-grained access to numerical content. BART's tokenizer preserves LaTeX symbols, which is why it was chosen over T5 despite the similar encoder-decoder architecture.

AMPS Pretraining Procedure. Models are pretrained on AMPS for one epoch using:

Optimizer: AdamW (Loshchilov and Hutter, 2019) with decoupled weight decay
Batch size: 128
Weight decay: 0.05
Objective: Standard autoregressive language modeling (next-token prediction)

The pretraining objective is:

$\mathcal{L}_{\text{pretrain}} = -\sum_{t=1}^{T} \log P_\theta(x_t \mid x_{<t})$

where $x_t$ is the $t$ -th token in the AMPS corpus, $P_\theta$ is the model's predicted probability distribution over the vocabulary, and $T$ is the total number of tokens in the pretraining sequence.

What it computes: the standard causal language modeling loss—for each position in the input sequence, the model predicts the next token given all previous tokens, and the loss is the negative log-likelihood of the actual next token under that prediction. The total loss is the average over all positions.

Why this form: autoregressive language modeling is the standard pretraining objective for decoder-only Transformers and is what GPT-2 and GPT-3 were originally trained with, making it the natural choice for a domain-adaptive pretraining phase that extends rather than replaces the original training. Alternatives like masked language modeling (BERT) or denoising objectives (BART) would require architectural changes incompatible with the GPT-family models.

The choice of a single pretraining epoch on AMPS reflects a practical constraint—the dataset is finite, and multiple epochs could lead to memorization—but also an empirical finding (consistent with standard transfer learning practice) that one epoch of domain-specific pretraining provides most of the benefit without overfitting.

MATH Fine-Tuning Procedure. After pretraining on AMPS, models are fine-tuned on the 7,500-problem MATH training set. The fine-tuning uses the same optimizer and hyperparameters as pretraining (AdamW, batch size 128, weight decay 0.05).

The fine-tuning objective is a mixed-format training approach. Let $\langle P \rangle$ denote the problem text, $\langle\text{Answer}\rangle$ denote the ground-truth final answer, and $\langle\text{Solution}\rangle$ denote the ground-truth step-by-step solution. The training data consists of an equal mix of two sequence types:

Final-answer-only format: "⟨P⟩ Final Answer: ⟨Answer⟩"
Full-solution format: "⟨P⟩ Full Solution: ⟨Solution⟩"

The model is trained to predict the tokens following the prefix (either the answer or the solution), with the loss computed only on those target tokens (the problem and prefix tokens are conditioning context, not prediction targets). The paper states they train with "an equal mix" to ensure the model learns both skills—producing compact final answers quickly and producing detailed step-by-step solutions—since these are used in different evaluation settings.

Why this mixed format? Training only on final answers would teach the model the answer formats but not how to derive them, losing the benefit of step-by-step solutions for learning mathematical reasoning. Training only on full solutions would teach the model to generate verbose outputs that include derivations, but might make it slower at the direct-answer task and could confuse the model about when to stop generating. The mixed format allows the model to learn both modes, with the prompting prefix ("Final Answer:" vs. "Full Solution:") acting as a control signal for which output format to produce. The paper confirms that training with solutions improves performance: ablating solutions during fine-tuning (training only on final answers) reduces GPT-2 1.5B accuracy from 6.9% to 6.3%, a 0.6 percentage point drop.

Training Infrastructure. The paper states models are "trained with 8 A100 GPUs, each requiring less than a day," indicating that the full pipeline—AMPS pretraining plus MATH fine-tuning—is computationally accessible to academic research groups, not requiring industrial-scale compute. This is an intentional design choice for a benchmark: the evaluation protocol and training recipe should be reproducible without massive resources.

GPT-3 Evaluation via OpenAI API. For GPT-3 experiments, two modes are used:

Few-shot: The prompt consists of 8 example problems with correct answers (but not full solutions, "due to space"), formatted consistently, followed by the test problem. The model generates up to 20 tokens at temperature 0 (greedy decoding). This tests the model's ability to infer the task from in-context examples without any gradient updates.
Fine-tuned: The OpenAI API's fine-tuning endpoint is used on the MATH training set, but without AMPS pretraining because "their API does not have the option to pretrain on datasets as large as AMPS." This means GPT-3 fine-tuned results are directly comparable to GPT-2 fine-tuned results without AMPS pretraining, isolating the effect of model scale alone.

The few-shot results for GPT-3 (both 13B and 175B) are marked with asterisks in Table 2 and are notably worse than the fine-tuned versions of the same models, highlighting that in-context learning alone is insufficient for MATH—the model needs task-specific fine-tuning to adapt to the answer format and mathematical reasoning patterns of the benchmark.

Evaluation Protocol

Prompting Formats. Models are evaluated using two distinct prompting strategies, corresponding to the two fine-tuning formats:

Direct answer prompting: The model receives "⟨P⟩ Final Answer:" and is expected to generate only the final answer (typically a short string like \frac{2}{3} or 7). This is the default evaluation mode used for all main results unless otherwise specified.
Full solution prompting: The model receives "⟨P⟩ Full Solution:" and is expected to generate a complete step-by-step derivation ending with a boxed answer. This mode tests whether the model can productively use "scratch space"—generating intermediate reasoning steps before committing to a final answer.

The paper's hypothesis is that for difficult problems, allowing the model to "think out loud" through step-by-step generation should improve accuracy because the model can use intermediate tokens to perform computations and check reasoning that would be impossible in a single forward pass to the answer. The result, however, is the opposite: generating full solutions decreases accuracy from 6.9% to 5.3% for GPT-2 1.5B. The authors hypothesize this is due to a "snowballing effect, in which partially generated 'solutions' with mistakes can derail subsequent generated text." In other words, once the model makes an error early in a solution, that error conditions all subsequent tokens, making the model increasingly likely to produce an incorrect final answer. This is a significant finding: the model has not yet learned to generate solutions that are self-consistent or self-correcting.

Beam Search Configuration. The paper uses beam search rather than greedy decoding or random sampling for answer generation. Beam search maintains multiple candidate output sequences and selects the one with the highest overall probability, which is standard for tasks where a single high-quality output is preferred over diverse outputs.

The specific beam sizes used are:

Beam size 20 when generating only the final answer (direct answer prompting). This relatively large beam allows the model to explore many possible completions of the final answer and pick the most likely one.
Beam size 10 when generating full step-by-step solutions. The smaller beam is presumably due to computational constraints—since solutions are much longer than final answers, maintaining 20 beams through hundreds of tokens would be prohibitively expensive.

The paper does not provide an ablation on beam size, but the choice reflects a practical tradeoff: larger beams increase the chance of finding the correct answer (by exploring more completions) but cost proportionally more compute.

Temperature settings are not specified for GPT-2 experiments in the main text, but GPT-3 experiments use temperature 0 (equivalent to greedy decoding within each beam). The HuggingFace defaults are used otherwise, which typically include temperature 1.0 with beam search (meaning the probabilities are unmodified before beam selection).

Answer Extraction and Comparison. The evaluation pipeline for each problem works as follows:

Token generation: The model generates tokens autoregressively until a stop condition is met (end-of-sequence token or maximum length reached).
Answer extraction: The generated text is parsed to find the \boxed{} command. The content between \boxed{ and the matching } is extracted. If no \boxed{} is found, or if the parsing fails (e.g., unmatched braces), the answer is treated as empty.
Normalization: Both the extracted answer and the ground-truth answer are passed through the normalization function described in Section 3.1, which applies the equivalence rules (fraction formats, spacing, variable ordering, etc.).
Exact match comparison: The normalized strings are compared. If they are identical, the problem is counted as correct. Otherwise, it is incorrect.
Aggregation: Accuracy is computed as the fraction of the 5,000 test problems where step 4 returns a match. Results are reported as percentages, optionally broken down by subject, difficulty level, or both.

The paper notes that answers are "unique after normalization," meaning the normalization rules are sufficient to collapse all mathematically equivalent but syntactically different answers into a single canonical form. This is a strong claim that the authors support by describing the specific equivalences handled, but it is worth noting that perfect normalization is theoretically impossible for arbitrary mathematical expressions (due to the undecidability of expression equivalence in general). However, for the constrained domain of competition mathematics answers—which are typically simple numbers, fractions, or algebraic expressions—the described rules likely cover the vast majority of cases.

Confidence Estimation. As an auxiliary analysis, the paper examines whether model confidence can be used to detect errors. Confidence is defined as "the average prediction probability of the tokens that make up a generated answer"—that is, for an answer consisting of $k$ tokens, confidence is:

$\text{confidence} = \frac{1}{k} \sum_{i=1}^{k} P_\theta(t_i \mid \text{context}, t_{<i})$

where $t_i$ is the $i$ -th token of the answer and $P_\theta(t_i \mid \dots)$ is the model's predicted probability for that token given the context and previous answer tokens. This is the geometric mean of token-level probabilities, a standard confidence metric for autoregressive generation.

The paper finds that GPT-2 1.5B is "highly overconfident, with confidences that are often around 100%," and that the AUROC for distinguishing correct from incorrect answers based on confidence is only 68.8% (where 50% is random and 100% is perfect separation). This indicates that the model cannot reliably assess the correctness of its own outputs—it assigns high probability to wrong answers with similar confidence as to right answers.

Human Evaluation Protocol. The paper's human baseline, described earlier, provides an important calibration point but is not a rigorous psychometric study. The limitations include:

Only 6 participants, all university students
Only 20 problems sampled from the test set
1-hour time limit (MATH problems, especially at higher difficulty, can take 10–20 minutes each for careful solution)
No calculator allowed (which disadvantages humans more than models, which can perform arithmetic perfectly)
Four of six participants were paper authors (potential for familiarity effects, though problems were randomly sampled)

The results should therefore be interpreted as "rough but informative" rather than as precise human baselines. The important qualitative finding is that MATH is challenging for humans with strong mathematical backgrounds and nearly impossible for current ML models—the best model's 6.9% accuracy is an order of magnitude below the weakest human participant's 40%.

Experimental Design Choices and Justifications

Why GPT-2/GPT-3 and Not Other Architectures? The paper's choice of autoregressive decoder-only models is pragmatic:

Decoder-only models are the state of the art for text generation — architectures like GPT-2, GPT-3, and their successors dominated few-shot and fine-tuned generation benchmarks in 2020–2021.
BART was tested but performed worse (4.9% vs. 5.4% for the smallest GPT-2), suggesting that the bidirectional encoder doesn't help for this task when the model has already been pretrained on AMPS. The paper speculates that the decoder-only architecture's causal attention may be better suited to autoregressive generation of mathematical notation.
T5 failed due to tokenization — its SentencePiece tokenizer strips many LaTeX control sequences, making it fundamentally unable to represent mathematical notation. This finding is important in its own right because it demonstrates that tokenizer choice can be the limiting factor for technical domains.

Why Beam Search and Not Sampling? Beam search with beam sizes of 10–20 is used throughout, which is standard for maximum-accuracy generation. The paper doesn't ablate this choice, but it is justified by the task: MATH requires exactly one correct answer per problem, and maximizing the probability of that answer is the goal. Sampling (with temperature > 0) would introduce stochasticity and could be useful for generating diverse candidates for voting or reranking, but the paper's focus is on single-pass accuracy without such post-hoc aggregation.

Why One Epoch of AMPS Pretraining? The paper provides empirical justification indirectly: the main comparison is between AMPS-pretrained models and non-AMPS-pretrained models of various sizes, and the finding that a 0.1B AMPS-pretrained model matches a 13B non-AMPS-pretrained model demonstrates the value of one epoch of domain-specific pretraining. Multiple epochs could potentially provide further gains (as standard in language model pretraining), but the paper does not explore this, likely because one epoch already saturates the benefit from a dataset of this size relative to the model capacities studied.

Why the Specific Upsampling/Downsampling Ratios? The 5× upsampling of Khan Academy and 2× downsampling of Mathematica is a heuristic to balance quality and quantity. The paper doesn't provide an ablation, but the logic is clear: Khan Academy problems are human-designed, pedagogically structured, and have verified step-by-step solutions. Mathematica problems are algorithmically generated, potentially more repetitive and less pedagogically varied. Upsampling the higher-quality data and downsampling the lower-quality data is a standard technique in dataset mixing when quality varies across sources.

Why 8 Few-Shot Examples for GPT-3? The number 8 is chosen "due to space"—GPT-3's context window at the time was 2048 tokens, and fitting both problems and their formatted answers plus the test problem forced a limit on the number of examples. The paper prioritizes showing the correct answer format and the expected output style over providing many examples, consistent with few-shot methodology where a small number of high-quality demonstrations often suffices.

Why Report Both Few-Shot and Fine-Tuned GPT-3 Results? The two settings test different capabilities: few-shot tests whether the model's pretrained knowledge can be extracted through in-context examples without weight updates, while fine-tuning tests the model's ability to adapt its parameters to the specific answer format and problem distribution of MATH. The poor few-shot performance (3.0% for GPT-3 13B) despite the model's massive scale highlights that MATH requires more than general language understanding—it requires precise compliance with formatting conventions and mathematical operations that are not well-represented in general pretraining data.

Why Evaluate on Partial Solutions? The experiment in Figure 5—where models are given 0% to 99% of the ground-truth solution as a "hint"—serves a specific diagnostic purpose. If models could perfectly leverage provided reasoning traces, they would achieve near-100% accuracy when given 99% of the solution (since the final answer is usually a direct consequence of the final step). The observed accuracy of only ~40% at 99% solution visibility indicates that the model cannot consistently identify the final answer even when it has seen the entire derivation—suggesting a fundamental difficulty in extracting answers from mathematical derivations, not just in generating the derivations themselves. This experiment is performed during both fine-tuning and evaluation, meaning the model is trained to expect partial solutions as input, so the limitation is not a distribution-shift artifact.

4. Key Insights and Innovations

Innovation 1: MATH Operationalizes "Problem-Solving" as Distinguished from "Calculation," Creating a Benchmark That Withstands Scaling

The most consequential intellectual move in this paper is not the dataset itself but the diagnostic distinction it draws between two fundamentally different kinds of mathematical competence: calculation (applying a known operation to given inputs) and problem-solving (selecting, combining, and chaining heuristics from a large space of possibilities to produce a novel solution path). Prior mathematical reasoning benchmarks—the DeepMind Mathematics dataset (Saxton et al., 2019), symbolic integration (Lample and Charton, 2020), MathQA (Amini et al., 2019)—conflated these categories by testing whether models could learn input-output mappings for well-defined operation types. A model achieving 95% accuracy on function integration or arithmetic with large numbers demonstrates, in the authors' framing, that Transformers can serve as neural calculators—impressive but categorically distinct from mathematical reasoning. The MATH dataset is constructed from competition problems (AMC 10/12, AIME) specifically because such problems cannot be solved by pattern-matching a learned operation template: they require recognizing which of many possible approaches applies, adapting general techniques to specific problem structures, and maintaining logical coherence across multiple inferential steps.

This distinction matters beyond mathematics. The paper positions mathematical problem-solving as a proxy for domain-general reasoning—the ability to navigate a combinatorially large space of possible actions using heuristics, to recognize when a promising approach is failing and switch strategies, and to verify that intermediate steps are logically sound. If this capability is fundamentally different from the pattern-completion that large language models excel at (and the paper's results strongly imply it is), then MATH serves as a stress test for the "scaling solves everything" hypothesis that was ascendant in 2021. The finding that a 15× increase in parameters (0.1B → 1.5B GPT-2) yields only a 28% relative improvement in accuracy (5.4% → 6.9%, Table 2), while the same scaling on natural language tasks often produces qualitative capability jumps, provides the first concrete evidence that some reasoning capabilities may lie outside the reach of simply training larger Transformers on more data. The extrapolation to ~10³⁵ parameters for 40% accuracy—whether taken literally or as a rhetorical device—is a direct challenge to the scaling-laws orthodoxy that dominated NLP at the time.

This innovation is fundamental, not incremental. It doesn't improve an existing approach; it redefines what "progress on mathematical reasoning" means by constructing a benchmark where the standard approach (scaling) demonstrably fails. The contrast with prior datasets is stark: Figure 2 shows state-of-the-art accuracy on HOList, HOLStep, DeepMind Math, and Symbolic Integration ranging from ~60% to near-100%, while MATH sits at 6.9%. This gap isn't merely quantitative—it reflects a qualitative difference in what the benchmarks measure. MATH is designed so that even the best models of 2021 perform at levels comparable to random guessing on hard problems (Level 5 accuracy ~4%, Figure 9), creating headroom for research that would not exist if the benchmark were already saturating.

Innovation 2: Step-by-Step Solutions as Training Data Improve Performance, but as Model-Generated "Scratch Space" They Degrade It—Revealing a Fundamental Asymmetry

The paper's most surprising and theoretically revealing finding is an asymmetry between consuming reasoning traces and producing them. When models are trained on step-by-step solutions (the mixed-format fine-tuning with an equal mix of final-answer and full-solution sequences), accuracy improves by approximately 0.6 percentage points—a 10% relative gain over training on final answers alone. When models are provided partial ground-truth solutions as input hints during inference (Figure 5), accuracy climbs monotonically with the fraction of the solution shown, reaching roughly 40% when given 99% of the derivation. Yet when models are asked to generate their own step-by-step solutions before producing an answer—using the full-solution prompting format—accuracy decreases from 6.9% to 5.3% relative to directly outputting the final answer.

This asymmetry is significant because it contradicts the natural hypothesis that "thinking out loud" should help. The standard intuition—drawn from human cognition, where writing down intermediate steps externalizes working memory and enables error-checking—suggests that generating intermediate tokens gives the model more serial computation to solve the problem, analogous to chain-of-thought reasoning. The paper's negative result demonstrates that this intuition fails when the model's own generated tokens are unreliable: a partially incorrect derivation conditions all subsequent tokens on a flawed premise, creating a "snowballing effect" where early errors compound rather than being caught and corrected. The model has not learned to verify its own intermediate steps—it generates them as it would any other text, without the metacognitive capacity to recognize when a step contradicts the problem's constraints or produces an implausible intermediate result.

This finding reframes the problem of teaching models to reason mathematically. It suggests that the bottleneck is not the model's ability to follow a reasoning chain (it can—the partial-solution experiment proves this) but its ability to generate a self-consistent chain and monitor it for errors. The dataset's value proposition shifts accordingly: MATH's step-by-step solutions are valuable primarily as training targets (teaching the model what correct reasoning looks like) and as diagnostic tools (probing where in a derivation the model fails), not as a test-time strategy that current models can productively deploy. This insight has aged well: subsequent work on chain-of-thought prompting (Wei et al., 2022) found that larger models can benefit from generating intermediate steps, suggesting the negative result here may be scale-dependent—but the paper's identification of the snowballing failure mode and the asymmetry between consuming and producing reasoning remains a foundational diagnostic concept for the field.

This is a conceptual innovation rather than a methodological one: it doesn't propose a new technique but identifies a previously unarticulated failure mode that any future approach must address. The fact that the failure manifests even in the model's own training domain (it was fine-tuned on MATH solutions) rules out simple distribution-shift explanations and points to a deeper limitation in autoregressive generation for multi-step reasoning.

Innovation 3: Mathematics-Specific Pretraining (AMPS) Provides a 130× Parameter Multiplier, Demonstrating That Domain Knowledge Dominates General Capacity for Specialized Reasoning

The paper's finding that a 0.1B parameter GPT-2 model pretrained on AMPS achieves 5.4% accuracy—comparable to a 13B parameter GPT-3 fine-tuned on MATH without AMPS pretraining (5.2%)—is more than an impressive ablation result. It recalibrates the relationship between scale and domain knowledge by showing that targeted pretraining data can substitute for a 130× increase in parameters. This has profound implications for how the field thinks about capability acquisition in language models: for mathematically-intensive tasks, the quality and relevance of pretraining data appear to dominate raw model capacity by a factor of over two orders of magnitude.

This finding is particularly striking because the 13B GPT-3 model is not a weak baseline—it has been pretrained on hundreds of billions of tokens of diverse internet text, including some mathematical content. The fact that its mathematical reasoning capability, even after MATH-specific fine-tuning, can be matched by a model with 0.8% of its parameters trained on 23 GB of curated mathematical content suggests that mathematical knowledge is surprisingly sparse in general web text. The paper's ablation comparing AMPS-only pretraining (6.2% accuracy) to AMPS + Math StackExchange pretraining (6.0% accuracy) reinforces this: adding diverse real-world mathematical discussion decreases performance, implying that noisy or incorrectly answered mathematics is worse than no additional mathematics at all. The StackExchange finding is a clean demonstration that data quality and correctness matter more than data quantity and diversity for mathematical reasoning, a principle that has direct implications for data curation in specialized domains.

This innovation is incremental in concept (domain-adaptive pretraining was well-established by 2021; see Gururangan et al., 2020) but fundamental in its quantitative implications. The 130× multiplier is not a small efficiency gain—it's a regime change that would justify building domain-specific pretraining corpora for any specialized reasoning domain where general internet text provides insufficient signal. The paper's construction of AMPS—combining human-designed pedagogical exercises (Khan Academy) with algorithmically-generated problems that exercise the computer algebra system's capabilities (Mathematica)—provides a template for how to build such corpora: use human-curated data for quality and pedagogical structure, use algorithmic generation for scale and coverage, and carefully balance the two so that quality isn't drowned out by quantity.

Innovation 4: Difficulty- and Subject-Level Stratification Reveals That Model Failures Are Systematic and Interpretable—Not Random Errors

While the headline result—6.9% aggregate accuracy—conveys that MATH is hard, the paper's more analytically valuable contribution is the fine-grained error analysis enabled by MATH's dual categorization by subject and difficulty level. Figure 9 shows that model accuracy decreases monotonically with human-assigned difficulty, from ~15% on Level 1 problems to ~4% on Level 5 problems, confirming that the difficulty labels capture a property that affects both human and machine performance. Figure 10 breaks this down further by subject × difficulty, revealing substantial variation: some subject-difficulty combinations show markedly different accuracy than others at the same difficulty level, indicating that model capability is not a single scalar but varies with the type of mathematical content.

This stratification transforms MATH from a single-number benchmark into a diagnostic instrument. Rather than asking "how good is this model at math?", researchers can ask "at what difficulty level does the model's accuracy drop below a useful threshold?" or "which mathematical subdomains benefit most from a given training intervention?" The finding that models achieve nonzero accuracy even on Level 5 problems (~4%, or about 1 in 25 correct) is itself informative: even on the hardest problems, models are not guessing uniformly at random but occasionally produce correct answers through some combination of partial reasoning and luck. This creates a gradient of difficulty where different research approaches can be evaluated at different points—techniques that help on Level 3 problems may be different from those that help on Level 5, and MATH's structure makes it possible to measure this.

The error detection analysis (AUROC of 68.8% for distinguishing correct from incorrect answers based on model confidence) adds another diagnostic layer: models cannot reliably assess their own correctness, producing highly confident wrong answers with similar token-level probabilities as correct ones. This is a practically important finding—it means that simple confidence thresholding cannot be used to filter out incorrect answers in a deployment setting—and it suggests that metacognitive calibration (knowing when you're likely wrong) is a distinct capability that MATH can be used to measure.

This innovation is architectural rather than algorithmic: it's about how the benchmark is structured to support scientific inquiry, not about any specific technique. The dual subject-difficulty taxonomy, the inclusion of step-by-step solutions for error tracing, and the answer formatting that enables exact-match evaluation are design choices that collectively make MATH a research platform rather than merely a leaderboard. This is the kind of benchmark design that enables the systematic, difficulty-conditioned analysis of test-time compute strategies that later work (including the compute-optimal scaling paper analyzed in the Reference Example) would build upon.

5. Experimental Analysis

Evaluation Methodology

Dataset. The MATH dataset consists of 12,500 competition mathematics problems sourced from contests including AMC 10, AMC 12, and AIME, accessed via the Art of Problem Solving website. The paper uses a 7,500/5,000 train/test split. All problems include a final boxed answer and a full step-by-step solution in LaTeX. Problems are tagged across seven subjects (Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, Precalculus) and five difficulty levels (1–5, following AoPS conventions), where Level 1 is easiest and Level 5 corresponds to AIME-level difficulty.
Base model(s). The primary models evaluated are autoregressive decoder-only Transformers from the GPT-2 family at four scales: 0.1B, 0.3B, 0.7B, and 1.5B parameters. GPT-3 is evaluated at approximately 13B parameters (Curie) and 175B parameters (Davinci) via the OpenAI API. BART-Large (0.4B parameters) is tested in Appendix A.5 as an architectural alternative. T5 was attempted but abandoned because its SentencePiece tokenizer strips many LaTeX symbols—the paper reports that after "a broad hyperparameter sweep lasting two weeks, its performance was not competitive." The GPT-2 models are chosen because their tokenizer processes numbers digit-by-digit (following Henighan et al., 2020), which provides fine-grained numerical representation, and because they represent the standard paradigm for text generation at the time.
Metrics. The primary metric is accuracy: the fraction of test-set problems where the model's extracted and normalized answer exactly matches the ground-truth answer. Answer extraction parses the text between \boxed{} delimiters in the model's output. Normalization handles equivalent fraction representations (e.g., \frac{1}{2} ≡ 1/2 ≡ 0.5), polynomial factor orderings (e.g., 4(x+1)(x-1) ≡ 4(x-1)(x+1)), coefficient formatting (5x ≡ 5*x), variable ordering conventions, and spacing/parenthesis variations. The paper also reports accuracy broken down by subject, difficulty level, and subject × difficulty combinations. An auxiliary metric is the AUROC for error detection based on model confidence, where confidence is defined as the average prediction probability of the tokens constituting the generated answer. AUROC of 100% means perfect separation of correct and incorrect answers by confidence; 50% means random.
Baselines. There is no single dominant baseline at the time of the paper's writing—the central finding is that MATH is a new benchmark where all models perform poorly. The paper's comparisons are primarily across model scales and training configurations: (1) GPT-2 models pretrained on AMPS then fine-tuned on MATH, compared against GPT-2 models fine-tuned on MATH without AMPS pretraining (though this latter result is not reported as a separate table row—the 0% baseline in Table 2 is the smallest AMPS-pretrained model); (2) GPT-3 evaluated few-shot (8 in-context examples with answers, no gradient updates) versus fine-tuned on MATH (without AMPS pretraining, due to API limitations); (3) BART evaluated against GPT-2 of similar scale; (4) StackExchange-pretrained models compared against AMPS-pretrained models as an ablation of pretraining data source quality.
Generation budget / compute accounting. Compute is measured implicitly through model parameter count—there is no test-time compute budget scaling analysis of the kind in the Reference Example paper. The key cost axis is total training FLOPs (AMPS pretraining + MATH fine-tuning), not inference-phase generation budget. At inference, beam search with beam sizes 10–20 is used, so per-problem compute scales linearly with beam size, but this is held constant across experiments rather than being systematically varied. The paper does not report FLOPs or wall-clock time for training or inference, only noting that training uses "8 A100 GPUs, each requiring less than a day."
Cross-validation / statistical protocol. No cross-validation or statistical significance testing is reported. The test set is a fixed 5,000-question split. All results are single-run accuracies computed on this split. The human evaluation uses a small random sample of 20 problems from the test set with 6 participants, explicitly described as providing "a rough but informative comparison to human-level performance" rather than a rigorous psychometric baseline. There is no confidence interval reporting or multiple-seed averaging for any model result.

Main Quantitative Results

Aggregate Performance on MATH and the Failure of Scale

The central result appears in Table 2: accuracy on the MATH test set across model sizes, subjects, and training configurations. The headline numbers:

GPT-2 0.1B (AMPS pretrained + MATH fine-tuned): 5.4% average across subjects. This is the smallest model tested with AMPS pretraining and serves as the reference baseline. Subject-level performance ranges from 2.8% (Counting and Probability) to 7.3% (Precalculus), indicating substantial variation across mathematical domains even at this low overall accuracy.
GPT-2 1.5B (AMPS pretrained + MATH fine-tuned): 6.9% average. This represents a 15× increase in parameters over the 0.1B model, yielding only a 1.5-percentage-point absolute improvement—a 28% relative improvement. Subject-level performance ranges from 4.8% (Number Theory) to 8.8% (Precalculus). The slow growth with scale is the paper's core empirical claim: simply making models larger provides diminishing returns on MATH, contrary to the log-linear scaling observed on most text-based tasks.
GPT-3 13B (few-shot, no AMPS pretraining, no MATH fine-tuning): 3.0% average. This result—marked with an asterisk in Table 2 to indicate few-shot evaluation—demonstrates that in-context learning from 8 examples is dramatically insufficient for MATH. The model performs worse than the smallest GPT-2 model with AMPS pretraining and MATH fine-tuning, despite having 130× more parameters and orders-of-magnitude more general pretraining data.
GPT-3 13B (MATH fine-tuned, no AMPS pretraining): 5.6% average. This is the most direct comparison to the AMPS-pretrained GPT-2 models: fine-tuning on MATH without domain-specific pretraining yields accuracy roughly comparable to a 0.1B GPT-2 model with AMPS pretraining (5.4%), despite a 130× parameter advantage. The paper frames this as AMPS providing "about as much accuracy increase as a 130× increase in parameters."
GPT-3 175B (few-shot, no AMPS pretraining, no MATH fine-tuning): 5.2% average. The largest model tested, evaluated purely few-shot, performs only slightly better than the fine-tuned 13B version and slightly worse than the fine-tuned 0.1B AMPS-pretrained GPT-2. This is the starkest evidence that general scale does not compensate for domain-specific pretraining and task-specific fine-tuning.

Table 2 also reports a row for "GPT-3 175B*" at 5.2% (few-shot) and "GPT-3 13B*" at 3.0%. These are not directly comparable to the fine-tuned rows because they use a different evaluation protocol (few-shot prompting with 8 examples + answer versus prompting with "Final Answer:" after fine-tuning).

Subject-level variation (Table 2, columns): Across all GPT-2 models, Geometry consistently shows the highest accuracy (5.7–8.7%), while Counting and Probability consistently shows among the lowest (2.8–5.4%). Precalculus and Prealgebra trade off for the top position depending on model size. This subject-level stratification reveals that "mathematical reasoning" is not monolithic—some mathematical domains are substantially more learnable than others at a given scale.

Difficulty-Level Analysis: Human Difficulty Tracks Model Difficulty

Figure 9 plots accuracy against MATH's five difficulty levels for GPT-2 0.1B and GPT-2 1.5B. The results:

Level 1 (easiest): GPT-2 1.5B achieves approximately 15% accuracy. GPT-2 0.1B achieves approximately 10%.
Level 5 (hardest): Both models achieve approximately 3–4% accuracy, with the 1.5B model showing only marginal advantage over the 0.1B model despite being 15× larger.

The monotonic decrease in accuracy with human-assigned difficulty confirms that MATH's difficulty labels capture a property that affects both human and machine performance—they are not merely artifacts of human problem-solving style. The collapse to near-random performance at Level 5 for both model sizes indicates a capability ceiling that additional parameters do not overcome.

Figure 10 breaks this down further into a subject × difficulty heatmap-like display (shown as a grouped bar chart across 7 subjects × 5 levels = 35 categories). The paper reports:

"accuracy can vary by level and subject substantially."

Specific observations visible in Figure 10: Level 1 Prealgebra for GPT-2 1.5B reaches approximately 30–35% accuracy—the highest single-subject-difficulty result in the paper—while Level 5 problems across most subjects for GPT-2 0.1B are at or below 5%. The interaction between subject and difficulty is not purely additive: some subjects show steeper difficulty degradation than others.

Problem and Solution Length as Difficulty Proxies

Figures 11a and 11b examine correlations between problem/solution length and model accuracy:

Problem length vs. accuracy (Figure 11a): Each point represents a subject at a specific difficulty level (excluding problems with Asymptote figures). There is a downward trend—longer problems are generally harder—but substantial scatter, indicating that length alone is an imperfect difficulty predictor. Precalculus Level 1 problems are among the longest but also among the highest-accuracy, showing that length does not uniformly predict difficulty across subjects.
Solution length vs. accuracy (Figure 11b): Similarly structured. Longer ground-truth solutions are loosely correlated with lower accuracy. The paper's framing is that "problems with long questions or ground truth solutions indeed tend to be more difficult than problems with short questions or solutions."

These analyses are descriptive rather than causal—they don't claim that length causes difficulty, only that length is a coarse signal that correlates with the difficulty labels and with model accuracy.

AMPS Pretraining: A 130× Parameter Multiplier

The paper's most striking quantitative comparison (Section 4.2, AMPS Pretraining subsection):

"Without pretraining on AMPS, a GPT-3 (13B) model fine-tuned on MATH attains 5.2% accuracy. In contrast, a GPT-2 (0.1B) model both pretrained on AMPS and fine-tuned on MATH attains 5.4%. Consequently AMPS increases accuracy about as much as a 130× increase in parameters."

This comparison involves two models that differ in: (1) base model architecture (GPT-3 vs. GPT-2), (2) parameter count (13B vs. 0.1B), (3) general pretraining data (whatever GPT-3 was trained on vs. whatever GPT-2 was trained on), and (4) AMPS pretraining (absent vs. present). Despite these confounds, the paper isolates AMPS as the explanatory variable, arguing that domain-specific pretraining data dominates parameter count by a factor of 130× for this task.

The StackExchange ablation strengthens this interpretation: a GPT-2 0.3B model pretrained on AMPS plus Math StackExchange (~3 GB of questions and answers) achieves 6.0% accuracy, which is worse than the 6.2% achieved by pretraining on AMPS alone. The paper's interpretation is that AMPS is "more useful for pretraining even than diverse real-world mathematics data," implying that noisy or incorrectly-answered mathematical content can actively degrade performance relative to training on a smaller, cleaner, curriculum-structured corpus.

Step-by-Step Solutions: Positive During Training, Negative During Inference

Training with solutions improves accuracy. The paper reports an ablation in Section 4.3 (The Benefits of MATH Solutions):

"If we fine-tune with only the final answer, the GPT-2 (1.5B) accuracy decreases by 0.6% to 6.3%."

This is compared to the standard mixed-format fine-tuning (equal mix of final-answer-only and full-solution sequences), which achieves 6.9%. The 0.6-percentage-point absolute improvement represents approximately a 10% relative gain from including solutions during training. The paper treats this as evidence that "providing models with step-by-step solutions at training time also increases accuracy."

Generating solutions at inference time decreases accuracy. In Section 4.3 (Scratch Space):

"We evaluated this for GPT-2 (1.5B) and found that this actually makes performance worse, dropping accuracy to 5.3%."

This is compared to the standard final-answer-only prompting, which achieves 6.9%. The 1.6-percentage-point drop is larger in absolute magnitude than the gain from training with solutions. The paper hypothesizes a "snowballing effect, in which partially generated 'solutions' with mistakes can derail subsequent generated text."

The combined finding—that consuming solutions helps but producing them hurts—is the paper's most theoretically significant empirical result. It demonstrates an asymmetry between supervised learning from correct reasoning traces (effective) and autonomous generation of reasoning traces (counterproductive at this model scale), which the paper uses to argue that current models lack metacognitive verification capabilities.

Partial Solution "Hints" Improve Performance

Figure 5 shows accuracy when models are conditioned on varying fractions of the ground-truth step-by-step solution before generating the final answer. This experiment uses GPT-2 0.7B, with the model fine-tuned and evaluated using prompts of the form "⟨P⟩ <Partial Step-by-Step Solution without Final Answer> Final Answer:".

Key data points from Figure 5:

0% of solution (the standard evaluation setting): approximately 6–7% accuracy, consistent with the 0.7B model's performance in Table 2.
~50% of solution: accuracy rises to approximately 15–20%.
99% of solution (the entire derivation except the final answer): accuracy reaches approximately 40%.

The monotonic improvement demonstrates that models can leverage provided reasoning traces—as they receive more of the correct derivation, they become increasingly able to produce the final answer. However, the 40% accuracy at 99% solution visibility means that even with nearly the entire correct reasoning chain provided, the model fails to extract the correct answer 60% of the time. The paper interprets this as evidence that the model struggles to identify the logical consequence of the final reasoning step, not just to generate the derivation. This experiment uses the ground-truth solutions from MATH, so the "hints" are perfectly correct—ruling out the possibility that noisy or incorrect intermediate reasoning causes the remaining failures.

Error Detection: Models Cannot Reliably Assess Their Own Correctness

For GPT-2 1.5B:

The model is "highly overconfident, with confidences that are often around 100%."
The AUROC for distinguishing correct from incorrect answers based on confidence is 68.8%.

An AUROC of 68.8% provides some signal above random chance (50%), but is far from what would be needed for reliable error detection in a practical system. The combination of high absolute confidence even on wrong answers (overconfidence) and poor separation between correct and incorrect answer confidence distributions (low AUROC) indicates that the model's internal probability estimates are poorly calibrated for this task—a finding that is consistent with broader literature on neural network calibration but demonstrated here specifically for mathematical reasoning.

Few-Shot vs. Fine-Tuned GPT-3: In-Context Learning Is Insufficient

The GPT-3 rows in Table 2 (marked with asterisks for few-shot) show:

GPT-3 13B few-shot: 3.0% (vs. 5.6% fine-tuned — a 46% relative improvement from fine-tuning)
GPT-3 175B few-shot: 5.2% (vs. no fine-tuned version available due to API restrictions)

The few-shot GPT-3 175B model, with approximately 1,166× more parameters than GPT-2 0.1B, achieves lower accuracy (5.2%) than the smallest AMPS-pretrained plus MATH-fine-tuned model (5.4%). This comparison reinforces the paper's argument that domain-specific pretraining and task-specific fine-tuning are more impactful than raw parameter count for mathematical reasoning.

The paper also reports that GPT-3 175B performs well on other reasoning-like tasks: approximately 40% on the hardest C-Test examples (where average humans score ~20%) and that scaling trends on LogiQA (Figure 6) suggest human-level performance will be reached "relatively soon." These results contextualize MATH's difficulty: tasks that appear to require logical reasoning are in fact yielding to scale, while MATH resists. This sharpens the paper's claim that MATH tests something qualitatively different from existing reasoning benchmarks.

BART Architecture: No Improvement from Bidirectional Encoding (Appendix A.5)

BART-Large (0.4B parameters), after AMPS pretraining and MATH fine-tuning, achieves 4.9% accuracy, compared to GPT-2 0.1B at 5.4%. The paper concludes that "models with a bidirectional encoder and unidirectional decoder do not yield marked changes in MATH accuracy." This is a negative result that suggests the bottleneck is not architectural (encoder-decoder vs. decoder-only) but rather something more fundamental about the model's ability to learn mathematical reasoning from the available training data.

Ablation Studies and Robustness Checks

AMPS pretraining vs. no AMPS pretraining: The comparison between GPT-3 13B fine-tuned without AMPS (5.2%) and GPT-2 0.1B with AMPS (5.4%) demonstrates that AMPS effectively substitutes for a 130× increase in parameters. This is reported in Section 4.2 under "AMPS Pretraining" and is the central evidence for the claimed parameter multiplier. However, it is not a clean ablation—it compares different model families (GPT-3 vs. GPT-2), different general pretraining data, and different parameter counts—so the 130× figure is a heuristic comparison rather than a controlled measurement.

Training with solutions vs. training with answers only: Removing step-by-step solutions from the MATH fine-tuning data (training only on "⟨P⟩ Final Answer: ⟨Answer⟩" sequences, without any full-solution examples) reduces GPT-2 1.5B accuracy from 6.9% to 6.3% (Section 4.3, The Benefits of MATH Solutions). This is a clean within-model ablation confirming that step-by-step solutions provide a small but reliable training benefit (approximately 10% relative improvement).

Generating solutions vs. generating answers directly at test time: Prompting GPT-2 1.5B with "Full Solution:" instead of "Final Answer:" reduces accuracy from 6.9% to 5.3% (Section 4.3, Scratch Space). This uses the same model, the same test set, and differs only in the prompting prefix, making it a clean comparison. The result is robust—it rules out the hypothesis that generating intermediate steps is beneficial for models at this scale.

AMPS + StackExchange vs. AMPS alone: A GPT-2 0.3B model pretrained on both AMPS and Math StackExchange achieves 6.0% accuracy, compared to 6.2% with AMPS alone (Section 4.2, AMPS Pretraining). This is a counterintuitive negative result: adding more diverse, real-world mathematical data reduces performance. The paper's explanation—that StackExchange data is noisier and may contain incorrect answers—is plausible but not formally tested (e.g., by filtering StackExchange for high-quality answers or comparing different StackExchange subsets).

Khan Academy upsampling and Mathematica downsampling: The pretraining data mixture uses 5× upsampling of Khan Academy and 2× downsampling of Mathematica (Section 4.1). No ablation is provided for this ratio. This is a significant unexamined design choice—it is possible that different mixing ratios would yield substantially different downstream accuracy, but the paper treats the chosen ratio as fixed.

Beam size for generation: The paper uses beam size 20 for final-answer generation and beam size 10 for full-solution generation (Section 4.1). No ablation compares different beam sizes. This matters because beam search can have complex interactions with accuracy—larger beams explore more completions but may also amplify model errors by selecting high-probability but incorrect sequences. The paper's choice is motivated by "default HuggingFace generation parameters, except that we use beam search," implying that beam size was chosen as a reasonable default rather than systematically optimized.

One epoch of AMPS pretraining: The paper pretrains for exactly one epoch on AMPS. No ablation compares different numbers of pretraining epochs (or different fractions of the AMPS data). This is a practical constraint choice—the dataset is large (23 GB), and one epoch already provides substantial benefit—but means the paper does not establish whether additional pretraining epochs would improve or degrade performance.

GPT-3 few-shot vs. fine-tuned (13B only): The direct comparison between few-shot (3.0%) and fine-tuned (5.6%) GPT-3 13B demonstrates that task-specific fine-tuning provides substantial gains over in-context learning. However, GPT-3 175B is only evaluated few-shot (5.2%), so there is no evidence on whether fine-tuning the larger model would yield proportionally larger improvements. This is a missing data point imposed by API limitations at the time.

Partial solution hints (Figure 5): This experiment—providing models with 0% to 99% of the ground-truth solution—shows that accuracy increases with more solution context, reaching approximately 40% at 99% visibility. The model used is GPT-2 0.7B, which achieves ~6–7% without hints. This is a strong demonstration that the model can utilize correct reasoning when provided, but it is not an ablation in the traditional sense—it changes the input the model conditions on, making it a probe of model capability rather than a comparison of training configurations.

Tokenizer choice (T5 vs. GPT-2 vs. BART): The failure of T5 (due to LaTeX-symbol-stripping tokenization) and the underperformance of BART (4.9% vs. GPT-2's 5.4% at comparable sizes) collectively demonstrate that architectural and tokenization choices matter for mathematical reasoning. This is reported informally—"after a broad hyperparameter sweep lasting two weeks, its performance was not competitive"—rather than with systematic ablation tables. The paper does not report what specific T5 accuracies were achieved, only that they were non-competitive.

Difficulty and length correlations (Figures 11a, 11b): These figures examine whether problem/solution length alone explains difficulty. The scatter plots show weak to moderate correlations with substantial variance, suggesting that length is a noisy proxy for difficulty at best. This is a robustness check against the hypothesis that MATH's difficulty structure is reducible to text length (i.e., that the benchmark isn't measuring reasoning but merely processing longer inputs).

Critical Assessment

Claim: "Accuracy remains relatively low, even with enormous Transformer models" (Abstract, Section 4.2)

What the experiments demonstrate: The best model tested (GPT-2 1.5B with AMPS pretraining) achieves 6.9% accuracy on MATH. GPT-3 175B in a few-shot setting achieves 5.2%. These numbers are indeed low in absolute terms and relative to model performance on other text-based tasks where Transformers approach or exceed human-level performance.

What is not demonstrated: The largest model evaluated with both AMPS pretraining and MATH fine-tuning is GPT-2 1.5B. The 175B model is only evaluated few-shot, without AMPS pretraining, and without the paper's full fine-tuning protocol. The claim that MATH remains difficult "even with enormous Transformer models" is supported for few-shot evaluation of the largest model (175B) and for fine-tuned evaluation of medium-sized models (up to 1.5B), but there is a substantial gap: no model larger than 1.5B was evaluated under the full pipeline (AMPS pretraining + MATH fine-tuning + beam search decoding). It is unknown whether GPT-3 175B, if it could have been AMPS-pretrained and MATH-fine-tuned, would have achieved substantially higher accuracy than 6.9%. The API limitations that prevented this are noted in the paper, but the claim implicitly extrapolates to "enormous Transformers" in general, which requires the (untested) assumption that the trend from 0.1B → 1.5B (only 28% relative improvement for 15× scale) would continue to 175B. This is a reasonable extrapolation but is an extrapolation, not a direct measurement.

Claim: "Simply increasing budgets and model parameter counts will be impractical for achieving strong mathematical reasoning if scaling trends continue" (Abstract, Section 5)

What the experiments demonstrate: The accuracy improvement from GPT-2 0.1B (5.4%) to GPT-2 1.5B (6.9%) is modest—a 1.5-percentage-point absolute gain for a 15× parameter increase. The extrapolation that achieving 40% accuracy would require ~10³⁵ parameters relies on assuming a log-linear relationship between parameter count and accuracy, which the paper acknowledges is an extrapolation. The paper states: "assuming a log-linear scaling trend, models would need around 10³⁵ parameters to achieve 40% accuracy on MATH."

Weaknesses in this extrapolation: (1) Only three GPT-2 data points (0.1B, 0.3B, 0.7B, 1.5B) are available for fitting the trend, and these four points span less than two orders of magnitude in parameter count. Extrapolating over 30 orders of magnitude from four points within a factor of 15 is extremely fragile—any curvature in the scaling law (and there almost certainly is curvature, given that accuracy is bounded above by 100%) would dramatically change the extrapolation. (2) The extrapolation assumes that the AMPS pretraining + MATH fine-tuning pipeline would continue to work at larger scales, but models with larger capacity might learn substantially more from the same fine-tuning data, potentially breaking the log-linear trend. (3) The paper shows that AMPS pretraining provides a 130× parameter multiplier (0.1B + AMPS ≈ 13B without AMPS), suggesting that data quality interventions can dramatically shift the effective scaling curve—the extrapolation to 10³⁵ parameters does not account for the possibility that better pretraining data, longer fine-tuning, or algorithmic improvements could steepen the scaling trend.

The extrapolation is best understood as a rhetorical claim about the insufficiency of scale alone rather than a precise quantitative prediction. The experiments genuinely support the qualitative point—scaling from 0.1B to 1.5B produces meager gains—but the specific 10³⁵ figure is speculative.

Claim: "Having models generate step-by-step solutions before producing an answer actually decreased accuracy" (Section 4.3, Abstract)

What the experiments demonstrate: GPT-2 1.5B prompted with "Full Solution:" achieves 5.3% accuracy, compared to 6.9% with "Final Answer:"—a 1.6-percentage-point drop. This is a clean, within-model, within-test-set comparison. The paper's explanation (snowballing errors) is plausible and consistent with the qualitative examples in Figures 3 and 4, which show generated solutions that contain logical errors and produce wrong answers.

What is not demonstrated: (1) Only one model scale (1.5B) is tested for this comparison. It is possible that larger models would benefit from solution generation rather than being harmed by it—a hypothesis that later work on chain-of-thought (Wei et al., 2022) would confirm in models at the 100B+ scale. This is not a flaw in the paper (which explicitly acknowledges the limitation and frames it as an interesting research direction) but it does bound the generality of the claim: at the model scales studied, generating solutions hurts. The paper cannot claim this holds for all Transformer scales based on their data. (2) The beam search configuration differs between the two prompting modes (beam size 20 for final answer, beam size 10 for full solution), which introduces a confound: part of the accuracy drop could be due to the smaller beam rather than the solution generation itself. The paper does not ablate beam size to separate these effects.

Claim: "Training on solutions increases relative accuracy by 10% compared to training on the questions and answers directly" (Section 1, Abstract)

What the experiments demonstrate: GPT-2 1.5B fine-tuned with solutions achieves 6.9%, compared to 6.3% without solutions—a 0.6-percentage-point absolute difference and approximately 10% relative improvement. This is tested at a single model scale and configuration, but the finding is clean and matches the paper's narrative.

What is not demonstrated: The effect size is small in absolute terms (0.6 percentage points) and is not tested across multiple model sizes. It is unclear whether this 10% relative benefit would compound at larger scales (yielding larger absolute gains) or remain a small fixed improvement. The paper does not analyze why training with solutions helps—whether it provides better regularization, teaches answer formatting, or genuinely improves mathematical reasoning capability—which limits the diagnostic value of the finding.

Claim: "Pretraining on AMPS enables a 0.1 billion parameter model to perform comparably to a fine-tuned model that is 130× larger" (Section 1, Abstract)

What the experiments demonstrate: GPT-2 0.1B with AMPS achieves 5.4%; GPT-3 13B without AMPS achieves 5.2% (fine-tuned) or 5.6% (the paper cites 5.2% in the AMPS Pretraining subsection but Table 2 shows 5.6% for GPT-3 13B fine-tuned—this is a minor inconsistency in reporting, where the comparison appears to use 5.2% as the non-AMPS fine-tuned number even though Table 2 lists 5.6%). The qualitative point—that domain-specific pretraining can compensate for large differences in model scale—is supported.

Limitations: This comparison crosses model families (GPT-2 vs. GPT-3), general pretraining data, and training procedures, making it an observation about the importance of domain-specific data rather than a controlled measurement of the data-to-parameters tradeoff. The 130× figure would be more convincing if it came from same-family models (e.g., GPT-2 0.1B + AMPS vs. GPT-2 13B without AMPS), but no such comparison exists because GPT-2 does not have a 13B variant. Additionally, GPT-3 13B may be undertrained for its size relative to GPT-2 0.1B (the GPT-3 paper reports scaling laws that may prioritize different compute allocations), so part of the performance gap could be due to pretraining efficiency rather than domain relevance.

Missing Experiments That Would Strengthen the Paper

Fine-tuned GPT-3 175B on MATH. The largest model evaluated with fine-tuning is 1.5B parameters. A fine-tuned 175B model—had it been possible via the API at the time—would directly test the claim that scaling provides diminishing returns and would significantly strengthen (or potentially weaken) the paper's central narrative.
Multiple random seeds or confidence intervals. All accuracy numbers are single runs on the fixed 5,000-question test set. With accuracy in the single digits, the expected number of correct answers for the 1.5B model is approximately 345 out of 5,000. The standard deviation of this binomial count is approximately 18, corresponding to roughly 0.36 percentage points in accuracy. Differences between model configurations (e.g., 6.9% vs. 6.3%) are small relative to this variability, and without confidence intervals, it is impossible to assess whether apparent differences are statistically reliable. Reporting accuracy to one decimal place (e.g., 6.9%) implies precision that the experimental design does not support.
AMPS pretraining for GPT-3. The paper's central finding about AMPS rests on a cross-family comparison. If GPT-3 had been AMPS-pretrained (which was infeasible due to API limitations), the comparison would be much cleaner. The paper acknowledges this limitation in the GPT-3 section but does not address how it affects the strength of the 130× multiplier claim.
Fine-tuned model without any AMPS pretraining. There is no row in Table 2 for a GPT-2 model fine-tuned on MATH without AMPS pretraining. Such a baseline would directly measure the contribution of AMPS within the same model family, removing the cross-family confound. The paper reports that StackExchange pretraining (without AMPS) gave lower accuracy, but that is a different ablation.
Scaling the fine-tuning data size. All fine-tuning uses the full 7,500 MATH training problems. Ablating the fine-tuning data size would test whether the model's poor accuracy is due to insufficient task-specific data or a fundamental capability ceiling. If accuracy scales with MATH training data size, then collecting more competition problems would be a practical path forward; if it plateaus, the limitation is in model architecture or pretraining.
Human performance on the full test set. The human evaluation uses 20 randomly sampled problems with 6 participants. While this is described as "rough but informative," a larger human baseline (e.g., 100 problems with diverse participants) would better calibrate the difficulty scale and provide more reliable reference points for model progress.

Where the Claims Hold Conditionally

The claim that MATH resists scaling holds at model sizes up to 1.5B parameters under the full pipeline. Whether it holds at 10B, 100B, or larger is an open question that the paper flags for future work. Subsequent research (including the compute-optimal scaling paper in the Reference Example, which uses PaLM 2 models at much larger scales) would partially fill this gap, but from the paper's own data, the claim is demonstrated only in the small-to-medium model regime.
The claim that generating solutions hurts accuracy holds for GPT-2 1.5B with beam size 10. Whether it holds at larger scales, with different decoding strategies, or with models specifically trained for self-verification, is untested. The paper's own hypothesis—that snowballing errors cause the degradation—implies that if models become better at generating correct intermediate steps, the sign of the effect could flip. This is exactly what later chain-of-thought research would find.
The claim that AMPS provides a 130× parameter multiplier holds for the specific comparison between GPT-2 0.1B (with AMPS) and GPT-3 13B (without AMPS). It should not be interpreted as a general law—the multiplier likely depends on the scale of the base model, the quality and quantity of domain-specific pretraining data, and the nature of the downstream task. A model already saturated with mathematical knowledge from general pretraining would benefit less from AMPS.
The extrapolation to 10³⁵ parameters for 40% accuracy is a projection of a trend fitted to four small-scale models over 15× parameter range. It should be treated as an argument that scale alone is insufficient—which the data genuinely support—not as a precise forecast of what would happen if we actually built a 10³⁵-parameter model.

6. Limitations and Trade-offs

The Benchmark's Difficulty Ceiling Creates a Research Signal, but Leaves the Hardest Problems Essentially Unsolved

The assumption or constraint. The paper's central empirical finding is that model accuracy collapses to near-random performance on the hardest MATH problems. For GPT-2 1.5B—the best model evaluated under the full AMPS pretraining + MATH fine-tuning pipeline—Level 5 accuracy is approximately 4% (Figure 9). The paper treats this as evidence that MATH successfully resists scaling, but it also means that essentially none of the hardest competition problems are being solved. At 4% accuracy on Level 5, a model solves roughly 1 in 25 problems correctly, and with only approximately 1,000 Level 5 problems in the test set (one-fifth of 5,000), this corresponds to roughly 40 correct answers out of 1,000—a number that could plausibly arise from lucky guessing or partial pattern-matching rather than genuine multi-step reasoning.

The consequence. The near-zero performance on hard problems means that MATH cannot distinguish between models or techniques that improve reasoning on the most challenging problems—the floor is too low to measure improvement. If a new training method or architecture raised Level 5 accuracy from 4% to 6%, the difference (approximately 20 additional correct answers in 1,000) would be difficult to detect reliably without confidence intervals or multiple seeds, which the paper does not provide. Worse, if the 4% accuracy reflects essentially random guessing on unsolvable problems rather than partial reasoning, then improvements to "genuine reasoning" might produce zero signal at this difficulty tier until a qualitative capability threshold is crossed. The benchmark's diagnostic value is therefore concentrated at difficulty Levels 1–3, where accuracy is high enough (5–15%) to reveal differences between models and techniques, but the hardest problems—precisely those that most require the "problem-solving heuristics" the paper claims to measure—provide negligible statistical signal.

What evidence exists in the paper. Figure 9 shows the accuracy collapse with difficulty, and Figure 10 further breaks this down by subject × difficulty, confirming that Level 5 problems across nearly all subjects have accuracy at or below approximately 5%. The paper acknowledges the low absolute numbers only indirectly—by noting that "our benchmark's easiest problems are more challenging than previous benchmarks"—but does not discuss the implication that the hardest problems provide almost no measurement signal at current capability levels.

Mitigation status. Not addressed. The paper presents the low accuracy as evidence of MATH's difficulty and as motivation for future work, but does not discuss the measurement-theoretic problem that a benchmark with near-floor performance on its hardest subset cannot characterize progress on the capability it most wants to measure. A partial mitigation would be to increase the number of hard problems in the test set (improving statistical power for detecting small improvements at low accuracy) or to report uncertainty quantification that makes the noise level explicit, but neither is done. The difficulty stratification is presented as an unqualified positive feature of the benchmark design without acknowledging this tradeoff.

The Benchmarked Models Are Small by Contemporary Standards, and the Extrapolation to "Enormous Transformers" Rests on a Single Cross-Family Comparison

The assumption or constraint. The paper's strongest claim—that MATH resists scaling and that "simply using bigger versions of today's Transformers will not solve our task in the foreseeable future" (Section 2)—is supported by direct evidence only for GPT-2 models up to 1.5B parameters (the largest model evaluated under the full AMPS pretraining + MATH fine-tuning + beam search pipeline) and by a cross-family comparison with GPT-3 models evaluated in a weaker protocol. The fine-tuned GPT-3 models (13B) were not AMPS-pretrained because the OpenAI API did not support it, and the GPT-3 175B model was evaluated only few-shot, without any MATH fine-tuning. This creates a substantial evidential gap: no model larger than 1.5B parameters was evaluated under the paper's best training configuration.

The consequence. The extrapolation from four GPT-2 data points (0.1B, 0.3B, 0.7B, 1.5B) over a 15× parameter range to a prediction about 10³⁵-parameter models (Section 5) spans more than 30 orders of magnitude beyond the measured data. This extrapolation treats a trend fitted to models spanning less than two orders of magnitude in parameter count as reliable over 30 orders of magnitude, which is statistically indefensible. More practically, the paper cannot rule out that a model at the 10B–100B scale—which would be "enormous" by the paper's own framing but is vastly smaller than 10³⁵—might achieve substantially higher accuracy, especially if combined with AMPS pretraining and MATH fine-tuning. The finding that AMPS provides a "130× parameter multiplier" (0.1B + AMPS ≈ 13B without AMPS) actually implies that data quality interventions can dramatically shift the effective scaling curve, making extrapolations from the small-scale data-poor regime even less reliable.

What evidence exists in the paper. Table 2 reports all model results. The largest fine-tuned GPT-2 is 1.5B, and the largest GPT-3 model (175B) is only evaluated few-shot at 5.2% accuracy. The paper explicitly notes the API limitation: "their API does not have the option to pretrain on datasets as large as AMPS" (Section 4.1). The scaling trend in Figure 9 uses only GPT-2 0.1B and 1.5B as its endpoints across difficulty levels. The LogiQA extrapolation in Figure 6 provides a contrasting example where scaling does appear to be solving the task—which the paper interprets as evidence that MATH is qualitatively different—but this inference requires the assumption that the GPT-2 scaling trend is representative of all Transformer scaling behavior on MATH, which is untested.

Mitigation status. The paper partially acknowledges the limitation by stating the model sizes tested and the API constraints, and by framing the 10³⁵ extrapolation as conditional ("assuming a log-linear scaling trend"). However, the strength of the rhetorical claims—"simply increasing budgets and model parameter counts will be impractical," "scaling is not currently solving MATH" (Abstract)—goes beyond what the data can support. A fairer characterization would be: at model sizes up to 1.5B parameters under the best available training pipeline, accuracy improves slowly with scale, and even a 175B model with few-shot prompting performs poorly, but whether the full pipeline (AMPS + fine-tuning + beam search) applied to a 10B–100B+ model would break the log-linear trend is an open empirical question that the paper cannot answer. Subsequent work—including the compute-optimal scaling paper in the Reference Example, which studies models at much larger scales—would partially address this gap, but the MATH paper itself does not.

AMPS Pretraining Is Demonstrated as Critical but Its Contribution Is Confounded with Model Family, General Pretraining Data, and Training Procedures

The assumption or constraint. The central evidence for AMPS's value is a cross-family comparison: GPT-2 0.1B pretrained on AMPS + fine-tuned on MATH (5.4%) versus GPT-3 13B fine-tuned on MATH without AMPS (5.2% or 5.6%, depending on where in the paper the number is cited). From this, the paper concludes that AMPS provides "about as much as a 130× increase in parameters" (Section 4.2). This comparison confounds at least four variables: (1) model architecture and training recipe (GPT-2 vs. GPT-3 differ in layer count, width, attention patterns, and training data distribution), (2) general pretraining data (GPT-2 was trained on WebText; GPT-3 on a substantially larger and different corpus), (3) parameter count (0.1B vs. 13B), and (4) AMPS pretraining (present vs. absent). Attributing the performance parity entirely to AMPS requires the assumption that GPT-2 0.1B and GPT-3 13B would perform identically on MATH if both were trained without AMPS—an assumption that is not tested and is almost certainly false given what we know about scaling effects on other tasks.

The consequence. The 130× multiplier figure is likely misleading as a quantitative claim about the substitutability of domain-specific pretraining data for model scale. A within-family comparison—e.g., GPT-2 0.1B with AMPS versus GPT-2 0.1B without AMPS, and GPT-2 1.5B with AMPS versus GPT-2 1.5B without AMPS—would directly measure AMPS's contribution at different scales, but no GPT-2 model without AMPS pretraining is evaluated in Table 2. As a result, the paper cannot establish whether AMPS provides a constant additive benefit (~5 percentage points regardless of scale), a multiplicative benefit (increasing with scale), or a diminishing benefit (most valuable for small models). The qualitative conclusion—that domain-specific pretraining data is highly valuable for mathematical reasoning—is almost certainly correct and has been borne out by subsequent work, but the specific 130× quantification is an artifact of the experimental design rather than a measured property.

What evidence exists in the paper. The AMPS evidence consists of: (1) the cross-family comparison (0.1B + AMPS ≈ 13B without AMPS, Section 4.2), (2) the StackExchange ablation where adding StackExchange data to AMPS reduces accuracy from 6.2% to 6.0% (Section 4.2), and (3) the training-with-solutions ablation where including solutions in MATH fine-tuning improves accuracy from 6.3% to 6.9% (Section 4.3). None of these directly measures AMPS's contribution within a single model family at multiple scales. The paper notes that GPT-3 models "do not pretrain on AMPS due to API limits" (Table 2 caption), but does not discuss how this confound affects the interpretation of the multiplier claim.

Mitigation status. The StackExchange ablation partially addresses the concern by showing that pretraining data quality matters (AMPS alone beats AMPS + noisier data), but it does not isolate AMPS's contribution relative to no domain-specific pretraining. The paper could have reported GPT-2 models fine-tuned on MATH without AMPS pretraining as an explicit baseline row in Table 2—the fact that it does not suggests either that this baseline was not run, or that it was run and the results were not competitive enough to include (which would itself be informative). This is a notable omission for a paper whose central practical contribution is the AMPS corpus.

Exact-Match Evaluation, Even with Normalization, Creates a Binary Signal That Obscures Partial Progress and Penalizes Near-Correct Answers

The assumption or constraint. MATH uses exact-match accuracy as its sole evaluation metric, with a normalization pipeline that handles certain equivalence classes (fraction representations, polynomial factor orderings, variable naming conventions) but necessarily treats all non-matching answers as equally incorrect. The paper states that "these rules cover nearly all ways that different generated or actual solutions can be equivalent in practice" (Section 3.1). However, this means that an answer that is correct except for a single sign error, a missed factor of 2, or an incorrectly simplified fraction is scored identically to an answer that is completely unrelated to the problem—both receive 0 credit.

The consequence. Exact-match evaluation produces a steep, discontinuous reward signal that may obscure genuine progress in mathematical reasoning capability. A model that improves from producing random numbers to producing answers that are structurally correct but contain minor arithmetic errors (e.g., producing $x^2 + 2x + 1$ instead of $x^2 + 2x + 2$ ) would show zero improvement in accuracy despite demonstrating substantially better understanding of the problem's structure. This is a particularly acute concern for MATH because the paper's own qualitative analysis of generated solutions (Figures 3 and 4) shows that models often produce "coherent and on-topic" derivations that contain logical mistakes—suggesting that there is meaningful variation in solution quality that the binary metric cannot capture. The paper reports error detection AUROC of 68.8%, confirming that the model's confidence provides some signal about correctness, but the binary evaluation discards this graded information entirely.

The binary metric also means that the benchmark cannot distinguish between different kinds of failure—a model that fails because it cannot parse the problem statement, a model that chooses the wrong mathematical approach, a model that applies the right approach but makes an algebraic error, and a model that correctly reasons through the entire problem but formats the answer incorrectly (e.g., using the wrong LaTeX command that the normalizer doesn't handle) all receive identical scores. This limits MATH's utility as a diagnostic tool for understanding how models fail and where interventions would be most impactful.

What evidence exists in the paper. The normalization rules are described in Section 3.1, and the answer extraction pipeline is described in Section 4.1. The paper acknowledges that partial-credit evaluation is desirable in principle—the partial solution experiment (Figure 5) provides "hints" of varying completeness and shows that accuracy improves with more solution context—but does not use this to motivate an alternative metric. The generated solution examples in Figures 3 and 4 provide qualitative evidence that models produce partially correct reasoning that receives zero credit under exact match: in Figure 3 (first example), the generated solution produces the answer "8" while the ground truth is "8," so this instance would be scored correct, but the derivation contains nonsensical steps, illustrating that exact match can also give credit to answers reached through incorrect reasoning.

Mitigation status. Not addressed as a limitation. The paper presents exact-match accuracy as a feature of MATH—it "allows MATH to be scored with exact match rather than with heuristic metrics such as BLEU" (Section 1)—and the normalization pipeline is presented as making exact match feasible where it would otherwise be impossible due to representation ambiguity. There is no discussion of the tradeoff between evaluation reliability (exact match eliminates ambiguity about what constitutes a correct answer) and evaluation informativeness (partial-credit metrics or step-level evaluation could provide richer signal about model capabilities). This is a defensible design choice for a benchmark, but it means that MATH is better suited to measuring whether models can produce perfect answers than to measuring incremental improvements in reasoning capability, which may be the more relevant signal for research progress.

The 7,500-Training-Problem Fine-Tuning Budget Is Fixed and Not Scaled with Model Size, Creating a Potential Confound Where Larger Models May Be Data-Starved

The assumption or constraint. All models, regardless of parameter count (0.1B through 1.5B), are fine-tuned on the same 7,500 MATH training problems for the same number of optimization steps (implicitly, one pass through the training data, since the paper does not specify multiple fine-tuning epochs). The paper does not ablate the fine-tuning data size or the number of fine-tuning epochs. This means that larger models—which have more parameters to adapt and typically require more data to fine-tune effectively without overfitting—receive the same amount of task-specific supervision as smaller models, potentially underestimating their capability.

The consequence. If the larger GPT-2 models (0.7B, 1.5B) are data-starved during fine-tuning—receiving only 7,500 examples to adapt 0.7–1.5 billion parameters—then their performance may reflect underfitting to the MATH task distribution rather than a fundamental inability to learn mathematical reasoning at that scale. The paper's core finding—that accuracy scales slowly with model size—could then be partially explained by insufficient task-specific data for the larger models rather than by a genuine ceiling on what Transformers can learn about mathematics. This would mean that the extrapolation to 10³⁵ parameters is confounded not only by the small scale range (as discussed in limitation 2) but also by a data constraint that disproportionately penalizes larger models. The fact that AMPS pretraining—which provides orders of magnitude more mathematical data—dramatically improves performance (the 130× multiplier claim) is consistent with the hypothesis that mathematical capability is data-limited rather than architecture-limited.

What evidence exists in the paper. The paper reports the training setup in Section 4.1: "Before fine-tuning on MATH, models pretrain on AMPS. We pretrain for one epoch... During fine-tuning, models predict final answers and solutions." No information is provided about the number of fine-tuning epochs or whether early stopping was used. The fine-tuning data size (7,500 problems) is reported in Section 3.1 but is not discussed as a potential bottleneck. The paper does not show learning curves (training or validation loss over fine-tuning steps) that would reveal whether larger models are underfitting.

Mitigation status. Not addressed. The paper treats the 7,500-problem fine-tuning set as fixed infrastructure and does not consider that larger models might need more data or that the observed scaling trend might steepen if fine-tuning data were scaled proportionally with parameters. This is an understandable omission given the paper's focus on the benchmark itself rather than on optimizing training procedures, but it weakens the claim that scale alone is insufficient—what the experiments actually demonstrate is that scale alone is insufficient given 7,500 fine-tuning examples, which is a weaker and more conditional claim. A training data scaling experiment (showing, for example, that 1.5B model accuracy plateaus even with more fine-tuning data) would have substantially strengthened this argument.

The Human Baseline Is Too Small and Homogeneous to Serve as a Reliable Calibration Point

The assumption or constraint. The paper reports human performance on MATH based on 6 participants (4 of whom are paper authors) solving 20 randomly sampled problems under a 1-hour time limit without calculators (Section 3.1, Appendix A.6). The results span from 40% (a PhD student who "does not especially like mathematics") to 90% (a three-time IMO gold medalist). The paper correctly describes this as providing "a rough but informative comparison to human-level performance" (Section 3.1), but then uses these numbers as reference points: 40% accuracy for models "would be impressive," and the IMO gold medalist's 90% (with errors "exclusively due to small errors of arithmetic") sets a de facto human ceiling.

The consequence. With only 6 participants and 20 problems, the human baseline has extremely wide uncertainty. The 40% participant solved 8 of 20 problems correctly; the 90% participant solved 18 of 20. The difference between these scores could reflect genuine skill differences, luck in the random problem sample, or time-management strategies under the 1-hour constraint (which forces a tradeoff between checking work and attempting more problems). The paper cannot characterize the distribution of human performance on MATH—there is no estimate of what fraction of the mathematically-trained population would score at each level, no data on how performance varies with time allocation, and no comparison of calculator vs. no-calculator conditions. The claim that "even 40% accuracy for a machine learning model would be impressive" implicitly treats the weakest human participant as a meaningful lower bound, but it is unclear whether a different random sample of 20 problems or a different participant with similar mathematical background would produce substantially different scores.

The homogeneity of the participant pool (all university students in computer science, most affiliated with the authors' research group) further limits generalizability. The paper does not report how the participants were selected beyond noting that "four of the participants are authors on the paper" and "the other two participants are friends or acquaintances of the authors" (Appendix A.6). This is an appropriate subject pool for a rough calibration, but the results should not be interpreted as establishing population-level human performance benchmarks.

What evidence exists in the paper. The human evaluation is described in Section 3.1 (main text) and Appendix A.6 (additional details). The paper reports the scores as raw fractions (8/20, 13/20, 14/20, 15/20, 18/20, 18/20) without confidence intervals. The instructions, time limit, and no-calculator constraint are specified. The paper is transparent about the small sample: "we were restricted to assessing six human participants and could not rely on crowdsourcing sites such as Amazon Mechanical Turk" (Appendix A.6).

Mitigation status. The paper acknowledges the limitation explicitly—"this survey was only to give a rough sense of human-level performance" (Appendix A.6)—and the main text describes the human results as "rough but informative" (Section 3.1). This is a reasonable disclaimer, but it does not change the fact that the human baseline is too noisy to serve as a reliable calibration point. A more informative human evaluation would require at minimum a larger problem sample (to reduce sampling variance), a larger and more diverse participant pool (to characterize population variation), and ideally a within-subjects manipulation of time constraints and calculator access. The paper's choice to include a human baseline despite these limitations is better than providing no human reference at all, but readers should treat the specific percentages (40%, 90%) as illustrative anecdotes rather than as benchmarks that models should be measured against.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose a new model, training algorithm, or architectural innovation. Its contribution is conceptual and diagnostic: it provides the first rigorous demonstration that mathematical problem solving—as distinguished from calculation—is qualitatively different from the text-based tasks that enormous Transformers were rapidly solving in 2021, and that scaling alone provides sharply diminishing returns on this capability. This finding challenges the prevailing narrative of the time, which held that most limitations of language models could be overcome by increasing model and dataset size along predictable log-linear trends.

The paper's impact is best understood not as a paradigm shift but as a boundary marker for the scaling hypothesis. Prior work from Kaplan et al. (2020) and Henighan et al. (2020) had established that Transformer performance scales predictably with compute on language modeling and many downstream tasks. The MATH paper identifies a specific capability—competition-level mathematical reasoning—where this relationship breaks down at accessible model scales. The extrapolation that achieving 40% accuracy would require approximately 10³⁵ parameters, while not a precise prediction, serves as a rhetorical and conceptual device that redirects research attention: if you believe scaling alone won't get us there in any practical timeframe, then the field must invest in algorithmic innovation rather than merely larger models.

This is not to say the paper disproves the scaling hypothesis—it does not, and cannot, given that it only tests models up to 1.5B parameters under the full pipeline. What it does is raise the evidentiary bar for claims that scaling alone suffices. The contrast the paper draws between MATH and LogiQA (Appendix A.2, Figure 6) is instructive: on LogiQA, scaling trends suggest human-level performance is imminent; on MATH, the same scaling methodology produces negligible gains. This forces a more nuanced view: scaling works for some reasoning-like tasks but not others, and the field needs to understand why. The paper's identification of the snowballing error problem—where models cannot productively use their own generated intermediate steps because early mistakes derail subsequent reasoning—provides one mechanistic hypothesis for this discrepancy.

The paper also recalibrates the role of domain-specific pretraining data. The finding that a 0.1B parameter model pretrained on AMPS matches a 13B parameter model without AMPS (a 130× parameter multiplier, however imprecisely measured) establishes that data quality and domain relevance can dominate raw model capacity for specialized reasoning tasks. This shifts the cost-benefit calculus for organizations deciding between training larger general-purpose models versus curating high-quality domain-specific data. It also provides a concrete template—the AMPS corpus combining human-designed pedagogical content (Khan Academy) with algorithmically-generated problems exercising a computer algebra system (Mathematica)—for constructing domain-specific pretraining corpora in other technical fields.

Perhaps most significantly, the paper establishes MATH as a durable research platform rather than a soon-to-be-solved benchmark. By designing the dataset around competition problems that require heuristic selection, multi-step chaining, and strategic reasoning—capabilities that current models demonstrably lack—the authors create headroom for years of research. The benchmark's dual categorization by subject and difficulty, its inclusion of step-by-step solutions, and its exact-match evaluation with careful normalization collectively enable fine-grained scientific inquiry rather than mere leaderboard competition. This is a deliberate departure from the pattern of benchmarks (SuperGLUE, SQuAD, etc.) that were rapidly saturated by scaling, and it represents a methodological contribution to how the field designs evaluation frameworks for capabilities that resist simple scaling solutions.

Follow-Up Research This Work Enables

Scaling the full pipeline (AMPS pretraining + MATH fine-tuning) to 10B–100B+ parameter models. The paper's central empirical claim—that scaling provides diminishing returns on MATH—is demonstrated only for GPT-2 models up to 1.5B parameters under the best training configuration. The largest model evaluated with the full pipeline is two orders of magnitude smaller than models that subsequently became standard (PaLM, GPT-3.5, GPT-4). A direct follow-up would train models at the 10B, 50B, and 100B+ scale with the exact AMPS pretraining + MATH fine-tuning recipe, measuring whether the log-linear scaling trend observed from 0.1B to 1.5B continues, steepens, or breaks. If accuracy jumps disproportionately at some scale threshold—as later chain-of-thought work suggests it might—this would fundamentally revise the paper's conclusion that scaling "will be impractical." If accuracy remains stubbornly low even at 100B+ parameters, the paper's core argument is powerfully validated. This experiment was infeasible for the authors in 2021 due to compute and API constraints but became tractable within a few years.

Training verifiers to detect errors in step-by-step solutions and using them to filter or guide generation. The paper's most revealing finding is the asymmetry between consuming solutions (which helps) and producing them (which hurts, due to snowballing errors). This suggests a clear research direction: train a separate model—a process-supervised verifier analogous to the PRM in the Reference Example paper—to score each step of a generated solution for correctness. During inference, use this verifier to prune low-scoring solution branches (via beam search or rejection sampling) or to provide feedback that the generator conditions on for revision. A strong experiment would fine-tune a verifier on MATH solutions labeled with step-level correctness (which could be derived automatically by checking whether each step is a valid logical consequence of prior steps using a symbolic engine, or by human annotation), then measure whether verifier-guided generation reverses the sign of the scratch-space effect—turning the 5.3% accuracy of solution generation (which is worse than direct answering at 6.9%) into accuracy that exceeds direct answering. The paper's partial-solution experiment (Figure 5), showing that models reach ~40% accuracy when given 99% of the ground-truth solution, establishes an upper bound on what perfect verification could achieve.

Curriculum-based fine-tuning that progressively increases problem difficulty. The paper demonstrates that model accuracy varies systematically with human-assigned difficulty (Figure 9) and that models achieve nonzero accuracy even on the hardest problems (~4% at Level 5). This creates the conditions for curriculum learning: fine-tune models first on Level 1–2 problems, then progressively introduce Level 3–4, and finally Level 5, rather than training on the uniform mixture of all difficulty levels used in the paper. The hypothesis is that models might learn more transferable heuristics from easier problems (where they get more frequent correct answer signals) and then apply those heuristics, with refinement, to harder problems. A strong experiment would compare curriculum-trained models against uniformly-trained models at multiple scales, measuring whether the curriculum steepens the accuracy-vs-scale curve—specifically, whether it raises Level 5 accuracy from the ~4% floor to a level where further improvements become measurable. The paper's subject × difficulty breakdown (Figure 10) also enables subject-specific curricula (e.g., train on Prealgebra before Intermediate Algebra within the same difficulty tier).

Augmenting MATH with automatically-generated problem variants to test generalization vs. memorization. With only 7,500 training problems, there is a risk that fine-tuned models are partially memorizing problem-solution patterns rather than learning generalizable mathematical reasoning. A follow-up could use the Mathematica-based problem generation infrastructure that the authors developed for AMPS to create variants of MATH-style problems—same mathematical concept, different numerical parameters or surface forms—and test whether models fine-tuned on original MATH problems transfer to these variants. If accuracy on variants is substantially lower than on the original test set, this would indicate that the reported 6.9% accuracy overestimates the model's genuine mathematical reasoning capability (because it includes a memorization component). Conversely, if transfer is high, models are learning something more abstract than surface patterns. This experiment would also produce an expanded training set that could address the potential data-starvation confound identified in the Limitations section (the 7,500-problem fine-tuning budget is fixed regardless of model size).

Integrating symbolic computation engines with neural generation for hybrid problem solving. The paper's qualitative analysis of generated solutions (Figures 3 and 4) shows that models often take reasonable high-level approaches but make arithmetic or algebraic errors in execution. This suggests a hybrid architecture: the language model proposes the high-level solution strategy (which theorem to apply, what substitution to make, how to set up the equation), and a symbolic engine (Mathematica, SymPy, or a theorem prover) executes the computation steps and verifies correctness. A concrete experiment would fine-tune a model to generate solution sketches that call external tools for specific operations (e.g., "solve the quadratic equation 3x² + 2x − 5 = 0 using the quadratic formula"), execute those calls via an API, and incorporate the results into the solution text. Accuracy would then be a function of the model's strategic reasoning (which it might do well) rather than its arithmetic precision (which it does poorly). This direction is directly motivated by the paper's finding that "models can consistently generate correct LaTeX and often perform steps that appear related to the question at hand, but still make many logical mistakes"—separating strategy from execution could mitigate the primary failure mode.

Developing and benchmarking metacognitive calibration methods using MATH's difficulty labels and solution structure. The paper reports that model confidence is poorly calibrated (AUROC of 68.8% for error detection, with near-100% confidence on many wrong answers). MATH's structure—difficulty levels, subject tags, step-by-step solutions—makes it an ideal testbed for developing methods that improve calibration. A concrete experiment would train an auxiliary calibration model that takes as input the problem text, the model's generated solution, and the model's internal representations (log probabilities, attention patterns, or hidden states) and predicts whether the answer is correct. The goal is to push the AUROC substantially above 68.8% and to produce confidence estimates that are reliable enough for downstream use (e.g., flagging answers for human review, or adaptively allocating more test-time compute to low-confidence problems). The difficulty labels provide ground truth for evaluating whether calibration quality varies with problem difficulty—do models know when they don't know on hard problems, or are they equally overconfident across all difficulty levels?

Practical Applications and Downstream Use Cases

Mathematics education and automated tutoring systems. A model that can generate step-by-step solutions to competition-level problems—even imperfectly—has direct application in educational technology. The paper shows that models fine-tuned on MATH solutions can produce "coherent and on-topic" derivations (Figures 3 and 4) even when the final answer is wrong. In a tutoring context, these partially-correct solutions could serve as starting points for student critique ("find the error in this derivation") or as hints of varying completeness (analogous to the partial-solution experiment in Figure 5, where providing 50% of the solution raises student-model accuracy from ~7% to ~15–20%). The fact that the paper demonstrates models can consume solutions more effectively than they can produce them suggests an asymmetric human-AI collaboration: the AI generates candidate solution steps, and the human (or a separate verifier) checks them. This is lower-stakes than fully autonomous problem-solving and leverages the model's strengths (generating plausible mathematical text) while mitigating its weaknesses (inability to verify correctness).

Generating training data for self-improving mathematical reasoning systems. The AMPS corpus demonstrates that algorithmically-generated mathematical problems with step-by-step solutions are valuable pretraining data. The 100 Mathematica scripts described in Table 1 and Appendix Figure 8 are themselves a practical contribution: they can be extended, modified, or used as templates to generate orders of magnitude more training data covering additional mathematical topics. For research groups building mathematical reasoning systems, this paper provides both a pretraining recipe (AMPS → MATH fine-tuning → beam search decoding) and a code base (the Mathematica scripts) that can serve as infrastructure for data generation pipelines. The paper's finding that adding noisy real-world data (StackExchange) reduces performance relative to clean synthetic data (AMPS alone, 6.0% vs. 6.2%) provides a concrete design principle: for mathematical reasoning, algorithmic generation with verified correctness may be more valuable than web-scraped data of uncertain quality.

Benchmarking foundation models on a task that resists "solved by scaling" claims. As of 2024, MATH remains a widely-used benchmark specifically because the paper demonstrated in 2021 that it would not be quickly saturated. For model developers and evaluators, MATH provides a standardized, automatically-gradable test of mathematical reasoning that produces meaningful differentiation between models—unlike benchmarks where near-ceiling performance makes comparisons uninformative. The paper's careful answer normalization scheme (handling fraction equivalence, polynomial factor ordering, variable naming conventions) makes the benchmark practical to use at scale without requiring human evaluation, while the difficulty and subject stratification enable fine-grained capability profiling. Organizations selecting between models for mathematically-intensive applications can use MATH performance broken down by difficulty level (Figure 9) and subject (Table 2) to identify specific capability gaps rather than relying on a single aggregate number.

Informing resource allocation between domain-specific pretraining and general model scale. The paper's headline finding—that AMPS pretraining provides a ~130× parameter multiplier for mathematical reasoning—has direct implications for teams deciding how to allocate compute budgets. If a 0.1B parameter model with 23 GB of curated mathematical pretraining data achieves accuracy comparable to a 13B parameter model without it, then for mathematically-intensive applications, investing in domain-specific data curation may be substantially more cost-effective than investing in larger models. This is a concrete, quantifiable tradeoff (however imprecise the specific 130× number) that can guide practical deployment decisions: for a fixed budget, should you fine-tune a large general-purpose model or build a domain-specific corpus and train a smaller model? The paper provides evidence favoring the latter for mathematical reasoning, with the caveat that this finding is demonstrated at small model scales and may not extrapolate to the regime where general-purpose models have much larger absolute capacity.

When to Prefer This Method

The paper does not propose a method with clear tradeoffs against named alternatives—it introduces a benchmark and a pretraining corpus, not a technique whose adoption requires choosing between competing approaches. The question of "when to use MATH vs. another mathematical reasoning benchmark" is implicitly addressed by the paper's design rationale but is not formulated as a decision rule. A forced comparison matrix would therefore be generic boilerplate rather than a reflection of the paper's own articulated tradeoffs.