AMO-Bench: Large Language Models Still Struggle in High School Math Competitions

🎯 Pitch

Even the strongest LLM scores just 52.4% on AMO-Bench—a new set of 50 original, IMO-level math problems—plummeting from >90% on saturated benchmarks like AIME. This drastic drop exposes how much current models still lack robust olympiad reasoning, while a discovered test-time scaling law reveals untapped capacity waiting to be unlocked.

1. Executive Summary

This paper introduces AMO-Bench, an advanced mathematical reasoning benchmark comprising 50 entirely original, human-crafted problems that are cross-validated to meet or exceed International Mathematical Olympiad difficulty standards, evaluating 26 large language models under a fixed sampling regime of 32 trials per problem (reporting AVG@32). The benchmark reveals that even the best-performing model — GPT-5-Thinking (High) — achieves only 52.4% accuracy, with most models scoring below 40%, while the benchmark also surfaces a test-time scaling trend (a near-linear relationship between log output length and accuracy across reasoning-effort settings) and evidence of latent capability via pass@k (top-tier models exceed 70% pass@32), establishing that current LLMs still struggle substantially on olympiad-level math reasoning despite possessing the initial capability to solve these problems when given sufficient sampling budget.

2. Context and Motivation

The Core Problem: We Are Running Out of Tests That Can Discriminate Between LLMs

Mathematical reasoning has become the gold-standard evaluation domain for large language models. Unlike open-ended conversation or creative writing—where quality is subjective and evaluation is contentious—mathematics offers an unambiguous correctness signal: a problem has a well-defined answer, and a solution either reaches it or doesn't. This makes math benchmarks uniquely valuable for tracking genuine progress in reasoning capabilities, as opposed to progress in style, fluency, or memorization of conversational patterns.

The fundamental problem AMO-Bench addresses is that existing math benchmarks are rapidly approaching saturation and can no longer serve as effective discriminators of reasoning ability among top-tier models. When multiple leading models score above 90% on the same test, the test has ceased to provide meaningful signal about which model reasons better—it only tells you that all of them are "good enough" for that difficulty level. This is a measurement problem with real consequences: without adequately challenging benchmarks, the field loses its ability to track genuine progress, compare methods, or identify where the remaining weaknesses lie.

Figure 1 in the paper illustrates this saturation vividly. On benchmarks like AIME24 and AIME25, top reasoning models from OpenAI, DeepSeek, Qwen, and Gemini all cluster between roughly 87% and 99% accuracy. AMO-Bench, by contrast, spreads these same models across a much wider range—from roughly 37% to 52%—restoring the benchmark's power to differentiate between models and to reveal meaningful performance gaps. This is not merely an academic concern: when benchmarks saturate, the research community can no longer tell whether a new model or technique represents genuine progress or merely incremental optimization on an already-solved problem class.

Why Saturation Happens: Two Distinct Mechanisms

The paper identifies two specific mechanisms driving benchmark saturation, and they operate through fundamentally different pathways:

Mechanism 1: Genuine capability improvement. The straightforward explanation is that LLMs are genuinely getting better at mathematical reasoning. Models like GPT-5, DeepSeek-V3.1, and Qwen3 have been trained with explicit reasoning-oriented objectives (reinforcement learning, chain-of-thought distillation, test-time compute scaling), and their performance on competition-level math problems reflects real advances in their ability to plan multi-step solutions, manipulate symbolic expressions, and verify their own intermediate reasoning. When models cross the 90% threshold on a benchmark like AIME24, part of the explanation is simply that they have mastered the reasoning patterns required by that difficulty level.

Mechanism 2: Performance leakage from data contamination. The more insidious explanation—and the one that motivates a key design choice in AMO-Bench—is that benchmarks derived from publicly available competition problems may be partially memorized by models during pretraining. AIME problems, for instance, are posted online with solutions, discussed on forums like Art of Problem Solving, and included in training datasets scraped from the web. When a model "solves" an AIME problem, it may be doing genuine reasoning, or it may be recalling a solution pattern it has already seen—and these two possibilities are observationally equivalent from the final answer alone.

The paper cites recent work (Sun et al., 2025; Balunović et al., 2025) that has raised concerns about data contamination in existing math benchmarks, and positions AMO-Bench as a direct response: all 50 problems are newly created by human experts and cross-checked against existing resources to ensure originality. This is a deliberate countermeasure against the data contamination problem that existing benchmarks cannot address.

The Problem with Harder Benchmarks: Proofs Don't Scale

The natural response to saturation is to create harder benchmarks. However, the paper identifies a critical bottleneck in how harder math evaluations have traditionally been conducted. The most difficult math competitions—most notably the International Mathematical Olympiad (IMO)—are proof-based: contestants must write full mathematical proofs, not merely produce a final answer. This creates a fundamental obstacle for automated evaluation at scale.

The paper explicitly calls this out:

"While recent efforts have incorporated problems from more difficult and newly held contests such as the International Mathematical Olympiad (IMO), these questions tend to be proof-based and require manual verification by experts. This reliance on expert review hinders the implementation of automated scoring processes, leading to inefficiency and inconsistency in large-scale evaluations and result reproductions."

This is not merely a convenience issue—it is a scalability and reproducibility problem. If evaluating a model's performance on a benchmark requires multiple mathematics PhDs to read and judge proofs, then:

Throughput is limited: You cannot evaluate many models, many samples per model, or many iterations of the same model during development.
Reproducibility is compromised: Different expert evaluators may apply different standards, and even the same evaluator may be inconsistent over time. This makes it difficult for independent researchers to verify published results.
Iteration speed is crippled: During model development, teams need rapid feedback on whether their changes improve reasoning. Manual grading introduces delays that make fast iteration impossible.

Prior benchmarks that have attempted to evaluate IMO-level reasoning (such as MathArena for IMO and USAMO; Balunović et al., 2025, and Proof or Bluff?; Petrov et al., 2025) have grappled with this tradeoff, generally opting for expert evaluation of proofs but accepting the associated costs and reproducibility limitations. AMO-Bench takes the opposite approach: it achieves IMO-level (or higher) difficulty while requiring only a final answer, enabling automatic grading without sacrificing challenge.

Prior Approaches and Where They Fall Short

The paper situates AMO-Bench within a landscape of increasingly challenging math benchmarks, organized along two axes: competition-level difficulty (problems requiring high-school to olympiad mathematical knowledge) and graduate/research-level difficulty (problems requiring advanced university mathematics).

Competition-level benchmarks. The lineage begins with MATH (Hendrycks et al., 2021), which covered topics from algebra to precalculus at the high-school competition level. As models saturated MATH (with scores crossing 90% on subsets like MATH500), the field shifted toward olympiad-level problems. OlympiadBench (He et al., 2024) introduced bilingual, olympiad-level challenges. Omni-MATH (Gao et al., 2024) expanded to multiple subjects with rigorous olympiad-oriented evaluation. MathOdyssey (Fang et al., 2025) focused on long-range compositional reasoning. BeyondAIME (ByteDance-Seed, 2025) collected problems styled after AIME but with increased difficulty.

The critical limitation shared by nearly all of these benchmarks is that their problems are drawn from existing competitions or published materials, making them vulnerable to data contamination. Even when problems are not directly memorized, models may have been exposed to similar problems, solution strategies, or stylistic patterns during pretraining, which inflates apparent reasoning ability. The paper's originality review stage—comparing candidate problems against existing datasets via n-gram matching and web search—directly addresses this gap.

A secondary limitation is difficulty calibration. Many existing benchmarks include a range of difficulty levels, some of which are no longer challenging for top models. For example, the first 10 problems in AIME are substantially easier than the later problems, and including them in a benchmark dilutes its ability to discriminate among top-tier models. AMO-Bench's difficulty review stage explicitly filters out problems that are "suitable for MO-level competitions" but lack adequate complexity, requiring that each problem meet or exceed IMO difficulty standards.

Graduate/research-level benchmarks. The other trajectory has been toward problems that test advanced mathematical knowledge rather than (or in addition to) reasoning ability. FrontierMath (Glazer et al., 2024) covers computationally intensive and abstract problems across most branches of modern mathematics. HARDMath2 (Roggeveen et al., 2025) focuses on approximation-based problems from applied sciences and engineering. HLE (Phan et al., 2025) (Humanity's Last Exam) constructs a multi-subject academic benchmark spanning human frontier knowledge. RealMath (Zhang et al., 2025) evaluates research-level mathematical tasks.

These benchmarks test a fundamentally different capability than AMO-Bench. They assess whether a model possesses extensive and deep mathematical knowledge (e.g., group theory, measure theory, differential geometry), whereas AMO-Bench assesses whether a model can perform complex reasoning within a bounded knowledge domain (high-school mathematics). The distinction is important: a model might fail FrontierMath because it doesn't know the definition of a cohomology group, and it might fail AMO-Bench because it cannot construct a multi-step combinatorial argument despite knowing all the necessary concepts. Both are valuable evaluation dimensions, but they measure different things. AMO-Bench deliberately stays within the IMO knowledge syllabus to isolate reasoning ability from knowledge breadth.

How This Paper Positions Itself

AMO-Bench positions itself at the intersection of three design constraints that no existing benchmark simultaneously satisfies:

Difficulty floor: All problems must meet or exceed IMO difficulty, verified by multiple human experts and confirmed by preliminary LLM evaluation (at least two state-of-the-art reasoning models must fail to consistently solve each problem).
Originality guarantee: All problems are newly created and cross-checked against existing competitions and online resources to prevent data contamination.
Automatic gradability: All problems require only a final answer (not a proof), enabling parser-based or LLM-based grading with demonstrated 99.2% accuracy.

The paper explicitly does not claim that AMO-Bench replaces all existing math benchmarks. Rather, it addresses a specific gap: the need for a sustainable, automatically gradable, contamination-free, maximally challenging benchmark that can continue to discriminate among top-tier reasoning models as they improve. The comparison in Figure 1 makes this positioning concrete: on easier benchmarks (AIME24, AIME25), the evaluated models are tightly clustered near the ceiling; on AMO-Bench, they are spread across a meaningful performance range, recovering the discriminatory power that saturated benchmarks have lost.

A subtle but important aspect of this positioning is that AMO-Bench is designed not just as a static snapshot of current capabilities, but as a benchmark with headroom for future progress. The paper's analysis of test-time scaling trends (Figure 7) and pass@k potential (Figure 8) demonstrates that models have substantial untapped capability on these problems—they can solve them with sufficient sampling budget—but cannot currently solve them reliably in a single attempt. This means AMO-Bench is unlikely to saturate immediately, because the path to high single-shot accuracy requires genuine improvements in reasoning reliability, not just broader sampling.

The Dual Role: Evaluation Tool AND Diagnostic Instrument

Beyond its primary role as an evaluation benchmark, AMO-Bench is positioned as a diagnostic instrument for understanding model reasoning behavior. The paper includes analyses that go beyond aggregate scores:

Reasoning efficiency analysis (Figures 5–6): By measuring average output token consumption alongside accuracy, AMO-Bench reveals the computational cost of reasoning at this difficulty level. The finding that top models consume ~37K tokens on AMO-Bench versus ~7K on AIME25 is itself a diagnostic signal about the nature of the reasoning required.
Test-time scaling analysis (Figure 7): By varying reasoning effort (Low/Medium/High settings) for the same model families, the benchmark reveals whether increased inference compute continues to yield improvements—a direct test of whether current models are compute-bottlenecked on these problems.
Pass@k analysis (Figure 8): By computing pass@k across multiple samples, the benchmark separates two failure modes: inability to solve a problem at all (low pass@k even at large k) versus inability to find the correct solution reliably on a single attempt (low pass@1 but high pass@k). The latter indicates that the model possesses the capability but lacks reliable reasoning selection, suggesting different improvement strategies (better verification, search, or self-correction) than the former (fundamentally new capabilities through pretraining or RL).

These diagnostic dimensions make AMO-Bench not just a harder test, but a richer source of information about where and how models fail on olympiad-level reasoning. The benchmark is designed to support not only the question "how good is this model?" but also "what should we work on next to make it better?"—which is the more important question for driving research forward.

3. Technical Approach

3.1 Reader Orientation

This paper does not propose a new model or algorithm; rather, it presents AMO-Bench, a systematically constructed evaluation benchmark—a dataset of 50 math problems plus an associated grading infrastructure—designed to measure advanced mathematical reasoning in large language models. The problem it solves is the measurement gap created by existing math benchmarks reaching performance saturation: when multiple top-tier models score above 90% on the same test, the test can no longer tell us which model reasons better or where genuine capability differences lie, so AMO-Bench creates a harder, contamination-free, automatically gradable instrument that restores discriminatory power among state-of-the-art reasoning models.

3.2 Big-Picture Architecture (Diagram in Words)

The AMO-Bench system has four major components, organized as a pipeline that flows from problem creation to model evaluation to result analysis:

Problem Construction Pipeline — a multi-stage human-expert-driven process that creates, validates, and filters 50 original math problems to guarantee difficulty, originality, correctness, and gradability. This is the "data generation" component, producing the benchmark itself.
Grading Infrastructure — a hybrid evaluation system that combines parser-based grading (for numerical, set, and variable-expression answers, covering 39 of 50 problems) with LLM-based grading (for descriptive answers, covering 11 of 50 problems), achieving 99.2% grading accuracy as validated on 1,000 human-checked responses.
Model Evaluation Protocol — a standardized sampling and scoring procedure that runs each evaluated LLM through all 50 problems 32 independent times (temperature=1.0 for reasoning models, 0.7 for non-reasoning), extracts final answers from a structured output format (\boxed{<answer>}), applies the appropriate grading method per problem, and reports AVG@32 (the average accuracy across 32 trials).
Diagnostic Analysis Framework — a set of post-hoc analyses that go beyond aggregate scores, including reasoning efficiency measurement (output token consumption vs. accuracy), test-time scaling analysis (varying reasoning effort levels within model families), and pass@k computation (estimating the probability that at least one of k samples is correct), all designed to reveal not just how well models perform but why they fail and where improvement potential lies.

Information flows as follows: human experts create candidate problems → multi-stage review filters and validates them → the 50 surviving problems form the benchmark → each evaluated LLM receives each problem with a structured output prompt → the model generates 32 independent responses per problem → the grading infrastructure extracts final answers and scores them → aggregate metrics and diagnostic analyses are computed across models.

3.3 Roadmap for the Deep Dive

First, the problem construction pipeline (Section 2.1 in the paper), because the benchmark's value derives entirely from its construction quality — understanding how problems are created, validated, and filtered is essential to trusting the evaluation results.
Second, the grading methodology (Section 2.3), since accurate, scalable grading is what distinguishes AMO-Bench from proof-based olympiad benchmarks and enables the large-scale evaluation reported in the paper.
Third, the evaluation protocol and metrics, covering the sampling strategy, temperature settings, output format requirements, and the choice of AVG@32 as the primary metric — these details determine what the reported numbers actually mean.
Fourth, the diagnostic analysis methods (pass@k, test-time scaling, output length analysis), since these are not merely supplementary results but integral components of the benchmark's design as a diagnostic instrument.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmark construction and evaluation paper whose core idea is that a sustainably challenging math evaluation must simultaneously guarantee difficulty (≥IMO), originality (no data contamination), and automatic gradability (final-answer format), achieved through a multi-stage human-expert pipeline with hybrid grading.

Problem Construction Pipeline

The benchmark's 50 problems emerge from a four-stage pipeline that progressively filters candidate problems through quality, originality, and difficulty gates. Each stage involves multiple human experts making independent assessments, creating redundancy that guards against individual judgment errors.

Stage 1: Data Creation. Mathematics experts from top universities and educational institutions independently design each problem from scratch. The paper specifies that these experts have "extensive backgrounds in high school mathematics competitions, either having won MO-level mathematics competition awards or possessing experience in competition problem design." This credentialing is important because it ensures that problem authors understand the specific reasoning patterns tested in olympiad-level mathematics — they are not merely mathematicians, but mathematicians with expertise in the competition genre.

For each problem, the author must produce three deliverables:

The problem statement (the question text provided to models).
A detailed step-by-step solution (the human-annotated reasoning path).
The final answer in the format required for grading.

The annotated solutions serve multiple downstream purposes: they enable the quality review stage (reviewers can verify logical correctness), they provide transparency for users of the benchmark (enabling error analysis and prompt engineering), and they serve as a difficulty signal (the paper analyzes solution length distribution as a proxy for reasoning complexity in Section 2.2).

Stage 2: Quality Review. Each candidate problem undergoes blind review by at least three experts, meaning reviewers evaluate problems without knowing who created them and without seeing each other's assessments. This blind protocol reduces social biases (deference to senior experts, reluctance to criticize colleagues) that could compromise evaluation quality.

The quality review assesses two specific criteria:

Semantic correctness: Whether the problem statement and solution are "semantically unambiguous and logically correct." This is more demanding than simply checking for typos — it requires verifying that the problem has a well-defined answer, that the solution validly derives that answer without logical gaps or hidden assumptions, and that there are no edge cases or degenerate interpretations that would make the problem ill-posed. In competition mathematics, subtle ambiguities in problem statements (e.g., whether zero is considered a positive integer, whether degenerate cases are included) can change the correct answer, and catching these requires domain expertise.
Syllabus compliance: Whether the mathematical knowledge required for the problem is "within the scope typically covered in MO-level competitions such as IMO." This is a deliberate constraint: AMO-Bench is designed to test reasoning within bounded knowledge, not breadth of mathematical education. The IMO syllabus covers high-school mathematics (algebra, geometry, number theory, combinatorics, sequences and functions) but explicitly excludes calculus and university-level mathematics. By enforcing this boundary, AMO-Bench ensures that models fail because of inadequate reasoning — not because they lack graduate-level mathematical knowledge.

Stage 3: Originality Review. This stage addresses the data contamination problem directly through multiple detection mechanisms:

N-gram matching against existing datasets: Candidate problems are compared against problems from existing benchmarks (AIME24/25 are explicitly mentioned) using 10-gram character-level matching. A 10-gram is a sequence of 10 consecutive characters; if a candidate problem shares long character sequences with known competition problems, it is flagged as potentially derivative. The choice of 10 characters as the threshold represents a balance: shorter n-grams (e.g., 5-grams) would produce many false positives from common mathematical phrasing, while longer n-grams (e.g., 20-grams) might miss near-duplicates with minor rewording.
Web search for similar content: Problems are searched against online resources to identify any highly similar content that might exist in forums, solution repositories, or training data scraped from the web. This addresses a broader contamination vector than n-gram matching alone, since a problem might be original in wording but conceptually identical to a widely-discussed competition problem.
Expert attestation during quality review: During the quality review stage, experts are explicitly asked "whether they have encountered highly similar questions in past competitions." This leverages human memory as an additional contamination check — a problem might slip past automated n-gram and web-search checks if it is similar but not identical to an obscure competition problem that an experienced expert would recognize.

This three-pronged originality review is more thorough than what most existing benchmarks employ. The paper does not claim it guarantees zero contamination — no feasible method can — but it establishes a substantially higher bar than benchmarks that simply republish existing competition problems without any originality screening.

Stage 4: Difficulty Review. This is the most technically interesting stage, because it operationalizes the concept of "difficulty" through a combination of human judgment and empirical LLM testing. Each candidate problem must satisfy two criteria:

Human expert verification: The problem must "meet or exceed the IMO difficulty standards, as verified by the human expert." This is a qualitative judgment made by experts familiar with IMO problem difficulty. It serves as a floor: problems that are suitable for olympiad-level competitions but are relatively easy within that category (the paper cites "the first 10 questions in AIME" as examples of MO-level problems that do not meet the IMO difficulty threshold) are filtered out. This is important because many competition benchmarks include a range of difficulty within the competition, and problems at the easier end of that range may already be saturated for top models.
Empirical LLM difficulty verification: The paper employs "multiple advanced reasoning models (such as GPT, DeepSeek, and Gemini series models) for preliminary evaluation, requiring that at least two such models fail to correctly and consistently solve the problem." The specific operationalization is defined in a footnote: each model is given 3 samples per problem, and if all 3 samples are correct, the model is deemed capable of "consistently" solving the problem. At least two models must fail this consistency check — meaning at least two state-of-the-art reasoning models must produce at least one incorrect answer in 3 attempts — for the problem to be included.

This empirical difficulty check is a clever design choice for several reasons. First, it makes "difficulty" operational rather than subjective: a problem is hard enough if current best models cannot solve it reliably. Second, it is forward-looking: as models improve, future benchmark creators can raise the bar by using even stronger models in the preliminary evaluation. Third, it connects the difficulty criterion directly to the benchmark's purpose: problems that top models can already solve consistently would not discriminate among those models, so they are excluded.

The requirement that at least two models fail (rather than just one) guards against model-specific weaknesses being misinterpreted as problem difficulty. A problem might be trivial for most models but challenging for one specific architecture due to an idiosyncratic weakness; requiring failure from at least two models reduces the chance that such problems are included.

Result of the pipeline. Of the candidate problems that enter this pipeline, only those surviving all four stages are included in the final benchmark of 50 problems. The paper does not report the initial number of candidate problems or the rejection rate at each stage, which is a minor transparency limitation — knowing the pipeline's selectivity would help assess how difficult it is to create problems meeting these standards and whether the benchmark can be feasibly expanded in the future.

Problem categorization. The 50 problems are classified into five categories aligned with standard olympiad syllabi: Algebraic Equations & Inequalities (11 problems, 22%), Functions & Sequences (13 problems, 26%), Geometry (5 problems, 10%), Number Theory (9 problems, 18%), and Combinatorics (12 problems, 24%). Geometry is underrepresented relative to typical olympiad distributions, which the paper acknowledges implicitly by reporting the category counts. This matters for interpretation because geometry problems often require spatial reasoning that current text-only LLMs handle differently from algebraic reasoning; a benchmark with fewer geometry problems may slightly overestimate models' overall mathematical reasoning capability compared to a balanced olympiad.

Solution length as a difficulty proxy. Figure 3b compares the token length distribution of human-annotated solutions across MATH500, AIME24, and AMO-Bench. The distribution is dramatically shifted toward longer solutions in AMO-Bench: while MATH500 solutions are predominantly under 512 tokens (93% in that bucket), and AIME24 solutions cluster in the 512–2048 token range, AMO-Bench solutions span 512 to over 4096 tokens, with 40% in the 2048–4096 range and 12% exceeding 4096 tokens (using the DeepSeek-V3.1 tokenizer). The paper interprets this as evidence that AMO-Bench problems "are inherently more challenging and require more complex reasoning to arrive at the final answer."

This interpretation relies on the assumption that solution length correlates with reasoning complexity, which is generally true for olympiad-style problems but not universally — a problem could have a long solution because it involves many mechanical steps without deep insight, or a short solution because it requires a single clever observation. Nevertheless, the magnitude of the shift (from <512 tokens dominating in MATH500 to >2048 tokens dominating in AMO-Bench) is large enough that it almost certainly reflects genuine differences in required reasoning depth.

Grading Methodology

The grading system is the component that enables AMO-Bench to achieve IMO-level difficulty while maintaining automatic gradability. It uses a hybrid approach that selects the grading method based on the answer type of each problem.

Answer type taxonomy. The paper classifies all 50 problems into four answer types, with explicit examples provided for each:

Numerical answers (e.g., Example 1: "Find the minimum value of $x_1 + x_2 + \cdots + x_{2024}$ " → answer: 1382935444). These are single numbers (integers or decimals) that can be compared exactly or to specified precision.
Set answers (e.g., Example 2: "Find all positive integers n such that..." → answer: {1, 2, 3}). These are finite sets of values, typically represented in set notation.
Variable-expression answers (e.g., Example 3: "Find the general term formula for this sequence" → answer: $\frac{(2+\sqrt{3})^{2^{2-n}} + (2-\sqrt{3})^{2^{2-n}}}{2}$ ). These are symbolic expressions containing variables, where correctness cannot be determined by simple string matching but requires verifying equivalence under variable substitution.
Descriptive answers (e.g., Example 4: "Find the minimum value..." → answer: a case analysis distinguishing n=3 from n≥4, with different minimum values for each case). These require multi-part or conditional answers that cannot be represented as a single number, set, or expression.

The distribution is 39 parser-gradable problems (types 1–3) and 11 LLM-gradable problems (type 4).

Parser-based grading (39/50 problems). For numerical, set, and variable-expression answers, the grading pipeline works as follows:

Answer extraction: Models are instructed via the query prompt template (reproduced in Appendix A, Example 5) to format their final answer as ### The final answer is: $\boxed{<answer>}$. The grading system extracts the content within the \boxed{} LaTeX environment. This structured output format is a deliberate design choice that constrains model behavior to produce machine-parseable responses, and the paper provides an example in the prompt to make the expected format unambiguous.
Parsing and verification: The extracted answer is processed using the math-verify library, which provides tools for parsing mathematical expressions and verifying equivalence with ground-truth answers. The paper does not detail the internal workings of math-verify, but its functionality generally includes symbolic simplification, expression normalization, and equivalence checking under standard mathematical interpretations.
Decimal precision requirement: If a model's answer contains a decimal approximation, it "must be accurate to at least four decimal places." This is a practical threshold: olympiad problems often have exact closed-form answers, but models sometimes produce decimal approximations instead of simplified radicals or fractions. Requiring four decimal places of precision ensures that approximately correct answers are not awarded credit while accommodating the reality that models may output numerical approximations.
Variable-expression verification: For expression-type answers, simple string matching is insufficient because equivalent expressions can have different forms (e.g., $\frac{x^2-1}{x-1}$ equals $x+1$ for $x \neq 1$ ). The paper's approach is to "assign multiple sets of values to the variables in the expression, then verify whether the values of the generated expression match that of the ground-truth expression." This is essentially testing: substitute several concrete values, evaluate both expressions numerically, and check if they agree. If the expressions disagree for any test value, they are not equivalent. The number of test values and their selection strategy are not specified, which is a minor detail that could affect grading accuracy for expressions with restricted domains or piecewise definitions.
Manual post-processing adjustment: The paper mentions that during preliminary evaluation, the authors "manually review the parsing results... and adjust the post-processing algorithms." This indicates that the parser-based grading is not purely automated in its initial setup — it requires human calibration to handle edge cases in how models format their answers (e.g., inconsistent LaTeX, unexpected whitespace, alternative notation). Once calibrated, it can run automatically, but the initial tuning requires human judgment.

LLM-based grading (11/50 problems). For descriptive answers that cannot be parsed into a simple mathematical object, the paper uses o4-mini (Low) as a grading model with a structured grading prompt (reproduced in Appendix A, Example 6). The grading prompt provides:

The original problem
The reference (ground-truth) answer
The student's (model's) answer
Instructions to determine equivalence and output "Correct" or "Incorrect"
A required output format (### Conclusion: Correct/Incorrect)
The decimal precision requirement (four decimal places) for any numerical components

Majority voting across grading samples. To increase robustness, "majority voting is performed across five independent grading samples for each response." This means that for each model response to a descriptive-answer problem, the grading LLM (o4-mini Low) is queried 5 separate times with the same grading prompt, and the final grade is the majority vote among these 5 judgments. The choice of 5 samples (an odd number to avoid ties) provides statistical reliability: if the grading model has, say, a 5% error rate, the probability that a majority of 5 independent judgments will be wrong is substantially lower than the individual error rate (assuming independent errors, the probability of ≥3 errors out of 5 from a 5% error rate is approximately 0.001, though in practice grading errors are likely correlated because they stem from the same ambiguous cases).

The paper specifies that the grading model is set to "Low" reasoning effort (o4-mini Low). This is a practical choice: descriptive answer grading requires semantic understanding but not the deep multi-step reasoning that the benchmark itself tests. Using the "Low" setting balances grading accuracy against cost and latency. Whether a higher-effort grading model would improve accuracy beyond the already-reported 99.2% is not investigated, but the reported accuracy suggests it is unnecessary.

Manual verification of LLM-based grades. During preliminary evaluation, the authors "manually verify the correctness of LLM-based grades for all descriptive answers and revise answer descriptions where needed to enhance grading accuracy." This is a crucial calibration step: the ground-truth descriptive answers are refined based on observed grading failures until the grading model can reliably assess equivalence. This process creates a feedback loop between answer specification and grading reliability — if the original answer description is ambiguous or underspecified, the grading model may produce inconsistent judgments, and the description is then revised to be more explicit. The final answer descriptions in AMO-Bench therefore represent not just the mathematical solution but a grading-optimized description designed to be machine-evaluable.

Grading accuracy validation. The paper reports a manual quality check on "1,000 responses generated by 10 different LLMs" finding that "the grading accuracy reached 99.2%." This validation is important but some details are unspecified: the distribution of responses across problem types (whether all 50 problems are represented), the distribution across models (whether error-prone models are adequately sampled), and the nature of grading errors (false positives vs. false negatives). A 99.2% accuracy on 1,000 samples means approximately 8 grading errors, which could be concentrated in a few problematic problems or spread thinly. Nevertheless, this is strong validation that the grading infrastructure is reliable enough for benchmark purposes — grading errors at this rate are unlikely to meaningfully change model rankings or aggregate conclusions.

Evaluation Protocol and Metrics

The evaluation protocol standardizes how models interact with AMO-Bench to ensure fair comparisons. The key design choices involve sampling parameters, output format constraints, and the primary metric.

Model selection. The paper evaluates 26 LLMs spanning both proprietary and open-source categories, and both reasoning (models with explicit chain-of-thought or extended thinking capabilities) and non-reasoning model types. The selection is deliberately broad: it includes frontier models from OpenAI (GPT-5, o4-mini, o3-mini, GPT-4.1, GPT-4o), Google (Gemini-2.5-Pro, Gemini-2.5-Flash), Anthropic (Claude-Opus-4, Claude-Sonnet-4.5), DeepSeek (DeepSeek-V3.1-Thinking, DeepSeek-R1-0528, DeepSeek-V3.1, DeepSeek-R1, DeepSeek-V3-0324), Qwen (Qwen3-235B-A22B-Thinking-2507, Qwen3-Next-80B-Thinking, Qwen3-Max-Instruct, Qwen3-Next-80B-Instruct), GLM (GLM-4.5), Kimi (Kimi-K2), and LongCat (LongCat-Flash-Thinking, LongCat-Flash).

The inclusion of both reasoning and non-reasoning models serves an important analytical purpose: it reveals whether enhanced reasoning capabilities (typically achieved through reinforcement learning or specialized chain-of-thought training) are necessary for olympiad-level mathematics, or whether general-purpose instruction-tuned models can compete. The results (Figure 4) show that reasoning models occupy all top positions, but some non-reasoning models (Qwen3-Max-Instruct at 28.8%, LongCat-Flash at 14.6%) substantially outperform lower-tier reasoning models (e.g., o3-mini Medium at 13.1%, DeepSeek-R1 at 10.9%), indicating that the reasoning/non-reasoning distinction is not an absolute performance boundary.

Sampling parameters. The paper specifies:

Temperature: 1.0 for reasoning models, 0.7 for non-reasoning models. The higher temperature for reasoning models is notable: reasoning models typically use chain-of-thought or extended thinking that benefits from exploration, and temperature 1.0 encourages diverse reasoning paths across the 32 trials. The lower temperature (0.7) for non-reasoning models is more standard for evaluation settings, trading off diversity for coherence.
Top-k and top-p: Both set to top-k=50 and top-p=0.95 for all models. These are relatively standard nucleus sampling parameters that truncate the low-probability tail of the token distribution while preserving most of the probability mass.
Maximum context/output length: Set to "the highest allowable limit for each model during inference." This is a critical design choice: the paper explicitly states that this avoids "underestimating the reasoning capabilities of the model due to restrictions on the token budget." Given that top models generate ~37K tokens on average for AMO-Bench problems (as reported in the analysis), restricting the output length could artificially cap performance by truncating solutions before completion.

Structured output format. All models are prompted using a template (Appendix A, Example 5) that instructs them to:

Solve the problem
Output the final answer in the format ### The final answer is: $\boxed{<your answer>}$
Provide an example of the expected format (### The final answer is: $\boxed{123}$)
Ensure decimal approximations are accurate to at least four decimal places

The structured format serves the grading pipeline: it provides a reliable extraction point for the math-verify parser. Without this constraint, models might output answers in inconsistent formats that resist automatic parsing. The prompt template is reproduced verbatim in the appendix, enabling exact replication.

Primary metric: AVG@32. The paper's main evaluation metric is AVG@32, defined as the average accuracy across 32 independent sampling trials for each model on each problem, then averaged across all 50 problems. Formally, for a model $M$ and the set of 50 problems $\mathcal{P}$ :

$\text{AVG@32}(M) = \frac{1}{50} \sum_{p \in \mathcal{P}} \frac{1}{32} \sum_{i=1}^{32} \mathbb{1}[\text{correct}(M, p, i)]$

where $\mathbb{1}[\text{correct}(M, p, i)]$ is 1 if the model's $i$ -th sampled response to problem $p$ is graded as correct, and 0 otherwise.

What AVG@32 computes: the expected accuracy if you run the model once on a randomly selected problem from the benchmark. Each problem contributes equally to the final metric (the outer average is over problems), and within each problem, the 32 samples provide an estimate of the model's single-shot success probability on that problem.

Why AVG@32 rather than pass@1 or pass@k: AVG@32 estimates the single-attempt accuracy with reduced sampling variance compared to pass@1 (which would use only one sample and be noisier). The choice of 32 samples is validated in Appendix B (Figure 9), which shows how AVG@k stabilizes as k increases: at small k, the estimated accuracy fluctuates substantially across different random seeds; at k=32, the estimates are stable and "rarely appears the reverse-order phenomenon" (where model A appears better than model B in one sampling run but worse in another). The paper demonstrates this stability by plotting AVG@k for multiple k values across 8 representative models, showing that the relative ordering of models stabilizes well before k=32.

AVG@32 is fundamentally different from pass@32, which asks "what is the probability that at least one of 32 attempts is correct?" AVG@32 answers "what is the expected accuracy on a single attempt?" — a more demanding standard that reflects real-world deployment where users typically receive one model response per query.

The AMO-Bench-P subset. To facilitate easier reproduction, the paper defines AMO-Bench-P, a subset containing only the 39 parser-gradable problems. Table 1 in Appendix C reports AVG@32 on both the full AMO-Bench and AMO-Bench-P. Performance on AMO-Bench-P is generally slightly higher than on the full benchmark, confirming that descriptive-answer problems are inherently more challenging. The subset enables rapid evaluation without the cost and complexity of LLM-based grading, which is valuable for model development cycles where frequent evaluation is needed.

Diagnostic Analysis Methods

Beyond the primary AVG@32 metric, the paper employs three diagnostic analyses that extract richer information from the evaluation data. These are not merely supplementary — they are integral to the benchmark's design as a tool for understanding model limitations.

Reasoning efficiency analysis (output tokens vs. accuracy). This analysis, presented in Figures 5 and 6, examines the relationship between how many tokens models generate and how accurate they are. The methodology is straightforward: for each model, compute the average number of output tokens across all 32 samples for all 50 problems, then plot this against AVG@32.

The paper reports specific token counts: GPT-5-Thinking (High) generates an average of approximately 37,000 output tokens on AMO-Bench, compared to approximately 7,000 on AIME25 and 6,000 on AIME24. This 5–6× increase in token consumption quantifies the additional reasoning effort demanded by AMO-Bench problems.

Figure 6 extends this analysis across multiple benchmarks (MATH500, AIME24, AIME25, HMMT25, BeyondAIME, AMO-Bench) for six representative models. For each model, the relationship between benchmark accuracy and average output length forms a consistent pattern: as accuracy decreases (harder benchmarks), output length increases. This validates output length as a proxy for problem difficulty and demonstrates that models adapt their reasoning effort to problem complexity — they don't produce verbose solutions for easy problems or terse solutions for hard ones, suggesting that the token consumption reflects genuine reasoning rather than stylistic verbosity.

Test-time scaling analysis (reasoning effort control). Figure 7 presents a more controlled analysis: for models that expose reasoning effort settings (GPT-5 with Low/Medium/High, o4-mini with Low/Medium/High, o3-mini with Low/Medium/High), the paper plots AVG@32 against the logarithm of average output length across these settings.

The key finding is a near-linear growth trend: as the logarithm of output length increases, AVG@32 increases approximately linearly. This is described as "highly aligned with earlier experimental observations from existing benchmarks such as MATH500 and AIME24 (Muennighoff et al., 2025)," referencing the s1 work on simple test-time scaling.

This analysis serves two purposes. First, it demonstrates that AMO-Bench is not saturated — increasing compute (and thus output length) continues to yield accuracy improvements, indicating headroom for future progress. Second, it validates that the benchmark captures the compute-accuracy scaling relationship observed on easier benchmarks, suggesting that the underlying reasoning dynamics (more tokens → more reasoning → higher accuracy) extend to olympiad-level difficulty.

Pass@k analysis. The paper computes pass@k for $k \in \{1, 2, 4, 8, 16, 32\}$ (shown in Figure 8 and with granular k values for stability analysis in Figure 9). Pass@k is defined as the probability that at least one of k independent samples from the model is correct for a given problem:

$\text{pass@k} = \mathbb{E}_{\text{problems}} \left[ 1 - \frac{\binom{n - c}{k}}{\binom{n}{k}} \right]$

where $n$ is the total number of samples (32 in this case), $c$ is the number of correct samples for that problem, and the expression $1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}$ is the unbiased estimator for the probability that at least one of k samples is correct, given that c out of n samples were observed correct.

What pass@k computes: it estimates what fraction of problems the model could solve if allowed k independent attempts and any correct answer counts as success. It answers the question: "if you give the model k chances per problem, how many problems can it solve at least once?"

Why pass@k matters: it separates two distinct failure modes that are conflated in AVG@32:

Capability gap: If pass@k is low even at large k, the model fundamentally cannot solve the problem — no correct solutions exist in its output distribution, and no amount of sampling will help.
Reliability gap: If pass@1 is low but pass@k grows rapidly with k, the model can solve the problem but does so inconsistently — the capability exists but the model fails to select the correct reasoning path most of the time.

Figure 8 reports pass@32 for top models: GPT-5-Thinking (High) reaches 82.0%, LongCat-Flash-Thinking reaches 86.0%, and Qwen3-235B-Thinking reaches 74.0%. These substantially exceed the corresponding AVG@32 scores (52.4%, 43.6%, and 47.8% respectively), demonstrating that the primary limitation is reliability, not fundamental capability — the models do possess the knowledge and reasoning skills to solve most problems, but they cannot consistently find the correct solution path in a single attempt.

The gap between pass@32 and AVG@32 quantifies the potential improvement from better reasoning selection. For GPT-5-Thinking (High), the gap is approximately 29.6 percentage points (82.0% − 52.4%), representing the maximum achievable gain from techniques like improved verification, self-consistency, or test-time search that could convert latent capability into consistent performance.

Stability analysis of AVG@k (Appendix B, Figure 9). This analysis demonstrates why 32 samples were chosen for the primary metric. Figure 9 plots AVG@k for k from 1 to 32 across 8 models, showing how the estimated accuracy evolves and stabilizes. At small k, the estimates are noisy and model rankings sometimes reverse (e.g., models with similar true accuracy may appear in different orders depending on the random seed). As k increases, the curves stabilize and converge toward consistent values. The paper asserts that at k=32, "it rarely appears the reverse-order phenomenon," meaning the relative ranking of models is reliable.

The choice of 32 thus represents a balance: enough samples to suppress sampling variance below practically meaningful levels, but not so many that the evaluation becomes prohibitively expensive. For 26 models × 50 problems × 32 samples = 41,600 total model queries, which is substantial but feasible for a one-time benchmark evaluation.

4. Key Insights and Innovations

Innovation 1: The Benchmark as a Measurement Instrument — Difficulty as a Deliberate Design Constraint, Not an Emergent Property

Prior math benchmarks treat difficulty as an emergent property of problem selection: curators gather problems from existing competitions, and whatever difficulty distribution those competitions happen to have becomes the benchmark's difficulty profile. If some problems are too easy for current models (e.g., the first 10 AIME problems), they dilute the benchmark's discriminatory power at the top end but remain in the dataset because that's what the competition contained. The field's implicit assumption has been that harder benchmarks naturally arise from selecting harder source competitions — moving from MATH to AIME to IMO — and that any problem from a sufficiently elite competition is adequately challenging.

AMO-Bench fundamentally reframes this. Difficulty is not an emergent property of source selection; it is a deliberate, operationalized design constraint enforced through a two-pronged verification stage (Section 2.1, Difficulty Review). The paper establishes a difficulty floor — every problem must meet or exceed IMO standards — and then enforces it through both human expert judgment AND empirical LLM testing (requiring at least two state-of-the-art reasoning models to fail to consistently solve each problem within 3 attempts). This is a conceptual shift from "we selected problems from a hard competition" to "we can prove, through empirical verification, that each individual problem is hard for current models."

What makes this intellectually distinctive is that it solves a sustainability problem that benchmarks derived from fixed competitions cannot address. When a benchmark uses AIME 2024 problems, the difficulty of every problem is fixed at creation time — and as models improve, the entire benchmark saturates simultaneously. AMO-Bench's empirical difficulty verification creates a relative difficulty guarantee (hard for current models) rather than an absolute one (hard by IMO standards). The two criteria are complementary: IMO standards provide a stable difficulty reference, while the LLM verification ensures the benchmark remains challenging at the time of release. Future iterations of the benchmark could raise the LLM verification bar (e.g., requiring failure from GPT-6 instead of GPT-5) while maintaining the same IMO floor, creating a benchmark that can be recalibrated to remain challenging as models advance.

This is more than an incremental refinement of existing benchmark construction — it is a methodological innovation in benchmark design philosophy. The paper is essentially proposing that difficulty should be treated as a verifiable claim about each problem rather than an assumed property of the problem's source. The evidence that this matters is in Figure 1: AMO-Bench spreads top models across a 37–52% range while AIME24/25 clusters them at 88–99%, demonstrating that the difficulty floor successfully discriminates where source-based benchmarks have saturated. Figure 3b provides corroborating evidence from the solution length analysis — the dramatically longer human-annotated solutions for AMO-Bench problems (40% in the 2,048–4,096 token range vs. 0% for MATH500) confirm that the difficulty floor selects for genuinely more complex reasoning.

Innovation 2: Solving the Proof-vs-Scalability Tradeoff Through Hybrid Answer-Type Grading

The central tension in evaluating olympiad-level mathematics has been between difficulty and gradability. The hardest math competitions (IMO, USAMO) require full proofs, which demand expert human evaluation — a process that is expensive, slow, inconsistent across evaluators, and fundamentally irreproducible. Prior benchmarks that have tackled IMO-level problems (MathArena, Balunović et al., 2025; Proof or Bluff?, Petrov et al., 2025) accepted this tradeoff, opting for expert proof evaluation and living with the associated scalability limitations. The alternative approach — restricting to final-answer problems — has historically meant accepting lower difficulty, because the hardest competition problems tend to be proof-based.

AMO-Bench's core insight is that this tradeoff is not inherent to the problem difficulty but to the answer format of existing competitions. The paper demonstrates that it is possible to design original problems at IMO-level (or higher) difficulty that require only a final answer, and that the range of answer types needed to cover such problems — numerical, set, variable-expression, and descriptive — can be handled through a hybrid grading system that achieves 99.2% accuracy (validated on 1,000 responses across 10 models).

The intellectual contribution is not the individual grading methods (parser-based grading via math-verify and LLM-based grading via o4-mini are both established techniques) but rather the systematic decomposition of olympiad-level answers into a taxonomy that maps to grading strategies. The paper identifies four answer types (Section 2.3, Examples 1–4) and assigns each to the most appropriate grading method: numerical/set/expression answers (39/50 problems) use efficient, deterministic parser-based grading; descriptive answers (11/50 problems) use LLM-based grading with majority voting across 5 samples. This taxonomy is not merely a convenience — it is how the benchmark guarantees that IMO-level difficulty does not require IMO-level grading resources.

The significance extends beyond this specific benchmark. The answer-type taxonomy is a transferable framework that other benchmark designers can adopt. It says: if you can classify your problems into these four answer types, you can achieve automatic grading at high accuracy regardless of problem difficulty. This decouples difficulty from grading cost in a way that prior work had not formalized. The 99.2% grading accuracy validation (the manual check of 1,000 responses) provides empirical evidence that the taxonomy works in practice, not just in theory.

This is a fundamental advance in benchmark methodology rather than an incremental improvement. Prior to this work, the community's implicit model was that harder math benchmarks necessarily require more expensive evaluation. AMO-Bench demonstrates that this is a false necessity driven by the proof-based format of existing competitions, not by anything inherent to difficult mathematics. The benchmark achieves what the paper claims in its introduction: "IMO difficulty standards" with "automatic and robust grading" — two properties previously treated as incompatible.

Innovation 3: The Diagnostic Triangulation — Separating Capability, Reliability, and Efficiency as Independent Dimensions of Reasoning Performance

Most math benchmarks report a single aggregate metric (accuracy, pass@1, or similar) and stop there. This collapses multiple distinct aspects of reasoning performance into one number: a model with 40% accuracy might achieve that through perfect performance on 40% of problems and total failure on 60%, or through 40% success probability on every problem with high variance. These two profiles imply fundamentally different limitations (knowledge gaps vs. reasoning unreliability) and suggest different improvement strategies (more pretraining vs. better verification/search), but the aggregate metric cannot distinguish them.

AMO-Bench introduces what can be described as diagnostic triangulation: the simultaneous measurement of three independent dimensions of reasoning performance that, together, paint a much richer picture than any single metric could. The three dimensions are:

Capability (pass@k): Does the model possess the knowledge and reasoning skills to solve this problem at all, given enough attempts? Measured via pass@32 (Figure 8), which estimates the fraction of problems where at least one correct solution exists in the model's output distribution. Top models reach 74–86% pass@32, establishing that the fundamental capability to solve most AMO-Bench problems exists.
Reliability (AVG@32): Can the model consistently deploy its capability on a single attempt? Measured via AVG@32 (Figure 4), which estimates single-shot accuracy. The same top models achieve only 44–52% AVG@32, revealing a capability-reliability gap of approximately 29–34 percentage points.
Efficiency (output tokens vs. accuracy): At what computational cost does the model achieve its performance? Measured via the relationship between average output length and AVG@32 (Figures 5–6), which shows that top models consume ~37K tokens on AMO-Bench vs. ~7K on AIME25 — a 5× increase in reasoning cost for substantially lower accuracy.

The intellectual contribution is not any one of these metrics individually (pass@k and accuracy are standard, and output length analysis has appeared in prior work) but rather their simultaneous interpretation as a diagnostic system. The triangulation reveals which bottleneck is most binding for each model:

For GPT-5-Thinking (High), the pass@32 of 82% vs. AVG@32 of 52.4% indicates a reliability bottleneck: the model can solve most problems but fails to select the correct reasoning path consistently. This suggests that improvements in verification, self-consistency, or test-time search would yield large gains.
For a hypothetical model with pass@32 ≈ AVG@32 at a low value, the bottleneck would be capability: the model simply cannot solve these problems, and more sampling won't help. This would suggest fundamental pretraining or RL improvements are needed.
For models in the same series with different efficiency profiles (e.g., DeepSeek-V3.1-Thinking vs. DeepSeek-R1-0528 in Figure 5), the efficiency comparison reveals whether newer models achieve better accuracy through more reasoning (more tokens) or better reasoning (more accurate per token).

This diagnostic framework is a conceptual advance in benchmark design philosophy. Prior benchmarks answer "how good is this model?" AMO-Bench additionally answers "what should we work on to make it better?" — which is the more actionable question for research. The evidence supporting this framework is distributed across the paper: Figure 4 establishes the reliability baseline, Figure 8 establishes the capability ceiling, Figures 5–6 establish the efficiency landscape, and Figure 7 (test-time scaling) demonstrates that all three dimensions respond to increased compute, suggesting they are measuring genuine aspects of reasoning rather than artifacts of the evaluation protocol.

This is not an incremental improvement over single-metric evaluation — it is a fundamentally different way of using a benchmark that treats the evaluation data as a multi-dimensional signal to be interpreted rather than a single number to be maximized. The paper does not explicitly name this as "diagnostic triangulation," but it is the clear intellectual architecture underlying Sections 3.2's three analysis subsections (reasoning efficiency, test-time scaling, pass@k potential).

Innovation 4: Originality as a Verifiable Benchmark Property — Operationalizing Contamination Resistance Through Multi-Stage Verification

Data contamination — the possibility that models have memorized benchmark problems during pretraining — has been a persistent concern in LLM evaluation, but the field has lacked a systematic methodology for preventing it at benchmark construction time. Most benchmarks acknowledge the contamination risk and then proceed to use publicly available problems, relying on post-hoc analyses (e.g., canary strings, n-gram overlap detection after the fact) to estimate contamination rather than prevent it. Even benchmarks that commission new problems (e.g., some recent competition-style evaluations) typically rely on the problem authors' assurance of originality without systematic verification.

AMO-Bench's originality review stage (Section 2.1) operationalizes contamination resistance as a multi-stage verification process rather than a good-faith assumption. The key conceptual move is treating originality as a testable property of each problem, analogous to how difficulty is tested. The verification uses three independent detection methods:

Automated n-gram matching (10-gram character-level overlap against existing datasets like AIME24/25) to catch textual similarities.
Web search to identify problems that are conceptually similar to publicly discussed content, even if textually distinct.
Expert attestation during quality review, where reviewers explicitly declare whether they've encountered similar problems in past competitions — leveraging human memory for problems that might evade automated detection.

This is intellectually distinctive because it shifts the burden of proof. In prior benchmarks, contamination was presumed absent unless evidence demonstrated otherwise. In AMO-Bench, originality must be demonstrated through passing all three verification stages. The benchmark does not claim to guarantee zero contamination — no feasible method can — but it establishes a documented, reproducible originality standard that future benchmarks can adopt or strengthen.

The significance extends beyond this specific benchmark. The originality review framework is a transferable methodology for any benchmark that claims to measure reasoning rather than memorization. The three-stage design (automated textual check, automated conceptual check via web search, human expert check) covers different contamination vectors and creates defense-in-depth. The specific parameters (10-gram matching, the web search procedure) can be tightened or relaxed depending on the domain, but the methodological structure — treating originality as a multi-stage verification problem — is the lasting contribution.

This is best characterized as a methodological innovation with practical implications, rather than a theoretical advance. The field already knew contamination was a problem; AMO-Bench provides a concrete, replicable protocol for addressing it at construction time rather than detecting it after evaluation. The evidence that this matters is indirect but compelling: Figure 1 shows that AMO-Bench produces substantially lower scores than benchmarks derived from publicly available competition problems (AIME24/25), and while this could reflect genuine difficulty differences rather than contamination effects, the systematic originality verification at least ensures that the difficulty signal is not conflated with memorization artifacts — a confound that existing benchmarks cannot rule out.

5. Experimental Analysis

Evaluation Methodology

Dataset. The evaluation uses the full AMO-Bench dataset—50 original, human-crafted mathematical problems cross-validated to meet or exceed IMO difficulty standards. All problems are newly created and verified for originality, with no pre-existing train/test split since the benchmark is designed exclusively for evaluation. The problems span five categories: Algebraic Equations & Inequalities (11), Functions & Sequences (13), Geometry (5), Number Theory (9), and Combinatorics (12).
Base model(s). The paper evaluates 26 large language models spanning both proprietary and open-source families, including reasoning models (those with explicit chain-of-thought or extended thinking capabilities) and non-reasoning models. Specific models include GPT-5-Thinking (High/Medium/Low), o4-mini (High/Medium/Low), o3-mini (High/Medium/Low), Gemini-2.5-Pro, Gemini-2.5-Flash, Claude-Opus-4, Claude-Sonnet-4.5, DeepSeek-V3.1-Thinking, DeepSeek-R1-0528, DeepSeek-V3.1, DeepSeek-R1, DeepSeek-V3-0324, Qwen3-235B-A22B-Thinking-2507, Qwen3-Next-80B-Thinking, Qwen3-Max-Instruct, Qwen3-Next-80B-Instruct, GLM-4.5, Kimi-K2, LongCat-Flash-Thinking, LongCat-Flash, GPT-4.1, and GPT-4o-20241120. The selection is deliberately broad to establish a comprehensive performance baseline across the current LLM landscape, and the inclusion of both reasoning and non-reasoning models enables analysis of whether enhanced reasoning training is necessary for olympiad-level mathematics.
Metrics. The primary metric is AVG@32—the average accuracy across 32 independent sampling trials per problem, averaged across all 50 problems. Formally, for model M: AVG@32(M) = (1/50) × Σ_{p∈P} (1/32) × Σ_{i=1}^{32} 𝟙[correct(M, p, i)], where the indicator is 1 if the i-th sampled response to problem p is graded as correct. This metric estimates expected single-shot accuracy with reduced sampling variance compared to pass@1. Secondary diagnostic metrics include pass@k (the probability that at least one of k samples is correct, estimated using the unbiased estimator), average output token consumption (measuring reasoning efficiency), and AVG@k stability curves (tracking how estimated accuracy converges as k increases). For the AMO-Bench-P subset (39 parser-gradable problems only), AVG@32 is also reported.
Baselines. The paper does not use formal baselines in the traditional sense—there is no baseline method or model being compared against. Rather, all 26 models are evaluated on equal footing, and comparisons are made across models, across benchmark difficulty levels (AMO-Bench vs. AIME24/25 vs. MATH500 vs. HMMT25 vs. BeyondAIME), and across reasoning effort settings (Low/Medium/High for models that expose this control). The key comparative reference points are: (1) model performance on existing benchmarks (Figure 1, sourced from Meituan LongCat Team, 2025a) to establish the saturation problem, and (2) performance across the AMO-Bench model leaderboard (Figure 4) to establish the new discriminatory baseline.
Generation budget / compute accounting. Compute is measured implicitly through two mechanisms. First, output token consumption is tracked for each model (Figures 5–6), providing a direct measure of inference-time compute spent on reasoning. Second, for models with controllable reasoning effort (GPT-5, o4-mini, o3-mini), the paper varies the reasoning effort setting (Low/Medium/High) and plots accuracy against the logarithm of average output length (Figure 7), treating output length as a proxy for inference compute budget. There is no explicit FLOP accounting or generation budget constraint—models are run with their maximum allowable context/output lengths to avoid "underestimating the reasoning capabilities of the model due to restrictions on the token budget" (Section 3.1). The temperature is set to 1.0 for reasoning models and 0.7 for non-reasoning models, with top-k=50 and top-p=0.95 for all models.
Cross-validation / statistical protocol. The paper does not employ cross-validation, since AMO-Bench is an evaluation benchmark rather than a training dataset. The statistical reliability of AVG@32 is established through stability analysis (Appendix B, Figure 9), which plots AVG@k for k from 1 to 32 across 8 representative models. The paper demonstrates that as k increases, the estimated accuracy converges and model rankings stabilize, with the claim that at k=32 "it rarely appears the reverse-order phenomenon" (where model A appears better than model B in one sampling run but worse in another). For LLM-based grading of descriptive answers, majority voting across 5 independent grading samples is used to increase robustness. Grading accuracy is validated through manual quality checks on 1,000 responses from 10 different models, achieving 99.2% accuracy.

Main Quantitative Results

Overall Model Performance on AMO-Bench

Figure 4 presents the headline results: the AVG@32 performance of all 26 evaluated models on AMO-Bench. The highest-performing model is GPT-5-Thinking (High) at 52.4%, establishing the current state-of-the-art ceiling. The next tier of reasoning models clusters between 34% and 48%: Qwen3-235B-A22B-Thinking-2507 at 47.8%, DeepSeek-V3.1-Thinking at 47.6%, GPT-5-Thinking (Medium) at 47.3%, LongCat-Flash-Thinking at 43.6%, o4-mini (High) at 40.2%, Gemini-2.5-Pro at 38.7%, GLM-4.5 at 36.8%, Qwen3-Next-80B-Thinking at 34.8%, DeepSeek-R1-0528 at 34.3%, and o3-mini (High) at 32.3%. The paper notes that "most models scoring below 40%," and indeed 20 of the 26 evaluated models fall below this threshold.

A striking pattern in Figure 4 is that non-reasoning models can outperform lower-tier reasoning models. Qwen3-Max-Instruct (28.8%) and LongCat-Flash (14.6%) are non-reasoning models that surpass o3-mini (Medium) at 13.1%, DeepSeek-R1 at 10.9%, and Claude-Opus-4 at 10.6%. The paper interprets this as evidence that non-reasoning models "demonstrate a performance exceeding expectations" and possess "significant potential in tackling complex reasoning tasks." However, the absolute performance of even the best non-reasoning model (28.8%) is still far below the top reasoning models (52.4%), indicating that dedicated reasoning training provides a substantial advantage at this difficulty level.

The gap between top open-source and proprietary models is relatively narrow: Qwen3-235B-A22B-Thinking-2507 at 47.8% is only 4.6 percentage points below GPT-5-Thinking (High) at 52.4%, and DeepSeek-V3.1-Thinking at 47.6% is only 4.8 points behind. The paper characterizes this as "recent open-source advancements are closing the gap with leading commercial models."

At the lower end of the leaderboard, several models score below 10%: DeepSeek-V3.1 at 9.8%, Kimi-K2 at 7.5%, DeepSeek-V3-0324 at 5.2%, GPT-4.1 at 4.1%, and GPT-4o-20241120 at 1.5%. These near-zero scores confirm that AMO-Bench is not merely harder than existing benchmarks—it is effectively impossible for older or non-reasoning-oriented models, validating the difficulty floor established during benchmark construction.

Comparison Across Benchmarks: AMO-Bench vs. Existing Math Evaluations

Figure 1 provides the critical comparative context by showing the performance of five top-tier reasoning models across AMO-Bench, HMMT25, AIME25, AIME24, and MATH500 (with data for non-AMO benchmarks sourced from Meituan LongCat Team, 2025a). The contrast is stark:

On MATH500, all five models score above 95%: LongCat-Flash-Thinking at 99.2%, GPT-5-Thinking (High) at 99.2%, Qwen3-235B-A22B-Thinking-2507 at 99.6%, Gemini-2.5-Pro at 98.0%, and DeepSeek-V3.1-Thinking at 98.8%. The benchmark is effectively saturated—it cannot discriminate among these models.

On AIME24, performance remains high and clustered: scores range from 90.6% (LongCat-Flash-Thinking) to 94.6% (Qwen3-235B-Thinking), with GPT-5-Thinking (High) at 92.0% being surpassed by Qwen3-235B-Thinking. This is slightly more discriminative than MATH500 but still places all models within a narrow ~4 percentage point band.

On AIME25, the spread widens somewhat: LongCat-Flash-Thinking at 83.7%, GPT-5-Thinking (High) at 84.8%, Qwen3-235B-Thinking at 83.8%, Gemini-2.5-Pro at 79.3%, and DeepSeek-V3.1-Thinking at 80.4%. While this provides more discrimination than AIME24, the models are still clustered in the 79–85% range.

On HMMT25, the pattern is similar to AIME25: scores range from 76.3% (LongCat-Flash-Thinking) to 89.3% (Gemini-2.5-Pro), with GPT-5-Thinking (High) at 85.5%.

On AMO-Bench, by contrast, the same five models are spread across a 36.8–52.4% range: LongCat-Flash-Thinking at 43.6%, GPT-5-Thinking (High) at 52.4%, Qwen3-235B-Thinking at 47.8%, Gemini-2.5-Pro at 38.7%, and DeepSeek-V3.1-Thinking at 47.6%. The gap between the best and worst of these five models is 15.6 percentage points on AMO-Bench, compared to 3.6 points on MATH500, 4.0 points on AIME24, 5.5 points on AIME25, and 13.0 points on HMMT25. AMO-Bench thus provides the widest spread among these benchmarks, recovering discriminatory power lost to saturation on easier evaluations.

The paper explicitly frames this as evidence that AMO-Bench addresses the saturation problem: "These impressive results also expose an emerging challenge: many existing mathematics benchmarks are approaching performance saturation and are becoming less effective for assessing further advancements in reasoning capabilities."

Reasoning Efficiency: Output Token Consumption as a Difficulty Signal

Figure 5 plots AVG@32 against average output token consumption for all 26 evaluated models. The paper reports specific token counts for top models on AMO-Bench: GPT-5-Thinking (High) generates an average of approximately 37,000 output tokens, compared to approximately 7,000 on AIME25 and 6,000 on AIME24 (these comparison figures are reported in the main text). This represents a roughly 5–6× increase in reasoning cost for substantially lower accuracy, quantifying the additional computational burden imposed by AMO-Bench problems.

The scatter plot in Figure 5 reveals a clear positive correlation between output length and accuracy: "higher-performing models tend to require more output tokens." First-tier models exceeding 40% AVG@32 (GPT-5-Thinking High, Qwen3-235B-Thinking, DeepSeek-V3.1-Thinking, LongCat-Flash-Thinking, o4-mini High) all generate more than 35,000 tokens on average. Models in the 25–35% range (o3-mini High, Qwen3-Max-Instruct) cluster around 15,000–20,000 tokens. Lower-performing models generally produce fewer tokens, though the relationship is not perfectly monotonic—some models produce substantial output without commensurate accuracy.

The paper also identifies an efficiency improvement trend within model families over time: "o4-mini (High) outperforms o3-mini (High) at similar or slightly increased token counts" and "DeepSeek-V3.1-Thinking shows significant gains compared to DeepSeek-R1-0528 with even significantly less output tokens." This suggests that newer reasoning models are not merely spending more compute—they are achieving better reasoning efficiency (more accuracy per token), indicating genuine architectural or training improvements beyond simple test-time scaling.

Figure 6 extends this analysis across multiple benchmarks (MATH500, AIME24, AIME25, HMMT25, BeyondAIME, AMO-Bench) for six representative models. The consistent pattern across all six models is that harder benchmarks (lower accuracy) correspond to higher average output length. The paper interprets this as: "the model output length could be an indicator of reasoning challenge in the benchmark." For example, in the GPT-5-Thinking (High) panel, AMO-Bench appears at the far right of the x-axis (highest token count, ~37K) and lowest on the y-axis (lowest accuracy, ~52%), while MATH500 appears at the far left (lowest token count, ~6K) and highest accuracy (~99%). The monotonic relationship between benchmark difficulty and output token consumption across all six models validates that tokens are being spent on genuine reasoning rather than stylistic verbosity.

Test-Time Scaling: Accuracy Improves with Log Output Length

Figure 7 presents a more controlled analysis of test-time scaling by varying reasoning effort settings (Low/Medium/High) within three model families: GPT-5, o4-mini, and o3-mini. For each model family, the paper plots AVG@32 against the logarithm of average output length across the three effort settings.

The key finding is that all three model families exhibit a "near-linear growth trend in AVG@32 as the logarithm of average output length increases." The paper explicitly states this is "highly aligned with earlier experimental observations from existing benchmarks such as MATH500 and AIME24 (Muennighoff et al., 2025)," referencing the s1 work on simple test-time scaling.

For GPT-5, AVG@32 increases from approximately 26% (Low, ~8K tokens) to approximately 47% (Medium, ~16K tokens) to 52.4% (High, ~37K tokens), with the log-linear relationship appearing roughly consistent across this range. For o4-mini, the progression goes from approximately 13% (Low) to approximately 30% (Medium) to approximately 40% (High). For o3-mini, the range is from approximately 5% (Low) to approximately 13% (Medium) to approximately 32% (High).

The paper interprets the continued log-linear scaling as evidence that "further increasing the inference budget will further drive improvements on AMO-Bench" and that "these results highlight the significant room for improving the mathematical reasoning in current LLMs." This is a forward-looking claim: unlike benchmarks where test-time scaling has plateaued (indicating that models have exhausted the available reasoning headroom), AMO-Bench still shows clear scaling trends, suggesting that future models with larger inference budgets (or more efficient reasoning per token) will continue to improve.

The use of logarithmic scaling on the x-axis is noteworthy: it implies that each doubling of output length yields a roughly constant absolute improvement in accuracy. If this trend continues, reaching 70% accuracy might require approximately 4× the current output length (two doublings), and reaching 90% might require 16× (four doublings)—a computationally expensive proposition but one that follows a predictable scaling law.

Pass@k Analysis: Latent Capability Exceeds Observed Performance

Figure 8 reports pass@k for k ∈ {1, 2, 4, 8, 16, 32} across 8 representative models. The pass@k metric estimates the probability that at least one of k independent samples is correct, revealing the model's latent capability separate from its single-shot reliability.

The paper reports specific pass@32 values (also displayed in the heatmap overlay on Figure 8): GPT-5-Thinking (High) reaches 82.0%, LongCat-Flash-Thinking reaches 86.0%, Qwen3-235B-Thinking reaches 74.0%, DeepSeek-V3.1-Thinking reaches 78.0%, Qwen3-Max-Instruct reaches 68.0%, LongCat-Flash reaches 54.0%, DeepSeek-V3.1 reaches 44.0%, and GPT-4.1 reaches 26.0%.

The capability-reliability gap—the difference between pass@32 and AVG@32—is substantial for top models:

GPT-5-Thinking (High): 82.0% − 52.4% = 29.6 percentage points
DeepSeek-V3.1-Thinking: 78.0% − 47.6% = 30.4 percentage points
Qwen3-235B-Thinking: 74.0% − 47.8% = 26.2 percentage points
LongCat-Flash-Thinking: 86.0% − 43.6% = 42.4 percentage points

The paper interprets this as evidence that "top-tier models achieve pass@32 rates exceeding 70%, suggesting they possess the initial capability to solve these challenging problems even if they do not consistently identify the correct reasoning path at present." The pass@k curves show "rapid growth as k increases from 1 to 8, followed by a sustained but gradual improvement as k continues to rise," indicating diminishing returns to additional sampling but with meaningful gains persisting even at larger k.

This analysis demonstrates that the primary limitation on AMO-Bench is reasoning reliability rather than fundamental capability. Models do possess the knowledge and skills to solve most problems—given enough attempts, they find correct solutions for 74–86% of problems—but they cannot consistently deploy this capability on a single attempt. This has direct implications for improvement strategies: better verification, test-time search, or self-consistency techniques could close the capability-reliability gap without requiring fundamentally new capabilities from pretraining or reinforcement learning.

Stability of AVG@k Estimates

Appendix B, Figure 9 demonstrates why 32 samples were chosen for the primary metric. The figure plots AVG@k for k from 1 to 32 across 8 representative models, showing how the estimated accuracy evolves and stabilizes as the number of samples increases. At small k (1–4), the curves are noisy and model rankings occasionally reverse—for example, a model that is truly better might appear worse in a particular sampling run due to variance. As k increases, the curves converge toward stable values, and the paper asserts that at k=32, "it rarely appears the reverse-order phenomenon." The specific numbers in the heatmap overlay show how the estimated AVG@k for each model converges: for GPT-5-Thinking (High), AVG@1 is approximately 58% (reflecting high variance from a single sample), AVG@4 is approximately 53.3%, AVG@8 is approximately 52.5%, AVG@16 is approximately 52.4%, and AVG@32 is 52.4%. The convergence by k=16 suggests that 32 samples provides a conservative margin above the point of diminishing returns in variance reduction.

Ablation Studies and Robustness Checks

Parser-based vs. LLM-based grading (AMO-Bench-P subset ablation): The paper reports AVG@32 on the AMO-Bench-P subset (39 parser-gradable problems only) in Appendix C, Table 1, alongside full AMO-Bench scores. Performance on AMO-Bench-P is consistently higher than on the full benchmark, confirming that descriptive-answer problems are inherently more challenging. For example, GPT-5-Thinking (High) scores 54.8% on AMO-Bench-P vs. 52.4% on full AMO-Bench, Qwen3-235B-Thinking scores 56.2% vs. 47.8%, and LongCat-Flash-Thinking scores 45.3% vs. 43.6%. The gap is not uniform across models—Qwen3-235B-Thinking shows an 8.4 percentage point difference while LongCat-Flash-Thinking shows only 1.7 points—indicating that descriptive-answer problems differentially challenge models, potentially reflecting differences in how models handle multi-part or conditional answers.

Grading accuracy validation: The paper reports a manual quality check on 1,000 responses generated by 10 different LLMs, finding grading accuracy of 99.2%. This validation is not an ablation per se but serves as a robustness check on the grading infrastructure. The paper does not break down grading accuracy by answer type (parser-based vs. LLM-based) or by problem, which would be informative for identifying whether grading errors are concentrated in specific problems or answer formats. The 0.8% error rate (~8 errors in 1,000 responses) is small enough that it is unlikely to meaningfully affect model rankings, but if errors are concentrated in specific problems (e.g., a particular descriptive-answer problem where the grading LLM consistently misjudges equivalence), those problems could introduce systematic bias.

Reasoning effort settings (test-time scaling control): Figure 7 serves as a controlled ablation of reasoning effort within fixed model families. By varying only the reasoning effort setting (Low/Medium/High) while holding the model architecture and training fixed, the paper isolates the effect of test-time compute on performance. The consistent log-linear relationship across all three model families (GPT-5, o4-mini, o3-mini) suggests that the scaling behavior is a general property of reasoning models on this benchmark rather than an artifact of a specific model's architecture or training. The fact that all three families show this pattern—despite different absolute performance levels—provides converging evidence that AMO-Bench captures genuine test-time scaling dynamics.

Cross-benchmark consistency (output length as difficulty proxy): Figure 6 provides a robustness check on the claim that output length reflects benchmark difficulty. By plotting six independent models across six benchmarks, the paper demonstrates that the monotonic relationship between difficulty and output length is not model-specific—it holds for GPT-5-Thinking (High), GLM-4.5, Qwen3-235B-Thinking, Gemini-2.5-Pro, DeepSeek-V3.1-Thinking, and LongCat-Flash-Thinking. If the relationship were spurious (e.g., driven by a single verbose model), it would not replicate across six independent model families. The consistency across models and benchmarks provides strong evidence that output length is a genuine proxy for reasoning complexity rather than an artifact of model-specific verbosity.

Model ordering stability across k (sampling variance control): Appendix B, Figure 9 demonstrates that the relative ordering of models stabilizes as the number of samples increases. This is important because the paper's conclusions about which models perform best depend on reliable ranking. The figure shows that at k=32, the ordering among the 8 representative models is stable—the paper claims it "rarely appears the reverse-order phenomenon." This validates the choice of 32 samples for the primary metric and provides confidence that the leaderboard rankings in Figure 4 reflect genuine performance differences rather than sampling noise.

Critical Assessment

Do the experiments demonstrate that AMO-Bench addresses benchmark saturation?

The central claim of the paper is that existing math benchmarks are approaching saturation and that AMO-Bench restores discriminatory power. The evidence for this claim is primarily in Figure 1 and the cross-benchmark comparison in Section 3.2. The data are compelling: on MATH500, the top five models cluster within a ~4 percentage point band (95–99%); on AIME24, within ~4 points (90–94%); on AMO-Bench, within ~16 points (37–52%). This is a clear demonstration that AMO-Bench spreads models across a wider performance range.

However, the claim that this spread reflects genuine reasoning differences rather than variance from a small test set requires scrutiny. With 50 problems and 32 samples per problem, the standard error of AVG@32 for a model with true accuracy around 50% is approximately sqrt(0.5 × 0.5 / (50 × 32)) ≈ 1.25 percentage points (treating each of the 1,600 problem-sample pairs as independent, though problem-level correlations likely increase this). The observed spread of 15.6 points between the best and worst of the top five models substantially exceeds this sampling error, supporting the claim that the differences are real. However, the paper does not report confidence intervals or standard errors, making precise assessment difficult. A 95% confidence interval of roughly ±2.5 percentage points around each AVG@32 estimate would mean that several models in the 34–48% range could overlap in their true accuracies, complicating fine-grained ranking claims.

A more fundamental limitation is that the saturation comparison in Figure 1 uses accuracy scores from Meituan LongCat Team (2025a) for the non-AMO benchmarks rather than re-evaluating all models under the same conditions. The sampling parameters, prompt format, and grading methodology may differ between the AMO-Bench evaluation and the external sources, making the cross-benchmark comparison less controlled than it appears. A within-study evaluation of the same models on all benchmarks under identical conditions would strengthen the saturation claim.

Do the experiments demonstrate that models possess latent capability exceeding their single-shot performance?

The gap between pass@32 and AVG@32 (Figure 8 vs. Figure 4) is substantial—29.6 percentage points for GPT-5-Thinking (High), 42.4 points for LongCat-Flash-Thinking. This is strong evidence that models can solve problems they cannot solve consistently. However, the interpretation that this represents "initial capability" that can be unlocked through better verification or search requires an important caveat: the pass@k metric treats any correct answer among k samples as success, but in practice, the challenge is knowing which sample is correct without access to ground truth. The paper does not evaluate whether existing verification methods (process reward models, self-consistency, majority voting) can actually close this gap on AMO-Bench—it only demonstrates that correct solutions exist in the output distribution, not that they can be reliably identified. This is a significant gap: the capability ceiling is measured, but the path to reaching it is not tested.

Additionally, pass@k estimates become less reliable at larger k. The unbiased estimator uses combinations from the 32 available samples, and for k close to 32, the estimator is based on a small effective sample size. The paper's pass@32 estimates (e.g., 82.0% for GPT-5-Thinking High) are extrapolations from 32 samples that ask: if you took 32 entirely new independent samples, what's the probability at least one is correct? The variance of this estimate depends on the problem-level pass@1 distribution—if some problems have near-zero pass@1 and others near-100%, the estimate is more reliable than if all problems have intermediate pass@1. The paper does not report variance estimates for pass@k, making the precision of these numbers uncertain.

Do the test-time scaling results (Figure 7) support the claim that AMO-Bench will not saturate soon?

The log-linear relationship between output length and accuracy is consistent with prior findings and suggests headroom for improvement. However, three considerations temper this interpretation. First, the scaling analysis uses only three data points per model (Low/Medium/High), and the "near-linear" claim is based on interpolation and extrapolation from these three points. Three points can always be fit with a line; more reasoning effort settings (or continuous control over compute) would provide stronger evidence for the functional form. Second, the analysis does not control for the possibility that "High" reasoning effort differs from "Medium" in ways beyond output length—different prompt strategies, different decoding parameters, or different model snapshots could confound the compute-accuracy relationship. Third, log-linear scaling on current benchmarks has historically been observed up to a point and then plateaued; the paper cannot distinguish whether AMO-Bench is in the linear regime of a sigmoid that will eventually saturate or whether it will continue scaling indefinitely. The paper's forward-looking claim that "further increasing the inference budget will further drive improvements" is reasonable given the current data but should be understood as a projection, not a demonstrated fact.

Do the experiments support the claim that output length indicates reasoning challenge?

The cross-benchmark analysis in Figure 6 shows a consistent monotonic relationship between benchmark difficulty and output length across six models. This is strong correlational evidence. However, correlation does not establish causation—it is possible that both benchmark difficulty and output length are driven by a third factor (e.g., the type of reasoning required, the mathematical domains covered) rather than output length being a direct reflection of reasoning complexity. The paper acknowledges this implicitly by describing output length as an "indicator" rather than a causal measure. A stronger test would involve an experimental manipulation: for the same benchmark, vary the difficulty (e.g., by selecting easier vs. harder subsets) and observe whether output length tracks the within-benchmark difficulty gradient. The paper does not perform this analysis, leaving open the possibility that benchmark-specific factors (prompt style, expected answer format, domain composition) drive the correlation.

What experiments would have strengthened the paper?

Several additional analyses could have strengthened the claims:

Confidence intervals on all metrics. The paper reports point estimates (AVG@32, pass@k) without uncertainty quantification. Bootstrap confidence intervals would clarify which model comparisons are statistically reliable and which fall within sampling noise.
Problem-level difficulty analysis. The paper could examine which specific problems are hardest (lowest AVG@32 across models), whether difficulty is consistent across models (do all models struggle on the same problems?), and whether problem category (algebra vs. geometry vs. combinatorics) predicts difficulty. This would provide diagnostic information about which reasoning skills are most challenged by AMO-Bench.
Verification method evaluation. Since the paper identifies a large capability-reliability gap, testing whether existing verification techniques (majority voting, best-of-N with a process reward model, self-consistency) can close this gap on AMO-Bench would directly test the actionable implication of the pass@k results.
Within-study cross-benchmark comparison. Re-evaluating a subset of models on AIME24/25 and MATH500 under the same sampling protocol (temperature, top-k, top-p, prompt format, 32 samples) would enable direct, controlled comparison of saturation levels without relying on external data.
Scaling with more fine-grained compute control. For models that allow it, testing more than three reasoning effort levels would provide stronger evidence for the claimed log-linear scaling relationship and enable better extrapolation to future compute budgets.
Human baseline. The paper claims IMO-level difficulty but does not report human performance on AMO-Bench. A human baseline (e.g., performance of IMO medalists or strong competition math students on these problems under time constraints) would contextualize the model scores and validate the difficulty claims.

What genuine weaknesses exist in the experimental design?

The most significant weakness is the small test set size (50 problems) combined with reporting only point estimates. With 50 problems, each problem contributes 2% to the overall AVG@32. A model that solves one additional problem (out of 50) gains 2 percentage points—meaning that fine-grained model comparisons (e.g., GPT-5-Thinking High at 52.4% vs. Qwen3-235B-Thinking at 47.8%) are based on differences of roughly 2–3 problems. This granularity is coarse enough that small differences in problem selection (e.g., if 2–3 of the 50 problems happen to be in domains where a particular model is unusually strong or weak) could meaningfully shift rankings. The paper's stability analysis (Figure 9) addresses sampling variance across the 32 trials but does not address the sensitivity of the benchmark to the specific 50 problems chosen. A benchmark with more problems (e.g., 200–500) would provide finer-grained discrimination and greater robustness to problem-specific effects.

A related weakness is that the difficulty review stage—which filters out problems that top models can consistently solve—creates a potential circularity in the benchmark's difficulty claims. The problems were selected partly because GPT, DeepSeek, and Gemini series models failed to solve them consistently in preliminary evaluation. The same model families (or their newer versions) are then evaluated on the final benchmark and found to score poorly, which partially reflects the selection criterion rather than an independent assessment of difficulty. This does not invalidate the benchmark—the IMO-level human expert verification provides independent difficulty validation—but it means the "models struggle on AMO-Bench" finding is partly by construction rather than a purely surprising empirical result.

The single benchmark domain (olympiad-level mathematics) limits the generalizability of the diagnostic insights. The paper's findings about test-time scaling, pass@k gaps, and output-length-vs-accuracy relationships are specific to mathematical reasoning. Whether similar patterns hold for other reasoning domains (code generation, logical deduction, scientific reasoning) is unknown but plausible. The paper does not claim broader generalizability, so this is a scope limitation rather than a flaw, but it means AMO-Bench answers the question "how well do LLMs reason about olympiad math?" rather than "how well do LLMs reason?" in general.

Finally, the lack of error analysis is a missed opportunity. The paper does not categorize the types of errors models make (calculation errors, logical gaps, misinterpretation of problem statements, incomplete case analysis, etc.), which would provide actionable diagnostic information. The human-annotated reasoning paths included with each problem could support such analysis—comparing model outputs to human solutions could reveal where models deviate from correct reasoning—but the paper does not perform this comparison. This is consistent with the paper's scope as a benchmark introduction rather than a comprehensive model analysis, but it means the diagnostic potential of the benchmark is demonstrated rather than fully realized.

6. Limitations and Trade-offs

1. Small Benchmark Size (50 Problems) Limits Statistical Precision and Robustness

The assumption or constraint. AMO-Bench consists of exactly 50 problems. As an evaluation benchmark, its discriminatory power depends on the number of independent test items: with fewer problems, each correct or incorrect answer shifts the aggregate score by a larger increment (2 percentage points per problem), and the benchmark becomes more sensitive to the specific problems selected rather than the underlying reasoning capability it aims to measure.

The paper does not explicitly justify the choice of 50 problems beyond the practical constraints of the construction pipeline (human experts creating and cross-validating original IMO-level problems is inherently slow and expensive). There is no discussion of whether 50 problems provides adequate statistical power for the model comparisons the paper draws.

The consequence. With 50 problems, the granularity of AVG@32 is coarse: each problem contributes 2% to the overall score. The gap between the top two models—GPT-5-Thinking (High) at 52.4% and Qwen3-235B-A22B-Thinking-2507 at 47.8% (Figure 4)—is 4.6 percentage points, which corresponds to roughly 2–3 problems out of 50. If even 2–3 of the 50 problems happen to be in domains where a particular model is unusually strong or weak (e.g., due to training data composition rather than reasoning ability), the relative ordering of top models could shift.

More importantly, without confidence intervals, it is unclear whether the observed gaps between closely-ranked models are statistically reliable. For a model with true single-shot accuracy of 50% on 50 problems with 32 samples each, the standard error of AVG@32 is approximately:

$\text{SE} \approx \sqrt{\frac{p(1-p)}{n_{\text{problems}} \times n_{\text{samples}}}} = \sqrt{\frac{0.5 \times 0.5}{50 \times 32}} \approx 1.25\%$

This calculation treats each problem-sample pair as independent, which overstates precision (problem-level correlations increase the true standard error). Even under this optimistic assumption, a 95% confidence interval would span roughly ±2.5 percentage points around each estimate, meaning that models within a ~5-point band may not be reliably separable. In practice, the true standard error is likely larger, and the paper's leaderboard (Figure 4) implies fine-grained rankings that the benchmark size may not fully support.

A deeper consequence is that AMO-Bench may be sensitive to the specific problem composition. With 50 problems, domain imbalances matter: Geometry comprises only 5 problems (10% of the benchmark, Figure 3a), so a model that is unusually weak at geometry (a distinct reasoning skill from algebra or combinatorics) would lose at most 5 problems, while a model weak at Functions & Sequences could lose up to 13 problems. This domain imbalance means the benchmark weights different mathematical reasoning skills unevenly, which is not necessarily a flaw—no benchmark perfectly balances all subskills—but the small total size amplifies the impact of this imbalance on model rankings.

What evidence exists in the paper. The paper does not directly measure the statistical reliability of model comparisons or the sensitivity of rankings to problem composition. The stability analysis in Appendix B (Figure 9) addresses one dimension of variance—sampling variance across the 32 trials per problem—and shows that AVG@k stabilizes by k=32, but this analysis conditions on the fixed set of 50 problems. It answers the question "if we ran the same models on the same 50 problems again, would we get the same ranking?" but not "if we had selected a different set of 50 problems from the same difficulty distribution, would the rankings be similar?" These are fundamentally different sources of variance, and the paper only addresses the first.

Figure 4 implicitly provides some evidence that the benchmark discriminates in a way that aligns with prior knowledge (reasoning models outperform non-reasoning models; frontier models from major labs cluster at the top), which provides face validity but does not quantify uncertainty. The paper also reports performance on AMO-Bench-P (Table 1, Appendix C), the 39-problem parser-gradable subset, where the top model order is slightly different: Qwen3-235B-Thinking (56.2%) edges out GPT-5-Thinking High (54.8%) on this subset, compared to the full benchmark where GPT-5 leads (52.4% vs. 47.8%). This reversal on a 39-problem subset of the same benchmark suggests that problem selection can indeed shift model rankings, even within the same underlying difficulty distribution.

Mitigation status. The paper does not address benchmark-size limitations directly. There is no discussion of confidence intervals, bootstrap standard errors, or problem-level variance decomposition. The stability analysis (Figure 9) is partial—it addresses sampling variance within problems but not problem-selection variance across the benchmark. The paper does not suggest expanding the benchmark in future work, though the construction pipeline description implies that expansion is feasible (the pipeline is designed for ongoing problem creation). The issue is essentially unacknowledged and unmitigated.

For a practitioner deciding whether to use AMO-Bench for model comparison, this is the most consequential limitation: the benchmark can reliably distinguish large performance gaps (e.g., GPT-5-Thinking High vs. GPT-4o at 1.5%) but may not reliably order models within the same tier (e.g., the 34–48% cluster of reasoning models in Figure 4). A user who treats the exact rank ordering as meaningful is over-interpreting the data given the benchmark's size.

2. The 2048-Sample Difficulty Estimation Cost in the Example Summary Is Not Applicable Here — But Benchmark Construction Difficulty Filtering Creates a Different Circularity Concern

The assumption or constraint. The difficulty review stage (Section 2.1) filters candidate problems by requiring that "at least two [state-of-the-art reasoning] models fail to correctly and consistently solve the problem," where "consistently" is operationalized as failing to achieve 3/3 correct samples for a given model. This means the final set of 50 problems is selected because certain model families (GPT, DeepSeek, Gemini series) could not solve them reliably during preliminary evaluation.

The paper is transparent about this: "We employed multiple advanced reasoning models (such as GPT, DeepSeek, and Gemini series models) for preliminary evaluation, requiring that at least two such models fail to correctly and consistently solve the problem" (Section 2.1). The difficulty review is described as a feature—a mechanism to ensure adequate challenge—not as a limitation.

The consequence. The problem selection criterion introduces a form of circularity into the headline finding that "LLMs struggle on AMO-Bench." The benchmark was explicitly constructed to include problems that GPT, DeepSeek, and Gemini models could not reliably solve. When the same model families (or their newer versions) are then evaluated on the final benchmark and found to score below 52.4% (Figure 4), this reflects the construction criterion at least as much as it reflects an independent empirical discovery about model capabilities.

This does not make the benchmark invalid—the IMO-level human expert verification provides independent difficulty validation that does not depend on LLM performance—but it means the "struggle" narrative is partly built into the benchmark design. The paper's finding that models perform worse on AMO-Bench than on AIME24/25 is a genuine reflection of increased difficulty, but the specific magnitude of the gap (e.g., 52.4% vs. 84.8% on AIME25 for GPT-5-Thinking High, Figure 1) is influenced by how aggressively the difficulty review filtered problems. A less aggressive filter (e.g., requiring only one model to fail, or allowing 2/3 success as "consistent") would produce an easier benchmark; a more aggressive filter (requiring all models to fail, or using more than 3 attempts) would produce an even harder one. The reported numbers are thus partly artifacts of the specific filtering threshold.

A subtler consequence: the problem set may be adversarially selected against the specific model families used in the difficulty review. If GPT-series models have systematic weaknesses in certain problem types (e.g., combinatorial case analysis) that DeepSeek-series models handle better, the difficulty filter might retain problems that exploit GPT's specific weaknesses. When GPT-5 is then evaluated on the final benchmark, its performance reflects both genuine difficulty and potential adversarial selection. The requirement that at least two model families fail mitigates this somewhat—a problem must challenge at least two different architectures to pass the filter—but does not eliminate it, especially if the two failing models share architectural similarities or training data biases.

What evidence exists in the paper. The paper provides the construction criterion explicitly (Section 2.1) and reports the final performance of multiple model families (Figure 4), including both the families used in difficulty filtering and families not used (Claude, Qwen, GLM, Kimi, LongCat). The fact that all evaluated models—including those not involved in filtering—perform poorly (Claude-Opus-4 at 10.6%, Kimi-K2 at 7.5%) provides some evidence that the difficulty is genuine rather than purely an artifact of adversarial selection against specific model families. If difficulty were solely due to adversarial filtering against GPT/DeepSeek/Gemini, we would expect models from other families to perform substantially better, but they generally perform worse.

However, the paper does not report which specific models were used in the difficulty review stage, how many candidate problems were rejected at this stage, or the distribution of model failure patterns (e.g., did the same two models fail on all retained problems, or did different subsets of problems challenge different models?). This information would help assess the degree of adversarial selection.

Mitigation status. The paper partially mitigates this concern through the dual difficulty criterion: human expert verification that problems meet IMO standards provides difficulty validation independent of LLM performance. The IMO difficulty floor is an absolute standard, not a relative one, so even if the LLM-based filtering introduces some circularity, the problems would be challenging regardless. The paper also notes that preliminary evaluation used "multiple" models from different families, and the two-model failure requirement provides some robustness against model-specific weaknesses.

However, the mitigation is incomplete. The human expert criterion establishes a difficulty floor but not a ceiling—a problem could meet IMO standards and still be solvable by current models, as evidenced by the fact that top models score 52.4% rather than 0%. The LLM-based filtering is what pushes the difficulty beyond "meets IMO standards" to "cannot be consistently solved by current frontier models," and this additional filtering is what creates the circularity. The paper does not report what fraction of IMO-level problems were rejected by the LLM filter, which would quantify the incremental effect of the model-based selection.

For a practitioner interpreting the benchmark results, the key implication is that AMO-Bench's scores should be understood as reflecting both genuine mathematical difficulty and a deliberate selection bias toward problems that current models find challenging. This is not a flaw per se—it is exactly what the benchmark was designed to do—but it means that AMO-Bench's difficulty relative to other benchmarks is partly by design rather than purely an empirical property of olympiad mathematics. Future models trained with different architectures or on different data distributions might find AMO-Bench easier or harder than expected based on extrapolation from current model trends.

3. The Capability-Reliability Gap Is Diagnosed but the Path to Closing It Is Not Tested

The assumption or constraint. The paper identifies a large gap between pass@32 (what models can solve with enough attempts) and AVG@32 (what models do solve on a single attempt). For GPT-5-Thinking (High), this gap is 29.6 percentage points (82.0% pass@32 vs. 52.4% AVG@32, Figures 8 and 4). The paper interprets this as evidence that models "possess the initial capability to solve these challenging problems even if they do not consistently identify the correct reasoning path at present" and that there is "significant room for improvement in the reasoning capabilities of LLMs."

The implicit assumption is that this latent capability is actionable—that techniques like majority voting, best-of-N selection with a verifier, or other test-time search strategies can convert pass@k potential into improved single-shot or budgeted performance. The paper does not test this assumption.

The consequence. The capability-reliability gap is diagnosed but unvalidated as a practical path to improvement. There are several reasons the gap might not be closable in practice:

The verification problem is unsolved for this difficulty level. To select the correct answer among k samples, you need a verifier that can distinguish correct from incorrect solutions without access to ground truth. On AMO-Bench, where problems require ~37K tokens of reasoning (Figure 5), verifying correctness is itself a reasoning task of comparable difficulty to solving the problem. Standard approaches—majority voting, outcome reward models, process reward models—may not be reliable enough at this difficulty level to effectively select the correct solution. The paper provides no evidence either way.
The correlation structure of model errors matters. If a model's errors are randomly distributed (each sample fails for independent reasons), then pass@k grows roughly as 1 − (1 − p)^k, and sampling can rapidly convert a moderate per-attempt success rate into high overall success. But if errors are systematic—the model makes the same mistake on the same problems across most samples—then pass@k plateaus quickly and additional sampling yields minimal gains. Figure 8 shows that pass@k does continue to improve with k (the curves have not fully plateaued at k=32), but the rate of improvement slows, suggesting a mix of random and systematic errors. The paper does not characterize this mix, which determines how much of the pass@k ceiling is actually reachable.
The cost of verification at scale may be prohibitive. Even if an effective verifier exists, running 32 independent solutions per problem (each consuming ~37K tokens) and then verifying them would cost 32× the already-high single-shot inference cost. For GPT-5-Thinking (High), a single AMO-Bench evaluation at AVG@32 already requires ~50 problems × 32 samples × 37K tokens ≈ 59 million output tokens. Adding verification would multiply this further. The paper does not discuss the cost implications of closing the capability-reliability gap.

What evidence exists in the paper. The paper provides clear evidence for the existence of the gap but no evidence for its closeability. Figure 8 shows pass@k curves for 8 models, demonstrating that latent capability substantially exceeds observed performance. The paper's test-time scaling analysis (Figure 7) shows that increased compute (via higher reasoning effort settings) improves accuracy within single-model rollouts—this is a different mechanism from post-hoc selection among independent samples. The paper does not evaluate any post-hoc selection strategy (majority voting, verifier-based selection, self-consistency) on AMO-Bench, nor does it compare the cost-effectiveness of increased single-sample compute vs. increased sampling with selection.

The paper acknowledges this implicitly by characterizing the pass@k results as showing "potential" and "room for improvement" rather than as a demonstrated path to improvement. The language is appropriately cautious: models "possess the initial capability" and there are "substantial opportunities" for improvement, but the paper stops short of claiming that these opportunities can be realized with current techniques.

Mitigation status. The paper does not attempt to mitigate this limitation—it is fundamentally an open question that the benchmark raises but does not answer. The diagnostic framework (measuring pass@k alongside AVG@32) is itself the contribution: it identifies the gap as a research target. The paper's role is to provide the measurement instrument; closing the gap is left to future work.

For a practitioner, this limitation means that the encouraging pass@32 numbers (70–86% for top models, Figure 8) should not be interpreted as near-term achievable performance. They represent an upper bound on what is possible with perfect verification, but the path from 52.4% AVG@32 to 82.0% pass@32 may be long and expensive. A developer deciding whether to invest in test-time compute scaling for olympiad-level math should treat the AVG@32 numbers, not the pass@32 numbers, as the current practical performance baseline.

4. Single-Domain, Single-Format Evaluation Limits Generalizability of Diagnostic Insights

The assumption or constraint. AMO-Bench evaluates mathematical reasoning exclusively, using only olympiad-style competition problems within the IMO knowledge syllabus (high-school mathematics excluding calculus). The paper does not claim broader generalizability—it presents AMO-Bench as a math benchmark—but the diagnostic analyses (test-time scaling, pass@k gaps, output-length-vs-accuracy relationships) are interpreted as general insights about LLM reasoning behavior rather than math-specific phenomena.

The paper's related work section (Section 4) explicitly distinguishes AMO-Bench from benchmarks that test broader mathematical knowledge (FrontierMath, HLE) and from benchmarks in other domains (code generation, scientific reasoning), positioning AMO-Bench within the olympiad-math niche. The diagnostic claims are bounded by this scope: the paper does not assert that the same scaling patterns hold for code generation or logical reasoning, but it also does not discuss whether the patterns might be math-specific.

The consequence. Several of the paper's key findings may not generalize beyond olympiad mathematics:

Test-time scaling (Figure 7). The near-linear relationship between log output length and accuracy may be specific to tasks where "thinking longer" directly maps to "exploring more reasoning paths" or "verifying intermediate steps more carefully." In mathematical reasoning, additional tokens correspond to additional derivations, case analyses, or verification steps—activities that have a natural nested structure. In other reasoning domains (e.g., open-ended creative writing, strategic planning), the relationship between output length and quality may be fundamentally different (diminishing returns, or even negative returns if verbosity introduces incoherence).
Pass@k as a capability measure (Figure 8). The interpretation of pass@k as "latent capability" assumes that correct solutions are meaningfully different from incorrect ones—that there exists a subset of reasoning paths that lead to the right answer, and the model sometimes finds them. In mathematics, this assumption holds because problems have well-defined correct answers and solutions are verifiable. In domains where correctness is ambiguous (summarization quality, dialogue helpfulness), pass@k computed against a single reference answer may not capture genuine capability.
Output length as difficulty proxy (Figures 5–6). The finding that output length correlates with benchmark difficulty may be specific to mathematical reasoning, where harder problems require more steps. In other domains, difficulty might manifest as requiring more nuanced judgment rather than more steps, which may not correlate with output length. A model might produce verbose but shallow reasoning for a hard ethical dilemma, or terse but insightful reasoning for a simple factual question.
The reasoning vs. non-reasoning model distinction (Figure 4). The finding that reasoning models substantially outperform non-reasoning models on AMO-Bench (top reasoning models at 34–52% vs. best non-reasoning at 28.8%) may not hold for tasks where extended chain-of-thought provides less advantage. The gap between reasoning and non-reasoning models on AMO-Bench quantifies the value of explicit reasoning for olympiad mathematics specifically, not for reasoning in general.

What evidence exists in the paper. The paper provides no cross-domain comparison. All analyses—AVG@32, pass@k, test-time scaling, output length correlation—are conducted exclusively on math benchmarks (MATH500, AIME24/25, HMMT25, BeyondAIME, AMO-Bench in Figures 1 and 6). The consistency of patterns across these math benchmarks (e.g., the monotonic difficulty-output-length relationship in Figure 6 holds across six models and six benchmarks) provides within-domain replication but does not demonstrate cross-domain generalizability.

The paper references Muennighoff et al. (2025) for the test-time scaling finding, noting alignment with "earlier experimental observations from existing benchmarks such as MATH500 and AIME24." This external reference provides some evidence that the log-linear scaling pattern extends across multiple math benchmarks, but the referenced work also focuses on mathematical reasoning, so the domain remains math-specific.

Mitigation status. The paper does not address generalizability. This is a scope limitation rather than a flaw—the paper is explicitly a math benchmark paper—but it means that readers should not extrapolate the diagnostic findings to other reasoning domains without independent verification. The paper's contributions (the benchmark itself and the diagnostic framework) are designed for mathematical reasoning evaluation, and the insights derived from them are correspondingly bounded.

For a practitioner using AMO-Bench to evaluate models intended for general reasoning tasks (code generation, scientific Q&A, multi-step planning), this limitation is important: strong AMO-Bench performance may indicate strong mathematical reasoning specifically, but it may not predict performance on non-mathematical reasoning tasks. Conversely, weak AMO-Bench performance does not necessarily indicate weak general reasoning—a model might be excellent at code generation or logical deduction while struggling with olympiad mathematics due to domain-specific knowledge gaps or unfamiliarity with competition-style problem structures.

5. Grading Methodology Has Untested Sensitivity to Edge Cases and Distribution Shift

The assumption or constraint. AMO-Bench employs a hybrid grading system: parser-based grading via math-verify for 39/50 problems (numerical, set, and variable-expression answers) and LLM-based grading via o4-mini (Low) with majority voting across 5 samples for 11/50 problems (descriptive answers). The paper reports 99.2% grading accuracy based on manual verification of 1,000 responses from 10 different models.

The implicit assumptions are: (1) the grading accuracy measured on this validation set generalizes to all models and all response types that future evaluations might encounter, (2) the parser-based grading (which uses math-verify for expression equivalence checking and requires ≥4 decimal places for numeric approximations) handles the full range of valid answer formats that models might produce, and (3) the LLM-based grading with o4-mini (Low) remains reliable as the underlying grading model may be updated or deprecated by its provider.

The consequence. Several failure modes could compromise grading accuracy in ways not captured by the validation:

Parser fragility to format variation. The structured output prompt (Appendix A, Example 5) instructs models to use \boxed{<answer>}, but models may produce answers with non-standard LaTeX, ambiguous notation, or formatting that math-verify cannot parse. The paper acknowledges performing "manual review of parsing results during the preliminary evaluation and adjusting post-processing algorithms" (Section 2.3), which suggests that parser failures did occur and required human intervention. As new models with different output distributions are evaluated, new parsing edge cases may emerge. The grading accuracy validation (1,000 responses, 99.2% accuracy) was conducted on a specific set of 10 models; a new model that formats answers differently could trigger unanticipated parsing failures.
Expression equivalence testing for variable-expression answers (Example 3). The paper's approach—substituting "multiple sets of values" and checking numerical agreement—is a sound but incomplete method for verifying mathematical equivalence. Two expressions can agree on many test points while being mathematically distinct (e.g., $\sin(x)$ and a polynomial approximation that matches at the test points). The paper does not specify how many test values are used, how they are selected, or whether the method handles expressions with restricted domains, singularities, or piecewise definitions. A model that produces a valid but unconventional expression might be incorrectly marked wrong if the test values happen to fall outside the expression's domain or if the numerical evaluation encounters precision issues.
LLM-based grading reliability over time. The grading model is o4-mini (Low), a specific version of a proprietary model. If this model is updated, deprecated, or made unavailable, reproducing AMO-Bench results becomes impossible without re-grading all historical responses with the new grading model, which could yield different results. This is a reproducibility concern that parser-based grading avoids: math-verify is an open-source library with fixed behavior, but o4-mini is a black-box API subject to change. The paper acknowledges LLM-based grading "may be less efficient and does not consistently guarantee accuracy" (Section 2.3) and uses majority voting across 5 samples as mitigation, but this addresses sampling variance within a fixed grading model, not distribution shift if the model changes.
The 99.2% accuracy may mask systematic errors on specific problems. The 0.8% error rate corresponds to approximately 8 errors in 1,000 responses. If these errors are randomly distributed, their impact on aggregate metrics is negligible. But if they are concentrated on specific problems—for example, a descriptive-answer problem where o4-mini (Low) consistently misjudges a particular type of model error—that problem's scores would be systematically biased. The paper does not break down grading accuracy by problem, making it impossible to assess whether errors are random or concentrated.
Asymmetric grading errors (false positives vs. false negatives). The paper reports overall grading accuracy but does not distinguish between false positives (marking an incorrect answer as correct) and false negatives (marking a correct answer as incorrect). These have asymmetric effects on benchmark fairness: a benchmark that tends to give undeserved credit (false positives) inflates all models' scores and reduces discriminatory power; a benchmark that tends to penalize valid answers (false negatives) unfairly disadvantages models that produce unconventional but correct solutions. Without the error-type breakdown, it is unclear whether AMO-Bench's small grading error rate systematically biases results in one direction.

What evidence exists in the paper. The paper provides the 99.2% accuracy figure from 1,000 manually verified responses (Section 2.3) as the primary evidence for grading reliability. It does not report accuracy broken down by problem, by answer type (parser-based vs. LLM-based), by model, or by error type (false positive vs. false negative). The manual review and adjustment of post-processing algorithms during preliminary evaluation is mentioned but not quantified—we do not know how many adjustments were needed or how frequent parsing failures were before adjustment.

For the variable-expression verification method, the paper states that "multiple sets of values" are assigned without specifying the number or selection strategy, making the method's rigor difficult to assess. The decimal precision requirement (≥4 decimal places) is clearly specified and is a reasonable threshold for olympiad problems where exact answers are expected, but edge cases (e.g., answers that are exactly on the boundary of the precision threshold due to numerical instability) are not discussed.

Mitigation status. The paper partially mitigates grading reliability concerns through:

The hybrid grading approach, which confines LLM-based grading to only 11/50 problems, limiting the exposure to the least reliable grading component.
Majority voting across 5 grading samples for descriptive answers, reducing the error rate from individual grading model noise.
Manual verification of LLM-based grades during preliminary evaluation with adjustment of answer descriptions to improve grading accuracy.
The structured output prompt (\boxed{} format), which constrains model output to a parseable format and reduces format-related parsing failures.

However, these mitigations leave important gaps: the reproducibility concern from reliance on a proprietary grading model is unaddressed; the concentration of grading errors across problems is unmeasured; the false-positive vs. false-negative breakdown is unreported; and the sensitivity of the grading pipeline to new model behaviors (format drift, novel notation) is untested. The 99.2% figure provides reasonable confidence that grading errors do not dominate the results, but it does not guarantee that grading is uniformly reliable across all problems, models, and future evaluation scenarios.

For a practitioner using AMO-Bench for model development (where frequent re-evaluation is needed), the reliance on o4-mini for descriptive-answer grading introduces a dependency on a proprietary API with uncertain long-term availability and stability. Using only the AMO-Bench-P subset (39 parser-gradable problems, Table 1) avoids this dependency but sacrifices coverage of the hardest problem category (descriptive answers, which the paper's data suggests are indeed harder—scores on AMO-Bench-P are consistently higher than on full AMO-Bench). This is a practical tradeoff that the benchmark documentation should address but currently does not.

6. No Human Baseline to Contextualize Model Performance

The assumption or constraint. AMO-Bench is described as meeting or exceeding IMO difficulty standards, with the difficulty floor verified by human experts. The paper evaluates 26 LLMs on these problems (Figure 4) and concludes that models "still struggle" with scores below 52.4%. However, the paper never establishes what "good" performance on AMO-Bench would look like in human terms. There is no human baseline—no report of how IMO medalists, strong competition math students, or even the problem authors themselves perform on these problems under comparable conditions.

The paper references the IMO as a difficulty standard throughout: "meet or exceed the IMO difficulty standards" (Section 2.1), "at least the difficulty standards of IMO" (abstract), and "the scope typically covered in MO-level competitions such as IMO" (Section 2.1). These references imply that human IMO-level performance is the relevant benchmark, but that benchmark is never measured.

The consequence. Without a human baseline, the interpretation of model scores is ambiguous:

52.4% might be excellent or terrible depending on context. The IMO is a competition where the median participant scores substantially below the gold medal threshold. If typical IMO participants (strong high-school mathematicians) would score, say, 30–50% on AMO-Bench under time constraints, then GPT-5-Thinking (High) at 52.4% represents superhuman performance. If IMO gold medalists would score 80–90%, then 52.4% represents substantial room for improvement. The paper's narrative implies the latter interpretation ("LLMs still struggle"), but this framing is unsupported without a human reference point.
The difficulty claims are unvalidated from a human perspective. The paper asserts that problems "meet or exceed IMO difficulty" based on expert judgment, but expert judgment about difficulty is calibrated relative to human performance. A problem that "feels like an IMO problem" to an expert might be solved efficiently by expert humans while confounding LLMs for reasons unrelated to reasoning depth (e.g., reliance on geometric intuition that text-only models lack, or algebraic manipulations that humans find routine but models execute unreliably). Without human performance data, it is impossible to distinguish problems that are genuinely hard for all intelligent systems from problems that are hard specifically for current LLM architectures.
The test-time scaling analysis (Figure 7) lacks a human efficiency reference. The paper shows that models benefit from increased test-time compute (more output tokens). But how does this compare to human time investment? If an IMO participant would take 30–60 minutes to solve a problem (producing the equivalent of a few thousand tokens of written reasoning), then models consuming ~37K tokens of reasoning might be far less efficient than humans even if they achieve similar accuracy. Without human efficiency data, the test-time scaling results cannot be evaluated in terms of whether LLMs are approaching human-level reasoning efficiency or merely achieving accuracy through vastly greater computational expenditure.

What evidence exists in the paper. There is no human baseline in the paper. The problem authors—described as "mathematics experts from top universities and educational institutions" with "MO-level mathematics competition awards or experience in competition problem design" (Section 2.1)—would be the natural human reference group, but their performance on AMO-Bench is not reported. The paper does not discuss the absence of a human baseline or acknowledge it as a limitation.

The difficulty validation relies entirely on expert judgment ("verified by the human expert") and LLM performance ("at least two such models fail to correctly and consistently solve the problem"), neither of which provides a quantitative human performance reference. The paper uses "IMO difficulty" as an absolute standard without quantifying what IMO difficulty means in terms of human solve rates.

Mitigation status. This limitation is entirely unaddressed. The paper provides no human baseline, does not discuss the need for one, and does not suggest it as future work. The closest the paper comes to human context is the credentialing of problem authors, which establishes that the problems were created by experts but does not indicate how those experts would perform on the problems.

For a practitioner interpreting AMO-Bench results, the absence of a human baseline means that model scores should be understood as relative (how models compare to each other) rather than absolute (how close models are to human-level mathematical reasoning). The claim that models "still struggle" is defensible in the sense that accuracy is far from 100% and models show large capability-reliability gaps, but the claim implies a human comparison that the paper does not support with data. A more precise framing would be: "AMO-Bench reveals substantial performance variation among current LLMs on olympiad-level mathematics, with even the best models achieving ~52% single-shot accuracy, and the benchmark provides headroom for continued improvement." This framing does not require a human baseline and accurately reflects what the paper demonstrates.

7. Implications and Future Directions

How This Work Changes the Landscape

AMO-Bench introduces a methodological shift in benchmark design philosophy rather than a paradigm shift in model development or evaluation theory. The paper does not propose new models, training techniques, or evaluation metrics—it proposes a new way of constructing benchmarks that treats difficulty, originality, and gradability as independently verifiable design constraints rather than emergent properties of problem source selection. This is a refinement and formalization of existing benchmark construction practices, but one that addresses a concrete and increasingly urgent measurement crisis.

The magnitude of the shift is best understood in terms of what it enables the field to do that was previously difficult or impossible: track genuine reasoning progress in the regime beyond 90% accuracy on existing benchmarks. Prior to AMO-Bench, a researcher developing a new reasoning technique—say, a novel test-time search algorithm—faced a measurement problem: if their technique improved model accuracy on MATH500 from 98% to 99%, the 1-point gain could reflect genuine improvement, saturation effects, or noise. On AIME24/25, where top models cluster at 88–99%, the same measurement ambiguity applies: gains near the ceiling are compressed by the bounded nature of the accuracy metric. AMO-Bench restores headroom by pushing the accuracy of top models down to the 37–52% range (Figure 4), transforming what would be a 1–2 point gain on saturated benchmarks into a potentially 5–10 point gain on AMO-Bench—a much larger signal relative to noise, enabling clearer discrimination between genuinely better and marginally different methods.

This is not a theoretical contribution—the underlying measurement theory (accuracy as a bounded metric, the need for headroom) is well understood—but it is a practical intervention at a critical moment in the field's trajectory. When frontier models are released every few months and each claims reasoning improvements, the absence of benchmarks that can reliably measure those improvements creates a credibility problem: claims of progress become unfalsifiable if all available tests are near saturation. AMO-Bench provides a falsifiable test—a benchmark where current state-of-the-art is 52.4% and where the log-linear test-time scaling trends (Figure 7) suggest substantial headroom for future improvement before saturation recurs.

The paper also contributes a reconciliation of contradictory impulses in benchmark construction. Prior work faced an apparent tradeoff: you could have IMO-level difficulty (requiring expert-evaluated proofs) or automatic gradability (restricting to simpler final-answer problems), but not both. AMO-Bench resolves this by demonstrating that the tradeoff is an artifact of existing competition formats, not an inherent property of mathematical difficulty. The answer-type taxonomy (numerical, set, variable-expression, descriptive) and the corresponding hybrid grading system (parser-based for 39/50 problems, LLM-based with majority voting for 11/50) show that problems at IMO-level difficulty can be designed with gradable answer formats. This resolution is transferable methodology: future benchmark designers in mathematics or adjacent domains (physics, formal logic, competitive programming) can adopt the same taxonomy and grading pipeline rather than accepting the proof-vs-scalability dilemma as inevitable.

The diagnostic framework—simultaneously measuring AVG@32, pass@k, and output-length-vs-accuracy relationships—shifts evaluation from a single-dimensional "how good is this model?" question to a multi-dimensional "where and why does this model fail?" question. This is a reframing of what a benchmark is for: not just ranking models, but informing improvement strategies. The capability-reliability gap (29.6 percentage points for GPT-5-Thinking High, Figures 4 and 8) transforms the evaluation from a terminal assessment into a diagnostic signal—telling researchers that the bottleneck is reasoning selection, not fundamental capability, and therefore that investment in verification or search techniques is likely to yield larger returns than investment in pretraining. This reframing does not require new metrics (pass@k is standard), but it does require benchmarks designed to support multi-dimensional analysis—and AMO-Bench provisions this by including human-annotated solution paths, tracking output token consumption, and designing problems with answer formats that enable both single-shot and best-of-k evaluation.

The field's research direction priorities shift as a consequence:

More attractive: Test-time search and verification techniques for mathematical reasoning become more attractive because AMO-Bench quantifies the capability-reliability gap they could close. A researcher working on process reward models or self-consistency methods now has a clear target: close the ~30-point gap between AVG@32 and pass@32 on AMO-Bench for top models. This is a well-defined, measurable objective that saturated benchmarks cannot provide.
More attractive: Benchmark design methodology—specifically, the construction of contamination-resistant, difficulty-calibrated evaluation sets—becomes more attractive because AMO-Bench provides a replicable pipeline. Researchers can adopt the four-stage construction process (data creation, quality review, originality review, difficulty review) and the answer-type taxonomy for new domains.
Less attractive: Chasing marginal gains on saturated benchmarks becomes less attractive because AMO-Bench demonstrates that those gains are difficult to distinguish from noise. A paper reporting +0.5% on AIME24 will face the question: "What does this gain look like on AMO-Bench?"—and if the gain doesn't transfer to harder problems, it may reflect overfitting to the benchmark rather than genuine reasoning improvement.
Less attractive: Relying exclusively on single-metric evaluation (accuracy only) becomes less attractive because AMO-Bench demonstrates the diagnostic value of multi-dimensional analysis. The pass@k and output-efficiency dimensions reveal model properties that aggregate accuracy conceals.

Follow-Up Research This Work Enables

Human baseline measurement on AMO-Bench. The paper's most conspicuous gap is the absence of human performance data. A concrete follow-up would recruit 20–30 participants at varying skill levels (IMO medalists, national olympiad participants, strong undergraduate math majors) to solve AMO-Bench problems under time-matched conditions (comparable total reasoning time to what models consume, or unrestricted time to measure ceiling human performance). This would establish: (1) whether GPT-5-Thinking (High) at 52.4% AVG@32 is above, at, or below expert human single-shot performance, directly testing the paper's implicit "struggle" narrative; (2) the human pass@k ceiling, providing a target for model development; (3) the human output-efficiency benchmark (~tokens of written reasoning per problem), enabling comparison with the ~37K tokens that top models consume. Without this, all model scores float in a human-vacuum that limits their interpretability. The paper's problem authors—credentialed as MO-level competitors—are a natural starting point for this human baseline.

Verification method evaluation on AMO-Bench to close the capability-reliability gap. The paper identifies a 29.6-point gap between AVG@32 (52.4%) and pass@32 (82.0%) for GPT-5-Thinking (High) but provides no evidence on whether this gap is closeable. A direct follow-up would evaluate standard post-hoc selection strategies on AMO-Bench: majority voting (select the most common final answer among k samples), best-of-N with a process reward model (if a PRM can be trained or adapted for olympiad-level math), and self-consistency (varying temperature or prompt phrasing). The key measurement would be the accuracy achieved at each budget k and whether the cost-effectiveness (accuracy gained per additional sample) justifies the inference cost. A negative result—e.g., majority voting provides minimal gains because model errors are systematic rather than random—would be equally valuable, indicating that the pass@k ceiling is not reachable with current verification techniques and that fundamental reasoning reliability must be improved rather than papered over with sampling. The AMO-Bench-P subset (39 parser-gradable problems, Table 1) enables this evaluation without the cost and complexity of LLM-based grading.

Error taxonomy construction using human-annotated solutions. Each AMO-Bench problem includes a detailed human-written solution path (Section 2.1)—an asset the paper does not exploit for error analysis. A natural follow-up would classify model errors on AMO-Bench into categories derived from comparing model outputs to human solutions: calculation errors (arithmetic mistakes), logical gaps (missing cases, invalid inferences), misinterpretation (solving the wrong problem), incomplete reasoning (correct approach but failure to reach final answer), and format errors (correct reasoning but answer extraction failure). This taxonomy would reveal whether different models fail for different reasons (e.g., does DeepSeek-V3.1-Thinking make different error types than Qwen3-235B-Thinking?), whether error patterns correlate with problem category (are geometry problems failed for spatial reasoning reasons that text-only models share?), and whether test-time scaling reduces specific error types or only increases correct-solution probability without changing the error distribution. The human solutions provide a normative reasoning standard against which model deviations can be measured—a resource that most benchmarks lack and that AMO-Bench uniquely provides.

Cross-domain benchmarking to test generalizability of diagnostic patterns. The paper's findings—log-linear test-time scaling (Figure 7), large capability-reliability gaps (Figure 8), and output-length-vs-difficulty correlation (Figure 6)—are established exclusively for olympiad mathematics. A strong follow-up would replicate the same diagnostic framework on benchmarks in adjacent reasoning domains that share structural properties with AMO-Bench: competitive programming (e.g., Codeforces problems requiring algorithm design), formal theorem proving (e.g., miniF2F or LeanDojo problems requiring proof construction), and physics problem-solving (e.g., Physics Olympiad problems requiring multi-step derivation). For each domain, the study would measure AVG@k, pass@k, output efficiency, and test-time scaling trends, then compare whether the scaling slopes, gap magnitudes, and efficiency patterns are domain-specific or universal. If the patterns are universal, it suggests that reasoning reliability is a domain-general bottleneck that improved architectures or training objectives could address; if they are domain-specific, it suggests that mathematical reasoning training (RL on math, chain-of-thought distillation) produces capabilities that do not transfer, and that domain-specific benchmarks remain necessary.

Difficulty estimation without full evaluation cost to enable adaptive benchmarking. The paper's difficulty review stage (Section 2.1) filters problems using preliminary evaluation with multiple LLMs—a process that requires running candidate problems through models and checking correctness. A methodological follow-up would ask: can problem difficulty be predicted from problem text alone, without running models? Using AMO-Bench's 50 problems as training data (with difficulty operationalized as the mean AVG@32 across the 26 evaluated models, providing a fine-grained continuous difficulty score), a researcher could train a difficulty prediction model that takes only the problem statement as input and outputs an estimated difficulty score. If such a model generalizes—predicting difficulty for held-out olympiad problems from other sources—it would enable rapid, cost-effective benchmark construction: generate many candidate problems, predict their difficulty, and retain only those in the target difficulty range. This would address the scalability limitation of the current pipeline (human creation is slow, and LLM-based difficulty verification is expensive) and enable larger benchmarks (200–500 problems) with the same quality guarantees.

Stress-testing the grading pipeline against adversarial or out-of-distribution model outputs. The paper's grading accuracy validation (99.2% on 1,000 responses from 10 models) establishes reliability under current model behavior, but it does not test robustness to deliberate or emergent format shifts. A stress-test would systematically probe the grading pipeline's failure modes: (1) generate model responses with intentionally varied answer formatting (e.g., non-standard LaTeX, alternative mathematical notation, answers embedded in paragraph text rather than boxed) and measure how often the parser-based grading fails; (2) construct edge cases for the variable-expression verification (e.g., expressions that are mathematically equivalent but differ radically in form, expressions with restricted domains where test-value substitution could miss equivalence) and measure whether the "multiple sets of values" approach catches all equivalences; (3) evaluate whether the LLM-based grading (o4-mini Low with majority voting) produces consistent judgments when the same response is graded multiple times with different random seeds, or when semantically equivalent reference answers are phrased differently. The results would either validate the robustness of the current pipeline or identify specific failure modes that future benchmark iterations should address—for instance, by adding answer-format normalization steps, increasing the number of test values for expression verification, or switching to a more capable grading model for descriptive answers.

Practical Applications and Downstream Use Cases

Model selection for mathematical reasoning products. Organizations building products that require reliable mathematical reasoning—online tutoring systems, automated homework grading, STEM education platforms, quantitative research assistants—currently lack benchmarks that discriminate among frontier models. A platform choosing between GPT-5, DeepSeek-V3.1, Qwen3, and Claude for a math tutoring feature would see all four scoring 90–99% on MATH500 and 80–95% on AIME24/25 (Figure 1), making differentiation impossible. AMO-Bench, by spreading these same models across 36.8–52.4% (Figure 4), provides actionable discrimination: GPT-5-Thinking (High) leads at 52.4%, with Qwen3-235B-Thinking and DeepSeek-V3.1-Thinking close behind at ~47.6–47.8%, and Gemini-2.5-Pro and GLM-4.5 further back at ~37–39%. The output efficiency dimension (Figure 5) adds a cost consideration: GPT-5-Thinking (High) achieves its leading accuracy at a cost of ~37K tokens per problem, while some models in the same accuracy tier may consume fewer tokens. A platform optimizing for accuracy-per-dollar rather than accuracy alone would use both the AVG@32 leaderboard and the output-length-vs-accuracy scatter to select the model that provides the best accuracy at their inference budget. For cost-sensitive applications (e.g., serving millions of student queries), the efficiency dimension may dominate: a model at 47% accuracy with 20K tokens per problem may be preferable to one at 52% with 37K tokens, depending on the value of each additional correct answer.

Monitoring progress in reasoning research with a non-saturating metric. AI research labs developing reasoning-focused models (OpenAI's o-series, DeepSeek's R-series, Qwen's thinking models, Anthropic's extended-thinking variants) need evaluation benchmarks that can track improvement over multiple model generations without immediately saturating. AMO-Bench at its current difficulty level provides a clear improvement trajectory: the top score is 52.4% (GPT-5-Thinking High), and the log-linear test-time scaling trend (Figure 7) suggests that compute scaling alone can drive continued gains. As labs release new model versions, AMO-Bench scores provide a monotonic (or at least non-saturating) signal of progress—a model improving from 52% to 60% to 68% over successive releases tells a clear story, whereas improvement from 97% to 98% to 99% on MATH500 is compressed. The multi-dimensional diagnostic framework (AVG@32 + pass@32 + output efficiency) additionally reveals how progress is achieved: a new model that improves AVG@32 while maintaining or reducing token consumption represents a genuine efficiency gain; one that improves AVG@32 by consuming more tokens represents brute-force scaling; one that maintains AVG@32 but substantially increases pass@32 indicates improved latent capability that hasn't yet been harnessed for single-shot performance. This enables labs to characterize their improvements precisely and make informed decisions about where to invest further research effort.

Curriculum design for reinforcement learning on mathematical reasoning. Training reasoning models via reinforcement learning (as in DeepSeek-R1, o1, and similar approaches) requires a reward signal that distinguishes better from worse reasoning. The common approach uses outcome-based rewards (correct final answer = 1, else = 0) on training problems. However, if the training problems are too easy (models already solve them reliably), the reward signal provides minimal learning signal—the model rarely experiences informative negative feedback. If they are too hard (models never solve them), the reward is always zero and no learning occurs. AMO-Bench's difficulty calibration—verified to challenge current frontier models—places it in the ideal "zone of proximal development" for RL training: problems are solvable (pass@32 of 74–86% for top models, Figure 8) but not trivial (AVG@32 of 37–52%), meaning models will sometimes succeed and sometimes fail, providing a rich reward signal. While AMO-Bench itself (50 problems) is too small to serve as an RL training set, its construction methodology can be scaled: use the same pipeline (expert creation + IMO difficulty verification + LLM-based difficulty filtering) to produce a larger training set of 500–2000 problems at comparable difficulty. The AMO-Bench evaluation set then serves as a held-out test to measure whether RL training on this difficulty-appropriate curriculum improves reasoning more effectively than training on easier problems that provide less informative reward signals. The paper's finding that test-time scaling yields log-linear improvements (Figure 7) suggests that models have substantial learning capacity at this difficulty level that RL could exploit.