Soohak: A Mathematician-Curated Benchmark for Evaluating Research-level Math Capabilities of LLMs

🎯 Pitch

When asked to recognize ill-posed math problems, today’s best AI models fail to pause and instead confidently hallucinate answers—no model exceeds 50% on SOOHAK’s refusal subset. Meanwhile, even the strongest systems solve fewer than a third of the benchmark’s 340 new, contamination-free graduate-level problems, revealing a vast gap between competition mastery and actual research reasoning.

1. Executive Summary

This paper introduces SOOHAK, a 439-problem mathematics benchmark newly authored from scratch by 64 mathematicians to evaluate research-level reasoning in LLMs, consisting of a Challenge subset (graduate-level and research-adjacent problems) and a Refusal subset (probing whether models recognize ill-posed problems rather than producing confident but unjustified answers). Leading closed models reach only 30.4% Avg@3 on Challenge (Gemini-3-Pro), while the strongest open-weight model reaches 13.9% (Kimi-2.5), and on Refusal no model exceeds 50%, establishing that research-level mathematical reasoning and refusal remain substantially unsolved for current frontier systems while model performance scales roughly linearly with both train- and test-time compute on Challenge but not on Refusal.

2. Context and Motivation

The Core Problem: Research-Level Mathematics Benchmarks Are Scarce and Fragile

The fundamental problem this paper addresses is that the evaluation landscape for frontier mathematical reasoning in LLMs has hit a ceiling. As models have progressed from struggling with basic word problems to achieving gold-medal performance on the International Mathematical Olympiad (IMO), the community lacks benchmarks that can reliably discriminate among the most capable systems. This matters for several concrete reasons the authors establish (Section 1, Section 2):

Frontier model development needs real signal. When leading closed and open-weight systems all score similarly on existing benchmarks, developers cannot tell whether architectural changes, scaling decisions, or post-training recipes are actually improving mathematical reasoning or merely optimizing for saturated metrics. The paper notes that benchmarks such as MATH and GSM8K, which once saw <10% accuracy from language models, have become "less discriminative at the frontier" (Section 2).
High-stakes evaluation demands integrity under contamination. If benchmark items overlap with training data — an increasingly documented phenomenon in contest-derived sets (Balunović et al., 2025) — scores overestimate true generalization. This is not a hypothetical concern: the paper cites work showing that contamination can substantially inflate perceived capabilities, making it impossible to know whether a model that scores well on a benchmark has actually learned to reason mathematically or has simply memorized problem-solution pairs encountered during pretraining.
Research-level mathematics represents a qualitatively different capability from olympiad problem solving. The paper draws a sharp distinction here. Olympiad problems test step-by-step reasoning within bounded, self-contained domains where all necessary background knowledge is part of a standard contest curriculum. Research-level problems, by contrast, require synthesizing ideas across specialized subfields, navigating folklore knowledge (facts known to practitioners but not packaged as theorems), and engaging with material that may exist only in niche papers. This is the kind of reasoning that advances mathematical knowledge itself — and it is precisely what next-generation AI systems would need to master to genuinely assist in mathematical research.

The Benchmark Supply Failure

The paper identifies a structural failure in how the community has sourced evaluation material:

Most benchmarks scrape publicly available sources. MATH (Hendrycks et al., 2021) pulls from high school competition websites. GSM8K (Cobbe et al., 2021) uses grade-school math word problems. Omni-MATH (Gao et al., 2025) and MathArena (Balunović et al., 2025) aggregate contest problems from multiple sources. These datasets scale quickly — you can collect thousands of problems by crawling competition archives — but they inherit two fatal weaknesses:

Training data overlap is inevitable. Any problem freely available on the internet is likely to appear in web-scale training corpora. The paper cites Balunović et al. (2025) documenting contamination in widely used contest-derived sets. Once contaminated, a benchmark cannot distinguish memorization from reasoning.
Saturation happens fast. Contest problems are designed for humans under time constraints, not for LLMs that can integrate vast mathematical knowledge. As models improve, the fraction of unsolved problems shrinks rapidly, and the benchmark loses discriminatory power.

Research-level benchmarks exist but are tiny. The paper explicitly names Riemann-Bench (Garre et al., 2026) with 25 problems and FrontierMath Tier 4 (Glazer et al., 2024) with 50 problems as exemplars of the state of the art before SOOHAK. While these benchmarks probe genuinely hard mathematics — moonshot problems, research-adjacent material — their size makes them unstable for model comparison. With only 25 or 50 items, a model's score can swing by several percentage points from random sampling variation alone, and it is impossible to perform fine-grained subfield analysis. The evaluation community simply cannot track progress reliably with such small sample sizes.

Human authoring is expensive and difficult to scale. The obvious solution — pay mathematicians to write new problems — faces genuine practical barriers. Expert mathematicians are scarce and expensive. Problems at research level can take days to craft (Section 3.3 reports that "a single SOOHAK Challenge problem often required one or more days of work"). Quality control is nontrivial: verifying that an answer is correct, that the problem is well-posed, and that no hidden assumptions are needed requires reviewers of comparable expertise. And the whole enterprise must be completed under confidentiality to prevent leakage before evaluation — a nontrivial operational constraint.

The Tension Between Access Control and Transparency

The paper identifies a growing tension in the benchmark community around how to handle contamination risk. The dominant strategy has been access-controlled benchmarks: withhold problems and answers behind authentication, as in Humanity's Last Exam (Phan et al., 2025), FrontierMath, and First Proof (Abouzaid et al., 2026). This approach reduces leakage because problems never appear in public training data.

However, the paper argues this trades away transparency and reproducibility:

"To mitigate these issues, some efforts rely on newly written questions or carefully controlled release strategies... A small number withhold problems or answers behind access controls to reduce leakage, thereby improving longevity at the cost of transparency and reproducibility." (Section 2)

The tension is real. If a benchmark's items are secret, independent researchers cannot audit them for errors, cannot verify that difficulty estimates are accurate, cannot replicate the evaluation pipeline, and cannot assess whether the benchmark fairly represents mathematical knowledge across subfields. The paper's position — embodied in its design choice to publicly release the dataset in late 2026 after a temporary embargo — is that contamination resistance can be achieved through novelty rather than secrecy. If problems are genuinely new (authored from scratch by domain experts), they do not appear in training data regardless of whether they are publicly available. The embargo period provides time for the benchmark to serve its immediate evaluation purpose (the South Korean Sovereign AI competition, as described in Appendix B.1) before opening the dataset for broader research use.

Where Prior Approaches Fall Short, Specifically

The paper's critique of existing work is multi-dimensional and worth unpacking:

Single-area coverage limits diagnostic power. AMO-Bench (An et al., 2025) is explicitly cited as an example of a benchmark "confined to a single mathematical area." This matters because model capabilities are not uniform across subfields — a system might excel at algebraic manipulation while struggling with geometric reasoning or probabilistic thinking. A benchmark that covers only one area cannot reveal these asymmetries, and more critically, cannot tell model developers where to invest improvement effort. The paper's MSC subfield breakdown in Appendix D.5 is a direct response to this limitation: by annotating problems with Mathematics Subject Classification codes and reporting per-subfield leaderboards, SOOHAK aims to provide diagnostic signal about which kinds of mathematics remain challenging.

Small benchmarks cannot support difficulty-stratified analysis. Both Riemann-Bench (25 items) and FrontierMath Tier 4 (50 items) are too small to partition into difficulty levels. Yet difficulty stratification is essential because different models may saturate at different thresholds — a model that scores 50% on "hard" problems and 90% on "medium" problems is qualitatively different from one that scores 70% on both, even if their aggregate scores are identical. SOOHAK's 340-item Challenge subset, combined with the 702-item SOOHAK-Mini companion, enables this kind of analysis.

No evaluation of research-relevant metacognitive capabilities. The paper identifies a gap that no prior benchmark addresses: the ability to recognize when a problem is ill-posed. In research mathematics, a crucial skill is diagnosing when a question as stated is contradictory, underspecified, or otherwise lacks a well-defined answer — and responding appropriately rather than producing a confident but wrong solution. The paper frames this as a measurement of "benchmark hallucination and overconfidence" (Appendix B.8), a failure mode where models apply their reasoning machinery to produce persuasive incorrect answers to unanswerable questions. No existing benchmark systematically measures this capability, despite its centrality to actual mathematical practice. The Refusal subset (99 problems) is a direct response to this gap.

Open-weight vs. closed-model gaps are poorly characterized at frontier difficulty. The paper observes that while open-weight systems (Qwen3, GPT-OSS, Kimi) have become competitive on olympiad-level benchmarks, their relative standing on research-level problems was unknown prior to SOOHAK. Table 2 quantifies this gap: the strongest open-weight model trails the strongest closed model by ~16.5 percentage points on Challenge (13.87 vs. 30.39). This differential is substantively larger than the gap on SOOHAK-Mini (~6 points), suggesting that open-weight models disproportionately struggle with the kind of material that is rare or absent in publicly available training data — precisely the "folklore" and "unpublished" mathematics that SOOHAK Challenge targets.

How This Paper Positions Itself

SOOHAK positions itself as solving the intersection of four constraints that prior benchmarks could only address individually:

Scale sufficient for statistical reliability (439 problems across Challenge + Refusal, 1,141 total with SOOHAK-Mini) — an order of magnitude larger than FrontierMath Tier 4 or Riemann-Bench.
Novelty ensuring contamination resistance (problems authored from scratch by 64–105 mathematicians, with an originality agreement and no-AI-use certification) — solving the same problem as access-controlled benchmarks but without permanent secrecy.
Breadth across mathematical subfields (Algebra, Analysis, Geometry/Topology, Probability/Statistics, Applied Math/CS, Logic) — enabling diagnostic subfield analysis rather than a single headline score.
A novel refusal evaluation that probes metacognitive awareness alongside raw problem-solving ability — measuring a capability that no prior benchmark addresses.

The paper does not claim to be the first research-level math benchmark, nor the first to use expert authors, nor the first to be contamination-resistant. Its contribution is executing all of these simultaneously at operational scale, with transparency about the collection process (the pipeline in Figure 1, the compensation structure, the failure modes documented in Appendix F), so that the community can interpret results with appropriate nuance rather than treating benchmark scores as opaque signals.

The paper also explicitly connects to a broader trend: the need for evaluation infrastructure that can keep pace with model improvement. The introduction frames this as an escalating challenge — models progress, benchmarks saturate, new benchmarks must be built, often on compressed timelines and with large budgets. SOOHAK represents one point in this cycle, and the retrospective in Appendix F is offered as practical guidance for the next iteration.

3. Technical Approach

3.1 Reader Orientation

SOOHAK is a dataset construction and evaluation paper — it does not propose a new model, training method, or algorithm. Rather, it describes a pipeline for creating, filtering, and deploying a contamination-resistant mathematics benchmark at scale. The core idea is that by paying domain experts (mathematicians) to author problems from scratch under strict originality and confidentiality constraints, and by routing those problems through model-gated difficulty filters, one can build a benchmark that is simultaneously large enough for statistical reliability, broad enough across mathematical subfields to provide diagnostic signal, novel enough to resist training-data contamination, and difficult enough to leave substantial headroom above frontier models — all while maintaining the transparency that permanently access-controlled benchmarks sacrifice.

3.2 Big-Picture Architecture (Diagram in Words)

The SOOHAK system has four major components, organized as a pipeline that flows from problem creation through filtering to final evaluation:

Contributor Recruitment and Submission Pipeline — Mathematicians (faculty, PhD students, postdocs, IMO medalists) are recruited via direct outreach and public web submission. They author original problems under an NDA, IP-transfer agreement, and no-AI-use certification, submitting them through a web portal with problem statement, solution, and final answer in plain LaTeX.
Multi-Stage Automated Screening and Gating System — Each submitted problem is attempted by a panel of baseline LLMs of increasing capability. Problems that survive progressively harder model-gates are routed into different benchmark splits: SOOHAK-Mini (failed by small models), SOOHAK Challenge (failed by large open models), and a rejected pool from which Refusal items are drawn.
Manual Review and Quality Control Layer — Two human reviewers audit the model-generated solutions from the screening stage against contributor-provided references, flag discrepancies, request clarifications, correct errors, and ban contributors attempting to submit LLM-generated questions. This layer also handles split assignment confirmation, bilingual translation, and the construction of the Refusal subset from ill-posed rejected submissions.
Evaluation Engine — A standardized inference pipeline that runs the final benchmark items against both closed and open-weight models, parses final answers from model outputs, and uses GPT-5-Mini as an LLM judge to compare parsed answers against gold answers via mathematical equivalence checking, producing Avg@3 and Pass@3 metrics.

Information flows linearly: contributor → submission portal → automated LLM screening → human review + quality control → split assignment → bilingual translation → final dataset → model evaluation → scoring. The Refusal subset branches off after quality control, drawing from items that were rejected for being ill-posed rather than routed into a reasoning split.

3.3 Roadmap for the Deep Dive

First, the contributor ecosystem — who the authors are, how they were recruited, what terms governed their participation, and what they were paid — because the entire benchmark's quality depends on the expertise and incentives of the people creating the problems.
Second, the automated gating system — how problems are routed into Mini vs. Challenge splits using LLM-failure thresholds — because this is the mechanism that operationalizes "difficulty" and determines the benchmark's discrimination profile.
Third, the manual review and quality control procedures — how human reviewers catch errors, correct submissions, and construct the Refusal subset — because this is where benchmark integrity is enforced.
Fourth, the Refusal subset construction and grading logic — how ill-posed problems are sourced and scored — because this is a novel evaluation dimension that no prior benchmark addresses.
Fifth, the evaluation protocol — model configurations, sampling, answer parsing, and the LLM-judge grading pipeline — because this determines how scores are computed and what they mean.
Sixth, the FLOPs-matched comparison framework that enables the pretraining-vs-inference tradeoff analysis.

3.4 Detailed, Sentence-Based Technical Breakdown

This is fundamentally a benchmark construction and evaluation paper. Its core technical contribution is a pipeline that translates domain-expert effort into a contamination-resistant, difficulty-stratified, multi-dimensional evaluation instrument. There are no learned components — no models are trained, no algorithms are optimized. Instead, the "technical approach" is the set of operational procedures, filtering rules, and evaluation protocols that determine what the benchmark measures and how reliably it measures it.

3.4.1 Contributor Ecosystem: Recruitment, Credentials, and Compensation

The benchmark's quality rests on a simple premise: problems authored from scratch by domain experts cannot be in any model's training data. But executing this at scale — 1,141 accepted problems across all splits — requires solving a non-trivial organizational problem: how do you convince enough mathematicians to invest substantial time creating original problems under strict confidentiality, and how do you ensure those problems are genuinely new and human-authored?

Recruitment strategy. The paper uses two channels. The primary submission system recruited 86 contributors across 31 organizations through direct outreach — "emailing mathematics departments and contacting individual PhD students and faculty" (Section 3.1) — supplemented by 14 contributors who submitted via the project website without prior contact. A separate ScienceBench bulk purchase (Appendix B.4) added 19 additional contributors (10 professors, 7 postdoctoral researchers, 2 PhD students), bringing the total to 105. The primary-system pool skews toward the directly recruited: "Most accepted questions came from the direct-outreach pool" (Section 3.1).

Credentials. Across the full collection, the contributor pool breaks down as: 48% faculty, 23% graduate students and postdoctoral researchers (3% master's students, 20% PhD students or postdocs), 25% undergraduates, and 5% with undisclosed affiliation (Section 3.1). The Challenge subset specifically was "limited to selected faculty members, postdocs, PhD students, and a small number of IMO medalists in the primary system" (Section 3.2), with additional supplementation from ScienceBench. This credential filtering matters: the Challenge problems target graduate-level and research-adjacent material that requires active research experience or advanced training to craft credibly.

Compensation structure. The paper allocated a total compensation pool of USD 260,000 (Section 3.1). Contributors could opt for monetary compensation, authorship on the dataset paper, or both. Payment was issued on a per-accepted-question basis "until the quota for each split was filled." The per-question rates were split-dependent and ranged from USD 36 to USD 3,623 per question, with a cap of USD 20,000 per contributor. The large range reflects the difficulty gating: Challenge problems that passed the highest model-failure gate commanded higher rates than SOOHAK-Mini problems. The paper reports that the top five earners (predominantly faculty members whose contributions were mostly routed into Challenge) received USD 68,000 in total, while the top five contributors by volume (all students, mostly routed into SOOHAK-Mini) submitted 434 questions in total with no overlap with the top-earner group (Appendix B.2). This divergence between volume and earnings is intentional — the compensation design rewards difficulty, not throughput.

Legal and confidentiality constraints. Every primary-system contributor was required to sign a submission agreement (Section 3.1) affirming that:

Each problem was originally authored without AI assistance (no ChatGPT, no LLM-based drafting).
They transferred intellectual property rights to the project (work-made-for-hire or copyright assignment), meaning they "were not allowed to reuse, republish, distribute, or create derivative versions of accepted material" (Appendix B.2).
They would maintain confidentiality (NDA) and not paste candidate questions into public chat interfaces.

These legal instruments serve two purposes. First, they create a legal deterrent against submitting LLM-generated or plagiarized material — the contributor certifies originality under a binding agreement. Second, they prevent leakage: by transferring IP and imposing confidentiality, the project controls when and how problems enter the public domain. The temporary embargo (public release in late 2026) is possible because the project owns the problems and can delay their publication.

Why this approach over alternatives? The obvious alternative — scraping problems from publicly available sources — is fast and cheap but yields contaminated benchmarks (as documented in Balunović et al., 2025, cited in Section 2). The other alternative — access-controlled benchmarks with permanent secrecy — solves contamination but sacrifices transparency. The SOOHAK approach of novelty-through-expert-authorship plus temporary embargo attempts to solve both: the problems are genuinely new, so they are contamination-resistant by construction rather than by secrecy, and the embargo provides a window for high-stakes evaluation (the South Korean Sovereign AI competition, Appendix B.1) before the dataset becomes a public good for research. The cost — USD 260,000 in compensation plus operational overhead — is the price of this dual property.

3.4.2 The Automated Gating System: Model-Based Difficulty Routing

The central design decision in any benchmark is how to partition problems into difficulty levels. SOOHAK uses an unusual approach: rather than relying on contributor self-assessment or human difficulty labels, it routes problems through a cascade of LLM evaluations and assigns splits based on which models fail to solve them.

The three-gate architecture. Each primary-system submission is attempted by a panel of baseline LLMs organized into three progressively harder gates (Section 3.2, Appendix B.3):

Gate 1 (small open models): Qwen3-8B and OpenThinker3-7B. A problem must fail these models to pass Gate 1. This means the small models' generated solutions, when graded, do not match the contributor's gold answer.
Gate 2 (mid-size open models): GPT-OSS-20B and Qwen3-32B. A problem must fail these models to pass Gate 2.
Gate 3 (large open models, the Challenge gate): GPT-OSS-120B, Qwen3-235B, and DeepSeek-R1. A problem must fail all three to pass Gate 3.

Split assignment rules. The routing is deterministic based on gate passage (Appendix B.3):

$\text{Split} = \begin{cases} \text{SOOHAK-Mini (Gate 1–2)} & \text{if passes Gates 1 and 2 but fails Gate 3} \\ \text{SOOHAK Challenge} & \text{if passes all three gates} \\ \text{Rejected} & \text{if fails either Gate 1 or Gate 2} \end{cases}$

where "passes a gate" means "all models in that gate fail to solve the problem," and "fails a gate" means "at least one model in that gate solves the problem."

What this operationalizes. A problem that ends up in SOOHAK-Mini is one that small open models (8B parameters) cannot solve but that large open models (120B–235B parameters with reasoning) can — this roughly corresponds to high-school olympiad through early undergraduate material. A problem that ends up in SOOHAK Challenge is one that even the strongest open-weight reasoning models cannot solve — this targets graduate-level and research-adjacent material. A problem that fails Gate 1 or Gate 2 (i.e., Qwen3-8B or GPT-OSS-20B can solve it) is rejected as too easy for any split in this benchmark family.

Why model-gated routing rather than human difficulty labels? The paper frames this as an operational necessity (Section 3.2): "These model-based criteria are enforced as a hard policy." But the underlying motivation is that model capability is the relevant difficulty metric for a benchmark whose purpose is discriminating among models. A problem that GPT-OSS-20B can solve at 80% accuracy provides no signal for frontier systems — it is functionally "easy" regardless of what a human mathematician would call it. Conversely, a problem that appears "medium" to a human but is unsolvable by all current models is effectively "hard" for evaluation purposes. The model-gated routing ties difficulty directly to the evaluation objective: split assignment reflects what current systems can and cannot do, which is exactly what the benchmark aims to measure.

The paper does note a softer, non-binding mapping: "SOOHAK-Mini often corresponds to high-school through lower-undergraduate material, including olympiad and textbook-like problems. Challenge targets graduate level and research-adjacent material" (Appendix B.3). But this is post-hoc interpretation, not the routing mechanism.

The ScienceBench supplement. The Challenge split was "additionally supplemented with bulk-purchased problems from ScienceBench" (Section 3.2), yielding 112 problems from 19 contributors (Appendix B.4). These problems underwent a different gating process: rather than the three-gate LLM cascade, they were subjected to "lightweight peer-review" where contributors could view other submitted prompts without solutions and attempt solving them (Appendix B.4). The paper notes that "every submission was solved by at least one model," so verification focused on confirming that the model solutions and contributor solutions "relied on the same core reasoning ideas." This introduces a methodological asymmetry: Challenge contains both gate-3-routed primary-system problems and peer-reviewed ScienceBench problems, with different filtering procedures and contributor pools. The paper treats them as a unified Challenge split in all reported results.

3.4.3 Manual Review and Quality Control

The automated gating system produces split assignments, but it does not guarantee correctness. The paper implements a human review layer to catch errors that the LLM-based screening might miss or, critically, introduce.

The review workflow (Appendix B.3). Two members of the organizing team manually audit the automated screening outputs. Specifically, they "read the model-generated solutions produced during the difficulty-rating stage and compare them against the provided reference answer/solution" (Appendix B.3). There are three possible outcomes:

Agreement: The model-generated solutions are incorrect (as expected for an item that passed its gate) or the model-generated correct solution matches the contributor's solution reasoning. The item proceeds.
Discrepancy: A model produces "a coherent and seemingly correct solution that conflicts with (or casts doubt on) the submitted reference answer." In this case, the reviewers flag the item and request clarification from the author. The author may revise and resubmit, or opt in to inclusion of the clarified item.
No response: If the author does not respond, the submission is excluded.

Error correction scale. The paper reports that through this process, "we corrected 87 submissions and banned multiple contributors for attempting to submit LLM-generated questions" (Appendix B.3). The banned contributors — described in Figure 1 as ">1 dozen banned" — represent a non-trivial fraction of the submitter pool, suggesting that the no-AI-use agreement alone was insufficient as a deterrent and that the review process served a genuine filtering function.

Consistency checking. For each primary-system submission, the paper performs "an automated consistency check by comparing the contributor's proposed answer against answers produced by multiple LLMs generated as part of our split-assignment pipeline" (Appendix B.3). Exact agreement across models and contributor provides supportive evidence of correctness. When answers disagree, the item and generated responses are returned to the author "for a delayed re-solve" — meaning the author must independently re-derive their solution at a later time, reducing the risk that an initial error persists uncorrected.

External validation. The paper acknowledges the fundamental difficulty of verifying answers at frontier difficulty: "obtaining independent third-party solutions is often prohibitively difficult for many SOOHAK-Mini items and most Challenge items" (Appendix B.3). To provide an independent quality estimate, an external evaluation organization examined a randomly sampled portion of the collection after the main collection phase and "flagged approximately 5% of items for potential issues." The paper is careful to note that "flagged does not imply incorrect" — flags included genuine errors as well as ambiguity and evaluator misunderstanding. After addressing all flagged cases via correction or clarification, the paper estimates that "the fraction of items with substantive errors is at most 5%."

Why this form of quality control? The paper explicitly compares its approach to other frontier benchmarks: "As benchmark difficulty increases, exhaustive answer validation becomes increasingly costly and sometimes infeasible in practice, a challenge encountered in other frontier-level evaluations such as Humanity's Last Exam and FrontierMath" (Appendix B.3, citing Skarlinski et al., 2025; Burnham, 2025). The 5% upper-bound error rate is presented as a transparent quality metric rather than a claim of perfection — the reader is meant to "interpret results with appropriate caution" (Appendix B.3) rather than treating benchmark scores as noise-free measurements.

3.4.4 The Refusal Subset: Construction, Sourcing, and Grading

The Refusal subset represents a genuinely novel evaluation dimension: measuring whether models can recognize when a mathematical problem is ill-posed rather than producing a confident but incorrect solution.

Sourcing (Appendix B.8). Refusal questions are "drawn from submissions that we rejected during quality control because they exhibited logical flaws, missing assumptions, or other issues that render the question unsolvable or non-unique as stated." The paper maintains "a pool of such items" and selects from this pool to create the 99-item Refusal subset. Critically, these items "are not included as Challenge questions" — they are a separate split drawn from the same rejected-submission pool. This means Refusal items share the same authoring process and mathematical domain as the reasoning splits, but they contain embedded flaws that make them unanswerable as written.

What constitutes a Refusal-worthy flaw. The paper does not provide a taxonomy of flaw types, but the description in Appendix B.8 suggests three categories: (1) logical contradictions — the problem statement contains internally inconsistent constraints; (2) missing assumptions — the problem is underspecified, so multiple answers could be correct depending on which assumption is filled in; (3) no unique answer — the problem as stated admits multiple valid solutions or no solution at all.

Grading logic. This is the most operationally novel aspect. The paper defines correctness on Refusal items not by answer matching but by diagnostic behavior:

"The solver should explicitly indicate that the question has no well-defined answer as written or that it is underspecified, ideally with a brief explanation of the issue. We mark responses as incorrect if they present a specific mathematical answer as though the item were well-posed, or if they refuse without engaging the mathematical validity of the prompt." (Appendix B.8)

This creates a binary classification: a response is correct if it (a) identifies the flaw and (b) refuses to produce a numeric answer. It is incorrect if it (a) produces a confident answer (regardless of whether that answer would be correct under some interpretation) or (b) refuses without diagnosing the mathematical issue (e.g., a generic "I cannot answer this" without demonstrating understanding of why).

Why this matters. The Refusal subset measures a capability that is orthogonal to standard problem-solving benchmarks. A model could score perfectly on Challenge — solving every well-posed problem — while scoring zero on Refusal by confidently "solving" every ill-posed problem. Conversely, a model could be mediocre at problem-solving but excellent at recognizing when problems are broken. The paper argues (Appendix B.8) that this capability is "intrinsic to research mathematics" because real mathematical research involves formulating problems, not just solving them — and formulation requires recognizing when a tentative problem statement is not yet well-posed.

The grading logic has an important asymmetry: false positives (refusing on a well-posed problem) are not measured because Refusal items are known to be ill-posed. The benchmark only measures false negatives — the failure to refuse when refusal is appropriate. This means Refusal scores should be interpreted as a lower bound on a model's true refusal capability; a model might refuse correctly on some well-posed problems too, but the benchmark does not capture that.

3.4.5 Evaluation Protocol: Models, Sampling, Parsing, and Judging

The final component is the standardized evaluation pipeline that produces the scores reported in Table 2 and throughout Section 5.

Model panel. The paper evaluates eleven systems spanning closed and open-weight families (Section 4). The closed systems are Gemini-3-Pro, Gemini-3-Flash, GPT-5 Medium, GPT-5-Mini Medium, Claude-Opus-4.5, Claude-Sonnet-4.5, and Grok-4.1-Fast. The open-weight systems are Qwen3-235B-A22B-thinking-2507, GPT-OSS-120B, Kimi-2.5, and GLM-5. For GPT-5, the paper notes (Section 4, footnote): "We evaluate with GPT-5.1, GPT-5.2, and GPT-5 using identical configurations. GPT-5 yielded the best performance, and thus we report its results in the table." Reasoning was enabled for all models.

Decoding configuration (Appendix D.1, Table 4). The paper follows "the decoding configuration recommended by the provider" for each model. Two conventions hold:

Temperature: Gemini, GPT, and GPT-OSS families are sampled at temperature 1.0, "in line with vendor guidance for their reasoning modes." The remaining systems (Claude, Grok, Qwen3, Kimi, GLM) use temperature 0.6, "the recommended setting for their thinking variants."
Reasoning: Enabled wherever exposed: "thinking variant for the Qwen3 family, medium reasoning effort for GPT-5, GPT-5-Mini, and GPT-OSS-120B by default, and provider-default extended thinking for Claude."
Top-p and top-k: Left at provider defaults.
Context budget: Default per-model context windows; the test-time scaling ablations in Appendix D.3 vary context budget to 81,920 tokens for select models.

Sampling and metrics (Section 4). For each model–question pair, the pipeline samples three independent responses and computes two metrics:

$\text{avg@3} = \frac{1}{N} \sum_{i=1}^N \left( \frac{1}{3} \sum_{j=1}^3 c_{i,j} \right)$

$\text{pass@3} = \frac{1}{N} \sum_{i=1}^N \mathbb{I}\!\left[ \max_j c_{i,j} = 1 \right]$

where $N$ is the number of questions in the split, $c_{i,j} \in \{0, 1\}$ indicates correctness for question $i$ and sample $j \in \{1, 2, 3\}$ , and $\mathbb{I}[\cdot]$ is the indicator function (1 if the condition is true, 0 otherwise).

What avg@3 computes: The expected fraction of correct answers when drawing a single sample from the model, estimated by averaging the three-sample correctness rate across all questions. It is an estimator of the model's per-sample accuracy under the given decoding configuration.

What pass@3 computes: The fraction of questions for which at least one of the three samples is correct. It measures coverage — if the model can produce a correct answer with any of three attempts, the question is counted as solved.

Why both metrics: avg@3 measures reliability (how often is a single answer correct), while pass@3 measures capability breadth (how many problems can the model solve at all). A model with high pass@3 but low avg@3 is inconsistent — it occasionally finds correct solutions but cannot reliably reproduce them. A model with high avg@3 and high pass@3 is both broad and reliable. The paper reports both but generally uses Avg@3 as the primary metric in text and tables.

Answer parsing and judging. This is a non-trivial component because mathematical answers can be expressed in many equivalent forms. The pipeline works as follows (Section 4):

Parse a final answer from each model response. The paper does not specify the parsing mechanism in detail, but the phrase "we parse a final answer from each response" implies an extraction step that isolates the model's claimed answer (likely the content after a marker like "Final Answer:" or the last mathematical expression in the output).
Judge equivalence using GPT-5-Mini as an LLM judge. The judge receives only the gold answer and the parsed answer — no question text, no solution, no model reasoning trace. It outputs a binary correctness label: "correct" if the parsed answer is mathematically equivalent to the gold answer, "incorrect" otherwise.

Why an LLM judge? Mathematical answer formats can vary widely — a fraction 3/2, a decimal 1.5, an expression $\frac{6}{4}$ simplified, a radical form $\sqrt{9/4}$ — and simple string matching would produce false negatives. An LLM judge can recognize equivalence across these representations. Using an independent model (GPT-5-Mini) rather than the model being evaluated prevents self-grading bias.

Why no question text in the judge's context? The deliberate exclusion of question text and solution from the judge's input prevents the judge from "solving" the problem itself, which could introduce its own errors (the judge might incorrectly mark a correct answer as wrong because it solves the problem differently). The judging task is reduced to pure mathematical equivalence checking, which is a simpler and more reliable capability than problem-solving.

Test-time scaling ablations (Appendix D.3, Table 6). Beyond the default configurations, the paper explores the effect of increased reasoning effort and extended output budgets on two model families with such variants:

GPT-OSS-120B: Evaluated at three settings: medium reasoning (default), hard reasoning, and hard reasoning with 81,920-token context budget.
Qwen3-235B-A22B-thinking-2507: Evaluated at two settings: default context and 81,920-token context (the model "has no exposed reasoning-effort parameter," Appendix D.3).

The context budget extension from default to 81,920 tokens represents a 5× increase in per-question token allocation (from the standard 16,384 tokens). The paper frames this as measuring "test-time scaling" — whether allocating more inference compute per problem improves performance, analogous to the test-time compute scaling studied in the reference example paper but applied directly through provider-exposed reasoning-effort and context-budget controls rather than through external search or revision mechanisms.

Qwen3 family parameter scaling (Appendix D.2, Table 5). To measure how performance scales with model size, the paper evaluates the full Qwen3 family from 0.6B to 235B parameters, including both the standard release and the -2507 thinking-tuned checkpoints. This provides a "pretraining compute scaling" axis that complements the test-time scaling ablations.

3.4.6 Human Baseline Protocol

The paper includes a human baseline to provide "an interpretable reference point for difficulty" (Section 6). This is not a learned component but a procedural specification that determines what the benchmark scores mean relative to human capability.

Participant selection and team structure (Section 6, Table 3, Appendix E). Five teams (A–E) were assembled with deliberately varied expertise profiles:

Team A (CS Major, IMO experience): 5 participants, including 7 IMO Honorable Mentions, 2 IMO Bronze medalists, 1 EGMO Bronze, 1 APMO Bronze. These are students who had strong olympiad backgrounds in high school but pursued computer science rather than pure mathematics at university.
Team B (Math Major, IMO experience): 5 participants, including 1 IMO Silver, 1 APMO Bronze, 1 WMTC Gold, 2 KMS Gold/Silver. Current mathematics undergraduates with sustained contest training.
Team C (Math Major, IMO Gold): 5 participants, including 2 IMO Gold, 1 IMO Silver, 2 ICPC Seoul Gold, 2 KMS Gold. The strongest contest credentials in the pool, combined with competitive programming experience.
Team D (Math Major): 5 participants, including 1 KMS Silver, 2 KOI Bronze, 1 ICPC Asia Pacific Bronze, SIMC 2.0 Champion. More informatics and programming-oriented, with contest mathematics background.
Team E (Math Researchers): 5 PhD holders in mathematics and computer science, "with most having completed their undergraduate studies in mathematics and having published mathematical research" (Appendix E).

Evaluation set construction. The evaluation set comprises 79 prompts sampled across benchmark splits: 49 from the Calibration (SOOHAK-Mini) subset and 30 from Challenge. The paper notes an intentional upsample of harder questions: "Challenge items are drawn from narrower subfields and exhibit higher variance even among strong solvers, so a small number of hard items is often insufficient to reliably differentiate PhD-level performance" (Section 6.1).

Test conditions. Human evaluations were conducted under a nominal 4.5-hour time budget. Participants were "permitted to use any non-AI tools, including programming environments, computer algebra systems, and internet search for reference material." Use of LLMs and AI-assisted search features was explicitly prohibited. Session conditions were "not fully standardized across sessions" — the paper reports this for transparency, noting that "such operational differences can introduce additional variance in measured performance at frontier difficulty" (Section 6.1). Participants were compensated USD 340 each.

Scoring. Scoring is purely outcome-based: "A prompt is counted as correct only if the final answer is correct. We do not award partial credit for attempted-but-incorrect or partially correct solutions" (Section 6.1). This aligns the human scoring protocol with the model scoring protocol — both are evaluated on final-answer correctness, not on solution quality or partial progress.

LLM evaluation on the human set. For fair comparison, models are also evaluated on the same 79-prompt subset rather than the full dataset, at Pass@1 (single sample per problem). This is a departure from the main evaluation (which uses Avg@3 and Pass@3 with three samples) and means the human-vs-model comparison in Figure 3 uses a different sampling protocol than the main results in Table 2.

Aggregation logic. The paper reports "Combined" human coverage as "the union of all questions solved by any human team" — a question is counted as solved if at least one team solved it. This is an upper bound on what the pool of 25 human experts can collectively achieve, analogous to Pass@k with k equal to the number of independent teams. It substantially overstates what any single human or team can achieve, but it provides a measure of the benchmark's tractability ceiling — of the 79 questions, humans can collectively solve at least 50.6%, establishing that the problems are not impossible, just hard.

Why this team structure? The varied expertise profiles are designed to reveal "which types of mathematical expertise the benchmark rewards and to what degree" (Section 6). If IMO medalists dramatically outperform PhD researchers, the benchmark primarily measures contest-style reasoning under time pressure rather than deep mathematical knowledge. If PhD researchers dramatically outperform undergraduates, the benchmark primarily measures research-level expertise. The actual results (Section 6.2) show the former — contest-trained undergraduates outperform PhD researchers — which the paper interprets as a "task-format mismatch, not an ability gap" (Section 6.2). This diagnostic function of the human baseline is arguably more important than the absolute human scores: it tells users what the benchmark is actually measuring, as opposed to what it intends to measure.

3.4.7 Composite Scoring for Multi-Dimensional Evaluation

The paper introduces three composite scores in Appendix D.4 that combine the per-split Pass@3 metrics into aggregate measures. These are not used in the main results but are offered as descriptive tools for interpreting the multi-dimensional nature of the benchmark.

Definitions (Appendix D.4):

$\text{Capability} = \frac{1}{2}\left(\text{SOOHAK-Mini} + \text{Challenge}\right)$

$\text{Avg-R} = \frac{1}{3}\left(\text{SOOHAK-Mini} + \text{Challenge} + \text{Refusal}\right)$

$\text{SOOHAK-R} = \frac{1}{2}\left(\text{Challenge} + \text{Refusal}\right)$

where each term on the right-hand side is the Pass@3 score for that split.

What Capability computes: The unweighted mean of the two reasoning splits, containing no signal from the Refusal dimension. A model that scores highly on both reasoning splits but poorly on Refusal receives a high Capability score.

What Avg-R computes: The three-way mean that treats Refusal as an equally weighted third dimension. The paper provides an algebraic decomposition:

$\text{Avg-R} = \text{Capability} - \frac{1}{3}\left(\text{Capability} - \text{Refusal}\right)$

This shows that Avg-R subtracts one-third of the gap between reasoning capability and refusal performance from the raw Capability score. A model with Capability = 70 and Refusal = 40 receives Avg-R = 70 − (1/3)(30) = 60 — a 10-point penalty for being "confidently wrong" on ill-posed problems. A model with Capability = 70 and Refusal = 70 receives Avg-R = 70 — no penalty.

What SOOHAK-R computes: A frontier-focused composite that pairs the hardest reasoning split (Challenge) with Refusal, ignoring SOOHAK-Mini. This is designed to surface models that are both capable on advanced mathematics and careful about recognizing ill-posed problems — a profile the paper argues is relevant for research-assistance applications.

Why these composites? The paper presents them as purely descriptive — "the metric is purely descriptive" (Appendix D.4) — but their construction reveals a normative stance: being correct on well-posed problems while being confidently wrong on ill-posed problems is worse than being correct on well-posed problems while appropriately refusing on ill-posed problems. The Avg-R metric operationalizes this by penalizing the Capability–Refusal gap. Figure 4 (Appendix D.4) demonstrates the ranking shifts: GLM-5 rises 3 ranks from Capability to Avg-R (because its Refusal score is high), Kimi-2.5 drops 3 ranks (because its Refusal score is low relative to its capability), and GPT-5 takes the top Avg-R rank from Gemini-3-Pro despite Gemini's higher Capability — because GPT-5's Refusal score (43.09) is higher than Gemini-3-Pro's (41.41), and this gap outweighs Gemini's advantage on the reasoning splits.

3.4.8 Bilingual Translation Pipeline

The paper constructs a parallel bilingual benchmark (English and Korean) through a machine-translation-plus-post-editing workflow (Appendix B.7). This is motivated by the project's funding context — the South Korean Sovereign AI competition (Appendix B.1) requires evaluation in Korean — and by the desire to provide a resource for multilingual mathematical reasoning research.

Workflow steps (Appendix B.7):

LaTeX preservation: All LaTeX spans (expressions, symbols, commands) are replaced with protected placeholders before translation. This ensures mathematical content is preserved verbatim — the translation system operates on natural language text only, not on mathematical notation.
Machine translation: A "domain-adapted machine translation system" generates a draft translation from the placeholder-protected text.
Placeholder restoration: The LaTeX placeholders are restored into the translated text, so all mathematical content appears identically in both languages.
Professional post-editing: Human translators edit the draft under a guideline emphasizing "semantic faithfulness, terminology consistency, standardized mathematical notation, and minimal paraphrasing." Changes to mathematical symbols or LaTeX are disallowed except when required by target-language grammar.
Independent QA: An independent reviewer performs quality assurance, checking for mistranslations, omissions, terminology inconsistencies, punctuation issues, and text–formula mismatches.
Automated checks: Scripts verify LaTeX renderability, formula equivalence, untranslated segments, and cross-item term inconsistencies. Mathematical terminology is normalized using a curated glossary based on the Korean Mathematical Society dictionary.
Serialization: Final bilingual items are serialized in a structured JSON format.
Security: All translation and QA steps are executed under strict security constraints, including "prohibitions on uploading full problems to external LLM interfaces or sharing files outside the secured workflow" — this prevents leakage through the translation pipeline itself.

Language distribution. Approximately 92% of items were originally authored in English (Appendix B.7). Every item is translated into the other language (Korean → English and English → Korean), producing a fully parallel dataset where each problem exists in both languages with verified mathematical equivalence.

Why this investment? The bilingual construction serves both the immediate evaluation need (Korean-language model evaluation for the Sovereign AI competition) and a longer-term research purpose: it enables studying whether mathematical reasoning capabilities transfer across languages, whether models perform differently on the same mathematics expressed in different natural languages, and whether translation quality affects evaluation validity. The paper does not report per-language scores in the main results, but the infrastructure exists in the dataset for future work to explore these questions.

3.4.9 Problem-Type Classification and Subject Annotation

Beyond split assignment (Mini vs. Challenge vs. Refusal), each question receives two additional annotations that enable fine-grained analysis.

Coarse problem-type labels (Appendix B.6). A GPT-5 classifier assigns each question one of four labels:

olympiad: Contest-style problems with short trick-based solutions.
undergrad: Standard university topics such as calculus, linear algebra, introductory algebra and analysis.
graduate: Graduate core and research-level techniques.
beyond: Specialized research problems requiring background beyond graduate core.

The paper reports the distribution across splits: SOOHAK-Mini contains 401 olympiad, 218 undergrad, 63 graduate, and 18 beyond items. Challenge contains 77 olympiad and 69 undergrad items — meaning that even the Challenge split contains some problems that are classified as olympiad or undergrad in style, despite having passed the Gate 3 model-failure requirement. The paper explains this as: "the classifier tracks style and background, not the collection gate" (Appendix B.6). A problem can have olympiad-style structure (short, trick-based) while still being difficult enough that large reasoning models fail it — difficulty and style are partially but not perfectly correlated.

LLM-assigned Mathematics Subject Classification (Section 3.5, Table 1). Each question is assigned to an MSC subject area using "a GPT-5-mini classifier that takes the question plus contributor keywords and maps the item to a fixed taxonomy." The resulting distribution (Table 1) shows concentration in Algebra & Discrete (680 items, driven by number theory at 269 and combinatorics at 131), followed by Analysis (233, driven by real analysis at 115), Geometry & Topology (175, driven by geometry at 95), with smaller portions in Applied/CS/OR (27), Probability & Statistics (25), and Logic (1).

Contributor-provided keywords (Section 3.5). In addition to the LLM-assigned subjects, problem contributors supplied keyword tags at submission time. The paper notes a qualitative difference between splits: "SOOHAK-Mini centered on computational and contest-like pattern finding through tags such as number theory, modular arithmetic, factorization, geometry, and combinatorics. The Challenge split develops a specialized long tail, including tags such as automorphism, abelian variety, Fano variety, Kazhdan–Lusztig polynomials, moduli space, Richardson varieties, Barratt–Eccles operad, and homotopical algebra" (Section 3.5). This keyword divergence confirms that the model-gated routing does in fact partition the dataset along a meaningful mathematical sophistication axis — Challenge items cluster around specialized, research-level concepts that are rare or absent in contest-style mathematics.

Why dual annotation (contributor keywords + LLM subjects)? The contributor keywords capture the author's intended mathematical framing — what subfield they believe the problem belongs to. The LLM-assigned subjects provide a standardized taxonomy that enables cross-benchmark comparisons and systematic subfield coverage analysis (as in the MSC breakdown in Appendix D.5). The combination allows checking for discrepancies between author intent and systematic classification.

3.4.10 Summary of Design Choices and Their Justifications

Expert authorship with legal agreements over public scraping: avoids training-data contamination and enables temporary embargo for high-stakes evaluation before public release. The cost (USD 260,000) is the price of this property.
Model-gated difficulty routing over human difficulty labels: ties split assignment directly to what current models can and cannot do, which is the relevant metric for a benchmark whose purpose is model discrimination.
Model-failure-based gating over model-success-based gating: problems that models fail are retained; problems they succeed on are rejected or placed in lower splits. This ensures the benchmark measures unsolved capability rather than saturated performance.
Human review of model-generated solutions over fully automated quality control: catches errors that LLM screening might miss (87 corrections) and detects AI-generated submissions (>1 dozen contributors banned).
The 5% upper-bound error rate over claiming perfection: transparently acknowledges the fundamental difficulty of answer validation at frontier difficulty, aligning with practices in Humanity's Last Exam and FrontierMath.
Refusal subset construction from rejected submissions over purpose-written refusal problems: ensures Refusal items share the same mathematical domain and authoring process as Challenge items, making the refusal-vs-solving comparison meaningful rather than confounded by domain shift.
LLM judge (GPT-5-Mini) for answer equivalence over string matching: handles mathematical answer format variation. Excluding question text from the judge's context prevents the judge from introducing its own solving errors.
Avg@3 and Pass@3 over single-sample accuracy: captures both reliability (how often is a single answer correct) and coverage (how many problems are solvable at all), providing a richer picture of model capability than either metric alone.
Temperature 1.0 for reasoning-mode models over lower temperatures: follows provider guidance for their reasoning modes, where higher temperature is presumably beneficial for the chain-of-thought or extended-thinking processes that these models employ.
Bilingual parallel construction over monolingual dataset: serves both the immediate Korean-language evaluation need (Sovereign AI competition) and enables future research on cross-lingual mathematical reasoning transfer.
Composite scores (Capability, Avg-R, SOOHAK-R) over single headline metric: reflects the benchmark's multi-dimensional design (reasoning + refusal) and penalizes models that are confidently wrong on ill-posed problems — operationalizing the normative stance that refusal capability matters alongside problem-solving capability.
Temporary embargo with planned public release over permanent access control: balances the immediate evaluation need (contamination-free scores for the competition) with the long-term research need (transparency, reproducibility, independent auditing). The embargo is "bounded by the conclusion of the funded evaluation initiative" (Appendix B.1), making it time-limited rather than indefinite.

4. Key Insights and Innovations

Innovation 1: Refusal as a First-Class Evaluation Dimension, Not an Afterthought

The paper's most conceptually distinctive move is elevating refusal — the ability to recognize and appropriately respond to ill-posed problems — from a nuisance variable that benchmark builders try to eliminate to a deliberately constructed evaluation axis with its own 99-item subset, its own scoring rubric, and its own leaderboard. This is not a minor addition; it fundamentally changes what the benchmark claims to measure.

What the field did before. Prior math benchmarks treated question validity as a quality-control problem to be solved during dataset construction. If a problem was found to be ill-posed — contradictory constraints, missing assumptions, no unique answer — it was fixed or discarded. The goal was a clean dataset where every item had exactly one correct answer. This made evaluation straightforward (string matching or equivalence checking) but created a blind spot: the benchmark measured only what models do when problems are well-posed, never what they do when problems are broken. The possibility that a model might confidently "solve" an unanswerable question was treated as a measurement error to be prevented, not as a capability to be evaluated.

SOOHAK inverts this. The Refusal subset is explicitly sourced from submissions that were rejected during quality control because they were ill-posed (Appendix B.8). Rather than discarding these items as defective, the paper repurposes them to probe a capability that the authors argue is "intrinsic to research mathematics" (Section 1): recognizing when a problem statement does not yet admit a well-defined answer. The grading logic is novel: correctness is defined by diagnostic behavior (identifying the flaw and refusing to produce a numeric answer), not by answer matching. A model that produces a confident but incorrect solution to an unanswerable question receives zero credit — even if its reasoning is internally coherent and its answer would be correct under some plausible interpretation.

Why this matters beyond raw performance. The Refusal subset recasts benchmark evaluation as a measurement of two orthogonal capabilities: problem-solving competence (can the model find the right answer when one exists?) and epistemic caution (can the model recognize when no right answer exists as stated?). A model that excels at the first while failing the second is, in the paper's framing, systematically overconfident — it applies its reasoning machinery uniformly regardless of whether the input is well-posed. This is a failure mode with real consequences: in research mathematics, confidently asserting a "solution" to an ill-posed problem wastes effort and potentially propagates errors. In higher-stakes domains, the same failure mode could be catastrophic.

The paper provides evidence that these capabilities are genuinely decoupled. Table 2 shows that GPT-5 — the strongest model on SOOHAK-Mini (72.22% Avg@3) and second-strongest on Challenge (26.37%) — scores only 43.09% on Refusal. GLM-5, a weaker reasoning model (9.61% on Challenge), scores the highest Refusal at 49.49%. The Qwen3 family is a "clear outlier in the other direction," scoring worst on Refusal despite non-trivial Challenge performance. These divergences are not noise; they are systematic evidence that problem-solving ability and epistemic caution are distinct model properties that do not covary strongly.

The paper's composite scores (Appendix D.4, Figure 4) operationalize the normative stance that these capabilities should be evaluated jointly. The Avg-R metric penalizes models whose Refusal performance lags behind their reasoning performance, causing ranking shifts: GLM-5 rises 3 ranks from a purely reasoning-based ranking, while Kimi-2.5 drops 3 ranks. GPT-5 overtakes Gemini-3-Pro in the Avg-R ranking despite lower reasoning scores, because GPT-5's Refusal advantage outweighs Gemini's reasoning advantage. This is not merely descriptive — it encodes a value judgment about what constitutes a "better" mathematical reasoner, and it provides a template for how future benchmarks might integrate metacognitive evaluation without abandoning standardized scoring.

What kind of contribution is this? This is a new diagnostic concept — not a metric gain or an architectural improvement, but a reframing of what mathematical reasoning evaluation should measure. It is incremental in execution (the Refusal subset is constructed from an existing rejected-submission pool; the grading logic is binary) but fundamental in implication: it establishes that benchmark builders should treat "knowing when you don't know" as a measurable capability rather than an error to be suppressed. The paper does not claim to have solved refusal evaluation — the 99-item subset is small, the grading logic is coarse, and the sourcing from rejected submissions introduces domain-specific biases — but it has made the case that refusal belongs on the evaluation agenda, and it has provided a concrete, reproducible method for incorporating it.

Innovation 2: Model-Gated Difficulty Routing as an Operationalized Construct

SOOHAK's method for assigning problems to difficulty tiers — routing each submission through a cascade of LLM evaluations and assigning splits based on which models fail to solve it — represents a philosophically different approach to difficulty labeling than what prior benchmarks employed. This is not just a different filtering mechanism; it embodies a different answer to the question "what does 'difficulty' mean for a benchmark?"

What the field did before. Prior benchmarks used one of three difficulty-signaling strategies. The most common was human-assigned labels: MATH (Hendrycks et al., 2021) used competition level (Algebra, Counting and Probability, etc. from AMC 10/12), while Omni-MATH (Gao et al., 2025) organized problems by olympiad stage. The second was contributor self-assessment: authors of newly written problems estimated their item's difficulty (as in FrontierMath and Riemann-Bench). The third was no explicit difficulty stratification at all — the benchmark was simply "hard" in aggregate, and per-item difficulty was left implicit.

All three strategies share an assumption: difficulty is a property of the problem itself (its mathematical content, required background, solution complexity), and human experts can reliably assess it. SOOHAK challenges this assumption by operationalizing difficulty as what current models can and cannot do. A problem is "Challenge" not because a mathematician judges it to be graduate-level, but because GPT-OSS-120B, Qwen3-235B, and DeepSeek-R1 — the strongest open-weight reasoning models available during collection — all fail to solve it. A problem is "Mini" not because it matches contest curricula, but because small models fail it while large models succeed. A problem is rejected entirely if even small models can solve it, regardless of how mathematically interesting it might be.

Why this matters. This operationalization has three consequences that human-assigned difficulty labels do not provide:

Difficulty is relative to model capability, not absolute. A problem that is "graduate-level" for humans might be trivially solved by frontier LLMs if it happens to match patterns well-represented in training data. Conversely, a problem that is "contest-level" might be extremely difficult for models if it requires a trick that is rare in training corpora. The model-gated routing automatically captures this relativity — problems that are easy for models end up in lower splits or are rejected, regardless of their human-assigned difficulty.
Difficulty is dynamic, not static. The paper acknowledges this implicitly by naming the specific models used for gating and the collection timeframe. If the benchmark were reconstructed in 2027 with stronger baseline models in the gates, the same problems might route differently — some current Challenge items might become Mini items, and some current Mini items might be rejected. This is a feature, not a bug: it means the gating mechanism can be recalibrated as model capabilities evolve, producing new benchmark versions that maintain discrimination at the frontier. The paper does not explore this dynamic potential, but the architecture supports it.
Difficulty is tied to the evaluation objective. A benchmark's purpose is to discriminate among models. A problem that all evaluated models solve at >95% provides no discriminatory power, regardless of its mathematical sophistication. The model-gated routing ensures that every item in SOOHAK Challenge was unsolved by at least the open-weight reasoning models in the gating panel during collection — meaning it retains discriminatory headroom for the frontier systems being evaluated.

Evidence that the gating works as intended. The paper provides several signals. Table 2 shows that SOOHAK-Mini scores range from 40.88% (Claude-Sonnet-4.5) to 72.22% (GPT-5), while Challenge scores range from 5.69% (Claude-Sonnet-4.5) to 30.39% (Gemini-3-Pro). The score drop between splits is large and consistent across all models — roughly 40–50 percentage points for the strongest models, confirming that the gating produced qualitatively distinct difficulty tiers. The Qwen3 size-scaling results (Table 5) show that Challenge Pass@3 rises from 2.94% at 0.6B to 15.29% at 32B, a roughly linear scaling trajectory that suggests the split is neither trivially easy (no headroom) nor impossibly hard (no signal). The Refusal split, by contrast, shows no such scaling (2.69% at 235B for Qwen3, lower than 28.28% at 32B), confirming that the gating produced a dimension that does not simply track parameter count.

The paper also reports a qualitative divergence that validates the routing: contributor keywords on SOOHAK-Mini center on "computational and contest-like pattern finding" while Challenge keywords develop "a specialized long tail" including terms like automorphism, abelian variety, Fano variety, and moduli space (Section 3.5). The model-gated routing didn't just produce harder problems — it produced problems from different mathematical domains, with different reasoning demands.

Limitations of this approach. The model-gated routing has two important caveats. First, gate model choice is arbitrary and time-bound. The specific models used (Qwen3-8B, GPT-OSS-20B, DeepSeek-R1, etc.) were the strongest available during the collection window. Different choices would produce different splits. The paper does not provide a principled framework for selecting gate models — this was an operational decision driven by available resources and the collection timeline. Second, the gating is a one-time filter, not a continuous calibration. Once a problem is assigned to Challenge, it stays there even if future models can solve it trivially. The paper's embargo period provides temporary protection against this, but after public release, the gating calibration will drift as models improve. The paper does not propose a mechanism for recalibration or versioning.

What kind of contribution is this? This is a methodological innovation in benchmark construction — a different way of thinking about what difficulty means and how to operationalize it. It is incremental in implementation (the gating is a simple pass/fail cascade) but fundamental in conceptual framing: it shifts the difficulty construct from a property of problems to a property of the problem–model interaction. This reframing has implications beyond SOOHAK — any benchmark that aims to discriminate at the frontier must contend with the fact that "difficulty" is not a stable human judgment but a moving target defined by model capability.

Innovation 3: Expert Authorship as Contamination Resistance Without Permanent Secrecy

SOOHAK's approach to contamination — paying 64–105 mathematicians to author problems from scratch under legal agreements, rather than scraping public sources or permanently withholding items — represents a third path in the contamination-resistance design space that prior benchmarks had not systematically explored at scale.

What the field did before. Prior benchmarks occupied two extremes. At one end: public-sourced benchmarks (MATH, GSM8K, Omni-MATH, MathArena) that are cheap to construct but vulnerable to training-data overlap because their problems were drawn from publicly accessible websites. As Balunović et al. (2025) documented, contamination in such sets can substantially inflate model scores, making it impossible to distinguish memorization from reasoning. At the other end: permanently access-controlled benchmarks (Humanity's Last Exam, FrontierMath, First Proof) that withhold problems and answers behind authentication, preventing contamination by never exposing items to training corpora. This approach solves the contamination problem but sacrifices transparency — independent researchers cannot audit the items for errors, cannot replicate the evaluation pipeline, and cannot verify that the benchmark measures what it claims to measure.

SOOHAK proposes a middle path: contamination resistance through novelty. If problems are genuinely new — authored from scratch by domain experts under an originality agreement and a no-AI-use certification — then they cannot appear in any model's training data, regardless of whether the dataset is eventually made public. The contamination resistance comes from the authoring process (creating content that did not previously exist), not from access control (keeping existing content secret). The public release planned for late 2026 can proceed without compromising contamination resistance because the problems were never in training data to begin with — they were created after all current models were trained.

Why this matters at scale. Expert-authored benchmarks are not new — Riemann-Bench and AMO-Bench both used expert authors. What is new is executing this approach at operational scale (1,141 accepted problems across all splits, 105 contributors, USD 260,000 in compensation, a four-month timeline) while maintaining quality control and confidentiality. The paper demonstrates that the model is feasible: you can pay mathematicians enough to produce genuinely challenging, original problems under legal frameworks that protect the benchmark's integrity, and you can build a review pipeline that catches errors and detects AI-generated submissions.

The paper is transparent about what this costs — not just financially (USD 36 to USD 3,623 per question, USD 20,000 per-contributor cap, USD 260,000 total pool, plus operational overhead for the ~USD 550,000 total budget reported in Appendix F) but also organizationally. The constraints were severe: four months end-to-end, NDA and IP-transfer agreements with every contributor, a manual review process that corrected 87 items and banned ">1 dozen" contributors for LLM-generated submissions, and a bilingual translation pipeline conducted under security constraints to prevent leakage through the translation process itself. The retrospective in Appendix F documents the failure modes observed during construction — incentive misalignment (rewarding difficulty led some contributors to optimize for that rather than quality), reviewer-hour constraints (not enough expert review capacity early in the project), and the structural limitation of unique-integer answers (which restricts the space of feasible problems and biases toward certain subfields).

Why this approach over permanent secrecy. The paper implicitly argues that transparency matters for benchmark credibility in ways that permanent access control cannot provide. When a benchmark's items are public, the community can: (1) audit for errors (the paper reports a ~5% upper-bound error rate from external validation — this figure is credible precisely because the items will be public); (2) verify that difficulty labels are accurate; (3) analyze subfield coverage and identify biases; (4) replicate the evaluation pipeline independently; (5) study why models fail on specific items, not just that they fail. Permanently access-controlled benchmarks can provide none of these properties — their scores are opaque signals whose interpretation depends entirely on trust in the benchmark builders.

The temporary embargo is the key design element that makes this possible. By delaying public release until late 2026 (after the South Korean Sovereign AI competition concludes, per Appendix B.1), the paper satisfies the immediate need for a contamination-free evaluation while preserving the long-term research value of a public dataset. The embargo is "bounded by the conclusion of the funded evaluation initiative" (Appendix B.1), making it a time-limited constraint rather than an indefinite restriction.

Evidence that the approach works. The paper does not directly measure contamination — by construction, truly novel problems cannot be contaminated. But it provides several indirect signals. The gap between closed and open-weight models on Challenge (16.5 percentage points: 30.39% for Gemini-3-Pro vs. 13.87% for Kimi-2.5) is substantially larger than on SOOHAK-Mini (roughly 6 points: 72.22% for GPT-5 vs. 66.07% for Kimi-2.5). The paper interprets this as evidence that "open-weight systems transfer less reliably to unpublished and research-adjacent mathematics" — precisely the material that would be absent from public training data. This interpretation is consistent with the paper's contributor interviews (Appendix B.5), where authors reported that "when relevant mathematics is absent from accessible papers, scattered across niche sources, or hidden behind paywalls, contributors found it substantially easier to write questions that remain challenging for frontier systems." In other words: the benchmark is measuring something that is genuinely scarce in public training data, and the open-vs-closed gap is a signal that novelty-based contamination resistance is working.

What kind of contribution is this? This is an operational demonstration with conceptual implications. The paper shows that expert authorship at scale is organizationally feasible — not easy, but feasible — and that the resulting benchmark exhibits properties (large open-vs-closed gap on advanced material, scaling behavior that suggests under-training rather than overfitting) consistent with genuine novelty. The contribution is not the idea of expert authoring (which predates SOOHAK) but the demonstration that it can be executed at the scale needed for statistically reliable evaluation, with the transparency needed for community trust, under the time and budget constraints that real benchmark construction projects face. The retrospective in Appendix F is arguably as important as the benchmark itself — it provides concrete guidance for future builders about what failed, what was expensive, and what they would do differently.

Innovation 4: The Challenge of Measuring Refusal Through Scaling Behavior

The paper's finding that Refusal does not scale with model size or test-time compute in the same way that Challenge performance does is not a metric gain but a diagnostic negative result that reveals something structural about current model capabilities and, by extension, about what the Refusal subset actually measures.

The core finding (Figure 2, Tables 5-6, Section 5). Challenge Pass@3 scales roughly linearly with Qwen3 model size: from 2.94% at 0.6B to 15.29% at 32B, adding roughly 3 points per doubling after the initial jump. Challenge also scales with test-time compute: extending GPT-OSS-120B from medium to hard reasoning gains 7.94 points, and extending context to 81,920 tokens adds another 3.24 points. Refusal shows no such patterns. Qwen3-235B scores 5.05% on Refusal, lower than Qwen3-32B at 28.28% — larger models can be worse at refusal. Qwen3 is described as "a clear outlier in the other direction, performing worst on SOOHAK Refusal across the panel" (Section 5). GLM-5, a model in the same open-weight class, scores 49.49% — nearly 10× higher. These are not gradual scaling trends; they are wild swings that suggest refusal capability is not a smooth function of model capacity.

Why this is conceptually significant. The finding challenges an implicit assumption in the field: that capabilities improve monotonically with scale, and that more compute (training or inference) makes models better at everything. The paper describes this explicitly: "Refusal does not show the same scaling patterns; what governs refusal and hallucination behavior we leave to future work" (Section 5). This admission of ignorance is informative — it tells the community that current scaling approaches (bigger models, longer reasoning chains) may not address the refusal failure mode, and may even exacerbate it (as in the Qwen3 case, where the largest model is the worst at refusal).

Interpretation. The paper does not claim to explain why refusal doesn't scale, but the juxtaposition of findings suggests a plausible mechanism. Models are trained to produce correct answers to well-posed problems — this capability improves with scale because larger models have more knowledge and better reasoning. Refusal requires recognizing when the problem itself is ill-posed, which is a different skill: it requires checking for internal consistency, verifying that assumptions are sufficient, and detecting that no unique answer exists. This metacognitive check is not explicitly trained in current post-training recipes (instruction tuning, RLHF, reasoning reinforcement learning) — there is no loss term that rewards the model for saying "this problem is broken." Consequently, scaling model size or reasoning budget improves problem-solving capability without necessarily improving — and potentially degrading — the ability to recognize when problem-solving is inappropriate.

The Qwen3 family result (largest model performs worst) is particularly suggestive. If refusal is suppressed by training objectives that reward producing correct answers — and larger models are more strongly optimized for this objective — then the largest, most capable reasoners might also be the most prone to confidently solving unanswerable problems. This is a form of Goodhart's law in benchmark design: the metric "produces correct answers to well-posed problems" does not imply "recognizes ill-posed problems," and optimizing for the former may directly harm the latter.

Evidence strength and limitations. The evidence for this finding comes from a single model family (Qwen3) and a small set of test-time scaling experiments (GPT-OSS-120B and Qwen3-235B). The paper does not have refusal scaling data for the closed models (Gemini, GPT-5, Claude) across model sizes or reasoning budgets — the Refusal scores in Table 2 are point estimates at a single configuration. The claim that "refusal doesn't scale" is therefore preliminary and model-family-specific; it may not generalize to all architectures or post-training recipes. The paper appropriately hedges: "what governs refusal and hallucination behavior we leave to future work" (Section 5).

What kind of contribution is this? This is a diagnostic negative result — a finding that a capability the benchmark measures does not improve with scale in the expected way, which reveals that current scaling approaches target a different capability than what Refusal measures. It is incremental in evidence (limited to two model families, small experiments) but fundamental in implication: if refusal is structurally independent of reasoning capability, then building models that are both good reasoners and appropriately cautious will require deliberate design beyond simply scaling up existing recipes. The paper has identified a capability gap that current training paradigms do not address, and it has provided a benchmark instrument for measuring progress on closing that gap.

Innovation 5: Human Baselines as Diagnostic Probes, Not Just Performance Targets

The paper's human baseline study serves a different purpose than typical benchmark evaluations. Rather than treating human performance as a target to be matched or exceeded — as in "model X exceeds human performance on benchmark Y" — SOOHAK uses its human baseline to diagnose what the benchmark actually measures.

What the field typically does. Standard practice, from ImageNet to MATH to FrontierMath, is to report an aggregate human score (often from a single expert or a small homogeneous group) and compare it to model aggregate scores. The comparison is treated as a validity check: if humans score much higher than models, the benchmark is "hard"; if models approach or exceed human performance, the benchmark is approaching saturation. The human score is a scalar used to position model performance on an absolute scale, but it reveals little about what kind of capability the benchmark rewards.

SOOHAK inverts this by deliberately constructing a heterogeneous human panel with five teams spanning distinct expertise profiles: CS majors with IMO experience, math majors with IMO experience, math majors with IMO gold medals, math majors with more programming focus, and PhD mathematics researchers. The performance variation across these profiles — not just the aggregate human vs. model comparison — is the diagnostic signal.

The key finding (Figure 3, Section 6.2). Contest-trained undergraduates ("Math Major with IMO experience" and "Math Major with IMO Gold") outperform PhD mathematics researchers on this benchmark, despite the researchers having more advanced mathematical knowledge. The paper interprets this not as evidence that undergraduates are "better at math" than PhDs, but as evidence that the benchmark's format — 4.5-hour time constraint, broad coverage across subfields, final-answer-only scoring — systematically advantages contest-style reasoning over deep research expertise. As the paper puts it: "We view this as a task-format mismatch, not an ability gap. Two factors likely drive this. The 4.5-hour budget incentivizes short-path solutions more natural to contest-trained mathematicians, and the benchmark's breadth limits the advantage of narrow research specialization" (Section 6.2).

Why this diagnostic framing matters. This finding has direct implications for how benchmark scores should be interpreted. If a model outperforms humans on SOOHAK, it does not necessarily mean the model has surpassed human research mathematicians — it means the model is better than humans at contest-style problem solving under time pressure on problems drawn from multiple subfields. A PhD researcher might take days to solve a problem within their specialty using deep theoretical machinery; the benchmark's format does not reward that approach. The human baseline thus reveals what the benchmark is measuring (contest-style breadth and speed) as distinct from what it might appear to measure (research-level mathematical competence). This is a form of construct validity analysis performed through human evaluation design, not through psychometric statistics.

The paper provides additional behavioral observations that reinforce this interpretation: "humans tend to avoid long, notation-heavy items even when they are not intrinsically difficult, whereas LLMs apply more uniform effort across question lengths" (Section 6.2). This reveals another format effect: LLMs have an advantage on problems that are not mathematically harder but are notationally denser, because they do not suffer from the attentional fatigue that makes humans skip such items under time pressure.

Evidence strength. The human baseline is based on a 79-problem subset, not the full benchmark. The per-team sample sizes (5 participants per team, one 4.5-hour session) are small, and the paper acknowledges that "session conditions were not fully standardized across sessions" (Section 6.1). The finding that contest-trained undergraduates outperform PhD researchers is directionally clear (Team B at 38.0% vs. Team E at 24.1%, Figure 3) but the absolute magnitudes should be treated with caution given the small sample. The paper does not report confidence intervals or statistical tests, which limits the strength of comparative claims.

What kind of contribution is this? This is a methodological contribution to benchmark evaluation practice — a demonstration that human baselines can and should be designed to reveal construct validity, not just to provide a scalar performance target. It is incremental in execution (five teams, one session each, 79 problems) but conceptually important: it provides a template for how future benchmarks might use heterogeneous human panels to understand what their instruments are actually measuring, as distinct from what they intend to measure. The finding that SOOHAK rewards contest-style reasoning under time pressure rather than deep research expertise does not invalidate the benchmark — it clarifies it, enabling users to interpret scores with appropriate nuance.

5. Experimental Analysis

Evaluation Methodology

Dataset. SOOHAK comprises two main subsets: Challenge (340 items) and Refusal (99 items), with a companion SOOHAK-Mini (702 items) tracking smaller and open-weight systems. All items are newly authored from scratch by 64–105 mathematicians under originality agreements; the full collection will be publicly released in late 2026, with model evaluations available upon request in the interim (Section 1, Section 3.5). The Challenge subset targets graduate-level and research-adjacent mathematics, routing problems through a model-gated pipeline that requires failure of all large open-weight reasoning models (GPT-OSS-120B, Qwen3-235B, DeepSeek-R1) during collection. The Refusal subset draws from submissions rejected during quality control for being ill-posed, containing contradictions, missing assumptions, or lacking unique answers (Appendix B.8).
Base models. The evaluation panel spans eleven systems: seven closed models (Gemini-3-Pro, Gemini-3-Flash, GPT-5 Medium, GPT-5-Mini Medium, Claude-Opus-4.5, Claude-Sonnet-4.5, Grok-4.1-Fast) and four open-weight systems (Qwen3-235B-A22B-thinking-2507, GPT-OSS-120B, Kimi-2.5, GLM-5). The paper evaluates these specific models because they represent the frontier of both closed and open-weight mathematical reasoning capability at the time of writing (Section 4). For family-scale analysis, the full Qwen3 parameter spectrum (0.6B to 235B, including -2507 thinking-tuned checkpoints) is evaluated on all splits (Appendix D.2). For GPT-5, the paper notes that GPT-5, GPT-5.1, and GPT-5.2 were all evaluated with identical configurations; GPT-5 yielded the best performance and is therefore reported in all tables (Section 4, footnote).
Metrics. The primary metrics are Avg@3 and Pass@3, computed from three independent responses per model–question pair (Section 4). Avg@3 is the mean correctness rate across all questions, averaging the three-sample binary correctness per question: it estimates per-sample expected accuracy under the given decoding configuration. Pass@3 is the fraction of questions for which at least one of the three samples is correct: it measures coverage, capturing whether the model can produce a correct answer with any of three attempts. On the Refusal subset, correctness is defined by diagnostic refusal behavior rather than answer matching: a response is correct only if it identifies the flaw in the problem and refuses to produce a numeric answer; confident answers to ill-posed problems are marked incorrect, as are refusals that do not engage with the mathematical validity of the prompt (Appendix B.8).
Baselines. The paper does not compare against prior benchmark scores or standardized baselines in the traditional sense, since SOOHAK is a new dataset. Instead, the primary comparative structure is cross-model ranking — all eleven evaluated systems are run on the same items under standardized protocols, and scores are compared across models within each split. The closest thing to a "baseline" is the performance of the weakest frontier models (Claude-Sonnet-4.5 at 5.69% on Challenge, Qwen3-235B at 8.04%) and the strongest models (Gemini-3-Pro at 30.39%, GPT-5 at 26.37%), which establish the achievable range on this benchmark. For the human comparison, the baseline is aggregate human team coverage (50.6% on the 79-problem evaluation set), providing an interpretable reference point for difficulty rather than a target to beat (Section 6, Figure 3).
Generation budget / compute accounting. The main evaluation uses a fixed per-question budget of three independent samples, with decoding configurations following provider recommendations (Appendix D.1, Table 4). For test-time scaling ablations, the paper varies two dimensions: reasoning effort (medium vs. hard for GPT-OSS-120B) and context budget (default vs. 81,920 tokens for GPT-OSS-120B and Qwen3-235B-A22B-thinking-2507, representing a 5× increase from the default 16,384-token allocation; Appendix D.3, Table 6). For parameter-count scaling, the paper evaluates the Qwen3 family at 0.6B, 1.7B, 8B, 14B, 32B, and 235B parameters (Appendix D.2, Table 5). Compute is measured implicitly through model size (parameters) and per-question token allocation (context budget × reasoning effort), without explicit FLOPs accounting.
Cross-validation / statistical protocol. The paper does not employ cross-validation or statistical significance testing on model scores. The evaluation is a fixed comparison: all eleven models, all splits, three samples per question, one-time scoring. The human baseline (Section 6) uses a 79-problem subset sampled across splits, with five independent teams evaluated once under a 4.5-hour time budget. The composite scores (Capability, Avg-R, SOOHAK-R) in Appendix D.4 are purely descriptive aggregations of Pass@3 scores with no confidence intervals. The paper does not report variance estimates, standard errors, or statistical tests for any model-to-model comparisons.

Main Quantitative Results

Overall Split Performance and Model Ranking

Table 2 reports the headline scores for all eleven models across SOOHAK-Mini, SOOHAK Challenge, and SOOHAK Refusal. The key aggregate findings are:

On SOOHAK-Mini (702 items), frontier models are strong but not saturated. GPT-5 achieves the highest Avg@3 at 72.22%, followed closely by Gemini-3-Pro at 71.70% and Grok-4.1-Fast at 70.66% (Table 2). Pass@3 scores are roughly 8–10 points higher: GPT-5 reaches 81.48%, Gemini-3-Pro reaches 80.63%. The strongest open-weight model on SOOHAK-Mini is Kimi-2.5 at 66.07% Avg@3 (74.93% Pass@3), followed by GLM-5 at 63.11% and GPT-OSS-120B at 61.02%. The gap between the best closed and best open-weight model on SOOHAK-Mini is approximately 6.1 percentage points (72.22% vs. 66.07%), which is meaningful but not enormous. Claude-Sonnet-4.5 is the weakest model overall at 40.88% Avg@3, indicating that SOOHAK-Mini spans a wide difficulty range and discriminates across the full model capability spectrum.

On SOOHAK Challenge (340 items), all models struggle substantially. Gemini-3-Pro leads with 30.39% Avg@3 (44.12% Pass@3), followed by GPT-5 at 26.37% (40.88% Pass@3) and Claude-Opus-4.5 at only 10.39% (18.82% Pass@3). The drop from SOOHAK-Mini to Challenge is dramatic: GPT-5 falls from 72.22% to 26.37% (a 45.9-point decline), and the strongest open-weight model (Kimi-2.5) falls from 66.07% to 13.87% (a 52.2-point decline). The best open-weight Challenge score is 13.87% (Kimi-2.5), with GPT-OSS-120B at 11.27% and GLM-5 at 9.61%. The gap between the best closed and best open-weight model on Challenge is approximately 16.5 percentage points (30.39% vs. 13.87%), roughly 2.7× larger than the gap on SOOHAK-Mini. The paper reports that 124 Challenge items remain unsolved by any evaluated model and 170 items are unsolved or missed in total (Section 5), establishing substantial headroom for future systems.

On SOOHAK Refusal (99 items), performance is poor and rankings invert relative to reasoning splits. GLM-5 achieves the highest Avg@3 at 49.49% (73.74% Pass@3) — the only model approaching 50% on this split. Closed models cluster between 41–43%: GPT-5 at 43.09%, Gemini-3-Flash at 43.10%, Gemini-3-Pro at 41.41%, GPT-5-Mini at 41.08%. Claude models score substantially lower (Claude-Opus-4.5 at 26.60%, Claude-Sonnet-4.5 at 27.61%). The Qwen3 family is a "clear outlier in the other direction" (Section 5): Qwen3-235B-A22B-thinking-2507 scores only 2.69% Avg@3 (5.05% Pass@3) — the worst Refusal performance in the panel, and dramatically worse than its Challenge score of 8.04% and its SOOHAK-Mini score of 56.22%.

The composite scores in Appendix D.4 (Figure 4) demonstrate how Refusal performance reshapes rankings. GPT-5 takes the top Avg-R rank from Gemini-3-Pro despite lower reasoning scores (Capability: Gemini-3-Pro first, GPT-5 second), because GPT-5's Refusal score (43.09%) exceeds Gemini-3-Pro's (41.41%), and this gap outweighs Gemini's advantage on reasoning. GLM-5 rises 3 ranks from a purely reasoning-based ranking to Avg-R. Kimi-2.5 drops 3 ranks because its Refusal score (29.97%) is low relative to its reasoning capability. These ranking shifts are systematic: they reflect the fact that reasoning ability and epistemic caution are not strongly correlated across models.

Open-Weight vs. Closed Model Gap Analysis

Table 2 reveals a pattern that the paper highlights as a central finding: the capability gap between closed and open-weight models widens substantially as difficulty increases. On SOOHAK-Mini, the gap between the best closed (GPT-5, 72.22%) and best open-weight (Kimi-2.5, 66.07%) is 6.15 percentage points. On SOOHAK Challenge, the gap between the best closed (Gemini-3-Pro, 30.39%) and best open-weight (Kimi-2.5, 13.87%) is 16.52 percentage points — 2.7× larger. The paper interprets this as evidence that "open-weight systems transfer less reliably to unpublished and research-adjacent mathematics, which is consistent with recent attempts to apply LLMs to unresolved mathematical problems relying on top-performing closed systems" (Section 5). On Refusal, the pattern inverts: the best open-weight model (GLM-5, 49.49%) exceeds every closed model by 6+ percentage points, and GPT-OSS-120B (43.77%) is competitive with the best closed models.

Subfield-Level Performance Heterogeneity

Appendix D.5 (Table 7) provides per-MSC-subfield accuracy across all 18 full-coverage models, restricted to subfields with at least 20 problems. The paper identifies three patterns:

Uniformly challenging subfields show low mean performance across all models. MSC 16 (Rings and algebras, 24 problems) has a mean accuracy of 14.6% and a best score of only 48.6% (Gemini-3-Pro). MSC 52 (Convex and discrete geometry, 30 problems) has a mean of 24.7% and a best of 57.8% (Gemini-3-Pro). These subfields are difficult for all models regardless of architecture or training recipe.

Uniformly easier subfields show high mean performance with compressed ranges. MSC 40 (Series and summability, 36 problems) has a mean of 69.8% and a best of 80.6% (Grok-4.1-Fast). MSC 11 (Number theory, 269 problems) has a mean of 57.8% and a best of 71.9% (Gemini-3-Pro). MSC 26 (Real functions, 115 problems) has a mean of 59.4% and a best of 75.1% (Gemini-3-Pro). These subfields are relatively tractable for current systems.

High-disagreement subfields show large gaps between the best and worst models, making them the most diagnostic for differentiating model capability. MSC 52 (Convex/discrete geometry) has a range of 57.8 percentage points between best and worst model. MSC 60 (Probability, 24 problems) has a range of 55.6 pp, with Grok-4.1-Fast leading at 59.7%. MSC 5 (Combinatorics, 131 problems) has a range of 48.6 pp, with Gemini-3-Pro leading at 51.9%. MSC 15 (Linear algebra, 33 problems) is the only subfield where an open-weight model leads: GPT-OSS-120B with hard reasoning and 81,920-token context scores 55.6% (Table 7).

The per-subfield leader rotation is notable: Gemini-3-Pro leads number theory (MSC 11), real analysis (MSC 26), combinatorics (MSC 5), and algebraic subfields (MSC 16). Grok-4.1-Fast leads series/summability (MSC 40), geometry (MSC 51), and probability (MSC 60). GPT-OSS-120B leads linear algebra (MSC 15). No single model dominates across all subfields, suggesting that mathematical capability is not a monolithic construct — different models have different subfield-specific strengths.

Compute Scaling: Parameter Count and Test-Time

Qwen3 parameter scaling (Appendix D.2, Table 5, Figure 2 left). The paper evaluates the Qwen3 family from 0.6B to 32B (standard release) and 4B to 235B (-2507 thinking-tuned checkpoints) to measure how performance scales with model size. On SOOHAK Challenge, Pass@3 rises from 2.94% (0.6B) to 5.59% (1.7B) to 8.82% (8B) to 12.06% (14B) to 15.29% (32B) — a roughly linear trajectory adding approximately 3 points per checkpoint. The -2507 thinking-tuned 235B model scores 15.00%, essentially flat relative to the 32B standard model (15.29%) and the 30B-A3B thinking model (15.59%), suggesting that Challenge performance does not benefit from further parameter scaling within the Qwen3 family beyond the ~30B threshold. On SOOHAK-Mini, scaling continues to help: 35.75% (0.6B) to 70.80% (32B). The -2507 235B model regresses to 67.66% on SOOHAK-Mini relative to the 30B-A3B at 71.23%, which the paper attributes to output-length bottlenecks — the 81,920-token context variant (Table 6) partially recovers to 70.23%.

SOOHAK Refusal does not scale with parameter count within the Qwen3 family. Pass@3 on Refusal shows a non-monotonic pattern: 11.11% (0.6B), 5.05% (1.7B), 5.05% (8B), 21.21% (14B), 28.28% (32B), then collapsing to 5.05% at 235B-A22B-thinking-2507 (Table 5). The largest model in the family performs worst on Refusal — a stark inversion of typical scaling expectations. The paper describes Qwen3 as "a clear outlier in the other direction, performing worst on SOOHAK Refusal across the panel" (Section 5), but within-family scaling shows this is not simply a Qwen3-specific failure mode: it is specifically the largest Qwen3 variant that fails most dramatically.

Test-time scaling (Appendix D.3, Table 6, Figure 2 middle). On GPT-OSS-120B, increasing reasoning effort from medium to hard yields gains of 1.57 points on SOOHAK-Mini (75.21% → 76.78% Pass@3) and 7.94 points on Challenge (18.53% → 26.47%). Further extending the context budget to 81,920 tokens adds another 4.13 points on SOOHAK-Mini (76.78% → 80.91%) and 3.24 points on Challenge (26.47% → 29.71%). The total gain from default medium reasoning to hard reasoning with extended context is 5.70 points on SOOHAK-Mini and 11.18 points on Challenge — nearly doubling Challenge performance relative to the default configuration. On Qwen3-235B-A22B-thinking-2507, extending context to 81,920 tokens lifts Challenge from 15.00% to 22.35% (+7.35 points) and SOOHAK-Mini from 67.66% to 70.23% (+2.57 points). Refusal shows inconsistent test-time scaling: GPT-OSS-120B drops from 60.61% (medium) to 52.53% (hard) to 55.56% (hard, extended), while Qwen3-235B improves from 5.05% to 9.09% with extended context. The paper explicitly notes: "Refusal does not show the same scaling patterns; what governs refusal and hallucination behavior we leave to future work" (Section 5).

Unsolved items (Figure 2 right). The fraction of Challenge items unsolved by any model in the panel decreases as more models are considered, but a substantial residual remains: 124 items are unsolved by any evaluated model in the default evaluation panel. The paper notes that this "exceeds smaller benchmarks such as Riemann-Bench (≥23/25)" (Section 5), meaning the number of items that no current system can solve is larger than the total size of some prior research-level benchmarks, establishing the scale of remaining headroom.

Human Baseline Results

Human team performance (Section 6.2, Figure 3, right panel). On the 79-problem human evaluation set (49 SOOHAK-Mini + 30 Challenge):

Combined human coverage (union of all questions solved by any team) reaches 50.6% — meaning at least one of the 25 human participants across five teams solved half the prompts.
Math Major with IMO experience (Team B) is the strongest single team at 38.0%, followed by Math Major with IMO Gold (Team C) at 31.6%.
Math Researchers (Team E, PhD holders) score 24.1% — substantially below the contest-trained undergraduate teams.
Math Major (Team D, more programming-oriented) scores 21.5%.
CS Major with IMO experience (Team A) scores lowest at 12.7%.

Model vs. human comparison (Figure 3, left panel). On the same 79-prompt set at Pass@1:

Gemini-3-Pro achieves 60.8% — the only model that exceeds combined human coverage (50.6%).
GPT-5 scores 43.0%, GPT-5-Mini scores 41.8%.
The strongest open-weight model on this subset is Qwen3-235B-thinking at 39.2%.
Claude-Opus-4.5 scores 30.4%, DeepSeek-R1-0528 scores 26.6%, Claude-Sonnet-4.5 scores 21.5%.

Interpretation of human results. The paper draws several conclusions from the human baseline (Section 6.2):

Contest training, not research experience, drives performance. The ordering (Math Major with IMO experience > Math Major with IMO Gold > Math Researchers) reveals that the benchmark's format — 4.5-hour time constraint, broad subfield coverage, final-answer-only scoring — systematically advantages contest-style reasoning under time pressure over deep research expertise. The paper explicitly frames this as "a task-format mismatch, not an ability gap" (Section 6.2).
Sustained pure-math training matters beyond contest credentials. CS Major with IMO experience (Team A, 12.7%) underperforms the pure-math teams (Team B, 38.0%; Team C, 31.6%) despite comparable olympiad credentials. The paper interprets this as evidence that "sustained undergraduate-level pure-math training provides an additional advantage on this benchmark" (Section 6.2), suggesting that immersion in mathematical coursework — not just contest problem-solving — builds capabilities that transfer to this evaluation.
Models apply uniform effort; humans avoid notation-heavy items. The paper observes that "humans tend to avoid long, notation-heavy items even when they are not intrinsically difficult, whereas LLMs apply more uniform effort across question lengths, yielding an additional coverage advantage beyond raw competence" (Section 6.2). This is a format effect that advantages models independently of their mathematical reasoning capability.
Collaboration strategy matters. Team C (Math Major with IMO Gold) "emphasized parallelism and internal competition, attempting 44 problems and solving 25 correctly," while Team B (Math Major with IMO experience) "emphasized division of labor with cross-checking, attempting 38 problems and solving 30 correctly" (Appendix E). Team B's higher solve rate on fewer attempts suggests that cross-checking improves accuracy at the cost of throughput — a tradeoff that the benchmark's time pressure penalizes but that yields higher final scores in this format.

Ablation Studies and Robustness Checks

The paper's ablation and robustness analysis is distributed across appendices and is relatively limited compared to model-training or method-development papers, reflecting the benchmark-construction focus. The key analyses are:

Qwen3 family parameter scaling across splits (Appendix D.2, Table 5). The full parameter spectrum from 0.6B to 235B is evaluated on all three splits, revealing that Challenge scales roughly linearly with model size up to ~30B parameters but plateaus thereafter, SOOHAK-Mini scales more smoothly across the full range but shows regression at 235B (attributed to output-length constraints), and Refusal shows non-monotonic behavior with the largest model performing worst. This ablation establishes that the three splits measure fundamentally different capability dimensions — they do not simply reflect difficulty on a single underlying scale.

Test-time compute scaling with reasoning effort and context budget (Appendix D.3, Table 6). On GPT-OSS-120B, the three-configuration sweep (medium reasoning, hard reasoning, hard with 81,920-token context) shows monotonic improvement on reasoning splits: SOOHAK-Mini gains from 75.21% to 76.78% to 80.91%, Challenge gains from 18.53% to 26.47% to 29.71%. On Qwen3-235B, the two-configuration sweep (default context, 81,920-token context) shows Challenge gains from 15.00% to 22.35% and SOOHAK-Mini gains from 67.66% to 70.23%. The Refusal pattern is inconsistent: GPT-OSS-120B drops on hard reasoning (60.61% → 52.53%) then partially recovers with extended context (55.56%), while Qwen3-235B nearly doubles (5.05% → 9.09%) but remains extremely low. No monotonic scaling relationship is observed for Refusal.

Model family variant comparison for GPT-5 (Section 4, footnote, Appendix D.1). The paper evaluates GPT-5.1, GPT-5.2, and GPT-5 under identical configurations; GPT-5 yields the best performance and is reported in all tables. This is a limited family-internal comparison rather than a systematic ablation, but it confirms that the reported GPT-5 scores represent the strongest configuration within that model generation.

Reasoning effort and context budget interaction (Appendix D.3). The GPT-OSS-120B sweep reveals that increasing reasoning effort from medium to hard provides larger gains on Challenge (+7.94 points) than on SOOHAK-Mini (+1.57 points), while extending context provides roughly comparable gains on both (+4.13 for SOOHAK-Mini, +3.24 for Challenge). This suggests that harder problems benefit more from deeper reasoning chains, while longer context helps uniformly across difficulty levels.

The 5% upper-bound error rate from external validation (Appendix B.3). An external evaluation organization examined a randomly sampled portion of the collection after the main collection phase and flagged approximately 5% of items for potential issues. After addressing all flagged cases via correction or clarification, the paper estimates that "the fraction of items with substantive errors is at most 5%." This is not an ablation in the traditional sense but serves as a quality-control robustness check: it establishes that the benchmark's correctness is bounded, not perfect.

Contributor pool composition and compensation structure (Appendix B.2). The paper reports that the top five contributors by volume (all students) submitted 434 questions (mostly routed into SOOHAK-Mini), while the top five earners (predominantly faculty, receiving USD 68,000 total) had no overlap with the volume group, their contributions being mostly routed into Challenge. This divergence confirms that the compensation design (rewarding difficulty over throughput) functioned as intended, steering expert effort toward the hardest problems.

ScienceBench integration and peer-review process (Appendix B.4). The 112 ScienceBench-sourced Challenge problems underwent a different gating process (contributor peer review rather than LLM-failure routing). The paper notes that "every submission was solved by at least one model," and verification focused on confirming that model and contributor solutions "relied on the same core reasoning ideas." This asymmetry — Challenge contains both gate-3-routed primary problems and peer-reviewed ScienceBench problems — is not tested as an ablation (scores are not reported separately), but the paper acknowledges it as a methodological difference.

Bilingual translation quality assurance (Appendix B.7). The translation pipeline includes professional post-editing, independent QA review, and automated checks for LaTeX renderability, formula equivalence, and terminology consistency. No quantitative translation quality metric is reported, but the paper describes the multi-stage process as a robustness measure ensuring that the Korean and English versions are semantically equivalent.

Composite score sensitivity to Refusal weighting (Appendix D.4). The three composite scores (Capability, Avg-R, SOOHAK-R) implicitly weight the Refusal dimension at 0, 1/3, and 1/2 respectively. The paper does not systematically vary weights, but the shift from 0 to 1/3 weighting produces ranking changes of up to 3 positions (Figure 4), demonstrating that the Refusal dimension is not redundant with reasoning performance and that composite scores are sensitive to how Refusal is incorporated.

Critical Assessment

Claim 1: "Leading closed models reach only 30.4% Avg@3 on Challenge, leaving substantial headroom." Strongly supported by Table 2. The best score is 30.39% (Gemini-3-Pro), and 124 items are unsolved by any evaluated model. The claim is straightforward and the evidence directly matches. However, the paper does not report confidence intervals on these scores. With 340 items and 3 samples each, the standard error on a 30.4% estimate is approximately sqrt(0.304 × 0.696 / 340) ≈ 2.5 percentage points. This means the true capability of these models on this benchmark is uncertain within a ~5-point range (95% CI), which is large relative to the gaps between top models (Gemini-3-Pro at 30.39% vs. GPT-5 at 26.37% — a 4.02-point gap that is within plausible estimation error). The paper's ranking of top models on Challenge should be interpreted with this uncertainty in mind.

Claim 2: "The strongest open-weight model reaches only 13.9% on Challenge." Supported with qualifications. Table 2 reports Kimi-2.5 at 13.87%, but this is with default decoding configurations. GPT-OSS-120B with hard reasoning and 81,920-token context reaches 29.71% Pass@3 on Challenge (Table 6), which is not directly comparable to the 13.87% Avg@3 (different metric, different configuration) but suggests that open-weight models may be capable of substantially higher Challenge performance with test-time compute scaling. The claim of "13.9%" is true for default configurations but understates what open-weight models can achieve with extended reasoning budgets — a caveat the paper does not emphasize in the headline claim.

Claim 3: "On Refusal, no model exceeds 50%." Supported. GLM-5 reaches 49.49% Avg@3, GPT-5 reaches 43.09%. However, the Refusal grading logic has an important asymmetry that the paper acknowledges: correctness is defined by diagnostic refusal behavior, but the paper does not evaluate whether models refuse on well-posed problems (false positives). A model that simply refuses to answer every question would score 100% on Refusal but 0% on Challenge; the benchmark does not measure this tradeoff directly. The composite scores (Avg-R) partially address this by penalizing low Refusal scores, but they cannot detect models that achieve high Refusal by being overly cautious on all problems. The Refusal scores should therefore be interpreted as a lower bound on true refusal capability — a model's actual tendency to recognize ill-posed problems may be higher than its Refusal score if it also refuses on some well-posed problems, but the benchmark does not capture this.

Claim 4: "Model performance scales roughly linearly with both train- and test-time compute on Challenge but not on Refusal." Supported directionally but with limited evidence. The parameter-count scaling claim rests on a single model family (Qwen3, 0.6B to 32B) and shows plateauing at 235B. The test-time compute scaling claim rests on two model families (GPT-OSS-120B with three configurations, Qwen3-235B with two configurations). These are small experiments — three data points for one family, two for another — and "roughly linear" is a strong characterization for such limited data. More importantly, the test-time scaling is measured through provider-exposed reasoning-effort and context-budget parameters, which are black-box controls whose effects on internal computation are not transparent. "Increasing reasoning effort from medium to hard" may increase compute by an unknown factor; the paper does not attempt to measure the actual FLOPs multiplier. The claim that Challenge "scales with test-time compute" is better supported than the claim that it scales "roughly linearly," given the limited data points and unknown compute multipliers.

Claim 5: "Open-weight systems transfer less reliably to unpublished and research-adjacent mathematics." This is an interpretation of the Challenge open-vs-closed gap (16.5 points), not a directly tested claim. The paper cannot verify why open-weight models underperform on Challenge — it could be because the material is absent from public training data (the interpretation offered), or because closed models have architectural advantages (different model scales, different post-training recipes, different reasoning implementations), or because the evaluated open-weight models are simply less capable overall and the Challenge gap is a proportional reflection of a general capability gap (open-weight models trail on SOOHAK-Mini as well, just by less). The paper's contributor interviews (Appendix B.5) provide anecdotal support — contributors "found it substantially easier to write questions that remain challenging for frontier systems" when relevant mathematics was "absent from accessible papers, scattered across niche sources, or hidden behind paywalls" — but this is qualitative and does not rule out alternative explanations. The claim is plausible but underdetermined by the evidence presented.

Missing experiments that would have strengthened the paper:

Confidence intervals on all reported scores. The paper reports point estimates without error bars, making model-to-model comparisons difficult to interpret. With 340 Challenge items and 3 samples each, the sampling error is non-trivial.
Per-split scores for the ScienceBench-sourced items vs. primary-system items within Challenge. If these subpopulations have different properties (as suggested by the different collection protocols), reporting them separately would allow assessment of whether the Challenge results are sensitive to sourcing method.
Refusal false-positive measurement. Evaluating whether models correctly solve well-posed problems vs. refuse them would complete the picture of refusal behavior. A model with high Refusal and low Challenge might be overly cautious rather than appropriately discriminating.
Direct comparison of test-time compute scaling across all models, not just two families. The test-time scaling experiments are limited to GPT-OSS-120B and Qwen3-235B. Extending this to the closed models (e.g., comparing GPT-5 at different reasoning efforts) would clarify whether the scaling patterns are general or model-specific.
Statistical tests for the human baseline comparisons. With 5 participants per team and one session each, the human scores have substantial variance. Without confidence intervals, the claim that contest-trained undergraduates outperform PhD researchers — while directionally clear (38.0% vs. 24.1%) — cannot be evaluated for statistical significance.
Per-language score reporting (English vs. Korean). The bilingual construction is a significant investment, but no per-language results are reported. This is a missed opportunity to demonstrate the benchmark's value for cross-lingual evaluation.

Strengths of the experimental design:

Multi-model, multi-split evaluation provides rich comparative signal. Eleven models across three splits, plus parameter scaling and test-time scaling variants, yields a much more nuanced picture of model capabilities than a single score on a single benchmark.
The Refusal split genuinely measures something different. The non-monotonic scaling behavior (Qwen3-235B performs worst on Refusal despite being the largest Qwen3 variant), the ranking inversions between reasoning and refusal (GLM-5 rises, Qwen3 falls), and the inconsistent test-time scaling all converge on the same conclusion: Refusal is not simply a harder version of the same task. This is a genuine empirical finding that validates the benchmark's multi-dimensional design.
The human baseline is diagnostically informative beyond providing a scalar target. The expertise-profile variation (contest-trained undergraduates > PhD researchers) reveals format biases that users should account for when interpreting scores. This is a methodological strength that future benchmark papers should emulate.
Transparency about quality limitations. The paper's willingness to report a 5% upper-bound error rate, acknowledge the compressed timeline and its consequences, and document failure modes in Appendix F is unusual in the benchmark literature and substantially increases trustworthiness. Many benchmarks report scores as if they were error-free; SOOHAK explicitly tells users "to interpret results with appropriate caution" (Appendix B.3).

6. Limitations and Trade-offs

Difficulty Estimation Cost Is Unaccounted for in Headline Efficiency Claims

The model-gated difficulty routing system — which assigns problems to SOOHAK-Mini vs. Challenge based on whether specific LLMs fail to solve them — requires evaluating every submitted problem against a panel of baseline models (Qwen3-8B, OpenThinker3-7B, GPT-OSS-20B, Qwen3-32B, GPT-OSS-120B, Qwen3-235B, DeepSeek-R1) during the collection phase. The paper is transparent that this gating is the mechanism that produces the benchmark's discrimination profile, but it does not account for the computational cost of this gating in any reported metric.

The paper does not explicitly quantify the total inference compute consumed by the gating pipeline. However, the scale can be estimated: each of the 1,141 accepted problems (plus an unknown number of rejected submissions) was attempted by up to seven models across three gates, with at least one sample per model. At inference costs appropriate to 8B–235B parameter reasoning models, this represents a substantial computational investment that is amortized over the benchmark's construction but not reflected in the per-question evaluation budget reported to users. A practitioner who wants to replicate this pipeline for a new benchmark would need to budget for this gating compute in addition to contributor compensation and review costs — the USD 550,000 total budget (Appendix F) covers personnel and operations but may not fully account for inference costs if models were run on donated compute (the acknowledgments thank EleutherAI and CoreWeave "for providing compute for some of the evaluations").

The consequence is that the benchmark's construction cost is underestimated in a way that matters for reproducibility. A team attempting to build a similar benchmark from scratch would face not only the contributor compensation costs (~USD 260,000) but also substantial inference costs to run the gating pipeline, which scale with the number of submissions × the number of gate models × the per-sample inference cost. The paper's retrospective (Appendix F) identifies the compressed four-month timeline as "a first-order bottleneck" but does not mention gating compute as a constraint — this may reflect the availability of donated compute (EleutherAI/CoreWeave) that a typical academic team would not have access to.

The paper partially mitigates this by using only open-weight models in the gating panel, ensuring that the gating models are freely available and can be run without API costs — but the inference compute itself (GPU-hours for models up to 235B parameters with reasoning enabled) remains substantial and is not quantified. The paper does not suggest future work on cheaper difficulty estimation, as this is a construction-phase cost rather than an evaluation-phase cost; however, for teams seeking to extend or recalibrate the benchmark with new gate models as capabilities evolve, this cost would recur.

Refusal Grading Measures False Negatives but Not False Positives

The Refusal subset evaluates whether models recognize ill-posed problems and refuse to answer them rather than producing confident but incorrect solutions. The grading logic (Appendix B.8) defines a correct response as one that "diagnoses the flaw instead of confidently producing a numeric answer." A response is marked incorrect if it "presents a specific mathematical answer as though the item were well-posed, or if they refuse without engaging the mathematical validity of the prompt."

This grading design creates a fundamental asymmetry: it measures false negatives (failing to refuse on an ill-posed problem) but cannot measure false positives (refusing on a well-posed problem). The Refusal subset contains only items known to be ill-posed; there is no complementary measurement of whether high-Refusal models also incorrectly refuse to answer well-posed Challenge or SOOHAK-Mini items. A model that simply refused to answer every question — outputting some variant of "this problem is ill-posed" regardless of the input — would score 100% on Refusal but 0% on the reasoning splits. The paper does not report per-model refusal rates on the reasoning splits, so it is impossible to distinguish between models that are genuinely discriminating (refusing only on ill-posed problems) and models that are simply more cautious overall.

The paper acknowledges this implicitly through its composite scores (Appendix D.4). The Avg-R metric combines Refusal with reasoning performance, so a model that scores 100% on Refusal by refusing everything would be penalized by its zero reasoning scores. However, for models with intermediate behavior — e.g., a model that refuses on 30% of well-posed problems and 60% of ill-posed problems — the Refusal score of 60% overstates its true discrimination ability, and the composite metrics only partially correct for this by incorporating reasoning performance. The Capability–Refusal gap used in the Avg-R decomposition (Appendix D.4) assumes that the two dimensions are independent, but they are not: a model's baseline tendency to refuse affects both scores, creating a confounding relationship that the composite metrics do not disentangle.

The paper does not attempt to address this limitation. Appendix B.8 describes the grading convention without discussing the false-positive problem. The human baseline study (Section 6) uses only well-posed problems from SOOHAK-Mini and Challenge — it does not evaluate human refusal behavior on the Refusal items. This is a significant gap because it leaves unanswered the question of whether the benchmark is measuring discriminating refusal (refusing appropriately based on problem validity) or simply baseline caution (a general tendency to refuse that happens to score well on Refusal). The paper states that "what governs refusal and hallucination behavior we leave to future work" (Section 5), which is an accurate characterization of the current state but does not resolve the measurement validity concern.

The Benchmark Measures Contest-Style Reasoning Under Time Pressure, Not Research-Level Mathematical Expertise

The human baseline study (Section 6) reveals a structural property of the benchmark that complicates its stated goal of evaluating "research-level math capabilities." The five human teams with varied expertise profiles show a clear performance ordering: contest-trained mathematics undergraduates (Teams B and C, with IMO medals and ongoing pure-math training) achieve 38.0% and 31.6% respectively on the 79-problem evaluation set, while PhD mathematics researchers (Team E) achieve only 24.1% (Figure 3). The paper interprets this explicitly: "We view this as a task-format mismatch, not an ability gap. Two factors likely drive this. The 4.5-hour budget incentivizes short-path solutions more natural to contest-trained mathematicians, and the benchmark's breadth limits the advantage of narrow research specialization" (Section 6.2).

This finding has direct implications for what the benchmark measures. If PhD researchers — the very population whose expertise the benchmark claims to target — underperform contest-trained undergraduates, then the benchmark is not primarily measuring research-level mathematical knowledge. It is measuring the ability to solve diverse mathematical problems quickly, under time pressure, with final-answer-only scoring. This is a legitimate capability to evaluate, and it correlates with mathematical training, but it is not the same as the deep, specialized reasoning that characterizes actual mathematical research. A research mathematician might spend days or weeks on a single problem within their specialty; the benchmark's format does not reward that mode of thinking.

The paper is transparent about this finding and does not attempt to hide it — the human baseline section is explicitly designed to reveal these format biases. The paper's abstract and introduction frame the benchmark as targeting "research-level math capabilities," but the human results suggest that the actual construct being measured is closer to "advanced contest-style reasoning under constraints." This is not a failure of the benchmark per se — it is a clarification of what the scores mean — but it means that users should not interpret a model's SOOHAK Challenge score as a measure of its ability to conduct mathematical research. It is a measure of its ability to solve novel, difficult mathematical problems under evaluation conditions that happen to advantage contest-style reasoning.

The paper partially mitigates this by reporting the human results transparently and by describing the observed behavioral differences (e.g., "humans tend to avoid long, notation-heavy items even when they are not intrinsically difficult, whereas LLMs apply more uniform effort across question lengths," Section 6.2). The composite metrics and subfield breakdowns provide additional diagnostic resolution. However, the paper does not offer a way to disentangle contest-style reasoning ability from research-level expertise in the model scores — a model that scores well on Challenge might be genuinely capable of research-level reasoning, or it might simply be very good at contest-style problem solving on novel material. The benchmark cannot distinguish these interpretations.

Single-Domain Scope Limits Generalization Claims

All problems in SOOHAK and SOOHAK-Mini are drawn from pure and applied mathematics, with the distribution concentrated in Algebra & Discrete (680 items, Table 1), Analysis (233 items), and Geometry & Topology (175 items), with minimal representation from Applied/CS/OR (27 items), Probability & Statistics (25 items), and Logic (1 item). The evaluation is conducted exclusively on mathematical problem solving — there is no extension to other reasoning domains such as code generation, scientific reasoning, or formal proof verification.

The paper's claims about model capabilities, scaling behavior, and the relationship between reasoning and refusal are therefore domain-specific by construction. The finding that Challenge performance scales roughly linearly with model size and test-time compute (Section 5, Figure 2) may be specific to mathematical problem solving, where additional computation translates relatively directly into extended reasoning chains and more thorough exploration of solution paths. The finding that Refusal does not scale (and that the largest Qwen3 variant performs worst on Refusal, Table 5) may be an artifact of how mathematical problem-posing interacts with model training — models trained on mathematical corpora may develop strong priors that problems are well-posed, leading to overconfidence on ill-posed items. In other domains (e.g., code generation, where ill-posed specifications are common in real-world settings), the refusal scaling relationship might be entirely different.

The paper does not claim cross-domain generalization. The title and abstract are explicit that the benchmark targets mathematical reasoning. The limitation is not that the paper overclaims, but that the insights it generates — about open-weight vs. closed model gaps on novel material, about the non-scaling of refusal behavior, about the contest-style bias in evaluation format — may not transfer to other reasoning domains without independent verification. A practitioner who wants to understand whether these patterns hold for code generation or scientific reasoning would need to construct analogous benchmarks in those domains, which the paper does not provide guidance for.

The paper makes no attempt to address domain generalization — this is outside its scope. The MSC subfield breakdown (Appendix D.5) provides within-mathematics variation that partially addresses the question of subfield-specific effects (Table 7 shows that different models lead different subfields), but this is still within a single broad domain. The paper's retrospective (Appendix F) identifies "the 'unique integer answer' format is becoming exhausted" as a structural limitation and suggests proof-assistant-based evaluation as a future direction, but this is about evaluation format within mathematics, not about extending to other domains.

Small-Sample and Non-Standardized Human Baseline Limits Comparative Claims

The human baseline study (Section 6, Appendix E) is designed to provide an interpretable reference point for model scores, and it succeeds in revealing format biases (contest-style advantage over research expertise). However, several aspects of the study design limit the strength of the quantitative human–model comparisons.

Sample size and statistical power. Each team consists of only 5 participants, evaluated in a single 4.5-hour session. The evaluation set is 79 problems, meaning that each team's score is based on at most 395 person-problem attempts (5 people × 79 problems, with teams not necessarily attempting all problems). With such small samples, the per-team scores have large standard errors: a team scoring 38.0% on 79 items has an approximate standard error of sqrt(0.38 × 0.62 / 79) ≈ 5.5 percentage points, meaning the 95% confidence interval spans roughly 27% to 49%. The gaps between teams (Team B at 38.0% vs. Team C at 31.6% vs. Team E at 24.1%) are within plausible sampling error — the paper reports no confidence intervals, no statistical tests, and no attempt to quantify the uncertainty in these estimates.

Non-standardized test conditions. The paper explicitly states that "session conditions were not fully standardized across sessions. We report this for transparency, as such operational differences can introduce additional variance in measured performance at frontier difficulty" (Section 6.1). The nature of these differences is not described — they could include time of day, physical environment, availability of computational tools, or level of collaboration permitted within teams. The paper reports that Team C "emphasized parallelism and internal competition" while Team B "emphasized division of labor with cross-checking" (Appendix E) — these are qualitatively different strategies that could produce different outcomes independently of the participants' mathematical ability. Without standardization, it is unclear whether performance differences across teams reflect differences in expertise or differences in collaboration strategy and test conditions.

Non-representative problem sampling. The 79-problem evaluation set upsamples harder problems: 49 from SOOHAK-Mini, 30 from Challenge. This is intentional ("We intentionally upsample harder questions," Section 6.1) because "a small number of hard items is often insufficient to reliably differentiate PhD-level performance." However, it means the human scores on this set are not directly comparable to the full-split model scores reported in Table 2 — the human evaluation set has a different difficulty distribution than either SOOHAK-Mini or Challenge individually. When the paper reports that Gemini-3-Pro at 60.8% exceeds combined human coverage at 50.6% (Figure 3), this comparison is on the 79-problem subset, not the full benchmark, and the model scores on this subset differ from their full-split scores (Gemini-3-Pro reports 71.70% on full SOOHAK-Mini, Table 2).

No human baseline on Refusal. The human evaluation set contains only well-posed problems from SOOHAK-Mini and Challenge (Section 6.1). Human performance on the Refusal subset is not measured. This means we do not know how humans perform on the task of recognizing ill-posed problems — whether PhD researchers would outperform contest-trained undergraduates (plausible, since research experience involves formulating well-posed problems), whether humans would score near-ceiling or near-floor, or whether humans exhibit the same non-scaling patterns that models show. The Refusal split's difficulty relative to human capability is entirely uncalibrated.

The paper mitigates these limitations through transparency — it reports the sample sizes, the non-standardization, the upsampling, and the absence of Refusal testing explicitly. The human baseline is framed as providing diagnostic signal about what the benchmark measures, not as a precise calibration point. However, the quantitative comparisons in Section 6.2 and Figure 3 ("Gemini-3-Pro at 60.8% is the only model that exceeds combined human coverage at 50.6%") are presented without uncertainty quantification, which may lead readers to overinterpret the precision of these comparisons. The paper does not suggest future work to strengthen the human baseline methodology.

Temporary Embargo Creates an Intermediate State Without Full Transparency or Full Protection

The paper adopts a temporary embargo strategy: the dataset will be publicly released in late 2026, with model evaluations available upon request in the interim (Section 1, Appendix B.1). This is positioned as a middle path between permanent access control (which prevents contamination but sacrifices transparency) and immediate public release (which enables transparency but risks contamination before the benchmark serves its evaluation purpose). The embargo is "bounded by the conclusion of the funded evaluation initiative" — the South Korean Sovereign AI competition (Appendix B.1).

This strategy creates an intermediate state during the embargo period (from the paper's publication to late 2026) where:

Independent researchers cannot audit the benchmark. The problems, solutions, and grading logic are not publicly accessible. The paper provides example problems (Section 3.3, Appendix C), describes the collection and gating pipeline, and reports aggregate statistics, but the actual items are withheld. This means that during the embargo period, the paper's claims about the benchmark's properties — difficulty distribution, subfield coverage, answer correctness — cannot be independently verified. The 5% upper-bound error rate from external validation (Appendix B.3) is reported but cannot be replicated.
Independent researchers cannot replicate the evaluation. While the paper reports model scores and describes the evaluation protocol (Section 4, Appendix D.1), the scores cannot be reproduced without access to the items. A researcher who wants to evaluate a new model on SOOHAK must request evaluation through the project organizers — a gatekeeping mechanism that centralizes evaluation control.
Model developers might train on the benchmark before it becomes public. The embargo prevents public web scraping, but the paper reports that model evaluations are "available upon request." This means that closed-model developers who request evaluation receive the problem statements (to run inference) and potentially the solutions (to verify correctness). If these developers retain the items and incorporate them into future training runs — either deliberately or inadvertently through logging — the benchmark could become contaminated before its public release. The paper's legal agreements with contributors govern the project's use of the items but cannot bind third-party model developers who receive items through the evaluation request process.

The paper acknowledges the embargo constraint but does not discuss the intermediate-state tension in detail. Appendix B.1 states that the embargo "reflects this funding context" and is "bounded by the conclusion of the funded evaluation initiative," and that "the project organizers welcome requests for model evaluation." The paper does not describe safeguards against contamination through the evaluation-request channel — for example, whether items are transmitted with contractual restrictions, whether they are deleted after evaluation, or whether the evaluation is conducted by the organizers on behalf of the requesting team rather than by sharing the items directly.

The consequence is that during the embargo period, the benchmark operates in a regime closer to access-controlled benchmarks (Humanity's Last Exam, FrontierMath) than to the transparent, community-auditable resource that the paper envisions for its post-embargo future. The paper's central argument — that novelty-based contamination resistance is superior to permanent secrecy — is only fully realized after the embargo lifts. Before that point, the transparency benefits are aspirational rather than actual. The paper does not discuss this temporal asymmetry or offer guidance on how the transition from embargoed to public will be managed to prevent last-minute training-data harvesting immediately before release.

7. Implications and Future Directions

How This Work Changes the Landscape

SOOHAK does not propose a new model, training method, or architectural innovation — it reconfigures how the field thinks about evaluating frontier mathematical reasoning at the moment when existing benchmarks have lost discriminatory power. The paper's contribution is less a paradigm shift than an operational reframing of the benchmark construction problem: it demonstrates that four constraints previously addressed in isolation (scale, novelty, breadth, and metacognitive evaluation) can be simultaneously satisfied at an operational budget (~USD 550,000) and compressed timeline (four months), and it provides a transparent account of what that costs — not just financially, but in terms of organizational complexity, failure modes, and residual quality limitations.

The most consequential conceptual move is the elevation of refusal from a quality-control nuisance to a first-class evaluation dimension. Prior benchmarks treated ill-posed problems as defects to be eliminated during construction. SOOHAK repurposes them as probes for a capability — recognizing when a problem is broken — that the paper argues is intrinsic to research-level mathematics. This changes the evaluation framing from "can the model solve well-posed problems?" to "does the model apply its reasoning machinery discriminately, solving problems when they are solvable and diagnosing flaws when they are not?" The empirical finding that Refusal does not scale with model size or test-time compute in the same way that Challenge performance does (Figure 2; Qwen3-235B performs worst on Refusal at 2.69% despite being the largest variant, Table 5) establishes that these are genuinely decoupled capabilities — not simply different points on a single difficulty spectrum. This finding reorients the evaluation agenda: it implies that model developers tracking only reasoning accuracy are optimizing a partial metric, and that metacognitive caution is a separate axis requiring separate measurement and, presumably, separate training interventions.

The paper also substantially clarifies the open-weight vs. closed-model capability gap on novel material. Before SOOHAK, the field knew that closed models generally outperformed open-weight models on reasoning benchmarks, but the magnitude and nature of the gap on genuinely novel problems was poorly characterized. Table 2 quantifies a differential that widens dramatically with difficulty: 6.1 points on SOOHAK-Mini (72.22% vs. 66.07%) balloons to 16.5 points on Challenge (30.39% vs. 13.87%). The paper's contributor interviews (Appendix B.5) provide a plausible mechanism — problems drawing on "folklore-level reasoning" and material "absent from accessible papers, scattered across niche sources, or hidden behind paywalls" are substantially harder for models trained primarily on public data. This finding redirects attention from the question "are open-weight models catching up?" (the answer on SOOHAK-Mini is "partially") to the more diagnostic question "catching up on what kind of material?" — and the answer on research-adjacent mathematics is "not yet, by a wide margin."

The human baseline methodology introduces a diagnostic function for human evaluation that goes beyond the standard "model vs. human" scalar comparison. By deliberately constructing heterogeneous teams spanning IMO medalists, mathematics undergraduates, and PhD researchers, the paper reveals that the benchmark's format rewards contest-style reasoning under time pressure more than deep research expertise (Team B at 38.0% vs. Team E at 24.1%, Figure 3). This is not a benchmark failure — it is a construct validity finding. It tells users that SOOHAK Challenge scores should be interpreted as measuring "advanced contest-style reasoning on novel material under constraints" rather than "research-level mathematical expertise." The paper's willingness to design a human baseline that reveals this limitation, rather than one that simply positions humans above models, is methodologically instructive for the field.

The model-gated difficulty routing operationalizes "difficulty" as a property of the problem–model interaction rather than a property of the problem alone. This is a conceptual shift from human-assigned difficulty labels (as in MATH, FrontierMath) to a construct where difficulty is defined by which current models fail to solve the problem. The paper's gating architecture (Gate 1: small open models; Gate 2: mid-size open models; Gate 3: large open reasoning models, Appendix B.3) makes this operationalization explicit and reproducible. The implication is that benchmark difficulty is not static — it drifts as model capabilities improve, and benchmarks that aim to discriminate at the frontier must either periodically recalibrate their difficulty thresholds (using new gate models) or accept that headroom will diminish over time. The paper does not propose a recalibration mechanism, but the architecture transparently supports one.

The retrospective in Appendix F changes the incentive structure for future benchmark construction. By documenting what failed — incentive misalignment (rewarding raw difficulty produced some low-quality items), reviewer-hour constraints (not enough expert review early in the project), the structural exhaustion of unique-integer answer formats — the paper provides a public good that reduces the cost of future efforts. This is unusual in the benchmark literature, where most papers present their final dataset as a fait accompli without discussing the construction process's failure modes. The paper's recommendation that future builders invest in "early review infrastructure with explicit rubrics," "incentive schemes aligned with multiple notions of quality," and "evaluation formats that broaden the space of measurable mathematical competence" (Appendix F) is actionable guidance derived from specific negative experiences.

Research directions that become more attractive:

Metacognitive evaluation as a standard benchmarking practice. The Refusal subset demonstrates that it is feasible to measure epistemic caution alongside raw capability. The community should now expect new reasoning benchmarks to include a refusal or calibration dimension — not as an afterthought, but as a core design element. Benchmarks that report only accuracy on well-posed problems are measuring an incomplete capability profile.
Difficulty-as-interaction operationalization. The model-gated routing provides a template for benchmarks that define difficulty relative to current model capability rather than human judgment. This is particularly valuable in fast-moving domains (code generation, scientific reasoning) where human difficulty labels rapidly decouple from what models can and cannot do.
Transparent benchmark construction with documented failure modes. The paper's Appendix F sets a standard for candor that, if adopted by the field, would substantially improve the trustworthiness of benchmark scores and reduce the hidden costs of benchmark construction.

Research directions that become less attractive:

Scraping public contest archives for frontier evaluation. The paper's documentation of contamination vulnerabilities in contest-derived sets (citing Balunović et al., 2025) and the operational demonstration that expert-authored benchmarks at scale are feasible makes the cost–benefit calculus for public-source scraping less favorable — at least for benchmarks targeting frontier discrimination.
Single-score headline metrics for mathematical reasoning. The paper's multi-dimensional design (Challenge + Refusal + Mini, with composite scores in Appendix D.4) demonstrates that model rankings shift substantially depending on which dimensions are weighted (GLM-5 rises 3 ranks, Kimi-2.5 drops 3 ranks when Refusal is incorporated, Figure 4). Reporting a single aggregate score would obscure these tradeoffs; the field should move toward multi-dimensional reporting.

Follow-Up Research This Work Enables

Calibrating refusal: measuring false-positive refusal rates on well-posed problems. The Refusal grading logic (Appendix B.8) measures false negatives — the failure to refuse on ill-posed problems — but provides no measurement of false positives — refusing to answer well-posed problems. A model could score 100% on Refusal by refusing to answer every question, and the current benchmark would not detect this. A direct follow-up would evaluate the same model panel on the Challenge and SOOHAK-Mini splits with a refusal-detection pass: for each model response to a well-posed problem, classify whether the model refuses (declines to produce a numeric answer) or attempts a solution. The resulting 2×2 confusion matrix per model (refuse/attempt × problem is well-posed/ill-posed) would distinguish genuinely discriminating models (high refusal on Refusal, low refusal on Challenge) from uniformly cautious models (high refusal everywhere) and uniformly overconfident models (low refusal everywhere, including on Refusal). The paper's Qwen3 results — 2.69% on Refusal, 8.04% on Challenge (Table 2) — already suggest Qwen3-235B is in the uniformly overconfident category, but this is an inference rather than a direct measurement. A study that adds per-item refusal annotations to the reasoning splits would close the most significant measurement validity gap in the current benchmark.

Training interventions targeting refusal capability and measuring transfer effects. The paper's finding that Refusal does not scale with model size or test-time compute (Figure 2, Tables 5–6) while Challenge does suggests that current post-training recipes do not optimize for epistemic caution. A natural experiment: take a base model with known Refusal performance (e.g., Qwen3-235B at 2.69% on Refusal, Qwen3-32B at 28.28%, Table 5) and apply a targeted fine-tuning intervention — for example, training on a dataset of well-posed and ill-posed mathematical problems with explicit "refuse" labels for the ill-posed ones, or using RLHF with a reward that penalizes confident answers to ill-posed problems. Then measure: (a) Refusal improvement on SOOHAK, (b) whether Challenge performance degrades (does teaching caution impair reasoning?), (c) whether the improvement generalizes to held-out ill-posed problems from a different mathematical domain. The large gap between Qwen3-235B (2.69%) and GLM-5 (49.49%) on Refusal — both open-weight models with non-trivial Challenge performance — establishes that refusal is trainable in principle (since GLM-5 achieves it), and the Qwen3 family's non-monotonic refusal scaling suggests that current training recipes may actively suppress it. A positive result (refusal improves without reasoning degradation) would provide a recipe for building more epistemically cautious models; a negative result (refusal and reasoning trade off) would reveal a fundamental tension with implications for deployment safety.

Continuous recalibration of the gating infrastructure as model capabilities evolve. The paper's model-gated routing uses a fixed set of gate models (Qwen3-8B, GPT-OSS-20B, DeepSeek-R1, etc.) frozen at the collection time. As capabilities improve, the gates' failure thresholds drift: problems that DeepSeek-R1 failed in 2025 may be solved by 2027 models, meaning they would route differently if the benchmark were reconstructed. A follow-up study would periodically re-evaluate the gating pipeline with updated baseline models and report: (a) what fraction of current Challenge items would be reclassified as Mini (or rejected) under 2027 gate models, (b) whether the benchmark's discrimination at the frontier degrades predictably over time, and (c) whether a versioning scheme — SOOHAK-2025, SOOHAK-2027, each with re-gated splits — maintains headroom more effectively than a static benchmark. This is not hypothetical: the paper reports that 124 Challenge items are unsolved by any evaluated model (Section 5), but this number will decrease as models improve, and a recalibration study would quantify the rate of headroom erosion and inform decisions about when to invest in a new benchmark version versus continuing to use the existing one.

Cross-domain replication of the refusal scaling finding. The paper's finding that Refusal does not scale with model size or test-time compute is based on a single mathematical reasoning benchmark with a specific refusal grading rubric. It is unknown whether this pattern generalizes to other domains where ill-posedness takes different forms. A cross-domain replication would construct analogous refusal subsets for: (a) code generation (ill-posed specifications with contradictory requirements, missing constraints, or ambiguous outputs), (b) scientific question answering (questions with false premises or insufficient information), (c) formal theorem proving (invalid theorem statements). For each domain, evaluate the same model families (Qwen3 parameter spectrum, GPT-OSS with varied reasoning effort) and test whether the non-scaling pattern replicates. If refusal scales linearly in code generation but not in mathematics, that would reveal domain-specific features of training data that encode well-posedness priors. If refusal fails to scale across all domains, that would strengthen the paper's implicit claim that current training paradigms structurally suppress epistemic caution. SOOHAK provides the template (refusal split sourced from rejected submissions, binary diagnostic grading) that makes these replications feasible.

Proof-assistant-based evaluation to escape the unique-integer answer bottleneck. The paper's retrospective (Appendix F) identifies "the 'unique integer answer' format is becoming exhausted" as a structural limitation: it restricts the space of feasible problems, systematically favors subfields where clean numeric end-answers are natural, and cannot evaluate proof-based reasoning. A concrete extension: select a subset of SOOHAK Challenge problems amenable to formalization (e.g., problems in algebra, number theory, or combinatorics with constructive solutions), encode them in Lean or Coq, and evaluate models on proof generation rather than final-answer prediction. The evaluation metric shifts from "does the final answer match?" to "does the generated proof compile and close the goal?" This would address the format bias the paper identifies (the benchmark rewards contest-style short-path solutions rather than deep reasoning) and would probe a capability — formal proof construction — that is closer to research mathematics than unique-integer problems. The paper's transparent documentation of subfield coverage (Table 1, Table 7) provides a map for which subfields are most amenable to formalization (algebraic subfields with constructive solution patterns) and which are least (analysis, geometry). The main challenge is the formalization cost — translating human-authored problems and solutions into proof-assistant code — which scales with the number of problems formalized; a pilot on 25–50 Challenge items would establish feasibility.

Per-language evaluation to leverage the bilingual construction. The paper invested substantially in a bilingual translation pipeline (Appendix B.7) that produces English and Korean versions of every item with verified mathematical equivalence, but reports no per-language results. A straightforward follow-up study would evaluate the same model panel on the English and Korean versions separately and report: (a) the per-language score gap for each model (do models perform differently on the same mathematics expressed in different languages?), (b) whether the gap correlates with the model's reported multilingual capability, (c) whether the gap is larger on Challenge than on SOOHAK-Mini (suggesting that advanced mathematical reasoning degrades more under language shift than routine problem solving), (d) whether the refusal gap varies by language (do models become more or less cautious about ill-posed problems when the prompt is in Korean vs. English?). The paper's bilingual infrastructure makes this experiment essentially zero-cost — the translated items already exist, and the evaluation pipeline supports both languages. The findings would inform whether multilingual mathematical benchmarks need per-language leaderboards, whether cross-lingual transfer of mathematical reasoning is a distinct capability worth tracking, and whether the current practice of reporting only English scores on bilingual benchmarks is masking meaningful variation.

Practical Applications and Downstream Use Cases

Stratified model selection for research-mathematics applications. An organization deploying LLMs for mathematical research assistance (e.g., a university math department, a theorem-proving research group, or an AI-for-science initiative) can use SOOHAK's multi-dimensional scores to select models based on their specific risk tolerance. The paper's finding that GPT-5 achieves the highest Avg-R composite (Figure 4) despite not leading on raw reasoning capability means that for applications where confidently wrong answers to ill-posed problems are costly (e.g., suggesting a flawed conjecture as a lemma for a proof), GPT-5's balanced capability–caution profile is preferable to Gemini-3-Pro's higher raw reasoning but lower relative refusal performance. Conversely, for applications where all problems are verified before use (e.g., generating candidate solutions that human mathematicians will check), Gemini-3-Pro's 30.39% Challenge Avg@3 (Table 2) may be more valuable than GPT-5's 26.37%, and the refusal dimension matters less. The per-MSC subfield leaderboard (Table 7) further enables subfield-specific model selection: a combinatorialist might prefer Gemini-3-Pro (leads MSC 5, combinatorics, at 51.9%), while a probabilist might prefer Grok-4.1-Fast (leads MSC 60, probability, at 59.7%). The benchmark's granularity transforms model selection from a single-score decision to a domain-informed risk-management decision.

Detecting and monitoring overconfidence regressions in model deployments. The paper's finding that the largest Qwen3 variant (235B) regresses dramatically on Refusal (2.69%, Table 5) while the mid-size variant (32B) scores 28.28% establishes that scaling model size can reduce epistemic caution. For organizations deploying LLMs in high-stakes mathematical settings — automated grading, mathematical proof checking, symbolic computation assistance — this creates a monitoring need. SOOHAK's Refusal subset provides a standardized instrument for regression testing: before deploying a new model checkpoint or a fine-tuned variant, evaluate Refusal performance alongside standard accuracy benchmarks. A drop in Refusal without a corresponding gain in reasoning accuracy signals that the new checkpoint has become more overconfident — it will produce confident answers to a broader set of ill-posed inputs — even if its accuracy on well-posed problems improves. The paper's finding that GPT-OSS-120B's Refusal drops from 60.61% to 52.53% when reasoning effort increases from medium to hard (Table 6) provides a concrete example of a capability–caution tradeoff that would be invisible in standard accuracy-only evaluation. Organizations can set Refusal thresholds based on their risk tolerance and use SOOHAK for ongoing calibration.

Training-data novelty auditing for open-weight model development. The paper's finding that the open-weight–closed gap widens from 6.1 points on SOOHAK-Mini to 16.5 points on Challenge (Table 2) provides a diagnostic signal for model developers. If an open-weight model's relative performance degrades sharply on Challenge compared to SOOHAK-Mini (as seen for Qwen3-235B, which drops from 56.22% to 8.04%), this suggests that the model's training data contains substantial coverage of contest-level and textbook-style mathematics but limited coverage of the research-adjacent, folklore, and unpublished material that Challenge targets. Developers can use this differential as a "data coverage stress test": a model whose SOOHAK-Mini-to-Challenge ratio is significantly worse than frontier closed models likely has gaps in training-data representation of advanced mathematics. The paper's contributor interviews (Appendix B.5) — which found that problems drawing on "folklore-level reasoning" and material from "niche research papers" were substantially harder for models — provide qualitative support for this interpretation. Developers can respond by targeted data acquisition: licensing mathematical literature, partnering with researchers to source unpublished problem sets, or using the benchmark's keyword distributions (Section 3.5) to identify underrepresented subfields.

Calibrating inference-compute allocation in mathematical reasoning pipelines. The test-time scaling results (Appendix D.3, Table 6) demonstrate that extending context budgets and reasoning effort yields larger gains on Challenge than on SOOHAK-Mini: for GPT-OSS-120B, moving from medium to hard reasoning improves Challenge by 7.94 points but SOOHAK-Mini by only 1.57 points. This provides an empirical basis for dynamic compute allocation: given a test-time compute budget, spend more tokens on problems that a difficulty classifier (trained on the benchmark's difficulty distributions, as the paper suggests in Section 5's discussion of future work) flags as Challenge-like, and spend fewer on Mini-like problems where additional reasoning yields diminishing returns. The paper's finding that Refusal does not benefit from test-time scaling (GPT-OSS-120B drops on hard reasoning, Qwen3-235B nearly doubles but remains at 9.09%, Table 6) adds a further dimension: if the goal includes appropriate refusal behavior, simply increasing the reasoning budget may be counterproductive. A pipeline that routes problems through difficulty estimation, then allocates compute based on predicted difficulty category (Mini: default budget; Challenge: extended budget; ambiguous/ill-posed: conservative budget to avoid overconfident answers) would operationalize the paper's insights about the non-uniform value of test-time compute across problem types.