ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judge

🎯 Pitch

Even GPT‑5‑high reaches only 65.9% when generating professional‑quality reports grounded in PhD‑level documents, yet the paper shows that a carefully debiased open‑weight judge can match proprietary evaluators at 1.68% of the cost. The real surprise is that fair automated assessment of open‑ended professional tasks is impossible without explicitly measuring and neutralizing each model's tendency to inflate its own scores.

1. Executive Summary

This paper introduces ProfBench, a rubric-based benchmark of over 7,000 expert-written response-criterion pairs across four professional domains — Physics PhD, Chemistry PhD, Finance MBA, and Consulting MBA — designed to evaluate LLMs on open-ended, real-world tasks requiring multi-page report generation grounded in provided documents. The paper studies two complementary evaluation axes: benchmarking models as LLM-Judges (binary verification of whether a response fulfills individual expert-written criteria, formulated as natural language inference) and benchmarking models as Report-Generators (producing lengthy, document-grounded responses scored by a selected judge against rubric criteria weighted by importance). A top-performing model like GPT-5-high achieves only 65.9% overall performance as a report-generator, confirming the benchmark's difficulty relative to saturated evaluations like AIME 25 (94.6%), while the authors develop a bias-mitigated, cost-optimized LLM-Judge — assembling a composite GPT-OSS-120B judge that adapts reasoning effort by domain and criterion type — matching the best proprietary judge (Gemini-2.5-Pro at 78.2% overall) at only 1.68% of the cost ( $0.70 vs.$ 1,320), and further reduce generation cost 4× through dynamic programming allocation of responses per task without compromising estimation robustness, establishing that fair, accessible professional-domain evaluation is achievable only when judge self-enhancement bias is explicitly measured and budget-aware allocation accounts for per-task variance heterogeneity.

2. Context and Motivation

The Core Problem: Verification Defines the Boundaries of What We Can Evaluate

The central tension this paper confronts is deceptively simple: the ease of verifying an answer should not dictate what kinds of problems we ask language models to solve, yet in practice, it does. The authors articulate this in their opening analogy (Section 1): solving a Sudoku puzzle is hard, but verifying a completed grid is trivial — check that no digit repeats in any row, column, or square. This asymmetry between generation difficulty and verification difficulty has been enormously productive for AI, enabling Reinforcement Learning with Verified Rewards (RLVR; Lambert et al., 2025; DeepSeek-AI et al., 2025) to drive rapid progress on tasks where correctness can be mechanically checked.

But the paper argues that this very productivity creates a selection bias in what the field measures and optimizes for. Verification-cheap tasks — competition math with unique numeric answers (AIME 25), competitive programming with unit tests (LiveCodeBench), instruction following with programmatic checks (IFBench), or multiple-choice science questions (MMLU-Pro, GPQA) — have become the de facto standard for evaluating LLM capabilities. These benchmarks usefully track progress along particular dimensions, but they collectively paint an incomplete picture of what "capable" means for a language model.

The gap the paper identifies is this: many real-world professional tasks that deliver genuine value — synthesizing financial analyses, writing research reports, evaluating M&A opportunities, proposing experimental designs — produce open-ended outputs with no unique correct answer and no mechanical verification procedure. A financial analyst's recommendation memo for a potential acquisition is not "correct" or "incorrect" in the way a math answer is; it succeeds to the extent that it addresses specific analytical requirements, incorporates the right data, applies sound reasoning, and communicates clearly. Evaluating such outputs requires substantive judgment that decomposes quality into multiple independently assessable criteria — precisely what professional rubrics provide.

The authors frame this concretely by noting that while exam-style questions "might predict how well PhD candidates (or other early-stage domain-experts) understand their field, people do not graduate from PhD programs (or become true-experts) simply based on how well they can pass these 'exam-style' questions." Real expertise manifests in the ability to produce original, valuable work — designing experiments, synthesizing disparate information, generating actionable recommendations — and benchmarks limited to closed-form answers cannot assess these capacities.

Why This Matters: The Real-World Deployment Gap

The practical stakes here are substantial and the paper articulates them through several interconnected arguments:

RLVR's applicability is bottlenecked by verifiability. If reinforcement learning from verified rewards drives much of the recent progress in LLM reasoning (as evidenced by DeepSeek-R1, o3, and similar models), then the set of tasks for which we can construct verified rewards effectively defines the frontier of what we can expect models to improve on through RL. Expanding that frontier requires expanding the class of verifiable tasks — and rubric-based evaluation represents a promising avenue for doing so because it decomposes complex, open-ended quality judgments into binary criteria that can (potentially) be automated. The paper positions itself as infrastructure for this expansion: a benchmark that demonstrates rubric-based evaluation is viable across multiple professional domains, combined with a cost-effective automated judge that makes iterative experimentation practical.

Professional tasks constitute a large fraction of economically valuable LLM applications. Consulting analyses, financial due diligence, scientific literature synthesis, and technical report generation are precisely the types of tasks that organizations would most like to automate or augment with LLMs. Yet the field's evaluation infrastructure — dominated by math competitions, coding challenges, and multiple-choice exams — provides limited signal about model readiness for these applications. A model that scores 95% on AIME 25 might still produce unusable consulting deliverables. ProfBench aims to fill this measurement gap directly.

The training-inference compute question extends to task formulation. The paper's broader argument, though not explicitly framed in these terms, connects to questions about how the ML community allocates its collective effort. If verification-cheap tasks dominate evaluation, then model developers optimize for those tasks — potentially at the expense of capabilities that matter more for deployment. Rubric-based evaluation, even when more expensive than programmatic checking, may be necessary to align optimization pressure with real-world utility. The authors' emphasis on reducing evaluation cost (to $0.70 for the full judge, down from thousands in prior work) is thus not merely a convenience feature but a strategic choice: cheap evaluation enables fast iteration, and without iteration, progress stalls.

Prior Approaches and Their Limitations

The paper situates itself relative to three classes of prior work, identifying concrete weaknesses in each.

Rubric-based benchmarks (PaperBench, HealthBench). Starace et al. (2025) demonstrated that human-written rubrics with LLM-judge scoring can evaluate open-ended professional tasks — specifically, reproducing ICML papers, with 8,316 response-criterion pairs across 20 tasks. Arora et al. (2025) applied a similar methodology to patient-physician conversations in healthcare, with 8,053 response-criterion pairs across hundreds of tasks. These works established the rubric paradigm's viability in specialized domains. However, the paper identifies a critical gap:

"there currently is no robust publicly available benchmark that makes use of rubric fulfillment across diverse professional domains beyond health and machine learning."

PaperBench covers a single domain (ML research reproduction) and HealthBench covers healthcare. Neither spans the range of professional expertise — physical sciences, business strategy, financial analysis — that would demonstrate generalizability. More practically, both are expensive to run: PaperBench's JudgeEval costs approximately $1,320 and HealthBench evaluation reaches$ 300 per round with o3 (Section 5.1), making iterative experimentation prohibitively costly for many researchers. The paper explicitly targets this cost barrier.

DeepResearch-Bench RACE (Du et al., 2025). This benchmark claims to cover PhD-level tasks across 22 domains with 100 total tasks, but the authors identify fundamental limitations:

First, the criteria (rubrics) are synthetically generated rather than human-written — and specifically, generated by Gemini-2.5-Pro, the same model family that serves as a reference. The authors note this creates a pervasive bias:

"there is a pervasive bias towards Gemini-2.5-Pro since the criteria and the reference 'high-quality' answer are both generated by this model. This results in Gemini-2.5-Pro being rated as by far the best performing—reaching more than 48.5 out of a maximum of 50 on each of the 4 axes, meaning >>97%."

When a model writes its own evaluation criteria, those criteria naturally align with its own output patterns — what the model considers "good" will be what it tends to produce. This makes the benchmark a measure of self-consistency rather than of task quality as judged by domain experts. The paper positions human-written, human-verified rubrics as the essential corrective.

Second, the tasks themselves lack domain specificity:

"many examples such as 'What are the investment philosophies of Duan Yongping, Warren Buffett, and Charlie Munger?' or 'How did Netflix manage to successfully adapt One Hundred Years of Solitude...' can be answered by an educated generalist (i.e., college graduate or equivalent experience) with a few straightforward internet searches."

The contrast with ProfBench is stark: ProfBench tasks require analyzing specific financial statements, evaluating deal structures, designing experimental protocols, or synthesizing scientific literature — tasks that demand professional training and cannot be adequately answered through general knowledge plus web search.

Third, with only 100 tasks across 22 domains (equally split between Chinese and English), DeepResearch-Bench provides an average of approximately 2.3 English tasks per domain — too few for reliable performance estimation within any single domain.

Standard benchmarks (MMLU-Pro, GPQA, HLE, AIME 25). These evaluate knowledge and reasoning through exam-style questions with unique correct answers. While they have driven progress and established important baselines, the authors locate their limitation in task format rather than difficulty:

"While some of these tests might predict how well PhD candidates... understand their field, people do not graduate from PhD programs... simply based on how well they can pass these 'exam-style' questions."

The gap is between demonstrating knowledge (answering a closed-form question correctly) and applying knowledge productively (producing a coherent, multi-faceted analysis that addresses a real professional need). The paper argues that rubric-based evaluation is necessary to bridge this gap, because open-ended outputs require multi-criteria quality assessment rather than binary correctness judgments.

Self-Enhancement Bias and Cost as First-Class Concerns

A distinctive contribution of the paper's motivation is its treatment of evaluation infrastructure quality — bias and cost — not as secondary implementation details but as first-class design constraints that determine whether a benchmark can fulfill its purpose.

Self-enhancement bias (Zheng et al., 2023) is the well-documented tendency of LLMs to rate their own outputs (or outputs from the same model family) more favorably than those from other models. The paper formalizes this concern into a measurable bias-index: the range of per-model bias (average difference between judge-predicted and human-annotated criterion-fulfillment) across the three response-generating models (o3, Grok4, R1-0528). If an LLM-Judge assigns systematically higher scores to o3 responses relative to human judgments, the benchmark's rankings become provider-biased rather than quality-reflective. The paper treats mitigation of this bias as a core evaluation metric — the overall judge quality is defined as Macro-F1 minus Bias-Index — asserting that benchmark fairness requires explicit attention to judge impartiality, not just judge accuracy.

This is a substantive departure from prior rubric-based benchmarks (PaperBench, HealthBench) which did not quantify or optimize for judge bias across response providers. The paper argues this is essential because "LLMs are known to have self-enhancement bias," and without measuring it, "fairness to various model responses" cannot be assessed.

Cost is similarly elevated to a central concern. The paper calculates upper-bound judge costs from public API pricing and reports them alongside performance metrics, explicitly treating cost as a tiebreaker between comparably-performing judges. The rationale is that "the cheaper judge can be accessible for more" — accessibility is not merely about convenience but about enabling the broader research community to iterate rapidly. When PaperBench's judge costs $1,320 and even HealthBench's costs$ 300 per evaluation round, only well-resourced labs can afford systematic experimentation. ProfBench targets costs two to three orders of magnitude lower to democratize access.

How This Paper Positions Itself

The paper positions ProfBench as filling three simultaneously unaddressed gaps:

Domain diversity: Existing rubric benchmarks cover single domains (ML, health); ProfBench spans four — two scientific (Physics, Chemistry PhD) and two business (Finance, Consulting MBA) — with tasks explicitly designed to require domain-specific professional knowledge rather than generalist reasoning plus web search.
Human-authored, human-verified rubrics: Unlike synthetically-generated criteria (DeepResearch-Bench), ProfBench's ~7,300 criteria are written by domain-expert annotators (44.7% PhD holders, 18.4% MBA holders, with an average of 5.24 years post-degree experience) and reviewed through an iterative feedback process where 41.4% of criteria were flagged for improvement before approval.
Fairness and accessibility as design goals: The benchmark includes explicit bias measurement and mitigation (through the Bias-Index and composite judge construction) and cost optimization (through adaptive reasoning effort, optimal sample allocation via dynamic programming, and selection of cost-efficient open-weight judges) — treating these not as afterthoughts but as integral to the benchmark's claims of practical utility.

The paper does not claim to introduce a fundamentally new evaluation paradigm — it explicitly builds on the rubric-based framework established by PaperBench and HealthBench, the LLM-as-judge methodology from Zheng et al. (2023), and the reasoning-effort-adaptation idea from Jung et al. (2025). Its contribution is in the systematic synthesis: bringing these elements together across multiple professional domains, measuring and mitigating their failure modes (bias, cost, variance), and demonstrating that the result is both challenging for frontier models and accessible to the broader community.

3. Technical Approach

3.1 Reader Orientation

ProfBench is not a single system but rather a benchmarking framework comprising two interconnected evaluation pipelines: one for assessing how well LLMs can judge whether a response satisfies an expert-written criterion (the LLM-Judge axis), and another for assessing how well LLMs can generate professional-quality, document-grounded reports that satisfy multiple such criteria (the Report-Generator axis). The core problem it solves is that existing benchmarks either cannot evaluate open-ended professional outputs at all (because they rely on unique correct answers for mechanical verification) or can only do so at prohibitive cost with undocumented bias toward specific model providers. The "shape" of the solution is a rubric-based evaluation paradigm — human experts decompose complex task quality into independently gradable binary criteria, human annotators label whether specific model responses satisfy each criterion to establish ground truth, and a carefully selected LLM-Judge automates the criterion-checking at scale with explicit measurement and mitigation of self-enhancement bias and a cost two to three orders of magnitude below prior work.

3.2 Big-Picture Architecture (Diagram in Words)

The ProfBench framework has five major components, organized into a data creation pipeline and two evaluation pipelines:

Expert-Curated Task and Rubric Data (the core artifact): 80 professional tasks (20 per domain: Physics PhD, Chemistry PhD, Finance MBA, Consulting MBA), each accompanied by 15–60 independently gradable criteria written and reviewed by domain-expert annotators. Each criterion specifies what a good response must include, is assigned an importance weight (Critical, Major, Minor, Additional), and receives binary fulfillment labels (Yes/No) from human annotators evaluating three reference model responses (o3, Grok4, R1-0528). This produces over 7,000 response-criterion pairs with ground-truth human judgments.
LLM-Judge Evaluation Pipeline (Section 4): Given a model response and a criterion, an LLM is prompted to output a binary judgment ("Yes" or "No") indicating whether the response satisfies that criterion. The pipeline evaluates over 40 models as judges, measuring (a) agreement with human annotations via Macro-F1, (b) bias toward specific response providers via a Bias-Index that captures the range of per-model score inflation/deflation relative to human ground truth, and (c) cost based on public API pricing. The pipeline produces a composite judge — GPT-OSS-120B with domain- and criterion-type-dependent reasoning effort — that matches the best proprietary judge while costing 1.68% as much.
Report-Generator Evaluation Pipeline (Section 5): Given a task prompt and grounding documents (truncated PDFs providing necessary factual context), a model generates a multi-page report. The selected composite LLM-Judge then evaluates the response against all criteria for that task, producing per-criterion binary judgments. These are aggregated into a weighted score using the importance weights assigned by annotators. Performance is estimated by averaging over 16 (or fewer, via optimal allocation) independent generations per task to quantify variance.
Cost and Variance Optimization (Sections 4.2, 5.1, Appendix G): An integer programming formulation solved via dynamic programming allocates generation budgets non-uniformly across tasks based on per-task variance estimates, enabling a 4× reduction in inference cost (from 16 to 4 average generations per task) without increasing estimator variance beyond acceptable levels.
Ablation Infrastructure (Section 6): A controlled experimental setup that removes grounding documents and/or adds web search capability to isolate the contribution of provided reference materials to model performance.

Information flows as follows: domain experts create tasks and rubrics → reference models generate responses → human annotators label criterion fulfillment → the labeled data serves as ground truth for both judge evaluation (comparing judge predictions to human labels) and for validating the composite judge's scoring against human-assigned scores for the three reference models → the validated judge evaluates new model-generated responses → per-criterion judgments are aggregated by weighted averaging → variance and cost are managed through optimal sample allocation.

3.3 Roadmap for the Deep Dive

First, the rubric creation and annotation pipeline: how tasks and criteria are designed, what quality controls ensure reliability, and how ground-truth labels are established. This is foundational because all downstream evaluation depends on the rubric quality and annotation reliability.
Second, the LLM-Judge task formulation and evaluation protocol: how criterion-checking is framed as binary natural language inference, what metrics capture both accuracy and fairness, and how the Bias-Index is defined and computed. This is essential for understanding what makes one judge better than another.
Third, the composite judge construction: how the authors select a specific model and adapt reasoning effort by domain and criterion type to simultaneously maximize agreement with humans, minimize bias, and minimize cost. This is the practical mechanism that makes the benchmark accessible.
Fourth, the Report-Generator evaluation protocol: how models are prompted with documents, how responses are scored by the composite judge, how importance-weighted aggregation works, and why the scoring schema is validated against human-annotated performance.
Fifth, the optimal allocation formulation for variance reduction: the integer programming objective, the dynamic programming solution, and how it achieves cost reduction without compromising estimation robustness.
Sixth, the ablation design for grounding documents: how the contribution of provided reference materials is isolated through controlled removal and web-search substitution.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmark construction and systematic evaluation paper whose core idea is that evaluating LLMs on open-ended professional tasks requires (1) expert-written, human-verified rubrics as ground truth, (2) explicit measurement and mitigation of LLM-Judge self-enhancement bias, and (3) cost optimization to make iterative experimentation practical for the broader community.

Rubric Creation and Annotation Pipeline

The foundation of ProfBench is a set of 80 professional tasks, each decomposed into 15–60 independently gradable criteria, with human annotators providing binary fulfillment labels for three reference model responses. The pipeline operates in four sequential stages, each with reviewer oversight and quality gates.

Annotator recruitment and qualification. The paper recruits 38 annotators from 8 countries, managed by a vendor, with specific qualification requirements per domain (Appendix C). For Consulting MBA tasks, annotators must have at least 2 years of work experience at top-tier consulting firms (McKinsey, BCG, Bain, Deloitte, PwC, EY, KPMG) or 4 years at another consulting firm. For Finance MBA, annotators need 2 years of experience at select investment banks (JPMorgan, Bank of America, CitiGroup, Wells Fargo, Goldman Sachs, Morgan Stanley, and peers). For Chemistry and Physics PhD, annotators must have completed a PhD at a Global Top 100 university with program selectivity factored in. Candidates undergo domain expertise testing and task-understanding evaluation before inclusion. Across the pool, 44.7% hold PhDs, 18.4% hold MBAs, and the remainder hold related bachelor's degrees with work experience; the average post-degree experience is 5.24 years. Annotators are paid "well-above minimum-wage following local standards, with hourly pay often exceeding full-time employment hourly pay in these professional fields to attract the most qualified annotators." Each annotator works on entire tasks end-to-end (prompt ideation through response annotation) to leverage personal expertise, and is limited to at most 5 tasks to ensure prompt diversity. Use of LLMs is disallowed at any annotation stage.

Prompt curation. Annotators create tasks that a professional might assign to a junior colleague, with the expectation of a "multi-page report" as output. The explicit instruction is to target difficulty that would challenge frontier LLMs (at the time of July 2025 collection: o3, Grok4, DeepSeek R1-0528). All prompts must be in English, text-only, and based only on publicly available documents (no proprietary sources). Annotators provide a rationale for how they conceived each prompt. The resulting prompts average 2,052.4 characters (standard deviation 997.4, range 696–5,339), typically containing 2–3 paragraphs with multiple related sub-questions. Chemistry and Physics tasks are shorter on average (1,617.4 and 1,799.2 characters respectively) while Consulting and Finance tasks are longer (2,360.8 and 2,432.2 characters respectively), reflecting the more open-ended nature of business analysis questions. Prompts are substantially more specific than those in comparable benchmarks — for example, a Finance MBA task (Figure 1) asks the model to "determine if Digital Realty Trust's management is effectively delivering shareholder value" by analyzing specific financial metrics including NOI margin, leverage, cost of capital, and FFO per share growth, requiring detailed quantitative work with provided financial statements.

Rubric creation. For each task, annotators write 15–60 criteria designed to be independently usable for grading (each criterion must make sense without reference to other criteria) while collectively capturing all aspects of response quality. The total across all 80 tasks is 7,347 response-criterion pairs. Each criterion includes a description, a justification (to encourage thoughtful creation), an importance weight at four levels — Critical (4 points), Major (3 points), Minor (2 points), Additional (1 point) — and one or more criterion types. The distribution across weights is roughly half Major (49.8%), with Critical (23.4%) and Minor (23.9%) each accounting for about a quarter, and Additional at 2.9%. By criterion type, Reasoning dominates at 62.9% of all criteria (assessing logical validity, causal/mathematical correctness, completeness of reasoning steps, and generalization/abstraction), followed by Extraction/Recall at 34.1% (assessing accurate retrieval of information with correct coverage and granularity), and Style at only 3.0% (formatting, clarity, conciseness, and tone).

A reviewer with substantial domain-specific task creation expertise provides feedback on each set of criteria, recommending whether to keep or improve each criterion. The paper reports that "41.4% are marked as needing improvement at various stages, indicating the high quality standard that we set for when reviewing this data." This iterative review process, with possible multiple back-and-forth rounds, is the key mechanism ensuring rubric quality: rather than accepting annotator criteria at face value, the benchmark requires domain-expert reviewer approval for every criterion.

Response annotation and ground-truth establishment. The vendor generates responses to each task using three models selected as best-performing at the time of collection (July 2025): OpenAI o3, xAI Grok4, and DeepSeek R1-0528, representing a mix of proprietary and open-weight models. Generation uses temperature 0.2 (except o3, which does not allow temperature setting and uses medium reasoning effort), with web search access (native for o3 and Grok4; SearXNG plus document upload for R1-0528). Responses were generated with the original un-truncated documents.

Annotators then score each of the three responses on each criterion with a binary Yes or No judgment, accompanied by a brief justification. The overall fulfillment rates across the three models establish a performance baseline: o3 fulfills 51.6% of criteria, Grok4 fulfills 47.4%, and R1-0528 fulfills 45.2%. By domain, criterion fulfillment averages 39.1% in Finance and 40.3% in Physics (harder domains) versus 56.5% in Consulting and 59.4% in Chemistry (relatively easier). These human annotations serve as the ground truth for both the LLM-Judge evaluation (comparing judge predictions to human labels) and for validating the composite judge's scoring schema.

Ground-truth reliability validation. To verify annotation quality, the authors conduct a validation experiment on 1,127 response-criterion pairs: two additional annotators with the same domain expertise independently re-annotate the same response-criterion pairs. Inter-annotator agreement is measured using Fleiss' κ (a chance-corrected agreement metric for multiple raters), yielding κ = 0.912. The authors characterize this as "excellent agreement among annotators," establishing that the ground-truth labels are highly reliable and that binary criterion-fulfillment judgments are well-defined enough for consistent human application. This is a critical validation step: if expert annotators could not agree on whether a response satisfies a criterion, the entire rubric-based evaluation paradigm would be undermined at the ground-truth level before even considering automated judging.

Grounding document curation. Annotators initially identified publicly available CSV or PDF documents supporting task answers, but initial documents were impractically large — some reaching 838 pages, with 200–400 page documents being "relatively common" and an average of 42.3 pages per file and 141.7 pages per task. Since many LLMs at the time had context windows limited to 128K–200K tokens (OpenAI o3/o4-mini, Anthropic Claude-Sonnet 4, and many open-weight models), these documents could not fit into context. The authors therefore had annotators perform a truncation step: each PDF is trimmed to at most 20 pages and each CSV to at most 100 rows and 10 columns, with annotators selecting the most relevant information. Post-truncation, files average 7.37 pages (standard deviation 3.10, maximum 15), and tasks average 24.69 pages total (standard deviation 19.28, maximum 85, minimum 3). The authors acknowledge this "will make tasks substantially easier" but frame it as "a necessary step in order to make benchmarking many current models feasible," with the intention to use original documents when context lengths increase. An additional constraint is imposed by provider API limits: each file must be under 10 MB, each task can include at most 10 files and 30 MB total, and CSV files (only 4 in total across ProfBench) are converted to PDF format using pandas and pdfkit (JazzCore, 2025) with table headers repeating on every page. Each task averages 3.35 files (standard deviation 2.71, maximum 10), with average file size 0.813 MB and average task size 2.723 MB.

Design choice: end-to-end annotator ownership. Each annotator works through all stages for their assigned tasks — prompt creation, rubric writing, and response annotation — rather than specializing in one stage. This is justified by the claim that annotators are "asked to create tasks that capture their personal expertise." The reasoning is that domain expertise is needed not just for creating a realistic prompt but also for judging what constitutes a good answer to that prompt; having the same person perform both ensures coherence between what is asked and how it is evaluated. The trade-off is that this limits annotator throughput (each task takes 10–20 hours) but increases quality.

Design choice: disallowing LLM assistance. The paper explicitly states that annotators cannot use LLMs at any stage. This is motivated by the critique of DeepResearch-Bench, where LLM-generated criteria created systematic bias toward the generating model. Human-only annotation avoids circularity where evaluation criteria encode the preferences of a specific model that will later be evaluated against those same criteria.

LLM-Judge Task Formulation and Evaluation Protocol

The LLM-Judge pipeline evaluates how well a language model can perform the binary criterion-checking task that the rubric-based paradigm depends on. The task is formulated as two-class natural language inference (NLI) / recognizing textual entailment (following Bowman et al., 2015): given a model response (the "premise") and a criterion description (the "hypothesis"), the judge must determine whether the response entails criterion fulfillment (Yes) or does not (No).

Prompt template. After experimentation with alternatives, the paper settles on a minimal template:

Response: <response>

Evaluate whether the response above satisfies this criterion: <criterion_description>. Only answer Yes or No.

They tested alternatives including "True/False" output (2.2% worse) and "1/0" output (4.5% worse), as well as more elaborate templates with role descriptions and reasoning instructions (no gains, substantially increased token cost). The authors hypothesize that Yes/No is most effective because these tokens "are more commonly used in natural language compared to others" and that reasoning LLMs can still perform reasoning in their thinking traces without explicit prompting. The paper explicitly does not provide the original task prompt to the LLM-Judge, following PaperBench's practice, because "the criteria are designed to be used independently of the prompt, and further providing the prompt might confuse the LLM-Judge."

Inference configuration. The paper uses different generation parameters for reasoning versus non-reasoning LLMs based on best practices from prior work (Yang et al., 2025; DeepSeek-AI et al., 2025). Non-reasoning LLMs use temperature 0 / top-p 0 (greedy decoding) with only 1 output token needed (Yes or No), making them fast and cheap. Reasoning LLMs use temperature 0.6 / top-p 0.95 and are allowed up to 32,000 output tokens for their thinking traces, with the post-thinking response expected to be Yes or No. This means non-reasoning judges are "substantially cheaper and faster than reasoning LLMs (approximately two to three orders of magnitude in practice)." The paper notes that for judges, web search and file upload are not required since criteria are self-contained. For cost reasons, experiments use only half of the dataset as the public set, with the other half held privately to mitigate test contamination concerns (Han et al., 2025). Reliability experiments with GPT-4.1 as judge showed that both Macro-F1 and Bias-Index differ by no more than 0.2% across three independent runs, justifying single-run evaluation for cost savings.

Evaluation metrics. The paper defines three complementary metrics for judge quality:

Agreement with human annotations: Macro-F1, computed as the unweighted average of per-class F1 scores across the Yes and No classes, treating human binary labels as ground truth and judge binary predictions as the comparison. Macro-F1 (rather than accuracy) is used because the class distribution may be imbalanced; it weights both fulfillment and non-fulfillment equally. The paper cites HealthBench and PaperBench as precedent for this choice.

Bias-Index: This metric captures the fairness of a judge toward responses from different model providers. It is computed in three steps:

First, for each response-generating model $m \in \{\text{o3}, \text{Grok4}, \text{R1-0528}\}$ , compute the per-model bias:

$\text{bias}_m = \frac{1}{N_m} \sum_{i=1}^{N_m} \left(c_i^{\text{model}} - c_i^{\text{human}}\right)$

where $N_m$ is the number of criteria evaluated for model $m$ 's responses, $c_i^{\text{model}} \in \{0, 1\}$ is the judge's binary prediction for whether the response fulfills criterion $i$ , and $c_i^{\text{human}} \in \{0, 1\}$ is the human ground-truth binary label for criterion $i$ .

What it computes: For each response-generating model, the average difference between the judge's predicted fulfillment rate and the human-labeled fulfillment rate. A positive bias means the judge is more lenient than humans (over-predicting Yes); a negative bias means the judge is stricter.

Second, compute the range of biases across the three response-generating models:

$\text{Bias-Index} = \max(\text{bias}_{o3}, \text{bias}_{Grok4}, \text{bias}_{R1-0528}) - \min(\text{bias}_{o3}, \text{bias}_{Grok4}, \text{bias}_{R1-0528})$

Why this form: The max-minus-min range captures the differential treatment of models rather than the absolute bias level. If a judge is uniformly strict (negative bias for all models by the same amount), rankings between models are preserved — the judge is harsh but fair. If a judge over-rewards o3 responses while under-rewarding Grok4 responses, the Bias-Index will be large because the per-model biases differ substantially. A low Bias-Index (ideally near 0) means the judge treats all model responses similarly relative to human judgments. This is directly motivated by the self-enhancement bias literature (Zheng et al., 2023), which shows that LLMs tend to favor their own outputs.

Overall score: The paper defines overall judge quality as:

$\text{Overall} = \text{Macro-F1} - \text{Bias-Index}$

This is a composite that rewards judges for aligning with human judgments while penalizing them for systematic bias toward specific models. The subtraction means a judge with perfect human agreement (Macro-F1 = 100%) but severe bias (Bias-Index = 20%) would score 80%, below a judge with slightly lower agreement (Macro-F1 = 88%) but negligible bias (Bias-Index = 2%) scoring 86%. The relative weighting implies that a 1% improvement in bias mitigation is valued equally to a 1% improvement in agreement, treating fairness as a first-class metric rather than a secondary concern.

Cost: The upper-bound cost of running the full LLM-Judge evaluation is calculated by multiplying input and output token counts by public API prices from OpenRouter (as of 2025), excluding discounts from caching. The paper notes that "in practice, caching alone will reduce the cost to as low as one-tenth as stated cost for judges whose costs are dominated by input tokens." Cost serves as a comparison dimension when judges have similar performance, with cheaper judges being more accessible.

Design choice: Macro-F1 over accuracy. The paper selects Macro-F1 over simple accuracy because the class distribution (Yes vs. No across all criteria) may be imbalanced — the three reference models fulfill criteria at rates of 45–52%, so neither class dominates. Macro-F1 ensures that a judge cannot achieve high scores by always predicting the majority class, forcing it to be good at both recognizing when criteria are fulfilled and when they are not. This is important because both types of errors matter: false positives (judging a bad response as fulfilling a criterion) would inflate model scores unfairly, while false negatives (judging a good response as failing) would unfairly penalize models.

Design choice: Bias-Index over per-model bias reporting. The paper could have simply reported per-model biases individually (as seen in Table 2, where columns o3, R1-05, and Grok4 show individual bias values). However, the Bias-Index aggregates these into a single summary statistic that captures the unfairness of the judge — the extent to which it differentially advantages some model providers over others relative to human judgment. The choice of max-minus-min range (rather than standard deviation or mean absolute deviation) is driven by the worst-case nature of the fairness concern: if a judge inflates o3 scores by 5% while deflating Grok4 scores by 5%, the 10% range directly captures the maximum distortion in model rankings that could occur. Standard deviation would underweight this bimodal pattern because it penalizes deviations from the mean, but the mean bias might be near zero while individual biases are far apart.

Composite Judge Construction: The GPT-OSS-120B Adaptive Judge

Rather than simply selecting the best-performing judge from Table 2, the authors construct a composite judge that adapts its reasoning effort based on the domain and criterion type of each evaluation instance. This is the central engineering contribution that achieves the cost-performance frontier.

The selection problem. From Table 2, the best overall proprietary judge is Gemini-2.5-Pro at 78.2% Overall (Macro-F1 79.2% minus Bias-Index 1.0%), costing $41.46 for the full evaluation with 967 output tokens on average. The best performing open-weight option at reasonable cost is GPT-OSS-120B, which in its high reasoning effort configuration achieves 75.4% Overall (Macro-F1 78.4% minus Bias-Index 3.0%) at$ 0.88, and in its low reasoning effort configuration achieves 76.7% Overall (Macro-F1 77.3% minus Bias-Index 0.6%) at $0.50. Neither configuration individually matches Gemini-2.5-Pro in Overall, but the high-effort variant performs better on certain domain-criterion combinations while the low-effort variant performs better on others.

The observed heterogeneity. Looking at the detailed breakdowns (Table 2), the paper observes: "the high reasoning effort version does better on Physics, Chemistry and Style-related criteria, while the low reasoning effort version does better on others." This domain- and criterion-type dependence is the key insight enabling adaptive reasoning — rather than applying uniform reasoning effort across all 7,000+ judgments, allocate high reasoning effort only where it helps and low reasoning effort where it doesn't, balancing quality and cost.

The composite judge specification. The selected judge is GPT-OSS-120B with high reasoning effort for Physics, Chemistry, and Style-type criteria and low reasoning effort for all other domains and criterion types (Finance, Consulting, Extraction/Recall, and Reasoning-type criteria). The motivation is domain-specific: Physics and Chemistry criteria often require multi-step reasoning about quantitative correctness and logical consistency that benefits from extended thinking, while Style criteria (formatting, significant figures, decimal places) similarly require careful verification. Finance and Consulting criteria are often more about retrieving specific facts from responses, which requires less extended reasoning.

The authors also note that they further fine-tune this selection by using "high for physics/chemistry/style + low for others" specifically, reporting this configuration achieves 78.2% Overall (Macro-F1 78.7% minus Bias-Index 0.5%) at $0.70. This matches Gemini-2.5-Pro's 78.2% Overall while costing 1.68% of Gemini-2.5-Pro's$ 41.46.

Cost comparison to prior work. The composite ProfBench judge costs $0.70 to run the full evaluation, compared to approximately$ 1,320 for PaperBench's JudgeEval (Starace et al., 2025) — a factor of roughly 1,886× cheaper. The paper attributes this to several factors: using an open-weight model with no per-token API premium, adapting reasoning effort so that expensive extended thinking is only used on the subset of criteria where it helps, and using a minimal prompt template that minimizes input tokens.

Design choice: reasoning effort adaptation over uniform effort. The alternative would be to use a single model configuration across all judgments. The paper's experiments show that the optimal uniform configuration (GPT-OSS-120B-low at 76.7% Overall) is 1.5 percentage points worse than the adaptive configuration (78.2% Overall), while the adaptive configuration adds only $0.20 in cost. The adaptation is inspired by Jung et al. (2025), who proposed escalating to more capable judges when necessary. Here, the escalation is not to a different model but to a higher-reasoning-effort configuration of the same model, avoiding the overhead of managing multiple model deployments.

Design choice: GPT-OSS-120B over alternatives. GPT-OSS-120B is an open-weight model, meaning it can be run without per-token API costs beyond infrastructure. At the time of writing, GPT-OSS-120B-low costs $0.50 for the full evaluation versus$ 41.46 for Gemini-2.5-Pro. Even if Gemini-2.5-Pro were slightly better (which the adaptive configuration shows they match), the cost difference of nearly 83× makes the open-weight option far more accessible for researchers who need to run the benchmark repeatedly during model development. The paper positions this as "democratizing access" — a researcher with a $100 budget could run the ProfBench judge 140 times with GPT-OSS-120B but only 2 times with Gemini-2.5-Pro.

Report-Generator Evaluation Protocol

The second evaluation pipeline assesses LLMs as producers of professional-quality reports, using the composite LLM-Judge as an automated scorer.

Task formulation. Each model is given a task prompt and the associated grounding documents (truncated PDFs) and asked to generate a response addressing the prompt. This formulation is inspired by "how human professionals commonly work when tackling real-world tasks" — consulting from provided reference materials rather than relying solely on parametric knowledge. The paper notes this is "a formulation that has not been applied in popular benchmarks such as HLE, GPQA and MMLU-Pro."

Inference configuration for report generation. The paper sets generation limits to accommodate long professional reports: up to 32,000 tokens for non-reasoning LLMs (the maximum possible sequence length for GPT-4.1) and 64,000 tokens for reasoning LLMs (the maximum for Claude 4 Sonnet and Gemini 2.5 Pro). For Google models, the default temperature of 1 is used because "temperature 0 induces highly repetitive generations." Kimi-K2-0911 uses the recommended inference temperature of 0.6, similar to reasoning models. The paper notes that despite the judge's affordability, judging criterion fulfillment costs approximately $3.50 per evaluation run (separate from response generation).

Scoring schema. For each generated response, the composite GPT-OSS-120B judge (with adaptive reasoning effort) evaluates every criterion for that task, producing a binary Yes/No per criterion. The overall score for a response is a weighted criterion-fulfillment rate:

$\text{Score} = \frac{\sum_{i=1}^{K} w_i \cdot \mathbb{1}[\text{criterion}_i \text{ fulfilled}]}{\sum_{i=1}^{K} w_i}$

where $K$ is the number of criteria for the task (typically 15–60), $w_i \in \{1, 2, 3, 4\}$ is the importance weight for criterion $i$ (1 = Additional, 2 = Minor, 3 = Major, 4 = Critical), and $\mathbb{1}[\cdot]$ is the judge's binary fulfillment prediction. The denominator normalizes by the sum of weights so that the score falls in $[0, 1]$ regardless of how many criteria a task has.

What it computes: A weighted average of criterion fulfillment where more important criteria contribute proportionally more to the final score. A response that satisfies all Critical criteria but misses some Minor criteria scores higher than a response that satisfies all Minor criteria but misses Critical ones. The final score is expressed as a percentage.

Why this form: The weighted average formalizes the intuition that not all criteria are equally important — getting a key financial calculation right matters more than formatting stock prices to two decimal places. The weights are assigned by domain experts during rubric creation, embedding professional judgment about what constitutes quality into the scoring function. Without weighting, a model could achieve high scores by optimizing for easily-satisfied style criteria while neglecting substantive analytical requirements.

Validation against human-annotated performance. To verify that the composite judge's scoring aligns with human judgment at the aggregate level, the authors compare judge-predicted scores to human-annotated scores for the three reference models (o3, Grok4, R1-0528). The paper reports the gaps:

R1-0528: judge-predicted 46.8% vs. human-annotated 46.1% (gap of +0.7 percentage points)
Grok4: judge-predicted 50.5% vs. human-annotated 51.8% (gap of −1.3 percentage points)
o3: judge-predicted 53.5% vs. human-annotated 52.7% (gap of +0.8 percentage points)

The gaps range from 0.7 to 1.3 percentage points, which the paper characterizes as validating that the scoring schema "is capable of scoring each model with only a 0.7 to 1.3% gap between judge-predicted and human-annotated performance across 3 models." This is critical: it demonstrates that the composite judge, despite being an imperfect criterion-by-criterion predictor (the Macro-F1 is 78.7%, not 100%), produces aggregate scores that closely track human-assigned scores through error cancellation — the judge's mistakes on individual criteria largely wash out when averaged across 30+ criteria per task.

Variance estimation. Rather than generating a single response per task, the paper generates 16 independent responses per model (sampling with the specified temperature) and reports the mean score and standard deviation across these 16 variants. This follows Bercovich et al. (2025) and is essential because stochastic generation means any single response may be unrepresentative of the model's typical performance. The standard deviation quantifies the within-task reliability: a model that scores 60% ± 2% is consistently performing at that level, while a model that scores 60% ± 12% has high variance and its estimated mean is less precise.

Design choice: importance-weighted scoring over unweighted average. The alternative would be to treat all criteria as equally important (unweighted criterion fulfillment rate). The paper does not ablate this choice explicitly, but the distribution of weights — roughly half Major, quarter Critical — means that equal weighting would undervalue Critical criteria (which experts deemed most important) and overvalue Minor and Additional criteria. The validation against human scores provides indirect evidence that the weighting scheme does not distort model rankings relative to holistic human judgment.

Design choice: separate judge for evaluation vs. human-as-ground-truth. The paper uses the composite LLM-Judge for evaluating new model responses, but validates the judge's aggregate scores against human annotations on the three reference models. This two-tier approach means the benchmark's ground truth is human judgment (for the reference models) while its scalable evaluation is automated (for all other models). The 0.7–1.3% validation gap bounds the automation error.

Optimal Sample Allocation for Variance Reduction (Appendix G)

The Report-Generator pipeline generates 16 responses per task by default, but this is costly — with 40 tasks in the public evaluation set, 16 responses per task means 640 total generations and $48 in generation costs (using o3). The paper introduces a method to reduce the average number of generations while maintaining estimation precision.

Motivation: heterogeneous per-task variance. Figure 3 illustrates that performance variance differs substantially across tasks. For Gemini-2.5-Flash (Thinking), the score on Task Chem-9 ranges from 11.2 to 63.8 across 16 samples (a range of 52.6 percentage points), while the score on Task Chem-4 ranges only from 82.0 to 96.1 (a range of 14.1 percentage points). This means that a fixed budget of $B$ generations per task is suboptimal: high-variance tasks benefit from more samples (reducing the standard error of their estimated mean), while low-variance tasks need fewer samples (their estimated mean is already precise).

Formulation as integer programming. The paper formalizes the allocation problem as minimizing the variance of the overall performance estimate subject to a total generation budget.

Let $n_i$ be the number of generations allocated to task $i$ , and $N$ be the total number of tasks (40 in the public evaluation set). The overall performance estimate $S$ is the average of per-task mean scores:

$S = \frac{1}{N} \sum_{i=1}^{N} \bar{s}_i$

where $\bar{s}_i$ is the mean score over the $n_i$ generated responses for task $i$ .

Assuming independent and identically distributed (i.i.d.) sampling of responses within each task and independence between tasks, the variance of $S$ is:

$\sigma^2(S) = \frac{1}{N^2} \sum_{i=1}^{N} \frac{v_i}{n_i}$

where $v_i$ is the variance of scores for task $i$ , estimated by generating 16 rollouts per prompt across four representative models (Gemini-2.5-Pro, Gemini-2.5-Flash Thinking, o3, o4-mini) and aggregating their variances.

What it computes: the variance of the estimator of the model's overall performance across all tasks. Each task $i$ contributes $\frac{v_i}{n_i}$ — its inherent variance divided by the number of samples — because with i.i.d. sampling, the variance of the sample mean $\bar{s}_i$ is $\frac{v_i}{n_i}$ . Summing across tasks (with the $\frac{1}{N^2}$ factor for averaging) gives the variance of the overall estimate.

Why this form: this is the standard formula for variance of a sum (or average) of independent random variables. The critical property is the $\frac{1}{n_i}$ dependence: doubling the number of samples for task $i$ halves its contribution to the overall variance. This diminishing-returns property is what makes allocation non-trivial — moving one sample from a low-variance task to a high-variance task reduces overall variance more than uniform allocation.

The optimization problem is:

$\min_{n_i} \sum_{i=1}^{N} \frac{v_i}{n_i}$

subject to:

$\sum_{i=1}^{N} n_i = B, \quad n_i \in \mathbb{Z}^+, \quad n_i \geq 1 \text{ for all } i$

where $B$ is the total generation budget. The constraint $n_i \geq 1$ ensures every task gets at least one generation (you cannot estimate a mean from zero samples).

Solution via dynamic programming. Given the small problem scale ( $N=40$ , $B$ up to a few hundred), the paper solves this exactly using dynamic programming — a standard technique for integer resource allocation problems. The DP state tracks the minimum achievable variance given a remaining budget and a subset of tasks to allocate. Because the objective is separable (sum of independent per-task terms) and the constraint is additive (sum of per-task allocations), dynamic programming finds the globally optimal integer allocation efficiently.

Results and practical configuration. The paper sets $B = 160$ for the public evaluation set of 40 tasks, meaning an average of $n_i = 4$ generations per task — a 4× reduction from the default 16 samples. The optimal allocation reduces standard deviation to approximately 50% of the naive point estimate (single generation per task). Figure 5 shows that optimal allocation consistently outperforms heuristic baselines: uniform allocation with mean aggregation, uniform allocation with median aggregation, and uniform allocation with min/max exclusion (removing the best and worst per-task scores before averaging). The dynamic programming solution finds allocations that are non-uniform — high-variance tasks like Chem-9 might receive 6–8 generations while low-variance tasks like Chem-4 might receive 1–2 generations.

Design choice: dynamic programming over heuristic allocation. A common heuristic for this problem is to allocate samples proportionally to estimated variance $v_i$ — tasks with higher variance get more samples. However, the $\frac{v_i}{n_i}$ objective means the marginal benefit of adding a sample to task $i$ is $\frac{v_i}{n_i} - \frac{v_i}{n_i + 1} = \frac{v_i}{n_i(n_i + 1)}$ , which depends not only on $v_i$ but also on the current allocation $n_i$ . Tasks that already have many samples benefit less from additional ones (diminishing returns). Simple proportionality ignores this non-linearity. Dynamic programming explicitly accounts for it, finding the allocation that equalizes marginal benefits across tasks at the optimum — a property the paper exploits to achieve lower variance than heuristic methods at the same budget.

Design choice: pre-computing variance estimates from past models. The variances $v_i$ are estimated from 16 generations across four representative models and then treated as fixed for all future evaluations. This assumes that per-task variance is a property of the task (and the evaluation rubric) — i.e., some tasks have inherently bimodal score distributions because a model either "gets" the core reasoning or doesn't — rather than a property of the specific model being evaluated. The paper does not validate this assumption directly, but the practical consequence is that the allocation is computed once and reused for all model evaluations, keeping the overhead minimal.

Ablation Design: Grounding Documents and Web Search (Section 6)

To quantify the contribution of the provided grounding documents to model performance, the paper conducts a controlled ablation with two additional configurations beyond the standard setting.

Configuration 1: LLM only (no documents). The model receives only the task prompt with no grounding documents. This tests whether the model's parametric knowledge alone can answer the professional tasks.

Configuration 2: Search capability (no documents, but web search). The model receives only the task prompt but is allowed to retrieve relevant documents through web search. This tests whether models can compensate for missing provided documents by finding the necessary information online. Since all grounding documents have publicly accessible URLs, this is a realistic test of whether web search can substitute for curated document provision.

Models tested. The ablation uses o3 and o4-mini, with both configurations compared against the standard setting (LLM + grounding documents). Cost calculations for the search configuration exclude search query costs, estimated at approximately $6.40.

Hypothesis and expected patterns. The central hypothesis is that professional tasks requiring analysis of specific documents (financial statements, research papers, technical specifications) cannot be adequately performed from parametric knowledge alone. If this holds, removing documents should substantially reduce performance, particularly on Extraction/Recall criteria that require retrieving specific facts from the provided materials. Adding web search should partially recover performance if models can locate the relevant documents online, but the recovery is expected to be incomplete because (a) search may retrieve different versions or related but non-identical documents, and (b) the large volume of retrieved content may exceed effective context utilization limits.

Design choice: two-step ablation (remove, then add search) rather than one-step. A single ablation comparing documents vs. no documents would conflate two effects: the value of having any reference information at all, and the value of having specifically curated, pre-selected reference documents. The two-step design separates these: the drop from standard to LLM-only measures the total contribution of reference information; the recovery from LLM-only to search measures how much of that value web search can recapture. This decomposition is practically important because if search fully recovered the document value, then the cost and complexity of curating grounding documents could be eliminated.

4. Key Insights and Innovations

Innovation 1: Bias-Awareness as a First-Class Evaluation Metric, Not an Afterthought

The paper's most conceptually distinctive contribution is elevating LLM-Judge bias from a known weakness to a measurable, optimizable quantity that is treated as co-equal with accuracy in defining judge quality. The dominant practice in the LLM-as-judge literature — including the rubric-based benchmarks ProfBench builds on (PaperBench, HealthBench) — has been to evaluate judges primarily by their agreement with human annotations (Macro-F1, accuracy, or correlation) while treating self-enhancement bias as a qualitative concern acknowledged in limitations sections. ProfBench makes a different move: it defines overall judge quality as Macro-F1 minus Bias-Index, meaning a 1% improvement in bias mitigation is valued identically to a 1% improvement in human agreement.

This is not an incremental tweak. It fundamentally reframes what "good judging" means. Under a pure-accuracy paradigm, a judge that achieves 88% Macro-F1 with a 10% provider bias range would be considered superior to a judge with 80% Macro-F1 and 1% bias range. Under ProfBench's formulation, the second judge wins (80 − 1 = 79 vs. 88 − 10 = 78). The paper is arguing, in effect, that a benchmark whose rankings are systematically distorted by model-provider identity is not a valid measurement instrument regardless of how well it tracks human judgments on average — because the distortion corrupts the very comparisons the benchmark exists to enable.

The field has known about self-enhancement bias since Zheng et al. (2023), but the typical response has been to acknowledge it as a limitation and move on. ProfBench's response is to operationalize it into a concrete metric (the max-minus-min range of per-model bias across three response providers) and to make that metric a binding constraint on judge selection. This is a diagnostic contribution: it provides a standardized, replicable way to measure whether an evaluation pipeline is provider-fair, which no prior rubric-based benchmark had done.

The practical consequence is visible in Table 2: several models with competitive Macro-F1 scores are revealed as unusably biased when the Bias-Index is applied. Llama-3.1-405B-Instruct achieves a strong 77.0% Macro-F1 but carries a 5.4% Bias-Index (driven by +11.5% bias toward o3 responses vs. +6.1% toward Grok4), dragging its Overall to 71.6%. The composite judge (GPT-OSS-120B adaptive) achieves only a slightly higher Macro-F1 (78.7%) but with a Bias-Index of 0.5%, yielding 78.2% Overall — the difference between a useful instrument and a systematically distorted one.

The significance extends beyond this paper: if the field adopts bias-awareness as a standard evaluation dimension for LLM-judges, it changes which judges get deployed in production evaluation pipelines. The paper's finding that increasing reasoning effort increases bias toward specific providers (visible consistently across the GPT-5, o3, and GPT-OSS families in Table 2) suggests a previously undocumented tension between accuracy and fairness that future judge-design work must navigate.

Innovation 2: Reasoning-Effort Heterogeneity as a Cost-Optimization Lever

The paper's second distinctive contribution is the empirical discovery and exploitation of domain- and criterion-type-dependent reasoning-effort utility: the finding that increasing a reasoning model's thinking budget helps on some evaluation subtasks but hurts (or provides no benefit) on others, and that this heterogeneity can be exploited to construct a composite judge that matches the best proprietary model at 1.68% of the cost.

This is fundamentally different from the standard "more reasoning = better" assumption that implicitly guided prior work. The naive approach to judge construction would be: pick the model configuration that gives the best overall score and use it uniformly. That would have produced either Gemini-2.5-Pro at 78.2% Overall and $41.46, or GPT-OSS-120B-low at 76.7% Overall and$ 0.50. The paper instead observes a non-uniform pattern: high reasoning effort improves performance on Physics (+2.8 percentage points over low effort for GPT-OSS-120B), Chemistry (+1.7), and Style criteria (+9.7 in one configuration), while low reasoning effort is better for Finance, Consulting, and Extraction/Reasoning criteria.

This pattern has a plausible causal explanation — Physics and Chemistry criteria often require multi-step verification of quantitative correctness (checking calculations, verifying that units are consistent, ensuring that causal chains are complete), which benefits from extended thinking, while Extraction criteria (did the model mention the correct number?) are often simple lookups that don't require reasoning. Style criteria like "rounds all stock prices to two decimal places" require systematic scanning of the response, which again benefits from reasoning. But the key intellectual move is not the explanation — it's the exploitation: rather than using a uniform reasoning budget, the paper constructs an adaptive judge that allocates high reasoning effort only to the domain-criterion combinations where it empirically helps.

This idea is inspired by Jung et al. (2025)'s "Trust or Escalate" framework, which proposed routing evaluation instances to more capable (and expensive) judges when cheaper judges are uncertain. But ProfBench adapts the concept in a novel way: the "escalation" is not to a different model but to a different inference-time configuration of the same model — specifically, the reasoning effort parameter that controls how many thinking tokens are generated before the final answer. This is a more granular and cost-efficient form of escalation that doesn't require managing multiple model deployments.

The significance is practical but substantial: if the community can replicate this pattern across other benchmarks and judge models, it establishes a general principle that judge cost can be optimized by matching reasoning depth to evaluation subtask characteristics rather than applying uniform effort. The $0.70 vs.$ 41.46 comparison is a concrete demonstration that this principle can reduce costs by nearly two orders of magnitude at equivalent quality.

Innovation 3: Per-Task Variance Heterogeneity as a Sample-Allocation Problem

The paper's third distinctive contribution is the formalization of benchmark evaluation as a resource-allocation problem under heterogeneous per-task variance, solved through dynamic programming to achieve cost reduction without compromising estimation robustness.

Prior benchmark evaluation practice — including PaperBench and HealthBench — typically uses uniform sampling: generate the same number of responses per task and report the mean. This is simple but implicitly assumes that all tasks contribute equally to estimator variance, which is false when some tasks have narrow score distributions (the model consistently succeeds or fails) while others have wide bimodal distributions (the model sometimes succeeds and sometimes fails on the same task). ProfBench's Figure 3 makes this visible: Chem-9 has a score range of 52.6 percentage points across 16 samples while Chem-4 has a range of only 14.1 points. The standard error of the per-task mean is $\sqrt{v_i / n_i}$ , so allocating samples uniformly across tasks with different $v_i$ is wasteful — low-variance tasks get more samples than they need, while high-variance tasks get fewer than they need.

The contribution here is not the mathematical formulation — allocating samples to minimize variance of a stratified mean is a standard survey-sampling problem — but rather the recognition that this problem exists in LLM benchmark evaluation and that it can be solved with off-the-shelf optimization techniques. The paper demonstrates that dynamic programming (feasible because the problem is small: 40 tasks, budget ≤160) can find allocations that reduce standard deviation to 50% of the naive point estimate at the same total budget, or equivalently, achieve the same precision with a 4× budget reduction.

This is a methodological contribution rather than a theoretical one: it provides a recipe that other benchmark designers can apply directly — estimate per-task variance from pilot runs on representative models, formulate the integer program, solve with DP, and use the resulting allocation for all subsequent evaluations. The paper shows that the optimal allocation consistently outperforms heuristic baselines (uniform allocation with mean/median aggregation, min/max exclusion) at all budget levels, confirming that the sophistication of the DP solution matters in practice rather than just in theory.

The broader implication is that benchmark cost and benchmark precision are not a fixed tradeoff — smart allocation can shift the frontier, making rigorous evaluation accessible to researchers with smaller compute budgets. This matters for the same reason the judge cost reduction matters: it lowers barriers to entry for the broader community.

5. Experimental Analysis

Evaluation Methodology

Dataset. ProfBench contains 7,347 human-written response-criterion pairs across 80 distinct tasks equally divided among four domains: Chemistry PhD, Physics PhD, Finance MBA, and Consulting MBA (20 tasks each). For evaluation, the paper uses only half of the dataset as the public set, holding the remaining half private to mitigate test contamination (Han et al., 2025). Three reference model responses per task (generated by o3, Grok4, and DeepSeek R1-0528) were annotated by domain-expert humans with binary criterion-fulfillment labels, establishing ground truth. Human validation on 1,127 response-criterion pairs with two independent re-annotators yielded Fleiss' κ = 0.912, confirming excellent inter-annotator agreement.
Base models for judge evaluation. Over 40 models spanning both proprietary (OpenAI GPT-4.1/5/o3/o4-mini families, Google Gemini-2.5, Anthropic Claude-Sonnet-4/Claude-3.5-Haiku, xAI Grok-4/3-mini) and open-weight (Qwen3, Kimi-K2, DeepSeek-V3.1/R1-0528, Llama-3.1/3.2/3.3/4, Llama-Nemotron, GPT-OSS-20b/120b) families. Models were selected to represent the diversity of available LLM capabilities at the time of evaluation (mid-2025), including both instruct and reasoning variants.
Base models for report-generator evaluation. A subset of 19+ models evaluated as report-generators, including GPT-5 (high/medium/low/minimal reasoning), GPT-5-mini/nano, o3, o4-mini, Gemini-2.5-Pro/Flash/Flash-Lite (thinking and non-thinking), Claude-Sonnet-4 (thinking), Grok-4, GPT-OSS-120b/20b, DeepSeek-V3.1 (thinking), Qwen3-235B/30B (instruct and thinking), Kimi-K2-Instruct, Llama-4-Maverick/Scout, and GPT-4.1/nano. Selection covers both proprietary and open-weight models at multiple scales.
Metrics for LLM-Judge evaluation. (1) Macro-F1: unweighted average of per-class F1 scores (Yes/No) comparing judge binary predictions to human ground-truth labels. (2) Bias-Index: the range (max minus min) of per-model bias across the three response-generating models (o3, Grok4, R1-0528), where per-model bias = average difference between judge-predicted and human-labeled criterion fulfillment for that model's responses. A low Bias-Index indicates fair treatment across providers. (3) Overall: Macro-F1 minus Bias-Index — the composite quality metric that penalizes biased judges. (4) Cost: upper-bound dollar cost based on public API pricing (OpenRouter, 2025) multiplied by input/output token counts, excluding caching discounts. Costs in Table 2 range from $0.02 (Llama-3.2-3B-Instruct) to$ 62.64 (Claude-Sonnet-4 thinking).
Metrics for Report-Generator evaluation. Weighted criterion-fulfillment rate (Score): for each response, the fraction of criteria fulfilled, weighted by importance level (Critical = 4, Major = 3, Minor = 2, Additional = 1), normalized by the sum of weights. Reported as a percentage. Additionally, Accuracy by Criterion Type (Extraction/Recall, Reasoning, Style) is reported to enable diagnostic analysis of model strengths and weaknesses. Performance is estimated by averaging across 16 independent generations per task (or fewer under optimal allocation), with standard deviation reported.
Baselines for LLM-Judge evaluation. No explicit baselines in the traditional sense — all 40+ models are compared against each other on the three metrics. The implicit baseline is the set of non-reasoning instruct models (e.g., GPT-4.1 at 75.4% Overall) against which reasoning models are compared, and proprietary models against which open-weight models are compared.
Baselines for Report-Generator evaluation. The primary reference point is GPT-5-high at 65.9% overall, establishing the performance ceiling. Within-model-family comparisons (e.g., GPT-5-mini vs. nano, Gemini-2.5-Flash vs. Flash-Lite) serve as scaling baselines. The three human-annotated reference models (o3 at 51.6% human fulfillment, Grok4 at 47.4%, R1-0528 at 45.2%) provide an anchor for interpreting judge-scored performance. For the ablation (Section 6), the standard configuration (LLM + grounding documents) serves as the baseline against which LLM-only and LLM + search configurations are compared.
Generation budget / compute accounting. For LLM-Judge evaluation, cost is measured in dollars based on API pricing, with per-judge token counts reported (input and output separately). Non-reasoning judges generate exactly 1 output token (Yes/No), making them ~100–1000× cheaper than reasoning judges that generate up to 32,000 thinking tokens. For Report-Generator evaluation, the generation budget is measured in number of responses per task (16 by default, reduced to an average of 4 under optimal allocation). Per-response generation limits are 32,000 tokens for non-reasoning models and 64,000 for reasoning models. Total evaluation cost includes both response generation and judge scoring ($3.50 for judging alone, plus generation costs).
Cross-validation / statistical protocol. No traditional cross-validation is used, since the task is benchmark evaluation rather than model training. Instead, the paper employs: (1) reliability assessment: three independent runs with GPT-4.1 judge confirmed Macro-F1 and Bias-Index differ by ≤0.2%, justifying single-run evaluation; (2) variance estimation: 16 independent generations per task per model, with mean and standard deviation reported following Bercovich et al. (2025); (3) judge validation: composite judge scores compared against human-annotated scores on the three reference models (gap of 0.7–1.3 percentage points); (4) inter-annotator agreement: Fleiss' κ on 1,127 re-annotated pairs to validate ground-truth reliability.

Main Quantitative Results

LLM-Judge Evaluation: Can Models Reliably Verify Professional-Domain Criterion Fulfillment?

The headline results appear in Table 2, which reports Macro-F1, Bias-Index, Overall score, and cost for over 40 judge models across four domains, three criterion types (Extraction, Reasoning, Style), and an "All" aggregate.

Best overall judge. Gemini-2.5-Pro (thinking) achieves the highest Overall score at 78.2%, with Macro-F1 of 79.2% across all domains and a Bias-Index of only 1.0%, at a cost of $41.46. Its sibling Gemini-2.5-Flash (thinking) trails by only 0.1 percentage points at 78.1% (Macro-F1 78.4%, Bias-Index 0.3%) at$ 7.92 — nearly identical quality at roughly one-fifth the cost. This near-parity between Pro and Flash variants suggests that model scale within the Gemini family provides negligible judging benefit on ProfBench, consistent with a pattern the authors highlight as "minuscule 0.1% model performance gains between Gemini-2.5-Flash-Lite and Gemini-2.5-Flash (non-thinking) as well as Gemini-2.5-Flash (thinking) and Gemini-2.5-Pro (thinking)."

Best non-reasoning judge. Among models restricted to a single output token (no extended thinking), GPT-4.1 achieves 75.4% Overall (Macro-F1 76.3%, Bias-Index 0.9%) at $11.31, followed closely by Kimi-K2-Instruct-0711 at 75.2% Overall (Macro-F1 77.6%, Bias-Index 2.4%) at only$ 0.81 — a 14× cost reduction for comparable quality. GPT-4.1-mini achieves 74.9% Overall at $2.26, nearly matching GPT-4.1's performance at one-fifth the cost.

The composite judge (GPT-OSS-120B adaptive). The authors construct a composite judge using GPT-OSS-120B with high reasoning effort for Physics, Chemistry, and Style criteria and low reasoning effort for all others. This configuration achieves 78.2% Overall (Macro-F1 78.7%, Bias-Index 0.5%), matching Gemini-2.5-Pro's 78.2% at a cost of $0.70 — only 1.68% of Gemini-2.5-Pro's$ 41.46. This is the judge used for all Report-Generator evaluations.

Open-weight vs. proprietary. The top open-weight models are competitive but trail the best proprietary models by small margins. Kimi-K2-0711 (instruct, non-reasoning) achieves 75.2% Overall at $0.81, only 0.2 percentage points behind GPT-4.1 at 75.4% but at 7.16% of the cost. In the reasoning category, GPT-OSS-120B-low achieves 76.7% Overall at$ 0.50, compared to Gemini-2.5-Pro at 78.2% and $41.46 — a 1.5 percentage point gap at 1.21% of the cost. Qwen3-235B-A22B-Thinking-2507 achieves 76.5% Overall at$ 1.84. The paper emphasizes that "the gap between closed-source and open-weight models is small," particularly given the enormous cost differential.

Model size scaling. Within model families, scaling from small to medium yields large gains, but scaling from medium to large yields diminishing returns (Table 2): GPT-4.1-nano → GPT-4.1-mini improves by +20.8 percentage points (54.1% → 74.9%), but GPT-4.1-mini → GPT-4.1 improves by only +0.5 points (74.9% → 75.4%). Llama-3.1-8B → 70B improves by +7.6 points (63.1% → 70.7%), but 70B → 405B adds only +0.9 points (70.7% → 71.6%). The jump from Llama-3.1-70B to Llama-3.3-70B (+3.4 points) exceeds the jump from Llama-3.1-70B to Llama-3.1-405B (+0.9 points), indicating "the importance of improvements in post-training recipe beyond model size alone."

Effect of reasoning/thinking. Enabling thinking on the same model generally improves Overall performance, though the magnitude varies (Table 2, comparing thinking vs. non-thinking variants): Claude-Sonnet-4 improves by 0.7 points (70.2% → 70.9%), DeepSeek-V3.1 improves by 4.8 points (72.8% → 76.6%), Gemini-2.5-Flash improves by 4.7 points (73.4% → 78.1%), and Gemini-2.5-Flash-Lite improves by 1.4 points (73.3% → 74.7%).

Reasoning effort scaling within a model. For the GPT-5 family (Table 2), increasing reasoning effort from minimal to low provides substantial gains — GPT-5 improves by 4.4 points (71.9% → 76.3%), GPT-5-mini by 7.9 points (66.7% → 74.6%), and GPT-5-nano by 18.6 points (55.0% → 73.6%). However, further increases (low → medium → high) yield inconsistent and sometimes negative returns: o4-mini peaks at low effort (76.8%), GPT-5-nano peaks at medium (72.7%), and gpt-oss-120b peaks at low (76.7%). The paper attributes this partly to "greater self-enhancement bias with more thinking, as the bias towards OpenAI o3 responses generally increases." For example, GPT-5-high's per-model biases are +1.0% (o3), −0.8% (R1-0528), and −1.3% (Grok4), yielding a Bias-Index of 2.3%, while GPT-5-minimal shows −0.5%, −5.6%, −5.0% with a Bias-Index of 5.1%.

Domain-specific judge performance. All judges perform worst on Chemistry (domain-averaged Macro-F1 across top models ~68–70%) and best on Physics (some models reaching ~89–90% Macro-F1). Style criteria show the widest variation: top reasoning models exceed 75% Macro-F1, while weaker models fall to the 50–60% range. This pattern is consistent with the observing note (Section 4.2) that "thinking with greater effort seems to lead to largest improvement in Physics, Chemistry and criteria related to Style," since these criteria "often entail multi-step reasoning to ensure that the response fulfills the criteria, instead of a simple answer-matching."

Cost disparities. The cost range is extreme: the cheapest functional judge (Llama-3.2-3B-Instruct) costs $0.02 but achieves only 58.3% Overall, while the most expensive (Claude-Sonnet-4 thinking) costs$ 62.64 for 70.9% Overall. The cost-effectiveness frontier is dominated by models like GPT-4.1-mini ( $2.26 for 74.9%) and GPT-OSS-120B-low ($ 0.50 for 76.7%), while several expensive models (Claude-Sonnet-4 at $62.64 for 70.9%, Grok-4 at$ 58.70 for 75.9%) deliver worse cost-quality ratios than much cheaper alternatives.

Report-Generator Evaluation: Can Frontier LLMs Produce Professional-Quality, Document-Grounded Reports?

The headline results appear in Table 3, which reports Score and standard deviation for each domain, overall weighted average, accuracy breakdown by criterion type (Extraction/Recall, Reasoning, Style), and average response characteristics (character length, input/output tokens, cost). Sixteen independent generations per task per model provide variance estimates.

Overall top-performing model. GPT-5-high achieves 65.9% overall (standard deviation 1.1%), confirming the benchmark's difficulty. By domain: Physics 49.3% (±2.6), Chemistry 70.6% (±2.1), Finance 63.7% (±2.8), Consulting 80.0% (±2.0). Physics is hardest and Consulting easiest, with a 30.7 percentage point gap between them. The paper contextualizes this against saturated benchmarks: GPT-5 achieves 94.6% on AIME 25, 87.0% on GPQA-Diamond, and 72.4% on SWE-Bench Verified (OpenAI, 2025), meaning ProfBench is "approximately as challenging as HealthBench, where GPT-5 reaches 67.2%."

Closed-source vs. open-weight models. Proprietary models dominate the top of the leaderboard (Table 3): GPT-5 (65.9%), o3 (61.4%), Gemini-2.5-Pro (60.3%), GPT-5-mini (60.3%). The best open-weight model, GPT-OSS-120B, achieves 54.9%, trailed by Qwen3-235B-A22B-Thinking at 54.0% and DeepSeek-V3.1 (Thinking) at 53.8%. The performance gap between closed and open is domain-dependent: "small for domains like Physics (<<1%), moderate for Chemistry and Consulting (9.2% and 9.6%) and particularly large for Finance (15.0%)." The paper hypothesizes this may result from open-weight models having more in-domain training data for Code and Math (similar to Physics problem-solving) and less emphasis on Chemistry, Consulting, and Finance.

Model size scaling. Within model families, larger models outperform smaller ones but with diminishing returns (Table 3): GPT-5-nano → GPT-5-mini improves by 10.2 points (50.1% → 60.3%), while GPT-5-mini → GPT-5 adds 5.6 points (60.3% → 65.9%). Gemini-2.5-Flash-Lite (Thinking) → Flash (Thinking) improves by 8.2 points (49.4% → 57.6%), while Flash (Thinking) → Pro adds 2.7 points (57.6% → 60.3%). Llama-4-Scout → Maverick improves by 4.0 points (35.4% → 39.4%). The pattern mirrors the judge evaluation finding: "improvement in ProfBench necessitates not only model scaling, but possibly further innovations in training techniques and data curation."

Effect of reasoning/thinking (within same model). Enabling thinking provides modest improvements for models where this is configurable (Table 3): Gemini-2.5-Flash improves from 56.8% (non-thinking) to 57.6% (thinking), a gain of 0.8 points. DeepSeek-V3.1 improves from 53.5% (non-thinking) to 53.8% (thinking), a gain of 0.3 points. Claude-Sonnet-4 improves from 53.5% (non-thinking) to 55.8% (thinking), a gain of 2.3 points.

Reasoning effort scaling (GPT-5 family). Increasing reasoning effort from minimal to high for GPT-5 (Table 3, bottom section) gradually increases overall performance: minimal 61.1%, low 62.9%, medium 65.2%, high 65.9% — a total improvement of 4.8 points. However, the paper notes that "when inferring with separate models of identical size trained for instruction following and thinking respectively, thinking does not necessarily confer an advantage." Indeed, Qwen3-30B-A3B-Thinking scores 44.6% while Qwen3-30B-A3B-Instruct scores 49.3% — a 4.7 point disadvantage for the thinking-trained variant. The authors attribute this partly to response length: the instruct version averages 11,167 characters versus the thinking version's 4,757 characters, and "longer responses tend to cover more content and hence increase the chance of satisfying more criteria."

Response length and verbosity experiments. To isolate the effect of response length on performance, the paper varies GPT-5's verbosity parameter while holding reasoning effort at medium (Table 3, bottom). Low verbosity produces average responses of 3,732 characters and scores 62.2% — a 3.0 point drop from medium verbosity (5,388 characters, 65.2%) despite a 30.7% reduction in length. High verbosity produces 7,133 characters and scores 66.3% — a 1.1 point gain with a 32.4% length increase. The paper concludes that "while response length does influence performance, its effect is minimal — typically within the standard deviation of the two verbosity settings." This suggests report quality, not mere length, drives score differences, though extremely short responses are penalized (Claude-3.5-Haiku at 1,784 characters scores only 27.6%).

Criterion-type breakdown. GPT-5-high's per-criterion-type accuracies (Table 3) reveal consistent difficulty ordering: Reasoning is hardest at 66.2%, followed by Extraction/Recall at 64.4%, with Style easiest at 65.3% (though Style's standard deviation of ±7.7 indicates high per-task variability). The finding that Reasoning criteria are not universally harder than Extraction criteria — the gap is small (1.8 points) and within standard deviations — suggests that professional-domain tasks demand both factual retrieval and logical reasoning in roughly equal measure.

Cost analysis. The cost of evaluating a single model as a report-generator varies enormously (Table 3): GPT-OSS-20B costs $0.75 for the full evaluation (16 samples across 40 tasks), while Claude-Sonnet-4 (thinking) costs$ 164.39. GPT-5-high costs $112.34, o3 costs$ 47.72, and the most cost-effective strong performer is GPT-OSS-120B at $1.35. The paper analyzes this tradeoff in Figure 4 (Appendix F), showing that "OpenAI models are on the Pareto Frontier at each price-point" while "Gemini-2.5 and Qwen3-Instruct-2507 models are close to the Pareto Frontier." Not all expensive models are good: Grok-4-0709 costs$ 122.78 for only 53.4% overall, nearly tied with Claude-Sonnet-4 (non-thinking) at 53.5% and $111.37.

Variance heterogeneity and optimal allocation (Figure 3, Figure 5, Appendix G). The per-task score distributions vary dramatically. Figure 3 visualizes this using box plots for Gemini-2.5-Flash (Thinking) across all 40 tasks: Chem-9 shows a 25th–75th percentile range from ~25% to ~50% (range of ~52.6 points across all 16 samples), while Chem-4 ranges from ~88% to ~95% (range of ~14.1 points). Similarly, Consulting tasks tend to have narrow, high-score distributions while some Finance tasks show wide bimodal distributions. The dynamic programming solution allocates samples non-uniformly: high-variance tasks receive up to 6–8 generations while low-variance tasks receive 1–2, reducing standard deviation to "only 50% of the naive point estimate" when the total budget is fixed at 160 (average 4 per task). Figure 5 confirms optimal allocation outperforms uniform allocation with mean/median aggregation and uniform allocation with min/max exclusion at all budget levels tested.

Ablation: Grounding Documents and Web Search (Section 6)

Table 4 presents results for o3 and o4-mini under three configurations: LLM only (no documents), LLM + search capability (no provided documents, but model can search the web), and LLM + grounding documents (the standard ProfBench configuration).

Removing grounding documents substantially degrades performance. For o3, the overall score drops from 61.4% (with documents) to 52.0% (LLM only) — a decline of 9.4 percentage points. For o4-mini, the drop is from 58.2% to 46.3% — a decline of 11.9 percentage points. The degradation is most severe for Extraction/Recall criteria: o3 drops by 17.1 points (60.4% → 43.3%), and o4-mini drops by 18.7 points (55.8% → 37.1%). Reasoning criteria show more modest declines: o3 drops by 7.0 points (61.8% → 54.8%), and o4-mini drops by 8.9 points (58.3% → 49.4%). Style criteria are largely unaffected (o3: 63.0% → 63.1%). This pattern confirms the hypothesis that grounding documents are most critical for information retrieval tasks and less essential for reasoning and formatting.

Web search partially recovers performance. Adding search capability (without providing the original documents) recovers some of the lost performance: o3 improves from 52.0% (LLM only) to 56.1% (LLM + search) — a recovery of 4.1 points, leaving a residual gap of 5.3 points below the full-document configuration (61.4%). o4-mini improves from 46.3% to 53.3% — a recovery of 7.0 points, with a residual gap of 4.9 points. The recovery is strongest for Reasoning criteria (o3: +3.1 points; o4-mini: +4.6 points) and Extraction (o3: +7.8 points; o4-mini: +11.3 points), suggesting search can locate some but not all of the necessary factual information. The paper notes the search configuration uses "0.12 to 0.24 million [input tokens] per task, or 4 to 12× as much as the original grounding documents," indicating that search retrieves substantially more content but with lower precision — the model must process more tokens to extract the same information.

Qualitative failure mode without documents. The paper reports that "without grounding prompts in such documents, models commonly respond with clarifying requests/questions (e.g., 'Please supply the REIT's Q1'25 NOI, total assets, total liabilities, shares outstanding and 3-month ADV …'), especially for finance and consulting." This is revealing: rather than hallucinating or generating plausible-sounding but incorrect analyses, models respond by requesting the specific data they need, suggesting they recognize when parametric knowledge is insufficient for the task.

Ablation Studies and Robustness Checks

Prompt template selection (Appendix B): The paper tested three binary output formats for the LLM-Judge: "Yes or No" (main template), "True or False" (alternative 1), and "1 or 0" (alternative 2). Using GPT-4.1 as the judge, the main template outperformed alternative 1 by 2.2% and alternative 2 by 4.5%. More elaborate templates with role descriptions, expert framing, and reasoning instructions produced no gains and increased token costs — leading to the adoption of the minimal template. This validates the sufficiency of a minimal prompting approach for this specific verification task.

LLM-Judge inference parameters: The paper tested non-reasoning models with temperature 0/top-p 0 (greedy decoding, 1 output token) and reasoning models with temperature 0.6/top-p 0.95 (up to 32,000 output tokens). Reliability experiments with GPT-4.1 confirmed that both Macro-F1 and Bias-Index differ by no more than 0.2% across three independent runs, justifying single-run evaluation. The reasoning model temperature of 0.6 (rather than 0) follows recommendations from Yang et al. (2025) and DeepSeek-AI et al. (2025), and is necessary because "temperature 0 induces highly repetitive generations" for some models (noted for Google models in the Report-Generator setup).

Report-Generator scoring schema validation (Appendix E): The composite judge's aggregate scores on the three reference models are compared against human-annotated scores: R1-0528 (judge 46.8% vs. human 46.1%, gap +0.7 points), Grok4 (judge 50.5% vs. human 51.8%, gap −1.3 points), o3 (judge 53.5% vs. human 52.7%, gap +0.8 points). All gaps fall within 0.7–1.3 percentage points, confirming that despite imperfect per-criterion accuracy, the judge's errors largely cancel when aggregated across 30+ criteria per task. This is the key validation that the automated evaluation pipeline produces rankings consistent with expert human judgment.

Criterion weight distribution: The paper reports that criterion weights are roughly half Major (49.8%), with Critical (23.4%) and Minor (23.9%) each about a quarter, and Additional at 2.9%. No formal ablation of the weighting scheme is presented (e.g., comparing weighted vs. unweighted aggregation), but the close alignment between weighted judge scores and holistic human scores on the three reference models provides indirect evidence that the weights do not distort model rankings.

Variance-based optimal allocation vs. heuristics (Figure 5, Appendix G): Four allocation strategies are compared: uniform with mean aggregation, uniform with median aggregation, uniform with min/max exclusion, and optimal allocation via dynamic programming. The optimal allocation consistently achieves lower variance at all budget levels. At a total budget of 160 (average 4 per task), the standard deviation is reduced to approximately 50% of the naive point estimate. The min/max exclusion heuristic underperforms due to discarding potentially informative extreme scores, while median aggregation underperforms mean aggregation because it ignores the magnitude of deviations.

Half-dataset public/private split: The paper uses only half the dataset for public evaluation, holding the remaining half private to mitigate test contamination concerns (Han et al., 2025). The private split is not evaluated in the paper but is mentioned as a safeguard. No analysis is provided on whether public and private splits yield consistent model rankings — this remains an open question for future release.

Google temperature sensitivity: The paper notes as a practical finding that "Google models use default temperature of 1 as we observe that temperature 0 induces highly repetitive generations," indicating that some model families have inference behavior that requires provider-specific tuning. This is not formally ablated but represents a robustness observation relevant to anyone attempting reproduction.

Critical Assessment

Does the benchmark genuinely require "professional knowledge" as claimed?

The paper's central claim is that ProfBench tasks "require PhD/MBA-level knowledge." The evidence for this is primarily process-based rather than outcome-based: annotators with relevant credentials (44.7% PhD, 18.4% MBA, average 5.24 years post-degree experience) created the tasks, and the tasks were designed to be "what annotators might ask their (junior) colleagues to help on." The outcome evidence is that GPT-5-high achieves only 65.9%, which is substantially lower than its performance on exam-style benchmarks like GPQA-Diamond (87.0%) or AIME 25 (94.6%). However, this performance gap could be attributable to factors other than knowledge requirements — the multi-page report format, the need to process grounding documents, or the rubric-based evaluation itself — rather than specifically to domain knowledge difficulty.

What would strengthen this claim: A controlled experiment where the same rubric criteria are applied to responses from (a) domain experts versus (b) educated generalists (e.g., college graduates without domain-specific training) versus (c) LLMs, to establish that domain expertise actually differentiates performance in the way the paper implies. Without such a baseline, the claim that tasks "require" professional knowledge is an assertion about task design rather than an empirically verified property. The paper's contrast with DeepResearch-Bench tasks — which the authors argue can be answered by "an educated generalist" — is qualitative and based on example inspection rather than systematic comparison.

Does the Bias-Index adequately capture judge fairness?

The Bias-Index as formulated (max-minus-min of per-model bias across exactly three models) is sensitive to the specific set of response-generating models chosen. The three reference models (o3, Grok4, R1-0528) were selected as best-performing at the time of collection (July 2025). If the Bias-Index were computed over a different set — say, three models all from the same provider — it would trivially be zero regardless of cross-provider bias patterns. Conversely, the Index captures only relative bias across these three specific models, not absolute bias toward all possible models or toward specific model characteristics (size, architecture, training data).

What the Bias-Index does capture: differential treatment across three provider-diverse models that the benchmark uses as reference anchors. This is a reasonable operationalization of the fairness concern for the specific benchmark structure, but it is not a general measure of judge impartiality. The paper could have strengthened this by: (a) showing that models not in the reference set (e.g., GPT-4.1, Claude-3.5-Haiku) show bias patterns consistent with the Bias-Index, or (b) validating that the Bias-Index computed on different subsets of reference models yields consistent rankings of judges.

Does the composite judge's cost advantage hold in practice?

The $0.70 cost for the composite GPT-OSS-120B judge assumes the model is run through an API with pricing comparable to OpenRouter's rates at the time of evaluation. However, as an open-weight model, the actual cost depends on infrastructure — self-hosting eliminates per-token API fees but incurs GPU rental or ownership costs. The paper's cost comparison to proprietary models ($ 0.70 vs. $41.46 for Gemini-2.5-Pro) is valid under the API pricing model, but researchers self-hosting GPT-OSS-120B on, say, an A100 would need to account for compute time rather than token pricing. The paper does not provide throughput or latency measurements that would enable such accounting.

Additionally, the cost of the adaptive routing itself — determining which reasoning effort to apply for each criterion based on domain and type — is trivially small (a lookup table), but the paper does not discuss whether this routing introduces any systematic biases. For instance, always using high reasoning for Physics criteria might subtly advantage models whose Physics outputs happen to align with how the high-reasoning judge evaluates responses, creating a domain-level analog of the Bias-Index pattern.

Are 16 samples per task adequate for reliable model ranking?

The paper uses 16 independent generations per task and reports standard deviations. For GPT-5-high, the overall standard deviation is ±1.1% on a score of 65.9%, meaning a 95% confidence interval of approximately ±2.2%. With only 1.5–4.8 percentage points separating GPT-5 (65.9%) from o3 (61.4%), Gemini-2.5-Pro (60.3%), and GPT-5-mini (60.3%), the ranking confidence is reasonable but not overwhelming — some adjacent models are separated by less than two standard deviations of each other's estimates. The optimal allocation approach (reducing to an average of 4 samples per task) will widen these intervals, potentially making finer-grained model comparisons unreliable. The paper shows that the optimal allocation reduces variance at a fixed budget, but does not report confidence intervals for model rankings under reduced sampling.

Does the 40-task public set provide sufficient domain coverage?

With 20 tasks per domain and 40 tasks in the public evaluation set (the half used for benchmarking), each domain contributes 10 tasks to the public evaluation. The paper reports per-domain scores (e.g., Physics 49.3% for GPT-5-high) based on these 10 tasks. With 10 tasks, an anomalous task can shift domain-level scores by several percentage points. The standard deviations reported in Table 3 (±2.0–2.8% for Physics, ±2.0–2.1% for Chemistry) reflect within-task generation variance but not the between-task sampling variance. The true uncertainty in domain-level estimates is larger than the reported standard deviations suggest, since the 10 tasks are themselves a sample from the universe of possible professional tasks in each domain. The paper does not estimate this component of uncertainty.

Does the document truncation undermine ecological validity?

The paper acknowledges that grounding documents were truncated from an average of 141.7 pages per task to 24.69 pages (an ~83% reduction) to fit LLM context windows, noting this "will make tasks substantially easier." This is a genuine limitation: the benchmark as deployed is a simplified version of the intended professional task. The paper frames this as a temporary compromise necessary for current context windows, but this means that performance on ProfBench should not be interpreted as "can perform the full professional task" — only "can perform the task given pre-selected, most-relevant document excerpts." The original-task-to-truncated-task gap is not measured. A useful future experiment would test a subset of models on original-length documents (even if most models fail due to context limits) to quantify this gap.

Are there missing baselines for the Report-Generator evaluation?

The paper evaluates each model independently but does not include obvious baselines that would help interpret absolute performance: (1) Human professional performance: what score would the domain-expert annotators themselves achieve on their own tasks under the same conditions (with the same truncated documents)? This would establish the ceiling. (2) Retrieval-only baseline: a system that simply extracts and quotes relevant passages from the grounding documents without any synthesis or reasoning. This would establish the floor for Extraction/Recall criteria. (3) Document-only baseline: the model receives only the grounding documents without the task prompt, to measure whether criteria can be satisfied by serendipitous document content. Without these, the reported scores (e.g., 65.9% for GPT-5-high) are difficult to interpret in absolute terms.

What the experiments do convincingly demonstrate:

ProfBench is substantially harder than exam-style benchmarks for frontier models. The 65.9% ceiling versus 94.6% (AIME 25) and 87.0% (GPQA-Diamond) is a genuine gap that cannot be explained by evaluation format alone.
LLM-Judge bias is measurable, varies substantially across models, and can be mitigated through model selection and adaptive reasoning effort. Table 2 provides comprehensive evidence, and the composite judge's 0.5% Bias-Index versus alternatives with 5%+ Bias-Index demonstrates that low-bias judges are achievable without sacrificing accuracy.
Cost and quality can be simultaneously optimized through adaptive reasoning effort and optimal sample allocation, not traded off. The GPT-OSS-120B composite judge matches the best proprietary judge at 1.68% of cost, and optimal allocation reduces generation cost 4× at equivalent precision.
Grounding documents are essential for performance on professional tasks, and web search only partially substitutes. The 9.4–11.9 point drop when removing documents, with only partial recovery from search, confirms that curated reference materials provide value beyond what general web access can recapture.
Model scaling within families shows diminishing returns for both judging and generation on professional-domain tasks. The consistent pattern of small gains from medium-to-large versus large gains from small-to-medium suggests that frontier models are approaching a capability plateau on these tasks that additional parameters alone may not overcome.

6. Limitations and Trade-offs

The 2048-Sample Difficulty Estimation Cost Is Not Amortized Into the Headline Efficiency Gains

The assumption or constraint. The paper's compute-optimal framework for test-time compute allocation fundamentally depends on knowing each prompt's difficulty before deciding how to spend the inference budget. The method for estimating difficulty — generating 2048 samples per question and averaging either ground-truth correctness (oracle bins) or PRM final-answer scores (predicted bins) — is explicitly acknowledged as expensive. The authors state in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

This means the headline $4\times$ efficiency gains (Figures 4 and 8, Section 5.3 and 6.2) are computed after difficulty is already known, without including the cost of learning it.

The consequence. In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter. Generating 2048 samples per question to estimate difficulty consumes more compute than the largest test-time budgets studied (256–512 generations). The reported $4\times$ figure is therefore best understood as an upper bound on achievable efficiency rather than a realized deployment gain. If difficulty estimation cost were amortized across the budget, the net efficiency improvement over best-of-N would shrink — potentially substantially, depending on how many questions share the same estimated difficulty distribution. For one-off queries, the overhead makes the approach impractical; for repeated queries from a stationary distribution, amortization helps but the paper provides no analysis of how many queries are needed to recover the initial investment.

What evidence exists in the paper. The cost is visible in the method description (Section 3.2): "For each question in the test set, the authors sample 2048 complete solutions from the base model and compute the pass@1 rate." The paper reports that predicted difficulty bins (using PRM scores without ground-truth labels) perform nearly as well as oracle bins (Figures 4 and 8, curves largely overlap), confirming that the approach works without answer access — but the 2048-generation cost remains in both cases. The paper does not report or ablate the performance of difficulty estimation with fewer samples (e.g., 16, 64, 256), which would directly quantify the cost-accuracy tradeoff.

Mitigation status. The authors flag this as "a key avenue for future work" (Section 3.2) and suggest "pretraining or finetuning models to directly predict difficulty of a question" (Section 8). No such model is developed or evaluated in the paper. The limitation is acknowledged but entirely unresolved — the framework's practical deployability hinges on future work that has not been done.

Hard Problems Show Near-Zero Improvement Regardless of Compute Budget

The assumption or constraint. The entire test-time compute scaling framework studied in this paper — PRM-guided search, iterative revisions, and compute-optimal combinations thereof — operates on the assumption that the base model produces correct solutions at some non-trivial rate. When the base model's pass@1 is near zero, no amount of search or revision can help because there are no correct solutions in the proposal distribution to find or refine.

The consequence. Across all methods studied — search, revisions, and their compute-optimal combinations — the hardest questions (difficulty bin 5) show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets up to 256 generations. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%, falling below the $14\times$ larger model's greedy performance at all $R$ values.

This means test-time compute can amplify existing capability but cannot create it. For problems genuinely outside the base model's capability range — those requiring knowledge, reasoning patterns, or problem-solving strategies not present in the pretraining distribution — the framework offers no path forward. Pretraining remains the only viable intervention for such problems. This is a fundamental capability bound, not an engineering limitation: it follows from the framework's architecture (search and revision operate on the base model's output distribution) and cannot be resolved by better hyperparameter tuning or improved allocation policies.

What evidence exists in the paper. The bin 5 results are consistently and starkly negative across every experiment. Figure 3 (right) shows beam search and best-of-N both at 1–3% for bin 5 at all budgets. Figure 9 shows the bin 5 compute-optimal scaling line flat near zero and below the $14\times$ larger model at all $R$ values. The paper is transparent about this:

"On the hardest questions (bin 5), no method makes meaningful progress — the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated." (Section 5.3)

The FLOPs-matched analysis in Section 7 reinforces this: hard questions show a −52.9% relative disadvantage for test-time compute versus the larger model at $R \gg 1$ .

Mitigation status. The paper acknowledges this limitation explicitly (Section 7 takeaway box, Section 8) but offers no mitigation beyond the observation that pretraining is necessary for these problems. This is not a weakness of the paper — it is a honest characterization of the boundary conditions — but it means the approach is fundamentally inapplicable to the hardest problem tier, which may be precisely the tier where assistance is most valuable.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect, followed by a correct target (Section 6.1). During training data construction, for each question, the authors identify correct and incorrect responses from 64 parallel samples, then construct multi-turn trajectories of 0–4 incorrect answers (with the last incorrect answer selected to have minimal character-level edit distance to the correct answer) followed by the correct answer. The model is fine-tuned with SFT only on the correct answer tokens.

The consequence. At test time, when the revision model produces a correct answer at some step in the chain, it may encounter this correct answer in its own context during subsequent revisions. Since it was never trained on trajectories containing correct in-context answers, it has no learned behavior for what to do when the current answer is already correct. The paper reports (Section 6.1):

"approximately 38% of correct answers get converted back to incorrect ones using a naive approach"

This means that as revision chains grow longer, the model has a substantial probability of undoing its own correct work. Without mitigation, longer chains would not monotonically improve and could actively degrade performance.

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1, though the paper does not provide a detailed breakdown of when reversions occur (e.g., early vs. late in chains, by difficulty bin). The mitigation — using majority voting or verifier-based selection across the entire chain rather than always taking the final revision — is what enables the sequential revision gains shown in Figure 6. Without within-chain selection, the revision model's per-step accuracy gains (Figure 6, left: from ~18.2% at step 1 to ~24–25% by steps 15–20) would be offset by correct-to-incorrect reversions in later steps.

Mitigation status. The paper mitigates this with a practical fix — selecting the best answer from any point in the revision chain using majority voting or verifier scoring, rather than taking the final output — but this is a patch, not a solution. The underlying problem (the model has no training signal for recognizing when its current answer is correct) remains. The paper does not explore training the model on trajectories that include correct in-context answers with a "stop revising" signal, which would be a more principled solution. The ReST $^{EM}$ experiment in Appendix K (Figure 16) further demonstrates the fragility of revision training: attempting to optimize the revision model with on-policy RL caused performance to degrade substantially with sequential revisions, suggesting the revision approach is sensitive to training methodology in ways that are not fully understood.

Verifier Over-Optimization Limits Scaling and Is Not Solved by the Compute-Optimal Policy

The assumption or constraint. All search-based test-time compute methods rely on the Process Reward Model (PRM) to score partial solutions and guide search. The PRM is a learned model trained with Monte Carlo rollout supervision (Section 5.1, Appendix D) and is imperfect — it makes errors in assessing whether a partial solution is on track to a correct final answer. As search becomes more aggressive (more beams, more lookahead steps, higher budgets), it exploits these imperfections, finding solutions that score highly under the PRM but are actually incorrect.

The consequence. This over-optimization is the primary bottleneck preventing unbounded improvements from additional test-time compute. The evidence is concrete and multi-faceted: beam search degrades easy-problem performance at high budgets (Figure 3, right — bin 1 accuracy decreases from ~78% to ~77% as budget goes from 4 to 256 while best-of-N improves from 68% to 88%); lookahead search — the strongest optimizer — paradoxically performs worst overall (Figure 3, left — lookahead search with $k=3$ underperforms all methods at the same generation budget because its extra per-step cost reduces the effective number of beams explored); and qualitative examples in Appendix M (Figures 29 and surrounding) show search producing degenerate outputs — low-information repetitive steps, overly short 1–2 step solutions — that score highly under the PRM.

The compute-optimal policy mitigates this by routing easy problems away from aggressive search toward best-of-N (which over-optimizes less), but it does not solve the underlying problem. On medium-difficulty problems where beam search is deployed, over-optimization still limits the scaling ceiling — the beam search curves in Figure 3 flatten and sometimes decline well before the budget is exhausted. The policy is essentially a way to stay below the over-optimization threshold per difficulty level, not a way to raise that threshold.

What evidence exists in the paper. Figure 3 (right) provides the clearest evidence: beam search ( $M=4$ ) on bin 1 (easiest) shows accuracy declining as budget increases, while best-of-N weighted shows monotonic improvement. Figure 3 (left) shows lookahead search (the strongest PRM optimizer) performing worst overall at matched generation budgets. Appendix M provides qualitative examples of degenerate search outputs. The paper explicitly identifies this as the central limitation of search-based methods:

"The degradation at high budgets is attributed to over-optimization of the PRM — search finds solutions that score highly under the PRM but are actually incorrect." (Section 5.3)

Mitigation status. The compute-optimal policy mitigates the symptom (by avoiding aggressive search on easy problems) but does not address the cause (PRM imperfection). The paper acknowledges this in Section 8: "improving verifier robustness is the key bottleneck for further scaling test-time compute, not improving search algorithms." No concrete approach to verifier improvement is developed or evaluated. Possible directions — adversarial training, ensemble verification, KL-constrained search — are mentioned only as future work. The current results are therefore specific to the verifier quality achievable with the Monte Carlo rollout training procedure, and the optimal policy thresholds would shift if verifier quality improved.

Single Benchmark and Single Model Family — Generalization Is Unverified

The assumption or constraint. All experiments in this paper use the MATH benchmark (Hendrycks et al., 2021) with PaLM 2-S* (Codey) as the base model. The MATH dataset consists of 12,000 training and 500 test questions drawn from high-school competition mathematics. PaLM 2-S* is a specific model from a specific model family (Google's PaLM 2 series), with a particular scale, training data composition, and capability profile. The paper states (Section 4) that the authors "believe this model is representative of the capabilities of many contemporary LLMs," but this is an assertion, not a finding.

The consequence. Several aspects of the paper's findings could be specific to the MATH-PaLM 2-S* combination and may not transfer to other settings. The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution — a model with different calibration, different error patterns, or different reasoning styles might exhibit different difficulty-dependent scaling curves, different over-optimization thresholds, and different optimal allocation policies. The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families. The finding that beam search helps on medium problems but hurts on easy ones might not replicate if the base model's capability-to-difficulty mapping differs.

The MATH benchmark itself is narrow: competition mathematics with exact answer verification. The difficulty-dependent patterns (beam search over-optimizing on easy problems, revisions helping on easy problems, search helping on medium problems) may not generalize to other reasoning domains — code generation (where correctness is verified by unit tests, not a learned PRM), logical reasoning, scientific question answering, or tasks requiring factual knowledge rather than deductive inference. The paper's explicit focus (Section 4) on "tasks where the model already possesses the necessary knowledge and the challenge is drawing complex inferences" means the framework is by design restricted to inference-heavy domains and may not apply to knowledge-recall-heavy tasks.

What evidence exists in the paper. The paper provides no out-of-domain or out-of-model-family experiments. The only model used as base is PaLM 2-S*. The only benchmark used is MATH. The $14\times$ larger model used in the FLOPs-matched comparison is from the same model family (implied by the parameter scaling approach), so even the pretraining-vs-inference comparison is within-family. The paper does not test on other math benchmarks (GSM8K, AMC, AIME), other reasoning domains, or other model families. The claim that PaLM 2-S* is "representative" is unsubstantiated.

Mitigation status. The authors acknowledge this as a limitation implicitly through scope statements (Section 4: "we believe this model is representative") and explicitly in Section 8, where extending to other domains and model families is listed as future work. The limitation is not mitigated in the current paper — all conclusions are conditional on the MATH-PaLM 2-S* setting, and the extent to which they generalize is unknown. A practitioner considering applying compute-optimal test-time scaling to a different model or task domain would need to replicate the core analyses (difficulty binning, strategy sweep, FLOPs-matched comparison) to determine whether the same patterns hold.

Search and Revisions Are Studied Independently, Not Combined

The assumption or constraint. The paper studies two complementary mechanisms for test-time compute — PRM-guided search (modifying the verifier/selection side) and iterative revisions (modifying the proposal distribution side) — but treats them as independent pipelines evaluated separately. Section 8 explicitly acknowledges:

"we did not experiment with PRM tree-search techniques in combination with revisions"

All compute-optimal results (Figures 4 and 8) are within a single mechanism: search strategies are compared against each other, and revision strategies (sequential vs. parallel ratios) are compared against each other, but there is never a configuration where beam search guides which revision branches to pursue, or where the revision model serves as the proposal distribution within a PRM search tree.

The consequence. The paper provides a lower bound on what test-time compute can achieve, not an upper bound. The two mechanisms have complementary strengths: revisions improve the quality of generated candidates (the model produces better initial attempts and refines them), while PRM search improves the selection among candidates (finding the best among generated options). A combined system — using the revision model as the proposal distribution within beam search, or using the PRM to decide when to revise versus when to restart — could yield performance beyond either mechanism alone, particularly on medium-difficulty problems where both mechanisms show individual benefits (Figure 3 right, bin 3: beam search helps; Figure 7 right, bins 3–4: intermediate sequential-to-parallel ratios are optimal).

The independence of the two pipelines also means the paper cannot answer questions about relative cost-effectiveness: for a given compute budget, is it better to invest in search, revisions, or some combination? The compute-optimal policy selects among strategies within each pipeline but never allocates budget across pipelines. This is a significant gap because a practitioner deploying test-time compute would need to make exactly this decision: given $N$ generations, how many should be spent on beam search exploration and how many on sequential revision refinement?

What evidence exists in the paper. No experiments combine search and revisions. The closest the paper comes is the FLOPs-matched comparison (Section 7, Figure 9), which shows separate results for search-based and revision-based compute-optimal scaling against the larger model, but never a combined configuration. The unifying framework in Section 2 (proposal distribution vs. verifier modifications) provides the conceptual scaffolding for combination, but the empirical work does not realize it.

Mitigation status. The authors explicitly flag this as future work (Section 8). The limitation is significant because it means the paper's central claim — that compute-optimal test-time scaling can improve efficiency by $4\times$ — may understate what is achievable with a fully integrated system combining both mechanisms. Conversely, combining them might introduce new failure modes (e.g., the revision model's output distribution might interact poorly with the PRM, which was trained on base model outputs — a distribution shift issue the paper already observes in Appendix J, Figure 15a). Without empirical evidence, the net effect of combination is unknown in both direction and magnitude.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose a new model architecture or training algorithm — it proposes a new evaluation paradigm for professional-domain LLM capabilities, and in doing so, it shifts the conversation around what benchmarks should measure and how they should be built. The shift is substantive but bounded: it is a methodological reframing of rubric-based evaluation rather than a paradigm-shattering breakthrough, but the specific choices the paper makes — treating bias as a co-equal metric with accuracy, exploiting reasoning-effort heterogeneity for cost reduction, and formalizing sample allocation as a variance-minimization problem — collectively raise the bar for what constitutes a well-engineered evaluation benchmark in the LLM era.

The most consequential conceptual contribution is the elevation of judge fairness from a qualitative concern to a quantitative, optimizable metric. The LLM-as-judge literature has known about self-enhancement bias since Zheng et al. (2023), but the field's response has been largely acknowledgment-based: papers note the bias exists, sometimes report per-model score differences, and then proceed with evaluation using whatever judge is most convenient or accurate. ProfBench makes a different move: it defines overall judge quality as Macro-F1 minus Bias-Index, operationalizing fairness as a direct penalty on the metric that determines judge selection. This means a judge with 88% agreement but severe provider bias (say, a 10-point range in per-model score inflation) is ranked below a judge with 80% agreement and negligible bias — an explicit assertion that rank-distorting unfairness invalidates a measurement instrument regardless of average accuracy.

The practical impact of this reframing is visible throughout the paper's judge selection process. The composite GPT-OSS-120B judge with adaptive reasoning effort achieves 78.2% Overall not because it has the highest Macro-F1 (Gemini-2.5-Pro edges it out at 79.2%), but because its Bias-Index of 0.5% is half of Gemini-2.5-Pro's 1.0%. The paper is effectively arguing that a slightly less accurate but substantially fairer judge produces more trustworthy model rankings. Whether the broader community adopts this framing — and whether the equal-weighting of Macro-F1 and Bias-Index in the Overall metric is the right tradeoff — remains to be seen, but the paper has made the question impossible to ignore for anyone building on its methodology.

The second landscape-shifting finding is more empirical than methodological: increasing reasoning effort in LLM-Judges can increase self-enhancement bias rather than reduce it. This is visible consistently across the GPT-5, o3, and GPT-OSS families in Table 2. GPT-5-minimal shows per-model biases of −0.5%, −5.6%, −5.0% (Bias-Index 5.1%), while GPT-5-high shows +1.0%, −0.8%, −1.3% (Bias-Index 2.3%) — an improvement but still non-trivial. o3-low shows +3.8%, +1.5%, +2.6% (Bias-Index 2.3%) while o3-high shows +2.0%, +0.5%, +0.8% (Bias-Index 1.5%). The pattern is not monotonic — for some models, low reasoning effort gives the lowest bias (GPT-5-nano-medium at 72.7% Overall vs. high at 71.9%), while for others, high reasoning effort reduces bias — but the key insight is that more thinking does not automatically produce fairer judging. This creates a tension that future judge-design work must navigate: the configuration that maximizes agreement with humans (often higher reasoning effort) may not be the configuration that minimizes provider bias, and the optimal operating point depends on the relative importance assigned to each.

The third contribution is the demonstrated viability of domain-diverse, cost-accessible rubric-based evaluation. Prior rubric-based benchmarks (PaperBench at $1,320 per evaluation, HealthBench at$ 300) were effectively restricted to well-resourced labs. ProfBench's composite judge at $0.70 and report-generator evaluation at$ 1.35– $112 (depending on model) makes systematic experimentation on open-ended professional tasks feasible for researchers with modest compute budgets. This is not merely a convenience feature — it is infrastructure that enables iterative improvement. If evaluating a single training run costs$ 1,320 in judge fees alone, experimentation slows to a crawl. At $0.70, rapid iteration becomes possible, which in turn makes it more likely that the research community will invest in improving model performance on these tasks. The paper's emphasis on cost is thus strategic: cheap evaluation drives fast progress, and fast progress requires cheap evaluation.

The paper also resolves a specific tension in the rubric-based evaluation literature. DeepResearch-Bench (Du et al., 2025) claimed to evaluate PhD-level tasks across multiple domains using synthetically-generated criteria, but the paper's critique — that synthetic criteria create systematic bias toward the generating model, resulting in Gemini-2.5-Pro scoring >97% — identifies a failure mode that the field might otherwise have overlooked. ProfBench's insistence on human-written, human-verified rubrics with documented inter-annotator agreement (Fleiss' κ = 0.912) establishes a quality standard that future rubric-based benchmarks will be measured against. The message is clear: if your evaluation criteria are generated by the same models you're evaluating, your benchmark measures self-consistency, not task competence.

Finally, the paper's optimal allocation formulation (Appendix G) introduces a methodological tool that other benchmark designers can adopt directly. The observation that per-task score variance is heterogeneous, that uniform sampling is therefore inefficient, and that dynamic programming can find allocations that reduce variance at fixed budget — this is not specific to ProfBench. Any benchmark where per-task performance variance can be estimated from pilot runs can apply the same technique to reduce evaluation cost or improve precision. The paper demonstrates that this produces consistent improvements over heuristic baselines (Figure 5), establishing it as a lightweight addition to the benchmark designer's toolkit.

Follow-Up Research This Work Enables

Calibrating the relationship between reasoning effort and self-enhancement bias. The paper observes that increasing reasoning effort sometimes increases bias and sometimes decreases it, but provides no systematic explanation. A targeted follow-up would ablate this relationship across a wider range of models and tasks: for a fixed set of judge models (GPT-5, o3, Gemini-2.5-Pro, Claude-Sonnet-4), sweep reasoning effort continuously (not just high/medium/low/minimal buckets) and measure both Macro-F1 and Bias-Index at each level. The hypothesis to test is whether there exists a "sweet spot" reasoning budget where bias is minimized, and whether this sweet spot varies predictably with model scale, training data composition, or domain. If such sweet spots exist and can be predicted from model characteristics, adaptive reasoning effort could be tuned more precisely than the paper's domain-based heuristic. If no consistent pattern emerges — if the bias-reasoning relationship is idiosyncratic to each model — that would itself be an important negative result, suggesting that bias mitigation cannot be automated through reasoning-effort tuning and requires other interventions.

Validating the Bias-Index against cross-provider model rankings. The Bias-Index is computed over exactly three response-generating models (o3, Grok4, R1-0528) that were selected as the best available in July 2025. A natural stress test is to compute the Bias-Index over a different set of response models — for instance, GPT-4.1, Claude-3.5-Haiku, and Gemini-2.5-Flash — and check whether the judge rankings produced by the two Bias-Indices are consistent. If a judge that appears fair under the original three-model set shows large bias under an alternative set, the Index is sensitive to the specific reference models chosen and may not capture general provider fairness. If rankings are stable across different reference sets, the Index is robust and can be adopted as a standard metric. This experiment requires annotating a modest number of additional response-criterion pairs (the three alternative models' responses on a subset of tasks) with human labels, which the paper's annotation pipeline could support at reasonable cost given the demonstrated inter-annotator reliability.

Combining PRM-guided search with iterative revisions for open-ended professional tasks. The paper you're analyzing studies PRM search and iterative revisions as independent test-time compute mechanisms. A natural and currently unaddressed question is whether a revision model fine-tuned on professional-domain outputs (using the ProfBench training data construction recipe) could serve as the proposal distribution within a PRM-guided beam search, and whether the combination yields gains beyond either mechanism alone on open-ended rubric-scored tasks. The experiment would: (1) fine-tune a revision model on ProfBench training data using the edit-distance-based pairing approach; (2) train a PRM on the same domain using Monte Carlo rollout supervision (since ProfBench criteria provide binary fulfillment labels, these could substitute for the "correct final answer" signal used in the MATH PRM training); (3) run beam search against the PRM using the revision model as the proposal distribution at each step; and (4) compare against beam search with the base model, sequential revisions alone, and best-of-N baselines. The hypothesis from the complementary-strengths framework is that revisions improve candidate quality on tasks where the model has partial competence (the medium-difficulty tasks in ProfBench's distribution), while PRM search improves selection among candidates — and the combination should outperform either alone, especially on tasks in the middle difficulty range where both mechanisms individually show benefit.

Measuring the document-truncation-to-original-document performance gap. The paper acknowledges that truncating grounding documents from an average of 141.7 pages per task to 24.69 pages "will make tasks substantially easier" but does not quantify this gap. A critical follow-up would evaluate a subset of models (at minimum, GPT-5-high, o3, and the best open-weight model) on the original, un-truncated documents for a sample of 5–10 tasks, even if most models fail due to context window limitations. The metric of interest is the score degradation: if models that achieve 65% on truncated documents drop to 30% on original documents, the benchmark's ecological validity as a measure of professional-task competence is substantially undermined — it would be measuring performance on a simplified proxy rather than the intended task. If the degradation is modest (say, <10 points), the truncation is a reasonable practical compromise. This experiment also serves as a forward-looking calibration: as context windows expand, the benchmark can transition to original documents, and this baseline quantifies how much harder the task will become. The paper's API cost reporting provides a natural framework for estimating the cost of such an experiment.

Establishing human professional performance baselines on ProfBench tasks. The paper reports that the three reference models (o3, Grok4, R1-0528) fulfill 45–52% of criteria as judged by human annotators, and that GPT-5-high achieves 65.9% via the composite judge — but provides no human performance baseline. What score would the domain-expert annotators themselves achieve on their own tasks under the same conditions (same truncated documents, same time constraints if applicable)? This is methodologically challenging because the annotators designed the tasks and criteria, creating a conflict of interest, but a feasible approach would be to recruit a separate set of domain experts (not the original annotators) to complete a subset of tasks, then have the original annotators score these human-generated responses against the same criteria. The result would establish a performance ceiling: if human experts achieve 85–90%, the remaining gap to GPT-5's 65.9% is substantial and meaningful; if humans achieve only 70–75%, the benchmark may be unrealistically difficult or the criteria may be too stringent. This baseline is essential for interpreting the absolute difficulty of ProfBench — without it, the 65.9% number is unanchored.

Stress-testing optimal allocation under distribution shift. The paper's dynamic programming solution for sample allocation uses variance estimates ( $v_i$ ) computed from four representative models (Gemini-2.5-Pro, Gemini-2.5-Flash Thinking, o3, o4-mini) and then applies the resulting allocation to all subsequent model evaluations. This assumes that per-task variance is a property of the task (and the evaluation rubric), not of the model being evaluated. A stress test would compute separate variance estimates for a held-out set of models not used in the allocation design (e.g., GPT-5-nano, Claude-3.5-Haiku, Llama-4-Scout) and check whether the optimal allocation derived from the original four models remains near-optimal for these new models. If the per-task variance ordering is consistent across model families — if Chem-9 is high-variance and Chem-4 is low-variance regardless of which model generates the responses — then the fixed-allocation approach is robust. If variance patterns are model-specific (some models have high variance on different tasks than others), then the allocation would need to be recomputed per model family, increasing overhead. This experiment would determine whether the cost savings from optimal allocation generalize or are contingent on the models used for variance estimation.

Practical Applications and Downstream Use Cases

Model selection for professional-domain deployment. Organizations evaluating LLMs for deployment on professional tasks — financial analysis at an investment bank, scientific literature synthesis at a pharmaceutical company, consulting-style deliverable generation at a strategy firm — currently lack benchmarks that measure the specific capabilities these tasks require. ProfBench provides a directly applicable evaluation: a firm considering whether to deploy GPT-5, o3, or an open-weight alternative for financial report generation can consult Table 3 and see that GPT-5-high achieves 63.7% on Finance MBA tasks while the best open-weight model (GPT-OSS-120B) achieves 45.5%, a gap of 18.2 percentage points. If the firm's use case is Consulting-focused, the gap narrows to 10.6 points (80.0% vs. 69.4%). The per-domain breakdowns enable cost-benefit analysis: a firm might choose GPT-5 for high-stakes financial analysis where the 18.2-point gap justifies the API cost, but deploy an open-weight model for consulting tasks where the gap is smaller and cost sensitivity is higher. The paper's cost reporting (GPT-5-high at $112.34 per full evaluation vs. GPT-OSS-120B at$ 1.35) makes this tradeoff quantitatively explicit.

Judge selection for automated evaluation pipelines. Teams building automated evaluation pipelines for open-ended model outputs — whether for RL reward signals, model selection during development, or production monitoring — face the judge selection problem that ProfBench's Table 2 directly addresses. The finding that GPT-4.1-mini achieves 74.9% Overall (Macro-F1 76.4%, Bias-Index 1.5%) at $2.26, while GPT-4.1 achieves only 0.5 points more at 5× the cost ($ 11.31), provides an actionable recommendation: for most use cases, the mini variant is the cost-effective choice. The Bias-Index column further enables filtering: a team concerned about provider fairness would avoid Llama-3.1-405B-Instruct (5.4% Bias-Index) or Claude-Sonnet-4 (5.0%) regardless of their Macro-F1 scores, and would instead select from the low-bias options like Gemini-2.5-Flash Thinking (0.3% Bias-Index, $7.92) or the composite GPT-OSS-120B judge (0.5% Bias-Index,$ 0.70). This is a concrete decision rule supported by the paper's data: for evaluation pipelines where model rankings matter, filter judges by Bias-Index first, then optimize for Macro-F1 per dollar.

Cost-efficient benchmark iteration during model development. A research team developing a model fine-tuned for professional-domain tasks needs to evaluate their model repeatedly during training — after each hyperparameter change, data mixture adjustment, or training run. At $1,320 per evaluation (PaperBench) or$ 300 (HealthBench), comprehensive evaluation is restricted to final checkpoints. At ProfBench's costs — $0.70 for the judge plus generation costs of$ 1–$112 depending on the model being evaluated — the team can afford to evaluate every intermediate checkpoint, enabling learning curves that reveal when improvements plateau, which domains benefit from which training interventions, and whether performance is improving uniformly or only on specific criterion types. The per-criterion-type accuracy breakdowns (Extraction/Recall, Reasoning, Style) further enable diagnostic analysis: if a training intervention improves overall score but degrades Extraction accuracy, the team can investigate whether the model is becoming better at reasoning at the expense of factual precision. This diagnostic granularity, combined with low cost, transforms ProfBench from a leaderboard benchmark into a development tool.

Calibrating expectations for domain-specific LLM capabilities. Organizations across industries are being asked to assess whether current LLMs are "ready" for professional-domain tasks. ProfBench provides a concrete, quantified answer: even the best model (GPT-5-high) satisfies only 65.9% of professional-quality criteria on average, with Physics PhD tasks at 49.3% and Finance MBA at 63.7%. These numbers — substantially below the 85–95% figures that frontier models achieve on exam-style benchmarks — provide an evidence-based counterargument to claims that LLMs can already replace professional analysts. At the same time, the Consulting MBA score of 80.0% suggests that certain professional domains may be closer to viable automation than others. The per-domain, per-criterion-type breakdowns enable nuanced capability assessments rather than blanket claims about "LLM competence." For a consulting firm evaluating whether to integrate LLMs into their workflow, the message is: the technology handles structured business analysis reasonably well (80% criterion fulfillment) but struggles with quantitative financial modeling (63.7%) and advanced physical sciences (49.3%) — deploy accordingly.