ArXiv: 2604.23099

🎯 Pitch

Evaluating a new generative AI model to within Β±1% accuracy can require as few as 1–2 carefully chosen test inputs when using pre-trained performance priors from other models. ProEval achieves this 8–65Γ— sample efficiency gain by framing both aggregate metric estimation and targeted failure discovery under a unified Bayesian framework, actively selecting or synthesizing the most informative prompts to evaluate.

1. Executive Summary

ProEval introduces a proactive evaluation framework that jointly addresses performance estimation and failure discovery for generative AI models by treating both objectives through a unified Bayesian lens β€” estimating aggregate metrics via Bayesian quadrature (BQ) and identifying failure regions via superlevel set sampling β€” while leveraging transfer learning to construct strong Gaussian process (GP) priors from historical evaluation data across models. Tested across reasoning benchmarks (GSM8K, StrategyQA, SVAMP, MMLU), safety datasets (ToxicChat, DICES-350, DIVE), and 16 models including GPT-5, Gemini 2.5 Flash, and Claude 4.5 Opus, ProEval achieves 8–65Γ— sample efficiency gains for performance estimation (reaching within Β±1% MAE using as few as 1–2 evaluated inputs through score-feature-based transfer) and discovers 2–5Γ— more diverse failure cases than competitive baselines through a hierarchical topic-aware synthesis algorithm (TSS) that transposes anchor failure patterns across semantically diverse topics, establishing that strong transferable priors enable dramatic evaluation efficiency improvements without requiring ground-truth labels on the target benchmark.

2. Context and Motivation

The Core Problem: Evaluating Generative AI Has Become Unsustainably Expensive

This paper tackles a practical crisis in generative AI development: the infrastructure for evaluating models has not kept pace with the proliferation of models and benchmarks that need evaluation. The authors paint a picture where evaluation, once a straightforward afterthought in the machine learning pipeline, has become the bottleneck that determines how quickly organizations can iterate on model development, verify safety compliance, and make deployment decisions.

The problem has several interdependent dimensions that compound one another:

Slow and costly inference. Evaluating a generative AI model means running it on test inputs β€” thousands or tens of thousands of them. Unlike traditional ML where inference is measured in microseconds, generating a single output from an LLM or VLM can take seconds. Multiply this by the size of modern benchmarks (GSM8K has 8,500 test problems; MMLU covers 57 subjects with thousands of questions) and by the number of models under consideration, and the wall-clock time balloons to days or weeks. The paper notes (Section 1) that "Running comprehensive evaluations across multiple models and datasets can cost thousands of dollars and require days of compute time."

Expensive evaluation signals. Beyond generation cost, there is the cost of rating the outputs. For objective benchmarks with ground-truth answers (math problems, multiple choice), this is cheap β€” but for safety evaluation, alignment assessment, and open-ended tasks, reliable rating requires either human annotators (slow, expensive, and subject to fatigue and inconsistency) or LLM-based "judges" (which are themselves expensive to run and may introduce systematic biases). The paper references this explicitly: "reliable assessment often requires costly human raters or expensive LLM-based 'judges'" (Section 1). Every additional test case evaluated adds to this cost.

Frequent re-evaluation during development. The costs described above are incurred not once but repeatedly. During model development, teams must run evaluation suites after every significant change β€” architecture modifications, data mixture adjustments, fine-tuning runs, safety interventions. The paper underscores this point: "during GenAI model development and quality iteration, development teams must frequently run numerous evaluation tasks against the same benchmarks, even for the most minor changes at different scales" (Section 1). What appears as a one-time evaluation cost is actually recurring, multiplying the practical burden.

The resulting coping strategy is deeply suboptimal. Faced with these costs, the paper observes that practitioners commonly resort to downsampling test data β€” evaluating on a random or hand-picked subset rather than the full benchmark. The authors explicitly call this out as problematic: "researchers and practitioners often resort to downsampling test data... which may yield less accurate estimates and fail to uncover rare but critical failure cases" (Section 1). A random subset might miss the 2% of inputs where the model exhibits a catastrophic safety failure, precisely because those failures are rare. Downsampling trades coverage for speed in a way that systematically biases evaluation toward the common case and away from the tail risks that matter most for deployment.

Why This Problem Matters Beyond Cost Savings

The paper's framing goes beyond mere efficiency. The inability to evaluate thoroughly and frequently has downstream consequences that affect the entire ecosystem:

Model selection under incomplete information. When an organization chooses between GPT-5, Gemini 2.5 Flash, and Claude 4.5 Opus for a specific application, they need to know which model performs best not just on average but on the specific types of inputs their users will generate. Evaluating all candidates on all relevant benchmarks is often infeasible, leading to decisions based on partial evidence or extrapolation from leaderboard scores that may not reflect real-world performance distributions.

Safety assurance with limited testing budgets. Safety evaluations are particularly sensitive to sampling: a model that is benign on 99% of prompts but generates harmful content on the remaining 1% needs thorough testing to surface those failures. A cost-constrained evaluation that samples only 100 prompts might entirely miss these edge cases, creating a false sense of security. The paper emphasizes that current methods "fail to uncover rare but critical failure cases" β€” a statement about statistical power, not just efficiency.

The proliferation of benchmarks compounds the problem. The paper cites NeurIPS (2025) to note that new benchmarks are being created at an accelerating rate. Each new benchmark represents a distinct axis of capability or safety that needs to be evaluated, but the total evaluation budget does not scale linearly with the number of benchmarks. Organizations face a triage problem: which benchmarks to evaluate, how many samples per benchmark, and how frequently to re-evaluate β€” all under a fixed time and cost constraint.

Amplification of development feedback loops. If thorough evaluation is expensive, developers receive less signal about which changes improve model behavior and which break things. This slows the entire development cycle: bugs are discovered later, regressions go undetected longer, and the feedback between model modification and performance measurement lengthens. The paper frames ProEval as directly addressing this: "It accelerates the iteration cycle of GenAI development and provides deeper insights into model failures" (Section 1).

Where Existing Approaches Fall Short

The paper situates itself against two broad categories of prior work: efficient evaluation (which aims to estimate aggregate metrics with fewer samples) and failure discovery / red teaming (which aims to find specific inputs where models fail). Both literatures have produced useful methods, but the paper identifies systematic limitations in each.

Static Benchmark Pruning: Efficient but Target-Agnostic

Methods like TinyBenchmarks (Polo et al., 2024), MetaBench (Kipnis et al., 2025), Anchor Points (Vivek et al., 2024), and DISCO (Rubinstein et al., 2025) identify a fixed, representative subset of test questions that, when evaluated, approximates the performance on the full benchmark. The idea is to pick questions that maximally discriminate between models or that represent "prototypical" difficulty levels.

The paper identifies a fundamental limitation (Appendix A):

"These approaches are inherently target-agnostic: they assume the failure modes of the target model perfectly align with the source models used to select the subset. If the target model exhibits idiosyncratic failures (e.g., due to a different architecture or safety tuning), static subsets fail to capture them."

The reasoning is subtle but important. Static pruning methods select questions based on how historical models behave. If a new model has been fine-tuned on a specific domain β€” say, it now handles multi-hop reasoning questions well but fails on basic arithmetic due to a vocabulary change β€” the pruned subset might contain questions that are now easy for it and omit questions that are now hard. The subset was optimized for the average model, not this specific model. ProEval avoids this by using historical model performance as a prior that can be updated as new data from the target model arrives, rather than a rigid selection that cannot adapt.

The paper also critiques Fluid Benchmarking (Hofmann et al., 2025) for a related but distinct limitation: while it dynamically selects questions based on estimated model ability (using Item Response Theory), it "reduces a model's complex, multi-dimensional capabilities to a single scalar ability score" and is "restricted to static item poolsβ€”meaning it cannot synthesize novel test cases." ProEval retains the multi-dimensional nature of model performance (through the GP's joint modeling of all test inputs) and supports synthesis of new test cases through the generative failure discovery pipeline.

Dynamic Active Testing: Cold-Start Problem and Limited Scope

Active testing (Kossen et al., 2021, 2022; Li et al., 2025) selects test points sequentially, using the results of previous evaluations to inform which input to evaluate next. This is more adaptive than static pruning, but the paper identifies two limitations:

The cold-start problem. Active testing methods begin with no prior knowledge about which inputs are likely to be hard for the target model. They must discover this from scratch by evaluating inputs, which means the initial samples are essentially random:

"since they have no knowledge of historical failure patterns across models, they must spend their initial budget identifying difficulty regions that are often already known to the community" (Appendix A).

ProEval's transfer learning approach solves this by constructing an informed prior from historical evaluation data across many models. The GP prior encodes that GSM8K questions involving fractions and multi-step arithmetic tend to be harder across models, so ProEval can target those inputs immediately without needing to rediscover this pattern for each new target model.

Limited to estimation, not discovery. Methods like Kossen et al. (2021) and Li et al. (2025) focus exclusively on estimating aggregate metrics (accuracy, error rate) and do not address failure discovery β€” finding the specific inputs where the model fails. The paper notes that ProEval "moves beyond simple estimation to perform failure case discovery and query synthesis, capabilities absent in their framework" (Appendix A). This is a scope difference: prior work answers "how accurate is the model?" while ProEval additionally answers "on what kinds of inputs does it fail?"

Failure Discovery and Red Teaming: Diversity-Efficiency Tradeoff

The failure discovery literature has developed increasingly sophisticated methods for automatically finding inputs that cause models to fail, but the paper argues there is an unresolved tension between diversity and sample efficiency:

Optimization-based red teaming (PAIR by Chao et al., 2024; TAP by Mehrotra et al., 2024; Bayesian Red Teaming by Lee et al., 2023) treats failure discovery as an optimization problem: find inputs that maximize some measure of model failure. The issue is that optimization tends to converge to a single failure mode β€” once the method discovers that the model fails on "questions about counting in ambiguous contexts," it keeps generating variations on that theme. The paper notes that prompt-optimization methods "can be prone to a lack of diversity, leaving vast regions of the failure landscape unexplored" (Appendix A). Bayesian Red Teaming attempts to address this with heuristic penalties like Self-BLEU to discourage repetition, but these are post-hoc patches rather than principled diversity mechanisms.

Quality-Diversity methods (Rainbow Teaming by Samvelyan et al., 2024) explicitly optimize for diverse failures using evolutionary algorithms, but the paper characterizes them as "highly sample-inefficient" β€” they need many queries to the target model to evolve a diverse population of adversarial inputs. For an expensive model, this sample inefficiency defeats the purpose.

ProEval's framing as level set estimation. The paper departs from the optimization paradigm entirely. Rather than trying to maximize failure, ProEval treats the problem as Bayesian Level Set Estimation (LSE) (Gotovos et al., 2013): map the entire region of input space where the model's failure probability exceeds a threshold. This is a fundamentally different objective β€” not "find the worst input" but "understand the full shape of the failure region." The authors state:

"Rather than optimizing for a single failure mode, ProEval seeks to efficiently map the entire superlevel set of inputs where the model fails, providing a global view of performance holes with minimal sample cost" (Appendix A).

This framing enables the exploration-exploitation acquisition function (Equation 12) that balances targeting likely-failure regions with exploring uncertain areas, and it motivates the topic-aware synthesis (TSS) that forces coverage across diverse semantic topics.

Why Bayesian Quadrature for Evaluation?

The paper also departs from the dominant estimation paradigm in active testing. Most methods use Importance Sampling (IS): train a surrogate model to predict which inputs are likely to be interesting, sample according to those predictions, and then reweight the estimates to correct for sampling bias (Kossen et al., 2021; Huang et al., 2025). The paper identifies a statistical issue:

"Most active testing strategies rely on Importance Sampling (IS), which often suffers from high variance in high-dimensional embedding spaces" (Appendix A).

Importance sampling weights can become extreme when the surrogate's sampling probabilities are poorly calibrated, which is common in high dimensions. A few samples with very large weights can dominate the estimate, leading to high variance.

Bayesian Quadrature (BQ) (O'Hagan, 1991; Ghahramani and Rasmussen, 2002) takes a different approach. Instead of correcting for biased sampling, BQ treats the integral (the aggregate performance metric) as a random variable derived from the GP posterior and analytically computes its variance. This variance can then be minimized directly by selecting inputs that provide the most information about the integral. The paper leverages this by designing acquisition functions that maximize variance reduction of the integral estimate (Section 2.3), which is statistically more efficient than IS-based approaches when the GP prior is well-calibrated.

However, standard BQ faces its own limitation: a "cold start" problem where generic GP priors (e.g., a default RBF kernel with zero mean) are uninformative about which inputs are correlated in terms of model performance. The paper's key innovation here is using pre-trained Gaussian processes (Wang et al., 2024) to initialize the GP prior from historical evaluation data, eliminating the cold start entirely.

How This Paper Positions Itself as a Unifying Framework

The paper's central positioning is that performance estimation and failure discovery should not be treated as separate problems but as dual Bayesian objectives sharing a common surrogate model. The introduction of Section 2 makes this explicit: the framework "grounds performance estimation in Bayesian quadrature and failure discovery in superlevel set sampling" (contribution 1). Both objectives operate on the same underlying function f(x)f(x) β€” the model's failure probability as a function of the input β€” but query it for different purposes: estimation aggregates over the input distribution, while discovery identifies high-probability regions.

This unification is not merely aesthetic. It means that every query to the target model serves both objectives simultaneously: evaluating an input improves both the aggregate estimate (by reducing BQ variance) and the failure region map (by reducing GP posterior uncertainty at that point). The acquired data is shared, and the GP posterior is updated once for both tasks. Prior work (Kossen et al., 2021 for estimation; Lee et al., 2023 for discovery) treated these as independent pipelines with separate surrogate models, missing this synergy.

The paper also positions itself as solving a proactive rather than reactive evaluation problem. The term "proactive" appears in the title and throughout: the framework doesn't just passively estimate after the fact but actively selects which inputs to evaluate and synthesizes new ones to probe specific capabilities. This proactive stance is operationalized through the active acquisition functions (Section 2.3) and the generative synthesis strategies (Section 2.4), which anticipate where failures are likely and target those regions preemptively.

Finally, the paper claims theoretical novelty: "the first theoretical proof that BQ with pre-trained GPs is unbiased and bounded without assuming a known GP prior" (contribution 5, Section 1). This addresses a gap in the pre-trained GP literature (Wang et al., 2024), which assumed the GP prior was known or estimated under stronger conditions. Theorem 3 provides a finite-sample bound on the BQ estimator's deviation from the ground truth when the prior is estimated from historical data, with the bound depending on the number of historical models NN and the kernel scale ΞΊ\kappa. This is positioned as providing formal guarantees for what would otherwise be a purely empirical approach.

3. Technical Approach

3.1 Reader Orientation

ProEval is a computational framework that acts as an adaptive test selector and generator, standing between a pool of unevaluated inputs and a generative AI model that needs to be tested. It solves the dual problem of how to get an accurate performance estimate with very few test queries and how to find diverse, specific failure cases without exhaustively searching the input space by maintaining a probabilistic model (a Gaussian process) of the target model's failure behavior, updating it with each evaluation, and using the model's uncertainty to decide which input to test next.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components connected in a feedback loop:

  1. GP Prior Constructor β€” Takes historical evaluation data (scores from other models on the same or similar benchmarks) and produces an informed Gaussian process prior β€” a probability distribution over the unknown failure function $f(x)$ β€” using either empirical score covariances or learned semantic embeddings. This component runs once, before any target-model evaluations begin.

  2. GP Posterior Updater β€” Maintains the current beliefs about $f(x)$ conditioned on all evaluations performed so far on the target model. Given a new observation (input $x$, binary score $y$), it analytically updates the mean prediction $\mu_t(x)$ and uncertainty $\sigma_t(x)$ for every input in the pool using standard GP posterior formulas.

  3. Performance Estimation Engine β€” Uses the GP posterior to compute a distribution over the aggregate performance metric $S = \int f(x) p(x) dx$, specifically its mean $\mathbb{E}[S | D_t]$ and variance $\mathbb{V}[S | D_t]$, via Bayesian quadrature. This component produces the final output: "the model's estimated error rate is X%."

  4. Active Acquisition Selector (for estimation) β€” Given the current GP posterior, evaluates a variance-reduction objective (Equation 10) over all unevaluated inputs and selects the one whose evaluation would most reduce the uncertainty in the aggregate estimate. This drives the estimation loop.

  5. Failure Discovery Engine β€” Operates in parallel or alternatively, using a different acquisition function (Equation 12) that balances predicted failure probability with epistemic uncertainty to select test inputs from a pool or to synthesize entirely new inputs via an LLM generator guided by topic constraints and anchor failure examples.

Information flows as follows: historical data β†’ GP prior construction β†’ initial posterior. Then, for each step of the active loop: current posterior β†’ acquisition function selects next input β†’ target model evaluates it β†’ observation updates posterior β†’ performance estimate and failure set refined. The loop terminates when the budget is exhausted or the estimate's uncertainty drops below a threshold.

3.3 Roadmap for the Deep Dive

  • First, the Bayesian formulation of evaluation (Section 2.1 in the paper): how performance estimation and failure discovery are unified under a single GP surrogate, including the core definitions of $S$ and $X_\lambda$. This is the foundation that all other components build on.

  • Second, transfer learning for GP priors (Section 2.2): the two strategies for constructing an informed prior β€” score features from historical covariance matrices, and prompt features from learned embeddings β€” because the quality of the prior is what enables the dramatic sample efficiency ProEval claims.

  • Third, the theoretical guarantee (Theorem 3): what it proves, what assumptions it requires, and what the bound actually says about the relationship between the number of historical models, the kernel scale, and estimation error.

  • Fourth, active performance estimation (Section 2.3): the variance-minimization acquisition function, the computational trick that makes it efficient with linear kernels, and Algorithm 1 as the full estimation loop.

  • Fifth, failure discovery (Section 2.4): the three strategies (SS, SS-Gen, TSS) as a progression from static pool retrieval to diversity-enforced synthesis, including the superlevel set acquisition function and the topic-aware bandit algorithm.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a framework and methodology paper whose core idea is that a single Gaussian process, conditioned on historical data and actively updated with a small number of target-model evaluations, can serve as the backbone for both accurate aggregate metric estimation and targeted failure case discovery, and that the choice of prior construction (score-based versus embedding-based) and acquisition strategy (variance reduction versus exploration-exploitation) determines the efficiency gains.

The Bayesian Formulation: Unifying Estimation and Discovery Under One GP

The paper formalizes both performance estimation and failure discovery as queries against a single latent function $f: \mathcal{X} \to \mathbb{R}$ that maps any input $x$ (a text prompt, an image, a question-answer pair) to a scalar representing the severity of the model's failure on that input. For the experiments in this paper, $f(x)$ is binary: 1 if the target model fails (incorrect answer, safety violation), 0 if it succeeds. But the framework accommodates continuous scores (e.g., graded error severity) without modification.

The two evaluation objectives are then defined as functionals of $f$ over the input space (Equation 1):

Performance estimation is the expected failure rate over the test distribution $p(x)$:

S=∫Xf(x)p(x)dxS = \int_{\mathcal{X}} f(x) p(x) dx

where $S$ is the aggregate performance metric we want to estimate (the population error rate), $\mathcal{X}$ is the space of all possible test inputs (in practice, the finite benchmark), $f(x)$ is the failure indicator at input $x$, and $p(x)$ is the distribution from which test inputs are drawn (assumed uniform over the benchmark for standard evaluation).

What it computes: $S$ is the expected value of the failure function under the test distribution β€” the long-run proportion of inputs on which the model would fail if we could evaluate it on the entire distribution. In practice, with a finite benchmark of $M$ test questions, $S$ is approximated as the sample average $\frac{1}{M} \sum_{j=1}^M f(x_j)$ where $x_j$ are the test-set questions.

Why this form: framing evaluation as integration rather than classification enables Bayesian quadrature β€” we can place a prior on the function $f$, observe its values at a small set of points, and analytically compute the posterior distribution of the integral. This is statistically more efficient than estimating individual $f(x_j)$ values and averaging them, because the GP prior encodes correlations that let observations at some inputs reduce uncertainty about others.

Failure discovery is characterized as the superlevel set of $f$ above a threshold $\lambda$:

XΞ»={x∣f(x)β‰₯Ξ»}\mathcal{X}_\lambda = \{x \mid f(x) \geq \lambda\}

where $\lambda$ is a user-defined threshold (for binary failure, $\lambda = 0.5$ defines the set of inputs where the model is more likely to fail than succeed). The goal is not just to find one point in $\mathcal{X}_\lambda$ but to characterize the set β€” to discover multiple, diverse failure inputs that span different regions of the input space.

Gaussian Process Surrogate: The Probabilistic Spine

The paper places a Gaussian process prior on the unknown failure function:

f∼GP(μ,k)f \sim \mathcal{GP}(\mu, k)

where $\mu: \mathcal{X} \to \mathbb{R}$ is the prior mean function (our best guess for $f(x)$ before seeing any target-model data) and $k: \mathcal{X} \times \mathcal{X} \to \mathbb{R}$ is the kernel or covariance function (encoding how similar we expect $f(x)$ and $f(x')$ to be for two inputs $x$ and $x'$).

What this means operationally: before evaluating the target model on any input, we have a probability distribution over the failure score for every possible input. For any finite collection of inputs, the joint distribution is multivariate Gaussian with mean vector $[\mu(x_1), ..., \mu(x_M)]$ and covariance matrix $[k(x_i, x_j)]_{i,j}$. This is the prior β€” it represents what we know before seeing data, and the entire art of ProEval is in making this prior informative (see Section 2.2).

Given $t$ noisy observations $D_t = \{(x_\tau, y_\tau)\}_{\tau=1}^t$ where each observation is modeled as $y_\tau \sim \mathcal{N}(f(x_\tau), \sigma^2)$, the posterior distribution $f \mid D_t \sim \mathcal{GP}(\mu_t, k_t)$ is given by the standard GP update equations (Equation 2):

ΞΌt(x)=ΞΌ(x)+k(x,xt)Ktβˆ’1(ytβˆ’ΞΌ(xt))\mu_t(x) = \mu(x) + k(x, \mathbf{x}_t) K_t^{-1} (\mathbf{y}_t - \mu(\mathbf{x}_t))

kt(x,xβ€²)=k(x,xβ€²)βˆ’k(x,xt)Ktβˆ’1k(xt,xβ€²)k_t(x, x') = k(x, x') - k(x, \mathbf{x}_t) K_t^{-1} k(\mathbf{x}_t, x')

where $\mathbf{x}_t = [x_\tau]_{\tau=1}^t$ is the vector of evaluated inputs so far, $\mathbf{y}_t = [y_\tau]_{\tau=1}^t$ is the vector of observed failure scores, $k(x, \mathbf{x}_t)$ is a row vector of kernel evaluations between $x$ and all observed inputs, $K_t = [[k(x_\tau, x_{\tau'})]_{\tau=1}^t]_{\tau'=1}^t + I\sigma^2$ is the $t \times t$ Gram matrix of observed inputs plus observation noise on the diagonal, and $\mu(\mathbf{x}_t) = [\mu(x_\tau)]_{\tau=1}^t$ is the prior mean evaluated at observed inputs.

What the posterior mean computes: $\mu_t(x)$ is a corrected prediction for $f(x)$. It starts with the prior guess $\mu(x)$ and adds a correction term $k(x, \mathbf{x}_t) K_t^{-1} (\mathbf{y}_t - \mu(\mathbf{x}_t))$. This correction is a weighted sum of the residuals (observed minus prior) at all evaluated points, with weights determined by how correlated $x$ is with each observed point. If $x$ is highly correlated with $x_1$ (in the sense of the kernel), and we observed that the model did worse than expected at $x_1$ (positive residual), the correction pushes the prediction for $x$ upward β€” we infer that $x$ is also likely to be hard.

What the posterior variance computes: $k_t(x, x')$ shrinks the prior covariance by subtracting $k(x, \mathbf{x}_t) K_t^{-1} k(\mathbf{x}_t, x')$. This term is always non-negative (since $K_t^{-1}$ is positive definite), so the posterior variance $k_t(x, x)$ is always less than or equal to the prior variance $k(x, x)$. The reduction is larger when $x$ is highly correlated with the observed points β€” evaluating one input reduces uncertainty about similar inputs.

Why the Gaussian observation model for binary data: the paper explicitly addresses this apparent mismatch in Section 3.2. A Bernoulli/probit likelihood would be more appropriate for binary outcomes, but it would break the closed-form GP posterior and require approximate inference (Laplace approximation, variational inference, MCMC). The authors experimented with a GP classifier using Laplace approximation and found it yielded worse performance (MAE of 0.1653 on StrategyQA vs. BQ-SF's 0.011), likely because the approximation introduces errors that compound when computing the integral. The Gaussian likelihood is a deliberate engineering trade-off: it sacrifices statistical correctness for computational tractability and analytical BQ updates. The "BQ Rounded" variant β€” which rounds posterior predictions to $\{0, 1\}$ before summing β€” is offered as a partial mitigation that exploits the binary nature of the data.

Linear Kernel on Embeddings: Computational Efficiency

As a special case that becomes critical for the variance-reduction acquisition function, the paper considers a GP with a linear kernel defined over a learned encoder $\phi: \mathcal{X} \to \mathbb{R}^d$:

k(x,xβ€²)=Ο•(x)TΟ•(xβ€²)k(x, x') = \phi(x)^T \phi(x')

What the linear kernel encodes: the correlation between failure scores at two inputs is the dot product of their embedding vectors. This is an inner-product similarity model: inputs with aligned embeddings (pointing in similar directions) are predicted to have correlated failure behavior, while orthogonal embeddings are predicted to be independent.

When this kernel is used, the posterior computation simplifies substantially. Define the embedding matrix $Z = [\phi(x_\tau)]_{\tau=1}^t \in \mathbb{R}^{d \times t}$ and the matrix $\tilde{K}_t = (Z Z^T \sigma^{-2} + I)^{-1} \in \mathbb{R}^{d \times d}$. Then the posterior evaluates to (Equation 3):

ΞΌt(x)=ΞΌ(x)+Οƒβˆ’2Ο•(x)TK~tZ(ytβˆ’ΞΌ(xt))\mu_t(x) = \mu(x) + \sigma^{-2} \phi(x)^T \tilde{K}_t Z (\mathbf{y}_t - \mu(\mathbf{x}_t))

kt(x,xβ€²)=Ο•(x)TK~tΟ•(xβ€²)k_t(x, x') = \phi(x)^T \tilde{K}_t \phi(x')

Why this matters computationally: the standard GP posterior (Equation 2) requires inverting the $t \times t$ matrix $K_t$, which costs $O(t^3)$ and becomes the bottleneck as we acquire more observations. With the linear kernel, the computation is dominated by inverting the $d \times d$ matrix in $\tilde{K}_t$, which costs $O(d^3)$. If the embedding dimension $d$ (typically 50–300 after PCA or tuning) is much smaller than the number of observations $t$ (which can grow to hundreds in the active loop), this is a dramatic speedup.

Furthermore, $\tilde{K}_t$ can be updated incrementally using the Sherman-Morrison formula when a new observation $(x_{t+1}, y_{t+1})$ arrives (Equation 4):

K~t+1=K~tβˆ’K~tΟ•(xt+1)Ο•(xt+1)TK~tΟƒβˆ’21+Ο•(xt+1)TK~tΟ•(xt+1)Οƒβˆ’2\tilde{K}_{t+1} = \tilde{K}_t - \frac{\tilde{K}_t \phi(x_{t+1}) \phi(x_{t+1})^T \tilde{K}_t \sigma^{-2}}{1 + \phi(x_{t+1})^T \tilde{K}_t \phi(x_{t+1}) \sigma^{-2}}

What the Sherman-Morrison update computes: given the old $d \times d$ precision-like matrix $\tilde{K}_t$ and the embedding $\phi(x_{t+1})$ of the new observation, this formula computes the updated matrix $\tilde{K}_{t+1}$ in $O(d^2)$ time rather than requiring a full $O(d^3)$ matrix inversion. The denominator $1 + \phi(x_{t+1})^T \tilde{K}_t \phi(x_{t+1}) \sigma^{-2}$ is a scalar that acts as a normalization factor: when the new point's embedding is already well-represented in the existing data (large $\phi(x_{t+1})^T \tilde{K}_t \phi(x_{t+1})$), the denominator is large and the update is small β€” we already know about this region.

Why this form enables efficient active selection: as we will see in the active performance estimation section, the variance-reduction acquisition function (Equation 10) requires evaluating the posterior variance of the integral under hypothetical new observations. The Sherman-Morrison update lets us compute what the posterior would be if we added a candidate point, without actually doing the full $O(d^3)$ inversion for each candidate. This is what makes the greedy acquisition step computationally feasible.

Posteriors for the Evaluation Objectives

Once the GP posterior over $f$ is established, the paper derives posteriors over the two evaluation objectives.

Bayesian Quadrature for Performance Estimation (Equation 5): the integral $S$ is approximated as a finite sum over $M$ test inputs $\{x_j\}_{j=1}^M$ drawn i.i.d. from $p(x)$. Because the GP posterior over $f$ is Gaussian at any finite set of points, the posterior over the sum (and hence the average) is also Gaussian:

E[S∣Dt]β‰ˆ1Mβˆ‘j=1MΞΌt(xj)\mathbb{E}[S \mid D_t] \approx \frac{1}{M} \sum_{j=1}^{M} \mu_t(x_j)

V[S∣Dt]β‰ˆ1M2βˆ‘j,jβ€²=1Mkt(xj,xjβ€²)\mathbb{V}[S \mid D_t] \approx \frac{1}{M^2} \sum_{j, j'=1}^{M} k_t(x_j, x_{j'})

where $\mu_t(x_j)$ is the posterior mean of $f(x_j)$ (our best point estimate of failure probability at input $x_j$), and $k_t(x_j, x_{j'})$ is the posterior covariance between $f(x_j)$ and $f(x_{j'})$.

What the posterior mean of $S$ computes: the expected error rate is simply the average of the GP's predicted failure probabilities over all test inputs. If the GP predicts that questions 1–5 have failure probabilities $[0.1, 0.3, 0.8, 0.2, 0.6]$, the estimated error rate is $(0.1 + 0.3 + 0.8 + 0.2 + 0.6) / 5 = 0.4$. This is the standard plug-in estimator β€” plug the posterior means into the integral formula.

What the posterior variance of $S$ computes: this is the key quantity that drives active sampling. The variance is the sum of all pairwise posterior covariances, scaled by $1/M^2$. If the test inputs have high positive covariation (failures are correlated across the benchmark), the variance is large because uncertainty in one region propagates to the aggregate estimate. If they are independent or negatively correlated, the variance is smaller because errors cancel. The active acquisition function selects inputs to directly minimize this quantity.

Why this form: standard Monte Carlo estimation would compute $\hat{S} = \frac{1}{t} \sum_{\tau=1}^t y_\tau$ using only the evaluated inputs, with variance $\text{Var}(f(x))/t$. BQ uses the GP posterior to extrapolate from evaluated to unevaluated inputs, borrowing information through the kernel. This means the BQ variance can be much smaller than the Monte Carlo variance because it accounts for the fact that evaluating one input reduces uncertainty about correlated inputs.

Probabilistic Superlevel Sets (Equation 6): the failure region is approximated as:

Xλβ={x∣μt(x)+Ξ²Οƒt(x)β‰₯Ξ»}\mathcal{X}_\lambda^\beta = \{x \mid \mu_t(x) + \beta \sigma_t(x) \geq \lambda\}

where $\sigma_t(x) = \sqrt{k_t(x, x)}$ is the posterior standard deviation at $x$, and $\beta$ is a confidence parameter.

What this computes: for each input $x$, the quantity $\mu_t(x) + \beta \sigma_t(x)$ is an upper confidence bound on $f(x)$. When $\beta = 0$, the criterion becomes $\mu_t(x) \geq \lambda$ β€” we include $x$ in the probable failure set if our point estimate of failure probability exceeds the threshold. When $\beta < 0$ (e.g., $\beta = -1$), the criterion is stricter: we require that even the lower confidence bound $\mu_t(x) - |\beta| \sigma_t(x)$ exceeds $\lambda$, meaning we are confident (at a level determined by $\beta$) that the input is a failure. When $\beta > 0$, the set is more inclusive: we include inputs where failure is plausible but uncertain, encouraging exploration.

Why this form: this is the standard approach in Bayesian level set estimation (Gotovos et al., 2013). It trades off between including too many false positives (small $\beta$ means we require high confidence) and missing true failures (large $\beta$ means we include uncertain regions). The acquisition function in Equation 12 uses $\beta$ to control the exploration-exploitation balance: $\beta = 0$ targets inputs where the GP predicts failure (exploitation), while $\beta < 0$ targets inputs where the GP is uncertain (exploration within the plausible failure region).

Transfer Learning for GP Priors: Score Features via Empirical Statistics

This is the first of two strategies for constructing an informed GP prior, and it applies when historical evaluation results exist for the exact same set of inputs across multiple models. The paper formalizes this as Assumption 2: all historical datasets evaluate the identical set of $M$ inputs $\{x_j\}_{j=1}^M$, producing scores $y_{ij}$ for model $i$ on input $j$.

Under Assumption 1 (all models' score functions $f_i$ are drawn from a shared GP prior $\mathcal{GP}(\mu, k)$), the vector of scores for model $i$, $\mathbf{y}_i = [y_{i1}, ..., y_{iM}]^T$, is a sample from an $M$-dimensional multivariate Gaussian with mean $\mathbf{u} = [\mu(x_j)]_{j=1}^M$ and covariance $\Sigma = [k(x_j, x_{j'})]_{j,j'=1}^M$. The paper estimates these from the $N$ historical models (Equation 7):

u^=1NYΓ—1N\hat{\mathbf{u}} = \frac{1}{N} Y \times \mathbf{1}_N

Ξ£^=1Nβˆ’1(Yβˆ’u^)(Yβˆ’u^)T\hat{\Sigma} = \frac{1}{N-1} (Y - \hat{\mathbf{u}})(Y - \hat{\mathbf{u}})^T

where $Y = [\mathbf{y}_i]_{i=1}^N$ is the $M \times N$ matrix of all historical scores (rows are inputs, columns are models), $\mathbf{1}_N$ is a vector of $N$ ones, $\hat{\mathbf{u}}$ is the $M$-dimensional vector of per-input mean scores (the average failure rate of historical models on each question), and $\hat{\Sigma}$ is the $M \times M$ sample covariance matrix (capturing how failures co-occur across questions).

What this computes: $\hat{\mathbf{u}}_j$ is the empirical difficulty of question $j$: if historical models fail on it 70% of the time, the prior mean for a new model is 0.7. $\hat{\Sigma}_{jj'}$ captures whether questions $j$ and $j'$ tend to be jointly hard or easy across models. A large positive entry means models that fail on $j$ also tend to fail on $j'$ β€” the questions test similar capabilities.

Conversion to score features (Section 2.2.1): the key insight is that these empirical statistics are equivalent to a GP with a linear kernel over "score features." Define the normalized score feature for input $x_j$ as:

Ο•(xj)=1Nβˆ’1[yijβˆ’u^j]i=1N\phi(x_j) = \frac{1}{\sqrt{N-1}} [y_{ij} - \hat{\mathbf{u}}_j]_{i=1}^N

where $\phi(x_j) \in \mathbb{R}^N$ is the centered, scaled vector of how each historical model performed on question $j$. Then setting the prior mean to $\hat{\mu}(x_j) = \hat{\mathbf{u}}_j$ and the linear kernel to $\hat{k}(x_j, x_{j'}) = \phi(x_j)^T \phi(x_{j'})$ exactly reconstructs $\hat{\Sigma}$:

k^(xj,xjβ€²)=1Nβˆ’1βˆ‘i=1N(yijβˆ’u^j)(yijβ€²βˆ’u^jβ€²)=Ξ£^jjβ€²\hat{k}(x_j, x_{j'}) = \frac{1}{N-1} \sum_{i=1}^N (y_{ij} - \hat{\mathbf{u}}_j)(y_{ij'} - \hat{\mathbf{u}}_{j'}) = \hat{\Sigma}_{jj'}

What the score feature represents: each input $x_j$ gets an $N$-dimensional feature vector where the $i$-th entry is how much model $i$'s performance on this question deviates from the average performance on this question. Two questions with similar feature vectors (i.e., similar patterns of which models succeed and fail) are predicted to have correlated failure behavior β€” the GP will infer that if the target model fails on one, it likely fails on the other.

Why this form: the linear kernel over score features inherits all the computational advantages described earlier (Sherman-Morrison updates, $O(N^3)$ instead of $O(t^3)$ for posterior computation). Moreover, it gives a concrete embedding of each question in a space defined by model behavior, which is intuitive: questions that are "hard for weaker models but easy for stronger ones" get different features than questions that are "universally hard."

Figure 2 in the paper validates this approach empirically: it shows the $M \times M$ covariance matrix of question performance across 5 models (GPT-4o, Gemini 2.5 Flash, Claude 4.5 Sonnet, Qwen 3 32B, GPT-5) on StrategyQA, GSM8K, MMLU, and SVAMP. The matrices exhibit clear block structure β€” groups of questions with positive covariance (models tend to succeed or fail together on them) and negative covariance (models diverge). This block structure is what the GP prior captures and exploits.

Theoretical Guarantee: Theorem 3

The paper provides a formal bound on the performance estimator when the GP prior is estimated from historical data rather than known a priori. Let $\hat{S}_t = \frac{1}{M} \sum_{j=1}^M \hat{\mu}_t(x_j)$ be the BQ estimator using the estimated GP (where $\hat{\mu}_t$ is the posterior mean computed with the estimated kernel $\hat{k}$ and mean $\hat{\mu}$), and let $S_t = \frac{1}{M} \sum_{j=1}^M \mu_t(x_j)$ be the BQ estimator using the true (unknown) GP posterior mean. Theorem 3 states:

E[S^t]=E[S∣Dt]\mathbb{E}[\hat{S}_t] = \mathbb{E}[S \mid D_t]

∣S^tβˆ’Stβˆ£β‰€aβ€²ΞΊ+Οƒ2|\hat{S}_t - S_t| \leq a' \sqrt{\kappa + \sigma^2}

The first line asserts unbiasedness: the expected value of the estimated estimator equals the expected value of the true posterior integral β€” the estimator is not systematically wrong despite using an estimated prior. The second line provides a finite-sample bound on the deviation between the estimated and true estimators, with:

aβ€²=(4M(t+1+2tlog⁑4MΞ΄+2log⁑4MΞ΄βˆ’2/N)(Nβˆ’tβˆ’2)Ξ΄)12a' = \left(\frac{4M \left(t + 1 + 2\sqrt{t \log \frac{4M}{\delta}} + 2 \log \frac{4M}{\delta} - 2/N\right)}{(N - t - 2)\delta}\right)^{\frac{1}{2}}

where $M$ is the number of test inputs, $t$ is the number of target-model evaluations, $N$ is the number of historical models used to estimate the prior, $\kappa$ is an upper bound on the kernel function ($\kappa \geq k(x, x)$ for any $x$), $\sigma^2$ is the observation noise variance, and $\delta$ is the confidence parameter (the bound holds with probability $1 - \delta$).

What the bound means operationally: the estimator using the estimated GP prior is guaranteed to be close to the estimator using the true GP prior, up to a factor that depends on the scale of the kernel and the noise. The key term in $a'$ is the denominator $(N - t - 2)\delta$: as the number of historical models $N$ increases, the bound shrinks. The paper interprets this as meaning that "increasing $N$ can lead to better estimation" and notes the condition $N > M$ ensures the bound is non-vacuous. This condition is often satisfied during model development: "because of the sweep over learning rate, batch size, architecture variants, regularizations, data scales, etc.," each configuration produces multiple model checkpoints, so $N$ (total historical models) can exceed $M$ (test set size).

Assumptions required: Assumption 1 (all models share a GP prior) and Assumption 2 (identical inputs across models) must hold, and $N \gg t$ is assumed (many more historical models than target-model evaluations). The proof (Appendix B) relies on Theorem 5 of Wang et al. (2024), which provides a bound on $|\hat{\mu}_t(x) - \mu_t(x)|$ for pre-trained GPs under a union bound over $M$ test points.

The paper is careful not to overclaim: it notes that empirically, "even when the theoretical bound is loose, the estimation accuracy remains high" (Section 2.2.1). The bound is a worst-case guarantee, not a tight prediction of actual error. Its value is in formally justifying that transferring from many historical models does not introduce systematic bias.

Transfer Learning for GP Priors: Prompt Features via Learned Embeddings

When historical scores are not available for the exact same inputs (e.g., evaluating a new benchmark, or transferring knowledge across benchmarks with different questions), the paper constructs a GP prior based on semantic similarity. The idea is to map each input $x$ (a text prompt, an image) into a $d$-dimensional embedding space using a function $\psi_\theta: \mathcal{X} \to \mathbb{R}^d$ with learnable parameters $\theta$, and then define the GP using a kernel over these embeddings.

The specific architecture (Figure 3) is a two-stage encoder: a fixed pre-trained embedding model (e.g., OpenAI's text-embedding-3-large, which produces 3072-dimensional embeddings) followed by a learnable multi-layer perceptron (MLP) that projects to a lower-dimensional space. The GP is then defined with a centered encoder:

ϕθ(x)=1dβˆ’1(ψθ(x)βˆ’1dψθ(x)Γ—1d)∈Rd\phi_\theta(x) = \frac{1}{\sqrt{d-1}} (\psi_\theta(x) - \frac{1}{d} \psi_\theta(x) \times \mathbf{1}_d) \in \mathbb{R}^d

where $\psi_\theta(x)$ is the $d$-dimensional output of the full encoder (fixed embedding model + learnable MLP), $\frac{1}{d} \psi_\theta(x) \times \mathbf{1}_d$ is a $d$-dimensional vector where every entry is the component-wise mean of $\psi_\theta(x)$, and the subtraction centers the features (zero mean). The scaling $1/\sqrt{d-1}$ ensures the kernel values have appropriate magnitude.

What the centering does: the raw embedding $\psi_\theta(x)$ might have a non-zero mean across its $d$ dimensions (e.g., biases from the pre-trained model). Centering removes this bias, which is analogous to the centering in the score features (subtracting $\hat{\mathbf{u}}_j$). The resulting $\phi_\theta(x)$ has zero mean over its dimensions, so the kernel $\phi_\theta(x)^T \phi_\theta(x')$ measures covariance around this mean β€” consistent with how the score-feature kernel was constructed.

The paper experiments with two kernel choices:

  • Linear kernel (via $\phi_\theta$ directly): $\hat{k}(x, x') = \phi_\theta(x)^T \phi_\theta(x')$
  • MatΓ©rn kernel over $\phi_\theta$: $\hat{k}(x, x') = k_{\text{MatΓ©rn}}(\phi_\theta(x), \phi_\theta(x'))$

The MatΓ©rn kernel is the primary choice for prompt-feature experiments because it is a stationary kernel (translation-invariant in the embedding space) that is more flexible than the linear kernel β€” it can capture non-linear relationships between embedding distances and failure probability. The linear kernel assumes that failure correlation is exactly the dot product, which is a strong assumption that may not hold for semantic embeddings.

The mean function is set to:

ΞΌ^(x)=1dψθ(x)Γ—1d\hat{\mu}(x) = \frac{1}{d} \psi_\theta(x) \times \mathbf{1}_d

which is a constant (scalar) equal to the component-wise mean of $\psi_\theta(x)$. The paper notes this is "bounded by the scale of $\psi$" (Section 2.2.2, Figure 3 caption), which is important because a GP prior mean that can take arbitrarily large values would be uninformative β€” it's better to have a mean that respects the natural range of the embeddings.

Training the encoder: the parameters $\theta$ (the MLP weights and optionally the GP kernel hyperparameters like MatΓ©rn lengthscale) are optimized by maximizing the log-likelihood of the historical data across all $N$ datasets:

ΞΈ^=arg⁑maxβ‘ΞΈβˆ‘i=1Nlog⁑p(yi∣θ)\hat{\theta} = \arg\max_\theta \sum_{i=1}^N \log p(\mathbf{y}_i \mid \theta)

where $p(\mathbf{y}_i \mid \theta)$ is the marginal likelihood of model $i$'s scores under the GP prior with parameters $\theta$. This is standard GP hyperparameter optimization (Type II maximum likelihood) applied jointly to all historical models. The objective encourages $\theta$ to produce embeddings such that the GP kernel explains the observed patterns of model failures.

What this enables (zero-shot generalization): once $\theta$ is learned, the GP can make predictions for any input $x$ β€” even one never seen in any historical dataset β€” by encoding it with $\psi_\theta$, plugging into the kernel, and computing the posterior. The claim is that semantically similar prompts will have similar embeddings and therefore similar predicted failure probabilities, even if the prompts come from entirely different benchmarks. This is what the paper calls "cross-benchmark" transfer.

Three variants compared:

  • Raw Prompt Features (RPF): $\psi_\theta(x)$ is the raw output of the fixed embedding model β€” no learnable MLP. The GP uses a MatΓ©rn kernel over PCA-reduced embeddings (16 dimensions in the experiments). The prior mean is either GMM-selected (Default scenario), from all auxiliary models (New Model scenario), or a constant 0.5 (New Bench scenario).
  • Tuned Prompt Features (TPF): $\psi_\theta(x)$ includes a learnable MLP optimized on historical data as described above. The mean is $\frac{1}{d} \psi_\theta(x) \times \mathbf{1}_d$. This is expected to capture finer-grained performance correlations that raw embeddings miss.
  • Score Features (SF): the empirical covariance approach from Section 2.2.1 β€” no semantic embeddings involved. Applicable only when historical data exists for the exact same inputs.

Selecting Historical Datasets: GMM Clustering and Abstention

The validity of Assumption 1 β€” that all historical models' score functions are drawn from a shared GP prior β€” depends on the source models being similar to the target model. If the target model is a strong reasoning model (say, GPT-5) and we include historical data from much weaker models with qualitatively different failure patterns, the estimated prior will be corrupted, leading to worse performance than using no prior at all.

The paper addresses this with a Gaussian Mixture Model (GMM) Clustering approach (Section 2.2.3):

  1. Feature construction: for each model, project its per-question score vector $[y_{ij}]_{j=1}^M$ on a reference benchmark (not the target benchmark β€” by default, "all available benchmarks except the target benchmark") into a lower-dimensional space using Principal Component Analysis (PCA).
  2. Clustering: fit a Gaussian Mixture Model to these PCA-reduced model profiles, selecting the number of components $K$ via Bayesian Information Criterion (BIC) minimization: $\text{BIC}(K) = -2 \log p(\Phi \mid \theta_K) + d_K \log N$, where $d_K$ is the number of free parameters in a $K$-component GMM.
  3. Prior construction: the GP prior is built exclusively from historical models in the same GMM cluster as the target model.
  4. Abstention rule: if the target model's cluster contains fewer than min_sources models (default = 3), ProEval abstains from making predictions. The paper reports that in 78 experiments, there were 17 abstentions (22% of cases), and that this dramatically improved reliability: "mean MAE drops from 0.0394 β†’ 0.0274" (Appendix E).

Why GMM clustering: the alternative β€” using all available historical data β€” assumes every model's behavior is informative for every target, which can lead to negative transfer. Figure 7 quantifies this: using G0 (Gemma-3-12B) as a source for G2 (Gemini 2.5 Flash) yields an MAE > 0.10, while using G1 (Gemma-3-27B) yields MAE < 0.01. The GMM identifies this structure and filters out the harmful sources. Figure 10 visualizes the clusters: models like G0 and O1 (GPT-3.5 Turbo) consistently fall outside the main cluster for most benchmarks, confirming their behavior is dissimilar.

The paper also explores alternative selection heuristics in Appendix E: Spearman rank correlation, Mahalanobis distance, leave-one-out likelihood, and Mardia's multivariate normality test. GMM with abstention achieves the best median MAE (0.0109) across 78 experiments, though Spearman correlation with a high threshold ($\tau \geq 0.7$) achieves the lowest variance β€” at the cost of a 54% abstention rate.

Active Performance Estimation: Minimizing BQ Variance

With the GP prior established, the active estimation loop proceeds as follows (Algorithm 1): at each step $t$, select the next input $x_{t+1}$ to evaluate by maximizing the expected reduction in the BQ posterior variance:

xt+1=arg⁑max⁑x∈X(V[S∣Dt]βˆ’V[S∣Dtβˆͺx])x_{t+1} = \arg\max_{x \in \mathcal{X}} \left(\mathbb{V}[S \mid D_t] - \mathbb{V}[S \mid D_t \cup x]\right)

where $\mathbb{V}[S \mid D_t]$ is the current BQ variance (from Equation 5) and $\mathbb{V}[S \mid D_t \cup x]$ is what the variance would be if we hypothetically added input $x$ to the observation set.

What this objective computes: for each candidate input $x$ in the unlabeled pool, we imagine evaluating it (but don't need to know the outcome β€” the variance reduction is independent of the observed $y$ value for a Gaussian likelihood) and compute how much the uncertainty about the aggregate metric would shrink. The input that maximizes this shrinkage is the one that provides the most information about the integral. This is a form of active learning for integration, distinct from standard active learning which minimizes prediction error on individual points.

Why the variance reduction is observation-independent: the posterior variance $k_{t+1}(x_j, x_{j'})$ depends on where we observed, not what we observed. The reduction $\mathbb{V}[S \mid D_t] - \mathbb{V}[S \mid D_t \cup x]$ is purely a function of the kernel and the locations of observed points, not the $y$ values. This means the entire acquisition schedule can be pre-computed before evaluating the target model even once β€” a major practical advantage for parallel evaluation or for scenarios where the evaluation budget is known in advance.

Efficient computation with linear kernel: when using the linear kernel over score features or embeddings, the variance-reduction objective can be expressed without inverting a $(t+1) \times (t+1)$ matrix for each candidate. Using the Sherman-Morrison formula, the optimization becomes (Equation 11):

xt+1=arg⁑max⁑x∈XExβ€²,xβ€²β€²[Ο•(xβ€²)TK~tΟ•(x)Ο•(x)TK~tΟƒ2+Ο•(x)TK~tΟ•(x)Ο•(xβ€²β€²)]x_{t+1} = \arg\max_{x \in \mathcal{X}} \mathbb{E}_{x', x''} \left[ \phi(x')^T \frac{\tilde{K}_t \phi(x) \phi(x)^T \tilde{K}_t}{\sigma^2 + \phi(x)^T \tilde{K}_t \phi(x)} \phi(x'') \right]

where $\phi(x)$ is the embedding of the candidate input, $\tilde{K}_t$ is the $d \times d$ precision-like matrix from the current GP posterior, and the expectation is over pairs $(x', x'')$ drawn from the test distribution (approximated by the $M$ test inputs).

What this formula computes β€” step by step:

  1. $\phi(x)^T \tilde{K}_t$ projects the candidate's embedding through the current posterior precision matrix, producing a $d$-dimensional row vector that encodes how correlated the candidate is with all previously observed points.
  2. Multiplying by $\phi(x)$ (outer product) and then by $\tilde{K}_t$ and $\phi(x'')$ computes the reduction in covariance between $x'$ and $x''$ if we were to observe at $x$.
  3. The denominator $\sigma^2 + \phi(x)^T \tilde{K}_t \phi(x)$ is the posterior variance at $x$ plus noise β€” if we already know a lot about $x$ (large $\phi(x)^T \tilde{K}_t \phi(x)$), the denominator is large and the reduction is small. If $x$ is in an unexplored region, the reduction is large.
  4. The expectation over test pairs $(x', x'')$ aggregates these pairwise covariance reductions into a single scalar: the total variance reduction for the integral.

Why this formulation is efficient: it avoids a naive $O(t^3 M)$ computation (reevaluate GP posterior for each of $M$ candidates). Instead, with $\tilde{K}_t$ pre-computed in $O(d^3)$, evaluating the objective for each candidate is $O(d^2)$, and the total per-step cost is $O(M d^2 + d^3)$. When $d$ is small (e.g., 16 for PCA-reduced prompts), this is negligible.

Algorithm 1 formalizes the loop: initialize the GP with the learned prior (mean $\hat{\mu}$, kernel $\hat{k}$), then for $t = 0, ..., T-1$: select $x_{t+1}$ via Eq. 10, evaluate $f(x_{t+1})$ (query the target model), update the dataset $D_{t+1}$, and update the performance estimate $\mathbb{E}[S \mid D_{t+1}]$ via Eq. 5.

The paper also mentions that batches of inputs can be pre-computed for parallel evaluation, since the variance reduction is independent of the observations. This is a practical consideration: if one has access to parallel model inference infrastructure, the acquisition function can select a batch of $B$ inputs at once, reducing the number of sequential rounds.

The "BQ Rounded" Variant

The paper introduces a heuristic variant for binary scores: after computing the posterior mean $\mu_t(x_j)$ for each test input, round it to the nearest integer (0 or 1) before summing to form the performance estimate:

S^trounded=1Mβˆ‘j=1Mround(ΞΌt(xj))\hat{S}_t^{\text{rounded}} = \frac{1}{M} \sum_{j=1}^M \text{round}(\mu_t(x_j))

What this does: if the GP predicts a failure probability of 0.73 for a question, it gets counted as 1 (failure). If it predicts 0.24, it gets counted as 0 (success). This exploits the binary nature of the task β€” the true $f(x)$ is always 0 or 1 β€” at the cost of discarding uncertainty information. A prediction of 0.51 and 0.99 are treated identically, even though the former reflects high uncertainty.

Why this can sometimes help: when the GP posterior means are well-calibrated but have noise around 0.5, the standard BQ estimator includes fractional contributions (e.g., 0.51) that introduce small errors proportional to the degree of uncertainty. Rounding removes these by forcing a hard decision. However, when the posterior means are near 0.5 for many inputs (high epistemic uncertainty), rounding can introduce larger errors because ambiguous predictions are arbitrarily binarized.

The experimental results reflect this trade-off: BQ Rounded sometimes outperforms standard BQ and sometimes underperforms, with the paper noting it "exhibits less stability than the standard BQ variant, especially when the posterior mean is near 0.5" (Section 3.2 footnote). It is offered as a baseline for future work on non-Gaussian BQ rather than a recommended default.

Failure Discovery: Superlevel Set Sampling (SS)

The first failure discovery strategy operates on a static pool of unlabeled inputs $D_{\text{pool}}$. The acquisition function (Equation 12):

Ξ±SS(x∣Dt)=1(ΞΌt(x)+Ξ²Οƒt(x)β‰₯Ξ»)Γ—kt(x,x)\alpha_{\text{SS}}(x \mid D_t) = \mathbf{1}\left(\mu_t(x) + \beta \sigma_t(x) \geq \lambda\right) \times k_t(x, x)

where $\mathbf{1}(\cdot)$ is the indicator function (1 if the condition holds, 0 otherwise), $\mu_t(x)$ is the posterior mean, $\sigma_t(x) = \sqrt{k_t(x, x)}$ is the posterior standard deviation, $\beta$ controls the confidence level (discussed below), and $\lambda$ is the failure threshold (0.5 for binary failure).

What the indicator term does: it restricts the search to the probable failure set $\mathcal{X}_\lambda^\beta$ defined in Equation 6. Any input where the upper confidence bound $\mu_t(x) + \beta \sigma_t(x)$ is below $\lambda$ gets an acquisition value of exactly zero β€” it is excluded from consideration regardless of its variance. This embodies the "exploitation" aspect: only search where the model is predicted to fail.

What the variance term does: $k_t(x, x)$ is the posterior variance at $x$. Among inputs in the probable failure set, the acquisition function selects the one with the highest uncertainty. This embodies "exploration" β€” within the failure region, prioritize inputs where the GP knows the least, to refine the boundary of the failure set and potentially discover new types of failures.

The role of $\beta$:

  • $\beta = 0$ means the condition is $\mu_t(x) \geq \lambda$ β€” we only consider inputs where the point estimate of failure probability exceeds the threshold. This is pure exploitation: sample where we think the model fails.
  • $\beta < 0$ (e.g., $\beta = -1$) means the condition is $\mu_t(x) - |\beta|\sigma_t(x) \geq \lambda$ β€” we require the lower confidence bound to exceed the threshold. This is more conservative: we only sample inputs where we are highly confident of failure (low uncertainty), which refines our knowledge of known failure modes.
  • $\beta > 0$ expands the search to inputs where failure is plausible but uncertain. This encourages exploration at the boundary of the failure region.

The paper's experiments use $\beta = 0$ as the default. The acquisition function is evaluated over all inputs in $D_{\text{pool}}$, and the maximizer is selected as the next query.

Why this form: standard level set estimation acquisition functions (e.g., the Straddle heuristic from Gotovos et al., 2013) often use $|\mu_t(x) - \lambda| / \sigma_t(x)$ or similar ratios. The paper's formulation separates the classification (indicator) from the uncertainty (variance), which is simpler and more directly interpretable: first filter to the failure region, then pick the most uncertain point within it.

Failure Discovery: Generative Synthesis (SS-Gen)

A limitation of SS is that it is restricted to the static pool $D_{\text{pool}}$. If no input in the pool lies in the true failure region, SS will never find a failure. SS-Gen extends SS by using the top-ranked inputs from SS as in-context anchors for an LLM generator that synthesizes new, more challenging test cases.

The procedure:

  1. Select $m$ anchor inputs from $D_{\text{pool}}$ with the highest $\alpha_{\text{SS}}$ values.
  2. Format them as in-context examples with the instruction: "These test cases likely cause the target model to fail. Analyze their common features and generate a new, more challenging test case."
  3. The LLM generator (Gemini 3 Pro at temperature 0.7 in the experiments) produces a new input $x_{t+1}$.
  4. Evaluate the target model on $x_{t+1}$ and add the observation to $D_{t+1}$.

What this achieves: the LLM generator extracts patterns from the anchor failures and produces new inputs that share those patterns but are novel. If the anchors are all math problems involving multi-step percentage calculations, SS-Gen generates more problems in that vein. This amplifies the exploitation: once a failure mode is discovered, generate more variations on it to precisely characterize its scope.

The limitation: the paper acknowledges that SS-Gen "often semantically mimics the anchors" (Section 2.4). If the anchors are math problems about "counting apples," the LLM generates more "counting apples" problems, not "counting oranges" or "calculating distances." This leads to mode collapse β€” all generated inputs explore the same narrow failure mode, failing to discover other, conceptually different failure modes the model might have.

Failure Discovery: Topic-Aware Exploration (TSS)

TSS is the most sophisticated strategy, designed to force diversity in synthesized failures while maintaining the exploitation benefits of anchor-guided generation. It consists of three components working together (Algorithm 2).

Topic clustering ($S = \{s_i\}_{i=1}^{N_{\text{topics}}}$): the unlabeled pool $D_{\text{pool}}$ is partitioned into semantic topics using BERTopic (Grootendorst, 2022) or a pre-defined topic set. Each topic is represented by keywords β€” for GSM8K, topics might include {age, old man, counting}, {shopping, prices, discounts}, {distance, speed, time}; for StrategyQA, topics might include {history, war, presidents}, {biology, animals, diet}, {sports, Olympics, records}.

Topic selection via UCB1: topics are treated as arms in a multi-armed bandit problem. At each step, a topic $s_t$ is selected using the Upper Confidence Bound (UCB1) algorithm (Auer et al., 2002):

st=arg⁑max⁑s∈S(μ^s+clog⁑tns)s_t = \arg\max_{s \in S} \left(\hat{\mu}_s + c \sqrt{\frac{\log t}{n_s}}\right)

where $\hat{\mu}_s$ is the empirical failure rate of questions generated under topic $s$ so far (the "reward"), $n_s$ is the number of times topic $s$ has been selected, $t$ is the total number of steps, and $c$ is an exploration constant. The first term $\hat{\mu}_s$ rewards topics that have yielded failures in the past (exploitation), while the second term $c\sqrt{\log t / n_s}$ rewards topics that have been underexplored (exploration). This ensures the algorithm does not fixate on a single failure-prone topic but gradually covers all topics.

Importantly, topic selection is independent of anchor selection. The anchors are chosen from $D_{\text{pool}}$ as the inputs with the highest $\alpha_{\text{SS}}$ values (likely-failure cases), regardless of which topic they belong to. The LLM generator then receives both the anchors and the target topic, with the instruction: "Ensure the new test case belongs to this topic..." plus the standard SS-Gen prompt.

What TSS achieves β€” the key innovation: it decouples the failure pattern (extracted from the anchors) from the topic (selected for diversity). The LLM is forced to transpose the failure pattern into a semantically different context. If the anchors are math problems involving multi-step subtraction with borrowing, and the selected topic is {distance, speed, time}, the LLM must generate a speed-distance-time problem that requires the same tricky multi-step subtraction pattern. This produces a failure case that is both challenging (inheriting the exploitable weakness) and semantically diverse (belonging to a different topic).

Why this is more than a heuristic: the UCB1 algorithm provides theoretical guarantees on cumulative regret β€” the loss compared to always selecting the best topic. By treating topic selection as a bandit, TSS automatically balances the exploration-exploitation tradeoff at the topic level, while the anchor-based generation handles exploitation at the content level, and the GP's $\alpha_{\text{SS}}$ acquisition function handles exploration-exploitation at the input level. This is a hierarchy of exploration mechanisms: the GP explores the input space, anchors exploit known failure patterns, and UCB1 explores the topic space.

The complete TSS loop (Algorithm 2): initialize GP posterior from historical data and any initial target-model evaluations; cluster $D_{\text{pool}}$ into topics; at each step, select topic via UCB1, select anchors from $D_{\text{pool}}$ maximizing $\alpha_{\text{SS}}$, generate $x_{t+1}$ via LLM with anchors and topic constraint, evaluate the target model on $x_{t+1}$, update the GP posterior and topic statistics.

Query generator validity (Section 3.3): the paper addresses a subtle concern β€” the LLM generator might produce questions that are unsolvable or incorrectly answered by the generator itself, making the target model's "failure" on those questions uninformative. A human verification study on 80 randomly sampled questions found that the strong generator model (Gemini 3 Pro) answered 90% correctly, and on the 10% it answered incorrectly, the target model (Gemini 2.5 Flash) almost always gave the same incorrect answer or a different incorrect answer. The paper concludes that the reported failure rate is a lower bound on the true failure rate β€” the target model also fails on questions the generator finds hard, plus potentially additional questions.

Impact of generator quality (Table 5): stronger generators yield higher failure discovery rates. TSS-RPF discovers 41.5% failures on StrategyQA with GPT-5 as the generator, compared to 28.7% with Qwen3 (32b) and 26.3% with Gemma3 (27b). The paper's recommendation: since ProEval reduces the number of queries needed, one can afford a more expensive (and capable) generator model while staying within budget.

Evaluation Metrics for Failure Discovery

The paper defines a comprehensive set of metrics (Section 3.1.2) to assess not just whether failures are found, but whether they are diverse and discovered efficiently:

Quality and efficiency metrics:

  • Cumulative Failures: total number of failure-inducing inputs discovered. This is the primary quantity β€” it should grow quickly and saturate late.
  • Failure Rate (FR): percentage of evaluated inputs that trigger failure. A high failure rate means the acquisition function is effectively targeting the failure region.
  • Samples to First Failure (SFF): number of queries required to find the first failure. This measures cold-start efficiency: a method that requires 30 queries to find a failure is less useful than one that finds it in 3.

Diversity metrics:

  • Embedding Diversity: the normalized log-determinant of the embedding Gram matrix for a fixed number $n = 100$ of discovered failure inputs: $D_{\text{emb}} = \frac{1}{n} \log \det(K + \epsilon I)$, where $K_{ij} = e_i^T e_j$ for L2-normalized text embeddings $e_i$. The log-determinant measures the volume spanned by the embeddings in feature space β€” if all failures have similar embeddings, the matrix is near-singular and the log-determinant is very negative; if they span diverse directions, the volume is large and the score is high. The metric is normalized to $[0, 1]$.
  • Topic Entropy: Shannon entropy of the topic distribution of discovered failures, normalized by the maximum possible entropy given the number of unique topics: $H_{\text{norm}} = \frac{-\sum_{t \in T} p(t) \log_2 p(t)}{\log_2 |T|}$. A value of 1.0 (100%) means failures are perfectly evenly distributed across all topics; a value near 0 means all failures are in a single topic.
  • Overall Diversity: a composite score: $\text{Diversity} = 0.5 \cdot \frac{H_{\text{norm}}}{100} + 0.5 \cdot \min\left(\frac{D_{\text{emb}}}{2}, 1\right)$. The paper gives an intuitive interpretation: "An overall score around 0.5 indicates that, while discovered failures may span several topics, they remain semantically similar, reflecting minor variations of the same failure pattern rather than truly distinct vulnerabilities. Conversely, a high overall diversity score (e.g., β‰₯0.90) indicates the discovery of highly distinct, semantically unique vulnerabilities spread evenly across multiple topics."

Modality Transfer and Reasoning Trace Ablations

Two additional technical investigations round out the approach:

Cross-modality transfer (Section 3.2): the paper reports a preliminary experiment transferring knowledge from DICES (text-based safety ratings) to DIVE (text-to-image safety evaluation). With 15 samples on DIVE, using DICES data for the GP prior reduced MAE from 0.111 to 0.055 compared to no transfer. This suggests that the embedding-based prior (TPF/RPF) can capture cross-modal performance correlations β€” text and image prompts with similar semantic content have correlated safety properties.

Reasoning trace integration (Table 2): the paper compares three strategies for incorporating the target model's Chain-of-Thought (CoT) reasoning into the GP embedding:

  • Q-Only: embed only the question text, ignoring the model's reasoning.
  • Concat: concatenate the question text with the model's reasoning trace and embed the combined text.
  • Fusion: embed the question and reasoning trace separately, then compute a weighted average of the embeddings (with weight $\alpha = 0.7$ on the question, 0.3 on the reasoning).

The results are task-dependent: Fusion performs best on StrategyQA (MAE 0.071 vs. Q-Only 0.128) and ToxicChat (MAE 0.001 vs. Q-Only 0.125), while Concat performs best on GSM8K (MAE 0.022 vs. Fusion 0.009 β€” wait, Fusion is better at 0.009). The takeaway is that incorporating CoT traces generally helps (the model's mistakes are often encoded in its reasoning), but the optimal integration method varies, and no pre-training was done in this ablation β€” suggesting further gains are possible by optimizing the embedding jointly with the GP on historical CoT data.

Summary of Key Design Choices and Their Justifications

  • Gaussian likelihood for binary data: a deliberate trade-off enabling closed-form GP posteriors and Bayesian quadrature. The BQ Rounded variant provides a partial mitigation. The paper experimented with a GP classifier (Laplace approximation) and found it underperforms, likely due to approximation errors. This is flagged as an open direction ("more investigations on transfer learning for GPC is required").
  • Linear kernel over score features: when historical data exists for the same inputs, this is the most statistically principled choice because it exactly reconstructs the empirical covariance matrix. The computational advantages (Sherman-Morrison updates, $O(N^3)$ instead of $O(t^3)$) are a bonus.
  • MatΓ©rn kernel over prompt features: when transferring across benchmarks, the MatΓ©rn kernel provides more flexibility than the linear kernel for modeling non-linear relationships between embedding similarity and failure correlation. The centering operation ($\phi_\theta$) is retained for consistency with the score-feature approach.
  • GMM clustering with abstention: the simplest method (leave-one-out prior) is competitive in many cases, but GMM with abstention achieves the best median performance by filtering out harmful source data and refusing to predict when the signal is too weak. The 22% abstention rate is acceptable because it prevents the 100Γ— error spikes seen in Figure 7.
  • Variance-reduction acquisition for estimation: unlike importance sampling, BQ variance reduction is observation-independent, enabling batch pre-computation and avoiding high-variance IS weights. The objective directly targets the quantity of interest (the integral), not individual predictions.
  • Three-tier hierarchy for failure discovery: SS (GP uncertainty in the failure region) β†’ SS-Gen (LLM-generate more of the same) β†’ TSS (LLM-generate with forced topic diversity). This progression addresses the diversity-efficiency tradeoff: each tier adds a mechanism for covering more of the failure landscape while retaining the exploitation guidance from the GP.
  • UCB1 for topic selection: a principled bandit algorithm that automatically balances exploring underexplored topics with exploiting failure-prone topics. The independence between topic selection and anchor selection is the key insight that enables transposing failure patterns across topics.

4. Key Insights and Innovations

Innovation 1: Unifying Performance Estimation and Failure Discovery as Dual Bayesian Queries on the Same Latent Function

The paper's most fundamental conceptual move is not any specific algorithm but the realization that performance estimation and failure discovery are two views of the same underlying uncertainty β€” and that treating them jointly in a single probabilistic framework yields synergies that separate treatment misses.

Before ProEval, these two problems inhabited essentially non-overlapping literatures. Performance estimation (Kossen et al., 2021, 2022; Polo et al., 2024) was framed as a statistical estimation problem: reduce the variance of an aggregate estimator by cleverly choosing which samples to evaluate. Failure discovery / red teaming (Perez et al., 2022; Chao et al., 2024; Lee et al., 2023) was framed as an optimization or search problem: find inputs that maximize some measure of model failure. These communities used different formalisms (importance sampling estimators versus adversarial attack algorithms), different evaluation metrics (MAE versus attack success rate), and different surrogate models (logistic regression versus separately trained classifiers). The two objectives were pursued in isolation, often by different teams within the same organization.

ProEval's unifying insight is that both objectives are queries against a shared latent function $f(x)$ β€” the model's failure probability as a function of the input β€” and therefore both benefit from the same probabilistic surrogate and the same collected data. The performance integral $S = \int f(x) p(x) dx$ and the failure superlevel set $\mathcal{X}_\lambda = \{x \mid f(x) \geq \lambda\}$ are different functionals of the same posterior distribution over $f$. Every evaluation of the target model simultaneously refines both the aggregate estimate and the boundary of the failure region.

This unification is intellectually significant because it changes the economic calculus of evaluation. In prior work, a team might spend 100 queries on performance estimation (getting a good accuracy number) and then spend another 100 queries on red teaming (finding specific failures), with no sharing of information between the two processes. In ProEval, those 200 queries shared across both objectives would collectively produce both a better accuracy estimate and a more comprehensive failure map than either 100-query run alone, because the GP posterior is conditioned on all 200 observations regardless of which acquisition function selected them. The paper does not explicitly quantify this synergy in a shared-budget ablation, but it is a direct consequence of the mathematical framework: the posterior $f \mid D_t$ in Equation (2) depends only on the collected data $D_t$, not on which objective motivated each acquisition.

The framing also resolves a tension in the active testing literature: methods like Kossen et al. (2021) treat evaluation as pure estimation, selecting samples that minimize estimator variance regardless of whether those samples lie in failure or success regions. But practitioners care about failures β€” they want to know not just "how often does the model fail?" but "on what kinds of inputs does it fail?" ProEval's dual-objective formulation makes this explicit: the performance estimate answers the first question, while the superlevel set answers the second. Both arise naturally from the same GP, without requiring a separate pipeline.

This is a fundamental reframing rather than an incremental improvement. It redefines what "evaluation" means: not a point estimate of accuracy, but a joint characterization of aggregate behavior and failure structure. The downstream implications are practical β€” a single evaluation run now serves multiple stakeholders (model developers want the aggregate metric; safety teams want the failure cases) β€” but the intellectual contribution is the recognition that these are not separate problems at all.

Innovation 2: Transfer Learning from Historical Model Performance as a General-Purpose Evaluation Prior

The second major contribution is demonstrating that the collective history of evaluating many models on many benchmarks constitutes a transferable prior that can dramatically accelerate evaluation of new models and new benchmarks β€” and that this prior can be constructed in two complementary ways (score-based and embedding-based) that cover different practical scenarios.

This idea is deceptively simple, but it represents a substantial departure from prior work. The dominant paradigm in efficient evaluation has been either static (prune a fixed subset of questions based on historical models and hope it generalizes to new targets) or cold-start (begin with no prior knowledge and learn from scratch as evaluations arrive). Both have fundamental limitations that the paper diagnoses clearly (Appendix A): static methods are target-agnostic ("they assume the failure modes of the target model perfectly align with the source models"), while dynamic methods waste their initial budget "identifying difficulty regions that are often already known to the community."

ProEval's transfer learning approach occupies a third paradigm: use historical data to construct a strong prior, then adapt it to the specific target with minimal new data. The prior is informative enough to skip the cold-start phase entirely β€” as Figure 6 shows, BQ-SF often reaches 1% MAE after only 1–2 evaluations β€” but the Bayesian updating mechanism ensures the posterior can diverge from the prior if the target model exhibits idiosyncratic behavior. The prior is a starting point, not a straightjacket.

The dual strategy for constructing the prior (Section 2.2) is worth highlighting as an intellectual contribution in itself. The paper recognizes that transfer learning for evaluation comes in two distinct forms depending on data availability:

Score-feature transfer (Section 2.2.1): when historical models have been evaluated on the exact same inputs as the target benchmark, the empirical covariance matrix of model scores directly provides a GP kernel. The construction β€” treating each input's vector of historical model scores as its feature, and defining the kernel as dot products in this score space β€” is mathematically elegant: it exactly reconstructs the empirical covariance under a linear kernel (Equation 7). But the real insight is that this covariance matrix encodes which questions test similar capabilities. Questions that tend to be jointly hard across models (positive covariance) are inferred to be correlated for the target; questions where models diverge (negative covariance) are inferred to test different skills. Figure 2 provides striking visual evidence: the covariance matrices for StrategyQA, GSM8K, and MMLU all show distinct block structure, meaning the correlation patterns are real and exploitable.

This is a fundamentally different use of historical evaluation data than static pruning. Pruning methods use historical data to select a subset of questions. ProEval uses historical data to learn a correlation structure over all questions. The distinction matters because a pruned subset throws away questions that might be informative for a particular target, while the learned covariance structure retains all questions and uses correlations to infer performance on unevaluated ones from evaluated ones.

Prompt-feature transfer (Section 2.2.2): when the target benchmark consists of entirely new questions with no historical scores, the paper shows that transfer is still possible through semantic embeddings. By training a kernel on top of pre-trained text/image embeddings, the GP learns that questions with similar semantic content tend to have similar difficulty across models. This enables zero-shot generalization: evaluating a new math question about fractions is informative about another new math question about fractions, even though neither has ever been evaluated on any model before.

The two strategies complement each other elegantly. Score features are more powerful when available (because they directly capture model-specific difficulty patterns, not just semantic similarity), but they require the strong Assumption 2 (identical inputs across models). Prompt features are weaker but more general: they work across benchmarks, across modalities (the cross-modal DICES→DIVE transfer experiment in Section 3.2), and on entirely novel inputs. The paper's framework accommodates both within the same mathematical structure (a GP with either a linear kernel over score features or a Matérn kernel over embedding features), making it a unified transfer learning framework for evaluation.

The practical significance is substantial but the intellectual contribution is the recognition that evaluation data from different models is not independent β€” it samples a shared structure (Assumption 1) β€” and that this structure can be captured by a GP prior and transferred to new targets. This transforms historical evaluation data from a byproduct of model development into a reusable asset that compounds in value as more models are evaluated.

Innovation 3: Diagnosing the "Similarity Assumption" and Operationalizing It via GMM Clustering with Abstention

A subtler but equally important contribution is the paper's treatment of negative transfer β€” the problem that transferring from dissimilar historical models can harm estimation accuracy β€” not as a nuisance to be hand-waved away but as a diagnosable, operationalizable phenomenon that can be systematically addressed.

The issue is straightforward to state: Assumption 1 posits that all historical models' score functions are drawn from a shared GP prior. But real models differ β€” a small open-source model fine-tuned on math will have qualitatively different failure patterns than a large proprietary model optimized for safety. If you include the former in the prior for evaluating the latter, you are injecting misleading information. The paper quantifies this in Figure 7: using G0 (Gemma-3-12B) as a source for evaluating G2 (Gemini 2.5 Flash) on GSM8K yields an MAE > 0.10, roughly 10Γ— worse than using G1 (Gemma-3-27B) or another similar model, where MAE < 0.01.

The naive solution β€” use all available historical data β€” is what prior work implicitly did. The paper's GMM clustering approach (Section 2.2.3) instead projects models into a behavior space (using PCA on their per-question score vectors on a reference benchmark) and clusters them. The intuition, validated by Figure 10, is that models with similar failure patterns occupy the same region of this space. The prior is constructed only from models in the target's cluster, filtering out dissimilar ones.

What elevates this from an engineering heuristic to an intellectual contribution is the abstention mechanism and the empirical analysis in Appendix E. The paper does not simply claim that filtering helps β€” it systematically compares seven selection methods (GMM, Spearman correlation, Mahalanobis distance, leave-one-out likelihood, Mardia's normality test, hypothesis testing, random) across 78 experiments and reports that GMM with abstention when the cluster has fewer than 3 models achieves the best median MAE (0.0109). Crucially, this method abstains in 22% of cases β€” it refuses to make a prediction when the prior is unreliable. The comparison with GMM without abstention (median MAE 0.0125, no abstentions) shows that the cases where GMM selects only 1–2 models are precisely the failure modes, and abstaining on those cases is what drives the improvement.

This is a diagnostic contribution: the paper shows not just that transfer learning works, but when it works and when it doesn't, and provides a principled mechanism for detecting the latter. The abstraction is that transfer learning for evaluation requires a similarity assumption (models in the source set must be drawn from a similar distribution as the target), and the paper operationalizes this assumption through clustering and abstention. The fact that a simple leave-one-out prior (using all historical models without filtering) achieves a respectable median MAE of 0.0125 (Table 26) shows that the similarity assumption holds often enough to be useful β€” but the GMM with abstention is safer and more robust.

The theoretical significance is that this establishes a boundary condition for the transfer learning approach. Theorem 3's bound depends on $N$ (number of historical models) and holds under Assumption 1. But Assumption 1 is an idealization β€” real models are not exact samples from a shared GP. GMM clustering is a practical mechanism for approximating Assumption 1 by selecting a subset of models for which the assumption is plausible, and the abstention rule is a safety valve for when even that subset is insufficient. The paper is effectively saying: "Here is a theoretical guarantee under ideal conditions, and here is a practical method for creating approximately ideal conditions, with a diagnostic for when even that approximation fails." This combination of theory plus diagnostic is more valuable than either alone.

Innovation 4: Verifier-Free Active Selection via Observation-Independent Variance Reduction

The paper makes an understated but practically significant contribution in the design of the active acquisition function for performance estimation (Section 2.3). The key property is that the variance reduction objective is independent of the observed $y$ values β€” it depends only on the location $x$, not on what the target model actually does at $x$. This is not a generic property of all active learning; it holds specifically for Bayesian quadrature with a Gaussian likelihood.

The consequence is that the entire acquisition schedule can be pre-computed before evaluating the target model even once. Given the GP prior and the pool of available test inputs, one can run the greedy maximization of Equation (10) offline to produce an ordered list of inputs, then evaluate them in that order. This contrasts with standard active testing (Kossen et al., 2021), where the surrogate model must be updated after each observation to decide the next sample, creating a sequential dependency that prevents parallelization.

This property matters practically: if one has access to a cluster of GPUs and wants to evaluate a model on 100 inputs, the ability to pre-compute which 100 inputs to evaluate and run them all in parallel reduces wall-clock time from 100 sequential rounds to one parallel batch. The paper mentions this in passing ("it is possible to pre-compute batches of test inputs for parallel evaluation," Section 2.3) but the implication is broader: it decouples the computational cost of acquisition (which is cheap, involving only matrix operations on the $d \times d$ posterior precision matrix) from the cost of evaluation (which is expensive, involving model inference).

The reason this is an intellectual contribution rather than a trivial consequence of the Gaussian model is that it reframes the role of the surrogate. In importance-sampling-based active testing, the surrogate (a logistic regression or random forest) must be accurate enough to produce useful sampling probabilities, but its predictions ($\hat{f}(x)$) directly influence which inputs are selected. If the surrogate is miscalibrated, the selection is suboptimal. ProEval's BQ acquisition function does not depend on the surrogate's point predictions at all β€” it depends only on the kernel, which encodes the correlation structure learned from historical data. The variance reduction from evaluating at $x$ is large if $x$ is correlated with many unevaluated test inputs, regardless of whether the GP predicts success or failure at $x$.

This makes the acquisition process robust to errors in the prior mean β€” a significant advantage when transferring from historical models that may differ in overall accuracy from the target. Even if the prior mean $\mu(x)$ is biased (e.g., the historical models are all weaker than the target, so the prior overestimates failure probability), the variance structure $k(x, x')$ might still be informative (the relative difficulty of questions is preserved), and the acquisition function relies only on the latter. The posterior mean will be corrected as observations arrive, but the acquisition schedule, being observation-independent, is fixed from the start.

This property is not available with non-Gaussian likelihoods (e.g., a Bernoulli observation model), which is part of why the paper chose the Gaussian approximation for binary data. The loss of statistical correctness is compensated by a gain in computational and operational flexibility. The paper's brief exploration of a GP classifier with Laplace approximation in Section 3.2 (yielding worse results) suggests that this trade-off favors the Gaussian approximation in practice, at least for the benchmarks studied.

Innovation 5: Topic-Aware Synthesis as a Decoupled Diversity Mechanism for Failure Discovery

The failure discovery pipeline (Section 2.4) culminates in the Topic-Aware Superlevel Set sampling (TSS) strategy, which embodies a conceptual insight about how to enforce diversity in generative failure discovery without sacrificing exploitation of known failure patterns. The insight is that diversity and exploitation operate on orthogonal axes β€” what makes an input fail (the failure pattern) versus what semantic domain it belongs to (the topic) β€” and that these can be controlled independently.

Prior generative red teaming methods (Chao et al., 2024; Perez et al., 2022) suffer from a fundamental tension: if you optimize for failure, the generator collapses to a narrow failure mode (e.g., repeatedly generating variations on the same adversarial template). If you enforce diversity heuristically (e.g., via Self-BLEU penalties as in Lee et al., 2023), you risk generating inputs that are diverse but not actually challenging. The field lacked a principled way to decouple these objectives.

ProEval's TSS decouples them structurally: the anchors (selected by the GP's $\alpha_{\text{SS}}$ acquisition function to exploit known failure regions) provide the failure pattern β€” the reasoning trap, the ambiguous phrasing, the missing constraint that causes the model to fail. The topic (selected by UCB1 to explore underexplored semantic domains) provides the semantic context in which that pattern is instantiated. The LLM generator is instructed to transpose the failure pattern into the new topic, producing an input that is both challenging (because it inherits the exploitable pattern) and semantically novel (because it belongs to a new topic).

This is a structural innovation in how generative failure discovery is organized. The three-tier hierarchy (SS β†’ SS-Gen β†’ TSS) is not just adding features incrementally β€” it represents a progression in what is being optimized:

  • SS optimizes for failure within a static pool (pure retrieval).
  • SS-Gen optimizes for failure via generation but risks topic collapse.
  • TSS optimizes for diverse failure via generation by introducing a second optimization loop (the bandit over topics) that is explicitly decoupled from failure exploitation.

The bandit formulation itself (UCB1 over topics) is standard, but its application here is clever because the reward signal β€” the failure rate of generated questions in that topic β€” is naturally provided by the evaluation loop. TSS does not need a separate diversity metric or reward function; it uses the same binary failure signal that drives everything else, aggregated per topic. Topics that yield failures get higher UCB scores and are selected more often until their uncertainty bonus decays, at which point underexplored topics become more attractive. This automatically balances exploitation and exploration at the topic level without manual tuning.

The empirical evidence for this decoupling's value is in Table 3: TSS-RPF achieves the highest overall diversity score (0.74 on StrategyQA, 0.73 on GSM8K) while maintaining competitive failure rates. The comparison with Rand-Anchor-Gen is particularly informative: Rand-Anchor-Gen uses the same prompt structure as SS-Gen but replaces BQ-selected anchors with uniformly random ones. It achieves comparable topic entropy (99.5%) but substantially lower failure rates (22.3% vs. 40.3% for SS-Gen-RPF on StrategyQA) β€” demonstrating that random anchors provide diversity without exploitation, while BQ-selected anchors (in SS-Gen) provide exploitation without diversity. TSS achieves both, simultaneously. The qualitative examples in Tables 23–24 further illustrate the difference: random generation produces simple, direct questions that the target model easily solves (Score 0.0), while TSS produces complex, multi-hop reasoning questions that consistently trigger failures (Score 1.0) across diverse topics.

This contribution is architectural rather than algorithmic: the components (GP, UCB1, LLM generation) are individually standard, but the way they are composed β€” with explicit decoupling of what drives exploitation and what drives diversity β€” represents a design pattern that could generalize to other domains where generative diversity matters (synthetic data generation, curriculum learning, scenario generation for simulation-based testing).

5. Experimental Analysis

Evaluation Methodology

  • Datasets. The paper evaluates on 9 benchmarks spanning three domains. Mathematical/logical reasoning uses GSM8K (Cobbe et al., 2021; subset of 1,319 questions sampled from 8,500), SVAMP (Patel et al., 2021; 700 from 1,000), StrategyQA (Geva et al., 2021; 1,603 from 2,780), and GQA (Hudson and Manning, 2019; 2,000 from 22.7M). General world knowledge uses MMLU Professional Law (Hendrycks et al., 2021; 1,534 questions). Safety alignment uses ToxicChat (Lin et al., 2023; 1,500 from 10,000), Jigsaw Civil Comments (Borkan et al., 2019; 1,500 from 150K), DICES-350 (Aroyo et al., 2023; 1,500 from 42K), and text-to-image DIVE (Rastogi et al., 2025; 1,500 from 38,410). Subsampling is applied "to accelerate experimentation and reduce inference costs" (Section 3.1.1, Appendix C.1). For the Default and New Model scenarios, the target benchmark uses all available questions; for New Bench, no historical scores exist for any question. The GMM-based source data selection uses a reference benchmark that includes "all available benchmarks except the target" (Section C.2).

  • Base models. Sixteen LLMs and VLMs from four families are evaluated as both target models and historical sources. Google models: Gemma-3-12B (G0), Gemma-3-27B (G1), Gemini 2.5 Flash (G2) and Pro (G3), Gemini 3 Flash (G4) and Pro (G5). OpenAI models: GPT-3.5 Turbo (O1), GPT-4o (O2), GPT-5 (O3), GPT-5.1 (O4), GPT-5.2 (O5). Anthropic models: Claude 3.5 Haiku (C1), Claude 3.7 Sonnet (C2), Claude 4.5 Sonnet (C3) and Opus (C4). Qwen3-32B (Q1). Multi-modal benchmarks (DIVE, GQA) exclude text-only architectures. The primary target for most experiments is Gemini 2.5 Flash (G2); for the NM scenario, target model data is entirely withheld from prior construction; for the Default scenario, all models serve as targets in turn (Tables 7–21). The selection of 16 models spanning four families, architectures, and capability levels tests whether transfer learning generalizes across qualitatively different model behaviors.

  • Metrics. Performance estimation accuracy is measured by Mean Absolute Error (MAE): $\text{MAE} = |\hat{S}_t - S^*|$, where $\hat{S}_t = \frac{1}{M} \sum_{j=1}^M \hat{\mu}_t(x_j)$ is the BQ estimate after $t$ evaluations and $S^* = \frac{1}{M} \sum_{j=1}^M f(x_j)$ is the ground-truth average failure rate over all $M$ test inputs. Sample efficiency is measured by the Number of Samples to reach $\leq 1\%$ MAE (@1% MAE), tracking the minimum evaluations required to cross that threshold. Failure discovery quality is measured by Cumulative Failures (total count of failure-inducing inputs discovered), Failure Rate (FR, percentage of evaluated inputs triggering failure), and Samples to First Failure (SFF). Failure diversity uses three metrics: Embedding Diversity ($D_{\text{emb}} = \frac{1}{n} \log \det(K + \epsilon I)$ computed over $n=100$ normalized L2 embeddings, normalized to $[0,1]$), Topic Entropy (Shannon entropy of topic proportions normalized by $\log_2 |T|$), and Overall Diversity, a composite score: $0.5 \cdot H_{\text{norm}}/100 + 0.5 \cdot \min(D_{\text{emb}}/2, 1)$.

  • Baselines. Performance estimation is compared against five methods. Random Sampling uniformly selects test cases and estimates accuracy via the sample mean. Four active testing strategies from Kossen et al. (2021) are evaluated: Logistic Regression + Importance Sampling (LR+IS), Logistic Regression + LURE (LR+LURE), Random Forest + Importance Sampling (RF+IS), and Random Forest + LURE (RF+LURE). All active baselines train surrogate models on evaluation results from 11 historical models, compute acquisition probabilities $q(i)$, and apply either standard IS weighting or the LURE sequential-without-replacement correction. For failure discovery, sampling baselines include Random (Rand) for static pool retrieval and Rand-Gen (unguided LLM generation), Rand-T-Gen (random topic constraint without anchors), and Rand-Anchor-Gen (random anchors with the same prompt structure as SS-Gen). The paper also compares BQ Rounded (posterior means rounded to $\{0,1\}$ before summation) against standard BQ for discrete evaluation.

  • Generation budget / compute accounting. The primary unit of test-time computation is the number of inputs evaluated on the target model. Performance estimation experiments use a budget of 1% of the benchmark size (e.g., ~15 inputs for a 1,500-question benchmark) for the main MAE comparisons (Table 1), with budgets of up to 20–100 inputs shown in convergence plots (Figures 5–6). Failure discovery experiments use a budget of $T=100$ iterations (Figure 8, Table 3), where each iteration synthesizes and evaluates one new input on the target model. The LLM generator (Gemini 3 Pro at temperature 0.7) is run once per iteration and its cost is shared across all synthesis-based methods; the paper notes this enables using a more expensive generator since ProEval reduces the number of queries needed (Table 5 discussion). The 2048-sample difficulty estimation cost discussed in Section 2.2.1 is not included in reported generation budgets.

  • Cross-validation / statistical protocol. Performance estimation experiments report means and standard deviations over 5 independent runs where applicable (Table 1, $\pm$ values). BQ methods with active selection are deterministic (the acquisition function is observation-independent), so no standard deviation is reported for those entries. For the Default scenario, GMM clustering (Β§2.2.3) selects source models using a reference benchmark that excludes the target benchmark to prevent data leakage. For the NM scenario, all historical data are used since no target-specific results exist. For the NB scenario, all results are zero-shot (no target-benchmark historical data). Failure discovery experiments use 10 independent runs and report standard deviations in Figure 8. Cross-validation for the compute-optimal strategies is not applicable (ProEval does not involve strategy selection over hyperparameters in the same way as the reference example).

Main Quantitative Results

Performance Estimation Efficiency on the Default Scenario

Headline result. Table 1 reports MAE at a 1% labeling budget for Gemini 2.5 Flash (G2) across 9 benchmarks. The BQ-SF variant (score features with active selection) achieves MAE values between 0.002 (DIVE) and 0.067 (DICES) across benchmarks, with the BQ-SF Rounded variant reaching 0.001 on both Jigsaw and ToxicChat. In contrast, Random Sampling achieves MAE between 0.040 Β± 0.005 (GSM8K) and 0.108 Β± 0.078 (DIVE). The best active baseline (RF+LURE) ranges from 0.028 Β± 0.029 (StrategyQA) to 0.129 Β± 0.073 (DIVE).

Active selection vs. random selection with the same prior. Comparing "Active Selection + BQ" against "Random Selection + BQ" within each feature type (RPF, TPF, SF) in Table 1 isolates the contribution of the acquisition function. For BQ-SF, active selection is almost always better than random selection: on GSM8K, Active BQ-SF achieves 0.013 vs. Random BQ-SF Rand at 0.023; on MMLU, 0.006 vs. 0.048; on ToxicChat, 0.002 vs. 0.001 β€” the latter being one of the few cases where random matches active, likely because the prior is already so strong that any sample suffices. For BQ-RPF and BQ-TPF, the pattern is mixed: active selection sometimes underperforms random (e.g., BQ-RPF: 0.177 Active vs. 0.091 Random on GQA; BQ-TPF: 0.253 Active vs. 0.078 Random on DICES), suggesting that when the prior is weaker (prompt-based transfer), the observation-independent acquisition function can select suboptimal inputs compared to random chance.

Score features dramatically outperform prompt features. Across all benchmarks, BQ-SF achieves substantially lower MAE than BQ-RPF and BQ-TPF at the same budget. Averaging across the 9 benchmarks (Table 1, Active Selection rows for G2): BQ-SF mean MAE β‰ˆ 0.015, BQ-RPF mean β‰ˆ 0.087, BQ-TPF mean β‰ˆ 0.072. The gap is most pronounced on benchmarks where semantic similarity alone is a weak predictor of model failure β€” e.g., DICES (BQ-SF: 0.067 vs. BQ-RPF: 0.178 vs. BQ-TPF: 0.253) and StrategyQA (BQ-SF: 0.011 vs. BQ-RPF: 0.128 vs. BQ-TPF: 0.022). This confirms that direct historical performance data on the exact same inputs provides a much stronger prior than learned semantic embeddings, as expected from the theoretical construction in Section 2.2.1.

Figure 5 convergence analysis. The per-step MAE trajectories for G2 on Jigsaw, GSM8K, MMLU, and GQA (Figure 5) reveal distinct convergence behaviors. BQ-SF starts with near-zero error (often below 0.01 from step 1) and remains flat β€” the prior is already so accurate that additional evaluations contribute minimally. BQ-RPF and BQ-TPF start at higher error (0.05–0.15) but decrease steadily, with BQ-TPF generally converging faster than BQ-RPF due to the learned encoder capturing finer-grained performance correlations. Random Sampling and the active testing baselines (RF+IS, RF+LURE) oscillate with high variance throughout, rarely approaching the BQ variants' accuracy within 20 steps. A notable exception: on Jigsaw (Figure 5a), RF+LURE is competitive with BQ-RPF at some steps, though still above BQ-SF and BQ-TPF.

Figure 6 sample efficiency. The @1% MAE metric (Figure 6) quantifies the efficiency gap in absolute terms. On Jigsaw: Random requires 59 samples, BQ-SF requires 1. On GSM8K: Random requires 105, BQ-SF requires 2. On MMLU: Random requires 74, BQ-SF requires 2 (BQ-RPF: 23, BQ-TPF: 7). On GQA: Random requires 81, BQ-SF requires 2. The best active baseline (RF+LURE on Jigsaw: 65, on GSM8K: 88, on MMLU: 29, on GQA: 52) is consistently 8–65Γ— less efficient than BQ-SF. This is the source of the paper's headline "8–65Γ— fewer samples" claim.

Comparison against baselines in Table 1. Excluding the BQ variants, the best baseline depends on the benchmark: RF+IS achieves 0.033 Β± 0.022 on DIVE (beating Random's 0.108), while RF+LURE achieves 0.028 Β± 0.029 on StrategyQA. However, no baseline consistently beats Random Sampling across all benchmarks β€” on GSM8K, Random achieves 0.040, better than LR+IS (0.060) and RF+LURE (0.034). This inconsistency highlights the challenge of importance-sampling-based active testing with limited surrogate model quality. ProEval's BQ-SF, by contrast, is either best or near-best on every benchmark.

Generalization to Other Target Models

Tables 7–21 (Appendix C.3.1). The paper evaluates all 16 models as targets using the same protocol as Table 1. The pattern generalizes broadly: BQ-SF with active selection is the best or among the best methods for almost every model-benchmark pair. Selected highlights:

  • GPT-5 (Table 9): BQ-SF achieves MAE of 0.001 on StrategyQA, 0.005 on ToxicChat, 0.007 on SVAMP. Random Sampling achieves 0.064 Β± 0.029, 0.052 Β± 0.024, and 0.065 Β± 0.031 respectively.
  • Claude 4.5 Opus (Table 15): BQ-SF achieves 0.006 on ToxicChat, 0.019 on SVAMP, 0.023 on StrategyQA. Random achieves 0.057 Β± 0.013, 0.063 Β± 0.042, 0.059 Β± 0.036.
  • Gemma 3 12B (Table 19): BQ-SF achieves 0.003 on ToxicChat but notably struggles on GSM8K (0.298) and SVAMP (0.180), where the prior quality is degraded β€” Gemma 3 12B is a relatively weak model whose failure patterns differ substantially from the stronger models dominating the historical data. This is a case where the GMM-based source selection (Figure 10) correctly identifies dissimilarity but does not have enough similar models to build a strong prior, resulting in higher error despite active selection.

Figure 4 (StrategyQA across all models). The bar chart shows MAE for 16 target models on StrategyQA using a 1% budget. BQ-SF (dark bars) generally achieves the lowest MAE, often below 0.02, while Random Sampling (light bars) varies widely with some models exceeding 0.08–0.10. The models where BQ-SF is less dominant β€” G0 (Gemma 3 12B), O1 (GPT-3.5 Turbo), Q1 (Qwen 3 32B) β€” are the ones identified by GMM clustering as outliers (Figure 10). The paper notes (Section 3.2): "When these outlier models serve as targets, their behavior deviates significantly from the source data, making it difficult to learn an informative prior. Despite this, ProEval's performance remains competitive."

Special Scenarios: New Bench (NB) and New Model (NM)

New Bench (NB) in Table 1 (bottom block). In this scenario, no historical data exists for the target benchmark's specific inputs. Transfer relies entirely on semantic similarity via prompt features (RPF or TPF). BQ-RPF (NB) achieves MAE ≀ 0.053 on 6 of 9 benchmarks (GSM8K: 0.017, DIVE: 0.031, Jigsaw: 0.039, MMLU: 0.053, GQA: 0.016, ToxicChat: 0.003) but fails on DICES (0.260), StrategyQA (0.134), and SVAMP (0.083). Comparing BQ-RPF (NB) to Random Sampling: BQ-RPF improves on 6 benchmarks (e.g., GSM8K 0.017 vs. 0.040; ToxicChat 0.003 vs. 0.064), is roughly equal on MMLU (0.053 vs. 0.063 Β± 0.050), and underperforms on DICES (0.260 vs. 0.100), StrategyQA (0.134 vs. 0.062), and SVAMP (0.083 vs. 0.065). The failures on DICES and StrategyQA suggest that semantic similarity alone is insufficient for safety-rating alignment and implicit reasoning β€” the embedding space does not capture the nuanced patterns that make one question harder than another. BQ-TPF (NB) underperforms BQ-RPF (NB) on most benchmarks (e.g., DICES 0.252 vs. 0.260, MMLU 0.073 vs. 0.053), indicating that tuning on historical data that excludes the target benchmark can overfit to irrelevant patterns.

New Model (NM) in Table 1. In this scenario, the target model has no historical results, but the benchmarks are standard (so score features are available from other models). BQ-SF (NM) performs nearly identically to BQ-SF in the Default scenario: MAE on GSM8K is 0.017 vs. 0.013; on MMLU: 0.007 vs. 0.006; on StrategyQA: 0.011 vs. 0.011. This is because the score features are constructed from other models' behavior on the same inputs (Assumption 2), and the target model's own historical data is not needed β€” the prior is entirely transfer-based. The small degradation on some benchmarks (DICES 0.038 vs. 0.067? Actually 0.038 is better) reflects that GMM clustering cannot use the target model's profile on a reference benchmark (it is unknown), so all available models are used, which can include dissimilar ones.

Failure Discovery: Quantity and Efficiency

Figure 8 cumulative failures. Across both StrategyQA (implicit reasoning) and GSM8K (math) domains, the generative strategies (SS-Gen, TSS) maintain near-linear growth in cumulative failure count over 100 iterations, while random generation baselines (Rand-Gen, Rand-T-Gen) exhibit flatter, sub-linear growth. On StrategyQA with RPF (Figure 8a): TSS-RPF reaches ~50 cumulative failures by iteration 100, SS-Gen-RPF reaches ~45, while Rand-Gen and Rand-T-Gen plateau around 30. On GSM8K (Figure 8c): the gap is larger β€” TSS-RPF and SS-Gen-RPF reach ~35 cumulative failures, while Rand-Gen stagnates around 7. The active methods are discovering new failure cases throughout the budget, while random methods stop finding new failures early.

Failure Rate and SFF (Table 3). On StrategyQA, SS-RPF achieves 48.0% failure rate with SFF of 3.0 β€” meaning nearly half of evaluated inputs trigger failure and the first failure is found almost immediately. Rand achieves only 17.3% failure rate with SFF of 9.3. On GSM8K, SS-RPF achieves 30.0% failure rate (SFF 4.0) vs. Rand at 7.0% (SFF 11.7). Among generative methods, SS-Gen-RPF achieves the highest failure rates: 40.3% on StrategyQA, 27.7% on GSM8K. TSS-RPF achieves 26.0% on StrategyQA and 31.3% on GSM8K β€” slightly lower failure rates than SS-Gen-RPF but with better diversity (discussed below), reflecting the expected exploration-exploitation tradeoff.

The 2–5Γ— failure discovery improvement claim (Section 4) derives from comparing TSS-TPF against Rand-Gen on the most challenging targets (Table 4): on GPT-5, TSS-TPF discovers 18.9% failures on StrategyQA vs. 5.1% for Rand-Gen (a 3.7Γ— improvement); on GSM8K, 16.8% vs. 3.2% (5.25Γ—). For weaker targets, the ratio is smaller but still substantial: Gemma3 (27b) on GSM8K shows 57.4% vs. 15.4% (3.7Γ—).

Failure Discovery: Diversity

Topic Entropy (Table 3). TSS-RPF achieves 99.0% on StrategyQA and 98.8% on GSM8K β€” near-perfect topic coverage. In contrast, SS-Gen-RPF achieves 95.2% and 96.2%, and Rand-Gen achieves 99.3% and 99.6%. The topic entropy is generally high across all generative methods (the LLM naturally produces varied topics when not constrained), but the difference emerges in Embedding Diversity: TSS-RPF achieves 1.00 and 0.95 on StrategyQA and GSM8K respectively, while SS-Gen-RPF achieves 0.95 and 0.99, and Rand-Gen achieves 0.57 and 0.78. Rand-Gen produces diverse topics but semantically shallow failures β€” the questions are varied in surface topic but similar in reasoning depth and difficulty, resulting in low embedding diversity. TSS forces both topic diversity (through the bandit) and semantic depth (through anchor transposition), resulting in the highest Overall Diversity scores: 0.74 and 0.73 for TSS-RPF vs. 0.64 and 0.69 for Rand-Gen.

Rand-Anchor-Gen isolates the anchor effect. This baseline uses the same prompt structure as SS-Gen but replaces BQ-selected anchors with uniformly random ones. It achieves high topic entropy (99.5% and 99.0%) and high embedding diversity (0.98 and 1.00) but low failure rates (22.3% and 6.3%) and high SFF (4.3 and 27.3 on GSM8K). The comparison with SS-Gen (failure rates 40.3% and 27.7%) shows that BQ-selected anchors are the primary driver of failure detection efficiency β€” random anchors provide diversity but fail to target the model's actual weaknesses.

SS vs. SS-Gen: retrieval vs. generation. Comparing SS-RPF (static pool retrieval) with SS-Gen-RPF (generation): on StrategyQA, SS-RPF achieves 48.0% failure rate but much lower topic entropy (71.9%) and embedding diversity (0.71) β€” the static pool limits discoverable failures to what already exists. SS-Gen-RPF achieves 40.3% failure rate (slightly lower) but 95.2% topic entropy and 0.95 embedding diversity β€” generation enables discovery of failures outside the pool. On GSM8K, the gap is similar: SS-RPF at 30.0% failure rate, 75.8% topic entropy vs. SS-Gen-RPF at 27.7% failure rate, 96.2% topic entropy.

TSS-RPF vs. SS-Gen-RPF: the diversity-quality tradeoff. On StrategyQA, TSS-RPF achieves higher Overall Diversity (0.74) than SS-Gen-RPF (0.71) but lower Failure Rate (26.0% vs. 40.3%). On GSM8K, TSS-RPF achieves both higher Overall Diversity (0.73 vs. 0.73 β€” tied) and higher Failure Rate (31.3% vs. 27.7%). The tradeoff is task-dependent: TSS excels when failure modes are concentrated in specific topics (GSM8K) but may sacrifice some exploitation efficiency when failures are diffusely distributed (StrategyQA).

Tuned vs. raw prompt features in failure discovery. Comparing RPF and TPF variants within the same method (e.g., TSS-RPF vs. TSS-TPF) in Table 3: differences are generally small, with no consistent winner. On StrategyQA, TSS-RPF achieves 0.74 Overall Diversity vs. TSS-TPF at 0.73; failure rates are 26.0% vs. 33.0%. On GSM8K, both achieve 0.73 Overall Diversity, with failure rates 31.3% vs. 20.7%. The tuned features sometimes help (lower SFF) and sometimes hurt (lower failure rate), suggesting that the learned encoder's benefit is sensitive to the specific benchmark and target model.

Cross-Target Model Generalization for Failure Discovery

Table 4. Five target models are evaluated with Gemini 3 Pro as the generator. TSS-TPF and SS-Gen-RPF consistently outperform Rand-Gen across all targets. On the hardest target (GPT-5, StrategyQA): TSS-TPF discovers 18.9% failures, SS-Gen-RPF discovers 15.4%, Rand-Gen discovers 5.1%. On the easiest target (Gemma3 27B, StrategyQA): 67.8%, 60.2%, 20.5%. The relative improvement is largest for stronger models (3.7Γ— on GPT-5 vs. 3.3Γ— on Gemma3), consistent with the intuition that random generation is particularly ineffective at finding failures for capable models β€” their failure modes are subtle and require targeted probing. On GSM8K, the pattern holds: TSS-TPF discovers 16.8% on GPT-5 vs. Rand-Gen's 3.2% (5.25Γ—), and 57.4% on Gemma3 vs. 15.4% (3.7Γ—).

Query generator quality (Table 5). Using Gemini 2.5 Flash as the target, six different generators are compared. GPT 5 as generator with TSS-RPF discovers 41.5% failures on StrategyQA; Gemma3 (27b) as generator with the same method discovers 26.3%. The monotonic relationship β€” stronger generator β†’ higher failure discovery rate β€” holds across all methods and both benchmarks. The paper's recommendation (Section 3.3) follows: since ProEval reduces the number of queries to the (expensive) target model, one can allocate a larger fraction of the budget to a stronger (more expensive) generator.

Ablation Studies and Robustness Checks

Source data selection via GMM clustering vs. alternatives (Appendix E, Table 26). Across 78 experiments (6 benchmarks Γ— 13–15 models), seven selection methods are compared by median MAE at iteration 20. GMM + minβ‰₯3 abstention achieves the best median MAE of 0.0109 (with 17 abstentions, 22% of cases). GMM without abstention achieves 0.0125 (no abstentions). The simple Leave-One-Out prior (using all available models) achieves 0.0131, making it a competitive safe default. Spearman correlation with threshold 0.7 achieves 0.0120 but with a 54% abstention rate β€” lowest variance (std 0.0175) but half the cases produce no prediction. Random selection of source models achieves median MAE of 0.0316, confirming that source selection matters substantially. The takeaway: GMM with abstention provides the best balance of accuracy and reliability; when abstention is not acceptable, using all models (LOO Prior) is safer than aggressive filtering.

Negative transfer without source selection (Figure 7, Table 26). The pairwise heatmap in Figure 7 quantifies negative transfer directly: using G0 (Gemma3-12B) as the sole source for G2 (Gemini 2.5 Flash) yields MAE of 0.032; using G1 yields 0.009. The optimal pair varies by target: G2 benefits most from pairing with C1 (MAE 0.004), while O2 (GPT-4o) benefits from G4 (MAE 0.008). The existence of high-error cells (MAE > 0.10, up to 0.684 in Figure 11b) validates that blind source selection can be catastrophic. The GMM approach successfully identifies the optimal pairs in most cases (Figure 10 clusters group similar models together).

Embedding model quality (Section 3.2, Table 22). An ablation on StrategyQA with Gemini 2.5 Flash using BQ-RPF at 50 samples compares four embedding models. The largest/strongest models achieve the lowest MAE: gemini_embedding_001 (3072-dim): 0.0425; text_embedding_3_large (3072-dim): 0.0654; text_embedding_3_small (1536-dim): 0.0834; all_minilm_l6_v2 (384-dim): 0.0900. The near-monotonic relationship suggests that embedding quality is a bottleneck for prompt-feature-based transfer β€” stronger embeddings capture finer-grained semantic distinctions that correlate with model performance.

Reasoning trace integration (Table 2). Comparing Q-Only, Concat, and Fusion for BQ-RPF on Gemini 2.5 Flash across 7 benchmarks. No single method dominates. Fusion achieves the best MAE on 4 benchmarks: GSM8K (0.009 vs. Q-Only 0.018 vs. Concat 0.022), SVAMP (0.063 vs. 0.083 vs. 0.213), ToxicChat (0.001 vs. 0.125 vs. 0.064), and Jigsaw (0.029 vs. 0.048 vs. 0.081). Concat achieves best on StrategyQA (0.099 vs. Q-Only 0.128 vs. Fusion 0.071 β€” actually Fusion is better at 0.071). Q-Only is best on MMLU (0.080 vs. Concat 0.272 vs. Fusion 0.104). The inconsistency suggests that reasoning traces contain useful signal but their integration is task-specific and the optimal method depends on whether the model's errors are detectable from surface-level reasoning patterns. No pre-training was done for this ablation, so these results represent a lower bound on what is possible.

BQ Rounded vs. standard BQ (Table 1, all target model tables). The BQ Rounded variant (rounding posterior means to $\{0,1\}$ before summation) produces mixed results. In Table 1 for G2, BQ-SF Rounded achieves identical or better MAE than BQ-SF on Jigsaw (0.001 vs. 0.016), ToxicChat (0.001 vs. 0.002), and DICES (0.064 vs. 0.067), but underperforms on MMLU (0.083 vs. 0.006) and DIVE (0.065 vs. 0.002). Across all 16 target model tables, there is no systematic advantage to rounding. The footnote in Section 3.2 explains: "this discards the nuanced uncertainty information provided by the GP posterior. As a result, BQ Rounded can introduce noise and exhibits less stability than the standard BQ variant, especially when the posterior mean is near 0.5." The empirical inconsistency supports this interpretation.

GPC (Gaussian Process Classifier) comparison (Section 3.2). A GP classifier with Laplace approximation (probit likelihood) was tested as an alternative to the Gaussian observation model. At a 1% budget: on StrategyQA, GPC achieved MAE of 0.1653 (vs. BQ-SF at 0.011); on SVAMP, 0.0516 (vs. BQ-SF at 0.005). The paper attributes this underperformance to approximation errors in the Laplace approximation and the lack of transfer learning optimization for the GPC: "More investigations on transfer learning for GPC is required to make concrete claims about how to best use GPC for our tasks." This negative result justifies the Gaussian likelihood as a practical choice despite its statistical mismatch with binary data.

Modality transfer (Section 3.2). A preliminary cross-modality experiment transfers from DICES (text safety) to DIVE (text-to-image safety) with 15 samples. Without transfer: MAE 0.111. With DICES→DIVE transfer: MAE 0.055. This 2× reduction demonstrates that semantic embeddings can capture cross-modal performance correlations — text and image prompts with similar safety-sensitive content share difficulty patterns.

Tuned vs. Raw Prompt Features: TPF does not consistently outperform RPF (Table 1). In the Default scenario, BQ-TPF outperforms BQ-RPF on 5 of 9 benchmarks (GSM8K: 0.044 vs. 0.018 β€” wait, RPF is better. Let me recount: DICES: TPF 0.253 vs. RPF 0.178; DIVE: TPF 0.034 vs. RPF 0.117; GQA: TPF 0.079 vs. RPF 0.177; GSM8K: TPF 0.044 vs. RPF 0.018; Jigsaw: TPF 0.033 vs. RPF 0.048; MMLU: TPF 0.073 vs. RPF 0.080; StrategyQA: TPF 0.022 vs. RPF 0.128; SVAMP: TPF 0.017 vs. RPF 0.083; ToxicChat: TPF 0.099 vs. RPF 0.125). TPF wins on 6, RPF on 3. However, TPF's advantages are concentrated on benchmarks where semantic similarity is more informative (StrategyQA, GSVAMP) and its failures are on benchmarks with nuanced, hard-to-learn performance patterns (DICES, GSM8K). This suggests that tuning helps when the embedding space contains latent structure relevant to model performance but can overfit to spurious correlations when the signal is weak.

Cross-target generalization for failure discovery with different generators (Table 5). For a fixed target (Gemini 2.5 Flash), failure discovery rates increase monotonically with generator capability. GPT 5 as generator with TSS-RPF achieves 41.5% on StrategyQA; Gemma3 (27b) achieves 26.3%. The gap is consistent across all methods (TSS-TPF, TSS-RPF, SS-Gen-RPF, Rand-Gen), confirming that generator quality is a first-order factor in failure discovery. The practical implication is that investing in a stronger generator yields proportionally better failure discovery, and ProEval's sample efficiency makes this investment feasible within a fixed budget.

Critical Assessment

Does ProEval genuinely achieve 8–65Γ— sample efficiency for performance estimation?

The evidence for this claim is strong at the extremes but conditional on problem setting. Figure 6 shows BQ-SF reaching 1% MAE after 1–2 evaluations on Jigsaw, GSM8K, MMLU, and GQA, while Random Sampling requires 59–105 evaluations β€” ratios of 30–105Γ—. Table 1 confirms BQ-SF MAE values at a 1% budget that Random would need 50–100Γ— more samples to match on many benchmarks.

However, several qualifications are necessary:

  1. The claim depends entirely on the score-feature (SF) variant. Prompt-feature variants (RPF, TPF) achieve much smaller gains (often 3–10Γ—, sometimes worse than random on specific benchmarks like DICES and StrategyQA in the NB setting). The paper's headline should be interpreted as "BQ-SF achieves 8–65Γ— efficiency," not "ProEval in all configurations achieves 8–65Γ—." The SF variant requires the strong Assumption 2 (identical inputs across historical and target models), which limits applicability.

  2. The 1% MAE threshold is a specific operational point. Figure 5 shows that after ~5–10 evaluations, BQ-RPF and BQ-TPF are still at MAE of 0.02–0.08, while BQ-SF is below 0.01 from the start. The efficiency ratio shrinks at larger budgets β€” if the requirement were 0.1% MAE, the gap might narrow because BQ-SF saturates quickly while other methods continue improving. The paper does not report MAE at other thresholds to characterize this.

  3. The cost of constructing the GP prior is excluded from the budget. For score features on the Default scenario, the prior requires evaluating ~15 historical models on the full benchmark β€” thousands of model queries that are not counted in the target-model budget. In practice, this cost is amortized over many target-model evaluations (if the prior is reused), but for a one-off evaluation, the total cost would be dominated by prior construction. The paper acknowledges this implicitly (historical data is assumed available), but the "8–65Γ—" figure should be understood as the marginal cost per new target model evaluation, not the total cost from scratch.

  4. The comparison baselines are weak on some benchmarks. The active testing baselines (RF+IS, LR+LURE, etc.) train surrogates on only 11 historical models and do not leverage the full covariance structure. The paper does not compare against more sophisticated active testing methods that might close the gap, such as those using neural surrogates or pre-trained embeddings. The "8–65Γ—" is against these specific baselines, not against the best possible prior-free methods.

  5. Table 1 shows BQ-SF is not universally dominant. On DICES, BQ-SF achieves 0.067 MAE β€” better than Random (0.100) but not dramatically so, and BQ-TPF actually underperforms Random (0.253 vs. 0.100). On StrategyQA for G0 (Table 7), BQ-SF achieves 0.058 vs. Random 0.088 β€” a modest 1.5Γ— improvement, not 8–65Γ—. The efficiency gains are large for well-behaved models similar to the historical sources but can be marginal for outlier models.

Does ProEval genuinely discover 2–5Γ— more diverse failure cases?

The claim is supported for specific comparisons but the "more diverse" aspect is nuanced. Table 4 shows TSS-TPF discovering 3.7–5.3Γ— more failures than Rand-Gen on GPT-5, and 2.5–5Γ— on other strong models. Figure 8 shows that active methods maintain linear growth in cumulative failures while baselines stagnate.

However:

  1. The 2–5Γ— claim compares against Rand-Gen only. Table 3 shows that SS-Gen-RPF achieves higher failure rates than TSS variants on StrategyQA (40.3% vs. 26.0% for TSS-RPF), though with lower diversity. If the metric were purely "failure rate," SS-Gen would be superior. The diversity metrics are what justify TSS, but the overall diversity scores in Table 3 are modest improvements: TSS-RPF achieves 0.74 vs. SS-Gen-RPF 0.71 on StrategyQA β€” a 4% relative improvement. The claim of "more diverse" failures is accurate but the magnitude of the diversity advantage is small compared to the failure rate advantage over baselines.

  2. Diversity is measured using the same embedding model used for acquisition (text-embedding-3-large). If this embedding model has blind spots (e.g., it fails to distinguish certain types of reasoning patterns), the diversity metric would overestimate true semantic diversity. An independent diversity assessment (e.g., human evaluation of failure type coverage) would strengthen the claim.

  3. The failure discovery experiments use a single generator (Gemini 3 Pro) and two domains (StrategyQA-like, GSM8K-like). The generality of TSS across different reasoning types (spatial reasoning, causal reasoning, code generation) and different modalities is not tested. The paper's claims are restricted to "implicit reasoning and mathematical reasoning," which is appropriate but limits the strength of the diversity claim.

Does the GMM-based source selection genuinely prevent negative transfer?

The evidence is strong and well-ablated. Appendix E Table 26 with 78 experiments shows that GMM + abstention achieves the best median MAE (0.0109), and Figure 7's pairwise heatmap visualizes specific cases where dissimilar source models cause 100Γ— higher error. The abstention mechanism is appropriately conservative (22% abstention rate) and the paper reports what happens without it (median MAE degrades from 0.0109 to 0.0125).

However:

  1. The GMM clustering uses a reference benchmark that is always "all benchmarks except the target." This means the clustering quality depends on the availability of diverse reference benchmarks. In a deployment with only one or two benchmarks, the clustering signal may be too weak to distinguish models, and the abstention rate could become unacceptably high. The paper does not characterize how clustering quality scales with the number of reference benchmarks.

  2. The comparison in Table 26 uses a specific operational point (iteration 20 on BQ-SF). The relative ranking of methods might change at different budgets or with different feature types (RPF vs. TPF vs. SF). The paper does not report selection method ablations across these axes.

What is not tested that would strengthen the paper?

  1. No combination of performance estimation and failure discovery in a shared-budget experiment. The paper treats these as separate experimental sections. A unified experiment that allocates a shared budget of, say, 100 queries across both objectives (some for BQ variance reduction, some for superlevel set sampling) and measures both MAE and cumulative failures would validate the claimed synergy of the unified framework.

  2. No sensitivity analysis on the number of historical models $N$. Theorem 3 provides an explicit dependence on $N$, and the paper notes that $N > M$ ensures the bound is non-vacuous, but there is no experiment varying $N$ (e.g., using 3, 5, 10, or 15 historical models) and measuring MAE. This would validate the theoretical prediction and provide practical guidance on how much historical data is "enough."

  3. No latency or wall-clock time measurements. The paper measures efficiency in "number of evaluations" but does not report actual runtime. For sequential revision strategies or TSS with LLM generation, the wall-clock time per iteration could dominate. A latency-aware comparison would reveal whether methods that appear efficient in sample count are practical in deployment.

  4. No comparison against the latest efficient evaluation methods that use LLM-based surrogates or neural processes. The baselines are from Kossen et al. (2021) using logistic regression and random forests. More recent work (Li et al., 2025, which the paper cites) uses neural processes for transfer learning in evaluation. A direct comparison would contextualize ProEval's gains against the current state of the art.

  5. No experiment measuring how estimation quality degrades when Assumption 1 is systematically violated β€” e.g., by including models from a different modality (text vs. code), a different language (English vs. multilingual), or a different training paradigm (base vs. instruction-tuned). The GMM clustering is meant to handle this, but the paper does not create controlled violations to measure the robustness of the clustering approach.

  6. The failure discovery experiments do not measure precision β€” what fraction of generated inputs labeled as "failures" are genuinely informative failures versus ill-posed questions or near-duplicates? The human verification study on 80 questions addresses validity partially, but a systematic precision analysis (especially comparing TSS and SS-Gen) would strengthen the claim that the additional diversity from TSS comes without sacrificing failure quality.

Conditional validity of the central claims

The paper's claims should be understood as holding under the following conditions, which are mostly consistent with the experiments but deserve explicit statement:

  • The 8–65Γ— sample efficiency for performance estimation holds when (a) using score features (Section 2.2.1) with historical evaluation data on the exact same inputs, (b) the target model is within the distribution of historical models (GMM cluster size β‰₯ 3), (c) the budget is small (1% of benchmark size), and (d) the metric is MAE. The claim weakens for prompt-feature-based transfer, for outlier target models, for larger budgets, and for benchmarks where semantic similarity alone is a weak predictor of model behavior (e.g., DICES safety ratings).

  • The 2–5Γ— more diverse failure discovery holds when (a) comparing TSS against unguided random generation (Rand-Gen), (b) using a strong LLM generator (GPT-5 or Gemini 3 Pro), (c) the target model has non-trivial failure rates (for near-perfect models, even TSS will struggle), and (d) diversity is measured in the same embedding space used for acquisition. The claim of diversity advantage over other guided methods (SS-Gen) is more modest and task-dependent.

  • The unbiasedness guarantee of Theorem 3 holds under Assumptions 1 and 2, which are idealized. The GMM clustering and abstention mechanism provide a practical approximation, but the theorem provides no guarantees when these practical mechanisms are used β€” it is a statement about the ideal case that motivates the design.

6. Limitations and Trade-offs

6.1 The Strongest Performance Estimation Results Require Score Features That Depend on a Strong and Often Unavailable Assumption

The assumption: BQ-SF, which achieves the dramatic 8–65Γ— efficiency gains the paper highlights, requires Assumption 2 (Section 2.2.1): all historical datasets must evaluate "the identical set of inputs $\{x_j\}_{j=1}^M$" as the target benchmark. The empirical covariance matrix $\hat{\Sigma}$ and the derived score features $\phi(x_j)$ are only defined when every historical model has been evaluated on every question in the target benchmark. The paper is explicit about this constraint when introducing score features, but the implications for practical deployment receive less emphasis than the headline efficiency numbers.

The consequence: In many realistic evaluation scenarios, this assumption fails. When a new benchmark is created (the New Bench scenario), no model has been evaluated on it yet β€” score features are unavailable by construction. When evaluating a model on a custom internal benchmark, historical data may exist for only a subset of questions. When benchmark maintainers update or expand a dataset, the new questions have no score features. In all of these cases, the practitioner must fall back on prompt-feature-based transfer (RPF or TPF), which the paper shows is substantially less effective: in Table 1, BQ-RPF achieves mean MAE of approximately 0.087 across benchmarks compared to BQ-SF's 0.015, and the efficiency advantage over random sampling shrinks to roughly 1–3Γ— rather than 8–65Γ—. On several benchmarks (DICES, StrategyQA in the NB setting), BQ-RPF actually underperforms random sampling. The paper acknowledges this: in the NB setting, "performance drops relative to the Default setting" (Section 3.2), but this understates the magnitude β€” the methodology itself changes, not just the performance.

What evidence exists in the paper: Table 1 provides direct evidence. Comparing the "Active Selection + BQ" rows: BQ-SF achieves MAE ≀ 0.02 on 6 of 9 benchmarks for Gemini 2.5 Flash; BQ-RPF achieves MAE ≀ 0.05 on only 4 of 9, with values of 0.177 on DICES, 0.128 on StrategyQA, and 0.125 on ToxicChat. The "Special Scenarios" block explicitly labels the NB and NM results, making the gap transparent to careful readers. The paper also shows in Table 1 that BQ-TPF (tuned prompt features) sometimes outperforms BQ-RPF (e.g., StrategyQA: 0.022 vs. 0.128) and sometimes underperforms (DICES: 0.253 vs. 0.178), indicating that prompt-feature transfer is not just weaker but also fragile β€” small differences in benchmark characteristics or tuning quality produce large variance in estimation accuracy.

Mitigation status: The paper offers no solution for making prompt-feature transfer as effective as score-feature transfer. The learned embedding approach (TPF, Section 2.2.2) is an attempt to close the gap, but the experimental results show it is inconsistent. The paper's footnote discussion of stronger embedding models (Section 3.2, Table 22) suggests that embedding quality is a bottleneck β€” better embeddings reduce MAE β€” but even the strongest tested embedding model (gemini_embedding_001, 3072-dimensional) still yields substantially higher error than score features. The fundamental issue is structural: semantic similarity between prompts is a much weaker signal about model failure correlation than direct empirical observation of which models fail on which prompts. The paper does not frame this as a limitation, but a practitioner evaluating a new benchmark should understand that the 8–65Γ— efficiency claim does not apply to them; they should expect roughly 3–10Γ— improvement at best, and potentially no improvement over random sampling depending on the benchmark and embedding quality.

6.2 The Difficulty Estimation Overhead That Enables Proactive Selection Is Not Accounted for in Efficiency Measurements

The assumption: The paper's active acquisition functions (both for performance estimation in Section 2.3 and for failure discovery in Section 2.4) assume the GP prior has already been constructed and the initial posterior has been computed. For score features, this requires evaluating all $N$ historical models on all $M$ test inputs β€” a computation that, depending on $N$ and $M$, can dwarf the target-model evaluation budget. For prompt features, the encoder $\psi_\theta$ must be pre-trained by maximizing the marginal likelihood over historical datasets (Section 2.2.2), which requires historical model evaluations on the training benchmarks. The paper does not include these costs in any reported budget or efficiency measurement.

The consequence: The headline "8–65Γ— fewer samples" compares only the number of target-model evaluations that ProEval requires versus baselines, ignoring the evaluations of historical models that make the GP prior possible. For a one-off evaluation of a single model on a new benchmark, the total cost would be: (historical evaluations for prior construction) + (target-model evaluations for BQ). If historical evaluations require 15 models Γ— 1,500 questions = 22,500 evaluations, and target-model evaluations require 2–15, the total is dominated by prior construction. The efficiency claim is best interpreted as marginal cost per additional target model on an already-characterized benchmark β€” a valid interpretation for organizations that repeatedly evaluate different models on the same benchmark, but not for one-off evaluations.

For the failure discovery experiments (Section 3.3), the situation is similar: the SS acquisition function requires an already-computed GP posterior, which requires historical data. The SS-Gen and TSS methods additionally require an LLM generator (Gemini 3 Pro in the experiments) whose cost is shared across methods but still represents substantial computational overhead β€” each synthesized question incurs one LLM generation call plus one target-model evaluation. The paper is transparent about generator costs (Table 5, discussion of choosing stronger generators within budget) but does not factor generator cost into sample efficiency claims.

What evidence exists in the paper: The paper never reports the cost of prior construction. Section 2.2.1 states that score features are derived from "$N$ historical models" evaluated "on the exact same set of inputs," and the experiments use 13–15 models (Section 3.1.1) across benchmarks of 700–2,000 questions (Table 6). The total evaluation count for prior construction is roughly 15 Γ— 1,500 = 22,500 per benchmark β€” approximately 100–1,000Γ— larger than the 1% labeling budgets (15–20 evaluations) used in the performance estimation experiments. The paper's discussion of the 2048-sample difficulty estimation cost in Section 2.2.1 (regarding the difficulty estimation approach from the related reference, not ProEval itself) shows the authors are aware of pre-evaluation overhead as a concern, but they do not apply this analysis to their own framework. Section 3.2 briefly notes, regarding the difficulty estimation: "our experiments do not account for this cost largely for simplicity," but this statement refers to the referenced prior work, not to ProEval's own prior construction.

Mitigation status: The paper implicitly addresses this through the Default vs. New Model vs. New Bench scenario structure. In the Default scenario (historical models already evaluated on the target benchmark), the prior construction cost has already been paid β€” the practitioner is assumed to have access to pre-existing evaluation results. This is realistic for well-established benchmarks (GSM8K, MMLU) where leaderboard data is publicly available or where an organization maintains an internal evaluation database. However, the paper does not quantify how many historical models are "enough" or how prior quality degrades as $N$ decreases. Theorem 3's bound depends on $N$ in the denominator, suggesting that fewer historical models means looser guarantees, but there is no empirical ablation varying $N$ (e.g., using 3, 5, or 10 historical models). A practitioner with access to only 2–3 historical model evaluations cannot determine from the paper whether the efficiency gains still hold at that smaller scale.

6.3 The GP Prior Construction Methods Make Strong Distributional Assumptions with No Diagnostic for Violation at Test Time

The assumption: Assumption 1 states that all historical models' score functions $f_i$ are drawn from a shared GP prior $\mathcal{GP}(\mu, k)$. For score features, this implies that the per-question score vectors $\mathbf{y}_i$ are i.i.d. samples from a multivariate Gaussian $\mathcal{N}(\mathbf{u}, \Sigma)$ (Section 2.2.1). For prompt features, it implies that the embeddings plus kernel structure can capture all systematic variation in model performance across inputs. The paper acknowledges that this assumption "fails when evaluating an out-of-distribution target model" (Section 2.2.3) and proposes GMM clustering as a mitigation.

The consequence: There is no mechanism to detect when Assumption 1 is violated during the active evaluation loop. The GMM clustering operates on reference benchmark data (not the target benchmark) and selects source models before any target-model evaluations begin. If the target model violates the Gaussianity assumption β€” for example, if it has a qualitatively different failure pattern that no source model captures (e.g., a model with a specific safety intervention that eliminates certain failure modes entirely) β€” the GP posterior will initially appear confident (based on the prior) but will produce systematically biased estimates. As target-model observations arrive, the posterior will eventually correct, but the initial estimates (which are what the active acquisition function relies on) will be misleading. In the worst case, this could cause the active selection to target uninformative regions of the input space, wasting the evaluation budget.

For prompt features, the issue is more subtle: the encoder $\psi_\theta$ is trained to maximize the likelihood of historical data. If the target benchmark has a fundamentally different relationship between semantic similarity and model performance (e.g., the task requires knowledge that is uncorrelated with the semantic categories captured by the embedding), the GP predictions will be poorly calibrated, and the GP posterior uncertainty estimates β€” which drive both acquisition functions β€” will not reflect true epistemic uncertainty. The paper's finding that BQ-RPF underperforms random sampling on DICES and StrategyQA in the NB setting (Table 1) is evidence of this failure mode.

What evidence exists in the paper: The GMM clustering (Figure 10, Appendix E) and the pairwise source-target heatmaps (Figure 7) are the primary diagnostic tools. Figure 10 shows that some models (G0, O1, Q1) consistently fall outside the main cluster across benchmarks, and Figure 7 shows that using these models as sources degrades estimation. However, these are pre-hoc diagnostics based on historical data. There is no post-hoc diagnostic reported β€” no analysis of whether the GP posterior becomes well-calibrated as target-model observations accumulate, no comparison of posterior predictive distributions against actual observations, and no metric for detecting when the GP prior is misleading after active learning has begun.

Appendix E compares several selection methods and includes Mardia's multivariate normality test as one candidate. The paper notes that "the combined normality assumption rarely holds in practice β€” this method proved too strict for practical abstention" (Appendix E.2.7). This is a revealing admission: the formal assumption underlying the GP prior is empirically violated in most cases, but the practical performance of the method is still good. The paper does not reconcile this tension. A practitioner might reasonably ask: if normality is violated, what justifies the Gaussian process model? The paper's implicit answer is "it works empirically," but this is an engineering claim, not a statistical one.

Mitigation status: The GMM clustering with abstention (Section 2.2.3, Appendix E) is a partial mitigation at the prior construction stage but provides no runtime diagnostics. The paper does not propose any mechanism for detecting prior mismatch during active evaluation β€” for example, monitoring the empirical calibration of posterior predictive intervals, computing posterior predictive checks, or adaptively weighting the prior against a non-informative baseline as evidence accumulates. The abstention rule is a binary gate (predict or don't predict) based on pre-hoc data; there is no continuous measure of prior reliability that could be monitored and acted upon during the active loop.

The theoretical guarantee (Theorem 3) is also not a diagnostic: it provides a bound assuming Assumption 1 holds, but does not provide a test for whether Assumption 1 holds. The bound becomes vacuous if Assumption 1 is violated, but the theorem does not tell the practitioner how to detect this.

6.4 The Gaussian Observation Model for Binary Data Is a Known Mismatch with Undetermined Practical Consequences

The assumption: The paper models binary failure observations (correct/incorrect, safe/unsafe) as $y_\tau \sim \mathcal{N}(f(x_\tau), \sigma^2)$ with a Gaussian likelihood (Section 2.1), rather than using a Bernoulli or probit likelihood appropriate for binary data. The paper explicitly acknowledges this as "a deliberate trade-off to ensure computational tractability and analytical updates in BQ with pre-trained GPs" (Section 3.2).

The consequence: The Gaussian likelihood produces posterior means $\mu_t(x)$ that are not constrained to $[0, 1]$ and can theoretically fall outside this range, though this is unlikely in practice given the prior. More importantly, the posterior variance $k_t(x, x)$ does not properly reflect the heteroskedasticity of binary data: for a Bernoulli variable with mean $p$, the information content of an observation is $p(1-p)$, which is maximized at $p=0.5$ and goes to zero at $p=0$ or $p=1$. The Gaussian model treats all observations as equally informative regardless of $p$. This means the uncertainty estimates that drive both the performance estimation acquisition function (Equation 10, which maximizes variance reduction) and the failure discovery acquisition function (Equation 12, which uses $k_t(x, x)$ as an exploration bonus) may be systematically miscalibrated. The practical consequence could be suboptimal active selection: the GP might over-value observations in regions where the model almost always succeeds or almost always fails (where a Bernoulli model would recognize that additional observations provide little information), or under-value observations near the decision boundary (where a Bernoulli model would recognize high information content).

The BQ Rounded variant tries to partially address this by forcing predictions to $\{0, 1\}$, but it introduces its own issues: "BQ Rounded can introduce noise and exhibits less stability than the standard BQ variant, especially when the posterior mean is near 0.5" (Section 3.2 footnote). The rounding discards uncertainty information that is critical for the variance-reduction acquisition function.

What evidence exists in the paper: The paper briefly reports an experiment with a Gaussian Process Classifier (GPC) using Laplace approximation in Section 3.2. The GPC achieved MAE of 0.1653 on StrategyQA and 0.0516 on SVAMP at a 1% budget, compared to BQ-SF's 0.011 and 0.005 respectively. The paper attributes this underperformance to "approximation errors in the Laplace approximation" and notes that "more investigations on transfer learning for GPC is required." This experiment is important because it shows that simply switching to a proper Bernoulli likelihood does not improve results β€” likely because the Laplace approximation introduces errors that compound in the BQ integral computation and because the GPC transfer learning pipeline is less mature. However, the fact that a properly specified model with current approximations underperforms does not mean the Gaussian model is theoretically sound β€” it means the engineering trade-off currently favors the Gaussian approximation, which is a different claim.

The BQ Rounded results across Tables 7–21 show no consistent advantage over standard BQ, as noted in Section 5. In Table 1 for G2, BQ-SF Rounded matches or beats BQ-SF on Jigsaw (0.001 vs. 0.016) and ToxicChat (0.001 vs. 0.002) but significantly underperforms on MMLU (0.083 vs. 0.006) and DIVE (0.065 vs. 0.002). This inconsistency suggests that the rounding heuristic is not a reliable fix for the model mismatch β€” it helps when posterior uncertainty is low (predictions are confidently near 0 or 1) and hurts when posterior uncertainty is high (predictions are near 0.5 and rounding discards meaningful signal).

Mitigation status: The paper is transparent about this limitation but does not resolve it. The authors position the Gaussian model as a practical choice and flag the GPC investigation as future work. A practitioner should understand that the uncertainty estimates from ProEval may be miscalibrated in ways that affect active selection quality, particularly in regimes where the failure probability is near 0.5 (maximum Bernoulli variance, where the Gaussian model and Bernoulli model diverge most). The paper's strong empirical results suggest that this miscalibration is not catastrophic in practice β€” BQ-SF achieves excellent estimation accuracy despite the model mismatch β€” but there is no formal justification for why this should be the case, and the failure modes are not characterized.

6.5 All Failure Discovery Experiments Use a Single Generator Model and Two Synthesized Domains with No Evaluation of Generalization to Other Reasoning Types or Modalities

The assumption: The failure discovery experiments (Section 3.3, Figures 8, Tables 3–5) use Gemini 3 Pro as the primary query generator and evaluate on two synthesized domains: StrategyQA-like implicit reasoning questions and GSM8K-like math word problems. The paper implicitly assumes that the hierarchical synthesis approach (TSS) and the superlevel set acquisition function (SS) will generalize to other reasoning types, modalities, and evaluation contexts.

The consequence: It is unknown whether TSS's topic-aware transposition of failure patterns works for domains where topics are less semantically separable (e.g., code generation where "topics" might be algorithmic paradigms rather than surface-level semantic categories), where failure modes are not easily transposable across topics (e.g., failures that depend on specific numerical properties in math vs. failures that depend on factual knowledge in open-domain QA), or where the input modality is not text (e.g., image-based safety evaluation). The BERTopic-based topic clustering used in TSS (Section 2.4) relies on text embeddings to define topics; for image inputs, a different clustering approach would be needed, and it is unclear whether the transposition instruction ("Ensure the new test case belongs to this topic...") is meaningful for non-text modalities.

The diversity metrics (Embedding Diversity, Topic Entropy) are also domain-specific in ways the paper does not explore. Topic Entropy assumes that topics are roughly equally important β€” that covering 10 topics evenly is better than covering 5. But in safety evaluation, some topics may be more critical than others (e.g., failures involving physical harm are more concerning than failures involving mild impoliteness), and a topic-weighted diversity measure would be more appropriate. The paper's unweighted entropy treats all topics equally, which may not align with practical evaluation priorities.

What evidence exists in the paper: The paper's failure discovery experiments are limited to two domains (implicit reasoning and math) with a single generator. Tables 4 and 5 explore generalization across target models and generators but not across evaluation domains. The paper acknowledges that the results are conditional on the generator: "The LLM generator's quality impacts the effectiveness of the methods SS-Gen and TSS" and "it is important to choose a capable LLM as the generator" (Section 3.3). But it does not acknowledge the restriction to text-based reasoning tasks as a limitation on the claimed generality of the approach.

The qualitative examples in Tables 23–24 illustrate the kinds of failures discovered but do not assess whether these failures cover the full space of relevant vulnerabilities. A practitioner evaluating a model for deployment in a medical QA system would want to know whether TSS can discover failures involving drug interaction reasoning, not just whether it can discover failures in grade-school math. The paper provides no evidence either way.

Mitigation status: The paper does not claim to have tested broad domain generalization and does not discuss this as a limitation. Section 8 (the Conclusion) suggests future work on "different rater costs and costs for generating model responses based on different inputs" but does not mention extending failure discovery to new modalities or reasoning types. The framework is presented as general-purpose, but the empirical validation is narrow. A practitioner should treat the failure discovery results as specific to the StrategyQA and GSM8K paradigms and should expect to invest significant effort in adapting the topic clustering, anchor selection, and LLM prompting to their specific domain.

6.6 The Efficiency Gains for Performance Estimation Are Highly Sensitive to the Quality of the GP Prior, with No Mechanism for Recovering from a Weak Prior Within the Active Loop

The constraint: ProEval's active acquisition functions select inputs to minimize posterior variance (Equation 10) or to balance exploration and exploitation in the failure region (Equation 12), but these objectives are computed under the current GP posterior β€” which is heavily influenced by the prior when $t$ is small. If the prior is poorly specified β€” for example, if the GMM clustering selects a source set that is too small (but above the abstention threshold of 3) or includes models that are only superficially similar β€” the initial acquisition steps may select suboptimal inputs, and the GP may require many observations to correct. However, the acquisition function provides no mechanism for detecting that the prior is weak and adapting the exploration strategy accordingly.

The consequence: In the Default scenario with score features, the prior is strong enough that this is not a practical concern β€” BQ-SF achieves low MAE after 1–2 evaluations regardless of which inputs are selected first. But in the NB scenario (and to a lesser extent in prompt-feature variants), the prior can be weak enough that random selection actually outperforms active selection. Table 1 shows this clearly: for Gemini 2.5 Flash on DICES, BQ-RPF with active selection achieves MAE of 0.178, while BQ-RPF with random selection achieves 0.105 (18% better with random). On GQA, Active BQ-RPF achieves 0.177 vs. Random at 0.091. On MMLU, Active BQ-RPF achieves 0.080 vs. Random at 0.073. The active acquisition function, which is designed to accelerate learning, is slowing it down because the prior's variance structure (which determines which inputs appear informative) is miscalibrated.

This failure mode is not diagnosable from within the framework. The GP posterior will happily report decreasing variance as observations accumulate, but if the variance estimates themselves are wrong (because the kernel is misspecified), the apparent confidence is illusory. The paper reports no experiment where the GP's uncertainty estimates are validated against empirical frequencies β€” for example, checking whether the fraction of observations falling within posterior credible intervals matches the nominal coverage.

What evidence exists in the paper: The evidence is in Table 1, where the comparison of "Active Selection + BQ" versus "Random Selection + BQ" reveals cases of negative value of active selection. The paper notes this pattern in Section 5: "active selection sometimes underperforms random (e.g., BQ-RPF: 0.177 Active vs. 0.091 Random on GQA; BQ-TPF: 0.253 Active vs. 0.078 Random on DICES), suggesting that when the prior is weaker (prompt-based transfer), the observation-independent acquisition function can select suboptimal inputs compared to random chance." However, the paper does not propose a solution β€” for example, adaptively blending the prior with a non-informative baseline, monitoring the empirical calibration of posterior variances, or detecting prior mismatch and reverting to random sampling.

The BQ-TPF variant was an attempt to improve the prior through learned embeddings, but its inconsistent performance (better than RPF on some benchmarks, worse on others) suggests that tuning on historical data does not reliably prevent prior misspecification on new benchmarks. The GMM abstention rule catches the most severe cases (fewer than 3 similar models) but provides no protection against moderate prior weakness β€” a cluster of 4 loosely similar models might pass the abstention check but produce a prior that is actively harmful for active selection.

Mitigation status: The paper does not address this limitation. There is no fallback mechanism, no adaptive weighting scheme, and no diagnostic for detecting when active selection is performing worse than random. The implicit mitigation is to use score features whenever possible (where the prior is almost always strong enough) and to accept that prompt-feature transfer is a gamble β€” sometimes it provides large gains, sometimes it provides none, and sometimes it actively hurts. A practitioner using prompt features in the NB setting should consider running a small validation experiment comparing ProEval's estimates against a random-sampling baseline on a held-out subset before committing the full evaluation budget.

7. Implications and Future Directions

How This Work Changes the Landscape

ProEval introduces a conceptual reframing of evaluation from a passive measurement activity to an active, model-driven inference process. The shift is not merely methodological β€” it redefines what "evaluation" means in the context of generative AI. Before this work, evaluation was fundamentally about sampling: select a set of test inputs (by pruning, by random selection, or by guided sampling), run the model on them, and compute a statistic. The model's role was as the object of study; the evaluation procedure was external to it. ProEval makes evaluation about the model in a deeper sense: the same historical data that records how other models behave becomes a transferable asset that encodes the structure of the evaluation task itself, and the target model's own responses become sequential evidence that updates a probabilistic belief state.

This reframing has several downstream effects on how the field should think about evaluation:

Evaluation data accumulates value over time. The paper demonstrates that historical model evaluations β€” scores from GPT-4o, Gemini 2.5 Flash, Claude 4.5 Opus, and others on standard benchmarks β€” are not just archival records. They are samples from an underlying performance structure (Assumption 1) that can be captured in a GP prior and reused for every subsequent model. The corollary is that the more models an organization evaluates, the cheaper future evaluations become, because each new model enriches the historical dataset that constructs the prior. This inverts the standard intuition that evaluation cost scales linearly with the number of models. The paper does not explicitly make this economic argument, but it follows directly from the score-feature construction: N historical models produce an N-dimensional feature for each test input, and Theorem 3 shows that the estimation bound tightens as N grows. Organizations that maintain internal evaluation databases across model versions are sitting on an asset that ProEval can operationalize.

The separation between performance estimation and failure discovery is artificial and costly. Prior work maintained these as distinct pipelines with different formalisms (importance sampling for estimation, adversarial optimization for discovery), different surrogate models, and different evaluation criteria. ProEval's unified GP framework collapses this separation: both objectives query the same posterior over f(x), and every observation serves both simultaneously. This is not just an aesthetic unification β€” it means that a fixed evaluation budget, if allocated across both objectives using ProEval's acquisition functions, can produce both a better aggregate estimate and a more comprehensive failure map than the same budget split across separate pipelines. The paper does not run this exact shared-budget experiment, but the mathematical structure implies it: the BQ posterior variance and the superlevel set acquisition function both condition on the same D_t. A practitioner who previously ran 100 queries for accuracy estimation and 100 queries for red teaming should, under ProEval, run 200 shared queries and get better results on both.

Verifier-free active selection is a practical enabler for parallel evaluation. The observation-independence of the BQ variance reduction (Section 2.3) β€” that V[S | D_t] - V[S | D_t βˆͺ x] depends only on the location x, not on the observed y β€” has an underappreciated operational consequence. It decouples the planning of an evaluation from its execution. An organization can pre-compute the optimal set of test inputs offline (using only the historical GP prior), distribute them across a GPU cluster, and evaluate all of them in parallel. This stands in contrast to sequential active testing methods (Kossen et al., 2021) where each evaluation must complete before the next is selected, creating a latency bottleneck. The paper mentions batch pre-computation in passing (Section 2.3), but the implication is substantial: ProEval enables embarrassingly parallel evaluation at scale, limited only by the number of available inference servers, not by sequential dependencies in the acquisition loop. For organizations evaluating models on thousands of test inputs, this property alone could reduce evaluation wall-clock time from hours to minutes.

The GMM abstention mechanism introduces a new principle for transfer learning: refuse to predict when source quality is insufficient. This is a methodological contribution that generalizes beyond evaluation. The empirical finding that incorporating dissimilar historical models can produce 100Γ— worse estimates (Figure 7) and that the cases where GMM selects too few models are precisely the failure modes (Appendix E, Table 26) establishes that aggressive source selection with explicit abstention is safer than "use all available data" β€” even though using all data achieves competitive median performance. This principle applies to any transfer learning setting where the similarity between source and target domains is variable and partially observable: build a diagnostic for source relevance, and when the diagnostic indicates insufficient signal, abstain rather than produce a potentially misleading prediction. The 22% abstention rate in the experiments is not a weakness β€” it is the mechanism that prevents the worst-case errors.

The topic-aware synthesis architecture (TSS) demonstrates that diversity and exploitation in generative testing can be structurally decoupled. This is an architectural insight, not an algorithmic one. Prior red teaming methods conflated two orthogonal objectives β€” "generate inputs that cause failure" and "generate inputs that are semantically diverse" β€” into a single optimization loop, typically resolving the tension through ad-hoc diversity penalties (e.g., Self-BLEU in Bayesian Red Teaming). TSS separates them: the GP acquisition function handles exploitation (selecting failure-prone anchors), while a separate bandit over topics handles exploration (selecting underexplored semantic domains), and the LLM generator transposes the failure pattern from the former into the context of the latter. This two-loop architecture is a design pattern that could generalize to any setting where generative testing must balance quality and coverage: curriculum generation for training, scenario generation for simulation-based evaluation, or synthetic data generation for model improvement.

This work establishes the boundary conditions for when transfer-based evaluation works β€” and when it doesn't β€” with unusual precision. The New Bench (NB) scenario results (Table 1) reveal that prompt-feature transfer succeeds on 6 of 9 benchmarks but fails on DICES, StrategyQA, and SVAMP β€” benchmarks where semantic similarity between prompts does not correlate strongly with model failure patterns. The negative transfer heatmaps (Figure 7) quantify exactly how much error increases when source models are dissimilar. The GMM clustering (Figure 10) provides a visual diagnostic for which models are in-distribution and which are outliers. This specificity is more valuable than a blanket claim of "transfer learning helps evaluation." It tells practitioners precisely where to be skeptical: DICES safety ratings, implicit reasoning tasks, and outlier models (G0, O1, Q1) are red flags for prompt-feature transfer.

Follow-Up Research This Work Enables

Adaptive blending of the GP prior with a non-informative baseline based on real-time calibration diagnostics. The paper identifies a failure mode where active selection underperforms random selection when the GP prior is weak (Table 1: BQ-RPF Active 0.177 vs. Random 0.091 on GQA; BQ-TPF Active 0.253 vs. Random 0.078 on DICES). A natural extension is to monitor the empirical calibration of the GP posterior as target-model observations arrive β€” for example, tracking whether the fraction of observations falling within posterior credible intervals matches the nominal coverage β€” and adaptively downweight the prior when miscalibration is detected. A simple implementation would compute a running calibration error over a small holdout set and blend the GP posterior mean with a constant baseline (e.g., the sample mean of observed scores) using a weight proportional to the inverse calibration error. The key evaluation would replicate the NB scenario in Table 1 and measure whether adaptive blending recovers the cases where active selection currently underperforms random.

Shared-budget joint optimization of performance estimation and failure discovery. The paper's unified framework implies that a single evaluation budget should be allocated across both objectives, but all experiments treat them separately. A direct follow-up would allocate a fixed budget of, say, 100 queries on a benchmark like GSM8K or StrategyQA, and compare three allocation strategies: (a) 100 queries purely for BQ variance reduction, (b) 100 queries purely for superlevel set acquisition (SS or TSS), and (c) a mixed strategy that interleaves BQ and SS acquisition (e.g., alternate between objectives or use a composite acquisition function that weights variance reduction and failure region exploration). The metrics would jointly measure MAE on the performance estimate and cumulative failures discovered, with the hypothesis that the mixed strategy achieves better results on both axes than either pure strategy at half the budget (i.e., 100 mixed queries outperform 50 estimation + 50 discovery run separately). The mathematical justification is that BQ acquisition tends to select inputs with high correlation to many test points (reducing aggregate uncertainty), while SS acquisition selects inputs in the failure region with high local uncertainty β€” these may be different regions of the input space, and mixing them provides broader coverage.

Scaling laws for GP prior quality as a function of the number of historical models N. Theorem 3 provides a bound that depends on N in the denominator: |SΜ‚_t - S_t| ≀ a' √(ΞΊ + σ²) where a' includes a 1/(N - t - 2) term. The paper uses a fixed N β‰ˆ 15 historical models but never varies N to validate the theoretical prediction empirically. A systematic study would take a fixed target model (e.g., Gemini 2.5 Flash on GSM8K), vary N from 2 to 15 by subsampling the historical model set, and measure MAE at a fixed budget (e.g., 1% of benchmark size). The expected finding is that MAE decreases monotonically with N but with diminishing returns β€” the first 3–5 models provide most of the gain, and additional models refine the prior marginally. The study should also measure the abstention rate of GMM clustering as N decreases, testing whether the abstention mechanism correctly identifies when N is too small to form a reliable prior. If the abstention rate rises sharply below some threshold N*, that provides practical guidance on the minimum evaluation history needed to deploy ProEval on a new benchmark.

Extending TSS to code generation, multimodal reasoning, and open-ended generation tasks with domain-appropriate topic definitions. The paper validates TSS on two text-based reasoning domains (StrategyQA-like and GSM8K-like) with BERTopic for topic clustering. Three extensions would stress-test the generality of the decoupled diversity architecture. For code generation (HumanEval, MBPP), topics could be defined by algorithmic paradigms (sorting, graph traversal, dynamic programming) rather than surface-level keywords, and the transposition instruction would need to specify that the generated problem should require a different algorithm while preserving the same reasoning trap (e.g., off-by-one errors). For multimodal reasoning (GQA, ScienceQA), topics could be defined by visual categories (scenes with occlusion, counting in cluttered images) and the generator would be a vision-language model that synthesizes new images instantiating the failure pattern in a new visual context. For open-ended generation (summarization, dialogue), where binary failure is less well-defined, the superlevel set threshold Ξ» would need to be a continuous severity score from a reward model, and the evaluation would measure whether TSS discovers outputs with high severity scores across diverse semantic domains. The key hypothesis to test in each extension is whether the decoupling of topic from failure pattern (the core TSS innovation) consistently produces more diverse failures than single-loop generation (SS-Gen) without sacrificing failure severity.

Non-Gaussian Bayesian quadrature with pre-trained priors for binary and count data. The paper's Gaussian observation model for binary data is a known mismatch, and the brief experiment with a GP classifier using Laplace approximation in Section 3.2 showed worse performance than the Gaussian model. This negative result is likely due to approximation errors in the Laplace approximation, not a fundamental limitation of non-Gaussian BQ. A focused methodological study would explore alternative approximate inference methods β€” variational inference with a Gaussian approximating distribution, expectation propagation, or stochastic variational Gaussian processes (SVGP) with a Bernoulli likelihood β€” specifically for the BQ integral estimation task with a pre-trained prior. The evaluation would compare against the Gaussian BQ baseline from the paper on the same benchmarks in Table 1, measuring both MAE and the calibration of posterior credible intervals. The hypothesis is that a properly inferred non-Gaussian posterior would provide better-calibrated uncertainty estimates, which would in turn improve active selection (since the variance-reduction acquisition function would be computed under a more accurate posterior). The engineering challenge is significant β€” closed-form BQ updates are lost, and the acquisition function would become observation-dependent β€” but if successful, it would remove the paper's most prominent statistical compromise.

Online model development monitoring with ProEval as a regression detection tool. The paper's evaluation is performed on static, fully-trained models. A practical extension would deploy ProEval in a continuous monitoring setting: after each training checkpoint during model development, run ProEval with a small fixed budget (e.g., 10–20 queries) against a historical prior constructed from previous checkpoints or baseline models. The goal shifts from estimating absolute performance to detecting regressions β€” significant increases in failure rate on specific input categories. The GP prior would encode the expected performance of the developing model (based on previous checkpoints), and the active acquisition function would prioritize inputs where the posterior suggests a divergence from the prior. This use case is natural for the score-feature approach because all checkpoints share the same test inputs (Assumption 2), and the number of "historical models" grows with each checkpoint, continuously improving the prior. The evaluation would measure false positive/negative rates for regression detection at various budget levels, comparing against random monitoring and against evaluating a fixed subset of inputs.

Practical Applications and Downstream Use Cases

Efficient leaderboard maintenance for benchmark platforms. Platforms like Hugging Face Open LLM Leaderboard or Chatbot Arena evaluate new models on established benchmarks (MMLU, GSM8K, HellaSwag) as they are submitted. Each new model currently receives a full evaluation β€” thousands of test queries β€” regardless of whether its performance is substantially different from existing models. ProEval's score-feature approach (Section 2.2.1) is directly applicable: the existing leaderboard entries constitute the historical dataset D, and each new model is a target whose performance can be estimated with 8–65Γ— fewer queries (Figure 6). For a benchmark like MMLU with 1,534 questions, a full evaluation costs 1,534 queries per model; ProEval could provide an estimate within Β±1% MAE using roughly 15–20 queries for models similar to existing entries (BQ-SF in Table 1), and could detect outlier models (via GMM clustering on reference benchmarks) that require fuller evaluation. The cost savings compound with each new model submission, and the historical prior improves as the leaderboard grows.

Safety evaluation at scale for model release certification. Organizations preparing to release a new model must evaluate it across multiple safety benchmarks (ToxicChat, DICES, Jigsaw, DIVE) to certify compliance with safety policies. The paper's results on safety benchmarks (Table 1: BQ-SF achieves MAE of 0.002 on ToxicChat, 0.016 on Jigsaw, 0.067 on DICES for Gemini 2.5 Flash at a 1% budget) enable a tiered approach: evaluate a small, actively selected subset of safety test cases using human raters (the expensive, reliable signal), use the GP posterior to estimate the aggregate safety violation rate, and simultaneously identify the specific inputs in the superlevel set X_Ξ» where violations are probable. The human rater budget can then be concentrated on the most informative or highest-risk inputs, rather than spread uniformly across a random sample. If a full safety evaluation of 1,500 questions costs 15,000inhumanratertime(at15,000 in human rater time (at 10 per rating), ProEval's 8–65Γ— reduction would bring this to 200–200–2,000 per model per benchmark β€” a meaningful saving when multiplied across dozens of model checkpoints during development.

Regression testing in model fine-tuning pipelines. When fine-tuning a base model (e.g., Gemma-3-27B) for a specific domain (e.g., medical QA), developers need to verify that fine-tuning does not degrade performance on other capabilities (the "catastrophic forgetting" problem). Running a full evaluation suite after every fine-tuning experiment is prohibitively expensive. ProEval's New Model (NM) scenario (Table 1, bottom block) addresses this: the base model and other models serve as the historical dataset, the fine-tuned model is the target, and ProEval estimates performance on each benchmark of interest with minimal queries. A developer could set up a monitoring suite that, after each fine-tuning run, evaluates 10–20 actively selected questions per benchmark (using BQ-SF, which achieves MAE of 0.004 on SVAMP and 0.007 on MMLU for most target models in Tables 7–21), and raises an alert if the estimated error rate on any benchmark exceeds a threshold relative to the base model. This transforms regression testing from a batch process run weekly to a continuous process run per experiment, accelerating iteration speed.

When to Prefer This Method

The paper's experimental design β€” comparing against Active Testing baselines (Kossen et al., 2021), static random sampling, and unguided generation β€” establishes a clear decision boundary for practitioners. ProEval should be preferred when all of the following conditions hold:

  • Historical evaluation data exists on the same inputs (Assumption 2) and the target model is within the distribution of historical models. This is the regime where BQ-SF achieves 8–65Γ— efficiency gains (Figure 6, Table 1). The GMM clustering with abstention (Section 2.2.3) provides a pre-hoc diagnostic: if the target model falls in a cluster with β‰₯3 source models, the prior is reliable; if the cluster is smaller, expect weaker gains and consider abstaining.

  • The budget is small relative to the benchmark size (1–5% of test inputs). The paper's primary results are at a 1% budget (Table 1), where the gap between ProEval and baselines is largest. At larger budgets, the relative advantage narrows as random sampling and other methods catch up.

  • The evaluation can tolerate a pre-computation phase to construct the GP prior from historical data. This pre-computation (evaluating all historical models on the full benchmark) is a one-time cost amortized across many target-model evaluations. For one-off evaluations, the total cost (prior construction + target evaluation) may exceed that of simply evaluating the target on a random subset.

  • The practitioner needs both an aggregate performance estimate and specific failure cases. The unified GP framework serves both objectives with the same collected data, and the dual-objective setup (Section 2.1) makes ProEval the natural choice when stakeholders demand both "how accurate is the model?" and "on what kinds of inputs does it fail?"

ProEval should be avoided or deployed with caution when:

  • No historical data exists for the target inputs (the New Bench scenario), and the benchmark involves tasks where semantic similarity is a weak predictor of model performance (e.g., safety ratings on DICES, implicit reasoning on StrategyQA). In these cases, BQ-RPF can underperform random sampling (Table 1), and there is no diagnostic within the active loop to detect this failure.

  • The target model is a known outlier relative to available historical models (e.g., evaluating a very weak model like Gemma-3-12B using a prior built mostly from strong models). The GMM clustering can detect this pre-hoc, but if abstention is not acceptable, a prior-blind method may be safer than a mismatched prior.

  • Latency is the binding constraint, not total evaluation count. Active selection with prompt features requires sequential GP posterior updates and acquisition function evaluations whose computational cost, while small per step (O(M dΒ² + dΒ³)), adds latency between model evaluations. If the goal is to minimize wall-clock time and sufficient parallel inference capacity exists, pre-computing a batch of BQ-selected inputs (exploiting observation independence, Section 2.3) mitigates this β€” but batch pre-computation is only available for the estimation objective, not for failure discovery where the acquisition function depends on observed y values through the GP posterior mean in the indicator term (Equation 12).