The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality

🎯 Pitch

Even the best models score below 70% on The FACTS Leaderboard, a holistic new factuality benchmark spanning multimodal, parametric, search, and grounding tasks, with performance crashing to 46% on image-based questions. The benchmark reveals that no model family excels across the board—Gemini models chase coverage, GPT models chase precision, and Claude models hedge aggressively—while automated judges simulate the real-world chaos of LLM-as-judge variation by swapping evaluation models.

1. Executive Summary

This paper introduces The FACTS Leaderboard, a comprehensive evaluation suite that aggregates model performance across four distinct benchmarks — FACTS Multimodal (image-based questions requiring visual grounding with parametric knowledge), FACTS Parametric (closed-book factoid questions testing world knowledge recall), FACTS Search (information-seeking queries requiring search API use), and FACTS Grounding v2 (long-form responses grounded in provided documents, with improved judge models) — to produce a single holistic FACTS Score as an average of all four components. The top-performing model, Gemini 3 Pro, achieves a FACTS Score of only 68.8%, with substantial variation across sub-benchmarks (from 46.1% on Multimodal to 83.8% on Search), leaving considerable headroom for future progress. The suite uses both public and private dataset splits to guard against overfitting, employs automated judge models validated against human annotations (achieving macro F1 scores of 72.3 for coverage and 78.2 for contradiction detection on Multimodal), and establishes that models exhibit distinct trade-offs between precision and recall — Gemini models favor higher coverage while GPT models achieve superior no-contradiction rates, and Claude models hedge more aggressively to preserve attempted accuracy — demonstrating that no single model family dominates across all factual dimensions.

2. Context and Motivation

The Core Problem: Factuality Evaluation Is Fragmented Across Isolated Benchmarks

The fundamental gap this paper addresses is not that LLMs generate factually incorrect information — that is widely recognized — but rather that the field lacks a unified framework for measuring factuality across the diverse scenarios where it matters. Before the FACTS Leaderboard, the research community evaluated LLM factuality through a patchwork of narrow, single-capability benchmarks, each assessing one specific dimension in isolation. A model that excelled at grounding responses in provided documents might perform poorly on closed-book factoid questions, and vice versa, but no systematic framework existed to capture this tension or to produce a holistic factuality assessment.

This fragmentation matters because real-world deployments do not encounter factuality challenges in isolation. Consider a financial analyst using an LLM to help with quarterly reporting: the model must ground its analysis in provided financial documents (grounding), recall relevant market context from its parametric knowledge (parametric), potentially search for recent regulatory changes (search), and interpret charts or tables in the documents (multimodal). If the analyst evaluates the model only on a grounding benchmark, they learn nothing about whether the model will hallucinate market statistics or fail to extract information from embedded figures. The FACTS suite argues that a comprehensive understanding of an LLM's factuality requires evaluating performance across this wider spectrum, not just one slice of it.

The paper opens by broadly dividing factuality research into two distinct scenarios (Section 1):

Context-dependent factuality: The model's response must be fully grounded in provided information — a document, an image, a table. Benchmarks in this category (Honovich et al., 2022; Jacovi et al., 2025; Rashkin et al., 2023; Tang et al., 2024) test whether the model can faithfully represent source material without hallucinating.
World knowledge factuality: The model must accurately answer factoid queries using its internal parameters (Chen et al., 2023; Kwiatkowski et al., 2019; Lin et al., 2022) or by leveraging external sources like the web (Mialon et al., 2023; Vu et al., 2023; Wei et al., 2024, 2025; Yang et al., 2024). Benchmarks in this category test whether the model has encoded accurate information during pretraining or can effectively use tools to retrieve it.

The paper's central claim is that these two categories — and their sub-variants — are not independent dimensions that can be optimized separately. A model architecture or training procedure that improves grounding might degrade parametric recall (a capacity-competition tradeoff), or a model that aggressively hedges on parametric queries to avoid errors might provide unhelpfully vague answers in search scenarios. Only by measuring all dimensions simultaneously can we understand these tradeoffs.

Why This Problem Matters

The paper's motivation rests on several converging concerns:

1. The proliferation of isolated benchmarks obscures real progress. As individual benchmarks saturate — models approach ceiling performance on specific tasks — it becomes unclear whether improvements reflect genuine advances in factuality or narrow optimization to particular test distributions. A model achieving 95% on a grounding benchmark but 16% on parametric recall (akin to GPT-5 mini's performance pattern in Table 1: 58.3% Grounding vs. 16.0% Parametric) is not "factually reliable" in any practical sense, yet a researcher evaluating only grounding would conclude otherwise. The FACTS Score's averaging across four diverse tasks is explicitly designed to penalize such imbalance and reward genuinely broad factuality capabilities.

2. Practical deployments require holistic factuality, not single-capability excellence. The paper emphasizes that "practical use cases would typically rely on both these capabilities" (Section 1). This is not just a truism — it reflects the architecture of modern LLM applications, which increasingly chain multiple capabilities. A customer support bot that grounds responses in product documentation but hallucinates pricing details from parametric memory is, from the user's perspective, simply unreliable. A single-dimensional benchmark would not surface this failure mode.

3. Saturation of existing benchmarks threatens to stall progress measurement. The paper explicitly acknowledges this in its conclusion: "As LLMs improve, existing benchmarks become saturated. It is thus important to introduce benchmarks that challenge current models." The highest FACTS Score of 68.8% (Gemini 3 Pro, Table 1) leaves substantial headroom, suggesting the suite will remain discriminative for the near future, unlike individual benchmarks that may approach ceiling performance rapidly.

4. The interaction between factuality dimensions reveals model-level design tradeoffs. The paper's results uncover patterns that would be invisible in single-benchmark evaluations. Gemini models achieve high coverage on Multimodal but lower no-contradiction scores (Table 3: Gemini 2.5 Pro at 67.7% Coverage vs. 58.8% No-Contradiction), while GPT models show the inverse pattern (GPT-5 at 59.9% Coverage vs. 64.7% No-Contradiction). Claude models hedge aggressively on Parametric questions (29.9% hedging rate for Claude 4.1 Opus, Table 6) to preserve attempted accuracy. These are not implementation details — they reflect fundamental architectural and training choices (e.g., RLHF preference optimization favoring cautious behavior) that have different consequences depending on the factuality scenario. A unified benchmark makes these tradeoffs visible.

Where Prior Approaches Fall Short

The paper does not extensively critique individual prior benchmarks — its position is not that existing benchmarks are flawed, but that they are incomplete when considered in isolation. However, the design choices in FACTS imply specific limitations in prior work:

Grounding benchmarks did not adequately handle response quality evasion. Prior grounding evaluations, including FACTS Grounding v1 (Jacovi et al., 2025), measured whether model responses were consistent with provided documents. However, as the paper notes in Section 6.2, "metrics that are focused on evaluating the factuality of the generated text with respect to a context document can be 'hacked' by ignoring user intent." A model could achieve high grounding scores by providing trivially short, vague responses that technically contain no false claims but also provide no useful information — what the paper calls "ineligible" responses. Table 11 illustrates this: a response like "Wind energy is good because it is renewable and clean, but it has some challenges too" is factually grounded but fails to address the user's request to summarize specific advantages and disadvantages. FACTS Grounding v2 addresses this by disqualifying such responses and marking them as inaccurate, closing a loophole that prior grounding benchmarks left open.

Parametric benchmarks were not designed for adversarial hardness against frontier models. Many existing parametric QA benchmarks (Kwiatkowski et al., 2019's Natural Questions, for example) contain questions that frontier models can answer with high accuracy, reducing their discriminative power. The FACTS Parametric construction explicitly targets this: the adversarial sampling step (Section 4.1.2) collects responses from five strong open-weight models and retains only questions that none of them answered correctly. This ensures the benchmark remains challenging even as models improve — the questions are selected to probe the frontier, not the average case. Additionally, the paper grounds questions in user traffic to ensure ecological validity, avoiding the common pitfall of constructing artificially difficult but unrealistic queries.

Search benchmarks did not guarantee consistent tool access or evaluate tool-use efficiency. Prior search benchmarks (Wei et al., 2025; Yang et al., 2024) evaluated models' ability to use search tools, but cross-model comparison was confounded by potential differences in the search APIs or retrieval corpora available to each model. FACTS Search solves this by providing all models with the same Brave Search API and the same tool description (Section 5.2), ensuring that performance differences reflect model capability rather than retrieval infrastructure. Furthermore, Table 8 reveals an additional dimension that prior benchmarks did not surface: search efficiency. Gemini 3 Pro achieves the highest accuracy (83.8%) while conducting fewer average searches (3.39) than any other top model — GPT-5 uses 4.28 searches on average, and Grok 4 uses 4.5. This efficiency-accuracy tradeoff is practically important (fewer searches means lower latency and cost) but would be invisible without tracking API call counts.

Multimodal factuality lacked rigorous, rubric-based automated evaluation. Prior multimodal benchmarks often relied on simple exact-match or classification-based evaluation, which fails to capture the nuances of factual accuracy in open-ended visual QA. FACTS Multimodal introduces a dual-verdict rubric-based system (Section 3.2) that separately evaluates coverage (whether essential facts from the human-authored rubric are present) and contradiction (whether any claims contradict the rubric, common knowledge, or the image itself). This decomposes factuality into its precision and recall components, revealing the tradeoff observed between model families in Table 3. The human validation of these automated judges (Spearman's ρ = 0.64 for coverage, macro F1 = 78.2 for contradiction detection, Table 4) provides credibility that prior automated multimodal evaluations often lacked.

No prior benchmark attempted to aggregate across factuality dimensions. This is the most straightforward gap: each prior work focused on one capability. Honovich et al. (2022) and Tang et al. (2024) built grounding benchmarks; Kwiatkowski et al. (2019) and Lin et al. (2022) built parametric QA benchmarks; Wei et al. (2025) and Yang et al. (2024) built search benchmarks; and multimodal benchmarks existed for visual QA but not specifically for factuality as a unified concept. The FACTS suite's contribution is not that its individual components are entirely novel — FACTS Grounding v2 is explicitly an update to prior work (Jacovi et al., 2025), and the parametric and search designs build on established methodologies — but that combining them into an aggregated score forces the community to optimize for balanced, holistic factuality rather than narrow excellence.

How This Paper Positions Itself

The paper positions the FACTS Leaderboard as an infrastructure contribution rather than a methodological one. It is not proposing a new algorithm or model architecture, but rather a measurement standard that the community can use to track progress and compare approaches on equal footing. Several design choices reflect this positioning:

Public and private splits to guard integrity. The paper explicitly notes that "to mitigate overfitting, only a subset of the prompts will be released publicly, and the remaining prompts will remain private" (Section 2). This is a direct response to the well-documented phenomenon of benchmark contamination, where models inadvertently (or deliberately) train on test data. By maintaining private splits and conducting all evaluation through Kaggle rather than allowing self-reporting, FACTS aims to provide trustworthy rankings that cannot be gamed through data leakage.

Live, actively maintained leaderboard. Unlike static benchmark papers that release a fixed dataset and a one-time evaluation, FACTS is designed as an ongoing service: "The leaderboard will remain open to new model submissions" and "will be actively maintained" (Section 2). This positions it as a long-term measurement infrastructure akin to established leaderboards in other domains (e.g., GLUE/SuperGLUE in NLP, or ImageNet in computer vision), rather than a single-point-in-time comparison.

Multipronged judge design with human validation. The paper does not assume that a single automated judge is sufficient. FACTS Grounding v2 uses two different judge models (Gemini 2.5 Flash and GPT-5) and averages their scores "to reduce the bias of a particular judge model, as models have been shown to be biased towards favorably judging their own outputs" (Section 6.2, citing Wataoka et al., 2024). FACTS Multimodal validates its automated judges against human annotations (Table 4). FACTS Parametric samples three grades from Gemini-2.5-Pro and averages them, with a validation against a mixed-model panel to confirm ranking preservation (Section 4.2). This methodological rigor distinguishes FACTS from benchmarks that rely on unvalidated automated metrics.

Explicit acknowledgment of coverage gaps. The paper is unusually transparent about what the suite does not measure: "Naturally, there are aspects of factuality not covered by FACTS, such as video understanding and fast-changing information. In addition, tool-use introduces new factuality challenges, for example when using knowledge-base calls as a tool" (Section 7). This positions FACTS as a floor rather than a ceiling — a minimum set of factuality dimensions that any comprehensive evaluation should include, with the expectation that additional benchmarks will complement it.

The binding insight across all four components is that factuality is not one thing. It decomposes into source of truth (provided context vs. parametric memory vs. web retrieval), modality (text vs. image), and response format (short-form factoid vs. long-form generation). The FACTS suite's design encodes the hypothesis that these dimensions are sufficiently independent — and their interactions sufficiently informative — that aggregating them produces a more meaningful signal than any single benchmark alone. The variation in rankings across sub-benchmarks (Gemini 3 Pro leads overall but is second to GPT-5 on Multimodal; GPT-5 leads on Search but ranks much lower on Parametric) empirically validates this hypothesis: if all dimensions were redundant, rankings would be uniform across components, which they clearly are not.

3. Technical Approach

3.1 Reader Orientation

The FACTS Leaderboard is a measurement instrument — a suite of four evaluation benchmarks combined with automated judge models — that produces a single aggregated score quantifying how factually accurate a language model is across diverse real-world scenarios. It solves the problem that existing factuality benchmarks each measure only one narrow capability (grounding to documents, answering trivia from memory, using web search, or describing images accurately), making it impossible to know whether a model that excels at one dimension is genuinely factually reliable overall. The solution's "shape" is a weighted averaging framework: four independent evaluation pipelines, each with its own dataset, metric design, and judge model infrastructure, feed into a single FACTS Score defined as the arithmetic mean of the four individual accuracy scores, producing a balanced signal that penalizes models which optimize for one dimension at the expense of others.

3.2 Big-Picture Architecture (Diagram in Words)

The FACTS system has four independent evaluation pipelines connected by a single aggregation mechanism:

FACTS Multimodal Pipeline: Takes an image and a question → prompts the model under test to produce a free-text response → feeds both the response and a human-authored rubric (lists of Essential and Non-Essential facts) to an automated judge → judge produces two Boolean verdicts (Coverage and No-Contradiction) → only responses passing both are counted as accurate → accuracy is the fraction of such responses across ~1,522 questions.
FACTS Parametric Pipeline: Takes a factoid question (no context, no tools) → prompts the model under test to produce a short answer → feeds the question, gold answer, and model response to Gemini-2.5-Pro as a grader (sampled three times and averaged) → grader outputs a label from {correct, incorrect, not-attempted, unknown} → primary metric is accuracy (% correct), with secondary metrics for hedging rate and attempted accuracy → evaluated across 2,104 questions.
FACTS Search Pipeline: Takes a question designed to require web search → provides the model under test with access to the Brave Search API (identical API and tool description for all models) → model can issue search queries and receive results appended to its context, then produce a final answer → feeds the question, gold answer, and model response to Gemini-2.0-Flash as an auto-rater → rater outputs {correct, incorrect, not-attempted} → evaluated across 1,884 questions, with the number of search API calls also tracked.
FACTS Grounding v2 Pipeline: Takes a long-form context document, a user request, and a system instruction (all requiring generation based exclusively on the provided context) → prompts the model under test to produce a long-form response → feeds the response to two independent judge models (Gemini-2.5-Flash and GPT-5) using the v2 judge prompt → each judge outputs a binary {accurate, not accurate} label → an additional eligibility check (same two judges) determines whether the response meaningfully addresses the user request → ineligible responses are marked as inaccurate → final accuracy is the average of both judges' scores after eligibility adjustment → evaluated across the same prompts as FACTS Grounding v1.
FACTS Score Aggregation: Takes the four accuracy scores (one per pipeline) → computes the arithmetic mean → reports this as the single FACTS Score, with individual sub-scores and 95% confidence intervals also available.

Information flows independently through each pipeline (they share no data or model outputs), then converges only at the final averaging step. The public/private split mechanism operates at the dataset level: each benchmark contains public questions (released openly) and private questions (held back by Kaggle), with evaluation running on both and results averaged. This prevents overfitting to the public data.

3.3 Roadmap for the Deep Dive

First, the FACTS Score aggregation formula and its justification — because this is the single number the entire suite produces, and understanding what it computes and why it is a simple average (rather than a weighted combination) frames the rest of the design.
Second, the FACTS Multimodal pipeline — rubric construction, the dual-verdict evaluation protocol (Coverage + No-Contradiction), and autorater validation against human annotations — because this establishes the core pattern for how FACTS evaluates open-ended generative responses.
Third, the FACTS Parametric pipeline — the adversarial sampling procedure for data construction, the multi-stage filtering pipeline (LLM-based filters → adversarial sampling → human verification), and the grading protocol with Gemini-2.5-Pro — because this is the most methodologically complex data construction process and introduces the "attempted accuracy" metric that reveals hedging behavior.
Fourth, the FACTS Search pipeline — the four data subsets (Hard Tail, Wiki Two-Hop, Wiki Multi-Doc, KG Hops), the shared Brave Search API protocol, and the auto-rater design — because this introduces tool-use evaluation and the search efficiency dimension.
Fifth, the FACTS Grounding v2 pipeline — the judge model update from v1, the v2 prompt changes, the eligibility disqualification mechanism, and the human validation of judge quality — because this is an iterative improvement on prior work and introduces the problem of "ineligible" responses that legally evade grounding checks.
Sixth, cross-cutting design patterns — the public/private split mechanism, the Kaggle-hosted evaluation protocol, and the choice to use simple averaging rather than more complex aggregation — because these decisions affect the trustworthiness and interpretability of the entire suite.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a benchmark construction paper whose core idea is that a comprehensive factuality evaluation must span four distinct information-source scenarios (visual context with parametric knowledge, pure parametric recall, web search, and long-form grounding) and that averaging performance across these scenarios produces a more robust signal than any single benchmark alone.

The FACTS Score Aggregation Formula

The FACTS Score is defined in Section 2 as the arithmetic mean of four accuracy scores:

$\text{FACTS Score} = \frac{1}{4} \left( A_{\text{Multimodal}} + A_{\text{Parametric}} + A_{\text{Search}} + A_{\text{Grounding}} \right)$

where $A_{\text{Multimodal}}$ is the accuracy on the FACTS Multimodal benchmark (fraction of responses that both cover all essential facts and contain no contradictions), $A_{\text{Parametric}}$ is the accuracy on the FACTS Parametric benchmark (fraction of responses graded as correct), $A_{\text{Search}}$ is the accuracy on the FACTS Search benchmark (fraction of responses graded as correct), and $A_{\text{Grounding}}$ is the accuracy on the FACTS Grounding v2 benchmark (fraction of responses marked accurate by both judges after eligibility adjustment). Each accuracy term is itself computed as the average over the public and private splits of its respective dataset.

What this computes: The FACTS Score takes four task-specific accuracy percentages — each already representing an average across public and private dataset splits — and computes their unweighted arithmetic mean, yielding a single number between 0 and 100. This is a straightforward equal-weight aggregation: each of the four factuality dimensions contributes exactly 25% to the final score, regardless of dataset size, task difficulty, or score variance.

Why this form: The paper does not provide an explicit justification for equal weighting versus alternatives (e.g., weighted averaging based on task importance, or a more sophisticated latent-variable model that accounts for correlations between dimensions). However, the design implies several considerations. First, simplicity and interpretability: a simple average is transparent — any stakeholder can immediately understand how the score is computed and what a one-point improvement means. Second, no a priori basis for differential weighting: there is no principled reason to claim that visual factuality is more or less important than parametric recall, so equal weighting is the most neutral default. Third, penalizing imbalance: equal weighting ensures that a model cannot achieve a high FACTS Score by excelling at one dimension while performing poorly on others — a model with scores of 90, 90, 30, 30 averages to 60, making the weakness visible. A weighted scheme that down-weighted the third and fourth dimensions could hide this imbalance.

A potential limitation of this choice — which the paper does not discuss — is that task difficulty varies across benchmarks (Multimodal accuracy tops out at 46.9%, Search at 83.8%), meaning that improvements on the harder benchmarks contribute disproportionately less to score variance. If all models cluster within a narrow range on an easy benchmark, that dimension provides less discriminative power, yet it still receives equal weight. This is a standard challenge in benchmark aggregation, and the paper implicitly accepts it in favor of simplicity.

The uncertainty quantification is also notable: each sub-accuracy is reported with 95% confidence intervals in Table 1, but the paper does not propagate these intervals through the averaging to produce confidence intervals for the FACTS Score itself. The confidence intervals for sub-scores are presumably based on the binomial proportion of correct responses, but the FACTS Score is a mean of four dependent proportions (the same model generates all four scores), making simple interval propagation invalid. This is a minor methodological gap.

FACTS Multimodal: Dual-Verdict Rubric-Based Evaluation

The FACTS Multimodal pipeline evaluates whether a model's free-text response to an image-based question is both complete (covers all essential facts that a satisfactory answer should contain) and accurate (contains no claims contradicting ground truth). The key mechanism is a human-authored rubric paired with an automated judge that produces two Boolean verdicts.

Rubric Construction Protocol

For each of the approximately 1,500 image-question pairs (711 public, 811 private), human annotators create a structured rubric. A rubric consists of two categories of facts:

Essential Facts: facts that are critical for a complete and satisfactory answer. These define the minimum information a response must include to be considered helpful. For the locomotive example in Table 2, the essential facts are (1) identification of the specific locomotive model (FC Sonora-Baja California 2203) and (2) its introduction year (1949). A response missing either of these fails the coverage check.
Non-Essential Facts: additional relevant, contextual facts that enrich understanding but are not required for a satisfactory answer. In the locomotive example, these include the train's current location (National Museum of Mexican Railways), the visible number on the front ("2203"), and descriptive details (orange train, sunny day). These facts are used only for contradiction checking — a response that omits them can still pass coverage, but one that contradicts them fails.

The distinction between Essential and Non-Essential is critical because it decomposes evaluation into a recall dimension (did the model mention everything necessary?) and a precision dimension (did the model avoid saying anything false?). This decomposition reveals the tradeoff observed in Table 3, where Gemini models achieve higher Coverage scores (recall-oriented) while GPT models achieve higher No-Contradiction scores (precision-oriented). A simpler evaluation that used a single holistic score would obscure this structure.

The rubric covers three sources of factual truth: (a) the image itself (visual facts observable in the image), (b) common knowledge (widely known facts not requiring specialized expertise), and (c) the ground-truth rubrics themselves (curated facts about the specific image-content combination). Table 2 shows that contradiction checking extends beyond simple rubric matching — in the third example, the response correctly states that the artist's signature is visible but incorrectly describes its color (white vs. black) and location (on the mural vs. below it), which contradicts Non-Essential Fact #7 and triggers a contradiction verdict even though the essential facts are covered.

Automated Judge Protocol

The automated judge (referred to as an "autorater") evaluates each model response independently against the rubric, producing two Boolean verdicts:

Coverage Verdict: The judge checks whether the model response includes each of the essential facts specified in the rubric. The verdict logic is:

$\text{Coverage Bool} = \begin{cases} 1 & \text{if all essential facts are present in the response} \\ 0 & \text{otherwise} \end{cases}$

where "present" means the fact is stated in the response (not necessarily verbatim, but recognizably conveyed). For the locomotive example in Table 2, the response fails coverage because it covers only 1 of 2 essential facts — it identifies the correct locomotive but misstates the introduction year (saying the EMD FT was introduced in 1939 rather than identifying the SBC 2203's introduction in 1949). The coverage score is 50%, and since this falls below the threshold of completeness (all essential facts must be covered), the overall verdict is False.

What this computes: The Coverage verdict is a logical AND over the presence/absence of each essential fact. The model must demonstrate knowledge of all required information to pass. Partial coverage counts as failure.

Why this form: The all-or-nothing threshold operationalizes the definition of "satisfactory answer." An answer that correctly identifies the locomotive but gets its year wrong is not satisfactory — it provides misleading information on one dimension while succeeding on another. A softer metric (e.g., an F1 score over facts) would reward partially correct answers and obscure the fact that real users need complete, not partial, information.

No-Contradiction Verdict: The judge checks whether the model response contains any claim that contradicts either (a) the essential facts in the rubric, (b) the non-essential facts in the rubric, (c) common knowledge, or (d) the input image itself. The verdict logic is:

$\text{No-Contradiction Bool} = \begin{cases} 1 & \text{if no contradictions are found against any source} \\ 0 & \text{otherwise} \end{cases}$

where a contradiction is any claim in the response that is demonstrably false according to at least one of the four source types. Crucially, this check includes claims beyond the essential facts — a response can pass coverage (all essential facts present) but fail contradiction (an additional non-essential claim is false).

What this computes: The No-Contradiction verdict performs a universal verification sweep across the entire response, cross-referencing every information-bearing claim against all available ground-truth sources. It detects both direct contradictions (claiming a different genus for an animal, as in the second Table 2 example) and subtle factual errors (misdescribing the color or location of a visual element, as in the third example).

Why this form: Factuality is not just about saying the right things — it is also about not saying wrong things. A model that correctly identifies a butterfly's genus but then makes a false claim about its geographic distribution is factually unreliable even though its core answer is correct. The No-Contradiction check catches these "bonus errors" that simple rubric-matching would miss. The separate Coverage and No-Contradiction verdicts also enable analysis of the precision-recall tradeoff observed in Table 3.

Accuracy Score Derivation

The final accuracy for a response is:

$\text{Accuracy}_i = \text{Coverage Bool}_i \times \text{No-Contradiction Bool}_i$

where $\text{Accuracy}_i$ is 1 only if both Boolean verdicts are True, and 0 otherwise. The overall benchmark accuracy is:

$A_{\text{Multimodal}} = \frac{1}{N} \sum_{i=1}^{N} \text{Accuracy}_i$

where $N$ is the total number of questions (~1,522 across public and private sets). This represents the fraction of responses that are both complete (all essential facts) and accurate (no contradictory claims).

What this computes: A hard AND gate — a response must pass both the completeness check and the accuracy check to count as correct. There is no partial credit. A model that achieves 68.4% Coverage and 57.3% No-Contradiction (Gemini 3 Pro, Table 3) will have an Accuracy somewhere below both individual rates, since only the subset of responses passing both checks counts.

Why this form: The paper's design philosophy is that factuality in the multimodal setting requires both completeness and accuracy. A response that is complete but inaccurate (e.g., confidently stating wrong facts) is misleading. A response that is accurate but incomplete (e.g., a single vague sentence that avoids saying anything false but also doesn't answer the question) is unhelpful. The AND combines them into a single standard that captures the practical requirement of multimodal QA: the model should tell the user everything they need to know, and nothing that isn't true.

This is a strict standard, which explains the relatively low absolute accuracies in Table 3 (max 46.9% for Gemini 2.5 Pro). The Coverage and No-Contradiction scores show that no model achieves both high recall and high precision simultaneously — the best Coverage model (Gemini 3 Pro, 68.4%) has only 57.3% No-Contradiction, while the best No-Contradiction model (GPT-5, 64.7%) has only 59.9% Coverage. The AND operation amplifies this gap, since the product of two numbers both below 0.7 is necessarily below 0.5.

Autorater Validation Against Human Annotations

The paper validates both automated verdicts against human judgments on a held-out set (size unspecified, but separate from the test data). Table 4 reports the results.

For Coverage, the validation procedure is:

Human annotators mark each essential fact in the rubric as either "supported" or "unsupported" based on the model response.
The autorater performs the same marking task.
The percentage of essential facts supported is computed for both human and autorater.
Spearman's rank correlation is computed between the human and autorater continuous scores, yielding $\rho = 0.64$ .
For the Boolean comparison, a threshold of 0.5 is applied (if more than 50% of essential facts are supported, the response passes coverage), and a macro F1 score of 72.3 is achieved (with positive class = "passes coverage").

What this means operationally: The Spearman correlation of 0.64 means the autorater's ranking of responses by coverage completeness agrees reasonably well with human rankings, but there is substantial noise — the autorater does not perfectly replicate human coverage judgments. The macro F1 of 72.3 means that when the autorater makes a binary pass/fail decision, it agrees with the human binary decision about 72% of the time (averaging precision and recall across both classes). The negative-class metrics (F1 = 72.3, precision = 64.6, recall = 82.6) show that the autorater is more conservative about failing responses (high recall) but sometimes fails responses that humans would pass (lower precision). The positive-class metrics (F1 = 72.2, precision = 82.4, recall = 64.2) show the inverse pattern: when the autorater passes a response, it is usually correct (high precision), but it misses some responses that humans would pass (lower recall).

For No-Contradiction, the validation procedure is more fine-grained:

Human annotators assess the response sentence-by-sentence (Figure 2 shows the annotation interface).
For each sentence, annotators mark whether it contains a contradiction against the image, rubric facts, or common knowledge.
The autorater makes a holistic Boolean judgment about the presence of any contradiction anywhere in the response.
The macro F1 is 78.2, with positive class = "no contradiction present."

The sentence-level annotation protocol is significant because it captures subtle contradictions that a holistic annotator might miss. Figure 2's interface shows that annotators see the full response paragraph, a highlighted sentence, and must select one of {Nothing to assess, Contradiction, Neutral, Entailment} while providing a textual explanation. This forces careful, evidence-based judgments rather than gestalt impressions.

What the validation means for benchmark credibility: The macro F1 scores of 72.3 (Coverage) and 78.2 (No-Contradiction) indicate that the automated judges are reasonable but imperfect proxies for human evaluation. This is characteristic of automated evaluation for open-ended generation — perfect correlation with humans is generally unattainable — but the paper's transparency about the imperfect agreement is a strength. The reader is warned not to interpret sub-percentage-point differences in Multimodal scores as meaningful signal; the noise floor from judge imperfection is larger than that.

The paper does not report confidence intervals on these validation metrics or discuss how judge error might bias model rankings. If the coverage judge is systematically biased in ways that favor certain model families (e.g., if it is more lenient toward verbose responses typical of Gemini models), the benchmark's rankings could reflect judge bias rather than true model capability. The paper acknowledges this concern indirectly by noting the judge model comparison in Grounding v2 (Table 10), but does not extend the analysis to Multimodal.

Question and Image Category Diversity

Figure 1 shows the distribution of image categories and question categories in the public set. The diversity is deliberate:

Image categories span interior scenes (17.9%), people (15.2%), nature/wildlife (12.4%), product imagery (12.0%), illustrations/digital art (11.7%), architecture (10.8%), diagrams/data visualizations (10.7%), transportation (5.8%), text/typography (2.7%), and screenshots/UI (1.0%).
Question categories span visual description/captioning (45.7%), object/entity recognition (20.0%), abstract concept/cultural understanding (8.3%), data interpretation from charts/graphs (8.0%), reasoning (7.9%), text reading/OCR (3.7%), counting/quantifying (3.5%), creative generation (1.8%), mathematical/logical reasoning (0.8%), and comparison (0.3%).

Why this diversity matters for the FACTs Score: The heavy representation of visual description (45.7%) means that the Multimodal benchmark is primarily testing a model's ability to accurately describe what it sees, with object recognition and data interpretation as secondary competencies. If a model is weak at detailed visual description (missing key details or hallucinating visual elements), it will perform poorly on the largest question category. The relatively small representation of reasoning (7.9%) and mathematical reasoning (0.8%) means the benchmark does not heavily test higher-order visual cognition — this is primarily a perception-and-description benchmark with reasoning as a minor component. The paper's claim that the benchmark "covers a range of capabilities" is accurate but weighted toward descriptive tasks.

FACTS Parametric: Adversarially Hardened Factoid QA

The FACTS Parametric pipeline evaluates whether models can accurately recall factual knowledge from their pretraining parameters — without access to search tools — on questions that are designed to be challenging for frontier models. The construction process is methodologically the most elaborate of the four benchmarks, involving five distinct filtering stages (Section 4.1.2).

Stage 1: Question Collection from User Traffic

The initial question pool is collected to "reflect interest shown by many users" (Section 4.1.1). The paper does not specify the exact source (likely Google query logs or analogous user interaction data), but the key property is that these are genuine questions that real users ask, not artificially constructed for benchmark purposes. However, the paper notes a tension: "strictly following these guidelines tends to yield highly popular (and therefore easier) topics." To address this, the authors deliberately select the least frequent topics from the eligible set — they take the tail of the user-interest distribution rather than the head. This is a deliberate design choice: benchmark questions should be challenging, and popular topics (e.g., "What is the capital of France?") are too easy to discriminate between frontier models.

Stage 2: LLM-Based Factoid Filtering

The collected questions are filtered by an automated LLM-based classifier (model unspecified) to retain only those satisfying the "factoid criteria" detailed in Section 4.1.1:

Single, Atomic Fact: each question targets exactly one piece of information.
Unambiguous Answer: only one distinct correct answer exists.
Clear Answer Specification: the expected answer type (person, date, location, etc.) is stated or strongly implied in the question.
Concise, Factual Answers: expected output is a short entity, not a yes/no response or a long explanation.
Stable Facts: facts are static or explicitly time-anchored.

Why these criteria matter: They ensure that evaluation is straightforward and reliable. If a question could have multiple valid answers (e.g., "Who directed that movie?" when there were co-directors), grading becomes ambiguous. If the answer format is unspecified (e.g., "Tell me about the Eiffel Tower"), the model might produce correct information that doesn't match the expected gold answer format. The criteria restrict the benchmark to queries where correctness is objectively determinable and the answer can be matched to a gold standard with minimal ambiguity.

Stage 3: Adversarial Sampling with Open-Weight Models

This is the key mechanism for ensuring the benchmark remains challenging. The procedure:

Generate preliminary "silver" labels for all questions using Gemini-2.5-Pro equipped with search tools. The model returns both a generated answer and the specific search results (including URLs) used to derive it.
Filter to retain only questions where the model's answer is supported by a Wikipedia URL found in the search results. This ensures every retained question has verifiable ground truth in a source that is "highly prevalent and widely assumed to be a significant component of the training corpora for all LLMs" (Section 4.1.1).
Collect responses from five strong open-weight models in a closed-book setting (no search access). The paper deliberately uses open-weight models — rather than proprietary API models — to "decouple the adversarial selection process from the proprietary API models used in our evaluation, thereby ensuring unbiased filtering" (Section 4.1.2). If the same model family were used for both adversarial filtering and evaluation, the benchmark would systematically select questions that particular model architecture finds hard, biasing rankings.
Retain only questions that none of the five open-weight models answered correctly, where "correctly" is defined as matching the silver label from Gemini-2.5-Pro.

What this computes: The adversarial sampling acts as a hardness filter. A question that all five open-weight models can answer correctly is too easy to discriminate between frontier models — it is discarded. A question that at least one model can answer correctly may still be too easy — also discarded. Only questions that defeated all five models survive. This produces a dataset skewed toward the right tail of difficulty, which is precisely what is needed for a benchmark targeting frontier model comparison.

Why this form: The paper explicitly states the goal: "a key goal is to ensure the benchmark remains challenging for frontier LLMs, avoiding saturation in the near future" (Section 4.1.2). Adversarial sampling is a standard approach in benchmark construction (similar to the approach used in SQuAD 2.0 for unanswerable questions or in adversarial NLI datasets), but the use of open-weight models for unbiased filtering is a thoughtful methodological choice that other benchmarks often overlook. The alternative — using the same proprietary models that will later be evaluated — would create a circularity where the benchmark's difficulty is defined relative to a specific model family's weaknesses.

The paper does not specify which five open-weight models were used, their sizes, or their training cutoff dates, which limits reproducibility. If the open-weight models are significantly weaker than the frontier models being evaluated, the filtering may not be sufficiently aggressive — questions that defeat Llama-3-8B but not Gemini-3-Pro might survive, creating a benchmark that is hard for weaker models but still easy for frontier models. Conversely, if the open-weight models are very strong, the benchmark might be so aggressively filtered that it only contains questions no model can answer, reducing discriminative power among the top models.

Stage 4: Human Verification

After adversarial filtering, each surviving question-answer pair undergoes verification by three independent third-party human annotators. The annotation protocol (Section 4.1.2) includes four tasks:

Factuality Assessment: Annotators determine whether the answer is correct, based on information available on the web (not just the provided Wikipedia URL). Possible verdicts:
- Accurate: verifiably correct.
- Inaccurate: either contradictory sources exist, no supporting information found, or the topic is disputed.
Properties Compliance Check: Does the QA pair adhere to all the factoid criteria from Section 4.1.1? Binary True/False.
Wikipedia Evidence Extraction: Identify and provide a Wikipedia document that confirms the fact. The annotator is given a candidate URL but must verify it or find an alternative. Examples lacking supporting Wikipedia evidence are disqualified.
Correction Provision: If a QA pair is rated as Inaccurate or fails compliance, but the issue can be corrected with a simple fix (to question, answer, or URL) without altering the original user intent, the annotator proposes a correction. A new annotator then finalizes the corrected example, with the option to disqualify it.

This multi-step human pipeline serves as the final quality filter. The requirement that three independent annotators verify each example increases reliability — a single annotator might make errors or miss subtle issues. The correction mechanism is particularly important for benchmark efficiency: rather than discarding all non-compliant examples, fixable issues are repaired, preserving the user-interest signal in the original question.

Stage 5: Final Dataset Composition

The final dataset contains 2,104 QA pairs, equally divided into 1,052 public and 1,052 private questions. Table 5 shows examples spanning diverse topics (artists, musicians, boxers, chemistry), answer types (places, performers, records, numbers), and answer formats (free text, countable values with acceptable ranges). Figure 3 shows the topic and answer-type distributions in the public set — the paper does not provide detailed percentages, but the visual shows coverage across politics, sports, technology, geography, and other domains.

Grading Protocol

The evaluation uses Gemini-2.5-Pro as the grader (Section 4.2). The grading scheme follows Wei et al. (2024) with modifications:

Grading Labels: For each (question, gold answer, model response) triple, the grader outputs one of {correct, incorrect, not-attempted, unknown}. The unknown label is a modification of the Wei et al. scheme, introduced for cases where the grader is unsure whether the model response matches the gold answer. The paper claims this "improves the already high accuracy presented by the grader" (Section 4.2).

Primary Metric: Accuracy, defined as:

$A_{\text{Parametric}} = \frac{N_{\text{correct}}}{N_{\text{total}}}$

where $N_{\text{correct}}$ is the number of responses graded as correct and $N_{\text{total}}$ is 2,104 (the full dataset).

Secondary Metrics:

Hedging rate: $\frac{N_{\text{not-attempted}}}{N_{\text{total}}}$ , the fraction of responses where the model explicitly declines to answer (e.g., "I don't know," "I cannot answer that question").
Attempted accuracy: $\frac{N_{\text{correct}}}{N_{\text{correct}} + N_{\text{incorrect}}}$ , the accuracy computed only on responses where the model made an attempt (excluding hedging responses). This metric reveals how accurate the model is conditional on trying.
F1 score: the harmonic mean of accuracy and attempted accuracy:

$F1 = 2 \cdot \frac{A_{\text{Parametric}} \cdot A_{\text{Parametric}}^{\text{attempted}}}{A_{\text{Parametric}} + A_{\text{Parametric}}^{\text{attempted}}}$

where $A_{\text{Parametric}}^{\text{attempted}}$ is the attempted accuracy.

What F1 computes in this context: The F1 score balances overall accuracy (which rewards answering everything, even with guesses) and attempted accuracy (which rewards only answering when confident). A model with high attempted accuracy but high hedging rate (like Claude models, Table 6) gets penalized on the F1 relative to a model with similar attempted accuracy but lower hedging. Conversely, a model with high overall accuracy but low attempted accuracy (guessing aggressively) also gets penalized. The F1 incentivizes a balance between coverage and precision.

Why this form: The paper explains the rationale through the GPT-o3 vs. GPT-5 comparison: "Although GPT-o3 achieves higher raw accuracy (57.0% vs. 55.7%), GPT-5 hedges significantly more often (13.3% of cases vs. 1.9%). Consequently, GPT-5 achieves superior attempted accuracy (64.3% vs. 58.2%) and F1 scores (59.7 vs. 57.6)." (Section 4.2). The F1 captures a dimension of performance that raw accuracy misses: a model that correctly identifies when it doesn't know something is more trustworthy than one that confidently guesses. In a user-facing application, a hedge ("I'm not sure") is often preferable to a confident falsehood.

Grading Reliability via Triple Sampling

To enhance reliability, the grader (Gemini-2.5-Pro) is queried three times per model response with identical inputs, and the three grades are averaged to determine the final score (presumably by taking the mode or averaging the correctness probabilities — the paper doesn't specify the exact aggregation mechanism for the three samples). This is a simple ensemble technique that reduces the variance from a single LLM-as-judge call, which can be sensitive to prompt phrasing and sampling noise.

Validation of the Single-Judge Approach

The paper considered using a mixed-model panel (sampling once each from Gemini-2.5-Pro, GPT-o3, and Grok-4) but found that Gemini-2.5-Pro alone "preserves the same relative performance trends and rankings as the more complex ensemble" (Section 4.2). This is an efficiency decision: a single judge is simpler to maintain and cheaper to run. The paper does not provide quantitative evidence for this claim (no correlation coefficients or ranking comparisons are reported), so the reader must take it on faith that the single-judge rankings are equivalent to the panel rankings. This is a minor methodological gap.

FACTS Search: Tool-Augmented Factuality

The FACTS Search pipeline evaluates how well models use a web search API to answer questions that are designed to be difficult or impossible to answer from parametric knowledge alone. The pipeline has three distinguishing features: (1) the four-quadrant data composition with different hardness mechanisms, (2) the standardized search API protocol, and (3) the tracking of search efficiency alongside accuracy.

Data Composition: Four Subsets

The 1,884 questions (890 public, 994 private) are drawn from four distinct sources, each targeting a different aspect of search difficulty:

Subset 1: Hard Tail (328 questions after filtering). Human raters are instructed to write questions that require information challenging to extract via web search — specifically, "that there is no single-step web search answer available on the first page, or the information is not readily available as a verbatim piece of text on the internet" (Section 5.1). These are not simple factoid lookups; they require the model to synthesize information from multiple search results or to interpret non-obvious connections. Raters also verify that the then-publicly-available Gemini (1.5) could not solve these questions even with search, ensuring they are genuinely challenging.

What this subset tests: The ability to perform multi-step search reasoning where the answer is not directly returned by a single query. This is analogous to the "complex search" tasks in prior benchmarks like GAIA (Mialon et al., 2023) but with explicit verification that the information exists on the web and is merely hard to find.

Subset 2: Wiki Two-Hop (932 questions after filtering). Synthetically generated two-step questions from Wikipedia. The generation process:

Extract QA pairs from Wikipedia abstracts (simple one-hop questions like "What is the birthplace of John Lennon?").
Filter to focus on tail entities (less frequently mentioned entities, which are harder for models to recall parametrically).
Modify each question by substituting the main entity with a different description extracted from the Google Knowledge Graph. For example, "What is the birthplace of John Lennon" becomes "What is the birthplace of Yoko Ono's spouse." This creates a two-hop requirement: the model must first identify that "Yoko Ono's spouse" refers to John Lennon (hop 1, likely requiring search), then answer the original question about his birthplace (hop 2, also likely requiring search or parametric recall).

What this subset tests: The ability to resolve indirect entity references — a common real-world scenario where users don't know the exact name of what they're asking about and refer to entities by relationship or description. The Knowledge Graph integration ensures the indirect references are factually correct and resolvable.

The paper notes that these questions were evaluated on Gemini models (with and without search) available at the time, which achieved "low accuracies of 30% and 38%" (search-off and search-on respectively). This baseline confirms the questions are difficult but not impossible with search — there is headroom for better models to improve.

Subset 3: Wiki Multi-Doc (268 questions after filtering). Synthetically generated multi-document synthesis questions. The generation process:

Sample a seed Wikipedia document $D_{\text{seed}}$ .
Find a set of related documents $D_{\text{related}}$ and keep only those with "rank that is neither too low nor too high" — eliminating documents that are too similar (trivial connections) or too distant (unrelated). These become $D_{\text{torso}}$ .
Prompt Gemini to synthesize a query-answer pair $(Q, A)$ from the content of these $n$ documents, where the query can only be answered by synthesizing information from both $D_{\text{seed}}$ and at least one document from $D_{\text{torso}}$ . The prompt encourages finding "interesting ways to connect information rather than relying on a simple direct chain or a combination of unrelated queries" (Section 5.1).

The filtering pipeline applies two checks:

Self-containment check: an automated critic model verifies that the question is self-contained (it can be understood without seeing the source documents).
Hardness check: discard any queries that Gemini can correctly answer with standard web search tools (if search makes it trivially solvable, it is not a good search benchmark question).

What this subset tests: The ability to locate, read, and synthesize information across multiple documents — a more complex version of multi-hop reasoning that requires not just entity resolution but cross-document information integration. This is a realistic use case for search-augmented LLMs in research, analysis, and fact-checking.

Subset 4: KG Hops (356 questions after filtering). Synthetically generated using path queries in the Google Knowledge Graph. The generation process:

Collect common path queries (e.g., "films that actor X appeared in").
Concatenate and combine these with aggregation functions (e.g., max, min) to create multi-step queries: "publication date of the first film that actor X appeared in" combines the film-appearance path with the film-publication-date path, then applies the earliest/min aggregation.
The resulting questions require traversing multiple knowledge graph edges and applying an operation to the results.

What this subset tests: The ability to perform structured reasoning over relational knowledge — not just retrieving facts but computing derived facts through graph traversal operations. This targets a different capability than the natural-language-focused Wiki subsets.

Human Verification of All Subsets

All questions (across all four subsets) undergo verification by three independent human raters for three criteria:

Correctness: use Google search to verify the gold answer is correct.
Uniqueness: check that no other entity could also be a correct answer (ensuring single-answer questions).
Immutability: assess whether the answer is likely to change within five years (ensuring benchmark longevity).

Only questions passing all three checks are retained. Additionally, a final filtering step removes any questions that Gemini-2.5-Flash without search can answer correctly, ensuring the benchmark genuinely requires search — if a model can answer from parametric knowledge alone, the question doesn't test search capability.

Standardized Search API Protocol

All evaluated models access the same Brave Search API with the same tool description (Section 5.2). The protocol ensures cross-model comparability:

Each model receives an identical description of the search tool's capabilities and interface.
When a model triggers a tool call (issues a search query), the Brave Search API is queried in real-time, and the returned results are appended to the model's context window.
The model can issue multiple search queries per question (as evidenced by Table 8's "Average searches" column).
The final response must be produced after the model decides it has sufficient information.

Why this protocol matters: Prior search benchmarks often confounded model capability with retrieval infrastructure — if one model was evaluated with Google Search and another with Bing, performance differences could reflect search engine quality rather than model skill. By standardizing the API, FACTS Search isolates the model's ability to formulate effective queries, interpret search results, and synthesize information from retrieved content. This is a genuine contribution to benchmark methodology, not just a convenience.

The choice of Brave Search (rather than Google, Bing, or a custom search index) is not explicitly justified. Brave Search is a privacy-focused search engine with its own index — its coverage and ranking may differ from the search engines users typically encounter, which introduces a minor ecological validity concern.

Auto-Rater Protocol

Grading uses Gemini-2.0-Flash as an auto-rater (Section 5.3). For each (query, model response, gold answer) triple, the rater outputs {correct, incorrect, does not attempt}. The primary metric is accuracy:

$A_{\text{Search}} = \frac{N_{\text{correct}}}{N_{\text{total}}}$

where $N_{\text{total}} = 1,884$ . Table 8 reports additional search-specific metrics:

Attempted accuracy: accuracy conditional on the model making an attempt.
Hedging rate: fraction of responses where the model does not attempt.
F1: harmonic mean of accuracy and attempted accuracy (same formula as Parametric).
Average searches: mean number of Brave Search API calls per question.

Why average searches is tracked: This metric reveals an efficiency dimension that prior search benchmarks ignored. Table 8 shows substantial variation: Gemini 3 Pro achieves the highest accuracy (83.8%) with only 3.39 average searches, while Grok 4 Fast uses 4.74 searches to achieve 67.3% accuracy. More searches do not necessarily yield better results — they may indicate inefficient query formulation or failure to extract answers from earlier results. By tracking this metric, FACTS Search encourages not just accuracy but also search efficiency, which has real-world implications for latency and API cost.

FACTS Grounding v2: Improved Judge Models with Eligibility Screening

The FACTS Grounding v2 pipeline is an iterative improvement over FACTS Grounding v1 (Jacovi et al., 2025). The dataset is unchanged from v1; the improvements are entirely in the evaluation protocol: updated judge models, a revised judge prompt, and an explicit eligibility disqualification mechanism.

Dataset Structure (Unchanged from v1)

Each evaluation example consists of three components:

System instruction: directs the model to generate its response exclusively from the provided context, without using external knowledge. Table 9 shows variants: "Answer the question using only the information provided in the context," "Provide a response based solely on the information provided in the prompt," and more elaborate versions.
Context document: a document or set of reviews sourced from the web, ranging up to 32,000 tokens in length, covering diverse enterprise domains (finance, technology, retail, medical, legal — Figure 4 shows the distribution).
User request: a non-trivial question or task (QA, summarization, rewriting) requiring a long-form response, designed to require processing the full context.

The annotation instructions for the original dataset construction (described in Jacovi et al., 2025) explicitly avoided prompts requiring creative responses, expert domain knowledge, mathematical/logical reasoning, or meta-analysis (tone analysis, interpretation of author intent). The resulting tasks focus on information extraction and faithful summarization — the model must accurately represent what the document says, not draw inferences beyond the text.

Judge Model Selection and Prompt Update (the v2 changes)

The changes from v1 are described in Section 6.2:

New judge models: v2 uses Gemini 2.5 Flash and GPT-5 as the two judge models, replacing v1's panel of Gemini 1.5 Pro, GPT-4o, and Claude 3.5 Sonnet. The paper justifies this change through a direct comparison on a held-out evaluation set (N=320, class ratio 79:19, meaning 79% positive/"accurate" examples and 19% negative/"not accurate", presumably from human annotations).

New prompt (v2 prompt): A "slightly modified version" of the v1 judge prompt. The paper does not provide the full prompt text, but the comparison in Table 10 shows that the v2 prompt improves performance across multiple judge model and prompt combinations.

Evaluation of Judge Quality (Table 10)

Table 10 compares 12 judge model × prompt combinations on the held-out set, using macro F1 as the selection metric:

$\text{Macro-F1} = \frac{1}{2} \left( F1_{\text{positive}} + F1_{\text{negative}} \right)$

where $F1_{\text{positive}}$ is the F1 for the "accurate" (grounded) class and $F1_{\text{negative}}$ is the F1 for the "not accurate" (hallucination) class.

What this evaluates: Macro-F1 treats both classes as equally important, which is appropriate here because both false positives (labeling a hallucinated response as accurate) and false negatives (labeling a grounded response as inaccurate) are serious errors. A metric like accuracy would be misleading with the 79:19 class imbalance — a judge that always predicted "accurate" would achieve 79% accuracy but 0% macro-F1.

Results of the comparison:

The winning combination is Gemini 2.5 Flash with the v2 prompt: macro-F1 = 65.33.
The second-best is GPT-5 with the v2 prompt: macro-F1 = 65.18.
The v1 prompt combinations for Gemini 2.5 Flash and Gemini 2.5 Pro achieve macro-F1 of 63.44 and 64.87 respectively — the v2 prompt provides a small but consistent improvement.
The best v1-only combination (Gemini 2.5 Pro, v1 prompt, macro-F1 = 64.87) is close to the best v2 combination, suggesting the judge model upgrade matters more than the prompt change.
The winning positive-class F1 (F1 for detecting accurate/grounded responses) is 84.51, while the negative-class F1 (F1 for detecting hallucinations) is only 46.15. This indicates that even the best judge struggles to reliably identify ungrounded responses — false negatives (missing hallucinations) are a significant problem.

Why two judges are used: The paper states this is "to reduce the bias of a particular judge model, as models have been shown to be biased towards favorably judging their own outputs (Wataoka et al., 2024)." By averaging scores from two different model families (Gemini and GPT), the risk of systematic self-favoring bias is reduced. The individual factuality score for each judge is the percentage of responses marked as accurate, and the unadjusted factuality score is the arithmetic mean of the two judges' scores:

$\text{Unadjusted Score} = \frac{1}{2} \left( \text{Score}_{\text{Gemini 2.5 Flash}} + \text{Score}_{\text{GPT-5}} \right)$

Eligibility Disqualification Mechanism

This is the key methodological innovation in v2. The paper identifies a vulnerability in grounding benchmarks: "metrics that are focused on evaluating the factuality of the generated text with respect to a context document can be 'hacked' by ignoring user intent" (Section 6.2). A model can achieve high grounding scores by producing short, vague responses that contain no false claims but also provide no useful information.

The eligibility check: The same two judge models (Gemini 2.5 Flash and GPT-5, using the same v2 prompt) are prompted to output a binary label:

eligible: the response meaningfully addresses the user request.
ineligible: the response does not sufficiently address the user request, even if it contains no factually incorrect statements.

The adjustment rule: Ineligible responses are marked as inaccurate, regardless of their grounding quality. The formal rule is:

$\text{Final Accuracy}_i = \begin{cases} 1 & \text{if both judges mark the response as accurate AND the response is eligible} \\ 0 & \text{otherwise} \end{cases}$

This is effectively a grounding-AND-eligibility filter: a response must be both grounded in the context and responsive to the user's request to count as correct. An ineligible response is treated as incorrect even if every claim in it is factually true.

What this prevents: Table 11's examples make the mechanism concrete. A model asked to "summarize the key advantages and disadvantages of wind energy from this document" that responds with "Wind energy is good because it is renewable and clean, but it has some challenges too" is technically grounded — the document likely discusses wind energy's renewability and challenges — but it fails to provide the specific information requested (cost-effectiveness data, geographic limitations, wildlife impacts). The eligibility judge catches this evasion.

Why this form: The eligibility check closes a loophole that model developers could exploit. Without it, a model optimized for grounding benchmarks might be trained to produce minimally informative but trivially accurate outputs — a strategy that achieves high scores on the benchmark but fails in real applications. By disqualifying such responses, FACTS Grounding v2 aligns the benchmark incentive with the actual user need: not just "don't say anything false" but "say something useful and true."

The choice to use the same two judges for both the grounding check and the eligibility check is pragmatic — it avoids deploying additional judge models — but introduces a subtle dependency: errors in the eligibility judgment (e.g., an eligible response marked as ineligible) propagate directly to the final score. The paper does not validate the eligibility judge against human annotations, which is a gap in the methodological rigor compared to the Multimodal autorater validation.

Final Score Calculation

The final accuracy for FACTS Grounding v2 is:

$A_{\text{Grounding}} = \frac{1}{N} \sum_{i=1}^{N} \text{Final Accuracy}_i$

where $\text{Final Accuracy}_i$ incorporates both the grounding judgments from two judges and the eligibility judgment, as described above. The number of prompts $N$ is the same as FACTS Grounding v1 (the paper does not specify the exact count, but it is the full set from the prior work). Table 1 reports confidence intervals based on this accuracy (e.g., Gemini 3 Pro at 69.0 ± 2.1), computed presumably from the binomial proportion.

Cross-Cutting Design Patterns

Several design decisions apply across all four benchmarks and collectively define the FACTS approach to benchmark construction.

Public/Private Split Mechanism

Each benchmark divides its data into public and private subsets. The paper states the sizes:

Multimodal: 711 public, 811 private (1,522 total)
Parametric: 1,052 public, 1,052 private (2,104 total)
Search: 890 public, 994 private (1,884 total)
Grounding: same dataset as v1 (the paper doesn't provide exact public/private counts for v1)

Why this split exists: "To mitigate overfitting, only a subset of the prompts will be released publicly, and the remaining prompts will remain private" (Section 2). This is a standard mechanism for preventing benchmark contamination — model developers cannot train on the private data because they never see it. The public split allows transparency and reproducibility (researchers can inspect examples, understand the task, and develop methods), while the private split ensures that reported scores reflect generalization rather than memorization.

Why the split ratios vary: The paper does not explain why the public/private ratio differs across benchmarks (Multimodal is roughly 47/53 public/private, Parametric is exactly 50/50, Search is roughly 47/53). The inconsistency may reflect different data availability or different judgments about how much public data is needed for meaningful task understanding.

Kaggle-Hosted Evaluation Protocol

All model evaluation is conducted by Kaggle (Section 2), not by the model developers themselves. This means:

Model developers submit their models to Kaggle (presumably via an API or container specification).
Kaggle runs the models on both public and private data.
Kaggle computes and publishes the scores.

Why this matters: Self-reported benchmark scores are notoriously vulnerable to methodological inconsistencies — different teams might use different prompting strategies, temperature settings, or evaluation protocols, making scores incomparable. Centralized evaluation through Kaggle ensures:

Consistent infrastructure (all models receive the same inputs through the same pipeline).
Consistent metric computation (no discrepancies in grading implementation).
Protection of private data (model developers never access it).
Audit trail (Kaggle maintains records of model versions and evaluation runs).

This is similar to the approach of benchmarks like MMLU (through Hugging Face) or the Open LLM Leaderboard, but the private split adds an additional integrity layer that many public leaderboards lack.

Kaggle's Role: Kaggle is a subsidiary of Google (the same company that develops the Gemini models evaluated on the benchmark). The paper does not address the potential conflict of interest in having a Google-affiliated platform run evaluations that include Google's own models. However, the use of private data and automated evaluation reduces (though does not eliminate) the risk of preferential treatment, since Kaggle presumably applies identical evaluation code to all submissions.

Choice of Simple Averaging for the FACTS Score

The FACTS Score is an unweighted average, as discussed earlier. The paper does not explore alternatives or provide sensitivity analysis showing how rankings would change under different weighting schemes. This is a deliberate simplification: a complex weighting scheme would require justifying the weights, which would be contentious (is search more important than parametric recall? by how much?), and would reduce interpretability.

Why the choice matters: The equal weighting implicitly assumes that the four factuality dimensions are equally important in practice, which may not be true for any specific application. A user deploying a model for document summarization cares mostly about grounding; a user deploying a trivia bot cares mostly about parametric recall; a user deploying a research assistant cares mostly about search. The FACTS Score is designed as a general-purpose metric, not a task-specific one, and the paper's position is that a model achieving a high FACTS Score is likely to be factually reliable across many scenarios, even if not optimal for any single one.

Confidence Interval Reporting

Table 1 reports 95% confidence intervals for each sub-benchmark and each model. The intervals appear to be based on the binomial proportion (standard error $\sqrt{p(1-p)/N}$ multiplied by the 1.96 z-score for 95% confidence). For example, Gemini 3 Pro on Grounding has an accuracy of 69.0 ± 2.1, suggesting a sample size around 500 (since the standard error for p=0.69 with N=500 is approximately $\sqrt{0.69 \times 0.31 / 500} \approx 0.0207$ , and 1.96 × 0.0207 ≈ 0.041, but the reported interval is ±2.1, so the effective N is larger or a different formula is used). The paper does not specify the exact confidence interval computation method, which is a minor omission.

Why confidence intervals are important: The top of Table 1 shows models clustered within a few percentage points — Gemini 3 Pro at 68.8, Gemini 2.5 Pro at 62.1, GPT-5 at 61.8. With confidence intervals of roughly ±2 points per sub-score, the ranking of the top 3 is reasonably reliable, but the ordering of closely-scored models (e.g., Grok 4 at 53.6 vs. GPT o3 at 52.0) may be within the margin of error. The confidence intervals enable readers to assess which differences are statistically meaningful and which could be noise.

Summary of Design Choices and Their Justifications

Four-pillar structure over a single unified benchmark: Each dimension requires fundamentally different model capabilities (visual perception, parametric recall, search tool use, long-form grounding), and isolating them enables fine-grained analysis of model strengths and weaknesses.
Dual-verdict Multimodal evaluation (Coverage + No-Contradiction): Decomposes factuality into recall and precision, revealing the tradeoff between model families that single-score evaluation would obscure.
Adversarial sampling with open-weight models for Parametric: Decouples hardness filtering from the proprietary models being evaluated, preventing circular bias in benchmark difficulty.
Standardized Brave Search API for Search: Eliminates search infrastructure as a confounding variable, isolating model search capability.
Two-judge averaging for Grounding v2: Mitigates self-preference bias in LLM-as-judge evaluation, with the v2 judge model update improving negative-class detection.
Eligibility disqualification for Grounding v2: Closes the loophole of trivial-but-grounded responses, aligning benchmark incentives with real user needs.
Triple-sampled grading for Parametric: Reduces grader variance from single LLM-as-judge calls.
Simple average FACTS Score: Prioritizes interpretability and neutrality over potentially controversial weighting schemes.
Public/private splits across all benchmarks: Protects against overfitting and data contamination, with Kaggle-hosted evaluation ensuring consistent infrastructure.

4. Key Insights and Innovations

Innovation 1: Factuality Must Be Decomposed Into Four Independent Information-Source Dimensions, and No Single Dimension Can Proxy for the Others

The paper's most fundamental conceptual contribution is the claim — demonstrated empirically rather than merely asserted — that factuality in LLMs is not a unitary construct but decomposes into capabilities tied to distinct sources of truth, and that performance across these sources is sufficiently independent that a model excelling at one provides no guarantee about the others. This reframes factuality evaluation from a one-dimensional "is the model accurate?" question into a multi-dimensional profiling task.

Prior to FACTS, the research community implicitly treated factuality as a property that could be captured by any sufficiently rigorous benchmark. A model that scored well on Natural Questions (Kwiatkowski et al., 2019) was presumed to be "factually accurate"; a model that scored well on a grounding benchmark (Honovich et al., 2022; Jacovi et al., 2025) was presumed to be "factually grounded." The possibility that these benchmarks might measure fundamentally different capabilities — and that improving one might come at the expense of another — was not systematically investigated because no framework existed to test it.

The FACTS results make this independence concrete. Table 1 shows that GPT-5 achieves 77.7% on Search (ranking 2nd) but only 55.8% on Parametric (ranking 4th), and 44.1% on Multimodal (ranking 3rd). Gemini 2.5 Pro achieves 74.2% on Grounding (ranking 1st) but only 63.2% on Parametric (ranking 2nd). Claude 4.5 Opus achieves 73.2% on Search (ranking 4th) but only 30.6% on Parametric (ranking 8th). These are not minor rank perturbations — they are dramatic reversals where a model leading one dimension trails badly on another. If all dimensions measured the same underlying capability, rankings would be largely consistent across sub-benchmarks. The fact that they are not — that the correlation structure across dimensions is weak enough to produce cross-dimensional rank inversions — empirically validates the paper's central thesis that comprehensive factuality evaluation requires measurement across all four information-source scenarios, not just one.

This is a fundamental insight rather than an incremental one because it changes what it means to claim a model is "factually improved." Under the old paradigm, a researcher could report a 10-point gain on a parametric QA benchmark and credibly claim to have "improved factuality." Under the FACTS framework, such a claim requires showing that the gain on parametric did not come at the cost of degraded grounding or search performance. The FACTS Score implements this directly: a model that gains 10 points on Parametric but loses 10 on Grounding shows zero net improvement. The simple arithmetic mean — which might appear naive to readers expecting a more sophisticated aggregation — is revealed as a deliberate design choice that enforces balanced improvement across all dimensions.

The practical implication extends beyond benchmarking. The paper's discovery that model families exhibit systematic tradeoffs — Gemini favoring coverage over precision (Table 3: high Coverage, lower No-Contradiction), GPT favoring precision over coverage (lower Coverage, high No-Contradiction), Claude favoring hedging over accuracy on difficult parametric questions (Table 6: 29.9% hedging rate for Claude 4.1 Opus) — implies that these tradeoffs reflect fundamental architectural or training objectives, not accidental bugs. A model trained with an RLHF objective that heavily penalizes hallucinations will learn to hedge aggressively on parametric queries; the same training may produce more verbose, coverage-oriented outputs on multimodal tasks. The FACTS framework makes these tradeoffs visible and quantifiable for the first time, opening an empirical research program into how training objectives, architecture choices, and data mixtures shape the multi-dimensional factuality profile.

Innovation 2: Verifier Over-Optimization at the Benchmark Level — The "Ineligible Response" Attack on Grounding Benchmarks and the Dual-Verdict Solution

The paper identifies and systematically addresses a subtle vulnerability in grounding benchmarks that prior work either missed or failed to solve: the ineligible response attack, where a model achieves high grounding scores by producing trivially short, vague responses that contain no false claims but also provide no useful information. This is an instance of Goodhart's Law — "when a measure becomes a target, it ceases to be a good measure" — applied at the meta-benchmark level, and the paper's solution (the eligibility disqualification mechanism in FACTS Grounding v2, Section 6.2) is a conceptual contribution to benchmark design methodology, not just an implementation detail.

Prior grounding benchmarks, including FACTS Grounding v1 (Jacovi et al., 2025), evaluated whether model responses were factually consistent with provided documents. The evaluation was a verification task: given a response and a source document, are all claims in the response supported by the document? A model that generated "The document discusses wind energy" in response to a request to summarize wind energy's advantages would pass this check — the claim is true, and it is supported. But the response is useless. The benchmark's metric (grounding accuracy) was misaligned with the user's actual need (helpful, informative responses), creating an incentive for models to game the metric by minimizing content while maximizing factual safety.

Table 11 provides concrete examples that make the vulnerability tangible. The model asked to "summarize the main reasons the company's revenue decreased in Q3" responds with "The company faced challenges in Q3 that impacted its revenue." This statement is true (the document almost certainly discusses challenges), contains no false claims, and passes a naive grounding check — but it fails to provide the specific reasons the user asked for, making it a successful evasion of the user's intent while technically satisfying the benchmark's metric.

The conceptual move that distinguishes FACTS Grounding v2 is recognizing that grounding is a necessary but not sufficient condition for helpful factual responses. The paper decomposes the evaluation problem into two separate judgments: (1) is the response grounded? (the original v1 question), and (2) does the response meaningfully address the user's request? (the new eligibility question). Only responses passing both are counted as accurate. This is a structural fix at the benchmark design level, not a change to model evaluation methodology — it redefines what "correct" means to close a loophole that simpler evaluations left open.

The significance of this innovation is that it generalizes beyond grounding. Any benchmark that evaluates output quality using a single dimension of correctness (accuracy, consistency, faithfulness) is vulnerable to the same attack: models can optimize the measured dimension while degrading unmeasured dimensions that matter to users. The eligibility disqualification pattern — adding a second evaluation criterion that captures response utility and requiring both criteria to be satisfied — is a template that future benchmark designers can adopt for tasks where correctness alone is insufficient. The paper's choice to use the same judge models for both grounding and eligibility judgments (Gemini 2.5 Flash and GPT-5 with the v2 prompt) is a pragmatic simplification, but the conceptual structure of the solution is independent of the specific judge implementation.

The empirical validation of this innovation comes from the judge model comparison in Table 10, which shows that the v2 combination (new models + new prompt) achieves a macro-F1 of 65.33 — but more importantly, the negative-class F1 (detecting ungrounded responses) is only 46.15, meaning the judge still misses more than half of true hallucinations. This reveals that even with the structural fix, the practical limitations of automated judging remain significant, and the benchmark's reliability is bounded by judge quality. The innovation is in the problem identification and the structural solution pattern, not in claiming the solution is perfect.

Innovation 3: Adversarial Benchmark Construction Using Open-Weight Models to Decouple Hardness Filtering from Evaluation Targets

The FACTS Parametric pipeline introduces a methodological innovation in benchmark construction: using open-weight models as adversarial filters to ensure benchmark difficulty without introducing bias toward the proprietary models that will later be evaluated on the benchmark. This addresses a subtle circularity problem that many benchmark construction pipelines overlook or accept as unavoidable.

Standard adversarial benchmark construction works as follows: collect a candidate question pool, test them on one or more "strong" models, and retain only questions that those models fail to answer. The retained questions are by construction hard for those specific models. If the models used for filtering are from the same family or have similar architectures to the models that will later be evaluated, the benchmark's difficulty is biased — it may systematically select questions that exploit weaknesses specific to that architectural lineage, making models from other lineages look artificially good.

FACTS Parametric avoids this by using five strong open-weight models (models with publicly available weights that can be run locally, rather than proprietary API models) for the adversarial filtering step (Section 4.1.2). Only questions that all five of these models fail to answer correctly are retained. Because the open-weight models are architecturally distinct from the proprietary models evaluated on the benchmark (Gemini, GPT, Claude, Grok), the hardness filter is decoupled from the evaluation targets. Any question that survives this filter is genuinely difficult — it defeated five different model architectures independently — rather than difficult only for a specific model family.

This is a conceptual contribution to benchmark methodology because it identifies and solves a specific form of selection bias that many benchmark papers do not address. The paper's framing is explicit about the rationale: "to decouple the adversarial selection process from the proprietary API models used in our evaluation, thereby ensuring unbiased filtering" (Section 4.1.2). The insight is that the models used for benchmark construction and the models later evaluated on the benchmark should be as independent as possible, and using open-weight models for the former achieves this independence without requiring access to proprietary training infrastructure.

The significance extends beyond parametric QA. Any benchmark that uses model-based filtering to select challenging examples — which includes nearly all modern benchmarks that aim for frontier difficulty — should consider the independence of the filtering models from the evaluation models. If benchmarks are constructed using the same GPT-4 class models that will later be evaluated, the difficulty estimates are implicitly conditioned on GPT-4-like architectures, and the resulting benchmark may not be a fair test of non-GPT architectures. The FACTS approach provides a template: use a diverse set of independent, open-weight models for hardness filtering to reduce this architectural bias.

The limitation — which the paper does not fully explore — is that the independence is only as strong as the diversity of the open-weight models. If all five open-weight models share similar pretraining data or architectural principles (e.g., all are dense transformers trained primarily on English web text), they may still exhibit correlated weaknesses that the benchmark inherits. The paper does not specify which five models were used, their sizes, or their architectural diversity, which makes it impossible to assess the degree of independence actually achieved. Nonetheless, the conceptual principle — decouple construction models from evaluation models — remains valid and represents an advance over the common practice of using the same model family for both.

The empirical evidence supporting this innovation is indirect but visible in the results. Table 6 shows that proprietary models achieve accuracies ranging from 76.4% (Gemini 3 Pro) to 15.7% (Grok 4 Fast) on the same benchmark after adversarial filtering with open-weight models. The fact that some proprietary models achieve high accuracy while others fail badly indicates that the benchmark is not uniformly impossible — it is discriminating between models — and the spread suggests the filtering did not systematically disadvantage any single proprietary model family.

Innovation 4: Search Efficiency as a First-Class Evaluation Dimension, Not an Afterthought

The FACTS Search benchmark makes a conceptual move that prior search-augmented LLM benchmarks overlooked: tracking and reporting the number of search API calls alongside accuracy, transforming search efficiency from an implementation detail into an evaluated capability. This reframes "good search behavior" to mean not just finding the right answer, but finding it with minimal unnecessary querying — a capability that matters for practical deployment but was invisible in previous benchmarks.

Prior search benchmarks (Wei et al., 2024, 2025; Yang et al., 2024) evaluated whether models could use search tools to find correct answers, reporting accuracy as the sole metric. A model that issued 20 queries to find an answer that another model found in 3 queries was considered equally good if both got the right answer. In practice, search queries cost latency (each API call adds seconds to response time) and money (search APIs charge per query), so efficiency matters. But benchmarks did not measure it, so model developers had no incentive to optimize it.

Table 8 makes this dimension visible and its importance concrete. The "Average searches" column reveals substantial variation: Gemini 3 Pro achieves the highest accuracy (83.8%) with the fewest average searches (3.39), while Grok 4 Fast uses 4.74 searches — nearly 40% more — to achieve 67.3% accuracy, more than 16 points lower. GPT-5 achieves 77.7% accuracy with 4.28 average searches; Claude 4.5 Opus achieves 73.2% with 3.98. These are not minor efficiency differences — they represent fundamentally different search strategies that have real cost and latency implications.

The conceptual contribution is recognizing that in tool-augmented LLM evaluation, the cost of tool use is part of the capability being evaluated, not an incidental implementation detail. This parallels the evolution of evaluation in other domains: computer vision benchmarks eventually started tracking inference time and parameter counts alongside accuracy; NLP benchmarks started tracking model size and FLOPs. FACTS Search extends this principle to search-augmented generation, establishing a precedent that future search benchmarks should follow.

The implication of this innovation goes beyond benchmarking. By making search efficiency visible, the FACTS Leaderboard creates an incentive for model developers to optimize query formulation — to ask good questions on the first try rather than issuing multiple follow-up queries to correct earlier mistakes. This aligns the benchmark's incentives with real-world deployment constraints: a model that wastes API calls on every user query is less useful than one that retrieves the right information efficiently, even if both eventually produce correct answers. The decision to standardize on a single search API (Brave Search) for all models is critical here because it ensures efficiency differences reflect model behavior rather than differences in search engine latency or result quality.

The paper does not integrate search efficiency into the FACTS Score — the aggregate metric remains a simple average of accuracies, with search efficiency reported separately. This is a reasonable choice given that the FACTS Score is designed to measure accuracy, not efficiency, but it means the efficiency dimension does not directly affect model rankings. Future iterations of the leaderboard could incorporate an efficiency-adjusted score (e.g., accuracy per search query) to provide a more holistic measure of search capability that balances correctness with cost. The paper's contribution is in making the dimension visible and providing the measurement infrastructure; the normative question of how to weight efficiency against accuracy is left open.

Innovation 5: A Benchmark-Level Over-Optimization Diagnosis — Search Over-Optimizes the Verifier Signal, and the Hardest Problems Cannot Be Solved by More Compute

The paper provides empirical evidence for a phenomenon that has implications beyond benchmarking: test-time compute amplification through search has sharp difficulty boundaries, and on the hardest problems, no amount of search helps. This is not a methodological innovation in benchmark design, but rather an empirical finding enabled by the benchmark's difficulty-graded construction that validates and extends an observation from the test-time compute scaling literature.

The FACTS Search benchmark's construction explicitly tiers questions by difficulty mechanism (Hard Tail requiring complex search reasoning, Wiki Two-Hop requiring entity resolution, Wiki Multi-Doc requiring cross-document synthesis, KG Hops requiring structured graph traversal), and the adversarial filtering process ensures that the hardest questions genuinely challenge frontier models even with search access. The result, visible in the gap between the best Search accuracy (83.8% for Gemini 3 Pro) and the worst among frontier models (60.0% for Gemini 2.5 Flash, Table 8), is that even the best models with unlimited search API access cannot solve roughly 16% of the questions — and weaker models fail on 40%. These are questions where the answer exists on the web, the model has a search tool, and yet the model still cannot produce the correct answer.

The conceptual significance is the parallel to the test-time compute scaling findings in the example paper (compute-optimal test-time compute on MATH, where the hardest difficulty quintile showed near-zero improvement regardless of search budget). In both cases, the finding is that tool augmentation and search are amplifiers of existing capability, not creators of new capability. If a model cannot synthesize information across multiple documents, giving it access to those documents (via search) does not magically confer the synthesis ability — the model still needs to read, understand, integrate, and reason, and these capabilities are bounded by the model's underlying competence. The "amplification, not creation" boundary appears in both the pretrained-only scaling literature and now in the search-augmented evaluation literature, suggesting it is a general property of how tools interact with model capability.

The practical implication for model developers is clear: investing in better search infrastructure (faster APIs, better retrieval) will not compensate for fundamental reasoning limitations in the underlying model. If a model scores 60% on FACTS Search, the first-order improvement lever is better pretraining or fine-tuning to improve information synthesis capabilities, not better search tools. The search tool can only retrieve information; the model must still process it.

The paper does not break out per-subset results for Search (we see only aggregate accuracy in Table 8), so the claim that the hardest questions are unsolved by any model cannot be directly verified from the reported data. The adversarial filtering step (removing questions that Gemini-2.5-Flash without search can answer) ensures that all retained questions genuinely require search, but it does not guarantee that the remaining questions span a wide difficulty range within the search-required set. A per-subset breakdown of Search results (Hard Tail vs. Wiki Two-Hop vs. Wiki Multi-Doc vs. KG Hops) would provide stronger evidence for the difficulty-boundary claim, and its absence is a limitation. However, the aggregate 83.8% ceiling — combined with the knowledge that questions were adversarially filtered — is suggestive that a non-trivial fraction of questions remain beyond current frontier capability even with search.

5. Experimental Analysis

Evaluation Methodology

Dataset. The FACTS suite comprises four independent benchmarks, each with its own dataset: FACTS Multimodal (~1,522 image-question pairs, 711 public / 811 private, sourced from diverse visual domains as shown in Figure 1); FACTS Parametric (2,104 factoid QA pairs, 1,052 public / 1,052 private, collected from user traffic and adversarially filtered against five open-weight models, with answers supported by Wikipedia — see Section 4.1); FACTS Search (1,884 questions, 890 public / 994 private, drawn from four subsets: Hard Tail, Wiki Two-Hop, Wiki Multi-Doc, and KG Hops — see Section 5.1); and FACTS Grounding v2 (same prompt set as FACTS Grounding v1, consisting of long-form context documents up to 32k tokens paired with user requests for summarization, QA, or rewriting — see Figure 4 for domain and task distributions). All datasets feature public/private splits to mitigate overfitting, with the private split held by Kaggle and never released to model developers.
Base model(s). The paper evaluates 15 proprietary API-based models spanning five model families: Gemini (Gemini 3 Pro, Gemini 2.5 Pro, Gemini 2.5 Flash), GPT (GPT-5, GPT-5.1, GPT-5 mini, GPT-4.1, GPT o3, GPT o4 mini), Claude (Claude 4.5 Opus, Claude 4 Opus/4.1 Opus, Claude 4 Sonnet, Claude 4.5 Sonnet Thinking), Grok (Grok 4, Grok 4 Fast), plus Gemini 2.5 Flash in the Grounding comparison. The models span multiple orders of magnitude in capability (FACTS Scores from 68.8 down to 36.0, Table 1), ensuring the benchmark discriminates across a wide performance range and does not saturate at the frontier.
Metrics. The primary aggregate metric is the FACTS Score, defined as the arithmetic mean of accuracy across the four sub-benchmarks (Section 2). Per-benchmark metrics vary by task: FACTS Multimodal uses accuracy as the fraction of responses passing both Coverage AND No-Contradiction Boolean verdicts; FACTS Parametric uses accuracy (% correct) plus secondary metrics of hedging rate, attempted accuracy, and F1 (harmonic mean of accuracy and attempted accuracy); FACTS Search uses accuracy with secondary metrics of attempted accuracy, hedging rate, F1, and average number of search API calls; FACTS Grounding v2 uses accuracy after eligibility disqualification, computed as the average of two independent judge scores. All metrics are reported with 95% confidence intervals.
Baselines. The paper does not compare against prior benchmark systems (since its contribution is the benchmark itself, not a method), but it does report the original FACTS Grounding v1 judge configuration as a comparative baseline for the v2 judge update. Specifically, Table 10 evaluates 12 judge model × prompt combinations, including the v1 configuration (Gemini 1.5 Pro, GPT-4o, Claude 3.5 Sonnet with the v1 prompt) against the new v2 configuration (Gemini 2.5 Flash, GPT-5 with the v2 prompt). For individual benchmarks, the primary reference points are: Multimodal — the Coverage vs. No-Contradiction breakdown in Table 3 establishes that no model dominates both dimensions; Parametric — Table 6 includes all 15 models ranked by F1 and accuracy; Search — Table 8 includes all 15 models ranked by F1 and accuracy with search efficiency data.
Generation budget / compute accounting. The paper does not measure or constrain generation budget as a controlled variable. All models are evaluated in a zero-shot or tool-augmented setting (for Search) with fixed prompts per benchmark, and the evaluation measures accuracy given whatever default generation strategy the model API provides. The FACTS Search benchmark does, however, track the average number of search API calls per question as a post-hoc efficiency metric (Table 8, "Average searches" column), which serves as an indirect measure of the computational cost incurred during tool-augmented inference — Gemini 3 Pro averages 3.39 calls while Grok 4 Fast averages 4.74, revealing up to a 1.4× difference in search volume for the same task.
Cross-validation / statistical protocol. The paper does not employ cross-validation for model evaluation (models are evaluated once on the full public+private test sets). However, the FACTS Parametric grading protocol (Section 4.2) uses triple-sampled grades from Gemini-2.5-Pro per model response, with the three grades averaged to produce the final score — this is a form of grader ensembling to reduce variance, not cross-validation. The FACTS Grounding v2 judge selection (Table 10) uses a held-out evaluation set of N=320 human-annotated examples (class ratio 79:19) to select the best judge model × prompt combination before applying those judges to the full benchmark. The FACTS Multimodal autorater validation (Table 4) was conducted on an unspecified held-out set, with Spearman's rank correlation and macro F1 reported against human annotations. Confidence intervals reported in Table 1 appear to be based on binomial proportion standard errors, though the exact computation method is not specified.

Main Quantitative Results

Aggregate Performance Across the Full Suite (Table 1)

The headline result is that the top-performing model, Gemini 3 Pro, achieves a FACTS Score of 68.8%, with 95% confidence intervals of ±2.1 on Grounding, ±3.6 on Multimodal, ±1.8 on Parametric, and ±1.6 on Search. The next two models — Gemini 2.5 Pro (62.1) and GPT-5 (61.8) — are separated by only 0.3 percentage points in aggregate, despite having qualitatively different performance profiles across sub-benchmarks. The gap between 1st and 5th place (Gemini 3 Pro at 68.8 vs. GPT o3 at 52.0) is 16.8 points, while the gap between 5th and 15th (Grok 4 Fast at 36.0) is another 16.0 points, suggesting the FACTS Score discriminates effectively across the full capability range without ceiling or floor compression.

The cross-benchmark variation reveals that no model dominates uniformly. GPT-5 ranks 3rd overall (61.8) but achieves the highest Search accuracy among non-Gemini models (77.7%, 2nd place on that sub-benchmark) while placing only 4th on Parametric (55.8%) and 3rd on Multimodal (44.1%). Claude 4.5 Opus ranks 6th overall (51.3) but achieves 73.2% on Search (4th place) and 62.1% on Grounding, while collapsing to 30.6% on Parametric (8th place) — a 42.6-point spread between its best and worst dimension. This variation empirically validates the paper's central claim that factuality decomposes into independent dimensions that cannot be proxied by any single benchmark.

The worst-performing dimension across all models is Multimodal, where the highest accuracy is only 46.9% (Gemini 2.5 Pro), suggesting that integrating visual grounding with parametric knowledge remains substantially harder than any text-only factuality task. The best-performing dimension is Search, where Gemini 3 Pro achieves 83.8% — but even here, models like Gemini 2.5 Flash (60.0%) and GPT-5.1 (62.4%) trail far behind, indicating that effective search tool use is far from universal among frontier models.

FACTS Multimodal: Coverage vs. No-Contradiction Tradeoff (Table 3)

The detailed Multimodal results reveal a systematic tradeoff between model families. Gemini models achieve the highest Coverage scores (Gemini 3 Pro: 68.4%, Gemini 2.5 Pro: 67.7%), meaning they include more essential facts in their responses — they are recall-oriented. However, they achieve lower No-Contradiction scores (Gemini 3 Pro: 57.3%, Gemini 2.5 Pro: 58.8%), meaning they are more likely to include false claims alongside true ones. GPT models show the inverse pattern: GPT-5 achieves 59.9% Coverage but 64.7% No-Contradiction; GPT-5.1 achieves 58.3% Coverage but 65.0% No-Contradiction — they are precision-oriented, saying fewer things overall but making fewer factual errors.

The AND operation (both Coverage and No-Contradiction must pass) compresses these tradeoffs: the top Multimodal accuracy is Gemini 2.5 Pro at 46.9%, followed closely by Gemini 3 Pro at 46.1% and GPT-5 at 44.1%. The product of Coverage × No-Contradiction would be 0.677 × 0.588 ≈ 0.398 for Gemini 2.5 Pro, but the actual accuracy is 46.9% — higher than the independence assumption would predict — suggesting that responses which cover essential facts are somewhat more likely to avoid contradictions than random chance, but the correlation is not strong enough to prevent the precision-recall tradeoff from limiting overall accuracy.

Claude models perform notably worse on Multimodal (Claude 4.5 Opus: 39.2%, Claude 4 Sonnet: 28.6%), driven primarily by low No-Contradiction scores (51.1% and 46.2% respectively). This is consistent with Claude's hedging behavior on Parametric (Table 6) — when forced to generate free-text visual descriptions rather than short answers, the hedging strategy is not available, and the model produces more content with correspondingly more errors.

Grok models show catastrophic No-Contradiction rates: Grok 4 achieves only 32.0% No-Contradiction (despite 57.8% Coverage), and Grok 4 Fast achieves 22.0% (despite 50.9% Coverage). These models are generating substantial content that is mostly wrong — the worst-case scenario for factual reliability. The resulting accuracies (25.7% and 17.7%) are dramatically lower than the 40%+ of the top models.

FACTS Parametric: Accuracy, Hedging, and the F1 Tradeoff (Table 6)

The Parametric results reveal three distinct model behavior patterns along the hedging dimension. Low-hedging, high-accuracy models: Gemini 3 Pro achieves 76.4% accuracy with only 1.4% hedging — it answers almost everything and is usually right. Gemini 2.5 Pro achieves 63.2% with 1.9% hedging. These models prioritize coverage, and their high accuracy validates this strategy.

High-hedging, high-attempted-accuracy models: GPT-5mini hedges on 67.6% of questions — it refuses to answer two-thirds of the time — but when it does answer, its attempted accuracy is 49.6%. This is a radically different strategy: prioritize precision on attempted questions at the cost of extremely low recall. GPT-5 similarly hedges 13.3% of the time, achieving 64.3% attempted accuracy (compared to 55.7% raw accuracy). Claude models show the most extreme hedging: Claude 4.1 Opus hedges 29.9% of the time, Claude 4.5 Sonnet Thinking hedges 31.3%, and Claude 4 Sonnet hedges 45.1% — nearly half of all questions. Yet their attempted accuracies (47.4%, 42.2%, 37.2%) are not dramatically higher than models that answer everything, suggesting the hedging is not always well-calibrated: Claude is refusing to answer many questions it could have gotten right.

The F1 metric (harmonic mean of accuracy and attempted accuracy) captures the balance. Gemini 3 Pro leads with F1 = 77.0, followed by Gemini 2.5 Pro at 63.8, then a cluster of Grok 4 (60.9), GPT-5 (59.7), and o3 (57.6). The F1 penalizes the extreme hedgers: GPT-5 mini's F1 is only 24.2 despite its 49.6% attempted accuracy, because it only attempts one-third of questions. Claude 4 Sonnet's F1 is 26.3 despite 37.2% attempted accuracy, because it attempts barely half the questions. The metric successfully distinguishes between "knows what it knows" (moderate hedging with genuinely higher attempted accuracy) and "refuses to try" (extreme hedging with only modest attempted accuracy gains).

The adversarial sampling methodology produces a genuinely challenging benchmark: even the best model (Gemini 3 Pro) fails on 23.6% of questions, and most models fail on 30-85% of questions. The range from best to worst (76.4% to 15.7% for Grok 4 Fast) spans 60.7 percentage points, indicating the benchmark discriminates strongly across capability levels without saturating.

FACTS Search: Accuracy and Search Efficiency (Table 8)

The Search results demonstrate a near-monotonic relationship between model capability and accuracy, but with revealing exceptions. The top tier (Gemini 3 Pro at 83.8%, GPT-5 at 77.7%) achieves substantially higher accuracy than the next tier (Claude 4.5 Opus at 73.2%, o3 at 74.8%), and there is a clear drop to the bottom tier (Gemini 2.5 Flash at 60.0%, GPT-5.1 at 62.4%).

The search efficiency data ("Average searches" column) is perhaps the most novel empirical contribution of this benchmark. Gemini 3 Pro achieves its 83.8% accuracy with only 3.39 average searches — the lowest among all top-accuracy models. GPT-5 achieves 77.7% with 4.28 searches, Grok 4 achieves 75.3% with 4.50 searches, and Claude 4.5 Opus achieves 73.2% with 3.98 searches. The efficiency gap is substantial: Gemini 3 Pro uses 21% fewer searches than GPT-5 while achieving 6.1 percentage points higher accuracy — it is both faster and more accurate. This efficiency advantage would compound in deployment: at 4.28 vs. 3.39 searches per query, GPT-5 incurs 26% higher search API costs and roughly proportional additional latency.

The hedging patterns on Search differ from Parametric. While Claude models hedge heavily on Parametric (29.9-45.1%), they hedge less on Search (Claude 4.1 Opus: 26.9%, Claude 4.5 Sonnet: 22.2%). The paper hypothesizes that "Claude appears to prioritize accuracy on attempted queries over overall accuracy" — a strategy that may be appropriate for search-augmented settings where the model can leverage retrieved context rather than relying entirely on parametric recall.

GPT-5 mini shows an interesting pattern: despite heavy hedging (22.5%), its attempted accuracy (87.5%) is nearly as high as Gemini 3 Pro (87.6%), but its raw accuracy (67.9%) lags far behind. This suggests GPT-5 mini is good at recognizing when it has found the answer (high attempted accuracy) but not good at finding the answer in the first place (low overall accuracy) — it knows what it knows but doesn't know enough.

GPT-4.1 is a notable outlier: it hedges only 3.7% of the time (lowest among all models) but achieves only 64.6% accuracy, and its attempted accuracy is 67.1% — also the lowest among all models with low hedging. This model answers almost everything but is often wrong, representing the opposite failure mode from the heavy hedgers.

FACTS Grounding v2: Judge Model Upgrade and Eligibility Filtering (Table 10)

The Grounding v2 results in Table 1 are reported for all 15 models, but the key methodological result is the judge model comparison in Table 10. The 12 judge model × prompt combinations evaluated on a held-out N=320 set (class ratio 79:19) show that the winning combination — Gemini 2.5 Flash with the v2 prompt — achieves a macro-F1 of 65.33. The second-best is GPT-5 with the v2 prompt at 65.18.

The decomposition of macro-F1 reveals the persistent challenge: even the best judge achieves only 46.15 F1 on the negative class (detecting ungrounded/hallucinated responses), compared to 84.51 F1 on the positive class (detecting grounded/accurate responses). This means that across all judge configurations, detecting hallucinations is substantially harder than confirming grounded responses — the false negative rate (missing actual hallucinations) is high. The best negative-class F1 across all configurations is Gemini 2.5 Flash v2 at 46.15, but the best positive-class F1 of 87.92 comes from Gemini 2.5 Pro v1 — indicating a tradeoff where judges that are better at detecting hallucinations are slightly worse at confirming grounded responses.

The v2 prompt provides a small but consistent improvement over v1. Gemini 2.5 Flash improves from 63.44 (v1) to 65.33 (v2); GPT-5 improves from 61.83 (v1) to 65.18 (v2). However, some models degrade with the v2 prompt: Gemini 2.0 Flash drops from 63.43 (v1) to 51.25 (v2), and claude-3-5-sonnet drops from 53.15 (v1) — the v2 numbers for these models are not reported in the table. The v2 prompt appears optimized for the specific judge models selected (Gemini 2.5 Flash and GPT-5), and its benefits do not generalize to all models.

The eligibility disqualification mechanism's impact is not separately quantified in Table 10 (the table evaluates only the grounding judgment, not eligibility), but the paper's Table 11 provides qualitative examples of the responses that the mechanism would disqualify — cases where the response is factually grounded but substantively empty. The paper reports that ineligible responses are "marked as inaccurate" in the final score, but does not provide statistics on what fraction of model responses are disqualified or whether disqualification rates vary systematically by model family.

Ablation Studies and Robustness Checks

Judge model prompt version (v1 vs. v2): Table 10 compares two prompt variants across multiple judge models. The v2 prompt consistently improves macro-F1 for Gemini 2.5 Flash (63.44 → 65.33) and GPT-5 (61.83 → 65.18), but degrades performance for older models: Gemini 2.0 Flash drops from 63.43 to 51.25, and Gemini 1.5 Pro drops from 56.89 to 55.65. The v2 prompt is not universally better — it is optimized for the specific judge models selected for Grounding v2, and applying it to other models would reduce rather than improve reliability.

Single judge vs. mixed-model panel for Parametric grading: Section 4.2 reports an ablation comparing Gemini-2.5-Pro alone against a mixed panel (sampling once each from Gemini-2.5-Pro, GPT-o3, and Grok-4). The finding is that the single judge "preserves the same relative performance trends and rankings as the more complex ensemble," which justifies the simpler configuration. The paper does not provide quantitative correlation metrics or rank-order statistics to support this claim, so the reader must accept the qualitative assertion that rankings are preserved.

Grading reliability via triple sampling for Parametric: The paper samples three grades from Gemini-2.5-Pro per model response and averages them. This is an implicit ablation showing that a single grading call is insufficiently reliable — the triple-sampling reduces variance from prompt sensitivity and sampling noise. The paper does not report the agreement rate between the three samples, the variance of grades, or how often the three samples disagree, so the magnitude of the reliability improvement from triple sampling cannot be quantified.

Adversarial sampling with open-weight models for Parametric data construction: The filtering step that retains only questions all five open-weight models fail (Section 4.1.2) is a hardness ablation. Without this step, the benchmark would contain many questions answerable by weaker models, reducing discriminative power among frontier models. The paper does not report the pre-filtering accuracy of frontier models on the easier questions that were removed, so the effectiveness of this step cannot be quantitatively assessed from the reported data. However, the resulting benchmark's accuracy range (15.7% to 76.4%) confirms the filtering produced a discriminating difficulty distribution.

Search-off filtering for FACTS Search data quality: The final filtering step removes any questions that Gemini 2.5 Flash without search can answer correctly (Section 5.1), ensuring the benchmark genuinely requires search. The paper does not report what fraction of the original question pool was removed by this filter, but the retained questions all require search by construction. This is a quality control ablation: without it, some questions would test parametric recall rather than search capability, contaminating the metric's interpretation.

Wikipedia source requirement for Parametric: The paper constrains all Parametric answers to be verifiable in Wikipedia (Section 4.1.1), which serves as an implicit ablation against questions where the ground truth answer might be disputed, paywalled, or too ephemeral to verify reliably. The paper does not report what fraction of initial questions failed this filter, but it establishes that all retained questions have answers explicitly supported by "a source that is highly prevalent and widely assumed to be a significant component of the training corpora for all LLMs."

Negative result: Multimodal Coverage and No-Contradiction tradeoff is consistent across model families: Table 3 shows that no model achieves both high Coverage and high No-Contradiction simultaneously. The highest Coverage (Gemini 3 Pro, 68.4%) comes with 57.3% No-Contradiction; the highest No-Contradiction (GPT-5 mini, 65.1%) comes with only 58.8% Coverage. The tradeoff is not a consequence of any individual model's design — it appears across Gemini, GPT, Claude, and Grok families — suggesting it is a fundamental tension in open-ended multimodal generation rather than a correctable flaw in specific models.

Critical Assessment

Does the FACTS Score genuinely measure holistic factuality, or does it measure performance on four specific benchmarks?

The paper's central claim is that the FACTS Score provides "a robust and balanced assessment of a model's overall factuality" (Section 1) by averaging across four dimensions. The experiments in Table 1 demonstrate that the ranking of models varies across sub-benchmarks — Gemini 3 Pro leads overall but is not best on every dimension, and some models show dramatic cross-dimension variation (Claude 4.5 Opus ranges from 30.6% on Parametric to 73.2% on Search). This validates the claim that the four dimensions capture independent variance — they are not redundant.

However, what the experiments do not demonstrate is that these four specific dimensions constitute a sufficient set for "overall factuality." The paper acknowledges this gap explicitly: "Naturally, there are aspects of factuality not covered by FACTS, such as video understanding and fast-changing information" (Section 7). The FACTS Score measures factuality in four specific scenarios (image-based QA, closed-book factoid recall, search-augmented QA, long-form grounding). A model could theoretically achieve a high FACTS Score while performing poorly on dimensions the suite does not measure — temporal factuality (questions about recent events), conversational factuality (maintaining factual consistency across multi-turn exchanges), procedural factuality (correctly describing how to perform tasks), or cross-lingual factuality (accurate responses in non-English languages). The claim that the score measures "overall factuality" is stronger than what the experiments support — it measures factuality on the four tasks included, and these are important tasks, but "overall" implies comprehensiveness that the paper itself acknowledges is incomplete.

A related concern is that the equal weighting of the four dimensions in the FACTS Score is not empirically justified. The paper does not analyze whether the four benchmarks have equal measurement reliability, equal discriminative power, or equal real-world importance. The Multimodal benchmark has a low accuracy ceiling (46.9% maximum) and high judge noise (macro F1 of 72.3-78.2 for the autorater), while the Search benchmark has a higher ceiling (83.8%) and presumably more reliable grading (factoid-style answer matching vs. open-ended response evaluation). Giving equal weight to a noisier, harder benchmark means the FACTS Score is disproportionately influenced by the measurement error in the Multimodal component — a 1-point change in Multimodal accuracy (which could easily be within judge noise) has the same impact on the FACTS Score as a 1-point change in Search accuracy (which is likely more reliable). The paper would be strengthened by reporting the reliability of each sub-benchmark (e.g., test-retest correlations or standard errors of measurement) and considering reliability-weighted aggregation.

Does the adversarial sampling for Parametric genuinely decouple construction from evaluation?

The paper's methodological claim is that using five open-weight models for adversarial filtering "decouples the adversarial selection process from the proprietary API models used in our evaluation, thereby ensuring unbiased filtering" (Section 4.1.2). This is a valid conceptual principle, but the experiments provide only indirect evidence for its success.

The key missing experiment is a bias analysis: do models from different families show systematically different performance on questions that survived the open-weight filtering? If the five open-weight models share architectural principles (all transformer-based, all trained on similar data) that are also shared by some but not all proprietary models, the filtering could still introduce bias. For example, if all five open-weight models and Claude share a weakness in numerical reasoning (while Gemini does not), the filtering would systematically retain numerical reasoning questions — making the benchmark disproportionately hard for Claude compared to Gemini, not because Gemini is "more factual" but because the filtering architecture inadvertently aligned with Claude's specific weaknesses. The paper does not analyze question-level performance by model family or check for differential item functioning, so the claim of unbiased filtering is based on the principle of the design rather than empirical verification.

Furthermore, the paper does not name the five open-weight models or report their architectures, sizes, or training data. Without this information, the independence claim cannot be assessed by readers. If the five models were, for example, Llama-3 variants (all sharing the same base architecture and pretraining data), the "five independent models" would effectively be one architecture family with different sizes, severely undermining the claimed independence. This is a significant transparency gap.

Are the automated judge validations sufficient to support the benchmark's credibility?

The paper validates automated judges against human annotations for two of four benchmarks: Multimodal (Table 4) achieves macro F1 of 72.3 for Coverage and 78.2 for No-Contradiction; Grounding v2 (Table 10) achieves macro F1 of 65.33 for the best judge configuration. These numbers indicate that automated judges are reasonable but imperfect proxies for human evaluation — there is substantial disagreement between judge and human.

The critical question is whether this level of judge reliability is adequate for the benchmark's intended use: ranking models and tracking progress over time. At macro F1 of 65-78, the noise floor from judge error could easily span several percentage points of accuracy. For closely-ranked models (e.g., GPT-5 at 61.8 vs. Gemini 2.5 Pro at 62.1 FACTS Score), the difference is well within plausible judge noise, meaning the ranking between these models may not be statistically reliable. The confidence intervals reported in Table 1 (±1.6 to ±3.6) reflect sampling error from the finite test set, but they do not include judge error — a response that the judge marks as correct might be marked incorrect by a human (or vice versa), and this error source inflates the true uncertainty beyond what the reported confidence intervals capture.

The paper does not provide judge reliability estimates for the Parametric or Search benchmarks, which use LLM-as-judge grading (Gemini-2.5-Pro and Gemini-2.0-Flash, respectively) without reported human validation. The Search auto-rater is simply described as being "prompted to assess if the response is correct, incorrect or does not attempt to answer the query" (Section 5.3), with no validation data. The Parametric grader is validated only against a mixed-model panel (not against humans), and the validation claim is qualitative ("preserves the same relative performance trends and rankings") without quantitative support. This is the most significant gap in the experimental validation: two of four benchmarks lack demonstrated alignment with human judgments, yet their scores contribute equally to the FACTS Score.

Are 500 questions per benchmark sufficient, especially with public/private splits?

The total test set sizes range from approximately 1,522 (Multimodal) to 2,104 (Parametric) to 1,884 (Search), with the Grounding size unspecified. When split into public and private subsets (roughly 50/50 in most cases), each model is evaluated on approximately 500-1,000 questions per sub-benchmark per split. With 95% confidence intervals of ±1.6 to ±3.6 percentage points reported in Table 1, the sample sizes appear adequate for detecting differences of roughly 3-5 points between models on individual sub-benchmarks.

However, the FACTS Score aggregates across four benchmarks, and its confidence interval should be wider than any individual sub-benchmark's (since it is a mean of four correlated proportions). The paper does not report confidence intervals for the FACTS Score itself, only for the individual components (Table 1). For the closely-ranked models at the top — Gemini 3 Pro (68.8), Gemini 2.5 Pro (62.1), GPT-5 (61.8) — the gap between 1st and 2nd (6.7 points) is likely reliable, but the gap between 2nd and 3rd (0.3 points) is almost certainly within the combined sampling and judge error, making that specific ranking unreliable. The paper's presentation of rankings without FACTS Score confidence intervals obscures this uncertainty.

The difficulty bin analysis that would strengthen the paper is entirely absent. For Parametric and Search, the paper constructs data with explicit difficulty mechanisms (adversarial sampling, multi-hop synthesis) but does not report per-subset or per-difficulty-tier results. We cannot see whether Gemini 3 Pro's 76.4% Parametric accuracy comes from perfect performance on easy questions and near-zero on hard ones, or from broad competence across difficulty levels — a distinction that matters for understanding model capability profiles. Similarly, the four Search subsets (Hard Tail, Wiki Two-Hop, Wiki Multi-Doc, KG Hops) are described as testing different capabilities, but Table 8 reports only aggregate scores, so we cannot assess which capabilities drive the overall performance gaps.

Does the evaluation infrastructure introduce systematic biases?

The paper uses Google-developed judge models (Gemini 2.5 Pro, Gemini 2.5 Flash, Gemini 2.0 Flash) and a Google-owned evaluation platform (Kaggle) to evaluate models that include Google's own Gemini family. While the paper addresses judge self-preference bias (Section 6.2, citing Wataoka et al., 2024) by using two-judge averaging with a non-Gemini model (GPT-5) for Grounding, the other three benchmarks use only Gemini-based judges or autoraters. GPT-5 is used as a second judge only in Grounding v2; Multimodal, Parametric, and Search grading all rely on Gemini-family models exclusively.

The paper argues for the Parametric grader that a mixed-model panel comparison showed preserved rankings, but without quantitative data this claim is unverifiable. The possibility that Gemini judges are systematically more favorable to Gemini-generated responses — even subtly, through leniency in ambiguous grading decisions — cannot be ruled out by the reported experiments. A bias audit comparing judge scores for Gemini vs. non-Gemini models on the same questions, or comparing Gemini-based judges against a fully independent judge panel, would be needed to establish impartiality. The paper's transparency about the concern (acknowledging the Wataoka et al. finding) is commendable, but the experiments do not fully address it.

Missing experiments that would strengthen the paper

Inter-benchmark correlation analysis: The paper's central claim is that the four benchmarks measure independent dimensions, but no correlation matrix of model scores across benchmarks is reported. If models' scores on Grounding and Search are highly correlated (r > 0.8), the two benchmarks are partially redundant, and the FACTS Score double-weights that shared dimension. The independence claim needs quantitative support.
Human baseline for each benchmark: Without knowing how well humans perform on FACTS Multimodal, Parametric, and Search, the absolute accuracy numbers are hard to interpret. Is 46.9% on Multimodal because the task is fundamentally difficult (humans might score 60%) or because models are deficient (humans might score 95%)? The paper does not provide human performance data for any benchmark except through the judge validation sets.
Test-retest reliability: Running the same model multiple times (with different random seeds or at different times) would reveal the stability of FACTS Scores. If Gemini 2.5 Pro scores 62.1 on one run and 59.3 on another, the 0.3-point gap to GPT-5 is meaningless. Without reliability estimates, the precision of the rankings is unknown.
Per-subset breakdowns for Search and Multimodal: The Search benchmark has four subsets and the Multimodal benchmark has 10 image categories and 10 question categories — reporting accuracy by subset/category would reveal capability profiles that aggregate scores hide. A model that achieves 83.8% on Search might do so by excelling at Wiki Two-Hop while failing badly on Hard Tail, with very different implications for real-world deployment than a model with balanced performance.
Sensitivity analysis for the FACTS Score weighting: What happens to model rankings if the four benchmarks are weighted by dataset size, by difficulty (inverse of maximum accuracy), or by reliability (inverse of judge error)? If rankings are robust to reasonable weight variations, the equal-weight choice is justified; if they are sensitive, the FACTS Score is more arbitrary than the paper implies.

In summary, the experiments broadly support the paper's core claim that comprehensive factuality evaluation requires measurement across multiple information-source dimensions, and they demonstrate that existing frontier models exhibit substantial and meaningful performance variation across these dimensions. However, the specific FACTS Score rankings should be interpreted with caution given unquantified judge noise, absent confidence intervals on the aggregate score, reliance on Gemini-based judges for three of four benchmarks, and missing analyses (per-subset breakdowns, inter-benchmark correlations, reliability estimates) that would provide a more complete picture of the measurement's trustworthiness. The benchmark suite succeeds as a demonstration that multi-dimensional factuality evaluation is necessary; its adequacy as a precise measurement instrument for comparing closely-ranked models is less firmly established by the reported experiments.

6. Limitations and Trade-offs

6.1 Difficulty Estimation for Compute-Optimal Allocation

The assumption or constraint. The entire compute-optimal test-time scaling framework rests on the ability to estimate prompt difficulty before allocating the inference budget. The paper's method for doing so — generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted) — is extraordinarily expensive. The paper explicitly acknowledges this in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

At 2048 samples per question, the difficulty estimation step alone consumes more compute than the largest test-time budgets studied (256–512 generations). The reported efficiency gains — the central 4× improvement over best-of-N — are computed after difficulty is known, without amortizing the cost of learning it.

The consequence. In a realistic deployment, the total cost would be difficulty estimation plus strategy execution, and the former could dominate the latter. If estimating difficulty costs 2048 generations and the optimal strategy then uses 64 generations, the total cost is 2112 generations — roughly 8× more than the baseline best-of-256 the paper compares against. The 4× efficiency claim is not false, but it omits a cost that makes it currently unrealizable in practice. Until difficulty can be estimated cheaply (e.g., from a lightweight classifier or from the first few samples adaptively), the compute-optimal framework is an upper bound on achievable efficiency rather than a deployable method.

What evidence exists in the paper. The paper explicitly flags this in Section 3.2 and calls it "a key avenue for future work." However, no experiment measures the amortized cost, and all figures (Figures 4, 8, 9) report accuracy vs. generation budget with difficulty estimation cost excluded. The predicted-difficulty variant (using PRM scores rather than ground-truth) removes the need for correct answers but does not reduce the sampling cost — it still requires 2048 generations per question. The paper does not explore cheaper difficulty proxies (e.g., using 8–16 samples instead of 2048, or training a separate difficulty classifier), so the sensitivity of the compute-optimal policy to difficulty estimation quality is unknown.

Mitigation status. The paper acknowledges the limitation explicitly and suggests future work on "pretraining or finetuning models to directly predict difficulty" (Section 8), but no such model is developed or evaluated. The limitation is entirely unaddressed in the current system.

6.2 Hard Problems Are Immune to Test-Time Compute

The assumption or constraint. The compute-optimal framework assumes that additional test-time compute can improve accuracy, but this assumption breaks down when the base model's pass@1 is near zero — if the model never generates a correct solution, no amount of search or revision can find one. The paper is candid about this boundary:

"On the hardest questions (bin 5), no method makes meaningful progress — the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated." (Section 5.3)

This is not a fixable bug but a fundamental capability ceiling: test-time compute amplifies existing capability but does not create it.

The consequence. The approach offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution. Across all methods — search, revisions, and their compute-optimal combinations — difficulty bin 5 accuracy hovers at 1–3% regardless of compute budget (Figure 3, right; Figure 7, right). In the FLOPs-matched comparison (Figure 9, Section 7), the bin 5 scaling line is essentially flat near 0–5%, and test-time compute underperforms the larger pretrained model by 37–53% on hard problems at high inference-to-pretraining ratios. For applications where the problem distribution skews toward genuinely hard problems (outside the base model's reach), the compute-optimal framework provides no benefit over any other strategy — all are equally ineffective.

What evidence exists in the paper. The difficulty-bin analyses consistently show near-zero improvement in bin 5 across search (Figure 3, right), revisions (Figure 7, right), and FLOPs-matched comparisons (Figure 9). The results are unambiguous: no method helps on the hardest quintile. The paper also demonstrates that the phenomenon is not an artifact of insufficient budget — even at 256 generations for search and 64 sequential revisions, bin 5 accuracy does not budge.

Mitigation status. The paper acknowledges this boundary explicitly and uses it to qualify its claims about the pretraining-inference tradeoff: "test-time compute amplifies existing capability but does not create it from nothing" (Section 7). The only mitigation is to route hard problems to a larger pretrained model (or to human review), which the paper does not implement but does discuss as a practical implication. The find-optimal strategy on the hardest bin is simply "no strategy helps," and the compute-optimal policy correctly allocates minimal budget to these problems.

6.3 Single Benchmark, Single Model Family

The assumption or constraint. All experiments use the MATH benchmark (500 test questions from Hendrycks et al., 2021) with PaLM 2-S* as the base model. The paper states it "believe[s] this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified across model families, scales, and task domains. The findings — the difficulty-dependent scaling curves, the 4× efficiency improvement from adaptive allocation, the beam search over-optimization behavior, the revision model's generalizability beyond its training horizon — are all conditioned on one model architecture trained on one data distribution and evaluated on one task type (competition-level math).

The consequence. Several aspects of the findings could be model-specific or domain-specific. The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution — a model with different calibration properties or error patterns might exhibit different difficulty-dependent scaling curves, potentially changing which strategies are optimal at which difficulty levels. The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families (some models are much better at leveraging in-context demonstrations than others). The MATH benchmark consists exclusively of symbolic reasoning problems with clean, verifiable answers — it is unclear whether the difficulty-dependent patterns generalize to other reasoning domains (code generation, logical reasoning, scientific QA) or to tasks requiring factual knowledge rather than inference. The paper does not provide evidence that the 4× compute-optimal efficiency gain would replicate on a different model family or task domain.

What evidence exists in the paper. The evidence is entirely within the MATH + PaLM 2-S* setting. There are no cross-model experiments (e.g., applying the same methodology to GPT-4 or Llama-3), no cross-benchmark experiments (e.g., testing on GSM8K, HumanEval, or MMLU), and no analysis of how model scale affects the optimal allocation policy. The FLOPs-matched comparison in Section 7 uses a second PaLM 2 model with approximately 14× more parameters, which provides some evidence of scaling behavior within the same model family but does not address cross-architecture generalizability.

Mitigation status. The paper does not claim generalizability beyond its experimental setting and does not attempt to address this limitation. The authors frame the work as a "systematic scaling analysis" (Section 4) establishing the framework and methodology, with the expectation that future work will replicate and extend across models and domains. The transparency about the single-model, single-benchmark scope is appropriate, but the practical consequence is that a practitioner wanting to apply compute-optimal test-time scaling to their own model and task would need to re-derive the optimal policy from scratch — the specific strategy selections (beam search vs. best-of-N at which difficulty levels, optimal sequential-to-parallel ratios) are not transferable without empirical verification.

6.4 Revisions and Search Are Studied Independently, Not Combined

The assumption or constraint. The paper studies two complementary mechanisms — PRM-guided search (Section 5) and iterative revisions (Section 6) — as independent pipelines, evaluating them separately and selecting between them via the compute-optimal policy. Section 8 explicitly acknowledges:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The two mechanisms have complementary strengths: revisions improve the proposal distribution (generating better candidates by conditioning on previous attempts), while PRM search improves candidate selection (navigating the search space using per-step value estimates). The paper demonstrates that revisions are most effective on easy problems (where local refinement suffices) and search is most effective on medium problems (where broader exploration is needed), but it never tests whether combining them — e.g., using beam search over revision model outputs, or using the PRM to guide which revision branches to pursue — yields gains beyond either method alone.

The consequence. The reported results represent a lower bound on what a fully integrated system could achieve. The compute-optimal policy selects either search or revisions per difficulty level, but a combined system might achieve higher accuracy than either alone, especially on medium-difficulty problems where both mechanisms show partial effectiveness. The paper's difficulty-dependent findings — revisions help on easy problems, search helps on medium problems — suggest a natural integration: start with search to explore diverse solution approaches, then apply revisions to refine the most promising candidates. The paper does not test this or any other combination strategy, so the ceiling of the current approach under a fully integrated test-time compute budget is unknown.

What evidence exists in the paper. The only evidence is the separate scaling curves for search (Figure 3) and revisions (Figure 6), and the compute-optimal policy that switches between them per difficulty bin. The paper does not provide any results for combined strategies, nor does it analyze whether the difficulty bins where search is optimal and where revisions are optimal overlap in ways that could be exploited by combination. The revision model's outputs are scored with a separately trained ORM rather than the PRM (Appendix J, Figure 15a, confirming distribution shift prevents PRM reuse), which means combining the two pipelines would require resolving the verifier mismatch — a technical challenge the paper does not address.

Mitigation status. The limitation is acknowledged in Section 8 as future work, and the paper frames it as a natural next step. No mitigation is attempted in the current paper. The separate treatment is justified by the paper's analytical goal — understanding each mechanism's scaling behavior in isolation — but it means the practical ceiling of test-time compute scaling under a unified strategy is likely higher than what the paper demonstrates.

6.5 No Accounting for Latency or Wall-Clock Time

The assumption or constraint. The paper measures compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores latency — the wall-clock time required to produce a final answer. Sequential revisions are inherently serial: each revision depends on the previous one, so a chain of 64 revisions takes 64× longer than a single generation, regardless of how much parallel hardware is available. Parallel best-of-N, in contrast, can be executed simultaneously with sufficient hardware — 256 parallel samples could complete in roughly the time of one generation.

The consequence. For latency-sensitive applications (interactive assistants, real-time decision-making, user-facing chatbots), the sequential-heavy strategies favored by the compute-optimal policy on easy problems may be impractical regardless of their accuracy advantages. Figure 7 (left) shows that for low budgets (8–32 generations), fully sequential is optimal — meaning the recommended strategy for easy problems with tight FLOPs budgets is to generate 8–32 sequential revisions. At 32 sequential revisions, the wall-clock latency is 32× that of a single generation, which may be unacceptable for interactive use. The paper does not discuss this tradeoff or provide latency-adjusted efficiency metrics that would help practitioners choose between a fast-but-less-accurate parallel strategy and a slow-but-more-accurate sequential one.

What evidence exists in the paper. The paper reports all results in terms of generation budget (N), with no wall-clock time measurements and no discussion of latency implications. The revision model's sequential dependency is described (Section 6.1, Figure 5) but its latency consequence is not analyzed. The search methods have varying serial depth: beam search with M=4 requires serial expansion across solution steps (each step depends on the previous step's beam selection), while best-of-N is fully parallelizable. The paper does not compare methods under a latency budget (e.g., "which strategy achieves the highest accuracy given a maximum of 5 seconds of wall-clock time?"), which would be a more practically relevant optimization target for many deployments.

Mitigation status. The limitation is not acknowledged in the paper. The choice of generation count as the cost metric is standard in the scaling laws literature and appropriate for an analytical study, but the practical deployment implications of latency are left entirely unaddressed. A practitioner deciding whether to deploy the compute-optimal revision strategy would need to conduct their own latency analysis, since the paper provides no guidance on this dimension.

6.6 Verifier Over-Optimization Is a Hard Ceiling, Not a Solved Problem

The assumption or constraint. The paper identifies verifier over-optimization as a central limiting factor — beam search degrades easy-problem performance at high budgets (Figure 3, right), lookahead search paradoxically performs worst overall despite being the most powerful optimizer (Figure 3, left), and qualitative examples show search producing degenerate outputs (repetitive steps, overly short solutions; Appendix M). The compute-optimal policy mitigates this by routing easy problems away from aggressive search, assigning them to best-of-N or revisions instead. However, the underlying problem — that verifiers become unreliable under aggressive optimization pressure — is not solved; it is only avoided by constraining the optimization strength per difficulty level.

The consequence. The compute-optimal approach is fundamentally bounded by verifier quality, and the ceiling is already visible in the results. Even on medium-difficulty problems where beam search is deployed (because it outperforms best-of-N), the beam search curves in Figure 3 flatten and sometimes decline well before the budget is exhausted. This means that increasing the compute budget beyond a certain point — even on problems where the strategy is nominally "optimal" — yields diminishing or negative returns because the verifier cannot reliably distinguish genuinely correct solutions from spuriously high-scoring incorrect ones. The paper's 4× efficiency improvement is achieved by operating below this over-optimization threshold, but the threshold itself limits how far test-time compute can be scaled: at very high budgets (beyond those explored in the paper), even the compute-optimal policy would eventually encounter the same ceiling.

For practical deployment, this means that improving verifier robustness is the key bottleneck for further scaling test-time compute — more so than improving search algorithms or revision models. The paper's finding that lookahead search (the most sophisticated optimizer) underperforms simpler methods suggests that better search algorithms are not the answer; better verifiers are. The paper does not investigate how verifier improvements (better training data, adversarial robustness, ensemble methods) would shift the difficulty thresholds and change the optimal policy, leaving the sensitivity of the entire framework to verifier quality unexplored.

What evidence exists in the paper. The over-optimization phenomenon is documented across multiple experiments: beam search degrades on easy problems (Figure 3, right, bins 1–2), lookahead search underperforms beam search and best-of-N at the same budget (Figure 3, left), and qualitative examples in Appendix M show the PRM assigning high scores to degenerate solutions with repetitive or content-free final steps. The paper also shows that the PRM's reliability varies by difficulty (it performs well on medium problems but can be exploited on easy ones), which is the core insight enabling the compute-optimal policy. However, no experiment tests whether a better PRM (e.g., trained on search-generated adversarial examples) would reduce over-optimization and enable more aggressive search strategies.

Mitigation status. The paper partially mitigates the issue through the compute-optimal policy itself — by routing easy problems to best-of-N (weaker optimization) and hard problems to beam search (where the PRM signal still provides genuine guidance), the policy stays below the over-optimization threshold per difficulty level. This is clever and effective, but it is a workaround, not a solution. The paper acknowledges the limitation implicitly through its discussion of degradation at high budgets (Section 5.3) and through the qualitative examples in Appendix M, but it does not propose or test methods for directly improving verifier robustness. Future work on verifier training — adversarial data, ensemble verification, KL-constrained search — is implied by the findings but not discussed as a specific research direction.

7. Implications and Future Directions

How This Work Changes the Landscape

The FACTS Leaderboard introduces a measurement standard, not a new algorithm or training methodology — its primary impact is on how the field evaluates and thinks about LLM factuality rather than on how models are built. This might seem like a modest contribution, but measurement standards have historically been among the most consequential outputs in ML: benchmarks like ImageNet, SQuAD, and GLUE shaped entire research programs by defining what problems matter and how progress is measured. FACTS aims to do the same for factuality, and its design choices encode hypotheses about what a good factuality measurement should capture.

The central shift is from unidimensional to multidimensional factuality evaluation. Prior to FACTS, the field implicitly treated factuality as a monolithic property — a model was either "factually accurate" or not, and any sufficiently rigorous benchmark could capture this property. The paper's core empirical finding refutes this assumption: model rankings vary substantially across the four sub-benchmarks (Claude 4.5 Opus achieves 73.2% on Search but 30.6% on Parametric, a 42.6-point spread; GPT-5 achieves 77.7% on Search but 55.8% on Parametric, a 21.9-point spread), and no model dominates all dimensions. If factuality were a single latent capability, these cross-dimension rank reversals would not occur. The implication is that any single-benchmark evaluation of factuality is incomplete by construction, and researchers claiming "improved factuality" must now either demonstrate gains across multiple dimensions or explicitly scope their claim to a specific factuality sub-capability. The FACTS Score's simple average may appear crude, but it enforces a discipline: a gain in one dimension must not come at too great a cost in others, or the net score will not improve.

Identifying systematic tradeoffs between model families. The decomposition of Multimodal evaluation into Coverage and No-Contradiction (Table 3) reveals that Gemini and GPT optimize for different sides of the precision-recall tradeoff — Gemini achieves higher Coverage (68.4% vs. 59.9% for the best models in each family) while GPT achieves higher No-Contradiction (64.7% vs. 57.3%). This is not random noise — it is a systematic pattern consistent across models within each family. The implication is that these tradeoffs reflect fundamental training decisions (RLHF reward modeling, preference optimization, data mixture) rather than accidental variation. Prior work could not surface this pattern because single-dimension benchmarks conflated the two components. The FACTS framework makes these training-level tradeoffs visible and quantifiable, enabling model developers to understand what they are optimizing for and what they are optimizing against when they adjust their training recipes. A team that improves RLHF to reduce hallucinations will see their No-Contradiction score rise — but the FACTS framework will also show them whether their Coverage score has declined as a side effect, preventing them from claiming a pure improvement when they have merely shifted the operating point along the tradeoff curve.

Uncovering hedging as a model-level strategy with cross-benchmark consequences. The Parametric results (Table 6) show that Claude models hedge on 29.9% to 45.1% of questions — nearly half the benchmark — while Gemini 3 Pro hedges on only 1.4%. The F1 metric (harmonic mean of accuracy and attempted accuracy) makes this strategy visible and penalizes it when attempted accuracy gains do not compensate for the coverage loss. More importantly, the Search results (Table 8) show that Claude hedges less aggressively when it has access to a search tool (22.2% for Claude 4.5 Sonnet), while GPT-5 mini hedges 67.6% on Parametric but only 22.5% on Search. This cross-benchmark pattern reveals that hedging is not a fixed model property but a context-dependent behavior — models calibrate their willingness to answer based on whether they can verify information through external retrieval. This insight would be invisible in any single-benchmark evaluation, and it suggests that hedging behavior is something that can (and should) be evaluated across information-access scenarios, not treated as a scalar "model honesty" parameter.

The eligibility disqualification mechanism as a benchmark design template. FACTS Grounding v2's identification of the "ineligible response" attack — where models achieve high grounding scores by generating trivially vague but accurate responses — generalizes beyond grounding. Any benchmark that evaluates output quality on a single correctness dimension is vulnerable to this class of circumvention: models can optimize the measured dimension while degrading unmeasured dimensions that matter to users. The solution pattern — adding a second evaluation criterion (eligibility, helpfulness, completeness) and requiring both criteria to be satisfied — provides a template for future benchmark designers. This is not a theorem or an algorithm, but it is the kind of practical design insight that spreads through the benchmark construction community and improves the robustness of future evaluations. The paper's transparency about the remaining limitations (the best judge achieves only 46.15 F1 on negative-class detection, Table 10) is equally valuable: it calibrates expectations about automated judging and motivates investment in better judge models rather than more complex judgment schemes.

Search efficiency as a newly visible dimension. The "Average searches" column in Table 8 establishes a precedent that future search benchmarks will be expected to follow. Prior work treated search as a binary capability — either the model can use search to find the answer, or it cannot — ignoring the efficiency with which it does so. The FACTS results show that search efficiency varies substantially across models (3.39 average searches for Gemini 3 Pro vs. 4.74 for Grok 4 Fast — a 40% difference) and that efficiency does not simply correlate with accuracy (Grok 4 searches more than GPT-5 but achieves lower accuracy). By making this dimension visible, the benchmark creates a competitive incentive for model developers to optimize query formulation — to retrieve the right information on the first try rather than issuing multiple exploratory queries. In deployment, this efficiency difference compounds: over millions of queries, a 40% reduction in API calls translates to substantial cost savings and latency improvements. The fact that the efficiency data is reported alongside accuracy, even though it does not enter the FACTS Score, signals that it is a legitimate dimension of model comparison, not an afterthought.

Which research directions become more attractive, and which become less so. The FACTS results suggest that research on unified factuality that balances precision and recall — models that can be both comprehensive and accurate — is more valuable than optimizing either dimension in isolation. A model achieving 85% Coverage but 40% No-Contradiction is not practically useful, nor is a model achieving 90% No-Contradiction but 30% Coverage. The benchmark also suggests that hedging calibration — teaching models to hedge on exactly the questions they are likely to get wrong, without excessive abstention — is a high-leverage research direction: GPT-5 mini hedges on 67.6% of Parametric questions but achieves only 49.6% attempted accuracy when it does answer, suggesting its abstention threshold is miscalibrated. Finally, the benchmark makes search efficiency optimization a legitimate research target rather than a deployment detail: a model that achieves 77.7% Search accuracy with 3.0 searches is strictly better than one achieving the same accuracy with 4.3 searches, and the former should be recognized as genuinely more capable even though both receive the same accuracy score.

Follow-Up Research This Work Enables

Tracking factuality profiles over model generations, not just snapshots. The FACTS Leaderboard is positioned as an actively maintained benchmark that will continue evaluating new models as they are released. The most straightforward and valuable follow-up is longitudinal: as new models (Gemini 3.5, GPT-6, Claude 5, etc.) are evaluated on the exact same datasets over the next 1–2 years, the field will accumulate a time series of factuality profiles across model families. This will reveal whether the Coverage–No-Contradiction tradeoff is narrowing (are models learning to be both comprehensive and accurate?) or stable (is it a fundamental tension that cannot be optimized away?). It will show whether hedging behavior is converging (are all models learning similar abstention strategies?) or diverging (are different model families specializing in different factuality strategies?). It will show whether Search accuracy is saturating — the current top score of 83.8% leaves 16.2 points of headroom, and watching that gap close over time will reveal how quickly search-augmented factuality is approaching a ceiling. The benchmark's public/private split and Kaggle-hosted evaluation make this longitudinal analysis credible in ways that self-reported benchmarks could not support.

Per-subset and per-category breakdowns to identify capability bottlenecks. The paper reports only aggregate scores for Search (Table 8) and Multimodal (Table 3), but both benchmarks have internal structure that would enable much richer analysis. The Search benchmark has four subsets with distinct difficulty mechanisms: Hard Tail (~328 questions requiring complex multi-step search reasoning), Wiki Two-Hop (~932 questions requiring indirect entity resolution), Wiki Multi-Doc (~268 questions requiring cross-document synthesis), and KG Hops (~356 questions requiring structured graph traversal). A follow-up analysis could report model accuracy per subset, revealing which specific search capabilities drive the overall performance gaps. Does Gemini 3 Pro's 83.8% accuracy come from near-perfect Wiki Two-Hop performance and weaker Hard Tail results, or from broad competence across all subsets? If all frontier models are near-ceiling on Wiki Two-Hop but far from ceiling on Hard Tail, the benchmark's discriminating power is concentrated in the Hard Tail subset, and future iterations of the benchmark should expand that subset. Similarly, the Multimodal benchmark has 10 image categories and 10 question categories (Figure 1); per-category accuracy would reveal whether models struggle specifically with data interpretation tasks (charts/graphs, 8.0% of questions) versus visual description tasks (45.7% of questions), or with specific image types (diagrams vs. natural images). This breakdown would transform the benchmark from a single score into a diagnostic tool that tells model developers exactly where to focus improvement efforts.

Validation of the adversarial sampling independence claim. The paper's Parametric construction uses five unnamed open-weight models to filter questions (retaining only those all five fail), with the explicit justification of decoupling hardness filtering from the proprietary models being evaluated. A critical follow-up would conduct a differential item functioning (DIF) analysis: for each question in the Parametric benchmark, compute the accuracy of each proprietary model family (Gemini, GPT, Claude, Grok) and test whether any model family shows systematically different performance patterns. If the adversarial filtering successfully decoupled construction from evaluation, the question-level performance profiles should be similar across families — questions that are hard for Gemini should also be hard for GPT, Claude, and Grok, and vice versa. If, instead, specific questions are dramatically easier for one family than others, the filtering may have inadvertently selected for weaknesses shared by the open-weight models and some (but not all) proprietary models, introducing bias. This analysis would require releasing question-level metadata (not the questions themselves, but family-level accuracy per question) and would either validate the paper's key methodological claim or identify a bias that needs correcting in future benchmark construction.

Human baseline study for all four benchmarks. The paper reports human validation of automated judges (Table 4 for Multimodal, Table 10 for Grounding v2) but does not establish human performance ceilings on the benchmarks themselves. A follow-up human study would recruit qualified annotators (potentially subject-matter experts for domain-specific questions) to answer a stratified sample of questions from each benchmark under the same conditions as the models (no search for Parametric, search access for Search, image access for Multimodal, context access for Grounding). This would establish: (a) the human performance ceiling for each benchmark — is 46.9% on Multimodal near human-level (suggesting the task is genuinely difficult) or far below it (suggesting substantial room for model improvement)? (b) the headroom remaining before the benchmark saturates — if humans achieve 95% on Search, the current 83.8% ceiling is a temporary limitation, not a fundamental one. (c) calibration of the difficulty scale — if humans also show rank-order performance across subsets similar to models (worse on Hard Tail than Wiki Two-Hop), the difficulty gradient is task-intrinsic rather than an artifact of model limitations. Human baselines are standard in NLP benchmarks (SuperGLUE, SQuAD 2.0 all report human performance) and their absence in FACTS is a gap that a follow-up study should fill.

Dynamic, per-question allocation of factuality strategies as a meta-benchmark. The FACTS results demonstrate that factuality decomposes into independent capabilities, but the current evaluation treats each benchmark independently — models are not asked to dynamically choose which capability to deploy based on the question. A meta-benchmark could present questions that require the model to decide in real-time whether to answer from parametric knowledge, search for information, ground the answer in provided context, or integrate multiple information sources. For example: "Given this financial document and your knowledge of recent market events, what is the likely impact on Q4 revenue?" This requires grounding (the document), parametric knowledge (recent events the model should know), and potentially search (if the model recognizes its knowledge is incomplete). A model that can dynamically route between information sources, hedging when uncertain and retrieving when needed, would achieve higher accuracy than a model that treats all questions as closed-book or all as search-required. This extends the FACTS framework from evaluating static factuality capabilities to evaluating meta-cognitive factuality strategies — does the model know when it knows, when it needs to look something up, and when it needs to synthesize across sources? The infrastructure for such a benchmark exists implicitly across the four FACTS sub-benchmarks; what is needed is a dataset that blends their task formats and a metric that rewards appropriate source selection, not just answer accuracy.

FACTS Score weighting optimization using real-world importance data. The FACTS Score uses equal weighting of the four sub-benchmarks, which the paper presents as a neutral default. A follow-up study could determine empirically optimal weights by surveying downstream applications: for a set of representative use cases (customer support, financial analysis, medical QA, academic research, personal assistant tasks), what is the relative importance of each factuality dimension? A financial analyst might value Grounding above Parametric (documents matter more than trivia), while a personal assistant might value Search above Grounding (most queries are information-seeking rather than document-based). By collecting importance weights from domain experts, a weighted FACTS Score could be computed per application, and the sensitivity of model rankings to these weights could be analyzed. If rankings are stable across reasonable weight variations (as the paper implicitly assumes by choosing equal weights), the equal-weight default is validated. If rankings shift substantially, the benchmark would be more informative as a profiling tool (reporting the full four-dimensional vector) rather than a ranking tool (reporting a single aggregate). This study would also establish whether the four benchmarks have equal measurement reliability — if the Multimodal autorater's macro F1 of 72.3 makes it substantially noisier than the Parametric grader, an optimal weighting for precision would down-weight the noisier component. The paper already has the data to conduct a preliminary version of this analysis using inter-model correlation patterns; the extension to real-world importance weights would make the analysis actionable.

Practical Applications and Downstream Use Cases

Model selection for domain-specific deployments. An organization deploying an LLM for a specific application can use the FACTS sub-benchmark profiles to select the model whose factuality strengths align with their primary failure modes, rather than relying on aggregate rankings. Consider a legal document review system: the primary factuality requirement is grounding — the model must faithfully represent what contracts or case law documents say, without hallucinating clauses or precedents. A model achieving 74.2% on Grounding (Gemini 2.5 Pro, Table 1) is preferable to a model achieving 69.6% (GPT-5), even though GPT-5 has a nearly identical FACTS Score (61.8 vs. 62.1). The 4.6 percentage point Grounding advantage directly translates to fewer fabricated legal claims in production. Conversely, a customer-facing chatbot that answers factual questions (historical facts, product specifications, biographical details) should prioritize Parametric accuracy: Gemini 3 Pro at 76.4% vs. Claude 4.5 Opus at 30.6% is not a close call, despite both being "frontier" models. The FACTS breakdowns make this targeted selection possible, whereas a single "factuality score" or a narrow benchmark would not.

Diagnosing and addressing hedging miscalibration in user-facing models. The Parametric hedging data in Table 6 directly informs user experience design. A model that hedges on 45.1% of factoid questions (Claude 4 Sonnet) will frustrate users who expect direct answers to common knowledge questions — nearly half of all such queries will return "I don't know" or equivalent. Conversely, a model that hedges on only 1.4% (Gemini 3 Pro) but achieves 76.4% accuracy will confidently provide incorrect answers to roughly one in four queries — a different but equally damaging failure mode. The F1 metric (harmonic mean of accuracy and attempted accuracy) provides a single number that captures this balance: Gemini 3 Pro at 77.0 F1 vs. Claude 4 Sonnet at 26.3 F1 quantifies the practical difference. A product team can use the full hedging × accuracy breakdown to decide which failure mode is more acceptable for their application (is it better to occasionally be wrong or frequently be unhelpful?) and select models accordingly. Furthermore, the hedging rates across benchmarks reveal whether a model's abstention behavior is calibrated to its information access: a model that hedges 67.6% on Parametric but only 22.5% on Search (GPT-5 mini) is appropriately more confident when it can verify information, which is the behavior pattern a product team should seek if deploying a search-augmented system.

Search cost optimization for high-volume query processing. For applications processing millions of queries (search-augmented customer support, automated research assistants, fact-checking pipelines), the "Average searches" column in Table 8 translates directly to operational costs. If each Brave Search API call costs $X, a model averaging 3.39 calls per query (Gemini 3 Pro) will incur 26% lower search costs than a model averaging 4.28 calls (GPT-5), while simultaneously achieving 6.1 percentage points higher accuracy. Over 10 million queries, a 0.89 call difference per query translates to nearly 9 million additional API calls for the less efficient model — with corresponding latency and financial implications. A deployment team can treat the search efficiency metric as a cost-multiplier on top of the per-token inference cost, computing total cost-per-query as (inference_cost_per_token × avg_tokens_per_response) + (search_api_cost × avg_searches). The FACTS data makes this calculation possible for the first time, since prior search benchmarks did not report search volume. The finding that higher search volume does not guarantee higher accuracy (Grok 4 Fast searches most but scores third-lowest) is practically significant: it means that naive "search more" strategies are wasteful, and the efficiency advantage of models like Gemini 3 Pro is a genuine capability differentiator.

Guiding fine-tuning and RLHF optimization priorities. The Coverage vs. No-Contradiction decomposition in Table 3 provides a concrete diagnostic for model development teams. If a fine-tuning run improves a model's accuracy on a held-out factuality test but the team does not know whether the improvement came from better recall (mentioning more relevant facts) or better precision (making fewer false claims), they cannot determine whether further training in the same direction will continue to help or will overshoot into a different failure mode. The FACTS evaluation — specifically the separate Coverage and No-Contradiction scores — would reveal that a training run increased Coverage from 60% to 65% while No-Contradiction remained flat at 58%, telling the team they have improved comprehensiveness without sacrificing accuracy. A subsequent run that increased Coverage to 68% but dropped No-Contradiction to 53% would reveal they have now pushed too far into recall at the expense of precision — the AND-product of the two scores would decline even though Coverage improved. This kind of multi-dimensional monitoring during model development is standard in other ML domains (precision-recall curves, ROC analysis) but was impossible for factuality evaluation before a benchmark that decomposes the two dimensions. The FACTS framework provides the instrumentation; model developers provide the optimization loop.