ReportBench: Evaluating Deep Research Agents via Academic Survey Tasks

ArXiv: 2508.15804

🎯 Pitch

Deep Research agents from OpenAI and Google produce reports with 95%+ factual accuracy, yet cite on average only 2% of the references that a human-written survey paper includes, operating with a precision-first, shallow-coverage strategy that still fabricates roughly 4% of cited-source links. ReportBench exposes these coverage-accuracy tradeoffs by reverse-engineering arXiv surveys into ground-truth benchmarks and automatically verifying every citation and factual claim against original sources.

---

1. Executive Summary

This paper introduces ReportBench, a systematic benchmark that evaluates the content quality of research reports generated by Deep Research agents by leveraging published arXiv survey papers as gold-standard references and applying reverse prompt engineering to create domain-specific evaluation tasks. The framework assesses reports along two critical dimensions—the quality and relevance of cited literature (measured via precision and recall of retrieved references against ground-truth bibliographies) and the faithfulness and veracity of all statements (verified through an agent-based pipeline that checks cited claims against original sources and validates non-cited claims via web-connected multi-model voting). Empirical evaluation reveals that commercial Deep Research agents from OpenAI and Google generate more comprehensive and reliable reports than standalone LLMs augmented with search tools, with OpenAI achieving 78.87% citation match rate and 95.83% factual accuracy, yet substantial gaps remain in reference recall (0.033 vs. a ground-truth average of 153 references per paper) and citation semantic consistency, establishing that current systems trade off breadth of coverage for per-claim accuracy and remain susceptible to both statement hallucination and fabricated citation links.

2. Context and Motivation

The Core Problem: We Cannot Reliably Evaluate AI-Generated Research Reports

The fundamental question this paper tackles is deceptively simple: when a Deep Research agent produces a multi-page research report in minutes—something that previously required human experts weeks to complete—how do we know whether the report is actually good? This is not merely an academic exercise. The stakes are high because these systems are increasingly positioned as tools for knowledge workers, scientists, and analysts who must trust that the reports they receive are factually accurate, properly sourced, and comprehensive. A report that confidently fabricates citations or misrepresents the content of cited papers can mislead downstream decisions in ways that are hard to detect without laborious manual verification.

The paper argues that evaluating research report quality decomposes into two axes—writing quality and content quality—and that the second axis, while more tractable to define objectively, currently lacks mature evaluation methodologies. This gap is the central problem ReportBench addresses.

Why This Is Urgently Important

The paper situates this problem against the backdrop of an accelerating deployment curve. Systems like OpenAI's Deep Research (OpenAI, 2025) and Google's Gemini Deep Research (Google, 2025) are already publicly available, promising to "autonomously execute extensive research tasks, including academic literature surveys, industry analyses, and market assessments" (Section 1). The paper cites a cascade of recent systems—Chen et al. (2025), Gottweis et al. (2025), Lu et al. (2024), Tang et al. (2025), Yamada et al. (2025), Zheng et al. (2025), Li et al. (2025)—indicating that this is not a niche capability but a rapidly maturing product category.

The practical significance is therefore immediate and concrete:

• Trust in automated knowledge synthesis. If a Deep Research agent produces a survey claiming that a particular medical treatment is effective, and that claim is based on hallucinated citations or misinterpreted sources, the consequences are not merely academic—they could influence clinical decisions, investment strategies, or policy recommendations. The paper's finding that even commercial products generate fabricated reference links (the "citation hallucination" example in Section 4, where gemini-2.5-pro invents a non-existent ResearchGate URL) demonstrates that this is not a hypothetical risk but an observed failure mode in deployed systems.

• Absence of monitoring infrastructure. Unlike translation or summarization—where metrics like BLEU or ROUGE provide imperfect but directionally useful signals—there is no widely accepted automated metric for "did this research report fairly and accurately survey the literature?" The paper argues that this absence prevents both users and developers from making informed comparisons between competing products or tracking improvements across model versions.

• Scale of the verification problem. A single Deep Research report can contain dozens of cited references and hundreds of individual factual claims. Manual verification by domain experts does not scale to the volume of reports these systems can generate. An automated, systematic evaluation framework is therefore a prerequisite for responsible deployment at scale.

Where Existing Evaluation Approaches Fall Short

The paper identifies four categories of prior evaluation work, each of which addresses only a fragment of the full research report evaluation problem. The paper's contribution is not to claim these prior approaches are wrong, but to argue that they are necessary but insufficient when applied to long-form, multi-source research reports generated by modern agents.

Fact-checking benchmarks isolate individual claims but ignore synthesis quality. The paper acknowledges that automated fact-checking has evolved into a mature, multi-stage pipeline: claim detection (Guo et al., 2022; Panchendrarajan & Zubiaga, 2024) identifies factual statements worth verifying, and evidence retrieval (Eldifrawi et al., 2024; Nanhekhan et al., 2025) finds supporting or refuting documents. Benchmarks like FEVER (Thorne et al., 2018), EX-FEVER (Ma et al., 2024), and SciFact (Wadden et al., 2020, 2022) evaluate these pipelines on both general-domain and scientific claims. These are valuable contributions, but the paper makes a precise criticism: "these benchmarks focus solely on fact-checking components, rather than evaluating the synthesized information as a whole, limiting their ability to assess recent long-form output from large language models (LLMs), such as full research reports" (Section 5, Fact Checking Evaluation). In other words, fact-checking benchmarks evaluate whether individual atomic claims are true, but they do not evaluate whether the report as a whole achieves adequate coverage, whether the selected references are the most relevant ones, or whether the narrative synthesis fairly represents the state of the field. A report could achieve perfect fact-checking accuracy while being a poor survey—for instance, by cherry-picking only papers that support a particular viewpoint while ignoring the broader literature.

Citation evaluation tools verify existence and format, not completeness. The paper points to existing citation verification tools that check "syntactic verification, existence verification, and semantic verification" (Barrot, 2025; Bairagi & Lihitkar, 2024) as well as benchmarks that evaluate citation quality through tasks like cited context identification, evidence sentence retrieval, and citation accuracy classification (Sarol et al., 2024). While these tools can flag a citation that doesn't exist or a statement that contradicts its cited source, the paper argues they share a critical blind spot: "the aspect of citation completeness—i.e., whether all relevant prior work on a given research topic has been cited—remains underexplored" (Section 5, Citation Evaluation). This is a subtle but important distinction. A report might cite real papers and accurately describe their content, yet still be a poor survey because it omits seminal works or fails to capture the breadth of the field. This dimension—which ReportBench addresses through reference overlap with expert-curated ground-truth bibliographies—is essentially absent from prior citation evaluation work.

Survey generation systems optimize for coherence, not factual grounding. The paper reviews the recent wave of automated survey generation systems: AutoSurvey (Wang et al., 2024) uses a two-stage LLM pipeline, HiReview (Hu et al., 2025) employs a taxonomy-driven framework, and SurveyForge (Yan et al., 2025) combines human outline structures with a scholar navigation agent for literature retrieval. These systems represent genuine progress in generating readable, structured, and topically coherent surveys. However, the paper notes limitations shared by these approaches: reliance on fixed or restricted citation scopes (meaning the systems cannot discover new relevant literature beyond what they are initially given), difficulty capturing human writing styles, and—most critically for evaluation—a focus on surface-level coherence rather than factual verification. The accompanying SurveyBench (paired with SurveyForge) evaluates reference, outline, and content quality, but ReportBench positions itself differently: it "focuses solely on well-defined and automatically verifiable dimensions of evaluation—namely, factual faithfulness and correctness" (Section 5, Survey Generation). This narrower scope is framed as a strength: by concentrating on dimensions that can be evaluated automatically and objectively, ReportBench avoids the subjectivity that plagues assessments of writing style or argumentation quality.

Deep Research evaluation benchmarks are surface-level. The paper identifies several existing benchmarks that evaluate capabilities relevant to Deep Research agents: BrowseComp (Wei et al., 2025), BrowseComp-ZH (Zhou et al., 2025), and WebWalker (Wu et al., 2025) test web retrieval and browsing abilities; SimpleQA (Wei et al., 2024), GAIA (Mialon et al., 2024), and Humanity's Last Exam (Phan et al., 2025) evaluate multi-hop factual reasoning; and Deep Research Bench (Bosse et al., 2025; Du et al., 2025) tests end-to-end report generation. The paper's criticism is direct: these benchmarks "often operate at a surface level and fall short of evaluating the core competencies essential for rigorous and reliable research" (Section 5, Deep Research Evaluation). What does "surface level" mean here? Based on the paper's methodology, it means that these benchmarks either test isolated skills (can the agent navigate to a webpage? can it answer a multi-hop question?) without evaluating the holistic quality of a complete research report, or they evaluate end-to-end reports but use coarse-grained metrics that do not decompose into citation-level and statement-level verification. ReportBench's contribution is the granularity: instead of giving a report a single overall score, it separately measures reference precision, reference recall, citation semantic consistency, and non-cited statement accuracy, producing a disaggregated profile that reveals where a system fails rather than just that it fails.

How This Paper Positions Itself

ReportBench is not positioned as a new agent or a new report generation method—it is an evaluation methodology and benchmark. Within that evaluation landscape, it occupies a specific niche that the paper argues is currently unfilled: end-to-end evaluation of research report content quality that is simultaneously comprehensive (covering both citation behavior and factual accuracy), fine-grained (decomposing into per-statement verification rather than holistic judgment), and automatically scalable (not requiring human expert annotation per evaluation instance).

The paper achieves this niche through two design decisions that distinguish it from prior work. First, it uses published arXiv survey papers as gold-standard references, which provides two things simultaneously: (a) a ground-truth bibliography against which retrieval recall can be measured without human labeling, and (b) a source of expert-validated content that serves as the implicit answer key for the "reverse prompt engineering" process. This is a clever bootstrap because it converts the expensive problem of "evaluate whether this report is good" into the tractable problem of "evaluate whether this AI-generated report on topic X recovers the same references and factual content as the expert-written survey on topic X." Second, it separates cited and non-cited statements into two distinct verification pipelines, acknowledging that these require fundamentally different evaluation strategies: cited claims can be checked against source documents (a relatively objective comparison), while non-cited claims require external web search and multi-model voting (an inherently noisier but necessary process).

The paper explicitly acknowledges that it is not evaluating writing quality—"leaving the assessment of writing quality to future work" (Section 1)—and is not attempting to evaluate the argumentative structure, novelty, or critical insight of generated reports. This scope constraint is important: it means ReportBench cannot tell you whether a report makes a compelling intellectual contribution, only whether it is factually grounded and well-sourced. The authors frame this as focusing on the dimensions that are "more clearly defined" (Section 1) and automatically verifiable, leaving subjective dimensions for future work.

In relation to the broader evaluation landscape, ReportBench most directly complements SurveyBench (Yan et al., 2025) and Deep Research Bench (Bosse et al., 2025; Du et al., 2025). Compared to SurveyBench, ReportBench claims advantages in scalability (automated construction pipeline vs. reliance on human outlines) and a sharper focus on objectively verifiable dimensions. Compared to Deep Research Bench, ReportBench provides finer-grained, per-statement verification rather than aggregate scoring. The paper's claim—one that the experiments in Section 3 are designed to support—is that this decomposition matters because it reveals failure modes (hallucinated citations, over-generation of low-quality references) that would be invisible in a single headline accuracy number.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily an evaluation methodology and benchmark construction paper — its core output is not a new model or agent, but a systematic framework for measuring the quality of research reports generated by Deep Research agents. The system being built is a two-part pipeline: first, a procedure for automatically constructing high-quality test tasks (prompts with ground-truth answers) from published survey papers on arXiv, and second, an automated evaluation engine that takes a generated research report as input and produces disaggregated quality scores across four dimensions: reference precision, reference recall, citation statement faithfulness, and non-cited statement factual accuracy. The problem this solves is the absence of scalable, objective evaluation for the end-to-end output of Deep Research agents — without ReportBench, comparing two such agents requires either subjective human judgment or proxy metrics (e.g., retrieval accuracy on isolated queries) that don't capture whether the final report is actually trustworthy.

3.2 Big-Picture Architecture (Diagram in Words)

The ReportBench framework has five major components, organized into two sequential phases (dataset construction, then evaluation):

1. Survey Paper Filter and Reference Extractor — takes the arXiv metadata snapshot as input, applies regex and LLM-based filters to identify peer-reviewed survey papers, and parses their LaTeX source to extract gold-standard bibliographies. Output: a corpus of 678 survey papers, each with a ground-truth reference list.

2. Reverse Prompt Engineer — takes each survey paper (full text + publication date) and uses an LLM to generate a research prompt whose ideal answer would be that survey paper, producing three prompt variants per paper at different granularity levels. Output: domain-specific research prompts with temporal cut-off constraints and explicit instructions not to cite the original source paper.

3. Domain Classifier and Test Set Sampler — uses Gemini 2.5 Pro to classify all prompts into 11 application domains, then downsamples to a balanced 100-prompt test set validated by four research experts. Output: the final ReportBench benchmark with 100 prompts.

4. Content Quality Evaluator — takes a generated report, extracts all URL citations, fetches the linked web pages, uses an LLM to extract scholarly article titles from those pages, and computes precision and recall of the extracted references against the gold-standard bibliography from the survey paper. Output: reference precision, reference recall, and average reference count per report.

5. Statement Factuality Verifier — splits the generated report's content into cited statements (those with explicit citation links) and non-cited statements (factual claims without citations), then runs two separate verification pipelines: (a) for cited statements, retrieves the cited source's full text, locates the most semantically relevant passage, and checks consistency between the statement and the source; (b) for non-cited statements, uses multiple web-connected LLMs with majority voting to verify factual correctness. Output: citation match rate (proportion of cited statements consistent with sources) and factual accuracy (proportion of non-cited statements verified as correct).

Information flows as follows: a survey paper enters the construction pipeline → its references are extracted and its text is reverse-engineered into a test prompt → the prompt is given to a Deep Research agent (or base model with search tools) → the agent generates a research report → the evaluation engine extracts the report's citations and statements → citations are matched against the gold-standard reference list to compute precision/recall → cited statements are verified against their source documents → non-cited statements are fact-checked via web search and multi-model voting → the four quality scores are produced.

3.3 Roadmap for the Deep Dive

• First, the core design philosophy: why the paper decomposes evaluation into content quality and statement factuality, and why it separates cited from non-cited claims for verification — this framing determines every downstream design choice.
• Second, the dataset construction pipeline (Phases I–III): how survey papers are identified, filtered, and turned into test prompts, because the quality of the benchmark determines the validity of all downstream measurements.
• Third, the content quality evaluation: how references are extracted from generated reports, matched against gold-standard bibliographies, and scored — this is the simpler half of the evaluation and establishes the retrieval performance baseline.
• Fourth, the cited statement verification pipeline: the three-stage process (extract, locate, verify) and why it is designed as interpretable sub-steps rather than a single LLM-as-a-judge call.
• Fifth, the non-cited statement verification pipeline: how multi-model web-connected voting works, why six independent judgments are collected, and how the voting threshold is applied.
• Sixth, the experimental configuration: which models serve as evaluators, how base models are augmented with search tools, and the specific settings that govern all experiments — this is necessary context because the evaluation framework itself uses LLMs, meaning the evaluator's reliability is part of the methodological story.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an evaluation methodology paper whose core idea is that the content quality of AI-generated research reports can be automatically and systematically assessed by (a) using expert-written survey papers as gold-standard references for what a good survey on a topic should cite and contain, and (b) decomposing the evaluation into separately verifiable sub-problems (reference overlap, citation faithfulness, non-cited claim accuracy) rather than relying on holistic human or LLM judgment.

---

The Core Design Philosophy: Why Decompose Content Quality into These Specific Dimensions?

The paper's evaluation framework is built on a specific claim about what makes a research report "good" in terms of content (as opposed to writing style, which is explicitly out of scope). The claim, stated in Section 1, is that content quality hinges on two critical factors: "(1) the quality and relevance of cited literature, and (2) the faithfulness and veracity of generated statements, whether derived from cited references or produced by the model."

This decomposition is not arbitrary — it reflects a fundamental asymmetry in how claims within a research report can be verified:

Cited statements make an explicit assertion about what a particular external source says. For example, "Smith et al. (2023) showed that method X achieves 95% accuracy on benchmark Y." Verifying this claim requires access to the cited source document and a comparison between what the report says and what the source actually says. This is, in principle, a closed-world verification problem: the ground truth is contained in the cited document, and the verification task is to check semantic consistency between two texts (the report's claim and the source's content). The paper designs a three-stage pipeline for this (extract citation-statement mappings, locate supporting passages in source documents, verify semantic consistency) that is intended to be interpretable and auditable, with intermediate outputs preserved for human inspection.

Non-cited statements are factual claims the model makes without attributing them to any source. For example, "Reinforcement learning from human feedback was first applied to language models in 2017" or "The transformer architecture uses self-attention to process sequences in parallel." Verifying these claims requires accessing external knowledge — there is no single source document to check against. This is an open-world verification problem that fundamentally cannot be reduced to text-to-text comparison. The paper addresses this through web-connected LLMs with majority voting, explicitly trading off the precision of source-document verification for the coverage of web-based fact-checking.

The paper further argues (in Section 2.2) that reference overlap with ground-truth bibliographies serves as a distinct evaluation dimension from statement-level verification. A report could cite all the right papers (high recall) but misrepresent what they say (low citation match rate), or it could perfectly describe the papers it cites (high match rate) but omit most of the important literature in the field (low recall). These are independent failure modes, and evaluating them separately is what enables the paper's later analysis to diagnose which failure modes characterize different Deep Research products (e.g., in Table 1, Gemini generates three times more cited statements than OpenAI but achieves lower precision, suggesting a breadth-over-accuracy trade-off).

This three-way decomposition — reference quality, cited statement faithfulness, non-cited statement accuracy — is the architectural backbone of ReportBench. Every design choice in the data construction and evaluation pipelines flows from the commitment to measuring these dimensions separately and automatically.

---

The Dataset Construction Pipeline: From arXiv Surveys to Test Prompts

The dataset construction process transforms published survey papers into a benchmark of 100 research tasks. The pipeline has three phases (Sections 2.1.1–2.1.3), and each phase includes specific filtering and quality-control steps that are essential for the benchmark's validity.

Phase I: Survey Paper Identification and Reference Extraction

The goal of this phase is to produce a corpus of high-quality survey papers with extracted, machine-readable bibliographies that serve as the ground truth for later reference-overlap evaluation.

Source and initial filtering. The pipeline starts from the "complete arXiv metadata snapshot" (arXiv.org submitters, 2024), which contains metadata for all papers submitted to arXiv. The first filter removes papers with submission dates before 2020-01-01, restricting the corpus to relatively recent work. This temporal filter serves two purposes: it ensures the surveys reflect current research landscapes (making them more representative of what a Deep Research agent would be asked to survey), and it ensures that the papers' cited references are also recent enough that their web pages are likely still accessible (important for the evaluation phase, which needs to fetch cited sources via URL).

Peer-review filter. To ensure quality, the paper only retains papers that "have undergone peer review and have been formally published." The method for identifying these papers is a two-step regex-based approach applied to the arXiv metadata:

1. Query titles for the keywords "survey" or "review" — this identifies papers that self-identify as literature surveys.
2. Search the comments field for "published" or "accepted" — this identifies papers that have completed the peer review process and been accepted at a venue.

This filtering is coarse and admits false positives. The paper gives the example of astronomy as a domain where the word "survey" often refers to observational sky surveys (e.g., the Sloan Digital Sky Survey) rather than literature surveys. To address this systematic false-positive problem, the pipeline adds a GPT-4o classification step: each paper's title and abstract are sent to GPT-4o (Hurst et al., 2024), which is prompted to produce a binary classification of whether the paper is a literature survey. The paper does not provide the exact prompt used for this classification, but the intent is clear: LLM-based semantic filtering removes papers where keyword matching fails to distinguish between disciplinary uses of the word "survey."

Reference extraction. For each surviving survey paper, the pipeline processes the paper's LaTeX source file to extract cited references. The extraction procedure has several steps:

1. Parse LaTeX citation commands (\cite{...}, \citep{...}, etc.) to identify which bibliography keys are cited in the main text.
2. Locate the corresponding bibliography database files (.bib files) referenced in the LaTeX source.
3. Retrieve the full bibliographic entry for each cited key from these databases.
4. Filter to retain only references explicitly cited in the main text (as opposed to references that exist in the .bib file but are never actually cited — a common artifact in LaTeX projects where bibliographies accumulate unused entries).

This extraction procedure is significant because it means the ground-truth bibliography is not the survey paper's full reference section as rendered in the PDF (which might include uncited entries or formatting artifacts), but rather the set of references that the paper's authors actually connected to specific claims in the text via citation commands. The paper states that this "mirrors the true citation pattern of the paper." After all filtering, 678 papers are retained.

Why this matters. This ground-truth bibliography is the basis for computing reference precision and recall during evaluation. Precision measures what fraction of the generated report's references appear in the gold-standard list (i.e., are relevant to the topic). Recall measures what fraction of the gold-standard references were successfully retrieved by the agent. The quality of this ground truth directly determines the validity of these metrics: if the gold-standard bibliography contained irrelevant references (false positives from the extraction process), precision measurements would be artificially inflated for any agent that happened to cite those irrelevant papers. The multi-stage filtering (date, keyword, peer-review, GPT-4o classification, LaTeX citation parsing) is designed to minimize this risk.

Phase II: Reverse Prompt Engineering

The insight driving this phase is that a survey paper can be viewed as the ideal answer to a specific research question — the question that motivated the human experts to write the survey in the first place. If we can reconstruct that question from the paper itself, we obtain a test prompt whose ground-truth answer (the set of references and factual content the survey contains) is known. The paper calls this "reverse prompt engineering."

The core procedure. For each survey paper, the pipeline:

1. Extracts the full text of the paper using a PDF parsing tool (the specific tool is not named in the paper).
2. Retrieves the paper's publication date (specifically, the "most recent update" date from arXiv metadata) — this will serve as a temporal cut-off for the generated prompt.
3. Prompts an LLM (the specific model is not named in Section 2.1.2) with the paper's full text and publication date, instructing it to generate a query whose ideal answer is precisely that paper.

Three prompt granularities. To increase diversity in the benchmark, the paper designs three types of prompt templates that produce different levels of detail:

• Sentence-level prompts: A short, high-level research question that captures the survey's broad topic. Example in Appendix A.3: "Provide a comprehensive survey of deep learning techniques for natural language processing, covering architectures from RNNs to Transformers."
• Paragraph-level prompts: A more detailed prompt that specifies particular subtopics, methods, or domains the survey covers, essentially compressing the survey's section structure into a few sentences of instruction.
• Richly detailed prompts: An elaborate prompt that closely mirrors the survey's scope, listing specific methods, evaluation criteria, and research questions the survey addresses. This variant is the most constrained and should produce the highest reference overlap if the agent does thorough work.

The paper uses all three prompt types in the construction process but samples randomly from them when creating the final 100-prompt test set, ensuring diversity in prompt specificity. The exact prompts used for each paper are not provided in the main text, though Appendix A.3 contains examples.

Temporal consistency constraint. Every generated prompt includes a cut-off date corresponding to the survey paper's most recent update. The paper gives an example instruction:

"Ensure only papers published before April 2025 are referenced."

This constraint is essential for evaluation validity. Without it, a Deep Research agent evaluated in July 2025 could cite papers published after the survey was written — papers that the human survey authors could not possibly have cited because they did not exist yet. These post-survey papers would count as false positives in the reference overlap computation (they would not appear in the gold-standard bibliography even if they are genuinely relevant to the topic), artificially penalizing the agent's precision. The temporal constraint ensures that both the human authors and the AI agent are searching within the same chronological window, making the reference overlap comparison fair.

Anti-hacking instruction. During early model evaluation, the authors observed "a phenomenon akin to prompt hacking" — models would ignore the temporal constraints and directly retrieve the original source survey paper, trivially achieving perfect reference overlap by simply copying its bibliography. To prevent this, every prompt is augmented with an explicit instruction:

"The model must refrain from citing the original paper corresponding to the given prompt."

This instruction attempts to force the agent to conduct original research — finding and synthesizing the relevant literature independently — rather than shortcutting by locating the survey paper itself. The paper does not report quantitative data on how often this hacking behavior occurred before the instruction was added, nor does it verify that the instruction fully prevents it, but the fact that the reported precision and recall numbers in Table 1 are far from 1.0 (OpenAI Deep Research achieves 0.385 precision and 0.033 recall) suggests that even if some hacking occurs, the agents are not simply reproducing the survey's bibliography.

What the ground truth consists of for each prompt. After Phase II, each test instance in the benchmark has: (a) a research prompt with temporal constraints and anti-hacking instructions, (b) a ground-truth reference list (the bibliography extracted in Phase I), and (c) the full text of the original survey paper (available for qualitative comparison but not directly used in the automated metrics). The paper does not extract individual factual claims from the survey papers to serve as ground truth for statement-level verification — instead, statement verification is handled through the source-document and web-search pipelines described later, which do not depend on the survey paper's content except as an implicit quality standard.

Phase III: Application Domain Distribution and Test Set Balancing

The 678 filtered survey papers are not uniformly distributed across academic disciplines. arXiv's coverage is heavily skewed toward computer science, physics, mathematics, and related STEM fields. To create a balanced test set that represents a reasonable diversity of research domains, the paper applies domain classification and downsampling.

Domain classification procedure. Each paper is classified into one of 11 categories using Gemini 2.5 Pro (Comanici et al., 2025), with classification based on the paper's title and abstract. The 11 categories are shown in the paper (Section 2.1.3) and include domains such as "Natural Language Processing," "Computer Vision," "Robotics," "Biology & Medicine," "Physics," and others. Importantly, an "Unknown" category is included "to reduce misclassification" — papers that don't clearly fit any predefined category are assigned to Unknown rather than being forced into an incorrect class, which would introduce noise into the domain-balanced sampling.

The distribution of the full 678-paper corpus across these domains is shown in Figure 2, which reveals the expected STEM skew. For example, NLP and Computer Vision categories dominate while domains like Biology & Medicine are less represented.

Test set construction. From the 678 papers, a total of 100 papers are down-sampled to create a balanced test set. The paper does not specify the exact downsampling algorithm (e.g., whether it's uniform across categories, proportional to category size, or something else), but the stated goal is to "create a balanced and general test set." Since each paper has three associated prompts (one per granularity level), the pipeline then randomly samples from these three prompt types for each paper to obtain "the final prompt with diversity." The result is a benchmark of exactly 100 prompts, which the paper names ReportBench.

Validation. The quality of the domain classification on the 100-prompt subset "was then reviewed and validated by four research experts." This human validation step is important because misclassification at this stage would cause the domain breakdown in the experimental analysis to be unreliable — if a computer vision paper were misclassified as NLP, any domain-specific performance patterns would be muddied. The paper does not report inter-annotator agreement or the number of corrections made by the human validators, but the inclusion of expert review distinguishes this benchmark from fully automated constructions that might propagate classification errors into the final results.

The ground-truth reference scale. The paper reports that the ground truth from ReportBench "includes an average of 153 references per paper" (Section 3.3). This number is crucial context for interpreting the recall scores in Table 1: OpenAI Deep Research achieves a recall of 0.033, meaning it retrieves approximately $0.033 \times 153 \approx 5$ of the ground-truth references on average. The paper explicitly notes this and argues that "recall should be considered a secondary signal rather than the primary focus of evaluation" because many of the 153 references support overlapping statements and a survey that cites 5 highly relevant papers might still be useful even if it misses 148 others. Whether this argument holds depends on the use case — for a comprehensive literature review, 3.3% recall would be unacceptable; for a quick overview, it might be sufficient.

---

Content Quality Evaluation: Reference Extraction and Matching

Once a Deep Research agent has generated a report in response to a test prompt, the first evaluation step is to assess the quality of the report's cited literature. This is the simpler half of the evaluation framework and produces three metrics: reference precision, reference recall, and the average number of references per report.

Step 1: URL extraction and normalization. Most Deep Research products use URL links to cite web pages — for example, a statement might be followed by a hyperlink to an arXiv abstract page or a conference proceedings entry. The paper adopts this URL-based citation format even when evaluating base models (which might not natively use URLs) by having them output references as web links. The advantage of URL-based citations, according to the paper, is that they "place the citation immediately adjacent to the corresponding statement, which ensures consistent performance even under chunked evaluation settings." This is a practical consideration: when evaluating long reports, the text must often be split into chunks for processing by LLMs with finite context windows, and URL citations embedded in the text are less likely to be separated from their associated statements than, say, a numbered reference list at the end of the report.

All URLs in the generated report are extracted, normalized (e.g., resolving redirects, stripping tracking parameters), and deduplicated. The normalization step prevents the same paper being counted multiple times if it appears under different URL variants (e.g., https://arxiv.org/abs/2401.0001 vs. https://arxiv.org/pdf/2401.0001). The paper does not specify the exact normalization rules used.

Step 2: Web page retrieval and scholarly article identification. For each unique URL, the pipeline retrieves the content of the linked web page via web scraping. The paper does not name the specific scraping tool used in the evaluation pipeline, though the base model experiments use Firecrawl (which returns pages in Markdown format). An LLM is then used to determine whether each page "corresponds to a scholarly article" and, if so, to extract the article title. This classification is necessary because generated reports might cite non-scholarly sources (blog posts, news articles, corporate websites) that should not be counted as academic references in the precision/recall computation — the gold-standard bibliography consists exclusively of academic papers, so non-scholarly URLs are irrelevant to the overlap metric.

The paper does not provide the specific prompt used for this classification, nor does it report the LLM used for this step (the evaluation configuration in Section 3.1 specifies the models used for statement verification but not for reference title extraction).

Step 3: Title matching and overlap computation. The extracted scholarly article titles are compared against the ground-truth reference titles from the survey paper's bibliography. The paper does not specify the exact matching algorithm — whether it uses exact string matching, fuzzy matching (e.g., Levenshtein distance), or semantic matching via embeddings. Given the variability in how paper titles can be formatted (e.g., capitalization, punctuation, arXiv ID prefixes), some form of normalization or fuzzy matching is almost certainly necessary, but the implementation details are absent.

Metrics computed. From the overlap between extracted and ground-truth titles:

• Precision $P$ is the fraction of the generated report's cited references that appear in the ground-truth bibliography:

  $P = \frac{|\text{extracted references} \cap \text{ground-truth references}|}{|\text{extracted references}|}$

  This measures relevance: of the papers the agent chose to cite, how many are actually papers that an expert would consider important for this topic?

• Recall $R$ is the fraction of the ground-truth bibliography that the generated report successfully cites:

  $R = \frac{|\text{extracted references} \cap \text{ground-truth references}|}{|\text{ground-truth references}|}$

  This measures coverage: of all the papers an expert survey on this topic would cite, how many did the agent find?

• Reference count is simply $|\text{extracted references}|$, the number of unique scholarly references in the generated report, averaged across all test prompts. This provides context: an agent might achieve high precision by citing only a handful of very safe, well-known papers, but this would come at the cost of low recall and low reference count.

Why precision and recall are computed separately rather than combined into an F1 score. The paper does not report F1 scores, and this is a deliberate choice. Precision and recall capture fundamentally different aspects of research quality: precision captures whether the agent's citations are trustworthy (low precision means the report is padded with irrelevant or marginal references), while recall captures whether the agent's literature search is thorough (low recall means important work is missing). A system that optimizes for F1 might achieve a reasonable score by, say, having moderate precision and moderate recall, but this single number would mask the fact that the system is mediocre at both dimensions. By reporting precision and recall separately, the paper enables analysis of trade-offs — which is exactly what happens in Section 3.3, where Gemini is shown to achieve higher recall at the cost of lower precision compared to OpenAI, suggesting a different underlying retrieval strategy.

---

The Cited Statement Verification Pipeline

While reference overlap measures whether the right papers are cited, it does not measure whether the report accurately represents what those papers say. A report could cite a highly relevant paper (contributing to high recall) but completely mischaracterize its findings (misleading the reader). The cited statement verification pipeline addresses this by checking every statement that includes an explicit citation against the actual content of the cited source.

The paper designs this as a three-stage structured validation pipeline rather than a single end-to-end LLM judgment. The motivation for this decomposition is explicitly stated: "Unlike traditional 'LLM-as-a-judge' approaches, which often suffer from instruction-following issues or biased scoring, our method decomposes the evaluation into fine-grained, interpretable, and verifiable steps. All intermediate outputs are retained for optional human inspection, thereby maximizing the reliability and transparency of the evaluation process." This is a methodological claim about evaluation design: by forcing the pipeline to produce explicit intermediate artifacts (statement-citation mappings, supporting passage selections, per-statement consistency verdicts), the evaluation becomes auditable. A human can inspect why a particular statement was flagged as inconsistent and determine whether the pipeline made an error or genuinely caught a hallucination. This transparency is absent when an LLM is asked to produce a single holistic score.

Stage 1: Cited Statement Extraction and Citation Mapping

The pipeline first identifies all statements in the generated report that contain explicit citation links. The paper uses the notation $S_{cited}$ to denote the set of cited statements.

An LLM (gpt-4o, as specified in Section 3.1) processes the report text and extracts two things for each citation-bearing passage: (a) the full text of the statement (sentence or multi-sentence claim), and (b) the specific citation link(s) associated with that statement. The output is a mapping $M: S_{cited} \rightarrow \text{URL}$ that associates each cited statement with the web page(s) it references.

The paper does not specify the prompt used for this extraction in the main text, but Appendix A.2.1 contains the prompt template for "Cited Statement Extraction." The extraction must handle cases where one statement cites multiple sources (e.g., "Smith et al. (2023), Jones et al. (2024) showed that...") and where one source supports multiple statements. The paper does not detail how these multi-mapping cases are resolved in the subsequent verification stages.

The number of cited statements extracted per report becomes a metric in itself: the "Count" column under "Cited Statements" in Table 1 reports the average number of citation-bearing claims per report. OpenAI Deep Research averages 88.2 cited statements per report (with considerable variation), while base models average far fewer (e.g., o3 at 16.16). This metric captures the report's density of sourced claims — a report with very few cited statements might be generating most of its content from the model's internal knowledge rather than retrieved literature, which would be reflected in the non-cited statement count and accuracy metrics.

Stage 2: Supporting Source Retrieval

For each cited statement $s \in S_{cited}$ and its associated URL, the pipeline retrieves the full content of the cited web page via web scraping. This is the same retrieval mechanism used in the content quality evaluation, but here the purpose is to find the specific passage within the source document that supports (or fails to support) the statement.

The LLM is prompted to "locate the most semantically relevant passage that supports the original statement" within the retrieved web page content. This is a retrieval task, not a verification task — the LLM is asked to find the passage that best matches the statement, not to judge whether it actually supports the statement. The output is a text snippet from the source document.

The paper provides the prompt for this step in Appendix A.2.3 ("Supporting Source Extraction"). The design choice to separate retrieval from verification is important: it means that if the verification stage later flags a statement as inconsistent, the pipeline can show both the statement and the passage it was compared against, enabling a human auditor to determine whether the inconsistency is genuine (the source doesn't support the claim) or a retrieval error (the pipeline found the wrong passage within the source).

Stage 3: Semantic Consistency Verification

The final stage compares each cited statement $s$ with its retrieved supporting passage $p$ and produces a binary judgment: consistent or inconsistent.

The LLM (again gpt-4o) is prompted to perform this comparison. The paper provides the prompt in Appendix A.2.4 ("Semantic Consistency Verification"). The core logic is: does the cited source actually say what the generated report claims it says? If the statement says "Smith et al. showed method X achieves 95% accuracy" but the source paper reports 92% accuracy, the statement should be flagged as inconsistent (even though 95% might be directionally similar). The paper notes that the evaluation metrics — particularly citation semantic consistency — are "sensitive to such discrepancies, allowing us to quantitatively capture and penalize these hallucination phenomena across model outputs" (Section 4).

Aggregating to a match rate. After all cited statements have been verified, the citation match rate is computed as:

$\text{Match Rate} = \frac{|\text{statements judged consistent}|}{|\text{total cited statements verified}|}$

This is reported as a percentage in Table 1 (e.g., OpenAI Deep Research: 78.87%). The match rate is the proportion of the agent's sourced claims that are actually supported by the cited literature. A match rate of 100% would mean every claim attributed to a source is faithful to that source; a match rate of 50% would mean half of the agent's citations are misleading or fabricated.

Average alignment score. Table 1 also reports an "Avg Score" column under "Cited Statements," which for OpenAI Deep Research is 88.2. This appears to be the average per-statement consistency score (possibly on a 0–100 scale, where each statement receives a confidence score rather than a binary label), but the paper does not define this metric explicitly. The relationship between the binary "Match Rate" and the continuous "Avg Score" is unclear — they could be derived from the same underlying judgments (with Match Rate being a thresholded version of the continuous score) or from different evaluation stages.

Why not use the PRM-like approach from the example paper? A natural question is why the paper doesn't train a dedicated verifier model (analogous to a Process Reward Model) for citation consistency checking, rather than relying on prompted gpt-4o. The paper doesn't address this directly, but the likely reason is practical: training a verifier of comparable quality to gpt-4o on this task would require a large corpus of (statement, cited-source, consistency-label) triples, which would itself require human annotation at scale — exactly the kind of expensive manual effort the paper's automated approach is designed to avoid. The choice of gpt-4o as evaluator represents a pragmatic trade-off: it leverages a strong pre-trained model's semantic understanding capabilities without requiring task-specific training data, at the cost of potential evaluator bias and unreliability (which the interpretable pipeline design partially mitigates).

---

The Non-Cited Statement Verification Pipeline

Non-cited statements are factual claims the model makes without attribution. The paper's examples in Section 4 illustrate the type: assertions about the history of a technique, descriptions of well-known methods, or claims about what a field has achieved. These statements cannot be verified against a single cited source document because the model has not provided one. The paper addresses this with a two-step validation process that relies on web-connected LLMs with majority voting.

Step 1: Non-Cited Statement Extraction

The pipeline first identifies all factual statements in the report that "do not have any citations." The LLM (again gpt-4o, as part of the evaluation framework) extracts these statements. The paper notes an important filtering step: content that is "general common sense or already supported by references" is removed. This filtering is necessary because not every uncited sentence in a report is a factual claim worth verifying. Transitional phrases ("In this section, we discuss..."), opinion statements ("This is a promising direction"), and statements that are obvious to any reader with basic domain knowledge ("Machine learning models require training data") would all be extracted as uncited sentences but would waste verification effort and likely produce noisy judgments. The filtering step, described in Appendix A.2.2 ("Non-cited Statement Extraction"), attempts to retain only "factual statements worth verifying."

The number of non-cited statements extracted per report is reported as a metric in Table 1: "Count" under "Non-Cited Statements." OpenAI Deep Research averages 38.9 non-cited statements per report, while Gemini averages 49.6. This metric captures how much of the report's content is the model's own unsupported assertions versus content attributed to sources.

Step 2: Multi-Model Web-Based Verification

Each extracted non-cited statement is independently verified by multiple web-connected LLMs. The paper's specific configuration (Section 3.1) uses:

• Two models: gemini-2.5-pro and gemini-2.5-flash. These are chosen because they have native web-search capabilities.
• Three independent judgments per model: Each model processes each statement three separate times, producing three verdicts per statement per model.
• Six total verdicts per statement (two models × three judgments each).

The independent judgments per model are important: even a single model can produce different verdicts on the same statement due to sampling variability, different search results on different attempts, or inherent stochasticity in the model's reasoning. By collecting three judgments, the pipeline captures intra-model consistency in addition to inter-model agreement.

Voting mechanism. The final decision for each statement is determined by majority voting across the six verdicts:

$\text{Verdict} = \begin{cases} \text{True} & \text{if at least 4 of 6 judgments are "True"} \\ \text{False} & \text{if at least 4 of 6 judgments are "False"} \\ \text{Tie} & \text{otherwise (3-3 split)} \end{cases}$

The paper does not explicitly state how ties are handled in the final accuracy computation — ties could be excluded, counted as incorrect, or treated as a separate category. The "proportion of votes" is recorded as a confidence score for each statement: a statement with 6/6 True votes has higher confidence than one with 4/6 True votes, even though both are classified as True under majority voting.

Factual accuracy computation. The factual accuracy for non-cited statements is:

$\text{Factual Accuracy} = \frac{|\text{statements with majority vote True}|}{|\text{total non-cited statements verified}|}$

This is reported as a percentage in Table 1. OpenAI Deep Research achieves 95.83% non-cited statement accuracy, the highest among all evaluated systems.

Why multi-model voting, and why these specific models? The paper's design rationale is stated explicitly: "This approach avoids relying on a single model and makes the validation more reliable." The vulnerability of single-model verification is well-documented: an LLM might exhibit systematic biases in fact-checking (e.g., being overly credulous toward statements that sound authoritative, or being overly skeptical of statements outside its training distribution). By requiring agreement between two different models (gemini-2.5-pro and gemini-2.5-flash, which have different architectures and training), the pipeline reduces the risk that a shared bias contaminates the verification. The choice of three judgments per model further reduces the impact of random errors (e.g., a single failed web search that returns irrelevant results).

The paper does not provide the exact prompt used for web-based statement verification in the main text, though Appendix A.2.5 contains the prompt template.

Limitations acknowledged implicitly. The paper does not explicitly discuss the reliability of web-based verification in Section 3, but the methodology implies several limitations. Web-connected LLMs are subject to the quality and recency of search results — a statement that is true but poorly indexed by search engines might receive False verdicts not because it's wrong but because the verification evidence is inaccessible. Conversely, a false statement that appears on many web pages (e.g., a widely propagated misconception) might receive True verdicts. The multi-model voting mechanism mitigates but does not eliminate these issues. The paper's reported accuracy numbers (92–96% for Deep Research products) should be interpreted with these caveats: they represent the proportion of non-cited statements that are verifiable as true via web search and model consensus, not an absolute ground-truth accuracy.

---

Experimental Configuration: Models, Tools, and Settings

The paper's experiments evaluate two categories of systems: commercial Deep Research products and base LLMs augmented with search tools. The configuration details in Section 3.1 are essential for understanding what is being compared and under what conditions.

Evaluator models. Different LLMs serve different roles in the evaluation framework:

• gpt-4o (Hurst et al., 2024): Used for cited statement extraction, supporting source retrieval, and semantic consistency verification (all three stages of the cited statement pipeline). Also used for the reference title extraction step in the content quality evaluation. The paper does not justify why gpt-4o was chosen over alternatives for these roles — it may be a practical choice based on availability, cost, or prior experience with its reliability on structured extraction tasks.

• gemini-2.5-pro and gemini-2.5-flash (Comanici et al., 2025): Used for non-cited statement fact-checking with web access enabled. These are chosen for their native web search capabilities.

The choice to use different models for different evaluation sub-tasks raises a subtle methodological question: are the evaluators biased in favor of or against particular evaluated systems? For example, if gpt-4o (the evaluator for citation consistency) shares architectural similarities with o3 (one of the evaluated base models), it might be more "lenient" when verifying o3's outputs. The paper does not discuss this potential evaluator bias.

Evaluated Deep Research products. Two commercial systems are evaluated:

• OpenAI Deep Research: The paper specifies that the "standard version of Deep Research, powered by the o3 model" was used. Responses were manually collected from the web-based interface during July 14–25, 2025.

• Gemini Deep Research: The paper specifies that both the "Gemini 2.5 Pro" and "Deep Research" toggles were enabled on the web interface to activate full research capabilities. Responses were collected during the same period.

Both products were used through their web interfaces, meaning the paper has no visibility into the internal architecture, prompting strategies, or tool-use configurations — the evaluation treats these systems as black boxes.

Evaluated base models. Four base models are evaluated: o3, gemini-2.5-pro, gemini-2.5-flash, and claude4-sonnet. Since these models lack native internet access, they are augmented with external tools:

• SerpAPI: Used for Google Search access. This provides the model with the ability to issue search queries and receive results (snippets, URLs, metadata).

• Firecrawl: Used for retrieving web pages in Markdown format. This provides the model with the ability to fetch and read the full content of specific URLs returned by SerpAPI.

The integration uses each model's native function call interface. This means the model can decide when to invoke search (based on its own judgment of when external information is needed) and when to read retrieved pages. Each model is capped at a maximum of five tool calls per instance. This cap is imposed "due to context length limitations" — reading multiple full web pages into the model's context window quickly consumes the available token budget, particularly for models with smaller context windows.

The five-call cap is a significant constraint. A research task might require searching for information across dozens of papers, reading their abstracts, and synthesizing findings. With only five tool calls, the model must be strategic about which searches to issue and which pages to read. This constraint likely explains part of the performance gap between base models and Deep Research products: the commercial products almost certainly perform many more than five search-and-retrieval operations internally, using iterative search strategies that the five-call cap prevents base models from replicating. The paper acknowledges this implicitly when noting that Deep Research products "significantly outperform their base models in coverage and factual grounding" (Section 3.4).

Data collection period. All responses were collected between July 14 and July 25, 2025. This bounded collection window is important for reproducibility: web search results and model behavior can change over time, so fixing the evaluation window ensures that all systems are evaluated under comparable conditions. A system evaluated in July 2025 might perform differently than the same system evaluated in September 2025 due to model updates, changes in web content, or modifications to the search APIs.

Why these specific systems? The paper does not provide a systematic justification for the selection of evaluated systems. The choice of OpenAI and Gemini Deep Research is natural — they are the most prominent commercially available Deep Research products. The choice of base models (o3, gemini-2.5-pro, gemini-2.5-flash, claude4-sonnet) covers the major model families from the leading AI labs, but the absence of other strong models (e.g., Claude 3.5 Opus, Llama 3 variants, or open-weight models with search augmentation) limits the generality of the base model comparison. The paper's findings about base models should be interpreted as characterizing the specific models tested, not as a universal claim about the gap between base models and Deep Research agents.

No training or fine-tuning. A notable absence from the methodology is any training or fine-tuning of evaluator models for the specific tasks they perform in the pipeline. All evaluation steps use off-the-shelf models with prompt engineering alone. This is a deliberate design choice that prioritizes ease of adoption and reproducibility — anyone with API access to gpt-4o and Gemini can replicate the evaluation — over the potentially higher accuracy of task-specific fine-tuned verifiers. The trade-off is that evaluator reliability depends entirely on the zero-shot/few-shot capabilities of the chosen models, which may vary across domains, statement types, and evaluation sub-tasks.

4. Key Insights and Innovations

Innovation 1: Survey Papers as Gold-Standard Ground Truth via Reverse Prompt Engineering

The most conceptually distinctive move in this paper is not the evaluation metrics themselves — precision, recall, and factuality verification all have precedent in the literature — but rather the method for obtaining ground truth without human annotation. The paper's key insight is that a published survey paper is the ideal answer to a research question we can reconstruct from the paper itself. By running this reconstruction in reverse — giving an LLM the survey's full text and publication date and asking it to generate the research prompt that would have motivated that survey — the paper creates a test instance where the ground-truth answer (the set of references the report should cite, and the factual content it should cover) is already known.

This framing is fundamentally different from how prior benchmarks construct evaluation tasks:

• Fact-checking benchmarks like FEVER (Thorne et al., 2018) and SciFact (Wadden et al., 2020) require human annotators to write claims and verify them against source documents. This creates high-quality data but does not scale — each claim requires expert effort to construct and verify. ReportBench's reverse-engineering approach eliminates this bottleneck: for any survey paper that passes the filtering criteria, a test prompt can be automatically generated.
• Survey generation benchmarks like SurveyBench (Yan et al., 2025) evaluate generated surveys against human-written outlines and reference lists, but the evaluation dimensions are broader (including writing quality and structure) and the construction pipeline is less automated. ReportBench's narrower scope — focusing exclusively on objectively verifiable dimensions — is what enables full automation.
• Deep Research evaluation benchmarks like Deep Research Bench (Bosse et al., 2025; Du et al., 2025) evaluate end-to-end report generation but typically use coarse-grained metrics that don't decompose into per-reference and per-statement verification. They also don't leverage the reverse-engineering trick to obtain ground-truth bibliographies.

The innovation here is architectural, not algorithmic. The paper does not propose a new model, a new training procedure, or a new metric formula. It proposes a new pipeline design pattern for evaluation — one that converts the expensive supervised learning problem of "evaluate whether this AI-generated report on topic X is good" into the self-supervised problem of "evaluate whether this AI-generated report on topic X recovers the references and factual content that a human expert survey on topic X contains." This is a clever reduction because it leverages the massive existing corpus of expert-written surveys on arXiv as a source of labeled data, sidestepping the annotation bottleneck that has limited prior evaluation efforts.

The scalability argument is worth examining carefully. The paper constructs 678 survey tasks from the initial arXiv corpus and downsamples to a balanced 100-prompt benchmark. But in principle, this pipeline could be run on every survey paper on arXiv that meets the filtering criteria, producing thousands of test prompts with zero additional human labeling. The only human-annotation step is the expert validation of domain classifications on the final 100-prompt subset, and even this is a quality check rather than a construction step. This contrasts sharply with benchmarks like FEVER, where every claim must be written and verified by humans, or SurveyBench, where human outlines serve as the backbone of evaluation. The practical implication is that ReportBench can scale with the growth of the academic literature — as more survey papers are published, more test tasks can be generated automatically.

The trade-off this innovation introduces is equally important to recognize. By using survey papers as the answer key, ReportBench evaluates whether an agent can reconstruct what a human expert survey contains — but this is not the same as evaluating whether the agent produces the best possible survey on a topic. A survey published in 2023 reflects the state of the field as understood by its authors in 2023, with their specific perspective, their specific selection of what to include and exclude, and their specific narrative framing. A Deep Research agent might produce a survey in 2025 that is genuinely superior to the 2023 human survey — including more recent work, identifying connections the human authors missed, or covering sub-areas the human authors chose to de-emphasize — and ReportBench would penalize it for deviating from the ground-truth reference list. The benchmark measures faithfulness to an existing expert synthesis, not the absolute quality of a novel synthesis. This is not a flaw per se — the paper is explicit that it targets "factual faithfulness and correctness" rather than novelty or insight — but it means ReportBench is a necessary rather than sufficient condition for research quality assessment.

The evidence that this pipeline actually works as intended comes from the experimental results in Table 1. The fact that precision scores are non-trivial (0.385 for OpenAI, 0.337 for claude4-sonnet) while recall scores are consistently very low (0.033 and 0.021, respectively) indicates that agents are finding some of the right papers but missing most of them. If the pipeline were fundamentally broken — if the reverse-engineered prompts were incoherent or the temporal constraints prevented agents from finding any relevant literature — both precision and recall would be near zero. The fact that precision is substantially above zero validates that the reverse-engineered prompts are meaningful and that the evaluation framework captures a real signal. The low recall, meanwhile, is itself a finding: current agents cannot comprehensively cover the literature that an expert survey would cite, which is a genuine limitation of current technology that the benchmark successfully reveals.

Innovation 2: The Cited/Non-Cited Statement Decomposition as an Evaluation Primitive

The paper's second conceptual contribution is its insistence that cited and non-cited statements in a research report require fundamentally different verification strategies, and that evaluating them through a single pipeline (or collapsing them into a single factuality score) would mask distinct failure modes. This decomposition is not just a practical engineering choice — it reflects a deeper insight about the nature of attribution in knowledge synthesis.

To understand why this decomposition is novel, consider how prior evaluation work handles attribution:

• Fact-checking benchmarks (FEVER, SciFact) verify claims against a provided evidence corpus, but they don't distinguish between claims that were attributed to a source by the model being evaluated and claims that the model generated without attribution. The verification task is the same regardless of whether the model cited a source — the evaluator checks external evidence — and the model's citation behavior is not part of the evaluation.
• Citation evaluation tools (Sarol et al., 2024) check whether citations exist, whether they are formatted correctly, and whether cited statements are faithful to their sources. But they don't evaluate non-cited statements at all — a report with perfect citation accuracy could still contain egregious factual errors in its unattributed claims, and citation evaluation would not detect them.
• LLM-as-a-judge evaluations for long-form text often produce a single holistic quality score that blends citation quality, factual accuracy, writing quality, and argumentation into one number. This loses the diagnostic signal that comes from separating different failure modes.

ReportBench's innovation is to treat the citation status of each statement as the primary axis of decomposition for the evaluation pipeline, and to design two entirely different verification architectures on either side of this divide. For cited statements, the verification is closed-world: there is a specific source document, and the task is to check semantic consistency between the report's claim and the source's content. This is a text-to-text comparison that can be decomposed into interpretable sub-steps (extract, retrieve, verify). For non-cited statements, the verification is open-world: there is no designated source, and the task is to determine whether an assertion about the world is true using web search. This requires a fundamentally different mechanism — multi-model voting with web access — that acknowledges the inherent uncertainty of open-world verification.

This decomposition produces a diagnostic capability that holistic evaluation lacks. Consider the results in Table 1:

• OpenAI Deep Research achieves 78.87% citation match rate and 95.83% non-cited factual accuracy. This is a system that is relatively careful about both its sourced claims and its unsupported assertions, with a small gap between the two.
• Claude4-sonnet achieves 73.67% citation match rate and 92.64% non-cited accuracy. Similar pattern, slightly lower on both.
• o3 achieves 31.43% citation match rate and 82.22% non-cited accuracy. This is a dramatically different profile: the model's sourced claims are mostly not supported by the cited sources (only 31% match rate), suggesting systematic hallucination when attributing claims to papers. Its non-cited claims are also weaker but the gap is proportionally smaller.
• gemini-2.5-flash achieves 44.88% citation match rate and a non-cited accuracy not separately highlighted but presumably higher. The citation fidelity problem is the dominant failure mode.

A single holistic factuality score would obscure these differences. o3 and gemini-2.5-flash might receive similar overall scores if non-cited accuracy compensated for poor citation consistency, but the nature of their failures is different, and the remedy is different. A developer trying to improve o3 would focus on its citation grounding mechanism — perhaps better prompting for source adherence or improved retrieval-to-generation alignment — while a developer trying to improve gemini-2.5-flash might focus on different aspects. The decomposition makes the evaluation actionable in a way that aggregate scores are not.

The paper provides concrete evidence for the importance of this decomposition in Section 4, where it presents two qualitative failure modes: statement hallucination (the model attributes a claim to a paper that doesn't actually say what the claim asserts — the author name error in the OpenAI Deep Research example) and citation hallucination (the model fabricates a reference that doesn't exist — the non-existent ResearchGate URL in the gemini-2.5-pro example). These are different bugs that would require different fixes, and they are detectable only because the evaluation separates citation verification from content verification and checks for both.

This conceptual move — treating citation status as the primary decomposer of the evaluation architecture — is not a radical theoretical innovation; it follows naturally from the structure of research reports. But prior work had not made it the organizing principle of evaluation design. ReportBench's contribution is to demonstrate that this decomposition yields measurably different information than holistic approaches, and that this information is practically useful for understanding where current systems fail.

Innovation 3: Citation Completeness as a Distinct and Measurable Evaluation Dimension

The paper identifies a blind spot in prior citation evaluation work and proposes a specific metric to address it: reference recall against an expert-curated bibliography as a measure of citation completeness. This is distinct from existing citation quality metrics (which verify that citations exist and are accurate) and from factuality metrics (which verify that statements are true). It asks a different question: of all the papers that should be cited in a comprehensive survey on this topic, how many did the agent actually find and cite?

Prior citation evaluation work, as the paper notes in Section 5, has focused on three dimensions:

• Syntactic verification: Is the citation formatted correctly? Does the DOI resolve? Is the author list complete? (Barrot, 2025).
• Existence verification: Does the cited paper actually exist? Is it a real publication or a hallucination? (Bairagi & Lihitkar, 2024).
• Semantic verification: Does the cited paper actually say what the report claims it says? (Sarol et al., 2024).

What's missing from this list is coverage — whether the set of cited references is the right set, not just whether each individual citation is correct. The paper's argument, made explicitly in the Section 5 discussion of citation evaluation, is that "citation completeness—i.e., whether all relevant prior work on a given research topic has been cited—remains underexplored." This is a subtle claim because it's not obvious that completeness should be an evaluation criterion. A short report that accurately summarizes the three most important papers on a topic might be more useful than a comprehensive report that lists 100 papers but says little about each. The paper's response to this objection is implicit in its experimental design: ReportBench reports completeness (recall) as a separate metric from accuracy (precision, citation match rate) rather than combining them, allowing users to decide how much they value coverage versus accuracy for their specific use case.

The innovation here is not the recall metric itself — information retrieval has used precision and recall for decades — but rather the operationalization of citation completeness for research report evaluation through the survey-paper-as-ground-truth construction. Before ReportBench, measuring citation completeness would have required human experts to enumerate, for each research topic, all the papers that a comprehensive survey should cite — a prohibitively expensive annotation task. The reverse prompt engineering approach solves this: the survey paper's bibliography is the expert-annotated list of relevant papers, created by domain experts through the normal process of writing and publishing a survey. This transforms an expensive supervised labeling problem into a data extraction problem.

The experimental results reveal that citation completeness is, quantitatively, the dimension where current systems perform worst. Table 1 shows:

• OpenAI Deep Research recall: 0.033
• Gemini Deep Research recall: 0.036
• claude4-sonnet recall: 0.021

Given that the average ground-truth bibliography contains 153 references, these recall scores correspond to finding approximately 3–5 of the expected papers. Even the best system finds only about 3.6% of the literature that an expert survey would cite. This is a striking finding that would be invisible in benchmarks that only measure precision or per-statement accuracy. A system could achieve 100% precision and 100% citation match rate while citing only one paper — it would score perfectly on existing citation quality metrics while being essentially useless as a research survey.

The paper anticipates the objection that recall is an unfair metric because a generated report is typically much shorter than a published survey (and therefore cannot cite 153 papers), and addresses it by noting that "many citations supporting the same or overlapping statements" contribute to the high ground-truth reference count, and that "recall should be considered a secondary signal rather than the primary focus of evaluation" (Section 3.3). This is a defensible position — recall is harder to interpret than precision because the "right" number of citations depends on report length and intended use — but the paper's decision to report recall rather than suppress it is a demonstration of the benchmark's philosophy: disaggregate, report everything, and let users decide which metrics matter for their application.

The practical significance of measuring completeness is that it reveals a fundamental capability gap in current Deep Research agents that precision-only evaluation would mask. These agents are reasonably good at verifying that the papers they cite are relevant (precision of 0.385 for OpenAI means over a third of cited papers match expert judgment), but they are extremely poor at ensuring they haven't missed important work. For applications where comprehensiveness matters — academic literature reviews, regulatory submissions, evidence-based policy reports — this is a critical limitation. For applications where a quick, high-precision overview suffices (a journalist's background research, a product manager's competitive landscape), high precision with low recall might be perfectly adequate. The value of ReportBench's design is that it surfaces this distinction quantitatively rather than burying it in a composite score.

Innovation 4: Interpretable, Auditable Verification Pipelines as a Design Principle

The paper's evaluation architecture embodies a methodological principle that is more often advocated than implemented: verification should be decomposable into interpretable sub-steps whose intermediate outputs can be inspected by humans. This is not a performance innovation — decomposing verification into stages almost certainly does not improve accuracy relative to an end-to-end LLM judge, and may introduce error propagation where mistakes in early stages cascade — but a transparency innovation that shifts the trust model for automated evaluation.

To appreciate what's distinctive here, it's useful to contrast ReportBench's cited statement verification pipeline with the dominant "LLM-as-a-judge" paradigm:

• LLM-as-a-judge: The evaluator LLM receives the report, the cited source (if any), and an evaluation rubric, and produces a score plus optionally a justification. This is simple to implement and flexible across tasks, but the justification is post-hoc — generated after the score is determined — and may not faithfully reflect the actual reasoning process. There is no way to verify whether the LLM actually compared the statement to the correct passage in the source, or whether it made a superficial judgment based on surface-level features. This makes the reliability of the evaluation dependent entirely on trust in the evaluator model.

• ReportBench's three-stage pipeline: The evaluator first extracts statement-citation mappings (output: a structured list of which statements cite which URLs). Then it retrieves the most relevant supporting passage from each cited source (output: a specific text snippet). Then it compares the statement to that retrieved passage (output: a consistency judgment). Each stage produces an explicit, inspectable artifact. If a human auditor questions a particular consistency judgment, they can examine the exact passage that the statement was compared against and decide whether the pipeline made an error at the retrieval stage (wrong passage) or the verification stage (correct passage but wrong judgment).

This architecture makes the evaluation debuggable in a way that end-to-end LLM judging is not. The paper's qualitative analysis in Section 4 demonstrates this: the authors manually inspected evaluation results and identified specific failure categories (statement hallucination, citation hallucination) that would be difficult to diagnose from a single aggregate score. A system receiving a low citation match rate could be failing because (a) it genuinely misrepresents sources, (b) the retrieval stage is finding the wrong passage in the source document, or (c) the verification stage is being overly strict in its consistency judgments. Without the intermediate artifacts, these are indistinguishable; with them, a developer can diagnose which component needs improvement.

The multi-model voting mechanism for non-cited statements applies the same principle in a different form. Rather than trusting a single model's web-based fact-checking, the pipeline collects six independent judgments (two models × three passes each) and applies majority voting. The proportion of votes serves as a confidence score, and a human auditor can examine cases where the vote was close (e.g., 4-2 or 3-3) to assess whether the disagreement reflects genuine ambiguity or evaluator error. This is a form of ensemble-based uncertainty quantification applied to evaluation — the spread of judgments across models and runs provides a signal about how reliable the evaluation is for each individual statement, which a single-model judgment cannot provide.

The paper does not claim that this architecture produces more accurate evaluations than a monolithic LLM judge. In fact, it's possible that a carefully prompted gpt-4o asked to holistically evaluate citation consistency would agree more often with human expert judgments than the three-stage pipeline, especially if errors propagate across stages. The paper's claim is different: this architecture produces more trustworthy evaluations because it makes the evaluation process transparent. For a benchmark that aims to be used by the research community to compare systems and track progress, trustworthiness of the metric may be more important than its raw correlation with human judgment. If users don't trust the metric, they won't use the benchmark; if the metric is transparent, users can calibrate their trust based on their own inspection of intermediate outputs.

This is a methodological contribution rather than a technical one — it's about how to design evaluation frameworks for complex generative outputs, not about what specific model or algorithm to use. The design pattern (decompose verification into inspectable sub-steps; use ensemble voting where ground truth is unavailable; retain intermediate outputs for optional human audit) is reusable beyond research report evaluation to any domain where the output is long-form, multi-source, and requires verification against external evidence. The specific implementation (gpt-4o for extraction and verification, Gemini models for web-based fact-checking) is a particular instantiation; the principle is the innovation.

5. Experimental Analysis

Evaluation Methodology

• Dataset. ReportBench consists of 100 research prompts derived via reverse prompt engineering from 100 peer-reviewed survey papers on arXiv, balanced across 11 application domains and validated by four research experts. Each prompt is paired with a ground-truth reference list extracted from the survey paper's LaTeX source, averaging 153 references per paper.

• Base model(s). Two commercial Deep Research agents are evaluated through their web interfaces: OpenAI Deep Research (powered by o3) and Gemini Deep Research (with Gemini 2.5 Pro and Deep Research toggles enabled). Four standalone base models—o3, gemini-2.5-pro, gemini-2.5-flash, and claude4-sonnet—are augmented with SerpAPI for Google Search and Firecrawl for web page retrieval, capped at five tool calls per instance due to context window constraints.

• Metrics. The framework produces six metrics: reference precision (fraction of a generated report's cited references that appear in the ground-truth bibliography), reference recall (fraction of ground-truth references successfully retrieved by the agent), average reference count per report, cited statement count (average number of citation-bearing claims), citation match rate (proportion of cited statements judged semantically consistent with their source documents), non-cited statement count (average number of unattributed factual claims), and non-cited factual accuracy (proportion of non-cited statements verified as correct via web-connected multi-model voting). The paper also reports an "Average Score" for cited statements whose exact definition is not specified but appears to be a continuous per-statement alignment score.

• Baselines. The primary baselines are the four standalone base models augmented with search tools (o3, gemini-2.5-pro, gemini-2.5-flash, claude4-sonnet). Within the Deep Research product analysis, OpenAI and Gemini serve as baselines for each other. The paper does not compare against prior survey generation systems (AutoSurvey, HiReview, SurveyForge) in the experimental section; the relationship to these systems is discussed in Section 5 (Related Work) but not benchmarked.

• Generation budget / compute accounting. For Deep Research products, there is no budget control—the agents are used as deployed, with their internal search and reasoning budgets determined by the product. For base models, the budget is a maximum of five tool calls (search or page retrieval operations) per instance, imposed by context length limitations. The paper does not measure or report the actual number of search queries, pages read, or tokens generated by the Deep Research products internally.

• Cross-validation / statistical protocol. The paper does not employ cross-validation. The 100-prompt test set is a single fixed split, validated by four research experts for domain classification quality. No confidence intervals, standard deviations, or statistical significance tests are reported for any metric in Table 1. The data collection period (July 14–25, 2025) serves as a temporal control: all systems are evaluated within the same window to ensure comparable web search conditions.

Main Quantitative Results

Product-Level Comparison: OpenAI vs. Gemini Deep Research

Table 1 is the central experimental artifact. The headline finding is that OpenAI Deep Research achieves substantially higher precision (0.385 vs. 0.145) and citation match rate (78.87% vs. 72.94%) than Gemini Deep Research, while Gemini generates more than three times as many cited statements (96.2 vs. 88.2 per report) and more references (32.42 vs. 9.89 per report).

Breaking this down dimension by dimension:

Reference retrieval quality. OpenAI's precision of 0.385 means that over a third of the references it cites—roughly 3.8 out of its average 9.89 references per report—are papers that appear in the expert survey's ground-truth bibliography. Gemini's precision of 0.145 means that fewer than one in seven of its cited references matches the ground truth, despite citing far more references overall (32.42 per report). Since Gemini generates roughly 3.3× more references but achieves less than half the precision, many of its additional citations are either irrelevant to the topic or point to sources outside the survey's scope. The paper characterizes this as Gemini "tend[ing] to over-generate citations without proportionally improving the coverage of high-quality references" and notes that "excessive citation may even introduce redundancy or dilute the relevance of retrieved content" (Section 3.3).

Recall is low for both systems and the gap is negligible: OpenAI achieves 0.033, Gemini 0.036. With an average ground-truth bibliography of 153 references, this translates to approximately 5 references recovered by either system. The paper explicitly frames recall as "a secondary signal rather than the primary focus of evaluation" because many of the 153 ground-truth references support overlapping claims, and a survey citing the 5 most central papers may be functionally adequate even though it misses 148 others.

Cited statement quality. Both systems perform reasonably well on citation faithfulness, but OpenAI holds a clear edge. OpenAI's citation match rate of 78.87% means roughly 4 out of 5 claims attributed to sources are actually supported by those sources; Gemini's 72.94% means roughly 3 out of 4 are supported. This 5.9 percentage point gap is meaningful at the scale of these systems—it translates to OpenAI producing approximately 15–20 more accurate cited claims per report than Gemini, assuming comparable total cited statement counts.

The paper also reports an "Avg Score" for cited statements (OpenAI: 88.2; Gemini: numbers not explicitly stated in the text for this metric but appear in Table 1). The relationship between this continuous score and the binary match rate is not defined, making it unclear whether they measure the same underlying phenomenon at different granularities or capture different aspects of citation quality.

Non-cited statement quality. Both products achieve high factual accuracy on unattributed claims, with OpenAI leading: 95.83% vs. 92.21%. Gemini compensates for lower per-statement accuracy by generating more non-cited statements overall (49.6 vs. 38.9 per report). This means that in absolute terms, both systems produce a comparable number of correct uncited claims (OpenAI: approximately 37.3 per report; Gemini: approximately 45.7 per report), but OpenAI makes fewer incorrect unattributed assertions (approximately 1.6 vs. 3.9 per report).

The precision-recall trade-off interpretation. The paper interprets these results as evidence that OpenAI and Gemini pursue fundamentally different research strategies: OpenAI prioritizes precision—citing fewer sources but ensuring those sources are relevant and accurately represented—while Gemini prioritizes breadth, generating many more citations and statements at the cost of lower per-item accuracy. The paper's language frames OpenAI's approach as preferable: "OpenAI achieves significantly higher precision," "OpenAI maintains a high average alignment score, suggesting stronger precision in citation usage." Whether this framing is objectively justified depends on the use case. A user who needs a comprehensive bibliography might prefer Gemini despite its lower precision; a user who needs a trustworthy summary of key papers might prefer OpenAI. The benchmark's value is that it makes this trade-off visible.

Model-Level Comparison: Deep Research Products vs. Their Base Models

The most consistent pattern in Table 1 is that Deep Research products dramatically outperform their corresponding base models on statement generation volume and citation match rate, even when retrieval performance (precision/recall) is similar. This is the paper's primary evidence that the Deep Research systems incorporate substantial additional engineering beyond simply wrapping a base model with search tools.

OpenAI Deep Research vs. o3. The retrieval performance is similar: precision (0.385 vs. 0.299) and recall (0.033 vs. 0.031) differ only modestly, and the average reference count is comparable (9.89 vs. 12.26). The paper interprets this as validation of "OpenAI's official disclosure that the retrieval and synthesis backbone of Deep Research is powered by the o3 model"—if Deep Research used an entirely different retrieval architecture, we might expect larger differences in retrieval metrics.

The divergence appears in statement generation and faithfulness, and it is dramatic:

• Cited statements: Deep Research averages 88.2 vs. 16.16 for o3—a 5.5× increase. This is not simply "o3 produces shorter reports"; the reports are of comparable overall length, but Deep Research attributes far more of its claims to specific sources rather than presenting them as general knowledge.
• Citation match rate: Deep Research achieves 78.87% vs. 31.43% for o3. This is the most striking gap in the entire table. o3's base model, when augmented with search tools, attributes claims to sources that do not support those claims nearly 70% of the time. This is not a retrieval failure—the sources are being found and cited—but a grounding failure: the model cannot reliably align what it writes with what the cited source says.
• Non-cited factual accuracy: Deep Research achieves 95.83% vs. 82.22% for o3. Even when making unattributed claims, Deep Research is substantially more reliable.

The paper's interpretation is that "Deep Research is not a direct output of o3, but rather likely incorporates an additional writing module, possibly optimized via fine-tuning or structured pipelines" (Section 3.4). This is a reasonable inference from the data, but it is speculation—the paper has no visibility into OpenAI's internal architecture. Alternative explanations consistent with the data include: (a) Deep Research uses o3 with extensive prompting and iterative refinement that the five-call-capped base model evaluation cannot replicate; (b) Deep Research uses a different decoding strategy (e.g., lower temperature, constrained generation) that improves grounding; (c) Deep Research applies post-hoc filtering or correction that the base model evaluation does not. The data cannot distinguish these hypotheses.

Gemini Deep Research vs. gemini-2.5-pro. The pattern mirrors the OpenAI comparison but with even larger magnitude differences:

• Retrieval trade-off inversion: Gemini Deep Research achieves lower precision (0.145 vs. 0.269) but higher recall (0.036 vs. 0.010) compared to gemini-2.5-pro. The base model is more selective but less comprehensive; Deep Research sacrifices per-reference relevance for broader coverage. Reference count diverges massively: 32.42 vs. 4.27 per report—a 7.6× increase.
• Statement generation: Deep Research produces 96.2 cited statements vs. 6.58 for the base model (14.6×) and 49.6 non-cited statements vs. 9.35 (5.3×). The 14.6× increase in cited statements far exceeds the 7.6× increase in references, suggesting that Deep Research generates more claims per cited paper as well as citing more papers overall.
• Citation match rate: Deep Research achieves 72.94% vs. 59.24% for the base model. The base model is better at citation grounding than o3 (59% vs. 31%) but still substantially worse than its Deep Research counterpart.
• Non-cited accuracy: Here the pattern reverses: gemini-2.5-pro achieves higher factual accuracy on unattributed claims (96.08% vs. 92.21%). This is the only metric where a base model outperforms its Deep Research product counterpart. The paper does not comment on this reversal, but it is noteworthy because it suggests the Deep Research pipeline may introduce factual errors in unattributed content (perhaps through more aggressive synthesis or extrapolation) that the base model's more conservative output avoids.

The paper attributes these gaps to "targeted optimization for thorough research and report generation" and speculates—based on the visible "plan" and "step-by-step reasoning" phases in the Gemini Deep Research web interface—that "the system functions more like a thoughtfully constructed multi-agent workflow or pipeline." Again, this is inference from external behavior rather than architectural knowledge.

Base Model Inter-Comparison

Among the four standalone base models evaluated with search tool augmentation (Table 1), claude4-sonnet is identified as the strongest overall:

• Precision: 0.337, second only to o3 (0.299 for gemini-2.5-pro, 0.237 for gemini-2.5-flash).
• Recall: 0.021, moderate among the group (o3 leads at 0.031).
• Reference count: 6.74 per report, moderate.
• Citation match rate: 73.67%, substantially better than o3 (31.43%), gemini-2.5-pro (59.24%), and gemini-2.5-flash (44.88%). This is claude4-sonnet's standout metric—it is the only base model whose citation faithfulness approaches Deep Research product levels (78.87% for OpenAI Deep Research).
• Non-cited accuracy: 92.64%, comparable to the Deep Research products and better than o3 (82.22%).

The paper characterizes claude4-sonnet as demonstrating "the most balanced performance." The implication is that citation grounding ability varies substantially across base model families—the 31.43% match rate for o3 is catastrophically low and suggests that o3's function-calling interface with search tools produces systematically unreliable citation behavior, while claude4-sonnet's interface appears to be substantially more reliable—but the paper does not investigate why this variation exists (e.g., differences in training data, instruction tuning, or function-calling implementation).

The most striking negative result in the base model comparison is o3's citation match rate of 31.43%. When o3 makes a claim and attributes it to a source, the source supports that claim less than one-third of the time. The paper does not analyze whether these failures are primarily due to (a) the model hallucinating paper content, (b) the model citing a paper that is topically relevant but does not contain the specific claim, or (c) the retrieval step in the evaluation pipeline (supporting source extraction) failing to locate the correct passage. The qualitative examples in Section 4 provide anecdotal evidence for type (a)—the OpenAI Deep Research example where "Kulkarni" is incorrectly attributed as an author—but systematic failure analysis is absent.

Qualitative Failure Analysis

Section 4 provides two representative error instances from manual inspection that concretize the quantitative findings:

Statement hallucination (OpenAI Deep Research, arXiv:2407.15186 test case). The agent generates a claim about "Kulkarni et al. (2025)" introducing RL fine-tuning for Text-to-SQL, citing paper arXiv:2503.23157v2. The cited paper does describe an RL reward scheme for Text-to-SQL, but does not include "Kulkarni" as an author. The paper notes that Kulkarni did publish on RL + Text-to-SQL, just not in this specific paper. The interpretation: "the model may have encountered similar data during training and mistakenly attributed Kulkarni's contribution to this cited paper." This is a cross-document entity confusion error where the model conflates two distinct but topically related papers.

Citation hallucination (gemini-2.5-pro, arXiv:2009.12619 test case). The agent generates a claim about vision-based in-vehicle crowd monitoring and cites a ResearchGate URL. The URL does not resolve—it is fabricated. Because no source content can be retrieved, the evaluation pipeline flags this as a citation mismatch. The paper classifies this as a "common error mode in function-call–driven retrieval: the model confidently invents plausible-looking reference links that nonetheless point to nothing."

These examples serve two purposes in the paper's argument: (a) they validate that the citation semantic consistency metric captures genuine failures rather than evaluation artifacts (the metric declined for these cases during evaluation, and manual inspection confirmed the errors), and (b) they illustrate that the failure modes differ qualitatively across systems (OpenAI misattributes authorship while pointing to real papers; Gemini fabricates entire references). The paper does not provide quantitative breakdowns of error types—we don't know what fraction of citation consistency failures are statement hallucinations vs. citation hallucinations vs. retrieval-stage errors in the evaluation pipeline itself.

Ablation Studies and Robustness Checks

The paper contains no formal ablation studies. There is no systematic investigation of how evaluation results change with different evaluator LLM choices, different voting thresholds for non-cited statement verification, different title-matching algorithms for reference overlap, or different prompt templates for the evaluation stages. The evaluation pipeline is presented as a fixed configuration without sensitivity analysis.

Several configuration choices function as implicit ablations in the experimental design, though they are not framed as such:

Deep Research product vs. base model comparison as an implicit pipeline ablation. By comparing OpenAI Deep Research (which incorporates internal retrieval, planning, and writing modules) with o3 augmented only with search and link-reading tools (capped at 5 tool calls), the experiment implicitly ablates the effect of the Deep Research pipeline beyond the base model's raw capabilities. The dramatic performance differences (5.5× more cited statements, 2.5× higher citation match rate) demonstrate that the pipeline matters enormously, but the experiment cannot isolate which component of the pipeline (search budget, iterative reasoning, specialized writing module, post-hoc verification) drives the improvement.

Three prompt granularities as an implicit prompt diversity check. The dataset construction generates three prompt types per survey paper (sentence-level, paragraph-level, richly detailed) and randomly samples from these when constructing the 100-prompt test set. This ensures that results are not dominated by a single prompt style. However, the paper does not report performance breakdowns by prompt granularity—we cannot assess whether some models perform better with more constrained prompts and others with more open-ended ones.

Multi-model voting with repeated judgments as an implicit reliability check. The non-cited statement verification collects three independent judgments per model (gemini-2.5-pro and gemini-2.5-flash). The fact that these are treated as separate votes implies that intra-model consistency is not perfect—if the same model always produced the same judgment on the same statement, collecting three judgments would add no information. The paper does not report intra-model agreement rates, inter-model agreement rates, or the distribution of vote splits (6-0, 5-1, 4-2, 3-3), which means we cannot assess how often the multi-model voting resolves uncertainty versus how often it simply amplifies systematic biases shared by both Gemini models.

Two evaluator models as an implicit evaluator diversity check. The choice to use gpt-4o for cited statement verification and Gemini models for non-cited verification means that the two verification pipelines use different evaluator families with different potential biases. The paper does not cross-validate by running cited statement verification with Gemini or non-cited verification with gpt-4o, so we cannot determine whether observed performance differences reflect genuine differences in the evaluated systems or artifacts of the evaluator assignment.

Absence of key ablations that would strengthen the paper:

• Evaluator model swap. Running the cited statement pipeline with Gemini models (or claude) instead of gpt-4o would assess evaluator robustness. If citation match rate rankings remain stable across evaluator models, the metric is likely capturing a real signal; if rankings flip, the metric is evaluator-dependent.

• Voting threshold sensitivity. The paper uses majority voting (4 of 6) for non-cited statements but does not report how factual accuracy numbers change under stricter (6 of 6) or more lenient (3 of 6) thresholds. This would reveal whether the reported accuracies (92–96%) are robust or highly sensitive to the threshold choice.

• Retrieval-stage quality assessment. The three-stage cited statement pipeline depends on the supporting source extraction step (Stage 2) correctly locating the relevant passage in the source document. The paper does not measure the accuracy of this retrieval step independently—we don't know how often a "mismatch" verdict is due to the pipeline retrieving the wrong passage rather than the statement genuinely contradicting the source.

• Prompt template sensitivity. The three evaluation stages (cited statement extraction, supporting source retrieval, semantic consistency verification) each use specific prompts (shown in Appendix A.2). The paper does not report whether results are stable under prompt variations—a critical concern for LLM-based evaluation where prompt sensitivity is well-documented.

• Domain-specific breakdown. Figure 2 shows the domain distribution of the benchmark, and the paper mentions domain classification as enabling "granular analysis," but no domain-specific results are reported. An ablation showing which domains are hardest for current systems (e.g., do biology surveys produce more hallucinations than NLP surveys?) would add practical value and test the robustness of the aggregate numbers.

Critical Assessment

The experiments in this paper demonstrate a specific and valuable capability: ReportBench can distinguish between the output quality of different Deep Research agents and their base models along dimensions that are automatically measurable. The evidence for this claim, presented in Table 1 and the qualitative examples in Section 4, is consistent and interpretable. However, the paper makes broader claims in its introduction and conclusion—about ReportBench serving as a "systematic benchmark to evaluate the content quality of research reports" and enabling "consistent evaluation of AI-generated research reports across multiple dimensions"—that the reported experiments only partially support.

What the experiments actually demonstrate vs. what they claim to demonstrate:

The experiments demonstrate that the evaluation pipeline produces non-trivial, interpretable measurements that differentiate systems in ways that align with qualitative observations (the commercial products outperform base models, OpenAI is more precise than Gemini, claude4-sonnet is the strongest base model). The measurements are face-valid: it is plausible that OpenAI Deep Research really does cite more relevant references and make more faithful claims than Gemini Deep Research, and the evaluation pipeline captures this.

What the experiments do not demonstrate is that these measurements correlate with human expert judgments of report quality. The paper never compares ReportBench's automated metrics against human evaluations of the same reports. This is a critical gap because the entire benchmark's validity rests on the assumption that precision, recall, citation match rate, and non-cited accuracy are good proxies for what a human domain expert would consider a "high-quality research report." Several failure modes are possible:

• High citation match rate but poor synthesis. A report could perfectly describe each cited paper's contribution individually while completely failing to identify connections, contradictions, or trends across papers—the core synthetic contribution that makes a survey useful. ReportBench's per-statement verification would give this report a high score while a human expert might judge it as shallow.
• High precision but poor reference selection. A report citing 5 perfectly relevant papers (precision = 1.0) might miss the 3 most important papers in the field because they were published in venues that the agent's search didn't index. A human expert would recognize this as a critical omission; ReportBench's recall metric would capture it, but only if the survey paper used as ground truth happens to cite those 3 papers—and the paper argues recall should be a "secondary signal."
• High factual accuracy on non-cited statements but misleading framing. A report could make numerous factually true statements about a field while presenting a narrative that is fundamentally misleading (e.g., suggesting a consensus where there is active debate, or omitting well-known limitations of a popular method). The web-based verification pipeline would validate each individual claim while missing the higher-level distortion.

The paper acknowledges these scope limitations in Section 1 ("leaving the assessment of writing quality to future work") and Section 5 (positioning ReportBench as focusing "solely on well-defined and automatically verifiable dimensions"), but the framing throughout the paper—"content quality," "comprehensive evaluation," "systematic benchmark"—implies a broader validity than the experiments support.

The missing human baseline. The most important experiment not run is a correlation study between ReportBench's automated metrics and human expert evaluations of the same generated reports. Without this, we cannot answer the most basic question about the benchmark: does a system that scores higher on ReportBench actually produce reports that domain experts prefer? This omission is particularly notable because the paper goes to substantial effort to establish survey papers as gold-standard references—implicitly claiming that matching an expert survey's reference list and factual content is a valid proxy for quality—but never validates this proxy against the thing it is proxying for. A study where, say, 3 NLP researchers rate 20 AI-generated reports on a topic and these ratings are correlated against ReportBench's metrics would transform the paper from "we built a pipeline that produces numbers" to "we built a pipeline that produces numbers that align with expert judgment."

The evaluator independence problem. The evaluation pipeline is itself implemented using the same family of models being evaluated (gpt-4o is an OpenAI model; the evaluated OpenAI Deep Research is powered by o3; gemini-2.5-pro serves both as an evaluated base model and as a web-connected verifier for non-cited statements). This creates a potential for evaluator bias that the paper does not address:

• When gpt-4o evaluates citation consistency for OpenAI Deep Research's outputs, is it more lenient than when evaluating Gemini's outputs? The models share architectural heritage and training data, which could produce systematic evaluator favoritism.
• When gemini-2.5-pro serves as a fact-checker for non-cited statements from gemini-2.5-pro (the base model evaluation), it is evaluating statements generated by a nearly identical system. Self-evaluation bias in LLMs is well-documented—models tend to rate their own outputs more favorably than outputs from other models.
• The paper could have mitigated this by cross-validating: running a subset of evaluations with evaluator models from different families and checking whether system rankings are stable. This experiment is absent.

The five-call cap as a confound. The base model evaluation caps tool calls at five per instance, while the Deep Research products are used as deployed, with unknown but presumably much larger internal search budgets. This means the base-model-to-Deep-Research comparison conflates two variables: (a) whether the system uses a specialized research pipeline versus raw model + tools, and (b) the search budget. The dramatic performance gap (5.5× more cited statements, 2.5× higher match rate for OpenAI Deep Research vs. o3) could be explained entirely by the search budget difference—o3 with 50 tool calls might close much of the gap, but we cannot know from these experiments. The paper acknowledges the five-call cap ("Due to context length limitations") but does not discuss its impact on the interpretation of the model-level comparison. An experiment where base models are given budget-matched tool calls (even if this means evaluating fewer prompts to stay within compute constraints) would isolate the pipeline effect from the budget effect.

Statistical reliability concerns. The 100-prompt test set, split across 11 domains, means some domains may have fewer than 10 prompts. The paper reports only point estimates in Table 1 (means) with no measures of variance, confidence intervals, or statistical tests. This matters because some of the comparisons the paper draws—particularly the "Gemini generates more than three times as many cited statements" and "OpenAI achieves notably higher citation match rate"—involve ratios of means where the underlying per-prompt variance could be large. A system that generates 200 cited statements on one prompt and 0 on the next would have the same mean as one that generates 88 consistently; without variance information, we cannot assess whether the differences in Table 1 reflect consistent performance differences or are driven by a few outlier prompts.

What the experiments do convincingly demonstrate:

Despite these limitations, the paper does convincingly demonstrate several specific and valuable findings:

1. Automated evaluation at this granularity is technically feasible. The pipeline successfully processes 100 diverse research prompts, extracts references and statements from generated reports, retrieves cited sources, and produces per-statement consistency judgments—all without human intervention except for the domain classification validation. This is an engineering achievement that enables the kind of systematic comparison that was previously impossible without expensive expert review.

2. Deep Research products are qualitatively different from base models with search. Even accounting for the search budget confound, the order-of-magnitude differences in statement generation volume and the large gaps in citation match rate cannot be plausibly attributed to random variation or evaluator noise. Something in the Deep Research pipeline—whether fine-tuning, multi-step reasoning, iterative writing, or all of the above—produces outputs that are structurally different from what base models produce.

3. Citation faithfulness is the single largest quality gap for base models. The finding that o3 achieves only 31.43% citation match rate when augmented with search tools—meaning that nearly 70% of its sourced claims are not supported by the cited sources—is a striking and practically important result. It suggests that simply giving a powerful LLM internet access and asking it to write a research report produces outputs that are dangerously unreliable in their attribution, and that substantial additional engineering is required to achieve the 70–80% match rates of the commercial products.

4. Non-cited factual accuracy is relatively high across all systems, with Deep Research products exceeding 92%. This is an encouraging finding for deployability: when these systems assert facts without attribution, those assertions are usually correct. The primary reliability risk is not general factual errors but citation-grounded errors—claims that are wrong and attributed to a source that does not support them, which is a more insidious failure mode because the citation creates an illusion of verification.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted for in Headline Efficiency Numbers

The assumption or constraint. The entire ReportBench evaluation pipeline depends on having a ground-truth reference list for each test prompt, which is extracted from a published survey paper's LaTeX source. This extraction process—identifying peer-reviewed survey papers, parsing their bibliographies, running GPT-4o classification to filter false positives, and generating reverse-engineered prompts—is performed once during benchmark construction. But the paper also uses the survey paper's full text and bibliography as the implicit answer key against which all metrics (precision, recall, citation match rate) are computed. This means that for any new research topic not covered by an existing published survey paper, the benchmark construction pipeline must be re-executed from scratch. The paper does not discuss the cost of this construction process, nor does it propose a method for evaluating reports on topics where no expert survey exists.

The consequence. The scalability claim—that ReportBench "can scale with the growth of the academic literature" because "as more survey papers are published, more test tasks can be generated automatically"—is true only for topics where surveys already exist. For genuinely novel or emerging research areas, or for interdisciplinary topics that cross traditional survey boundaries, no gold-standard reference list exists, and ReportBench provides no evaluation mechanism. This limits the benchmark's coverage to mature, well-surveyed fields (which is also where the arXiv corpus skews toward STEM). A practitioner evaluating a Deep Research agent on a cutting-edge topic—precisely the kind of topic where trustworthy synthesis is most valuable because no expert survey yet exists—cannot use ReportBench without first commissioning a human expert survey, defeating the purpose of automated evaluation.

What evidence exists in the paper. The paper implicitly acknowledges this constraint through its dataset construction choices: filtering for papers published after 2020-01-01, requiring peer review and formal publication, and applying GPT-4o classification to confirm survey status. These filters select for mature, peer-reviewed survey papers—exactly the type of paper that exists only after a field has stabilized sufficiently for experts to write comprehensive reviews. The paper does not report how many arXiv papers were excluded by these filters, nor does it characterize what fraction of active research topics have associated published surveys. The final benchmark of 100 prompts covers 11 domains, all within the arXiv corpus's STEM-heavy distribution (Section 2.1.3, Figure 2).

Mitigation status. Not addressed. The paper does not propose a method for constructing evaluation tasks in the absence of published surveys, nor does it acknowledge this as a limitation of the approach. The "Future Work" discussion is absent from the main text (there is no explicit future work section), and the conclusion only mentions hoping "ReportBench will serve as a valuable tool for the research community to monitor, compare, and further improve the reliability of AI systems" without discussing domain coverage expansion.

---

No Validation Against Human Expert Judgments of Report Quality

The assumption or constraint. ReportBench evaluates generated reports exclusively through automated metrics: reference precision/recall against a ground-truth bibliography, semantic consistency of cited statements against source documents (judged by gpt-4o), and factual accuracy of non-cited statements (judged by web-connected Gemini models with majority voting). The paper asserts that these metrics capture "content quality," defined as the "quality and relevance of cited literature" and "the faithfulness and veracity of generated statements" (Section 1). But the paper never validates whether these automated metrics correlate with what human domain experts would consider a high-quality research report.

The consequence. The benchmark's metrics could systematically diverge from human quality judgments in ways that are invisible without a human baseline. Several plausible divergence scenarios exist:

• A report could achieve high precision and citation match rate by accurately describing a small set of well-known papers while completely failing to synthesize them—identifying no connections, contradictions, or trends. A human expert would judge this as a poor survey despite high scores on all of ReportBench's dimensions.
• A report could achieve high factual accuracy on non-cited statements by making only safe, obvious claims ("Machine learning has many applications") while avoiding the kind of substantive, potentially contentious synthesis that makes a survey valuable. The benchmark would reward conservatism.
• The temporal cut-off constraint in the prompts ensures that agents cannot cite papers newer than the original survey. A human expert reading a 2025 AI-generated survey might reasonably value the inclusion of very recent (post-survey) papers that the original human survey could not have cited—these would be penalized by ReportBench's precision metric as false positives.

Without a human correlation study, we cannot determine whether systems that score higher on ReportBench actually produce reports that domain experts prefer. The benchmark may be optimizing for the wrong thing.

What evidence exists in the paper. The paper contains no experiment comparing automated metrics against human evaluations. None of the four research experts who validated the domain classification (Section 2.1.3) appear to have been asked to evaluate the quality of generated reports. The qualitative examples in Section 4 are manual inspections that confirm specific errors detected by the pipeline (statement hallucination, citation hallucination), but they are cherry-picked illustrations of pipeline-detected failures, not systematic comparisons of pipeline scores against human quality ratings. The paper provides no inter-annotator agreement data, no correlation coefficients, and no discussion of how the metrics would be validated against ground-truth human judgments.

Mitigation status. Not addressed. The paper implicitly treats the survey paper's reference list and factual content as the ground-truth definition of quality—if an agent recovers the same references and makes factually accurate statements, the report is "good." This is a defensible operationalization but an unvalidated one. The paper does not acknowledge this as a limitation, nor does it discuss the need for human validation studies in future work.

---

The Evaluator Models Are from the Same Families as the Evaluated Systems

The assumption or constraint. The evaluation pipeline uses gpt-4o (an OpenAI model) for cited statement extraction, supporting source retrieval, and semantic consistency verification, and uses Gemini models (gemini-2.5-pro and gemini-2.5-flash) for non-cited statement web-based fact-checking. The evaluated systems include OpenAI Deep Research (powered by o3, which shares architectural lineage with gpt-4o) and Gemini Deep Research (based on Gemini 2.5 Pro). This means the evaluator models and the evaluated systems come from the same AI labs and likely share training data, architectural principles, and possibly evaluation benchmarks.

The consequence. Self-evaluation bias in LLMs is a well-documented phenomenon: models tend to rate outputs from their own family more favorably than outputs from other families, either through explicit preference or through shared blind spots (both models in a family may fail to detect the same types of errors because they were trained on similar data). In ReportBench's configuration, this creates two distinct bias risks:

• gpt-4o evaluating OpenAI Deep Research outputs: If gpt-4o and o3 share similar representations of what constitutes "semantic consistency" or exhibit similar behaviors around citation interpretation, gpt-4o may be more lenient when judging whether an OpenAI Deep Research statement is supported by its cited source, because both models' "understanding" of the source document aligns. This would artificially inflate OpenAI's citation match rate relative to non-OpenAI systems.
• Gemini models evaluating Gemini Deep Research outputs: When gemini-2.5-pro serves as a fact-checker for non-cited statements generated by gemini-2.5-pro (the base model evaluation), it is effectively evaluating outputs from a nearly identical system. A shared factual error in Gemini's training data would go undetected because the evaluator shares the same misconception.

The paper's experimental results are consistent with possible evaluator bias—OpenAI Deep Research achieves the highest citation match rate (78.87%, evaluated by gpt-4o), while Gemini Deep Research achieves the highest recall (0.036, but the recall computation does not involve Gemini models as evaluators, making this metric more trustworthy)—but the paper provides no evidence to distinguish genuine performance differences from evaluator artifacts.

What evidence exists in the paper. The experimental configuration is documented in Section 3.1: "For statement extraction, supporting source extraction, and semantic consistency verification, we adopt gpt-4o. For the fact-checking of non-cited statements, we employ two web-connected models: gemini-2.5-pro and gemini-2.5-flash." The paper does not run any cross-evaluator experiment—for example, verifying a subset of cited statements with a Claude model to check whether the gpt-4o-produced match rates are stable across evaluator families, or using a non-Gemini web-connected model for non-cited statement verification. The paper does not discuss evaluator bias as a potential confound.

Mitigation status. Not addressed. The paper does not acknowledge evaluator-model bias as a methodological concern, does not propose cross-validation across evaluator families, and does not discuss the implications for benchmark trustworthiness. The explicit design choice to retain intermediate outputs for "optional human inspection" (Section 2.2) provides a partial mitigation—a human auditor could spot evaluator bias by manually checking a sample of consistency judgments—but the paper reports no such audit and the benchmark's primary value proposition is automation that avoids the need for manual inspection.

---

The Base Model Comparison Is Confounded by Unequal Search Budgets

The assumption or constraint. The base model evaluation caps each model at five tool calls (search queries or page retrievals) per instance "due to context length limitations" (Section 3.1). The Deep Research products (OpenAI and Gemini) are evaluated as deployed through their web interfaces, with no visibility into or control over their internal search budgets. The paper does not report the actual number of searches, pages read, or reasoning steps performed by the Deep Research products during report generation.

The consequence. The comparison between Deep Research products and their base models conflates at least two variables: (a) pipeline engineering — whether the system uses specialized multi-step reasoning, iterative writing, or post-hoc verification beyond what a raw model + tool-calling interface provides, and (b) search budget — the number of distinct information retrieval operations the system can perform. The paper interprets the dramatic performance gaps (OpenAI Deep Research produces 5.5× more cited statements than o3, with 2.5× higher citation match rate; Gemini Deep Research produces 14.6× more cited statements than gemini-2.5-pro) as evidence of "targeted optimization" and "additional writing modules" (Section 3.4). However, these gaps could be partially or entirely explained by search budget differences. If o3 were given 50 tool calls instead of 5, it might close much of the gap with OpenAI Deep Research on statement generation volume (finding more papers to cite) and possibly on citation match rate (having more opportunities to verify claims against sources before writing them). The five-call cap makes the base models artificially weak baselines, and the magnitude of the performance gap attributable specifically to pipeline engineering—as opposed to search budget—is unidentifiable from the reported data.

What evidence exists in the paper. The search budget asymmetry is documented in Section 3.1: base models are "capped the maximum number of tool calls at five per instance," while the Deep Research products are used as deployed with no budget constraint. The paper notes that Deep Research products "significantly outperform their base models in coverage and factual grounding" (Section 3.4) and attributes this to "task-specific model fine-tuning or pipeline design beyond standalone LLM capabilities," but never discusses the search budget confound. The visible "plan" and "step-by-step reasoning" phases in the Gemini Deep Research interface are cited as evidence of pipeline complexity, but these phases themselves consume search operations that the five-call-capped base model cannot replicate regardless of its reasoning capabilities.

Mitigation status. Not addressed. The paper does not run an experiment where base models are given budget-matched tool calls (e.g., allowing 20 or 50 calls on a smaller set of prompts), does not report the actual number of search operations performed by the Deep Research products, and does not discuss the budget confound as a limitation of the model-level comparison. The conclusion that "Deep Research products significantly outperform their base models" is stated without qualification about the budget asymmetry.

---

The 100-Prompt Test Set Provides No Statistical Reliability Guarantees

The assumption or constraint. The final ReportBench benchmark consists of exactly 100 prompts, balanced across 11 application domains (Section 2.1.3, validated by four research experts). The paper reports only point estimates in Table 1—means for each metric across the 100 reports—with no confidence intervals, standard deviations, standard errors, or statistical significance tests for any comparison between systems.

The consequence. With 100 prompts distributed across 11 domains, some domains may contain as few as 5–8 prompts (assuming roughly uniform distribution after downsampling, though the paper does not confirm exact per-domain counts). The per-prompt variance in metrics like citation match rate and reference precision could be substantial—a system might achieve 90% match rate on one prompt and 40% on another, depending on the topic's difficulty, the availability of accessible web sources, and the specificity of the reverse-engineered prompt. Without variance estimates, several of the paper's comparative claims are unjustified:

• The claim that OpenAI achieves "significantly higher precision (0.385)" than Gemini (0.145) is stated without evidence that this 0.24 difference exceeds what would be expected from sampling variation across 100 prompts. If the per-prompt precision standard deviation is, say, 0.3 for both systems, the difference would not be statistically significant at conventional thresholds.
• The claim that "claude4-sonnet demonstrates the most balanced performance" (Section 3.4) involves comparing claude4-sonnet's metrics to three other base models across six dimensions—a multiple-comparison problem where some differences will appear favorable by chance even if all models were equally capable.
• The paper draws conclusions about relative strengths and weaknesses (e.g., "Gemini tends to over-generate citations") that imply consistency across prompts, but the per-prompt distribution of citation counts could be highly skewed, with a few prompts generating most of the citations and others generating almost none.

The absence of statistical reporting means a practitioner cannot assess whether the performance differences in Table 1 are reliable enough to inform a choice between systems, or whether they reflect noise that would reverse with a different sample of 100 prompts.

What evidence exists in the paper. Table 1 reports means only. There are no error bars, no confidence intervals, no p-values, and no discussion of statistical methodology anywhere in the paper. The data collection period (July 14–25, 2025) is specified, and the domain classification was validated by four research experts, but these are procedural controls rather than statistical ones. The paper reports counts alongside means (e.g., cited statement count, non-cited statement count) but provides no distributional information for these counts.

Mitigation status. Not addressed. The paper does not acknowledge the small sample size or the absence of statistical reporting as limitations. The 100-prompt test set size may have been constrained by the cost of running evaluations against commercial API-based products (which require manual collection from web interfaces for the Deep Research agents), but the paper does not discuss this constraint or its implications for the reliability of the reported results.

---

ReportBench Cannot Evaluate Writing Quality, Synthetic Insight, or Argumentation

The assumption or constraint. From the outset, the paper explicitly restricts its scope to content quality, defined as reference quality/relevance and statement faithfulness/veracity, and defers writing quality evaluation to future work (Section 1: "leaving the assessment of writing quality to future work"). The evaluation metrics—reference precision/recall, citation match rate, non-cited factual accuracy—measure whether a report cites the right papers and makes factually true claims. They do not measure whether the report is well-organized, clearly written, intellectually insightful, or useful as a synthesis.

The consequence. This scope limitation means ReportBench provides a necessary but radically insufficient picture of research report quality. Several dimensions that distinguish an excellent survey from a mediocre one are completely invisible to the benchmark:

• Synthetic insight: A report could accurately describe 10 individual papers (high citation match rate) while making no attempt to identify trends, contradictions, open problems, or relationships between them. This would be a poor survey but would score well on all of ReportBench's metrics.
• Argumentative structure: A report could present true facts in a disorganized order, jumping between topics without logical flow. Since ReportBench evaluates individual statements atomically rather than assessing the report's structure, this failure mode is invisible.
• Critical evaluation: A survey that uncritically reports every paper's claimed results without discussing methodological limitations, conflicting findings, or replication concerns would be misleading despite being factually accurate on a per-statement level. ReportBench's verification pipeline checks whether a claim matches its source, not whether the source itself is reliable or whether the report appropriately contextualizes its findings.
• Accessibility and clarity: A report written in impenetrable prose with undefined jargon would be useless to its intended audience even if every cited reference is relevant and every claim is factually true.

The paper is transparent that these dimensions are out of scope, but the framing throughout—"comprehensive evaluation," "systematic benchmark," "content quality"—implies a completeness that the benchmark does not deliver. A practitioner who uses ReportBench as their sole quality gate for a Deep Research agent would be selecting for systems that are factually accurate and well-sourced but potentially incoherent, uninsightful, or poorly argued.

What evidence exists in the paper. The scope limitation is explicitly stated in Section 1: "we decompose the evaluation of research reports generated by LLMs into two core dimensions: writing quality and report content. Due to the subjectivity of writing-style evaluation, while the criteria for assessing content quality can be more clearly defined, this work focuses primarily on the evaluation of report content, leaving the assessment of writing quality to future work." The evaluation metrics described in Section 2.2 and reported in Table 1 all fall under the "content quality" umbrella—there are no metrics for organization, clarity, argumentation, or insight. The qualitative examples in Section 4 illustrate factual errors (hallucinated authors, fabricated URLs) but do not assess whether the surrounding text is well-written or insightful.

Mitigation status. Acknowledged but deferred. The paper explicitly states writing quality evaluation is left to future work. No partial mitigation—such as incorporating an LLM-based holistic quality score alongside the fine-grained metrics, or measuring structural properties like section coherence—is attempted. The paper positions this as a deliberate scope constraint rather than a limitation to be solved, arguing that content quality dimensions are "more clearly defined" and thus amenable to automated evaluation. Whether this is a reasonable scope choice or a fundamental limitation depends on the use case: for applications where factual accuracy is paramount and writing quality is secondary (e.g., evidence synthesis for policy decisions), ReportBench may be sufficient; for applications where the report's communicative effectiveness matters (e.g., educational materials, executive summaries), ReportBench's metrics are necessary but far from sufficient.

7. Implications and Future Directions

How This Work Changes the Landscape

ReportBench does not introduce a new model, training algorithm, or agent architecture. It introduces something more foundational to the research ecosystem: a reproducible, automated methodology for measuring the factual reliability of AI-generated research reports. This is primarily a methodological contribution — a shift in how the field evaluates long-form knowledge synthesis systems — rather than a technical advance in how those systems are built. The magnitude is best characterized as a new diagnostic capability that fills a specific, well-defined gap in the evaluation landscape, rather than a paradigm shift that reorients the entire field.

The paper's core intervention is to demonstrate that the content quality of a research report can be decomposed into separately measurable dimensions — reference precision, reference recall, citation faithfulness, and non-cited statement accuracy — and that these dimensions can be automatically computed at scale without human annotation, provided one has access to expert-written survey papers as ground truth and a suite of strong evaluator LLMs. This is not the first paper to advocate for fine-grained evaluation of long-form generation, but it is among the first to operationalize it for the specific output modality — multi-source research reports with explicit citations — that is becoming the default interface for Deep Research products.

The practical consequences of this diagnostic capability are twofold. First, it enables apples-to-apples comparison between commercial products (OpenAI Deep Research vs. Gemini Deep Research) and between products and their underlying base models (OpenAI Deep Research vs. o3, Gemini Deep Research vs. gemini-2.5-pro) along dimensions that developers and users actually care about: "does this report cite real, relevant papers?" and "can I trust that the claims attributed to those papers are accurate?" Before ReportBench, answering these questions required manual expert review that simply does not scale to the volume of reports these systems can generate. The paper provides the first systematic evidence, using a consistent methodology, that OpenAI's product achieves higher precision and citation match rates while Gemini's product generates broader but less reliable coverage — a finding that would be labor-intensive to establish through ad-hoc manual comparison.

Second, ReportBench's decomposition reveals where current systems fail, not just that they fail. A single aggregate quality score — "this report is a 7/10" — provides no guidance to a developer about which component to improve. The paper's finding that o3 (the base model) achieves only a 31.43% citation match rate, while OpenAI Deep Research (powered by o3) achieves 78.87%, tells a specific story: the base model's raw generation capability is insufficiently grounded in retrieved sources, and the Deep Research pipeline — whatever it does internally — substantially bridges this gap. A developer at a company building a competing Deep Research product can look at this result and identify citation faithfulness as the highest-priority bottleneck for their own base-model-plus-search pipeline. Without the decomposition, the developer would know only that the competitor's product "scores higher" — not which specific capability accounts for the difference.

The paper also partially reconciles an implicit tension in prior work between breadth and accuracy in automated research synthesis. Prior survey generation systems (AutoSurvey, HiReview, SurveyForge) optimized for comprehensiveness — generating surveys that cite many papers and cover many sub-topics — but did not systematically evaluate whether the cited papers actually support the claims made about them. ReportBench's results suggest that breadth and accuracy are in tension for current systems: Gemini Deep Research generates 3.3× more references and 14.6× more cited statements than gemini-2.5-pro, but achieves lower precision (0.145 vs. 0.269) and lower citation match rate (72.94% vs. 59.24%). The benchmark surfaces this trade-off quantitatively, enabling developers to make informed choices about where to position their systems on the breadth-accuracy frontier rather than optimizing for one dimension unaware of the cost to the other.

A more speculative consequence of this work concerns incentive alignment in the Deep Research product market. As of mid-2025, the primary differentiator between commercial Deep Research products is largely subjective — users form impressions based on a handful of example queries and the overall polish of the output. If ReportBench-style evaluation becomes standard (and the paper explicitly releases "datasets, prompts, and evaluation scripts" to encourage adoption), product teams will face pressure to improve on the specific dimensions the benchmark measures: reference precision, citation match rate, factual accuracy. This is simultaneously an opportunity (metrics-driven improvement tends to be faster than intuition-driven improvement) and a risk (Goodhart's law: when a metric becomes a target, it ceases to be a good metric). The paper's scope limitation — it does not evaluate writing quality, synthetic insight, or argumentation — means that over-optimizing for ReportBench scores could produce reports that are factually impeccable but intellectually shallow. The benchmark's transparency (intermediate outputs are retained for human inspection) partially mitigates this by enabling audits, but the risk is real and the paper does not discuss it.

The paper also implicitly establishes that current Deep Research agents are far from saturating the benchmark's difficulty. The highest recall achieved by any system is 0.036 — meaning even the best agent retrieves fewer than 4% of the references that an expert-written survey on the same topic includes. The highest citation match rate is 78.87% — meaning more than one in five cited claims is not supported by the source. These are not numbers that suggest the evaluation is too easy or that the metrics are about to become uninformative due to ceiling effects. There is substantial headroom for improvement on every dimension, which makes the benchmark practically useful as a progress-tracking instrument for the next several years of development.

Follow-Up Research This Work Enables

Human correlation study for ReportBench's automated metrics. The most critical unvalidated assumption in the paper is that ReportBench's automated metrics — reference precision/recall, citation match rate, non-cited factual accuracy — align with what human domain experts consider a high-quality research report. A study where, say, five NLP researchers and five computer vision researchers each rate 20 AI-generated reports (10 from OpenAI Deep Research, 10 from Gemini Deep Research) on overall quality, trustworthiness, and comprehensiveness, and where these ratings are correlated against ReportBench's per-dimension scores, would validate or refute the benchmark's construct validity. The study should measure both aggregate correlation (do high-scoring reports on ReportBench receive high human ratings?) and dimension-specific correlation (does citation match rate specifically predict human judgments of citation trustworthiness, or do humans weight other factors more heavily?). A positive result would substantially strengthen the paper's claim that ReportBench measures "content quality"; a negative or weak result would redirect effort toward developing metrics that better capture what humans value.

Cross-evaluator robustness assessment. The current evaluation pipeline uses gpt-4o for cited statement verification and Gemini models for non-cited verification. This creates an evaluator bias confound: gpt-4o may be systematically more lenient or strict when evaluating outputs from models in its own family (o3 and OpenAI Deep Research). A follow-up study should replicate the full evaluation on at least a 20-prompt subset using evaluator models from different families — for example, running cited statement verification with Claude (Anthropic) and Gemini (Google) in addition to gpt-4o, and running non-cited verification with a non-Gemini web-connected model. The key result would be the rank correlation of system scores across evaluator configurations. If OpenAI Deep Research maintains its lead in citation match rate regardless of which evaluator model is used, the result is robust to evaluator bias. If rankings flip when a non-OpenAI evaluator is used, the reported results are partially or entirely an artifact of evaluator in-group bias, and the benchmark community needs to standardize on evaluator-independent verification methods (e.g., multi-evaluator ensembles, human-validated reference judgments for a calibration subset).

Base model evaluation with budget-matched search. The current base model evaluation caps tool calls at five per instance, while the Deep Research products are used as deployed with unknown internal search budgets. This conflates pipeline quality with search budget. A follow-up experiment should evaluate base models at multiple search budget levels (5, 10, 20, 50, 100 tool calls) on a fixed subset of, say, 20 prompts, and plot the performance-by-budget curve for each model. Plotting the Deep Research product's performance as a horizontal line (since its internal budget is unknown) would reveal: (a) whether base models can match or approach Deep Research performance given sufficient search budget, (b) whether the performance gap narrows at higher budgets (suggesting the gap is primarily a budget effect) or persists (suggesting genuine pipeline quality differences), and (c) the marginal return on additional search operations for each model family, which would inform cost-efficiency decisions in practical deployments.

Citation faithfulness failure taxonomy at scale. The paper's qualitative analysis identifies two failure modes — statement hallucination (misrepresenting a real source) and citation hallucination (fabricating a non-existent source) — but provides no quantitative breakdown of their relative frequencies. A follow-up study should manually annotate a random sample of, say, 200 statements flagged as citation-inconsistent by the pipeline across multiple systems, classifying each failure as: (a) statement hallucination (source exists but doesn't support the claim), (b) citation hallucination (source doesn't exist or URL is fabricated), (c) retrieval error (pipeline retrieved the wrong passage from a real source), or (d) verification error (pipeline incorrectly flagged a consistent statement). The resulting confusion matrix would quantify the precision of the evaluation pipeline itself (how often do pipeline-flagged inconsistencies correspond to genuine errors?) and reveal which failure types dominate for different systems — information that would directly guide engineering priorities. If, for example, 40% of flagged inconsistencies are actually retrieval errors (type c), the evaluation pipeline's supporting source extraction step needs improvement before it can serve as a reliable evaluator; if 60% are genuine statement hallucinations (type a), the research priority is improving generation-to-source alignment in the agents themselves.

Domain-specific difficulty characterization. The 100-prompt benchmark is balanced across 11 domains, but the paper reports only aggregate metrics. A follow-up analysis should report per-domain results for each system, identifying which domains (e.g., NLP, biology, physics, robotics) produce the highest and lowest citation match rates, which produce the widest precision-recall gaps between systems, and which, if any, exhibit ceiling or floor effects (all systems performing similarly well or poorly). This would serve two purposes: (a) it would identify domains where current agents are already reliable (where further investment yields diminishing returns) versus domains where reliability is catastrophically low (where improvement is urgent), and (b) it would test whether the benchmark's domain balancing actually matters — if system rankings are consistent across domains, the balancing was unnecessary; if rankings flip across domains (e.g., Gemini outperforms OpenAI in physics but not in NLP), the balancing is essential and the aggregate numbers obscure important interaction effects.

Dynamic difficulty estimation and adaptive evaluation allocation. The paper's evaluation framework applies all metrics uniformly to every report, regardless of the report's apparent quality or the domain. A follow-up could develop a two-stage evaluation protocol: first, compute reference precision and citation match rate on a small sample of statements (say, 10 randomly selected cited statements); if the report scores above a threshold on these initial checks, expend the full evaluation budget (verifying all statements, running the full web-based fact-checking pipeline); if it falls below, flag the report as low-quality and skip the expensive full evaluation. This would substantially reduce the average evaluation cost while maintaining high recall for detecting quality failures — analogous to the adaptive difficulty estimation the paper identifies as a bottleneck in its own methodology (Section 3.2 of the context paper provides a precedent). The key measurements would be: what fraction of reports are flagged by the rapid initial check, what is the false-negative rate (reports that pass the initial check but would have been flagged by full evaluation), and what is the total computational cost savings relative to uniform full evaluation?

Practical Applications and Downstream Use Cases

Continuous monitoring of commercial Deep Research products. The paper's finding that OpenAI Deep Research achieves 78.87% citation match rate and 95.83% factual accuracy while Gemini Deep Research achieves 72.94% and 92.21% respectively (Table 1) establishes a quantitative baseline for the current generation of products. A research news service (e.g., an AI-focused newsletter, an analyst firm covering AI tools) could periodically re-run ReportBench against the latest versions of these products — say, quarterly — and publish trends showing whether factual reliability is improving, plateauing, or regressing with each update. This would serve a function analogous to benchmark tracking for consumer CPUs or smartphone cameras: providing an independent, methodology-transparent signal of product quality that consumers and enterprise purchasers can use to make adoption decisions. The specific metrics to track would be citation match rate and non-cited factual accuracy (directly related to user trust), with reference precision and recall providing context about comprehensiveness.

Quality gate in automated survey generation pipelines. The paper's demonstration that base models augmented with search tools produce citation match rates as low as 31.43% (o3) means that naive "LLM + search API" pipelines are not safe for producing research-grade output without additional verification. An organization building an internal automated literature review system (e.g., a pharmaceutical company generating weekly summaries of new publications in a therapeutic area, or a legal firm producing case law surveys) could integrate ReportBench's evaluation pipeline as a post-generation quality gate. Before a generated report is delivered to the end user, the pipeline would automatically compute reference precision and citation match rate; reports falling below a configurable threshold (e.g., precision < 0.3 or match rate < 70%) would be routed for human review rather than delivered directly. This provides a scalable safety net — automated generation with automated verification — that leverages the evaluation framework's interpretability: when a report is flagged, the specific inconsistent statements are available for the human reviewer to inspect, rather than the reviewer needing to verify the entire report from scratch.

Calibration target for open-source Deep Research agent development. As of mid-2025, the development of open-source Deep Research agents (e.g., STORM, gpt-researcher, or custom LangChain-based research pipelines) is proceeding without clear quantitative targets for what "good enough" means. This paper establishes that commercial products achieve roughly 78–79% citation match rate and 92–96% factual accuracy. An open-source development team can now set concrete engineering targets: "achieve ≥ 75% citation match rate and ≥ 90% non-cited accuracy on ReportBench before we consider our system production-ready." This converts an ambiguous quality goal ("make the reports trustworthy") into a measurable, continuously trackable objective. The benchmark's public release (code, data, evaluation scripts) makes this immediately actionable for any team with API access to the required evaluator models. A team that achieves parity with commercial products on these metrics can credibly claim their system is competitive on factual reliability, even if it lags on subjective dimensions like writing quality that ReportBench does not measure.