Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections

ArXiv: 2603.12180

🎯 Pitch

Top AI agents match human accuracy on complex document QA but succeed on almost entirely different questions, relying on brute-force search instead of strategic planning—and they need 10× more steps to do it.

1. Executive Summary

This paper introduces MADQA, a benchmark of 2,250 human-authored questions over 800 heterogeneous PDF documents, designed to evaluate whether multimodal agents demonstrate genuine strategic reasoning or merely stochastic trial-and-error search. The core contribution is a novel evaluation protocol that measures the accuracy-effort trade-off through the Kuiper statistic (quantifying how well an agent's computational investment correlates with success) alongside a formal decomposition of agent behavior into retrieval, navigation, and comprehension failures. The best agentic system, Gemini 3 Pro with a BM25 search tool, matches human accuracy at ~82% but exhibits fundamentally different competencies—sharing only κ = 0.24 item-level agreement—and relies on brute-force search to compensate for weak strategic planning, establishing that frontier agents close the accuracy gap only when granted generous compute budgets (failing dramatically in the low-step regime, where humans achieve ~50% first-query accuracy versus the agent's ~12%) while leaving an 18% oracle retrieval gap that search alone cannot bridge.

2. Context and Motivation

The Core Problem: We Don't Know Whether Agents Reason or Just Search

The fundamental question this paper tackles is deceptively simple: when a multimodal agent successfully answers a complex document-grounded question, does it achieve that answer through genuine strategic reasoning, or through stochastic trial-and-error search? This question matters because it cuts to the heart of what "agency" means in AI systems. An agent that brute-forces its way through a document collection — issuing many search queries, examining many pages, and essentially performing an exhaustive exploration — is fundamentally different from one that decomposes the question, plans a retrieval trajectory, and iteratively refines its search based on intermediate findings. The former scales poorly with corpus size and query complexity; the latter scales gracefully because it knows what to look for and when to stop.

This gap in understanding is significant for several practical reasons:

Deployment cost and latency. If agents succeed primarily through brute-force exploration, the computational cost of deploying them in production settings — where document collections can contain thousands or millions of pages — becomes prohibitive. The paper's finding that a leading agent (Claude Sonnet 4.5 RLM) processed over 270 million input tokens at a cost of $850 just to evaluate 500 test questions (Section 5) illustrates this acutely. An agent that cannot distinguish between questions requiring one search query and those requiring twenty will waste enormous resources on routine tasks.

Interpretability and trust. An agent that genuinely plans its retrieval — that decomposes "Which lesson plan suggests a lower instructor ratio: Firearms or New Mexico Justice System?" into sub-queries targeting each document, then compares values — produces an auditable reasoning trace. An agent that issues a dozen slightly reformulated queries until something surfaces yields no such trace. In high-stakes enterprise settings (legal document review, financial compliance, medical record analysis), the distinction between strategic reasoning and stochastic search is the difference between an assistant you can trust and one you must constantly supervise.

Capability boundaries. Understanding whether current agents reason or search determines where we should expect them to fail. A search-dominated agent will degrade gracefully as the corpus grows larger (it searches more, taking more time but potentially still succeeding). A reasoning-dominated agent will fail catastrophically when it encounters question types or document structures outside its planning capabilities — even in small corpora. Knowing which regime we are in informs deployment decisions, risk assessment, and research priorities.

Conflicting Signals from Prior Benchmarks

The paper is motivated by a genuine fragmentation in the evaluation landscape that makes it impossible to answer the reasoning-versus-search question from existing evidence. The authors identify three axes of fragmentation (Section 1.1 and Table 1):

1. Format: Text-only agentic benchmarks miss visual complexity. Benchmarks like Researchy Questions (Rosset et al., 2025) and BRIGHT (Su et al., 2025) capture the complexity of multi-step "agentic research" — decomposing questions, iterating on partial findings, synthesizing across diverse sources. However, they operate over HTML or plain text. Real-world enterprise documents — PDFs containing tables, forms, figures, checkboxes, handwritten annotations, and spatial layouts that encode meaning — require visual comprehension that text-only benchmarks cannot assess. An agent that excels at BRIGHT may fail completely when asked to "interpret the checkbox in the Delaware State Preschool Yearbook that indicates whether a quality standard was met," because that checkbox's meaning is encoded purely in its visual presence or absence, not in extractable text.

2. Scope: Domain-specific PDF benchmarks lack multi-step planning requirements. Benchmarks like FinRAGBench-V (Zhao et al., 2025) and ViDoSeek (Wang et al., 2025) confront agents with visually rich PDFs — tables, charts, forms — but restrict evaluation to narrow verticals (finance, slide decks) or rely on single-step metrics. A benchmark that asks "What was the revenue in Q3 2023?" tests whether the agent can retrieve and read a single value from a known document type, but not whether it can plan: "First find which document covers Q3 2023, then locate the revenue table, then extract the specific cell." The distinction between single-step retrieval and iterative planning is exactly what the "agentic" property (Section 1.2, Property 5) is designed to capture, and its absence from prior benchmarks means we have no rigorous signal on whether agents can plan retrieval trajectories.

3. Data provenance: LLM-generated questions risk bias toward similar models. General-purpose document benchmarks like ViDoRE (Faysse et al., 2025) and ViDoRe v3 (Loison et al., 2026) attempt to bridge the format-scope gap but suffer from methodological concerns. When questions are generated by LLMs (even with human verification), the generating model shapes the distribution of question types, linguistic patterns, and difficulty. A GPT-4-generated question may be systematically easier for GPT-4-based agents because it reflects the same reasoning patterns and vocabulary preferences. This circularity makes it impossible to separate genuine capability from evaluation bias. The authors explicitly contrast this with fully human-authored benchmarks, where questions reflect how humans actually decompose information needs rather than how models do.

Additionally, benchmarks like VIMDoc (Kim et al., 2025) and DOUBLE-BENCH (Shen et al., 2025) recycle documents from older datasets (e.g., DocVQA, Wikipedia), increasing the risk of data contamination — models may have memorized answers during pretraining rather than demonstrating genuine document comprehension at evaluation time. The paper's guessability analysis (Section 2.3, Appendix E.2) quantifies this concern: frontier models correctly "guess" 9–15% of MADQA answers without seeing any documents, with approximately 8% attributable to training data contamination from publicly available documents.

Where Prior Evaluation Paradigms Fall Short

The paper identifies specific limitations along multiple dimensions that together prevent the field from measuring what it claims to measure:

No accuracy-effort trade-off measurement. Prior benchmarks evaluate agents solely on correctness — did they produce the right answer? This metric is blind to how the answer was reached. An agent that answers 80% of questions correctly by issuing 50 search queries per question is fundamentally different from one that achieves the same accuracy with 3 queries, yet they would appear identical on a leaderboard. The paper frames this as a construct validity problem (Bean et al., 2025): if the construct we want to measure is "agentic reasoning" — the ability to plan and execute efficient retrieval — but the metric only measures final-answer correctness, then the benchmark does not validly measure its intended construct. The Kuiper statistic and the cumulative difference methodology (Section 3.3) are introduced specifically to close this measurement gap.

No decomposition of failure modes. Prior benchmarks report aggregate accuracy, which obscures where agents fail. Does an incorrect answer result from: (a) failing to find the right document at all (retrieval failure), (b) finding the right document but the wrong page (navigation failure), or (c) finding the right page but misreading the content (comprehension failure)? These failures demand different solutions — better search query formulation, better within-document navigation, or better visual reading capabilities, respectively — but without decomposing them, researchers cannot diagnose their systems' bottlenecks. The paper's error taxonomy (Section 5.1, Appendix H.3) provides this decomposition, revealing that retrieval failures account for 35.7% of all agent errors, navigation for 23.0%, comprehension for 28.8%, and refusals for 12.6% — but these proportions vary drastically across models (Figure 8), with weaker models dominated by refusals and stronger models shifting toward comprehension as their primary remaining bottleneck.

No human comparison with matched tools. Prior benchmarks either compare agents to humans with unrestricted document access (which tells us little about retrieval strategy — humans can just read everything) or lack human baselines entirely. By giving human annotators the same BM25 search tool as the agents (Section 4, Appendix C.2), the paper enables direct comparison of retrieval strategies: how many queries do humans issue? How do they reformulate after failures? What is their first-query accuracy? The finding that humans achieve ~50% accuracy on their first search query while the best agent achieves only ~12% (Figure 9) quantifies a specific strategic deficiency — inability to formulate effective initial queries — that aggregate accuracy masks entirely.

Saturation of existing benchmarks. Benchmarks like DocVQA and MP-DocVQA, while pioneering, have been extensively studied and are approaching performance ceilings. The paper's Classical Test Theory-based split creation (Section 2.4, Appendix D) explicitly reserves a "Sentinel Pool" of 100 hard items that no current model can solve (0% baseline accuracy), ensuring the benchmark retains discriminative power even as models improve. This is a deliberate design choice absent from prior benchmarks, which typically become saturated and lose their ability to rank frontier systems.

How This Paper Positions Itself

The paper positions MADQA not as another document QA benchmark, but as a measurement instrument for a specific, well-defined construct: agentic reasoning over document collections. This is operationalized through six formal properties (Section 1.2) that any valid benchmark for this construct must satisfy:

• Extractive: Answers must be lexically grounded in the evidence, preventing systems from succeeding through parametric knowledge alone.
• Multi-hop: Evidence may span disjoint pages or documents, forcing integration across retrieval steps.
• Closed-world: Answers must be derivable solely from the provided corpus, isolating document comprehension from world knowledge.
• Grounded: Evidence sets must be minimal, penalizing "lazy citations" that cite entire documents.
• Agentic: No single retrieval query can surface all evidence, necessitating planning, navigation, and aggregation.
• Visual: Non-textual modalities (layout, tables, figures) may be necessary for answering, testing multimodal comprehension.

These properties are not merely descriptive — they are enforced by construction through the annotation protocol (Section 2.2) and validated through explicit analyses (Section 2.3). The lexical overlap analysis shows that even trigram matching achieves only 51% recall — confirming that simple keyword search cannot solve the benchmark (Property 5). The parametric knowledge analysis quantifies the upper bound on "free" performance at 9–15% (Property 3). The visual necessity analysis finds that 57.2% of questions require structural or visual comprehension beyond free text (Property 6).

The paper's position is that no prior benchmark simultaneously satisfies all six properties while maintaining fully human-authored questions over fresh, heterogeneous documents. This gap means the field lacks a valid instrument for measuring whether agents genuinely reason about document collections or merely search stochastically through them. MADQA is designed to fill this gap, and the evaluation protocol — with its Kuiper calibration metric, error taxonomy, and human-agent comparison — is designed to produce the specific diagnostic signals that answer the reasoning-versus-search question.

The paper also explicitly positions itself within the construct validity framework of Bean et al. (2025), arguing that benchmark design should begin with a clear definition of the construct to be measured and then validate that the benchmark actually measures that construct rather than confounded alternatives (Section 2.3). This is a methodological contribution distinct from the benchmark itself: it provides a template for how future benchmarks should be designed and validated, moving the field away from ad-hoc dataset creation toward principled measurement.

3. Technical Approach

3.1 Reader Orientation

This paper introduces MADQA, a benchmark and evaluation framework — not a new model or algorithm — designed to measure how multimodal AI agents reason over collections of PDF documents. The core problem it solves is that prior benchmarks conflate "getting the right answer" with "reasoning strategically," making it impossible to tell whether an agent's success comes from genuine planning or from brute-force trial-and-error search through the document collection. The "shape" of the solution is a carefully constructed dataset of 2,250 human-authored questions paired with a multi-metric evaluation protocol that jointly measures answer correctness, evidence attribution, and — crucially — the efficiency with which agents allocate computational effort, enabling precise diagnosis of whether agents are reasoning or merely searching.

3.2 Big-Picture Architecture (Diagram in Words)

The MADQA framework has five major components, each responsible for a distinct measurement objective:

1. Document Corpus (800 heterogeneous PDFs): The "world" that agents must navigate — 18,619 pages across 63 fine-grained categories in 13 high-level domains, sourced from DocumentCloud. The corpus is specifically curated to contain clusters of related documents (e.g., sequential annual reports from the same organization, menus from different restaurants) to enable realistic cross-document multi-hop reasoning.

2. Question-Answer-Evidence Triples (2,250 human-authored items): Each item consists of a natural language question q, an extractive answer a (tokens that appear physically in the source documents), and a minimal evidence set E (a set of specific page identifiers from which a can be derived). The annotation protocol enforces six formal properties — Extractive, Multi-Hop, Closed-World, Grounded, Agentic, and Visual — that together define the construct of "agentic document collection VQA."

3. Principled Dataset Splits (Classical Test Theory-based partitioning): The 2,250 items are partitioned into Train (1,550 items), Development (200 items), and Test (500 items) sets using item response statistics computed from 16 diverse models. The Test set is explicitly engineered to maximize discriminative power (selecting items with high point-biserial correlation) while reserving a "Sentinel Pool" of 100 items that no current model can solve, ensuring long-term benchmark relevance.

4. Multi-Metric Evaluation Protocol: A suite of three complementary metrics — LLM-based Accuracy (answer correctness), Page F1 and Doc F1 (evidence attribution at two granularities), and the Kuiper statistic (effort calibration) — that together measure not just whether an agent succeeded but how it succeeded and at what computational cost.

5. Baseline Implementations (spanning static RAG, agentic search, recursive decomposition, and human performance): A set of reference systems — from managed RAG services (Gemini File Search, OpenAI Assistants) to BM25-based iterative agents to Recursive Language Models — evaluated under identical conditions, plus a human baseline using the same search tools, enabling direct comparison of retrieval strategies and efficiency.

Information flows through the framework during evaluation as follows: a question q is presented to a system under test → the system interacts with the corpus C through whatever retrieval and reasoning mechanisms it implements (search queries, page views, decomposition, aggregation) → the system produces a predicted answer â and (optionally) a set of cited evidence pages Ê → the evaluation harness scores â against gold answer a using the calibrated LLM judge (Accuracy), scores Ê against gold evidence E at page and document granularity (Page F1, Doc F1), and computes the Kuiper statistic over the system's recorded effort trajectory (number of tool calls or steps) across all test items.

3.3 Roadmap for the Deep Dive

• First, the benchmark's formal task definition and six desired properties, because understanding what is being measured — and what is explicitly excluded — is prerequisite to understanding how the benchmark is constructed and why each design choice matters. These properties (Extractive, Multi-Hop, Closed-World, Grounded, Agentic, Visual) are the "specification" that everything else implements.

• Second, the document corpus curation and question annotation pipeline, because the validity of the entire benchmark rests on whether the data actually instantiates the desired properties. I will walk through document sourcing, annotator selection and training, the annotation protocol, quality assurance, and the resulting statistics.

• Third, the construct validity analyses, because the paper explicitly commits to the measurement philosophy of Bean et al. (2025): benchmarks must validate that they measure what they claim to measure, not confounded alternatives. I will explain the lexical overlap analysis (proving the benchmark cannot be solved by keyword matching), the parametric knowledge analysis (quantifying the upper bound on "free" performance from training data contamination), and the visual necessity taxonomy (characterizing what types of visual comprehension the benchmark requires).

• Fourth, the Classical Test Theory-based split creation, because how the benchmark is partitioned into train/dev/test sets directly determines its statistical power, its resistance to saturation, and the reliability of leaderboard rankings.

• Fifth, the evaluation protocol in full detail, covering the LLM judge for answer correctness (including calibration against human judgments, bias correction, and the handling of multi-part answers), the Page F1 and Doc F1 attribution metrics, and the Kuiper calibration statistic (including its mathematical construction, interpretation, and sensitivity to the choice of effort measure).

• Sixth, the baseline implementations, covering the BM25 MLLM Agent, managed RAG services, HEAVEN, Claude Agent with Semtools, Recursive Language Models, MDocAgent, ColBERTv2 + LLaMA, and the human baseline collection — including the specific tools, prompts, and hyperparameters used, so that the evaluation results are reproducible and interpretable.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmark construction and evaluation methodology paper whose core idea is that measuring agentic reasoning over document collections requires: (1) a dataset that enforces six formal properties distinguishing agentic reasoning from single-step retrieval or parametric recall, (2) a principled test-set creation methodology that maximizes discriminative power and prevents premature saturation, and (3) a multi-metric evaluation protocol that jointly measures answer correctness, evidence attribution, and effort calibration — the latter being the key innovation that enables answering the paper's motivating question of whether agents reason (efficiently) or merely search (brute-force).

Formal Task Definition and the Six Desired Properties

The paper defines Agentic Document Collection Visual Question Answering through a formal specification and six properties. The task itself (Definition A.2 in Appendix A.2) is: given a corpus C of multi-page PDF documents and a natural language query q, produce an answer a (a sequence of tokens) and a minimal evidence set E ⊆ P(C), where P(C) denotes all pages in the corpus. The answer and evidence must satisfy six properties that are enforced by construction during dataset creation.

Property 1: Extractive. Answer tokens must appear physically in the evidence pages:

$\forall t_\ell \in a : \exists p \in E \text{ such that } t_\ell \in T(p)$

where T(p) denotes the token sequence of page p.

What it computes: a hard constraint on the answer format — every token in the predicted answer must be present in the text of at least one page in the evidence set. This means abstractive answers ("the revenue grew substantially") are invalid; only concrete, lexically grounded values ("$1.2M") are acceptable.

Why this form: the extractive constraint eliminates two confounds. First, it prevents systems from succeeding via parametric knowledge — if the answer must be a string present in the document, an agent cannot simply recall facts from pretraining. Second, it makes evaluation objective and auditable: correctness reduces to verifying that the answer string appears in the cited evidence. This is why the benchmark uses an LLM judge rather than exact string match (Section 3.1) — to handle legitimate variations in formatting while still enforcing extractive grounding.

Property 2: Multi-Hop. The evidence set may require multiple disjoint pages:

$|E| \geq 1, \text{ with } |E| > 1 \text{ for multi-hop questions}$

Multi-hop questions further decompose into cross-page (all evidence within one document: ∃D_i : E ⊆ D_i) and cross-document (evidence spanning multiple documents: ∄D_i : E ⊆ D_i).

What it computes: a classification of questions by the number of distinct evidence locations required. The annotation protocol targeted approximately 20% multi-hop questions, achieved at 17.3% overall (8.3% cross-page, 9.0% cross-document).

Why this form: multi-hop requirement is what distinguishes "lookup" from "reasoning." A question that can be answered from a single page tests retrieval + reading. A question requiring integration across pages (e.g., comparing values from two annual reports, or bridging: find entity X in document A, then find its property in document B) tests whether the agent can plan a trajectory, retain intermediate findings, and synthesize across retrieval steps. The separation of cross-page and cross-document is diagnostic: cross-page failures indicate within-document navigation problems; cross-document failures indicate inter-document retrieval problems.

Property 3: Closed-World. The answer must be derivable solely from C, independent of parametric knowledge θ:

$a = f(C, q) \text{ where } f \text{ uses no knowledge from } \theta \setminus C$

What it computes: a constraint on the information sources permitted for answering. The annotation protocol explicitly instructs annotators to create questions answerable entirely from the provided documents, strictly excluding external world knowledge.

Why this form: the closed-world assumption isolates document comprehension ability from general knowledge. If a model answers "What was the total excess permit revenue in Minnesota for the 2014-2019 period?" by recalling the value from training data rather than extracting it from the documents, the benchmark is measuring memorization, not agentic reasoning. The guessability analysis (Section 2.3, Appendix E.2) quantifies the extent to which this assumption holds: 9-15% of answers can be guessed without documents, with approximately 8% from contamination and 3% from random chance on binary questions.

Property 4: Grounded Attribution. The evidence set must entail the answer and be minimal:

$\text{ENTAILS}(E, a) = \text{True} \land E \text{ is minimal}$

where minimality requires that no proper subset E' ⊂ E suffices to derive a.

What it computes: a constraint on the evidence annotation. For each question, annotators identified the smallest set of pages from which the answer could be derived. If a value appears on both page 3 and page 15 of a document, annotators select only the most plausible or first occurrence, not both. If comparing two values requires pages from two documents, both pages are included.

Why this form: minimal evidence enables the Page F1 and Doc F1 metrics (Section 3.2) to serve as attribution quality measures. If the gold evidence were non-minimal — e.g., including entire documents rather than specific pages — then an agent citing the correct document but wrong page would receive partial credit for Doc F1 while being penalized on Page F1, providing a misleadingly optimistic attribution score. Minimality ensures that Page F1 genuinely measures whether the agent found the precise evidence, not just the general vicinity.

Property 5: Agentic. The task requires iterative retrieval that cannot be solved in a single forward pass:

$\nexists \text{ query } q' : \text{RETRIEVE}(q', C) \supseteq E \text{ in one step}$

What it computes: a structural constraint on question difficulty — there exists no single search query that would retrieve all evidence pages at once. This means the agent must decompose the question, issue multiple queries, and aggregate across retrievals.

Why this form: this is the property that distinguishes MADQA from standard RAG benchmarks. In a typical RAG evaluation, the question is asked, a retriever fetches the top-k passages, and the model answers — one retrieval, one generation. The agentic property guarantees that this single-step approach will fail because the evidence is distributed across pages that cannot be co-retrieved by any single query. The lexical overlap analysis (Section 2.3, Appendix E.1) validates this: even with perfect trigram matching, recall is only 51% — half of all evidence pages contain no trigram overlap with the question, meaning no keyword-based single query could retrieve them.

Property 6: Visual. Non-textual modalities in the evidence — spatial layout, table structure, figures, graphical elements — are necessary for deriving the answer:

$\nexists a' : \text{ENTAILS}(T(E), a') \land \text{EQUIV}(a', a)$

where T(E) denotes the linearized textual content stripped of visual structure. This means that even with perfect OCR, the answer cannot be derived from text alone.

What it computes: a constraint that the answer depends on information encoded visually, not textually. The visual necessity taxonomy (Section 2.3, Appendix E.3) classifies each question by whether visual comprehension of tables, structured forms, spatial layout, or visual artifacts (handwriting, checkboxes, charts) is required.

Why this form: this property ensures that the benchmark measures multimodal comprehension, not just text extraction. If all answers could be found by running OCR and searching the resulting text, a text-only model would perform as well as a multimodal one, and the benchmark would not measure visual understanding. The finding that only 42.8% of questions are solvable from free text alone confirms that this property is substantive: the majority of questions genuinely require understanding document structure that is lost in text linearization.

The six properties work together as a coherent measurement framework. Extraction (1) ensures answers are grounded. Multi-hop (2) ensures answers require integration. Closed-world (3) ensures integration is over the corpus, not parametric knowledge. Grounded (4) enables precise attribution measurement. Agentic (5) ensures the retrieval process requires planning. Visual (6) ensures the planning must account for non-textual information. A benchmark satisfying all six simultaneously can validly claim to measure agentic multimodal reasoning over document collections; a benchmark missing any one property measures something else.

Document Corpus Curation

The corpus was manually curated from DocumentCloud, a public repository of primary-source documents contributed by journalists, researchers, and organizations. The curation strategy was not random sampling but intentional clustering: the authors specifically sought groups of up to 30 related documents — sequential annual reports from the same organization, menus from different restaurants in the same city, legal filings from related cases — because these clusters enable realistic cross-document multi-hop questions (e.g., "compare the revenue from 2019 and 2020 across two separate annual reports" or "which restaurant has the most expensive spaghetti across these five menus").

The resulting corpus contains 800 PDF documents totaling 18,619 pages and 12.2 million tokens (using Qwen3 tokenizer). The length distribution is heavily right-skewed: median 5 pages, mean 23.3 pages, maximum 859 pages (an expense document). This heterogeneity is deliberate — it means agents must handle both single-page posters and book-length reports within the same evaluation, preventing strategies that assume uniform document length.

Documents span 63 fine-grained categories grouped into 13 high-level domains (Table 2): Financial (131 docs, 460 questions), Reports (127 docs, 360 questions), Government/Regulatory (105 docs, 304 questions), Legal (69 docs, 182 questions), HR/Employment (68 docs, 159 questions), Reference (62 docs, 218 questions), Miscellaneous (56 docs, 92 questions), Events (43 docs, 88 questions), Financial/Tax (39 docs, 82 questions), Media/Publishing (31 docs, 113 questions), Technical (29 docs, 68 questions), Education (26 docs, 68 questions), and Cases/Logs (14 docs, 55 questions). The category-level breakdown (Table 6 in Appendix B.1) reveals fine-grained document types including Annual Reports, 990 Forms, Verdict Forms, News/Journal articles, Expense Reports, Lesson Plans, Toxicology Reports, Inspection Reports, Performance Reviews, Termination Letters, Financial Disclosure Reports, Financial Statements, Form 8-K filings, Incident Reports, Tax Filings, Guides, Market Reports, Arrest Reports, Sustainability Reports, Search Warrants, Climate Action Plans, Missing Person Posters, Meeting Agendas, Public Records Requests, Contact Lists, Campaign Finance Reports, Resumes, Warrants, Notifications, Fines, Restaurant menus, Yearbooks, Data Sheets, Contracts, Audit Reports, Firearms Discharge Reports, Manuals, Patents, Employee Handbooks, Letters, Settlements, Budgets, Building Permits, Invitations, Leaderboards, Posters, Damage Reports, Contest rules, Crop Reports, Biographies, Guaranties, Case Logs, Conference Agendas, Marriage/Birth records, NDAs, Job Offers, Specifications, Speed Camera records, Diplomas, Flight Plans, and SEC Filings.

Layout diversity was quantified by running the Granite-Docling MLLM (Livathinos et al., 2025) over all pages to extract layout elements (text, tables, figures, lists, headers, checkboxes, footnotes, etc.), then computing per-element z-scores across categories (Appendix B.2). The resulting heatmap (Figure 2, with full detail in Figure 11) reveals systematic domain-specific patterns: financial and government documents exhibit high table density (z > 2 for tables in 990 Form, Budget, and Financial Statement categories), technical documents are figure-heavy (z > 2 for pictures in Specification, Patent, and Data Sheet categories), legal documents show elevated text density with minimal visuals (high z for text blocks, low z for pictures and tables in Settlement and Verdict Form categories), and reference materials like catalogs and guides contain diverse lists. This heterogeneity ensures the benchmark tests robustness across layout types — an agent that performs well on text-heavy legal documents may fail on table-dense financial filings, and vice versa.

Why this curation strategy: the combination of clustered documents (enabling multi-hop), broad domain coverage (preventing overfitting to any single document type), and heterogeneous layouts (requiring visual robustness) is what makes the benchmark's properties achievable. Without document clusters, cross-document reasoning would be artificial (comparing unrelated documents). Without broad domains, the benchmark would measure domain expertise rather than general agentic capability. Without layout heterogeneity, the visual property would be trivial (all documents share the same structure).

Question Annotation and Quality Assurance

The annotation process required over 1,200 hours of professional annotation work and proceeded through multiple stages designed to eliminate ambiguity, errors, and annotation artifacts.

Annotator selection and training. The authors contracted a professional data vendor with full-time employees experienced in labeling QA datasets over PDF documents. Twenty candidate annotators were initially provided identical guidelines and asked to annotate the same group of documents. Their annotations were subjected to a two-step verification: (1) GPT-5 was given the human-annotated oracle evidence pages for each question — if the model failed to answer correctly using the perfect context, the instance was flagged for manual review (indicating either the evidence was insufficient or the answer was ambiguous), and (2) only annotators with zero errors in manual review were selected to participate in the full MADQA annotation. This two-step filter ensures that the resulting dataset is solvable by construction: if GPT-5 with oracle evidence can answer every question, then the questions are well-formed and the evidence is complete.

Annotation protocol (Appendix C.1). Annotators followed strict guidelines designed to enforce the six properties:

1. Solvability constraint: Questions must be answerable entirely from the provided documents, strictly excluding external world knowledge. This enforces the Closed-World property (3). Generic questions like "What is the capital of Poland?" are forbidden; questions must require document access.

2. Unambiguity constraint: Questions must be specific enough to pinpoint a unique answer but must not reveal the source location too easily. Two anti-patterns were explicitly prohibited: self-referential location cues ("What is the title of this document?" — ambiguous in a collection of many documents) and trivial retrieval cues ("What is the email on page 4?" — turns the task into "go to page 4 and read," bypassing the retrieval challenge). Acceptable questions assume uniqueness in the corpus but do not indicate where: "What is the email address of Site Director William Wood?"

3. Evidence completeness: For every question, annotators collected the minimal set of evidence pages required to answer. For multi-hop questions, this means tagging all pages contributing to the final answer, not just the page containing the final value. For example, if answering "Which lesson plan suggests a lower instructor ratio: Firearms or New Mexico Justice System?" requires reading the Firearms lesson plan (page 5 of document A) and the New Mexico Justice System lesson plan (page 12 of document B), both pages are included in the gold evidence set — even if the answer comes from comparing values across them, neither page alone suffices. If evidence appears multiple times (duplication across documents), only the most plausible or first occurrence is noted.

4. Multi-hop targeting: Annotators were instructed to ensure approximately 20% of questions required multiple pieces of evidence. These questions fall into specific reasoning categories:

   • Bridging: The answer to part A is required to find part B (e.g., "What is the national bird of the nation that has a negative carbon footprint?" — find the nation in Doc A, then find its bird in Doc B).
   • Comparison: Aggregating values across documents (e.g., "Which country has won more soccer World Cups: Argentina or Brazil?").
   • Common Properties: Intersection of attributes (e.g., "Who is the only person to win an Olympic medal and a Nobel prize?").

Supervision, checks, and corrections. A domain expert from the authors' institution maintained constant synchronization with contractors to resolve ambiguities in real-time and enforce strict adherence to the guidelines. After the initial dataset was composed, the authors reviewed questions that were not answered correctly by any tested baseline model or any human annotator from the human baseline study. This led to two types of corrections:

• Replacement (< 1% of questions, mostly typos): Questions where the intended answer was genuinely unrecoverable due to annotation error.
• Extension (< 5% of questions, mostly adding optional, desirable context): Cases where the gold evidence set was expanded to include pages that, while not strictly necessary for answering, provided helpful context that made the answer unambiguous.

Annotation statistics (Table 2, Section 2.2). The final dataset contains 2,250 QA pairs distributed across 800 documents, averaging 2.8 questions per document. The multi-hop distribution closely matches the 20% target: 17.3% of questions are multi-hop (8.3% cross-page within the same document, 9.0% cross-document requiring multiple documents). The remaining 82.7% are single-hop, meaning all required evidence appears on a single page — though that page may still require visual interpretation (tables, forms, figures) and may not be retrievable by a single keyword query (satisfying the Agentic property even for single-hop questions if the evidence page has no lexical overlap with the question).

Why this annotation methodology: the combination of professional annotators (not crowdworkers), multi-stage verification (both automated via GPT-5 and manual by domain experts), and iterative refinement based on baseline model performance distinguishes MADQA from benchmarks that rely on semi-automated annotation pipelines or single-pass human labeling. The GPT-5 verification step is particularly clever: it uses a frontier model as an "oracle reader" — if the model cannot answer even when handed the correct evidence, the annotation is flawed. This catches errors that human review might miss (e.g., subtle ambiguities in answer format, subtly insufficient evidence that a human annotator, knowing the answer, would overlook). The iterative refinement based on model and human performance — replacing genuinely unsolvable questions and extending evidence where context was insufficient — ensures the benchmark is solvable by construction, eliminating the common benchmark pathology where some fraction of "errors" are actually malformed questions.

Construct Validity Analyses

Following the measurement philosophy of Bean et al. (2025), the paper explicitly validates that MADQA measures its intended construct (agentic multimodal reasoning) rather than confounded alternatives (lexical pattern matching, parametric knowledge recall, or text-only comprehension). Three analyses establish this construct validity.

Analysis 1: Lexical Overlap vs. Reasoning (Appendix E.1).

This analysis tests whether the benchmark can be solved by simple "Ctrl+F" keyword matching — a proxy for whether the Agentic property (5) genuinely holds. For each question, the authors extract n-grams (n ∈ {1, 2, 3}) from the question text after removing stopwords and common question words, then measure three quantities across the full 18,619-page corpus:

• Pages matched: how many corpus pages contain at least one question n-gram.
• Precision: what fraction of matched pages are gold evidence pages.
• Recall: what fraction of gold evidence pages are retrieved.

The results (Table 8) are stark:

Interpretation: Unigrams match nearly the entire corpus (median 4,125 pages — 22% of all pages) with 0.03% precision — the question vocabulary is so common that it appears everywhere, drowning any signal in false positives. Bigrams reduce the candidate set dramatically (median 24 pages) but precision remains terrible at 2.6% — only 1 in 40 matched pages is actually evidence. Trigrams achieve the best precision (1.9%) and a manageable candidate set (median 1 page), but recall drops to 51% — half of all gold evidence pages contain no trigram overlap with the question, meaning they would be completely invisible to any trigram-based retrieval strategy regardless of ranking.

Why this matters: this analysis directly validates the Agentic property. Since even perfect trigram matching misses half of evidence pages, no single keyword query — no matter how cleverly constructed — can retrieve all evidence. Agents must use semantic understanding to bridge the lexical gap between question vocabulary and document vocabulary. The benchmark cannot be reduced to a keyword search problem.

Analysis 2: Parametric Knowledge vs. Grounding (Appendix E.2).

This analysis quantifies the upper bound on "free" performance — what fraction of answers can models guess correctly without seeing any documents. This establishes how much of measured accuracy genuinely reflects document comprehension versus training data contamination.

Procedure: Six frontier models (Claude Haiku 4.5, Claude Sonnet 4.5, Gemini 2.5 Flash, Gemini 2.5 Pro, GPT-5, GPT-5 Mini) were prompted (Prompt 1) to answer all test questions without access to any document text or images. The prompt explicitly instructed models to guess even when uncertain, selecting randomly for yes/no or binary choice questions. Each correctly guessed answer was then classified by GPT-5 Mini (Prompt 2) into three categories:

• Yes/No: Questions with binary yes/no answers (random baseline: 50%).
• Other Binary Choice: Questions explicitly listing two options (random baseline: 50%).
• Memorization: All other questions requiring recall of specific facts, names, dates, or values.

Results (Table 9): Guessability ranges from 9.1% (Claude Haiku) to 15.2% (GPT-5), with an average of 11.2%. Decomposing this:

• Approximately 3.0% (average across models) stems from random chance on yes/no and binary questions — the expected 50% accuracy on these question types, given their frequency in the test set.
• Approximately 5.1% stems from memorization — models correctly recalling specific facts (names, dates, values) from training data.
• The remaining fraction (total minus random) — approximately 8.2% — represents total knowledge-based answering, including both memorization and above-chance performance on yes/no/binary questions.

Examples of memorized facts that GPT-5 answered correctly without documents: "The 17,082 metric tons of SO2 emissions reported by TVA in 2020 were 99% below the peak level from which year?" → 1977; "In what year was Chicken Annie's Original established?" → 1934; "Who was the Special Inspector General for Afghanistan Reconstruction in 2014?" → John F. Sopko. These are public facts from documents that were likely indexed on the web and incorporated into pretraining data.

Why this matters: the 9-15% guessability represents an upper bound on performance achievable without document comprehension. When models achieve 80%+ accuracy with document evidence, the additional 70+ percentage points represent genuine document-grounded reasoning — not prior knowledge. This validates the Closed-World property (3): while some contamination exists (expected for public documents), it accounts for a small fraction of total performance, and the benchmark's signal is dominated by document comprehension. The explicit decomposition into random chance vs. memorization also provides a diagnostic: as models are trained on larger web corpora, the memorization component may increase, and the benchmark's difficulty should be recalibrated accordingly.

Analysis 3: Visual Perception (Appendix E.3).

This analysis quantifies Property 6 (Visual) — what fraction of questions genuinely require visual comprehension beyond what could be extracted from clean text.

Taxonomy: Each question was classified (using Gemini 3 Flash with Prompt 3, provided the question, gold answer, and rendered images of all gold evidence pages) into one of five mutually exclusive primary categories:

• Free Text (42.8%): Answer present in free-flowing narrative text; clean OCR extraction would be sufficient; no table, form, or layout comprehension required.
• Structured (28.8%): Requires navigating structured forms (permits, licenses, certificates, applications) to locate specific fields — distinguished from tables by the label-value pair structure rather than row-column correlation.
• Tabular (21.0%): Requires understanding tabular relationships — correlating rows and columns, comparing values across rows, or aggregating data from multiple cells.
• Artifacts (6.7%): Visual artifacts not captured above, including handwritten text (4.1%), charts/diagrams (1.8%), images/photos (0.3%), signatures, checkbox/tick mark status, stamps and seals.
• Other Spatial (0.7%): Requires understanding spatial positioning on the page (e.g., "at the bottom of the page") where position matters beyond what table/form structure captures.

Key finding: Only 42.8% of questions can be answered from free text alone. The remaining 57.2% require some form of visual or structural comprehension — understanding table structure (21.0%), navigating form layouts (28.8%), or interpreting visual artifacts (6.7%). While most questions (87.7%) require only a single visual modality, 12.3% benefit from multiple — most commonly Structured + Artifacts (7.0%), typically occurring when forms contain handwritten entries or checkbox fields.

Why this matters: this validates that the benchmark genuinely requires multimodal comprehension. A text-only system that OCRs documents and searches the resulting text could, in principle, solve the 42.8% of free-text questions. It would fail on the 57.2% involving structure and artifacts — unless it could infer tabular relationships and form structure from linearized text, a capability that varies significantly across architectures and is precisely what the benchmark is designed to measure. The taxonomy also provides a diagnostic lens: by reporting accuracy separately for each visual category, researchers can determine whether their system's failures are concentrated in table reading, form navigation, or artifact interpretation, enabling targeted improvements.

The construct validity framework as a whole establishes that MADQA measures what it claims to measure: the Agentic property is validated by the lexical overlap analysis (no single query suffices), the Closed-World property by the guessability analysis (contamination explains only 8-12% of performance), and the Visual property by the perception taxonomy (57% of questions require non-textual comprehension). These analyses are not post-hoc justifications — they are integral to the benchmark's design, following the principle that benchmarks should be validated as measurement instruments, not merely created as datasets.

Principled Splits Creation via Classical Test Theory

Rather than using random train/dev/test splits, the paper applies Classical Test Theory (CTT) (Crocker & Algina, 1986) to construct a test set that maximizes statistical power while preserving long-term relevance. This is a methodological contribution distinct from the dataset itself: it provides a principled framework for benchmark split creation that could be applied to other benchmarks.

Item statistics computation. For each of the 2,250 questions, two metrics are computed from the predictions of 16 diverse models (Table 7) spanning five model families (Google, Anthropic, OpenAI, Zhipu AI, Alibaba) and a wide capability range (ANLS* scores from 0.300 to 0.688):

• Item Difficulty (p_j): The mean accuracy of all 16 models on item j. Formally: $p_j = \frac{1}{M} \sum_{i=1}^{M} X_{ij}$ where M = 16 is the number of models and X_{ij} \in \{0, 1\} is the correctness of model i on item j. A lower p_j indicates a harder question. Note: this "p-value" is distinct from statistical hypothesis testing p-values; it follows CTT terminology.

• Item Discrimination (r_{pbis, j}): The point-biserial correlation between model performance on item j and total model performance (excluding item j to prevent autocorrelation). Formally: $r_{pbis, j} = \text{Corr}(X_{\cdot, j}, S_{-j})$ where S_{-j} = \sum_{k \neq j} X_{\cdot, k} is the total score of each model on all items except item j, and X_{\cdot, j} is the vector of binary outcomes on item j across all models. High discrimination means that strong models tend to get the item right and weak models tend to get it wrong — the item effectively separates capability levels.

Why these metrics: difficulty alone is insufficient for test set selection. A test set of uniformly hard questions (low p_j) would have low variance — all models perform poorly, and rankings become noise-dominated. A test set of uniformly easy questions (high p_j) would saturate quickly — once a model reaches ceiling performance, it cannot be distinguished from stronger models. Discrimination (r_{pbis}) identifies items that produce informative variance: questions where performance correlates with overall model quality, making them useful for ranking systems. The CTT approach optimizes for this informativeness.

Selection strategy (illustrated in Figure 5):

1. Sentinel Pool: Items with p_j ≤ 0.1 (solved by ≤ 10% of models) are reserved for a Sentinel Pool of 100 items (20% of the 500-item test set). These items typically have low discrimination (because all models fail them, there is little variance to correlate with overall performance) but are crucial for headroom preservation — they guarantee that the benchmark contains questions no current model can solve. As models improve over time, these items will transition from uniformly failed to discriminating (as strong models begin to solve them while weak models still fail), and eventually to uniformly solved, at which point the benchmark can be refreshed with new sentinel items. This design prevents the benchmark from saturating prematurely — a common failure mode where benchmarks lose discriminative power once frontier models approach ceiling performance.

2. Discrimination-maximizing selection: The remaining 400 items (80% of the test set) are drawn from items with p_j > 0.1. These items are stratified into 9 uniform difficulty bins (0.1 < p ≤ 0.2, 0.2 < p ≤ 0.3, ..., 0.9 < p ≤ 1.0), and within each bin, items are greedily selected in descending order of discrimination (r_{pbis}) until the budget is filled. This ensures the test set spans the full difficulty range (maintaining coverage of easy questions that weaker models can solve) while maximizing the discrimination signal within each difficulty stratum.

3. Development set: A separate 200-item Development set is selected using the same methodology but from the remaining items not selected for the Test set, ensuring non-overlapping splits. The remaining 1,550 items form the Train set — released with full annotations to facilitate RL-based optimization and prompt engineering.

Validation of the split quality. The resulting Test set (n = 500) achieves a Spearman rank correlation ρ > 0.85 with the complete benchmark (2,250 items) — meaning the relative ranking of models on the 500-item test set strongly predicts their ranking on the full dataset. This validates that the CTT-based selection preserves the benchmark's discriminative power while reducing evaluation cost by 78% (from 2,250 to 500 items). Figure 13 shows the difficulty distribution shift: the full dataset has mean difficulty 0.58 (slightly skewed toward easier items), while the Test set has mean difficulty 0.45 (shifted toward harder items due to the Sentinel Pool and discrimination-maximizing selection).

Why this methodology: random splits are the default in benchmark construction but are suboptimal for three reasons. First, they provide no guarantee of discriminative power — a random 500-item subset might, by chance, contain mostly easy or mostly hard items, producing rankings that are noisy or insensitive to capability differences. Second, they provide no mechanism for headroom preservation — as models improve, a randomly selected test set saturates uniformly, with no reserved hard items to continue discriminating frontier systems. Third, they waste annotation budget — if some items provide no discrimination signal (e.g., universally easy items that all models solve), including them in the test set adds evaluation cost without adding ranking information. The CTT approach directly optimizes for the properties that make a test set useful: high discrimination (items that separate strong from weak models), broad difficulty coverage (items at all capability levels), and explicit headroom (items beyond current frontier capabilities).

Evaluation Protocol: Answer Correctness

The primary correctness metric is LLM-based Accuracy, designed to balance the Extractive property (answers must be lexically grounded in the evidence) with the practical need to accept semantically correct responses that differ in surface form from the ground truth.

Answer representation. Answers are represented as lists of strings to accommodate multi-part responses (e.g., "list all board members" → ["Alice Smith", "Bob Jones", "Carol Williams"]). Ground truth answers may have multiple valid alternatives — the gold set G = {G^{(1)}, ..., G^{(k)}} contains all acceptable answer formats. The final score for a prediction P is: $\text{Score}(P) = \max_k \text{Metric}(P, G^{(k)})$ where Metric applies the two-stage evaluation pipeline described below, and the max over alternatives ensures that if the prediction matches any valid format, it receives full credit.

Two-stage evaluation pipeline:

Stage 1: Exact match. If the predicted answer (as a list of strings) exactly matches any gold alternative in all positions, the score is 1.0 and no LLM judge is invoked. This handles the ~50% of cases where agreement is perfect by construction.

Stage 2: LLM judge. If no exact match exists, an LLM judge (Gemini 2.5 Flash with function calling for structured output) evaluates semantic correctness using a chain-of-thought prompt (Prompt 4) developed through multiple rounds of human calibration.

LLM judge evaluation criteria (applied in strict sequence):

1. Refusal detection: Does the answer refuse or claim inability to respond? If yes → score 0.0 (Incorrect).
2. Content matching: Does the core meaning match any ground truth variant? If content is wrong or different → score 0.0.
3. Critical errors (any single one → score 0.0):
   • Missing scale qualifiers (e.g., predicting "50" when the gold answer is "$50 million" — the magnitude is wrong).
   • Binary questions (yes/no, true/false) without an explicit "Yes" or "No" in the answer.
   • Wrong entity or value (different person, company, number from what was asked).
   • Partial list with wrong items mixed in (if some items are correct but others are fabricated).
4. Format issues (only evaluated if content is correct): If the gold answer expects a list but the prediction is a comma-separated string with short, atomic items → score 0.5 (Partial — semantically correct but format violation).
5. Verbosity (only evaluated if content is correct):
   • Acceptable extra content: qualifying phrases, relevant context, clarifying details → score 1.0 (Correct).
   • Unacceptable extra content: unrequested details, over-specific precision → score 0.5 (Partial).
   • Deal-breaking verbosity: multi-sentence responses, full paragraphs, conversational preambles → score 0.0 (Incorrect).

Judge calibration. The prompt was iteratively refined through multiple rounds of human review on 100 stratified samples, identifying systematic disagreements and adjusting to address edge cases — particularly around list formatting, acceptable verbosity levels, and unit qualifier handling. This calibration improved human-LLM agreement from 82% to 90% on held-out samples.

Final validation. Excluding the ~50% exact-match cases (where agreement is perfect by construction), the calibrated judge achieves quadratic-weighted Cohen's κ = 0.88 with human judgments — indicating "almost perfect" agreement per Landis & Koch (1977). Sensitivity (probability LLM judges correct when human judges correct) was measured at q_1 = 0.980 on a 200-sample human evaluation. Specificity (probability LLM judges incorrect when human judges incorrect) was measured at q_0 = 1.000.

Bias correction. Following Lee et al. (2026), the paper applies Rogan-Gladen correction to adjust for LLM judge bias: $\hat{\theta} = \frac{\hat{p} + q_0 - 1}{q_0 + q_1 - 1}$ where \hat{p} is the raw LLM judgment score (the fraction of predictions judged correct), q_1 = 0.980 is measured sensitivity, and q_0 = 1.000 is measured specificity. Confidence intervals account for both test sample variance and calibration uncertainty.

What this computes: given the raw accuracy from the LLM judge (which may be biased — systematically more lenient or strict than a human evaluator), the bias-corrected accuracy \hat{\theta} estimates what a perfect human evaluator would assign. Under the measured calibration parameters (q_0 = 1.000, q_1 = 0.980), the correction is minimal (the LLM judge is well-calibrated) but the formula is included for methodological rigor.

Why this form: the Rogan-Gladen correction is the standard approach for adjusting prevalence estimates when a diagnostic test has known sensitivity and specificity. It accounts for the fact that even a highly accurate judge (κ = 0.88) has non-zero error rates, and these errors propagate to aggregate accuracy estimates. Without correction, a judge that is 98% sensitive and 100% specific would slightly underestimate true accuracy (because it misses 2% of truly correct answers). The correction adjusts for this, producing an unbiased estimate. The paper's transparency in reporting both raw and corrected scores — and the calibration parameters used — enables readers to assess the magnitude of correction and compare against other benchmarks that may or may not apply similar corrections.

Design choice: LLM judge over string-matching metrics. The authors initially considered ANLS* (Peer et al., 2025), a normalized edit-distance metric designed for document processing tasks. However, they found it "too strict — even after adding alternative answers, 35% of predictions where ANLS* assigned zero score were actually correct per human review" (Section 3.1 footnote). The LLM judge was motivated by this finding: exact-match or fuzzy-match metrics systematically penalize valid formatting variations that humans recognize as equivalent. The trade-off is that the LLM judge introduces its own bias (hence the calibration and correction), but the measured κ = 0.88 suggests this bias is small relative to the error reduction from accepting valid surface-form variations.

Only 5% of correct predictions are over-verbose (Section 5.1), indicating that when systems find the right evidence, they almost always provide the requested extractive answer — the LLM judge's verbosity criteria rarely trigger. This validates that the Extractive property is not overly restrictive: well-functioning agents naturally produce concise, grounded answers.

Evaluation Protocol: Retrieval and Attribution

Attribution is measured through two F1 metrics at different granularities, comparing the set of pages (or documents) cited by the agent against the gold minimal evidence set.

Page F1. For question i, let R_i be the set of unique page identifiers (tuples of (document_id, page_index)) cited by the agent, and let G_i be the gold minimal evidence set. Precision and recall are:

$P_i = \frac{|R_i \cap G_i|}{|R_i|}, \quad R_i = \frac{|R_i \cap G_i|}{|G_i|}$

The F1 score is the harmonic mean:

$\text{F1}_i = \begin{cases} 2 \cdot \frac{P_i \cdot R_i}{P_i + R_i} & \text{if } P_i + R_i > 0 \\ 0 & \text{otherwise} \end{cases}$

Edge cases: if the agent cites nothing (|R_i| = 0), P_i = 0 and F1_i = 0. The final benchmark Page F1 is the arithmetic mean over all N test samples: Score = (1/N) * Σ_i F1_i.

Doc F1. Identical computation, but R_i and G_i contain document identifiers only (not page numbers). This relaxes the localization requirement: an agent that correctly identifies the document but cites the wrong page (e.g., the Table of Contents instead of the specific clause on page 42) receives partial credit at the document level even though Page F1 penalizes the page-level error.

What these metrics compute together: Page F1 measures precise localization — did the agent cite the exact pages containing the evidence? Doc F1 measures coarse localization — did the agent at least identify the correct documents? The gap between Doc F1 and Page F1 diagnoses "last-mile" navigation failures: high Doc F1 with low Page F1 means the agent knows which document to look at but cannot pinpoint where within it. Low Doc F1 means the agent fundamentally failed to find the relevant documents.

Why this form: attribution is a core component of the RAG Triad (Madzou, 2024) — answer correctness, context relevance, and groundedness. A system can produce a correct answer (high Accuracy) with poor attribution (low Page F1) by either guessing correctly from parametric knowledge (violating Closed-World), citing irrelevant pages while coincidentally producing the right answer (spurious correlation), or citing entire documents rather than specific pages (lazy attribution). Separating Accuracy from Page F1/Doc F1 prevents these confounds from inflating perceived performance. An answer that is correct but poorly attributed is flagged as suspicious — it may be relying on memorization rather than genuine retrieval.

Design choice: page-level rather than bounding-box-level evidence. The paper deliberately chose page-level evidence annotations over finer granularity (bounding boxes, table cells, token spans). The rationale (Appendix A.3): a pilot study showed high inter-annotator agreement at page level, which also "aligns with how humans navigate documents and how retrieval systems operate." Finer granularity would introduce annotation noise — different annotators might draw slightly different bounding boxes around the same evidence — and would not correspond to how retrieval systems typically work (most systems retrieve pages or chunks, not pixel regions). For agentic evaluation, "page-level evidence suffices to verify correct navigation without introducing annotation noise."

Evaluation Protocol: Efficiency and Calibration (The Kuiper Statistic)

This is the paper's most novel metric and the one that directly addresses the motivating question: do agents reason (efficiently) or search (brute-force)? The Kuiper statistic quantifies effort calibration — whether the computational effort an agent expends on a problem correlates with success.

Motivation. Standard accuracy metrics are blind to computational cost. An agent that achieves 80% accuracy by issuing 50 search queries per question is fundamentally different from one that achieves 80% accuracy with 3 queries, but they appear identical on a leaderboard. The Kuiper statistic captures this difference by measuring whether accuracy conditioned on effort departs from the agent's overall mean accuracy. A well-calibrated agent should solve easy problems with low effort and only expend high effort on hard problems — but it should eventually succeed on those hard problems (or at least recognize futility and stop). A poorly calibrated agent either: (a) expends high effort on problems it ultimately fails (diminishing returns without recognition of futility), or (b) expends uniformly high effort regardless of problem difficulty (no strategic triage).

Formal construction. Given N test items, for each item i we observe a tuple (s_i, y_i), where:

• s_i ∈ ℕ is the effort — the number of discrete steps (tool calls, search actions) taken by the agent on item i.
• y_i ∈ {0, 1} is the outcome — binary correctness of the final answer.

Let \bar{y} = (1/N) * Σ_i y_i be the global mean accuracy.

The evaluation samples are sorted by ascending effort. Let π be a permutation of indices {1, ..., N} such that:

$s_{\pi(1)} \leq s_{\pi(2)} \leq \cdots \leq s_{\pi(N)}$

The Cumulative Difference sequence D = (D_0, D_1, ..., D_N) is defined as:

$D_0 = 0$

$D_k = \sum_{j=1}^{k} \left(y_{\pi(j)} - \bar{y}\right) \quad \text{for } k = 1, \ldots, N$

What this computes: D_k is the running sum of deviations from the mean, accumulated over items ordered by increasing effort. At step k, we have considered the k lowest-effort items. If the accuracy on these k items is above the global mean \bar{y}, the sum increases (positive slope — the agent is performing better than average on low-effort items). If the accuracy is below the global mean, the sum decreases (negative slope — the agent is performing worse than average on high-effort items).

Interpretation of the cumulative difference curve (Figure 6):

• Upward segments (positive slope) indicate effort ranges where accuracy exceeds the global mean — the agent is efficient in this regime, solving problems correctly with relatively little computation.
• Downward segments (negative slope) indicate effort ranges where accuracy falls below the global mean — the agent is inefficient, expending significant computation on problems it ultimately fails.
• Flat segments indicate effort ranges where accuracy roughly equals the global mean — effort and success are uncorrelated.

For a perfectly calibrated agent, D_k would start positive (easy problems, low effort, high accuracy), remain positive or plateau at zero (harder problems, higher effort, but accuracy proportional to effort investment), and never go deeply negative. For a poorly calibrated agent, D_k would go strongly negative at high k — the agent spends many steps on problems it fails, dragging the cumulative deviation below zero.

The Kuiper statistic summarizes the calibration quality into a single scalar:

$K = \max_{0 \leq k \leq N} D_k - \min_{0 \leq k \leq N} D_k$

What this computes: the total range of the cumulative deviation curve — the difference between its highest peak and lowest trough. A lower K indicates that accuracy is relatively independent of effort: P(y = 1 | s) is roughly constant across the effort spectrum. This can mean either: (a) the agent is well-calibrated, solving easy problems efficiently and avoiding wasted effort on hard problems, or (b) the agent's accuracy is uniformly low regardless of effort, producing a flat curve (zero range). This ambiguity is why Kuiper must be interpreted jointly with accuracy — low Kuiper with high accuracy is good; low Kuiper with low accuracy means the agent never succeeds regardless of effort.

A higher K indicates distinct regimes of over-performance and under-performance: the agent is much better than average at low effort and much worse than average at high effort. For agentic systems, this typically manifests as a sharp decline at high step counts — the agent invests many steps in problems it cannot solve, indicating poor strategic termination (failure to recognize when additional search is futile).

Why this form: the Kuiper statistic is chosen over alternatives (e.g., Pearson correlation between effort and accuracy) because it is non-parametric — it makes no assumption about the functional form of the relationship between effort and accuracy. A Pearson correlation would measure linear association; the Kuiper statistic captures any deviation from uniformity, including non-monotonic patterns (e.g., accuracy might be high at very low effort, dip at moderate effort, and recover at high effort). The cumulative difference method (Kloumann et al., 2024) on which Kuiper is based is specifically designed for comparing paired samples without distributional assumptions.

Sensitivity to effort measure (Appendix F.4). The paper investigates whether the choice of effort metric affects Kuiper scores. Four effort measures are compared where available: step counts (tool calls), total tokens (input + output), generated tokens (output only), and execution time. Pairwise Spearman rank correlations between measures are high (ρ = 0.72 to 0.95 for models achieving ≥ 60% accuracy, Table 11), indicating that sample orderings are largely preserved across definitions. The resulting Kuiper values vary by at most 20% across metrics (Table 12). For example, Claude Sonnet 4.5 BM25 Agent has Kuiper = 35.4 under step counts vs. 37.8 under generated tokens vs. 36.4 under total tokens. The default metric used in the paper is step counts (tool calls).

Why step counts as the default: step counts are interpretable (one step = one tool invocation or LLM call), system-agnostic (does not depend on tokenizer or hardware), and directly capture the agent's decision-making — each step represents a choice to continue exploring rather than terminate. Token-based measures conflate decision-making with verbosity (a model might produce verbose but content-free responses that inflate token counts without representing meaningful search effort). Time-based measures conflate computational effort with API latency and hardware variability.

Important caveat (Section 7, Limitations): "A 'step' can bundle different amounts of computation, retrieved context, or latency depending on tool design and model interface. As a result, Kuiper scores are most interpretable when comparing methods under a consistent agent/tooling regime." Cross-system Kuiper comparisons (e.g., BM25 Agent vs. RLM vs. Managed RAG) should be interpreted cautiously because the definition of a "step" differs — a BM25 Agent step is one search tool call, an RLM step is one recursive sub-LLM invocation, and a Managed RAG step is opaque (the service handles retrieval internally). Within-system comparisons (e.g., different models using the same BM25 Agent framework) are directly interpretable.

Baseline Implementations

The paper implements seven categories of baselines, spanning the design space from static retrieval to fully agentic exploration. I describe each in sufficient detail for reproducibility.

Baseline 1: BM25 MLLM Agent (Appendix G.1).

This is the primary agentic baseline, used for the broadest model coverage (17 systems across five model families, plus the human baseline using the same search tool). It couples text-based retrieval with multimodal reasoning in an iterative loop.

Index construction. Each document page's OCR-extracted text is indexed using the Whoosh search library. Each indexed entry contains: source filename, page number, and full OCR content. The index supports Boolean operators (AND, OR, NOT), phrase matching (via quotes: "Bell 407"), and wildcard patterns (* for multiple characters, ? for single character). The default retrieval returns k = 5 pages per query.

Agent loop (Algorithm 1). The agent operates for up to T = 10 maximum iterations. At each iteration:

1. The model receives the conversation history (system prompt + question + all previous search results and their rendered images) and can choose to call the search tool or the answer tool (via structured output / tool forcing).

2. If search is called with a natural language query, the Whoosh index returns the top-5 matching page identifiers. Each page is rendered as a high-resolution PNG image (for Anthropic's API, images exceeding the 5MB limit are progressively downscaled using Lanczos resampling). The rendered images — not the OCR text — are appended to the conversation. This means the agent sees the actual document pages as images, enabling it to leverage visual understanding for layout, tables, and figures.

3. If answer is called, the agent must produce: (a) an answer list (one-element list for single answers, multi-element for lists), and (b) citations specifying the exact source file and page number for each piece of evidence.

4. On the final iteration (t = T), the model is forced to provide an answer via constrained decoding (OpenAI) or tool forcing (Anthropic) — it cannot search further.

System prompt (Prompt 5): The agent is instructed that "the answer is definitely in the documents" and to "try different terms (synonyms, abbreviations, rephrasing)" if search returns no results. The prompt emphasizes concise, extractive answers: "Use as few words as possible (exact document words preferred). Do not write full sentences."

Why this design: the BM25 MLLM Agent represents a minimal agentic architecture — search + read, iterated. It has no explicit planning module, no memory beyond the conversation history, and no learned retrieval policy. Its success or failure is purely a function of the underlying MLLM's ability to: (a) formulate effective search queries from the question, (b) interpret search results (page images), (c) decide when to reformulate vs. answer, and (d) extract precise answers from relevant pages. This makes it an ideal "probe" for the benchmark: if a model performs poorly on this architecture, the failure can be attributed to deficiencies in these specific sub-capabilities, localized through the error taxonomy (retrieval vs. navigation vs. comprehension).

Baseline 2: Managed RAG Services (Appendix G.2).

Two proprietary RAG-as-a-Service offerings are evaluated as industry-standard reference points: Gemini File Search and OpenAI Assistants File Search. Both abstract the entire retrieval pipeline — chunking, embedding, indexing, query reformulation, re-ranking — behind a single API.

For Gemini File Search, all PDF documents are uploaded to a file search store. At query time, the Gemini model receives access to this store as a tool; one API call triggers retrieval, context injection, and answer generation atomically. The system returns grounding metadata indicating which document chunks were retrieved, though the internal retrieval strategy is opaque.

For OpenAI Assistants, documents are uploaded to a vector store with OpenAI's proprietary chunking and embedding. An Assistant configured with the file search tool automatically retrieves relevant chunks. The system operates via a threads/runs abstraction; each question creates a thread, triggers a run, and returns results with inline citations.

Common prompt (Prompt 6): Both services receive a minimal prompt instructing structured JSON output with answer (list) and citations (file + page). No task-specific fine-tuning or prompt engineering is applied — representing "out of the box" performance.

Why these baselines: Managed RAG services represent the deployment path of least resistance — what a practitioner would obtain by uploading documents and querying. They serve as a lower bound on agentic capability (since they perform single-shot retrieval with no iterative refinement) and as a calibration point for the cost of agentic approaches (is the additional computation of iterative search worth the accuracy gain?). The finding that agentic systems consistently outperform their managed RAG counterparts (Gemini 3 Pro BM25 Agent at 82.2% vs. Gemini 3 Pro File Search at 78.6%, Table 3) quantifies the value of iteration.

Baseline 3: HEAVEN (Hybrid Visual Retrieval, Appendix G.3).

HEAVEN (Kim et al., 2025) is a multi-vector retrieval system that combines single-vector efficiency with multi-vector accuracy. It operates in two stages:

Stage 1: DSE (Document Screenshot Embedding, Ma et al., 2024a) retrieves k_1 = 200 candidate pages using efficient single-vector embeddings.

Stage 2: ColQwen2.5 (Faysse et al., 2024) re-ranks candidates using multi-vector late interaction, producing k_2 = 5 final pages. A Visual-Snippet (VS) page reduction factor of r = 15 and query filter ratio ρ = 0.25 control efficiency. Stage weights are α = 0.1 and β = 0.3.

The retrieved pages are passed to a VLM with Prompt 7, which instructs extractive answering with page-level citations. All hyperparameters follow paper defaults (Table 13).

Why this baseline: HEAVEN represents the state-of-the-art in static (non-agentic) visual retrieval for document collections. It serves as an upper bound on what retrieval quality can be achieved without iterative agentic planning — if an agentic system cannot outperform HEAVEN's retrieval, the agentic overhead is not justified.

Baseline 4: Claude Agent with Semtools (Appendix G.4).

This baseline uses the Claude Agents SDK integrated with semtools, a suite of CLI tools for semantic document processing. The agent has access to three Unix-style utilities:

1. parse: Converts PDFs to Markdown using LlamaParse backend.
2. search: Performs semantic keyword search over text files using static embeddings, with configurable context windows (--n-lines), top-k retrieval (--top-k), and distance thresholding (--max-distance).
3. workspace: Manages persistent embedding caches for document collections.

The agent uses Claude's claude code system prompt preset with bash execution capabilities. Tool permissions allow Bash, Read, Glob, Grep, and Search operations. The CLAUDE.md configuration (Prompt 8) provides usage patterns for composing these tools into retrieval pipelines. The agent is instructed to search the converted Markdown versions of all PDFs and produce structured output with answer and page-level citations.

Why this baseline: this represents a more flexible agentic architecture than the BM25 MLLM Agent — the agent can compose arbitrary Unix pipelines, manage persistent state via workspaces, and choose between semantic and exact search. Its performance relative to the simpler BM25 Agent reveals whether tool flexibility improves agentic reasoning (by enabling more sophisticated retrieval strategies) or degrades it (by increasing the action space and making planning harder).

Baseline 5: Recursive Language Models (RLMs, Appendix G.5).

RLMs (Zhang et al., 2025a) represent a task-agnostic inference paradigm where the entire document corpus is loaded as a variable in a REPL (Read-Eval-Print Loop) environment. The model can write and execute code to: (1) chunk the corpus programmatically, (2) launch recursive sub-LLM calls via llm_query() to process subsets, and (3) aggregate results. The implementation uses the RLM library with Mistral OCR 3 for document conversion, augmented with explicit page markers ([PAGE N] ... [END PAGE N]) for citation extraction.

The model receives the full corpus text (linearly concatenated, with file and page markers) as a variable and a prompt (Prompt 9) instructing it to search and answer. The raw RLM response is post-processed through a structured output extraction step (using the same underlying model) to normalize answers and citations.

Why this baseline: RLMs represent the unconstrained extreme of the agentic design space — no fixed retrieval pipeline, no predefined search tools, just raw code execution with recursive LM calls. The finding (Section 5) that RLMs incur catastrophic computational overhead without matching constrained agents' accuracy (Claude Sonnet 4.5 RLM: $850 cost, 270M input tokens, 70.5% accuracy vs. Claude Sonnet 4.5 BM25 Agent: 80.6% accuracy) is a key empirical result supporting the paper's claim that "retrieval constraints are essential for cost-effective reasoning."

Baseline 6: MDocAgent and ColBERTv2 + LLaMA.

MDocAgent (Han et al., 2025): A fixed five-stage pipeline of specialized agents (General, Critical, Text, Image, Summarizing) that coordinates parallel text and image retrieval for document understanding.

ColBERTv2 + LLaMA-3.1-8B: A simple open-source baseline with late-interaction retrieval (Santhanam et al., 2022). Relevant pages are retrieved via ColBERTv2 and fed to Llama 3 8B for answer generation.

Why these baselines: MDocAgent represents a purpose-built document understanding system with explicit multi-agent coordination; ColBERTv2 + LLaMA represents a minimal open-source baseline. Together they span the spectrum from specialized architecture to off-the-shelf components.

Baseline 7: Human Performance (Appendix C.2).

To contextualize model performance and enable direct behavioral comparison, human annotators completed the test set under two conditions:

Human BM25 Agent: Annotators used the same Whoosh-based BM25 search engine as the BM25 MLLM Agent, accessed through a custom Streamlit web application (Figure 12). The interface provides: the question, a search bar supporting Boolean operators/phrase matching/wildcards, an interactive PDF viewer with page navigation, and the ability to mark pages as evidence. Complete interaction trajectories were logged: every search query with timestamp, results, page views, and final evidence selection. This enables direct comparison of search strategies, query reformulation behavior, first-query accuracy, and effort calibration between humans and agents using identical retrieval tools.

The 500 test questions were divided into 20 batches of 25, and annotators received standard instructions: read the question, search, navigate, mark evidence, enter the answer.

Human Oracle Retriever: A separate condition where annotators were shown the gold evidence pages directly — eliminating retrieval difficulty and reducing the task to reading comprehension. This establishes the upper bound on accuracy achievable with perfect retrieval.

Why these baselines: the human BM25 Agent is the paper's most informative baseline — it answers the question "how well could an intelligent agent perform using only keyword search?" The comparison between human BM25 Agent (82.2% accuracy, Kuiper = 14.6) and the best agentic system (Gemini 3 Pro BM25 Agent: 82.2% accuracy, Kuiper = 25.8) reveals that while frontier models match human accuracy, their effort calibration is substantially worse. The human oracle retriever (99.4% accuracy) reveals the 17.2% "Oracle Gap" — the accuracy ceiling imposed by imperfect search, which neither humans nor agents can exceed without better retrieval. The finding that human comprehension errors nearly disappear with perfect retrieval (Section 5.2) confirms that the benchmark's difficulty for humans lies primarily in retrieval, not reading comprehension — a property that should also hold for sufficiently capable agents.

4. Key Insights and Innovations

Innovation 1: Effort Calibration as a First-Class Evaluation Dimension

The field's dominant evaluation paradigm for document QA — and agentic systems more broadly — treats computation as an external resource: report accuracy, and if two systems achieve the same accuracy, they are equivalent. The paper's most fundamental conceptual move is to reject this premise and instead treat computation as an internal behavior that reveals the nature of the agent's reasoning process. The Kuiper statistic operationalizes this move by measuring not just whether an agent succeeded, but whether the effort it invested correlated with success — a signal that cleanly separates strategic reasoning from brute-force search.

What the field did before this innovation: Prior benchmarks (DocVQA, MP-DocVQA, MMLongBench-Doc, ViDoRE) evaluated agents on answer correctness alone, with occasional retrieval-quality metrics (recall@k, MRR) that measured whether the right documents were found but not how much computation was expended to find them. The implicit assumption was that accuracy subsumes all other dimensions of capability — if system A is more accurate than system B, system A is better, regardless of whether it used 3 queries or 300. This assumption persists even in recent agentic benchmarks: BRIGHT and Researchy Questions report accuracy and retrieval metrics but provide no mechanism for evaluating whether agents allocate computation efficiently across problems of varying difficulty.

The metric closest to Kuiper in prior work is expected calibration error (ECE), which measures whether a model's reported confidence matches its actual accuracy. But ECE requires models to produce explicit confidence scores (which most agentic systems do not), and measures calibration of stated uncertainty, not calibration of computational investment. The paper's behavioral approach — inferring implicit uncertainty from observed effort, then measuring whether that effort correlates with success — is novel.

Why this is a fundamental shift, not incremental: The Kuiper statistic does not merely add another column to a leaderboard; it reframes what "evaluation" means for agentic systems. An agent is no longer characterized by a single accuracy number but by an accuracy-effort profile — a curve that reveals whether performance degrades gracefully or catastrophically as problems become harder (requiring more steps). Two agents with identical 82% accuracy can have dramatically different profiles: the human baseline achieves this with Kuiper = 14.6 (near-uniform efficiency), while Gemini 3 Pro achieves it with Kuiper = 25.8 (systematically wasting effort on problems it fails, Figure 10). The human is doing something qualitatively different — strategic termination, recognizing futility — that the agent lacks, and the Kuiper statistic surfaces this difference where accuracy alone would declare equivalence.

This reframing has downstream consequences beyond this paper. It suggests that agent evaluation should follow the template of computational learning theory, where algorithms are characterized by their sample complexity or regret bounds at different budget levels, rather than the template of static benchmarks, where systems converge to a single score. The paper's cold-start analysis (Figure 9) — showing that Gemini 3 Pro starts at ~12% first-query accuracy vs. humans at ~50% — reinforces this: the "best" system at convergence (after 9 steps) is not the best system under a constrained budget (≤ 3 steps). The Kuiper framework makes this budget-dependence explicit and measurable.

Tie to evidence: The Kuiper scores in Table 3 span a wide range (14.6 for humans to 73.2 for GPT-5 Mini BM25 Agent) and are not monotonic in accuracy (GPT-5 BM25 Agent achieves 77.7% accuracy but Kuiper = 52.6, substantially worse than Gemini 3 Pro at 82.2% and Kuiper = 25.8). The cumulative difference curves (Figure 10) visually confirm the interpretation: human and RLM curves remain relatively flat (low dependency of accuracy on effort), while agent curves show sharp declines at high step counts — the signature of brute-force search applied to problems the agent cannot solve.

Innovation 2: Decomposing Agent Failures into a Retrieval–Navigation–Comprehension Cascade

Prior benchmarks report aggregate accuracy, treating all errors as equivalent. The paper introduces a diagnostic failure taxonomy that decomposes every incorrect prediction into a four-stage cascade — retrieval failure (wrong document), navigation failure (wrong page within right document), comprehension failure (right page, wrong answer), and refusal (no answer) — using only citation metadata, requiring no subjective LLM classification for the error type assignment.

What the field did before this innovation: Error analysis in document QA has been largely qualitative (example-based) or coarse (text-vs-vision ablations). Benchmarks like DocVQA and InfographicVQA provide per-category accuracy breakdowns (by answer type, by document category) but do not decompose why an answer is wrong. Recent work like ViDoRe v3 reports retrieval metrics separately from answer metrics, but the two are not linked per-question — there is no way to determine, for a specific incorrect answer, whether retrieval failed or the reader model failed. This means a system with poor retrieval but excellent reading and a system with excellent retrieval but poor reading could produce identical accuracy and retrieval-recall scores, yet require entirely different improvements.

Why this is distinctive at the idea level: The cascade is not just a classification scheme — it is a measurement instrument for diagnosing bottlenecks. The key insight is that the taxonomy is mechanistic: each stage corresponds to a specific sub-capability that can be independently improved. Retrieval failure → improve search query formulation. Navigation failure → improve within-document localization (table of contents reading, section header awareness). Comprehension failure → improve visual reading (table interpretation, form field extraction, handwriting recognition). Refusal → improve strategic persistence (don't give up too early). This maps directly to actionable research directions, unlike aggregate accuracy which only says "get better."

The cascade also reveals how error composition shifts with model capability (Figure 15): weaker models are dominated by refusals (GPT-4.1 Nano: 48.2% of all predictions are "no answer"), mid-tier models by retrieval failures, and the strongest models by comprehension failures. This progression — from "can't even try" to "can find things but can't read them" — is a developmental trajectory that aggregate accuracy obscures. It tells researchers where to invest effort: if your model is in the retrieval-dominated regime, improving the reader will not help because the reader never sees the right pages; if it is in the comprehension-dominated regime, improving retrieval will yield diminishing returns.

Tie to evidence: The error decomposition in Figure 8 shows that Claude Sonnet 4.5 has the lowest retrieval failure rate among agents (4.0%) but the highest comprehension failure rate among top models (8.6%) — it finds content effectively but misinterprets it. This is a fundamentally different failure signature from GPT-5 (retrieval failure 11%, navigation failure 4%, comprehension failure 7%), despite similar aggregate accuracy. The retrieval-vs-comprehension scatter (Figure 16) reveals two distinct clusters of mid-tier models: retrieval-limited systems (Qwen3-VL, GLM-4.6V Flash) and comprehension-limited systems (GPT-4.1 family, Claude Haiku 4.5). These clusters would be invisible in an accuracy-only leaderboard.

Innovation 3: Human-Agent Comparative Analysis Reveals Complementary Competence, Not Just a Performance Gap

The paper provides the first controlled comparison of human and agent search strategies over document collections under identical retrieval tools (the same BM25 index, the same search interface). This design choice — giving humans the same constrained tool rather than unrestricted document access — enables isolating the strategic component of retrieval from the perceptual component, and reveals a finding more nuanced than "humans are better": humans and agents succeed on different questions, with remarkably low item-level agreement (Cohen's κ = 0.24 between humans and Gemini 3 Pro, Figure 20), despite identical aggregate accuracy (~82%).

What the field did before this innovation: Human baselines in document QA benchmarks are rare and, when present, typically provide humans with unrestricted document access (oracle retrieval) — establishing an upper bound on reading comprehension but telling us nothing about retrieval strategy. No prior benchmark has instrumented human searchers with the same retrieval interface as agentic systems, logged complete interaction trajectories, and compared query formulation strategies, first-query accuracy, and effort calibration head-to-head.

Why this is significant beyond raw performance: The low item-level agreement (κ = 0.24) reveals that humans and agents have complementary competencies, not a simple skill hierarchy. Of the 107 items where humans and Gemini 3 Pro disagree, 54 are solved only by humans and 53 only by the agent — a near-perfect split. The error profiles are qualitatively different: human-specific failures are dominated by comprehension errors (64%), reflecting attention fatigue on complex extractions (negation blindness, temporal reasoning failures, role conflation, as catalogued in Table 16). Agent-specific failures split evenly between retrieval (43%) and comprehension (43%). This complementarity suggests that hybrid human-agent pipelines — where the agent handles routine retrieval and the human verifies or handles edge cases — could exceed the accuracy ceiling of either alone, a finding with direct implications for enterprise deployment architectures.

The cold-start analysis (Figure 9) reveals a specific strategic deficiency: humans achieve ~50% accuracy on their first search query; the best agent achieves ~12%. Humans craft precise initial queries from context; agents issue vague queries and recover through iteration. This is not a capability gap in the traditional sense — the agent can eventually find the answer — but a planning gap: the agent cannot anticipate, from the question text alone, what search terms will be effective. This suggests that improvements in query formulation (possibly through fine-tuning on search trajectories, or through learned "query expansion" policies) could close the cold-start gap without improving retrieval or reading capabilities at all.

Tie to evidence: The pairwise agreement matrix (Figure 20) shows that model-model pairs within similar capability tiers share moderate agreement (κ ≈ 0.4–0.6), while the human row shows uniformly low agreement with all agents (κ = 0.06–0.24). This pattern — agents agreeing with each other more than with humans — is characteristic of systems that share systematic biases (similar query formulation strategies, similar visual reading failures). The human baseline serves as an "out-of-distribution" probe that exposes these shared biases, which cross-validation within the model family would miss.

Innovation 4: Retrieval Constraints as an Enabler, Not a Limitation, for Cost-Effective Agentic Reasoning

A common intuition in agent design is that providing more tools, more flexibility, and fewer constraints should enable better reasoning — the agent can choose the optimal strategy for each problem. The paper provides strong counter-evidence: the most unconstrained approach evaluated, Recursive Language Models (RLMs), which provide the model with programmatic access to the entire corpus and the ability to write arbitrary code for processing it, incurs catastrophic computational overhead without matching the accuracy of constrained agents. Claude Sonnet 4.5 RLM processed over 270 million input tokens at a cost of $850 (for 500 test questions) yet achieved only 70.5% accuracy, compared to the same model in the BM25 Agent framework at 80.6% accuracy (Figure 7, Table 3).

What the field did before this innovation: The dominant trend in agentic system design has been toward increasing flexibility: ReAct (Yao et al., 2023) gives agents interleaved reasoning and acting; Reflexion (Shinn et al., 2023) adds verbal self-reflection; Tree-of-Thoughts enables branching exploration; RLMs represent the logical endpoint where the agent can write arbitrary programs. The implicit assumption is that flexibility enables optimal strategy selection — the agent, being intelligent, will choose the right tool for the right problem. The paper challenges this assumption: unconstrained flexibility creates an action space so large that the agent cannot effectively navigate it, leading to inefficient exploration (processing far more tokens than necessary) without corresponding accuracy gains.

Why this is a fundamental insight, not just an empirical observation: The finding reframes the relationship between agent architecture and efficiency. The paper does not merely show that RLMs are expensive (which would be an implementation detail); it shows that the constraints imposed by simpler architectures (BM25 search with a fixed retrieval interface, limited iterations) are beneficial because they structure the agent's exploration. The BM25 Agent has exactly two actions: search (with natural language) or answer. This constrained action space forces the agent to formulate semantically meaningful queries (rather than writing code to grep the corpus, which would match lexically but miss semantically related content) and to commit to an answer within a bounded number of steps (rather than recursing indefinitely). The constraints are a form of inductive bias that channels the agent's reasoning into productive strategies.

This insight connects to broader themes in AI: the "bitter lesson" (Sutton, 2019) that methods leveraging computation outperform methods leveraging human knowledge, but also the "less is more" principle that constrained search often outperforms unconstrained search when the evaluation function is imperfect (as the BM25 retrieval function inevitably is). The paper's design space analysis (Appendix I, Table 18) explicitly frames this as a morphological choice — Control Flow architectures range from single-step RAG to fully unconstrained RLMs — and the empirical results provide evidence for where on this spectrum the optimal trade-off currently lies.

Tie to evidence: Figure 7 plots accuracy vs. cost (log-scale USD) for leading models under three paradigms: Managed RAG, BM25 Agent, and RLM. The RLM points (red) are systematically to the right (higher cost) and often lower (worse accuracy) than the corresponding BM25 Agent points (green). Claude Sonnet 4.5 RLM costs approximately 10× more than Claude Sonnet 4.5 BM25 Agent while achieving 10 percentage points lower accuracy. The Kuiper statistics reinforce the interpretation: RLMs exhibit more uniform effort-accuracy relationships (Kuiper = 22.9 for Gemini 3 Pro RLM, Kuiper = 30.0 for GPT-5.2 RLM, Figure 10) compared to BM25 Agents (Kuiper = 25.8–73.2), suggesting RLMs spread computation more evenly across problems rather than concentrating it where it helps — precisely the failure mode of an agent that cannot distinguish easy from hard problems and applies uniform (excessive) computation to both.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use MADQA, the paper's newly introduced benchmark of 2,250 human-authored questions grounded in 800 heterogeneous PDF documents (18,619 pages, 12.2M tokens). The primary evaluation is on the Test set (n = 500 questions), selected via Classical Test Theory to maximize discriminative power while preserving a Sentinel Pool of 100 items that no current model can solve. A Development set (n = 200) is used for strategy selection and prompt engineering; the remaining 1,550 items form the Train set, released with full annotations.

• Base model(s). The paper evaluates 17 models across five model families in the BM25 MLLM Agent configuration alone (Table 3), plus additional models in managed RAG, RLM, and specialized baselines. Families include Google Gemini (3 Pro, 2.5 Pro, 2.5 Flash), Anthropic Claude (Sonnet 4.5, Haiku 4.5), OpenAI GPT (5, 5.2, 5 Mini, 5 Nano, 4.1, 4.1 Mini, 4.1 Nano), Zhipu AI GLM (4.6V, 4.6V Flash), and Alibaba Qwen3-VL (235B Thinking, 32B Thinking, 8B Thinking, plus Instruct variants for item response analysis in Table 7). The diversity of model families, scales (from 8B to frontier), and capabilities (ANLS* scores from 0.300 to 0.688 in Table 7) is deliberate — the paper aims to characterize how agentic behavior changes with model capability, not just to identify the current best system.

• Metrics. Three complementary metrics are reported for all evaluated systems (Table 3):

  Accuracy (Section 3.1, Appendix F.1): LLM-based semantic correctness judged by Gemini 2.5 Flash, calibrated to human judgments at quadratic-weighted Cohen's κ = 0.88 on non-exact-match cases. Answers are scored against gold alternatives using a chain-of-thought evaluation across five criteria (refusal detection, content matching, critical errors, format issues, verbosity), producing a three-way classification: Correct (1.0), Partial (0.5), or Incorrect (0.0). Bias correction via Rogan-Gladen formula using measured sensitivity (0.980) and specificity (1.000) is applied. The paper reports accuracy with confidence intervals accounting for both test sample variance and calibration uncertainty.

  Page F1 and Doc F1 (Section 3.2, Appendix F.2): Harmonic mean of precision and recall over the set of cited pages (or documents) compared to the gold minimal evidence set. Page F1 uses page identifiers (document, page_index) as units; Doc F1 uses document identifiers only. Edge cases: zero citations → F1 = 0. The gap between Doc F1 and Page F1 diagnoses "last-mile" navigation failures — high Doc F1 with low Page F1 indicates correct document identification but wrong page citation.

  Kuiper statistic (Section 3.3, Appendix F.3): Measures effort calibration by computing the range of the cumulative difference curve D_k = Σ_{j=1}^k (y_{π(j)} - ȳ), where items are sorted by ascending effort (step count), y_i is binary correctness, and ȳ is global mean accuracy. K = max(D_k) - min(D_k). Lower K indicates effort-invariant performance (well-calibrated termination or uniformly effort-independent behavior); higher K indicates systematic over-performance at low effort and under-performance at high effort — the signature of brute-force search with poor strategic termination. Kuiper is reported only for agentic systems (non-agentic systems operate with fixed computational budgets, making effort calibration meaningless).

• Baselines. Seven categories of baselines span the design space from static retrieval to fully agentic exploration (Section 4):

  1. BM25 MLLM Agent (Appendix G.1): The primary agentic baseline, evaluated with 17 models. Couples Whoosh-based BM25 search with vision-language model reasoning in an iterative loop (max 10 iterations, 5 results per query). The agent formulates search queries, receives rendered page images (not OCR text), and decides when to answer vs. search further. This is the framework used for the error taxonomy, query reformulation analysis, and human-agent comparison.

  2. Managed RAG Services (Appendix G.2): Two proprietary RAG-as-a-Service offerings — Gemini File Search and OpenAI Assistants File Search — evaluated "out of the box" with minimal prompt engineering. These represent industry-standard single-shot retrieval baselines where chunking, embedding, and retrieval are handled opaquely by the service provider.

  3. HEAVEN (Kim et al., 2025; Appendix G.3): State-of-the-art static visual retrieval combining DSE (single-vector, k₁ = 200) with ColQwen2.5 re-ranking (multi-vector late interaction, k₂ = 5). Evaluated with GPT-5.2 and GPT-4o as the reader model.

  4. Claude Agent with Semtools (Appendix G.4): Claude Agents SDK with Unix-style semantic tools (parse, search, workspace). The agent can compose arbitrary bash pipelines for document processing, representing a more flexible agentic architecture than the BM25 Agent.

  5. Recursive Language Models (RLMs) (Zhang et al., 2025a; Appendix G.5): Unconstrained programmatic document processing where the entire corpus is loaded as a REPL variable and the model writes code to chunk, query sub-LLMs, and aggregate. Evaluated with GPT-5.2, Gemini 3 Pro, and Claude Sonnet 4.5.

  6. MDocAgent (Han et al., 2025) and ColBERTv2 + LLaMA-3.1-8B (Santhanam et al., 2022): Specialized multi-agent document understanding and minimal open-source baselines, respectively.

  7. Human Performance (Appendix C.2): Two conditions — Human BM25 Agent (same search tool as agentic baselines, complete trajectory logging) and Human Oracle Retriever (gold evidence pages provided directly). 500 test questions annotated by professional annotators.

• Generation budget / compute accounting. The primary unit of computational effort is step count — the number of tool calls (search queries) issued by agentic systems. For non-agentic systems (managed RAG, HEAVEN, ColBERTv2), the budget is fixed at 1 retrieval step. For agentic systems, the maximum is T = 10 iterations. Additional cost measures include inference cost in USD (Figure 7) and token counts (for RLMs). The Kuiper statistic uses step count as the default effort measure; Appendix F.4 validates that token-based and time-based measures correlate strongly with step counts (Spearman ρ = 0.72–0.95 for models ≥ 60% accuracy) and produce Kuiper values within 20% of the step-based values.

• Cross-validation / statistical protocol. The LLM judge's calibration is validated on a 200-sample human evaluation subset, measuring sensitivity (q₁ = 0.980) and specificity (q₀ = 1.000) for Rogan-Gladen bias correction. Confidence intervals for accuracy are computed accounting for both test sample variance and calibration uncertainty. The CTT-based test set selection uses predictions from 16 models to compute item difficulty and discrimination; the Sentinel Pool (p ≤ 0.1) and discrimination-maximizing selection (p > 0.1, stratified into 9 bins) produce a 500-item test set with Spearman ρ > 0.85 correlation with the full benchmark. The human baseline was collected using 20 batches of 25 questions each, with all batches completed by the same annotator pool.

Main Quantitative Results

Aggregate Results: The Full Leaderboard

Table 3 presents the primary evaluation results. I organize the discussion around the paper's four central empirical findings.

Finding 1: Simple agentic systems can outperform strong, static RAG.

The best-performing system, Gemini 3 Pro BM25 Agent, achieves 82.2% accuracy, representing a statistically significant improvement over its optimized non-agentic counterpart, Gemini 3 Pro File Search (78.6%). This 3.6 percentage point gap holds consistently across the GPT family: GPT-5 BM25 Agent (77.7%) substantially outperforms GPT-5 File Search (49.6%) — a 28.1 point gap — and GPT-5.2 BM25 Agent (67.8%) outperforms GPT-5.2 File Search (50.0%) by 17.8 points. The only exception is Gemini 2.5 Flash, where File Search (71.8%) outperforms the BM25 Agent (58.5%) — the authors note this model performs "exceptionally well with Google's managed RAG," suggesting the managed service's proprietary retrieval and chunking strategies compensate for the model's weaker query formulation capabilities.

The agentic advantage appears most dramatically for models that struggle with the managed RAG interface. GPT-4o HEAVEN achieves 48.6%, while the weakest BM25 Agent (GPT-4.1 Nano) achieves only 19.5% — indicating that agentic iteration amplifies existing model capability rather than compensating for fundamental weaknesses. The agentic advantage is not uniform: it requires the model to formulate effective search queries, interpret search results, and decide when to stop. Models incapable of these sub-tasks (GPT-4.1 Nano, Qwen3-VL 8B) perform worse as agents than in static RAG configurations where retrieval is handled externally.

Finding 2: Specialized solutions punch above their weight.

In the class of smaller models and specialized architectures, M3DocRAG achieves 61.6% accuracy and MDocAgent achieves 63.8%, both rivaling or exceeding larger commercial models. These systems significantly outperform the simple open-source baseline ColBERTv2 + LLaMA-3.1-8B (40.2%) and several BM25 Agents using much larger models (GPT-4.1 Nano at 19.5%, GLM-4.6V Flash at 46.0%, Qwen3-VL 8B at 47.3%). The performance gap — over 20 points between M3DocRAG and ColBERTv2 + LLaMA — highlights the potential of domain-specific innovations, particularly vision-aware retrieval (M3DocRAG encodes pages as images, capturing visual cues that text-only retrieval misses) and multi-agent coordination (MDocAgent's five-stage pipeline of specialized agents).

Finding 3: Retrieval constraints are essential for cost-effective reasoning.

Figure 7 plots accuracy against inference cost (log-scale USD). The most unconstrained approach — Recursive Language Models — incurs catastrophic overhead without matching constrained agents. Claude Sonnet 4.5 RLM processed over 270 million input tokens at a cost of $850 (for 500 test questions) yet achieved only 70.5% accuracy. The same model in the BM25 Agent framework achieved 80.6% accuracy — over 10 percentage points higher — at substantially lower cost. Gemini 3 Pro RLM (73.8%, Kuiper = 22.9) similarly underperformed its BM25 Agent counterpart (82.2%, Kuiper = 25.8). The RLM paradigm's lack of constraints leads to inefficient information processing: the model can write arbitrary code to process the corpus, but without structured retrieval interfaces, it explores the corpus inefficiently, processing orders of magnitude more tokens without corresponding accuracy gains.

The Kuiper statistics partially explain this: RLM systems show more uniform effort-accuracy relationships (flatter cumulative difference curves in Figure 10) compared to BM25 Agents, suggesting RLMs spread computation evenly across problems rather than concentrating it where it helps. However, the RLM Kuiper values (22.9–42.3) are not uniformly better than BM25 Agents (25.8–73.2) — some BM25 Agents show worse calibration, indicating that the Kuiper statistic measures calibration quality independently from the architecture's inherent efficiency.

Finding 4: Agents achieve superior page-level attribution.

Comparing Page F1 and Doc F1 across systems in Table 3 reveals a consistent pattern: managed RAG services often achieve high Doc F1 (Gemini 3 Pro File Search: 94.2% Doc F1) but substantially lower Page F1 (70.1%) — a 24.1 point gap. Agentic systems narrow this gap: Gemini 3 Pro BM25 Agent achieves Doc F1 of 90.2% and Page F1 of 78.5% — an 11.7 point gap, nearly half that of the managed RAG counterpart. The human BM25 Agent achieves similar numbers: Doc F1 93.4%, Page F1 79.3%, a 14.1 point gap.

This pattern indicates that managed RAG services successfully identify the correct document but struggle with precise within-document localization — they cite the right file but the wrong page. Agentic systems, through iterative search and page-level examination, localize more precisely. The diagnostic value of reporting both metrics is clear: high Doc F1 can mask "last-mile" navigation failures that Page F1 exposes. For systems where only Doc F1 is reported, these failures would be invisible.

GPT-5.2 BM25 Agent shows a particularly large gap: Doc F1 83.7%, Page F1 67.6% — a 16.1 point gap — suggesting that even with agentic iteration, some models struggle with within-document navigation. Claude Sonnet 4.5 BM25 Agent achieves the highest Page F1 among agents (79.1%), consistent with its lowest retrieval failure rate (4.0%, Figure 8) and aggressive query reformulation (highest mean cosine drift per step, 0.38, Figure 24).

Disaggregated Results: Error Decomposition

Figure 8 and the analyses in Section 5.1 and Appendix H.3 decompose system predictions into five categories: Correct (exact), Correct (verbose), Comprehension Failure (right page, wrong answer), Navigation Failure (wrong page, right document), Retrieval Failure (wrong document), and No Answer / Refusal. This decomposition reveals that the composition of errors, not just the total error rate, distinguishes strong from weak agents.

Aggregate error distribution. Across 17 BM25 MLLM Agent systems (8,499 predictions, 3,273 incorrect), the breakdown is: Retrieval Failure (35.7%), Comprehension Failure (28.8%), Navigation Failure (23.0%), and No Answer / Refusal (12.6%) — Table 17.

How error composition shifts with model capability. Figure 15 orders systems by increasing accuracy and shows the composition of incorrect predictions. The progression is striking: GPT-4.1 Nano (19.5% accuracy) is dominated by refusals — 48.2% of all predictions are "no answer," indicating the model abandons search prematurely. As capability increases, refusals diminish and retrieval failures become the dominant bottleneck: Qwen3-VL 8B (47.3% accuracy) shows 25.2% retrieval failure, Gemini 2.5 Pro (60.1%) shows 21.4%. At the highest capability levels, comprehension failures dominate: Claude Sonnet 4.5 (80.6% accuracy) shows 8.6% comprehension failure — the highest among top models — but only 4.0% retrieval failure, the lowest among all agents. This progression — from "can't even try" to "can find things but can't read them" — is a developmental trajectory that aggregate accuracy obscures.

Distinct error profiles at similar accuracy levels. The retrieval-vs-comprehension scatter (Figure 16) reveals that models at similar accuracy levels can have fundamentally different failure signatures. GPT-5.2 (67.8%) and Claude Haiku 4.5 (68.2%) achieve nearly identical accuracy, but GPT-5.2 has a higher retrieval failure rate (~11% vs. ~5%) while Claude Haiku has a higher comprehension failure rate (~11% vs. ~7%). These are different problems requiring different solutions: GPT-5.2 needs better query formulation; Claude Haiku needs better visual reading. An accuracy-only leaderboard would rank these models identically and suggest they are equivalent; the error decomposition reveals they would benefit from entirely different improvements.

Only 5% of correct predictions are over-verbose (Section 5.1), confirming that when systems find the right evidence, they almost always provide the requested extractive answer — the LLM judge's verbosity criteria rarely trigger. Among errors, 87.4% involve a concrete retrieval, navigation, or comprehension failure — not hallucination or refusal — confirming that the benchmark primarily tests information-seeking ability rather than generation quality.

Multi-Hop Complexity Analysis

The benchmark's multi-hop subset (17.3% of questions) is analyzed in Appendix H.1, revealing non-obvious complexity patterns.

Cross-document multi-hop is easier than same-document multi-hop. Counter-intuitively, cross-document questions (n = 186) show higher overall accuracy (75.7%) compared to same-document questions (61.2%) — Table 14 vs. Table 15. The authors hypothesize that cross-document questions often involve explicit comparisons (e.g., "which event occurs earlier?"), providing clearer task structure, while same-document reasoning requires subtle integration of information across distant sections of a single coherent document. This finding has implications for benchmark design: increasing the number of documents in a corpus does not necessarily increase difficulty if most questions involve explicit across-document comparisons; difficulty is driven by the relationship between evidence pieces, not merely their physical separation.

Semantic distance, not physical distance, drives same-document difficulty. For same-document multi-hop questions, accuracy drops by 38 percentage points when evidence spans semantically dissimilar contexts (semantic distance > 0.6: 34.8% accuracy) compared to semantically similar contexts (semantic distance < 0.15: 72.4% accuracy) — Table 14. Physical page distance shows near-zero correlation with accuracy (r = -0.06), while semantic distance shows moderate negative correlation (r = -0.26). Figure 14 visualizes this: accuracy varies strongly across semantic distance bins but is largely flat across physical distance bins. This means that integrating information from page 3 and page 42 of the same document is easy if both pages discuss related concepts, but integrating information from adjacent pages 3 and 4 can be very hard if the content is semantically disjoint. Retrieval difficulty is a function of conceptual coherence, not spatial proximity.

Error types differ by evidence complexity. Figure 17 breaks down errors by hop type for the top four systems. Cross-page (same-document) questions are consistently hardest, with error rates rising from ~13% for single-evidence to ~35% for cross-page. Retrieval failure accounts for a larger share of cross-page errors, suggesting that navigating within long documents is harder than finding separate documents — within-document retrieval requires finer-grained localization and is more susceptible to false positives (sections that mention the target concept but in the wrong context).

Error types differ by document domain. Figure 18 aggregates errors across the top four systems by document domain. Event-related documents are easiest (92% accuracy), likely due to structured formats with clear named entities. Media/Publishing (69%) and Financial (74%) documents are hardest, with Financial documents showing the highest retrieval failure rate — consistent with the challenge of navigating dense numerical tables across many pages.

Search Trajectory Analysis

The paper logs complete search trajectories for all BM25 MLLM Agents and the human baseline, enabling analysis of how retrieval succeeds or fails, not just whether it does.

First-query advantage. Figure 22 plots cumulative document and page retrieval rates by search iteration. Humans find the gold document on their first query ~80% of the time, while the best agent (Gemini 3 Pro) starts at ~70%. The gap narrows by iteration 4–5, but the human first-query advantage persists in page-level retrieval — humans not only find the right document sooner, but also the right page within that document.

Recovery after initial failure. Figure 23 measures, for questions where the first search query fails to retrieve the gold document, what fraction eventually succeed. The human baseline needs recovery for only 118 questions — far fewer than most agents (typically 250–380) — indicating that humans craft more precise initial queries. Among cases requiring recovery, top agents match human recovery rates: Claude Sonnet 4.5 and Gemini 3 Pro recover in >90% of cases, matching humans (~97%). Weaker models rarely recover: Gemini 2.5 Pro recovers in 56% of cases, GPT-4.1 Nano in only 12%. Much of the accuracy gap between strong and weak agents can be explained by recovery capability — not first-query precision.

Cold-start efficiency gap. Figure 9 plots accuracy as a function of step limit N, revealing a severe disparity. Human annotators achieve ~50% accuracy on their very first query, while Gemini 3 Pro starts at only ~12% — a 38 percentage point cold-start gap. The agent eventually converges near human accuracy after 9 steps, but this convergence requires steep, compute-intensive recovery. The human has an "illusion of infinite budget" — strategic calibration that enables efficient initial queries — that the agent lacks entirely. This finding directly ties to the Kuiper statistic: humans achieve low Kuiper (14.6) because they solve most problems with low effort; agents achieve higher Kuiper because they require high effort on many problems that humans solve quickly.

Query Reformulation Analysis

Appendix H.6 analyzes how agents reformulate search queries after initial failures, quantifying the relationship between reformulation magnitude and success.

Reformulation magnitude predicts success. Figure 24 shows the distribution of mean cosine drift per step (1 - cosine similarity between consecutive query embeddings) across systems. Top-performing systems reformulate more aggressively: Claude Sonnet 4.5 has the highest median drift (0.38), while GPT-4.1 Nano barely changes its queries (median drift 0.10). The right panel of Figure 25 shows a clear positive correlation between per-system mean drift and accuracy on reformulated questions — systems that make larger semantic jumps between queries are more likely to eventually find the answer.

Distribution of drift for correct vs. incorrect predictions. The left panel of Figure 25 shows drift distributions conditioned on correctness. Interestingly, the median drift for correct predictions (0.250) is only marginally higher than for incorrect predictions (0.239), and the distributions overlap substantially. This suggests that reformulation magnitude is necessary but not sufficient — aggressive reformulation helps on average, but some aggressive reformulations still fail (likely because the new query is effective but the underlying model cannot read the retrieved pages).

Single-query performance gap. Every system performs better on questions answered in a single query than on multi-query questions. The gap quantifies multi-step difficulty: Human achieves 86.4% on single-query vs. 79.5% on multi-query (a 6.9 point gap), while GPT-4.1 Nano drops from 46.1% to 13.2% — a 32.9 point gap. Weak models cannot effectively reformulate, so initial failure is catastrophic; strong models recover from initial failure, narrowing the single-vs-multi-query gap.

Human-Agent Comparative Analysis

Section 5.2 and Appendices H.2, H.4 provide the first controlled comparison of human and agent search strategies under identical retrieval tools.

Same accuracy, different competencies. Although humans and Gemini 3 Pro both achieve ~82% accuracy, pairwise item agreement is remarkably low: Cohen's κ = 0.24 (Figure 20). Of 107 disagreement items, 54 are solved only by humans and 53 only by the model — a near-perfect split. Model-model pairs at similar accuracy levels agree substantially more (κ ≈ 0.43), sharing systematic weaknesses. Human-specific failures are dominated by comprehension errors (64%), reflecting attention fatigue on complex extractions — negation blindness, temporal confusion, role conflation (Table 16). Model-specific failures split evenly between retrieval (43%) and comprehension (43%).

Perfect retrieval eliminates human reasoning errors. The human oracle retriever achieves 99.4% accuracy — close to ceiling. The comprehension errors that appear in the BM25 condition (attention fatigue, role conflation) nearly disappear when the annotator is focused on the right page. This confirms that the benchmark's difficulty for humans lies primarily in retrieval, not reading comprehension, and suggests that sufficiently capable agents with better retrieval would similarly approach oracle-level accuracy.

Humans calibrate effort better. Humans achieve a Kuiper statistic of 14.6 — below every agent system, whose scores range from 22.9 (Gemini 3 Pro RLM) to 73.2 (GPT-5 Mini BM25 Agent) — Table 3. Figure 10 visualizes the cumulative difference curves: the human curve remains relatively flat, while agent curves (particularly GPT-5 and GPT-5 Mini) show sharp declines at high step counts. Humans recognize when initial queries fail and change strategy; agents persist through minor query reformulations and spend compute on problems they ultimately fail to solve.

Response time inversely correlates with accuracy. Human median response time was 2 minutes (mean 3.3 minutes), with 50% of responses between 1–4 minutes. Questions answered in under one minute achieved 86% accuracy; those requiring over ten minutes achieved only 68%. This inverse correlation is the opposite of what a brute-force searcher would exhibit — it reflects strategic triage: humans spend more time on harder problems, and those harder problems are inherently less likely to be solved, producing lower accuracy conditional on high effort. The Kuiper statistic captures this pattern as a downward-sloping cumulative difference curve at high effort, which is exactly what Figure 10 shows for humans.

Human-agent complementarity. The low item-level agreement (κ = 0.24) combined with identical aggregate accuracy (~82%) and complementary error profiles (humans fail on comprehension, agents on retrieval) suggests that hybrid human-agent pipelines could exceed the accuracy ceiling of either alone. The paper does not test such a pipeline, but the data provides a clear rationale: route questions to the agent first, flag low-confidence or high-effort cases for human review, and leverage human comprehension strengths where agent retrieval succeeds but agent reading fails.

Ablation Studies and Robustness Checks

The paper's ablation studies are distributed across the appendices, each validating a specific component of the benchmark or evaluation protocol.

Effort measure choice for Kuiper statistic (Appendix F.4): The Kuiper statistic is computed under four effort definitions — step counts, total tokens, generated tokens, and execution time — for six systems where all measures were available (Table 12). For models achieving ≥ 60% accuracy, pairwise Spearman correlations between measures range from ρ = 0.72 to 0.95 (Table 11). The resulting Kuiper values vary by at most 20% across metrics: e.g., Claude Sonnet 4.5 BM25 Agent ranges from 35.4 (steps) to 37.8 (generated tokens). This confirms that the calibration assessment is robust to the choice of effort measure, though the paper appropriately cautions that cross-system Kuiper comparisons should be made under consistent tooling regimes (Section 7, Limitations).

LLM judge calibration (Appendix F.1): The LLM judge (Gemini 2.5 Flash) was calibrated against human judgments on two separate samples. On a 100-sample stratified set used for iterative prompt refinement, agreement improved from 82% to 90%. On a 200-sample held-out evaluation set, the judge achieved sensitivity q₁ = 0.980 and specificity q₀ = 1.000. The quadratic-weighted Cohen's κ = 0.88 on non-exact-match cases indicates "almost perfect" agreement. The Rogan-Gladen correction is applied but is minimal given the high specificity — the LLM judge's bias is small relative to its reliability. The paper transparently reports calibration parameters, enabling readers to assess correction magnitude.

ANLS vs. LLM judge (Section 3.1, footnote):* The authors initially considered ANLS* (Peer et al., 2025) but found it "too strict — even after adding alternative answers, 35% of predictions where ANLS* assigned zero score were actually correct per human review." This negative result motivated the LLM judge, and the 35% false-negative rate of string-matching metrics is a substantive finding for the document QA community: exact-match and fuzzy-match approaches systematically undercount accuracy on extractive tasks with formatting variation.

N-gram solvability analysis (Appendix E.1): The lexical overlap study (Table 8) shows that even trigram matching achieves only 51% recall and 1.9% precision — confirming that the benchmark cannot be reduced to keyword search. This validates the Agentic property (Property 5) by construction.

Parametric knowledge baseline (Appendix E.2): Six frontier models were prompted to answer questions without documents. Guessability ranges from 9.1% (Claude Haiku) to 15.2% (GPT-5), with ~8% from training data contamination and ~3% from random chance — Table 9. This quantifies the upper bound on "free" performance and validates the Closed-World property (Property 3). The decomposition into yes/no, binary choice, and memorization categories is a methodological contribution: future benchmarks can use this template to report contamination baselines transparently.

Visual necessity taxonomy (Appendix E.3): The Gemini 3 Flash-based classification of questions by visual modality requirement (Table 10) finds that only 42.8% of questions are solvable from free text alone, validating the Visual property (Property 6). The taxonomy provides a diagnostic lens for future research: systems can be evaluated separately on free-text, structured, tabular, and artifact questions to localize visual comprehension failures.

CTT-based split quality (Appendix D): The 500-item Test set achieves Spearman ρ > 0.85 with the full 2,250-item benchmark — validating that the CTT-based selection preserves ranking information while reducing evaluation cost by 78%. Figure 13 shows the difficulty distribution shifts from mean 0.58 (full dataset) to 0.45 (Test set), reflecting the deliberate inclusion of harder, more discriminating items.

Item difficulty spectrum (Appendix H.4): Of 500 test items, 23.8% are universally easy (>90% of models correct), 8.4% are universally hard (<10% correct), 66.8% are discriminating, and only 5 items (1.0%) are anomalous — easy for weak models but hard for strong ones (Figure 21). The near-absence of anomalous items confirms the benchmark's sound psychometric properties: difficulty increases monotonically with model capability. Among 86 items where humans fail, 24 have model difficulty ≥ 70%, revealing human-specific cognitive limitations on questions that most models solve reliably — further evidence of complementary human-agent competencies.

Domain- and evidence-complexity-based error analysis (Appendix H.1, Appendix H.3): Errors decomposed by document domain (Figure 18) show Media/Publishing (69% accuracy) and Financial (74%) as hardest domains; Events (92%) as easiest. Errors decomposed by evidence complexity (Figure 17) show cross-page questions are hardest for all top systems, with consistently elevated retrieval failure rates.

Failure cascade visualization (Figure 19): For four representative systems, the cumulative funnel from total questions → document found → correct page found → correct answer is visualized. This makes the progressive loss at each stage immediately visible: e.g., Gemini 3 Pro loses 17% of questions to wrong-document and wrong-page errors combined, with the remaining failures being comprehension errors on correctly-located evidence.

Critical Assessment

Claim 1: "Simple Agentic Systems Can Outperform Strong, Static RAG"

This claim is supported with important qualifications. Table 3 shows that Gemini 3 Pro BM25 Agent (82.2%) outperforms Gemini 3 Pro File Search (78.6%), and the GPT family shows dramatic improvements (GPT-5: 77.7% agentic vs. 49.6% static). However, the claimed superiority is not universal: Gemini 2.5 Flash File Search (71.8%) substantially outperforms Gemini 2.5 Flash BM25 Agent (58.5%). The paper acknowledges this exception but does not deeply investigate why this model-family-specific reversal occurs. Is Google's managed RAG using Gemini-specific optimizations (query reformulation, chunking strategies) that the generic BM25 Agent lacks? Is Gemini 2.5 Flash a weak query formulator that benefits from having retrieval handled by the managed service? The exception is noted but unexplained, which limits the generality of the claim.

A deeper issue is that the static RAG baselines (Managed RAG, HEAVEN) are not directly comparable to the agentic baselines in terms of retrieval architecture. Managed RAG services use proprietary chunking, embedding, and retrieval algorithms that may be more or less suited to specific model families. HEAVEN uses visual embeddings (ColQwen2.5) that the BM25 Agent lacks. The agentic advantage could be partly an artifact of BM25 being a strong-enough retriever that iterative refinement adds value, whereas the managed services might use weaker retrievers that single-shot approaches cannot compensate for. A fairer comparison would give both agentic and static systems access to the same underlying retrieval index. The paper's design — agentic systems use BM25, static systems use whatever the service provides — confounds architecture (agentic vs. static) with retrieval quality (BM25 vs. proprietary vs. visual embeddings).

Claim 2: "Effort Calibration Is a Distinct Axis from Accuracy"

This claim is strongly supported by the Kuiper statistics in Table 3 and Figure 10. GPT-5 BM25 Agent achieves 77.7% accuracy but Kuiper = 52.6, substantially worse than Gemini 3 Pro BM25 Agent (82.2%, Kuiper = 25.8). The wide range of Kuiper values among systems with similar accuracy (e.g., RLM systems at Kuiper 22.9-42.3 vs. BM25 Agents at 25.8-73.2) confirms that calibration is not redundant with accuracy.

However, the paper does not establish what constitutes a meaningful difference in Kuiper. Is Kuiper = 25.8 (Gemini 3 Pro) significantly better than Kuiper = 35.4 (Claude Sonnet 4.5)? The statistic has no associated confidence intervals or significance tests. The paper reports that Kuiper varies by up to 20% depending on effort measure choice (Appendix F.4) — does this mean differences smaller than 20% should be considered noise? Without establishing the reliability of Kuiper differences, the ranking of systems by calibration quality is suggestive but not definitive.

A deeper validity question: Kuiper measures whether effort correlates with success, but the interpretation of high Kuiper as "poor calibration" assumes that effort should be uniform across problems. This assumption may not hold for all agent architectures. An agent that deliberately spends more effort on harder problems and succeeds at a lower rate on those harder problems will have high Kuiper (because high-effort items have below-average accuracy), but this might reflect appropriate resource allocation — the agent correctly identifies hard problems and invests more, even if those problems remain hard. The paper interprets high Kuiper as evidence of "unproductive loops" (Section 5) and "persisting in searches that fail" (Section 5.2), but this interpretation requires the additional assumption that the extra effort does not improve outcomes relative to what would have happened with less effort. The Kuiper statistic alone cannot distinguish between "wasted effort" and "appropriate effort on inherently hard problems." The cold-start analysis (Figure 9) partially addresses this by showing that human first-query accuracy vastly exceeds agent first-query accuracy, suggesting agent effort is compensatory rather than strategic, but this is a separate analysis not directly tied to Kuiper values.

Claim 3: "Humans and Agents Succeed on Different Questions Despite Similar Accuracy"

This claim is strongly supported by the item-level agreement analysis. Cohen's κ = 0.24 between humans and Gemini 3 Pro (Figure 20) is remarkably low for systems with identical aggregate accuracy. The 54-53 split of human-only vs. agent-only successes on disagreement items is a striking pattern that would not emerge from random noise (which would produce asymmetric splits around the accuracy difference). The error type analysis — humans failing on comprehension (64%), agents splitting evenly between retrieval and comprehension — provides a mechanistic explanation for the complementarity.

The evidence would be stronger with a decomposition of which types of questions drive the disagreement. Are agent-only successes concentrated in specific document domains, visual categories, or question types? The per-domain error analysis (Figure 18) aggregates across the top four systems but does not break out human-vs-agent disagreements. Such a decomposition would reveal whether the complementarity is actionable (e.g., agents excel at structured data extraction from dense tables while humans excel at narrative reasoning across long documents) or merely idiosyncratic.

A practical limitation: the human baseline was collected from professional annotators familiar with the benchmark's document types and annotation guidelines. These annotators may not be representative of "typical" human performance — they are domain-aware, motivated, and working in controlled conditions. The 50% first-query accuracy may overstate what an average user would achieve. The paper does not report inter-annotator agreement or variance for the human baseline, making it difficult to assess whether the human-agent gap would replicate with a different annotator pool.

Claim 4: "Retrieval Constraints Are Essential for Cost-Effective Reasoning"

This claim is supported but the causal mechanism is not isolated. Figure 7 shows that RLMs cost dramatically more than BM25 Agents without matching their accuracy. However, the RLM framework differs from the BM25 Agent in multiple dimensions simultaneously: (1) it uses Mistral OCR 3 for document conversion rather than rendered page images, (2) it provides programmatic corpus access rather than structured search, (3) it has no built-in iteration limit, (4) it uses a different prompt format, and (5) it processes linearly concatenated text rather than per-page retrieval. The cost difference could be driven by any or all of these factors, not specifically by the "retrieval constraints" the paper emphasizes. An ablation that gave RLMs access to the same BM25 index as a tool, or that gave BM25 Agents programmatic corpus access without search constraints, would isolate the effect of the retrieval interface from other RLM-vs-Agent differences.

The cost analysis in Figure 7 uses USD, which depends on API pricing and tokenizer implementations that vary across providers and change over time. This makes the cost comparison non-reproducible — a reader evaluating the same systems six months later would encounter different pricing. The paper would be strengthened by reporting FLOPs or total tokens as the primary cost measure, with USD as supplementary context. Appendix F.4 reports token counts for some systems but does not provide the per-system token totals that would enable FLOPs-based comparison.

Claim 5: "An 18% Oracle Gap Reveals Retrieval Is the Primary Bottleneck"

This claim derives from comparing Human Oracle Retriever (99.4%) against the best agentic system (82.2%) — Table 3. The 17.2 percentage point gap is substantial, but its interpretation as "the primary bottleneck" requires careful qualification. The human oracle retriever eliminates both retrieval difficulty and the cognitive load of search strategy. Human annotators shown the correct pages can focus entirely on reading comprehension, without the attentional demands of formulating queries, evaluating search results, and maintaining state across retrieval steps. The 99.4% accuracy may reflect not just perfect retrieval but also reduced cognitive load, which improves comprehension even on the same pages.

An alternative interpretation: the oracle gap represents the combined effect of imperfect retrieval (search tool limitations) and the cognitive overhead of managing the search process (strategic planning load). These are distinct factors that current experiments cannot separate. To isolate retrieval quality from planning overhead, one would need to compare agent performance with oracle retrieval (providing agents the gold evidence pages directly, eliminating search entirely) against human oracle performance. If agents with oracle retrieval approach 99.4%, the gap is purely retrieval. If agents with oracle retrieval remain substantially below humans, there is a comprehension gap independent of retrieval. This experiment is not reported, making the "primary bottleneck" claim speculative rather than demonstrated.

Missing Experiments That Would Strengthen the Paper

1. Oracle retrieval for agents. Providing agentic systems with gold evidence pages would isolate comprehension capability from retrieval capability, revealing whether the oracle gap is driven by retrieval, reading, or both. This is the most important missing experiment for the paper's central claims.

2. Ablation of visual vs. text-only agent input. The BM25 MLLM Agent provides rendered page images; an ablation providing OCR text instead would measure the contribution of visual comprehension to agent performance. The visual necessity taxonomy (Appendix E.3) classifies questions by whether visual modality is beneficial, but performance on these categories under text-only vs. visual input is not reported. This experiment would directly validate the Visual property's practical importance.

3. Ablation of retrieval quality within agentic framework. All BM25 Agents use the same Whoosh index. An experiment varying the retrieval backend (BM25 vs. dense retrieval vs. hybrid) while keeping the agent architecture fixed would measure how retrieval quality interacts with agentic iteration. The paper shows that agentic iteration improves over static RAG, but does not show whether this improvement is larger or smaller with better retrievers.

4. Kuiper with confidence intervals. The paper reports Kuiper as point estimates without quantifying uncertainty. Bootstrap confidence intervals would establish whether observed Kuiper differences (e.g., 25.8 vs. 35.4) are statistically reliable. Without this, the calibration ranking is suggestive but cannot support strong comparative claims.

5. Scaling analysis across model sizes within the same family. The BM25 Agent is evaluated with GPT-5, GPT-5.2, GPT-5 Mini, GPT-5 Nano, GPT-4.1, GPT-4.1 Mini, and GPT-4.1 Nano — a range of scales within the GPT family. How do accuracy, error composition, and Kuiper change as model size increases? The paper reports these as rows in Table 3 but does not analyze the scaling trend within model families, which would reveal whether agentic capability emerges smoothly or discontinuously with scale.

6. Combining PRM-style verifier search with agentic retrieval. The paper's agentic baselines use no learned verifier or reward model to guide search termination. A natural extension — agents that use a confidence estimator to decide when to stop searching — is not explored, leaving open whether the Kuiper calibration failures reflect the absence of learned termination policies that could be trained from the logged trajectories.

6. Limitations and Trade-offs

Difficulty Estimation Cost and the Missing Practical Deployment Story

The assumption or constraint. The paper’s entire compute-optimal framework rests on estimating prompt difficulty before deciding how to allocate the inference budget. The method for doing so — generating 2,048 samples per question and averaging the PRM’s final-answer score — is extraordinarily expensive. The paper explicitly acknowledges this (Section 3.2):

"estimating difficulty in this way still incurs additional computation cost during inference… our experiments do not account for this cost largely for simplicity"

At 2,048 samples per question, the difficulty estimation step alone consumes more compute than the largest test-time budgets studied (256–512 generations). The paper frames this as "an exploration-exploitation tradeoff — compute spent assessing difficulty versus compute spent solving the problem — flagging it as a key avenue for future work." But crucially, no practical difficulty estimator is developed or evaluated. The predicted (non-oracle) bins still require 2,048 samples — they simply replace ground-truth correctness with PRM scores, removing the need for labels but not the computational cost.

The consequence. The reported 4× efficiency gains over best-of-N (Figures 4, 8) are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter. A system that generates 2,048 samples to learn that a question is "easy" and then uses 4 samples to solve it has not achieved 4× efficiency over best-of-256 — it has used 2,052 samples total, which is more expensive than just running best-of-256 uniformly.

This gap is not a mere implementation detail — it is the difference between a demonstration that compute-optimal scaling could work (given an oracle or impractical estimator) and evidence that it does work under realistic constraints. The paper’s central practical claim — that difficulty-adaptive allocation yields efficiency gains — cannot be evaluated without accounting for the cost of the adaptivity mechanism itself. For practitioners, the question is not "does compute-optimal scaling improve over best-of-N given free difficulty labels?" but "does compute-optimal scaling improve over best-of-N after paying for difficulty estimation?" The paper answers the former but not the latter.

What evidence exists in the paper. The paper reports that predicted difficulty bins (using PRM scores instead of ground-truth labels) perform nearly as well as oracle bins — the curves "largely overlap" in Figures 4 and 8. This demonstrates that if you are willing to pay the 2,048-sample cost, you can estimate difficulty without ground-truth access. But it does not address what happens if you use fewer samples, a lightweight classifier, or a dynamic estimation scheme. The paper acknowledges this gap explicitly (Section 8) by calling for future work on "pretraining or finetuning models to directly predict difficulty of a question."

Mitigation status. Not mitigated in the current work. The paper flags this as the primary bottleneck for practical deployment and suggests two directions: (1) lightweight difficulty classifiers trained on question text alone, and (2) adaptive estimation that amortizes difficulty assessment into the problem-solving process (e.g., using the first few samples as a difficulty signal and allocating the remaining budget accordingly). Neither is implemented or evaluated. Until this gap is closed, the headline efficiency gains should be understood as upper bounds on what is achievable, conditional on solving the difficulty estimation problem separately.

Hard Problems Remain Essentially Unsolved — Test-Time Compute Cannot Create Capability

The assumption or constraint. The paper’s framework assumes that the base model already produces correct solutions at some non-trivial rate. The compute-optimal policy amplifies this existing capability by allocating more test-time compute to problems where the base model has a fighting chance. The paper is transparent about this boundary (Section 7):

"test-time compute provides essentially zero benefit regardless of budget, meaning that some capabilities can only be acquired through pretraining, not recovered at inference time"

The consequence. On the hardest questions (difficulty bin 5), all methods — search, revisions, and their compute-optimal combinations — show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets (up to 256 generations). In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5% — no amount of additional test-time compute helps.

This is a fundamental limitation, not an engineering challenge. The revision model relies on the base model generating at least some correct solutions that it can learn to refine. The verifier-guided search relies on the base model’s proposal distribution containing correct solutions that search can find. When pass@1 is near zero, the proposal distribution is empty of correct solutions — there is nothing to find or refine. Test-time compute amplifies existing capability but cannot create it from nothing.

For practitioners, this means the compute-optimal framework offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model’s training distribution. If your deployment includes a substantial fraction of problems where the base model’s pass@1 is near zero — which is likely for any production system pushing beyond routine queries — the framework provides no guidance except "train a bigger model" or "collect more training data." The paper’s FLOPs-matched analysis (Figure 9) makes this explicit: for hard problems at moderate-to-high inference-to-pretraining ratios, pretraining is almost always more effective.

What evidence exists in the paper. The difficulty-bin analyses (Figures 3 right, 7 right, 9) consistently show bin 5 accuracy near floor levels. The FLOPs-matched comparison (Section 7) quantifies this as a sharp boundary: test-time compute is preferable on easy-to-medium problems at low inference-to-pretraining ratios (R ≪ 1), but pretraining dominates on hard problems across nearly all regimes. The paper’s error taxonomy for the BM25 Agent (Figure 8) shows that weaker models are dominated by refusals and retrieval failures — they cannot even begin to search effectively — which is a different failure mode from bin 5 but reflects the same underlying principle: some capabilities require pretraining.

Mitigation status. Not mitigated, and the paper does not claim to mitigate it. The authors acknowledge this as a fundamental boundary condition and frame it as a finding rather than a bug: "the failure case is equally informative: on the hardest problems… test-time compute provides essentially zero benefit." The implication for future work is that combining test-time compute with better pretraining (or retrieval, or tool use) is necessary to push beyond this ceiling. The Sentinel Pool design in MADQA (Section 2.4) — reserving items no current model can solve — explicitly anticipates that benchmarks need to maintain headroom beyond what test-time scaling alone can achieve.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate — Sequential Refinement Can Destroy Correct Answers

The assumption or constraint. The revision model was trained solely on sequences where all in-context answers are incorrect followed by a correct target (Section 6.1). This data construction was necessary — the paper needed incorrect-to-correct trajectories to teach the model to revise — but it creates a blind spot: the model never sees a correct answer in context during training, so it has no signal for what to do when the current answer is already right.

The consequence. At test time, the revision model may encounter correct answers in its context (produced during earlier revisions) and incorrectly "revise" them into wrong answers. The paper reports (Section 6.1) that:

"approximately 38% of correct answers get converted back to incorrect ones using a naive approach"

This is a catastrophic failure mode for sequential revision: the model’s improvement over successive steps (Figure 6, left — pass@1 rising from ~18% to ~24%) is partially offset by the reversion problem. The paper mitigates this with majority voting or verifier-based selection across the entire chain — rather than taking the final revision output, the system evaluates all answers in the chain and selects the best one. This works because even though 38% of correct answers are reverted, there exists at least one correct answer somewhere in many chains, and the selection mechanism can (imperfectly) recover it.

But this patch is fragile. If the selection mechanism is imperfect — and verifier quality degrades under distribution shift from revision model outputs (Appendix J, Figure 15a shows the base-LM PRM underperforms the revision-specific ORM) — some correct answers will be lost. More fundamentally, the reversion problem means the revision model is not a monotonic improver: longer chains do not necessarily produce better answers, and the optimal strategy involves retrospective selection across the chain rather than progressive refinement. This is a sharp departure from the intended semantics of "iterative revision" — the model does not consistently make answers better; it makes them different, sometimes better and sometimes worse, and external selection is required to extract value from the chain.

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1. Figure 6 (left) shows that pass@1 improves gradually across revision steps (from ~18% to ~24%), but this is the best answer selected from each step, not the sequential trajectory — it does not show reversion directly. The paper does not report the probability that a correct answer survives k additional revision steps, which would directly characterize the reversion problem’s severity as a function of chain length. The ReST^EM experiment (Appendix K, Figure 16) provides convergent evidence: attempting to optimize the revision model with RL-style training caused performance to degrade substantially with sequential revisions — "fully sequential performance drops to approximately 33.5% compared to roughly 38.5% at the optimal ratio" — suggesting the reversion problem is sensitive to training methodology and can worsen under optimization.

Mitigation status. Partially mitigated through within-chain selection (majority voting or verifier-based), which treats the revision chain as a source of diverse candidates rather than a refinement trajectory. The paper does not attempt to train the model to recognize when no revision is needed — a more principled solution that would address the root cause rather than the symptom. The revision training data could, in principle, be augmented with "correct → correct" trajectories (where the model is taught to output the same answer when the current answer is already right), but this is not explored.

No Combination of Search and Revisions — The Two Axes Are Studied Independently

The assumption or constraint. The paper studies two complementary test-time compute strategies — PRM-guided search (modifying the verifier) and iterative revisions (modifying the proposal distribution) — as independent mechanisms. The unifying framework in Section 2 explicitly positions them as complementary axes, but the experiments treat them separately: Section 5 evaluates search algorithms against the PRM using the base model’s (few-shot prompted) proposal distribution; Section 6 evaluates revisions using a separately trained ORM, without any PRM tree-search.

Section 8 explicitly acknowledges this gap:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The consequence. This is not merely an experimental omission — it means the paper’s results represent a lower bound on what the compute-optimal framework could achieve. The two mechanisms have complementary, difficulty-dependent strengths that the paper itself documents: revisions are most effective on easy problems (where the model’s initial output is roughly correct and needs local refinement — Figure 7, right, bin 1–2), while PRM search is most effective on medium-hard problems (where the model needs to explore qualitatively different solution strategies — Figure 3, right, bin 3–4). A combined system — using the revision model as the proposal distribution within beam search, or using the PRM to guide which revisions to pursue — could exploit both mechanisms’ strengths simultaneously, potentially yielding gains beyond either alone.

The FLOPs-matched comparison (Section 7) reinforces this gap: revisions outperform search on the FLOPs-matched analysis (Figure 9, left shows positive gains across all difficulties at R ≪ 1; Figure 9, right shows negative gains on medium and hard problems at moderate R). But this comparison treats them as alternatives rather than complements — a combined system might exceed both, particularly on medium-hard problems where search finds the right strategy and revisions refine the execution.

For practitioners, the paper’s current guidance is incomplete: it tells you how to allocate budget between sequential and parallel sampling for revisions, and how to choose between best-of-N and beam search for verifier-guided search. But it does not tell you how to allocate budget between revisions and search, or how to combine them. The compute-optimal policy chooses the best single strategy per difficulty bin, not the best combination.

What evidence exists in the paper. The evidence is entirely circumstantial — the difficulty-dependent patterns from Sections 5 and 6 show complementary strengths, but no experiment tests their combination. Appendix J (Figure 15a) shows that the base-LM PRM underperforms the revision-specific ORM on revision model outputs, confirming distribution shift as a practical obstacle to combining them — the PRM trained on base model outputs does not transfer to revision model outputs. But this is a measurement of the obstacle, not an attempt to overcome it (e.g., by training a PRM specifically on revision model outputs, or by using the revision model’s ORM within a search framework).

Mitigation status. Not mitigated. The paper frames this as future work (Section 8): "The natural next step is to use the revision model as the proposal distribution within beam search" and suggests specific architectures (PRM-guided revision pursuit, combined sequential-parallel with verifier-guided pruning). The current results are explicitly a lower bound, and the paper’s value is partly in establishing the complementary strengths that motivate the combination.

Single Benchmark, Single Model Family, Single Task — Generalization Is Unproven

The assumption or constraint. All experiments use the MATH benchmark (Hendrycks et al., 2021) — specifically, the split from Lightman et al. (2022) with 12,000 training questions and 500 test questions — and all use PaLM 2-S* (Codey) as the base model. The paper states (Section 4) that it "believe[s] this model is representative of the capabilities of many contemporary LLMs" and argues MATH is appropriate because "test-time compute is expected to help most when the model already possesses the necessary knowledge and the challenge is drawing complex inferences — mathematical reasoning fits this profile."

The consequence. Several aspects of the findings could be model- or benchmark-specific:

• PRM quality and over-optimization behavior depend on PaLM 2-S*’s output distribution. A model with different calibration properties, different error patterns, or different sensitivity to beam search depth would exhibit different difficulty-dependent scaling curves — potentially different thresholds for when search helps vs. hurts, and different optimal strategies per difficulty bin.

• The revision model’s ability to learn from edit-distance-paired trajectories depends on the base model’s in-context learning capabilities and the quality of the edit-distance heuristic for pairing incorrect and correct solutions. This heuristic assumes that structurally similar but incorrect solutions are "close" to correct solutions in a meaningful sense, which may not hold for all model families or task types.

• The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning and producing discrete, verifiable answers. It is unclear whether the difficulty-dependent patterns — beam search hurting easy problems (Figure 3, right), revisions excelling on easy problems (Figure 7, right), the sharp bin 5 failure boundary — generalize to other reasoning domains: code generation (where unit tests provide a different verifier signal), logical reasoning (where solutions have different structure), scientific QA (where factual knowledge may interact with reasoning), or tasks requiring open-ended generation rather than closed-form answers.

• The 500-question test set, split into five difficulty quintiles of ~100 each, then further split by two-fold cross-validation, means the compute-optimal policy is selected based on ~50 questions per fold per bin. This is a small sample — the selected strategies may not be robust, and the paper does not report confidence intervals on the compute-optimal scaling curves (Figures 4, 8), making it difficult to assess whether the observed gains are statistically reliable.

What evidence exists in the paper. The paper provides no cross-model or cross-benchmark validation. The choice of MATH is explicitly justified (Section 4) as a reasoning-heavy, knowledge-light benchmark that isolates inference capability from factual recall. The choice of PaLM 2-S* is justified as "representative" but this claim is not tested. The few alternative approaches attempted — PRM800k training data (from GPT-4-generated solutions) was found "largely ineffective" for PaLM 2 models (Section 5.1), and ReST^EM revision training degraded performance (Appendix K) — actually suggest that findings are sensitive to model-specific characteristics and training procedures.

Mitigation status. Not mitigated. The paper is a single-benchmark, single-model study, and the authors do not claim otherwise. This is a standard limitation of scaling law analyses — the computational cost of the experiments (generating 2,048 samples per question for 500 questions, training PRMs and revision models, sweeping over search hyperparameters at multiple budget levels) makes cross-model or cross-benchmark replication expensive. The defense is that the framework (difficulty-conditioned allocation, the proposal-verifier decomposition, the Kuiper-style calibration analysis) is general and should transfer, even if the specific thresholds and optimal strategies are model-dependent. But this defense is untested — a reader cannot distinguish between "this framework generalizes" and "this framework happens to work for PaLM 2-S* on MATH."

The 14× Larger Model Baseline Is Not Compute-Optimally Trained — FLOPs-Matched Comparison Favors Test-Time Compute

The assumption or constraint. The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately 14× more parameters, where the larger model scales parameters only while holding training data fixed. The paper acknowledges (Section 7) that this follows the LLaMA paradigm (Touvron et al., 2023) rather than compute-optimal pretraining (Hoffmann et al., 2022):

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the 14× larger model uses only greedy decoding — no majority voting, no best-of-N, no search, and no compute-optimal allocation of its own. This is a deliberate design choice to isolate the effect of test-time compute, but it makes the baseline substantially weaker than a fair comparison would warrant.

The consequence. Both design choices — parameter-only scaling and greedy-only decoding — systematically favor test-time compute in the FLOPs-matched comparison:

• Compute-optimal pretraining would produce a stronger baseline. A Chinchilla-optimal model trained with 14× more total FLOPs (scaling both parameters and data equally per Hoffmann et al., 2022) would likely outperform the parameter-only-scaled model used here. The paper’s reported advantages of test-time compute over pretraining — e.g., +27.8% relative improvement on easy questions at R ≪ 1 (Figure 1, top-right bar chart) — would shrink or potentially reverse against a properly compute-optimal larger model. The magnitude of this effect is unknown because the paper does not compare against a Chinchilla-optimal baseline.

• Greedy-only decoding for the larger model is an asymmetric handicap. The paper’s central claim is that test-time compute can substitute for pretraining compute. But the comparison gives the smaller model the full benefit of the compute-optimal framework (difficulty estimation, strategy selection, search/revision at multiple budget levels) while giving the larger model none of these benefits — not even a simple best-of-N baseline. A fair comparison would give the larger model some test-time compute budget as well, then ask: given a fixed total FLOPs budget, what is the optimal split between pretraining and inference compute? The current comparison answers a narrower question: can a small model with optimized inference beat a large model with no inference optimization? The answer is yes (for easy problems at low inference-to-pretraining ratios), but this is less surprising and less actionable than the broader claim.

What evidence exists in the paper. The paper is transparent about the parameter-only scaling choice (Section 7) and the greedy decoding choice (the 14× larger model's performance is shown as stars in Figure 9, placed at three x-axis positions corresponding to the three R values). The FLOPs accounting (Section 7) uses standard approximations from scaling laws:

X = 6ND_pretrain  (pretraining FLOPs)
Y = 2ND_inference  (inference FLOPs)

and derives the test-time compute multiplier for the smaller model to match total FLOPs under parameter-only scaling. The derivation is correct, but the interpretation as evidence that test-time compute can substitute for pretraining depends on the baseline being competitive.

Mitigation status. The paper explicitly flags this as a limitation and a direction for future work (Section 7): "leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." The parameter-only baseline is presented as "representative of a canonical approach to scaling pretraining compute" (LLA-based scaling) rather than as the optimal pretraining strategy. But the paper’s framing throughout — e.g., the title of Section 7's takeaway box ("FLOPs-Matched Comparison: Test-Time Compute Can Substitute for Pretraining — With Sharp Boundaries") and the presentation in Figure 1 — implies a stronger claim than the baseline supports. A more accurate summary would be: "test-time compute can beat a parameter-scaled (but not compute-optimally trained) larger model using greedy decoding, on easy-to-medium problems." This is still a useful finding, but it is substantially weaker than "test-time compute beats pretraining."

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the evaluation of multimodal document agents from a correctness-only paradigm to one where computational effort is a first-class signal about reasoning quality. Before MADQA, the field's default assumption was that an agent achieving 82% accuracy is equivalent to any other agent achieving 82% accuracy — the score subsumes all relevant dimensions of capability. The paper demolishes this assumption with a single, devastating finding: the human baseline and Gemini 3 Pro both achieve ~82% accuracy on MADQA (Table 3), yet share only κ = 0.24 item-level agreement (Figure 20) — they succeed on fundamentally different questions, with different error profiles (humans: 64% comprehension failures, 2.2% retrieval failures; Gemini 3 Pro: 43% retrieval failures, 43% comprehension failures). Aggregating these two systems into a single accuracy number erases precisely the information that distinguishes strategic reasoning from stochastic search.

The magnitude of this shift is not paradigm-level — the paper does not introduce a new model architecture or learning algorithm — but it is a substantive reframing of what "evaluation" means for agentic systems. The Kuiper statistic (Section 3.3) operationalizes this reframing: instead of reporting a scalar accuracy, report an accuracy-effort profile that reveals whether an agent's computational investment correlates with success. A well-calibrated agent solves easy problems with low effort and only expends high effort on problems where that effort pays off; a poorly calibrated agent expends high effort on problems it ultimately fails (Kuiper = 52.6 for GPT-5 BM25 Agent at 77.7% accuracy, vs. Kuiper = 25.8 for Gemini 3 Pro at 82.2% — Table 3). The cold-start analysis (Figure 9) provides the most visceral demonstration: humans achieve ~50% accuracy on their very first search query; the best agent achieves ~12%. The agent can eventually recover (converging to human-level accuracy by step 9), but the recovery requires brute-force iteration — the signature of a system that does not know what it is looking for until it has looked everywhere.

This reframing has methodological and practical consequences for the field:

Prior contradictions become explicable. The paper resolves a tension in the document QA literature that was previously attributed to benchmark quality or model capability: why do some studies find that agentic iteration helps while others find it adds cost without benefit? The Kuiper analysis provides the diagnostic: agentic iteration helps when the agent can formulate queries that bridge the semantic gap between question and evidence (Claude Sonnet 4.5: 4.0% retrieval failure, Kuiper = 35.4), and hurts when the agent reformulates ineffectively (GPT-4.1 Nano: median query drift 0.10, Figure 24; 48.2% refusal rate, Figure 8). The conflicting prior findings reflect differences in underlying model capability for query formulation, not differences in benchmark difficulty or architectural choices. This converts a confusing set of contradictory results into a coherent picture with clear boundary conditions: test agentic architectures only on models that can actually use them.

Research directions that become more attractive:

• Learned termination policies. The Kuiper analysis identifies a specific failure mode — agents persist in unproductive search loops rather than recognizing futility and either escalating to a human or providing a qualified answer. The paper's error taxonomy (Figure 15) shows that weaker models are dominated by refusals (giving up too early), while stronger models are dominated by retrieval and comprehension failures (not giving up early enough, but still failing). This suggests a "Goldilocks" problem — the optimal termination policy depends on model capability and question difficulty — that is ripe for learning-based solutions. The logged trajectories from the BM25 Agent framework (5,800+ search queries with outcomes, Appendix H.5) provide training data for a termination classifier that could substantially improve Kuiper scores without improving underlying retrieval or reading capabilities.

• Cold-start query formulation. The 38 percentage point first-query accuracy gap between humans (50%) and the best agent (12%) — Figure 9 — is not a retrieval problem (the agent eventually finds the evidence) and not a reading problem (the agent can read the evidence once found). It is a planning problem: the agent cannot anticipate, from the question text alone, what search terms will be effective. This is a well-scoped, measurable capability that could be targeted through fine-tuning on human search trajectories (the logged queries from the human baseline, Appendix H.5) or through learned query expansion policies.

• Human-agent collaborative pipelines. The complementarity finding — humans and agents succeed on different questions with κ = 0.24 agreement — is not just an academic observation. It directly implies that a hybrid system routing easy-looking or agent-suited questions to the model and hard-looking or human-suited questions to a human reviewer could exceed the 82% ceiling of either alone. The paper does not test such a pipeline, but the data provides a concrete routing signal: questions where the agent's initial search returns high-confidence matches are likely agent-suited; questions where the agent struggles (many reformulations, low retrieval precision) are candidates for human escalation.

Research directions that become less attractive:

• Unconstrained agentic exploration. The RLM results (Claude Sonnet 4.5 RLM: $850 cost, 270M input tokens, 70.5% accuracy vs. 80.6% for the BM25 Agent — Figure 7) provide a strong negative signal: giving agents more tools, more flexibility, and fewer constraints does not improve reasoning and catastrophically increases cost. The paper's finding that retrieval constraints are beneficial — they structure the agent's exploration and prevent wasteful computation — suggests that future research should focus on better constraints (learned retrieval interfaces, adaptive budget allocation) rather than fewer constraints (arbitrary code execution, unbounded recursion). The action space of agentic document QA is large enough that unconstrained exploration is inefficient; the inductive bias of structured search is necessary for cost-effective reasoning.

• Static RAG as a competitive baseline for complex document tasks. The managed RAG baselines consistently underperform their agentic counterparts (Table 3), and the Page F1 gap (Gemini 3 Pro File Search: Doc F1 94.2%, Page F1 70.1% — a 24.1 point gap) reveals that even strong managed services struggle with precise within-document localization. For applications requiring exact page-level citation — legal document review, financial compliance, medical record analysis — static RAG is insufficient regardless of the underlying retriever quality, because single-shot retrieval cannot adapt to the semantic structure of long documents. The field should shift investment from improving static retrieval architectures to developing agentic systems that can navigate documents iteratively.

Follow-Up Research This Work Enables

1. Training query formulation policies from human search trajectories. The paper logs complete search trajectories for both human annotators and 17 agent systems using the same BM25 search index (Appendix H.5, H.6). Humans achieve 50% first-query accuracy while the best agent achieves 12% (Figure 9) — a gap that is entirely attributable to query formulation, since both use the same retrieval backend. A natural follow-up would fine-tune an MLLM on human search trajectories: given a question q, predict the initial search query q' that a human would issue. The logged data provides 500 query trajectories from human annotators (one per test question) with outcomes, enabling supervised fine-tuning. A strong result would close >50% of the first-query accuracy gap (from 12% to at least 31%), demonstrating that query formulation is learnable from human demonstrations. A negative result — human trajectories are too idiosyncratic or context-dependent to transfer to model behavior — would suggest that the cold-start gap requires fundamentally different query representations (e.g., structured query plans rather than natural language strings).

2. Learning termination classifiers from agent error signatures. The error taxonomy (Figure 8, Appendix H.3) decomposes every incorrect prediction into retrieval failure, navigation failure, comprehension failure, and refusal — all deterministically computable from citation metadata, requiring no subjective judgment. For each question q and agent trajectory τ (sequence of search queries, retrieved pages, and intermediate answers), the outcome is known. A termination classifier trained on these trajectories could predict, after k steps, whether the agent is likely to succeed or whether further search is futile. The key experiment: train on trajectories from strong models (Gemini 3 Pro, Claude Sonnet 4.5) and test on held-out trajectories from the same models, measuring whether early termination (stopping the agent when the classifier predicts failure) improves Kuiper scores without reducing accuracy. The paper's Kuiper analysis (Table 3) shows that GPT-5 BM25 Agent (accuracy 77.7%, Kuiper = 52.6) has substantially worse calibration than Gemini 3 Pro (82.2%, Kuiper = 25.8) — a learned termination policy trained on the better-calibrated model's trajectories should close this gap.

3. Ablating retrieval quality within the agentic framework to isolate the "agentic advantage." The BM25 MLLM Agent evaluation uses a single retrieval backend (Whoosh BM25). The paper shows that agentic iteration improves over static RAG (Gemini 3 Pro BM25 Agent: 82.2% vs. Gemini 3 Pro File Search: 78.6% — Table 3), but this improvement could be an artifact of BM25 being a weak-enough retriever that iteration adds value — with a stronger retriever, the agentic advantage might diminish or disappear. A crucial ablation: replace BM25 with a dense retriever (e.g., ColQwen2.5 embeddings) in the same agentic framework, and measure whether the agentic-vs-static gap shrinks. If the gap persists, the agentic advantage is genuine — it reflects the value of iterative planning, not compensation for weak retrieval. If the gap disappears, the paper's central claim about agentic systems "outperforming strong, static RAG" is retrieval-dependent and may not generalize to production systems with state-of-the-art retrieval. This experiment is straightforward to implement (swap the Whoosh index for a vector database) and would substantially strengthen (or qualify) the paper's conclusions.

4. Measuring cross-model generalization of the error taxonomy. The error decomposition (Section 5.1, Appendix H.3) shows that error composition shifts systematically with model capability: weaker models are dominated by refusals, mid-tier models by retrieval failures, strongest models by comprehension failures (Figure 15). But all error analyses use the same BM25 Agent framework — it is unclear whether these patterns reflect properties of the models or properties of the framework interacting with model capability. A cross-framework replication — applying the same error decomposition to RLM trajectories, Managed RAG outputs, and Claude Agent with Semtools — would test whether the "refusal → retrieval → comprehension" progression is a universal developmental trajectory for document agents, or whether it depends on the retrieval interface. This is primarily a measurement contribution: it would establish the error taxonomy as a framework-independent diagnostic tool rather than a BM25-Agent-specific analysis.

5. Combining visual and text-only agent inputs to characterize when visual encoding matters. The visual necessity taxonomy (Appendix E.3) classifies 57.2% of questions as requiring visual comprehension — but this classification was done by an MLLM (Gemini 3 Flash) judging from gold evidence pages. The practical question is whether agent performance degrades when visual input is replaced with text-only input (OCR or Markdown conversion). An ablation: run the BM25 Agent with (a) rendered page images (the default), (b) OCR text from the same pages, and (c) Docling-converted Markdown, measuring accuracy separately for each visual category (free text, structured, tabular, artifacts). The hypothesis: visual input is essential for tabular questions (where spatial relationships between cells encode meaning) and artifact questions (handwriting, checkboxes), but text input may suffice for structured forms (where label-value pairs survive linearization) and free-text questions. This ablation would directly quantify the practical value of visual encoding for agentic document QA and identify which document types justify the higher cost of visual models.

6. Testing whether the Kuiper statistic responds to architectural improvements in the expected direction. The paper introduces Kuiper as a calibration metric but never demonstrates that it is responsive — that improving an agent's strategic behavior produces lower Kuiper scores. A validation experiment: take a poorly calibrated agent (e.g., GPT-5 Mini BM25 Agent: Kuiper = 73.2, Table 3) and apply an intervention known to improve strategic behavior — e.g., adding a hard step limit of 3 instead of 10, or adding a confidence threshold that forces early answering — and measure whether Kuiper decreases. If Kuiper responds as expected (lower Kuiper when the agent is forced to commit earlier), it validates the metric as a measure of calibration quality. If Kuiper does not respond, or responds unpredictably, the metric may be capturing noise or artifacts of the effort distribution rather than genuine calibration. This is a methodological validation experiment that would increase confidence in Kuiper as a leaderboard metric.

Practical Applications and Downstream Use Cases

Cost-efficient enterprise document review with human-in-the-loop routing. The complementarity finding — humans and agents succeed on different questions with κ = 0.24 agreement (Figure 20) — directly motivates a triage architecture for enterprise settings (legal document review, financial compliance, contract analysis). An organization with a corpus of thousands of documents and a stream of information-seeking queries could: (1) route each query first to an agentic system with a constrained budget (e.g., max 3 search steps), (2) flag queries where the agent's confidence (proxied by retrieval precision, answer format, or a learned termination classifier) falls below a threshold, and (3) escalate flagged queries to human reviewers. The paper's data suggests this pipeline could achieve accuracy above the 82% ceiling of either humans or agents alone — since they fail on complementary question types — while keeping human review costs manageable (only ~50% of queries would require escalation, based on the 54-53 split of human-only vs. agent-only successes on disagreement items). The Kuiper analysis provides the diagnostic for tuning the escalation threshold: set it to minimize Kuiper (i.e., minimize wasted human effort on questions the agent would have solved).

On-device or low-latency document QA with cold-start-optimized models. The cold-start analysis (Figure 9) reveals that the best agent achieves only ~12% first-query accuracy, requiring 4-5 iterations to match human-level performance. For latency-sensitive applications — mobile document assistants, real-time customer support over policy documents, interactive research tools — this iteration overhead is unacceptable. A model fine-tuned on human first-query formulations (as described in Follow-Up Direction 1) could close much of this gap, enabling single-query accuracy of ~30-40% (still below the human 50%, but 2-3× better than current agents). This would make agentic document QA viable in settings where users expect sub-second response times, not the multiple seconds required for iterative search. The paper's logged human trajectories (500 questions with full search histories) provide the training data; the key engineering challenge is distilling the query formulation capability into a small, fast model suitable for on-device deployment.

Benchmark design methodology for agentic capabilities. Beyond the specific MADQA dataset, the paper's methodological contributions — Classical Test Theory-based split creation (Section 2.4), construct validity analysis (Section 2.3), effort calibration measurement (Section 3.3), and error decomposition from citation metadata (Section 5.1) — constitute a template for building agentic benchmarks in other domains. An organization building a benchmark for agentic code generation, scientific literature review, or multimodal data analysis could directly adapt: (1) define desired properties analogous to the six MADQA properties (Extractive, Multi-Hop, Closed-World, Grounded, Agentic, Visual), (2) validate that the benchmark measures those properties through lexical overlap, parametric knowledge, and modality necessity analyses, (3) use CTT to select a discriminative test set with a Sentinel Pool for headroom preservation, (4) report Kuiper alongside accuracy to capture effort calibration, and (5) decompose errors using citation metadata to provide diagnostic signals. The paper's explicit commitment to the construct validity framework of Bean et al. (2025) — and its demonstration that this framework can be operationalized for a complex, multimodal task — lowers the barrier for other benchmark creators to adopt principled measurement practices.