Rethinking Agentic Search with Pi-Serini: Is Lexical Retrieval Sufficient?

ArXiv: 2605.10848

🎯 Pitch

Your expensive dense retriever might be overkill—PI-SERINI hits 83.1% accuracy and 94.7% evidence recall on deep research benchmarks using nothing but tuned BM25 and smart tool-use, outperforming dense-based systems while slashing costs by up to 10×. The secret isn't a fancier retriever but simply giving the agent enough retrieval depth and letting a capable LLM decide when to browse versus read.

1. Executive Summary

This paper revisits whether lexical retrieval can suffice for deep research as LLMs become more capable in an agentic loop, introducing PI-SERINI, a deliberately minimal search agent that separates retrieval, result browsing, and document reading into distinct tool calls to manage cached rankings and selective evidence acquisition. On BrowseComp-Plus with BM25, PI-SERINI paired with gpt-5.5 achieves 83.1% answer accuracy and 94.7% surfaced evidence recall, outperforming released search agents that use dense retrievers while reducing evaluation cost by 3.3×–10×. Controlled ablations demonstrate that BM25 tuning alone improves answer accuracy by 18.0% and surfaced evidence recall by 11.1% over default settings, while increasing retrieval depth further improves surfaced evidence recall by 25.3% over shallow retrieval, establishing that previously reported BM25 underperformance stems primarily from ill-configured parameters and insufficient retrieval depth rather than from inherent lexical retrieval limitations.

2. Context and Motivation

The Core Question: Is Lexical Retrieval Fundamentally Insufficient for Deep Research?

The paper's central motivating question is deceptively direct: Does a lexical retriever suffice as LLMs become more capable in an agentic loop? This question matters because the field has largely accepted a particular answer — that lexical methods like BM25 are insufficient — without rigorously testing whether that answer reflects inherent limitations of lexical retrieval or merely reflects poor configuration and shallow retrieval depth in the baselines that produced it.

The paper frames this as a question that "naturally arises when building deep research systems" (Section 1). Deep research systems — where LLMs iteratively search, inspect documents, and synthesize evidence across multiple sources — are fundamentally gated by their retriever's ability to find relevant documents. The standard industry assumption, implicit in the widespread adoption of dense retrievers and reasoning-aware retrieval models, is that lexical methods cannot bridge the semantic gap between complex natural language queries and the documents that contain the needed evidence.

This paper challenges that assumption directly, not by proposing a new retrieval model, but by asking whether the evidence for BM25's inadequacy is actually evidence of configuration failure rather than methodological failure.

Why This Question Is Important

The retriever choice has cascading practical consequences that the paper identifies across three dimensions:

Efficiency and Cost. Dense retrievers require embedding models that add latency, compute cost, and infrastructure complexity. BM25, by contrast, is computationally cheap, requires no model serving infrastructure, and can be deployed with mature, well-optimized search engines like Lucene. If BM25 can achieve competitive retrieval effectiveness, the cost savings are substantial — the paper reports 3.3×–10× reductions in evaluation cost compared to dense-retriever baselines (Table 1). For organizations running deep research systems at scale, this translates directly to reduced inference costs and simpler deployment architectures.

Scientific Clarity About System Bottlenecks. The paper draws an explicit parallel to an earlier moment in information retrieval research, citing Yang et al. (2019) and their observation that the field risked "overemphasizing retriever improvement while overlooking other opportunities." The concern is that if BM25's apparent weaknesses stem from configuration choices (shallow retrieval depth, default parameters tuned for short passages rather than long documents) rather than from lexical matching itself, then the research community is misallocating effort — investing heavily in more sophisticated retrievers while neglecting the agent–retriever interaction as the actual bottleneck. The paper's trajectory logging, which distinguishes surfaced documents from previewed documents from opened documents from cited documents, is designed to diagnose exactly this: whether the retriever fails to find evidence, or whether the agent fails to use evidence it has already been given.

The Changing Nature of LLM Capabilities. The paper's research question is explicitly contingent on LLM capabilities: "as LLMs become more capable in an agentic loop." This is not a timeless question about BM25 vs. dense retrievers. It is a question about whether more capable frontier models — with better reasoning, better tool use, and better ability to formulate and refine search queries — can compensate for the semantic matching limitations of lexical retrieval through iterative query reformulation and more sophisticated evidence inspection strategies. If they can, then the practical value proposition shifts: invest in better agent design and better LLMs, rather than in more expensive retrieval infrastructure.

Conflicting Prior Evidence and Where Existing Approaches Fall Short

The paper identifies several specific shortcomings in prior work that created the conditions for this question to be asked:

Released BM25 Baselines Understate Lexical Retrieval Capacity. On BrowseComp-Plus, released baselines from Chen et al. (2025) pair BM25 with o3 and gpt-5 at answer accuracies of 50.8% and 58.3% respectively (Table 1, rows a and c). These numbers are substantially below the dense-retriever baselines (qwen3-embed-8b) at 66.3% and 73.0% (rows b and d). The natural interpretation — and the one the field largely adopted — is that BM25 is simply worse. But the paper demonstrates that these baselines used shallow retrieval depth (top-5 results returned directly to the agent) and default BM25 parameters (k1 = 0.9, b = 0.4 in ANSERINI) that are tuned for passage retrieval on shorter documents, not for the long-document evidence search characteristic of BrowseComp-Plus. When PI-SERINI reconfigures BM25 with k1 = 25, b = 1 and retrieval depth 1000, the same gpt-5 + BM25 pairing jumps to 74.6% accuracy — exceeding the dense-retriever baseline. This is not a new retrieval model; it is the same BM25, configured appropriately for the task.

Dense Retrievers and Reasoning-Aware Retrievers Do Not Guarantee Better End-to-End Performance. The paper points to systems like Meng et al. (2026), which pairs gpt-5.2 with qwen3-embed-8b (a strong dense retriever) and achieves only 45.1% accuracy (Table 1, row e), compared to PI-SERINI with the same LLM and BM25 at 70.5% (row j). Similarly, AgentIR-4B (Chen et al., 2026), a reasoning-intensive retriever specifically trained for deep research, achieves 68.1% accuracy (row f), lower than several PI-SERINI BM25 configurations. These results do not prove that BM25 is better than dense retrievers in any absolute sense, but they demonstrate that retriever choice alone does not determine end-to-end system quality. The gap between Meng et al. (2026)'s 45.1% and the paper's 70.5% using the same LLM (gpt-5.2) suggests that agent design — how the agent searches, manages context, and interacts with retrieved documents — is at least as important as retriever quality.

Prior Studies Confound Retriever Quality with Configuration and Interaction Design. The paper argues that existing comparisons between lexical and dense retrievers in search agents conflate three separate factors:

1. Retriever configuration (BM25 parameters k1 and b): Default settings optimized for short-passage retrieval will underperform on long-document corpora where term frequency saturation and document length normalization behave differently. The grid search in Figure 3 shows that ANSERINI's default (k1 = 0.9, b = 0.4) falls in a low-performing region, while tuned settings (k1 = 16–25, b = 1.0) substantially improve evidence recall.

2. Retrieval depth (how many documents are returned to the agent): Shallow retrieval (top-5 or top-10) may discard relevant documents before the agent ever sees them, even if BM25 correctly ranks them highly. The ablation in Figure 2 shows that increasing retrieval depth from 5 to 1000 improves surfaced recall from 70.5% to 95.8%, with most of the gains concentrated below k = 100.

3. Agent–retriever interaction design (what tools the agent has for inspecting results): Baseline agents receive a single list of top-k results and must work with whatever fits in that view. PI-SERINI separates retrieval, result browsing, and document reading into distinct tool calls, enabling the agent to cache deep rankings, paginate through results, and selectively read only promising documents. This separation prevents retrieval depth from overwhelming the context window.

Because prior work did not isolate these factors, the observed performance gap between BM25 and dense retrievers could not be attributed to lexical matching limitations versus poor configuration choices. The paper's contribution is precisely this isolation.

How This Paper Positions Itself

The paper positions itself as a re-assessment study, not a new method proposal. Its posture is diagnostic rather than constructivist: separate the factors that influence end-to-end performance, measure their individual contributions, and determine whether the field's prior conclusions about BM25 were justified.

This is explicitly connected to the tradition of "critically examining neural hype" in information retrieval (Yang et al., 2019), where seemingly strong neural gains were shown to evaporate when properly compared against well-tuned lexical baselines. The paper's parallel is direct: just as Yang et al. (2019) showed that weak BM25 baselines created an illusion of neural model superiority, this paper shows that poorly configured BM25 baselines created an illusion of lexical retrieval inadequacy for deep research.

The paper's positioning is also agentic-loop-aware — it does not evaluate BM25 as a standalone retriever but as a component within a ReAct-style agent that can issue multiple queries, refine them based on observations, and selectively inspect results. This matters because BM25's well-known semantic matching weakness (it cannot match "dangerous" to "risky" or "eclectic" to "diverse") may be compensated by an agent's ability to try multiple query formulations until it finds one whose lexical terms match the target documents. The paper is testing whether this compensation is sufficient, rather than claiming BM25 has no limitations.

Finally, the paper introduces PI-SERINI not as the main contribution but as a necessary tool for the assessment. The agent's three-tool interface (search, read_search_results, read_document) and its trajectory logging across four document sets (surfaced, previewed, opened, cited) are designed to make the retriever–agent interaction transparent. Without this instrumentation, one cannot distinguish "the retriever didn't find the document" from "the retriever found it but the agent didn't inspect it" from "the agent inspected it but failed to use it in the answer." The paper positions PI-SERINI as enabling exactly this diagnostic decomposition, which prior baselines could not support.

The paper's conclusion is deliberately nuanced: "The lesson is not that BM25 is enough everywhere, but that under-configured baselines can hide weak agent–retriever interaction and miss the opportunities in designing more effective and cost-efficient search agents" (Section 8). This is a call for better experimental hygiene in deep research evaluation — properly configuring baselines, controlling for retrieval depth, and measuring not just final answer quality but the intermediate evidence inspection behavior that determines whether stronger retrievers are actually the limiting factor.

3. Technical Approach

3.1 Reader Orientation

PI-SERINI is a deliberately minimal search agent — essentially an LLM wrapped in a ReAct loop with a constrained tool API — that performs deep research by iteratively searching a document corpus, browsing cached search results, and selectively reading promising documents. The system solves the problem of isolating whether BM25 lexical retrieval is genuinely insufficient for deep research, or whether prior negative results stem from poor retriever configuration and shallow retrieval depth, by separating retrieval from result inspection so that the agent can cache deep rankings (up to 1000 documents) while only loading into its context window the specific evidence it chooses to inspect.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components connected in a loop:

1. LLM Agent — a frontier language model (e.g., gpt-5.5, claude-opus-4.7) that runs a ReAct loop: it reasons about the question, decides which tool to call, observes the tool output, and iterates until it produces a final answer. It operates under a system prompt that defines a multi-step retrieval workflow and a required response format.

2. Retrieval Controller — the central isolation layer that mediates all communication between the LLM agent and the search backend. It exposes exactly three tools (search, read_search_results, read_document), maintains session-local search caches keyed by search_id, implements rank-based and line-based pagination, and enforces tool blocking when the time budget is nearly exhausted. This controller is where retrieval depth (1000 hits per query) is handled without overwhelming the agent's context window.

3. Search Backend — ANSERINI BM25 running over the BrowseComp-Plus corpus (100,195 documents), configured with tuned parameters k1=25, b=1, returning up to 1000 ranked documents per query. The backend is stateless from the agent's perspective; all state management (cached rankings, pagination cursors) lives in the retrieval controller.

4. Time-Budget Steering Module — a timer-based policy that triggers at 70% of the per-query timeout (0.7 × 300 = 210 seconds), injects a "submit now" instruction into the agent's context, and blocks further tool calls to force completion before the hard 300-second deadline.

5. Trajectory Logger — records four distinct document sets per query: D_surfaced (all documents returned by search calls), D_previewed (documents whose excerpts were shown via read_search_results), D_opened (documents fully read via read_document), and D_cited (documents cited in the final answer). This instrumentation enables the diagnostic decomposition that is central to the paper's research question.

Information flows cyclically: the agent receives the question → reasons → calls search with a query string → the retrieval controller queries ANSERINI, caches the top-1000 ranking, and returns only ranks 1–5 → the agent may call read_search_results to browse deeper into the cached ranking (ranks 6–15, 16–25, etc.) → the agent may call read_document to read specific documents by docid in 200-line chunks → the agent reasons about the observed evidence → loops back to issue refined search queries or proceeds to synthesize a final answer → at 210 seconds, the time-budget module interrupts with a submission steer → at 300 seconds, the process is forcibly terminated if no answer has been produced.

3.3 Roadmap for the Deep Dive

• First, the formal problem formulation — how the paper casts deep research as iterative retrieval-and-reasoning in a ReAct loop, the notation for interaction trajectories, and what the system outputs (final response plus document evidence sets). This provides the formalism needed to understand what the agent is doing and what is being measured.

• Second, the retrieval controller and its three-tool API — the design rationale for separating retrieval, browsing, and reading into distinct tool calls, the exact arguments and outputs of each tool, how search caches work, and how pagination (rank-based for browsing, line-based for reading) prevents retrieval depth from overwhelming the context window. This is the central architectural contribution and must be understood in detail to see how PI-SERINI differs from baseline agents.

• Third, the BM25 configuration and retrieval depth choices — the specific parameter values (k1=25, b=1) and retrieval depth (1000), why these differ from defaults, and how they were selected via grid search. This is the "well-configured" part of the paper's claim about BM25 sufficiency.

• Fourth, the agent prompt and workflow — the 11-rule system prompt that defines the retrieval strategy (prefer short lexical searches, browse before re-querying, inspect promising candidates, etc.), the required response format (Explanation, Exact Answer, Confidence), and the subtle design choices embedded in these rules.

• Fifth, the time-budget steering mechanism — the two-stage policy (submit steer at 0.7T, hard termination at T), why a wall-clock timeout replaces the fixed iteration cap used in prior work, and how tool blocking is implemented.

• Sixth, the trajectory logging and evaluation infrastructure — the four document sets, the gold-answer LLM judge configuration, and how the paper computes answer accuracy, calibration error, and the three recall metrics (surfaced, previewed, behavior).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and empirical analysis paper whose core idea is that BM25 lexical retrieval can suffice for deep research when properly configured, used with sufficient retrieval depth, and paired with a tool interface that lets the agent manage cached rankings and selectively acquire evidence. The technical contribution is not a new model but a carefully instrumented agent architecture that isolates retriever quality from surrounding design choices.

Formal Problem Formulation

The paper formulates deep research as an iterative retrieval-and-reasoning problem within a ReAct-style agentic loop (Yao et al., 2023). The formulation is:

Given a user query, a search agent — materialized by an LLM — iteratively reasons about the question, calls tools to interact with a document corpus, observes the tool outputs, and eventually produces a final answer. The agent operates over an interaction trajectory defined as:

$H_T = (\tau_1, a_1, o_1, \ldots, \tau_t, a_t, o_t, \ldots, \tau_T, a_T)$

where $\tau_t$ is the reasoning trace generated by the LLM at turn $t$, $a_t$ is the action (tool call) selected by the agent at that turn, and $o_t$ is the observation returned by the environment in response to that action. $T$ is the total number of turns before termination.

What this represents: the complete interaction history of one query run. The agent's policy $\pi$ samples both the reasoning trace and the action conditioned on all prior history: $\tau_t, a_t \sim \pi(\cdot \mid H_{t-1})$. For intermediate turns ($t < T$), the action is a tool call, and the environment responds with an observation. For the final turn $t = T$, the action produces the response and terminates the loop. The paper treats the outputs as both the final response $R$ and the set of documents $D$ observed during retrieval.

Why this formulation matters: it makes explicit that the agent's decisions at each turn depend on everything that happened before — the queries it tried, the results it saw, the documents it read, and its own reasoning about what evidence is still missing. This is the "agentic loop" in the paper's research question. The trajectory is the unit of analysis for understanding why a system succeeded or failed: did the retriever not find the evidence, or did the agent not inspect evidence it had been given?

The paper adds that "the action $a$ can be one of three search-related tool calls" (search, read_search_results, read_document), which is the concrete instantiation of this abstract formulation. This three-tool decomposition is what enables the diagnostic trajectory logging described later.

The Retrieval Controller and Three-Tool API

The retrieval controller is the central architectural contribution of PI-SERINI. It is the "main isolation point" between the LLM agent and the search backend (Appendix A, Figure 4). Its job is to manage the mismatch between what BM25 can provide (deep rankings of 1000 documents) and what the LLM can process (a finite context window). It does this by exposing three tools that make retrieval, result browsing, and document reading distinct, controllable decisions for the agent.

Design rationale for separating tools. Baseline search agents on BrowseComp-Plus use a single retriever tool that directly returns the top-k search results, with k=5, inserting all five document excerpts into the agent's context window on every search call. This conflates retrieval depth with context management: if you want deeper retrieval (k=100 or k=1000), every search call would consume enormous context, making it impractical. By separating search (which caches a deep ranking but only shows the top 5), browsing (which paginates through the cached ranking), and reading (which loads full documents on demand), PI-SERINI lets the agent decide how much context to spend on inspecting search results versus reading documents versus issuing new queries. The retrieval depth of 1000 is always available in the cache, but only the documents the agent explicitly chooses to preview or read consume context.

Tool 1: search

The search tool issues a lexical query to the ANSERINI BM25 backend and caches the returned ranking under a session-local identifier.

Arguments:

• reason: a brief rationale string (at most 100 words) explaining why this query is being issued. The system prompt requires this as the first argument for every tool call, making the agent's search strategy explicitly recorded in the trajectory.
• query: a raw query string. The prompt instructs the agent to prefer "short lexical searches over long natural-language rewrites," which biases the agent toward BM25-friendly keyword queries rather than verbose semantic queries that BM25 would handle poorly.

Runtime behavior:

• The backend is queried with query_mode = plain (not structured Lucene syntax) and a backend request limit = 1000 hits.
• The full 1000-document ranking is cached under a newly generated search_id (e.g., s1, s2, ...), storing the raw query, query mode, and ranked document list.
• The agent immediately sees only the first page: ranks 1–5, formatted as excerpts with docids, along with the search_id, the list of cached docids, the list of displayed docids, and metadata about timing and truncation.
• At most 32 active search_id entries are retained in the session cache; older entries are evicted first.

What the agent receives and does not receive: the agent sees 5 excerpts but knows (via the tool output metadata) that up to 1000 documents are available in the cached ranking. It does not receive the full ranking in its context window. This creates an explicit decision point: does the agent browse deeper into this ranking (via read_search_results), read a specific document (via read_document), or issue a new query (via another search call)?

Why this design: it decouples recall (how many relevant documents are somewhere in the cached ranking) from precision (how many of the top-ranked documents the agent inspects). A shallow-retrieval baseline with k=5 has recall bounded by 5; PI-SERINI's recall is bounded by 1000, but the agent only pays the context cost for the documents it actually inspects. The paper's ablation in Figure 2 shows that surfaced recall improves from 70.5% at k=5 to 95.8% at k=1000, confirming that deep caching substantially increases the probability that relevant evidence is somewhere in the ranking, even if the agent doesn't inspect it all.

Tool 2: read_search_results

The read_search_results tool browses an existing cached ranking without issuing a new backend query. This is the pagination mechanism for search results.

Arguments:

• reason: brief rationale (at most 100 words).
• search_id: the identifier returned by a previous search call.
• offset: optional 1-indexed rank offset; defaults to 6 (the page after the initial top-5 display from search).
• limit: optional number of ranked hits to show; defaults to 10.

Runtime behavior:

• The tool retrieves the cached ranking for the given search_id and returns the formatted page for ranks offset through offset + limit - 1.
• Output includes the formatted page (doc excerpts with docids), the list of displayed docids, a next_offset value when more hits remain, and truncation metadata if the formatted page exceeds PI's output limits.
• No new query is sent to ANSERINI; the ranking was already retrieved and cached by the search call.

What this enables: the agent can explore progressively deeper into a single ranking without re-executing the query. For example, after a search returns ranks 1–5, the agent can call read_search_results with offset=6 to see ranks 6–15, then offset=16 for 16–25, and so on. This is a browsing metaphor: the agent "scrolls" through search results, deciding at each page whether any document looks promising enough to read in full.

Why this matters for the research question: one hypothesis for why shallow retrieval baselines underperform is that relevant documents often rank below position 5 — they are in BM25's ranking but outside the agent's viewport. read_search_results tests this hypothesis directly: by measuring previewed recall (what fraction of relevant documents the agent actually browses via this tool), the paper can distinguish "BM25 didn't rank the document highly" from "the agent chose not to browse deep enough to find it." The Figure 2 ablation shows that previewed recall saturates around k=50 at 74.7%, while surfaced recall continues climbing to 95.8% at k=1000 — meaning that deeper rankings contain more relevant documents, but the agent does not (or cannot) browse deeply enough to inspect them all.

Tool 3: read_document

The read_document tool fetches the full text of one document by its docid, using line-based pagination to handle the long documents characteristic of BrowseComp-Plus.

Arguments:

• reason: brief rationale (at most 100 words).
• docid: the document identifier obtained from search or read_search_results output.
• offset: optional 1-indexed line offset; defaults to 1 (start from the beginning).
• limit: optional maximum number of lines; defaults to 200.

Runtime behavior:

• If the document is found in the backend, the tool returns the formatted document lines for the range offset through offset + limit - 1, along with the returned line range, total line count, a backend truncation flag, next_offset when more lines are available, timing metadata, and output-truncation metadata when applicable.
• The agent can continue reading the same document by issuing subsequent read_document calls with offset set to the next_offset from the previous call, effectively streaming the document in 200-line chunks.

Why line-based pagination: BrowseComp-Plus documents are long — the median document has approximately 2,000 tokens and the 90th percentile has approximately 14,000 tokens (cited from Chen et al., 2025, Figure 4). Loading an entire document into context would be expensive and likely exceed context limits for multiple documents. Line-based pagination lets the agent read just enough to determine relevance before committing more context, or read only the specific sections containing the evidence it needs. The system prompt explicitly instructs: "When reading a document, start with offset=1 and a moderate limit. If it is truncated and still relevant, continue reading the same document."

Why this matters for the research question: read_document is where the agent actually acquires the evidence it will use in its final answer. The gap between previewed documents (those whose excerpts the agent saw via read_search_results) and opened documents (those the agent chose to read in full) measures how well the agent identifies promising candidates from excerpts alone. The gap between opened documents and cited documents measures how well the agent extracts and synthesizes evidence from documents it actually read. The paper's trajectory logging across these four tiers (surfaced, previewed, opened, cited) enables decomposition of search agent failures into retrieval failures vs. inspection failures vs. synthesis failures.

Spill-File Behavior

The paper documents a practical detail about handling tool output that exceeds the agent harness's display limits (Appendix A.3). When a search page or document chunk, after formatting, exceeds PI's line or byte truncation limits, the visible output in the agent's context is truncated, and the complete output is saved to a temporary "spill file" under a pi-search spill directory. This prevents information loss while respecting truncation limits. The spill directory is cleaned up on session shutdown or process exit. The paper does not report how often spill files are used or whether agents are aware of them (the prompt does not mention spill files, suggesting agents operate only on the truncated visible output).

BM25 Configuration and Retrieval Depth

The paper's central empirical claim — that BM25 can suffice when "well-configured" and used at "sufficient retrieval depth" — depends on specific parameter choices that differ substantially from defaults. These choices were not arbitrary; they were selected through a grid search (Section 6, Figure 3) and then held fixed across all PI-SERINI runs.

BM25 parameters. The paper uses k1 = 25 and b = 1, which are dramatically different from ANSERINI's defaults (k1 = 0.9, b = 0.4). The BM25 scoring function itself is not formally presented in the paper, but its two key parameters control:

• $k_1$ — the term frequency saturation parameter. Higher values mean the score continues to increase with additional term occurrences rather than saturating early. The default k1 = 0.9 is appropriate for short passages where a term appearing 3 times vs. 5 times conveys little additional relevance signal. The tuned k1 = 25 is appropriate for long documents (BrowseComp-Plus documents average ~5,179 words) where a term appearing 50 times vs. 10 times is a strong signal that the document is substantially about that concept. With k1 = 0.9, the term frequency component $\frac{tf}{k_1 + tf}$ saturates quickly; with k1 = 25, it grows more slowly and continues discriminating between documents with different levels of topical focus.

• $b$ — the document length normalization parameter, ranging from 0 to 1. b = 0 means no length normalization (long documents are not penalized for their length). b = 1 means full length normalization relative to the average document length. The default b = 0.4 means moderate normalization — long documents are somewhat penalized, which makes sense when retrieving short passages where verbosity dilutes relevance. The tuned b = 1.0 means full normalization, which may seem counterintuitive — why penalize long documents? The paper's grid search (Figure 3) shows that b = 1.0 is optimal, but does not explain the mechanism. A plausible interpretation: in a corpus where documents vary dramatically in length (from very short to ~14k tokens at the 90th percentile), full normalization prevents extremely long documents from dominating the ranking simply because they contain many occurrences of common query terms by chance. With full normalization, a long document must have a higher density of query term occurrences to match a shorter document's score, effectively normalizing for length-induced term frequency inflation.

Grid search procedure. The paper samples 100 queries from BrowseComp-Plus and performs a grid search over k1 and b, measuring surfaced evidence document recall as the objective. Figure 3 visualizes the results as a heatmap. The key finding is that ANSERINI's default setting (k1 = 0.9, b = 0.4) falls in a low-performing region, marked with an ×. The best-performing configuration is near k1 = 16 and b = 1.0 (marked with a ⋆). The paper notes that the tuned setting used in all experiments (k1 = 25, b = 1) and the best grid-search setting (k1 = 16, b = 1) produce similar downstream results (Table 3: 82.0% vs. 81.1% accuracy on the 100-query subset), suggesting the optimum is a plateau rather than a sharp peak.

Retrieval depth. The search tool requests up to 1000 documents from ANSERINI for every query. This depth is chosen to maximize surface recall — the paper's Figure 2 shows surfaced recall increasing from 70.5% at k = 5 to 95.8% at k = 1000, with most gains occurring below k = 100 (86.2% at k = 100). The practical cost of this depth is near-zero for BM25 (which can score and sort 100,000 documents in milliseconds using inverted indices) but would be substantial for dense retrievers (which require computing embeddings and performing nearest-neighbor search over the full corpus for each query).

Why 1000 and not more: the paper does not explicitly justify the ceiling of 1000, but it is likely determined by a combination of (1) diminishing returns (the Figure 2 curve flattens substantially after k = 100), (2) cache memory constraints (32 cached rankings × 1000 docids each = manageable memory footprint), and (3) practicality (beyond 1000, the agent would need to browse through an impractical number of pages to inspect all results).

Why this parameter configuration matters for the paper's claim: the central argument is that previously released BM25 baselines underperformed because they used default parameters tuned for passage retrieval and shallow retrieval depth (k = 5). The paper's contribution is not discovering that these parameters matter (this is well-known in IR) but rather demonstrating that the field's conclusions about BM25's inadequacy for deep research were based on experiments that used the wrong parameters for the task. If researchers had used k1 = 25, b = 1, and k = 1000 in their baselines, the apparent performance gap relative to dense retrievers would have been much smaller or nonexistent.

Agent Prompt and Retrieval Workflow

The system prompt (Appendix A.1) is an 11-rule instruction set that defines the agent's retrieval strategy, tool-use behavior, and response format. It is not a prompt template in the traditional sense (it does not contain few-shot examples); it is a detailed workflow specification.

Rule 1: Start with a concise raw query based on the original question. This biases the initial search toward the literal terms in the question, which is appropriate for BM25 since it matches on exact lexical overlap. It discourages the agent from paraphrasing or expanding the query before seeing what the literal terms retrieve.

Rule 2: Prefer short lexical searches over long natural-language rewrites. This is the most directly BM25-aware instruction in the prompt. BM25 performs poorly on long natural language queries because it treats every term as a required or weighted match — verbose queries introduce noise terms that dilute the signal from key content terms. By instructing the agent to keep queries short and keyword-like, the prompt aligns the agent's behavior with BM25's strengths.

Rule 3: Browse the current ranking with read_search_results before repeatedly rewriting the query. This prevents a common failure mode where agents issue many query reformulations without fully inspecting any single ranking. The instruction encodes a search strategy: go deep before going wide. If the cached ranking already contains relevant documents but they are below position 5, browsing is more efficient than issuing a new query that may or may not surface the same documents.

Rule 4: If a promising candidate document appears in the ranking, inspect it with read_document. This operationalizes the "selective evidence acquisition" design principle: the agent should not read every document; it should make a judgment from the excerpt and only invest context in reading documents that appear relevant.

Rule 5: When reading a document, start with offset=1 and a moderate limit, then continue if truncated and still relevant. This encodes the line-based pagination strategy: incremental reading prevents wasting context on irrelevant documents.

Rule 6: Use search refinements only when they add a genuinely new clue from what you already saw. This further discourages redundant querying and encourages the agent to extract actionable clues from observed evidence that can narrow or redirect the search.

Rule 7: Every tool call must include reason as the first argument, kept specific, under 100 words, and focused on the clue, gap, candidate, or ranking issue. This is the mechanism that makes the agent's search strategy observable in the trajectory logs. The reason strings (visible in the Appendix C example trajectories) document the agent's intent at each step: what gap it is trying to fill, what candidate it is investigating, or why it is reformulating.

Rule 8: As soon as you have enough evidence, stop using tools and answer in plain assistant text. This encourages early stopping rather than exhaustive search, which is important for cost efficiency. Combined with the time-budget steering, it creates a "satisficing" rather than "optimizing" agent behavior.

Rule 9: Final response must use exactly this format. The required format is:

Explanation: {explanation with cited docids in square brackets, e.g., [123]}
Exact Answer: {succinct final answer}
Confidence: {confidence score between 0% and 100%}

This structured output enables automated evaluation: the LLM judge (Section 4.1, Appendix A.5) extracts the Exact Answer for comparison with ground truth, and the Confidence field enables calibration error computation. The instruction to cite docids inline ([123]) creates the D_cited document set used in behavior recall computation.

Rule 10: If you receive a user steer telling you to submit now, stop using tools immediately and answer right away. This is the agent-side behavior for the time-budget steering mechanism. It instructs the agent to treat the submission steer as an override command, not a suggestion.

Rule 11: Keep Exact Answer concise and directly responsive to the question. This prevents verbose answers that might confuse the LLM judge or include extraneous information.

Why this prompt design matters: it encodes assumptions about BM25's strengths and weaknesses into the agent's default behavior. The preference for short lexical queries, the bias toward browsing before re-querying, and the incremental document reading strategy are all adaptations to BM25's characteristics. A different retriever (e.g., a dense retriever good at handling long natural-language queries) might benefit from a different prompt. The paper does not ablate the prompt design, so the extent to which PI-SERINI's performance depends on these specific instructions versus the tool interface itself is unknown — but the prompt is part of the "well-configured" claim, not an incidental detail.

Time-Budget Steering

The paper uses a wall-clock time budget rather than a fixed iteration cap to control agent execution. This design choice is motivated by practical relevance: "deep research is inherently time-sensitive, requiring systems to balance answer quality against realistic latency and cost constraints" (Section 1).

Two-stage policy. The time-budget mechanism has two thresholds:

1. Submit-now trigger at 0.7T. For the main experiments, the per-query timeout $T = 300$ seconds. At $\text{floor}(0.7 \times 300 \times 1000) = 210,000$ milliseconds (210 seconds), if the agent is still active, the system injects a submission steer into the agent's context. This steer is a user message saying that the time budget is nearly exhausted and instructing the agent to stop using tools and produce its best answer immediately from the evidence collected so far. From that point onward, all three tools (search, read_search_results, read_document) are blocked — any attempt to call them returns an error. The agent is expected to use the remaining 90 seconds for reasoning and answer synthesis.

2. Hard termination at T. If the agent still has not produced a final answer by 300 seconds, the process is forcibly terminated and the query is marked as timed out.

Why time-budget over iteration cap. Prior work (Chen et al., 2025; Meng et al., 2026) uses a fixed maximum number of iterations (e.g., 100 tool calls), which does not account for variability in per-iteration latency. A query that requires reading long documents (which take longer to process) may time out in wall-clock time even if it stays within the iteration budget, while a query with many fast searches may complete quickly but hit the iteration cap. Time-budget steering aligns the termination condition with the practical constraint: how long a user is willing to wait. It also makes cost more predictable, since the wall-clock time bounds the total tokens consumed (the agent cannot run indefinitely).

Why 300 seconds. The paper does not explicitly justify the 300-second timeout, but it is presumably chosen as a practical balance: long enough for the agent to conduct a multi-step search with several document reads, short enough to keep evaluation tractable (830 queries × 300 seconds ≈ 69 hours of wall-clock time if run sequentially, though parallel execution reduces this).

Why the 70% threshold. The paper does not discuss alternative thresholds, but the design choice reflects a tension: an earlier steer (e.g., at 50% of the budget) might prematurely terminate useful search, while a later steer (e.g., at 90%) might not leave enough time for the agent to synthesize an answer from complex evidence. The 70% threshold reserves 30% of the total budget for answer synthesis after the last tool call, which is approximately 90 seconds in the 300-second setting.

Interaction with answer format. The final response format (Explanation, Exact Answer, Confidence) is still required after the submission steer. This means the agent must not only produce an answer but also explain it and estimate its confidence — a non-trivial reasoning task — within the remaining time budget. The paper does not report how often agents fail to produce a properly formatted response after the submission steer, or whether answer quality degrades when produced under time pressure. This is a potential confound: the time-budget mechanism may prioritize faster-but-worse answers from agents that could have produced better answers with more time.

Trajectory Logging and Evaluation Infrastructure

PI-SERINI's trajectory logging is specifically designed to enable the diagnostic decomposition that is central to the paper's research question: distinguishing retrieval failures (the retriever didn't find the evidence) from inspection failures (the agent didn't look at the evidence it had) from synthesis failures (the agent looked at the evidence but didn't use it).

Four document sets. The paper defines four disjoint (conceptually; they may overlap in practice) document sets:

1. $D_{\text{surfaced}}$ — all documents returned by any search call during the trajectory. This is the union of all cached rankings across all queries. It represents what the system made available to the agent. Surfaced recall measures what fraction of the ground-truth evidence documents (or gold documents) appear somewhere in this set.

2. $D_{\text{previewed}}$ — the subset of surfaced documents whose excerpts were shown to the agent through read_search_results calls. This represents what the agent chose to inspect at the excerpt level. Previewed recall measures what fraction of ground-truth evidence the agent actually saw, even briefly, in search result pages.

3. $D_{\text{opened}}$ — the subset of surfaced documents that the agent read via read_document calls. This represents what the agent invested significant context in examining. Opened recall (not separately reported in the main results) would measure what fraction of evidence the agent read in full.

4. $D_{\text{cited}}$ — the subset of surfaced documents that the agent cited in its final answer (via the [docid] notation in the Explanation field). This represents what evidence the agent actually used to construct its answer.

Behavior recall. The paper reports "Behavior Recall" in Table 1 as recall over the union $D_{\text{opened}} \cup D_{\text{cited}}$. This is the fraction of ground-truth evidence documents that the agent either read in full or cited — a measure of how effectively the agent translated surfaced documents into actually-used evidence. The gap between surfaced recall and behavior recall measures the inspection and synthesis losses.

Why four sets. The diagnostic power comes from ratios between these sets:

• If surfaced recall is high but previewed recall is low, the retriever is finding evidence but the agent isn't browsing deeply enough into search results. This would suggest that read_search_results pagination is insufficient or that the agent's browsing strategy is suboptimal.

• If previewed recall is high but behavior recall is low, the agent is seeing evidence in excerpts but failing to (a) recognize it as relevant and open it, or (b) extract and cite it in the answer. This would suggest that the agent's evidence evaluation or synthesis is the bottleneck.

• If surfaced recall is low, the retriever genuinely cannot find the evidence, and no amount of agent improvement will help.

Gold-answer LLM judge. Answer correctness is evaluated by an automated LLM judge (Appendix A.5). The judge receives the question, the agent's full final response, and the benchmark-provided correct answer. It runs with gpt-5.3-codex in JSON mode with thinking = low. Each judge call has a 180-second timeout and runs in an isolated PI environment.

The judge is instructed to determine whether the extracted final answer is "semantically equivalent to the known correct answer," allowing "harmless wording differences, equivalent formatting, and added correct detail." For numerical answers, it allows "small formatting differences and obvious equivalent forms." If the response does not contain an extractable final answer, the judge sets extracted_final_answer to null and correct to false.

The judge outputs a JSON object with the schema:

{
  "extracted_final_answer": string | null,
  "correct_answer": string,
  "reasoning": string,
  "correct": boolean,
  "confidence": number
}

The correct field is the binary accuracy metric reported in Table 1. Calibration error (also reported in Table 1) is the discrepancy between the model's self-reported confidence (the Confidence field in the agent's response) and empirical correctness — the paper reports the absolute value without specifying the exact computation (e.g., expected calibration error vs. maximum calibration error).

Metrics computed from the trajectory. The paper reports:

• Accuracy: fraction of queries where the judge returns correct = true.
• Calibration error: discrepancy between agent-reported confidence and empirical correctness (computation method not specified).
• Surfaced recall (evidence and gold): fraction of ground-truth evidence/gold documents that appear in $D_{\text{surfaced}}$.
• Previewed recall (evidence and gold): fraction of ground-truth evidence/gold documents that appear in $D_{\text{previewed}}$.
• Behavior recall (evidence and gold): fraction of ground-truth evidence/gold documents that appear in $D_{\text{opened}} \cup D_{\text{cited}}$.
• Cost: total USD cost computed from token pricing (Appendix B) including input tokens, output tokens, and cached read tokens.

Token pricing and cost computation. The paper provides exact per-model token pricing in Appendix B (Table 4). Costs are computed from the actual token counts logged during each run, accounting for the three-tier pricing structure (input, output, cache read) that different providers use. For example, gpt-5.5 costs $5.00/1M input tokens,$30.00/1M output tokens, and $0.50/1M cache read tokens. The cache read tier is substantially cheaper (10× less than input for gpt-5.5), which makes PI-SERINI's prefix-cache-friendly agentic loop a major cost savings mechanism.

Baseline Agent Differences

The paper compares PI-SERINI against released baselines from Chen et al. (2025), Meng et al. (2026), and Chen et al. (2026). The key architectural differences that matter for interpreting the results are:

Single-tool vs. multi-tool interface. Baseline agents use a single retriever tool that directly returns the top-k search results (k = 5) as part of the tool output. They do not have separate browsing or reading tools. Under the paper's logging protocol, only $D_{\text{surfaced}}$ is defined for these baselines, since they do not use additional tools beyond search. This means surfaced recall for baselines is bounded by 5 (assuming one query) or by the total number of unique documents returned across all search calls (since each call returns 5 new documents). PI-SERINI's surfaced recall is bounded by approximately 32 × 1000 = 32,000 (32 cached rankings, each with 1000 documents), though in practice far fewer unique documents are surfaced because queries overlap.

Fixed iteration cap vs. time-budget steering. Baseline agents use a maximum iteration cap (100 iterations for Chen et al., 2025; not specified for Meng et al., 2026, but Table 2 shows an average of 73.8 tool calls per query, suggesting a high cap). PI-SERINI uses the 300-second timeout with the two-stage steering policy. This difference is not directly controlled for in the paper's experiments — the Table 3 ablation tests three termination conditions (timeout 300s, max 100 iterations, timeout 3600s) and finds that the timeout 300s setting achieves both lower cost and comparable or better accuracy than the max-iteration setting, suggesting the comparison is not unfair to PI-SERINI.

Retrieval depth. Baseline agents use k = 5; PI-SERINI uses k = 1000 with pagination. The paper's Figure 2 shows this is the single largest factor in surfaced recall improvement, and the paper's core argument is that this difference — not retriever type — explains most of the performance gap.

BM25 configuration. The paper does not explicitly state what BM25 parameters the Chen et al. (2025) baselines use, but since they use default ANSERINI (which has k1 = 0.9, b = 0.4 as defaults), the assumption is that these baselines used default parameters. The paper's Table 3 shows that tuning BM25 parameters from default to k1 = 25, b = 1 improves accuracy by 18.0 percentage points and surfaced recall by 11.1 percentage points on the 100-query subset, making this a substantial confound in the baseline comparison.

Summary of Design Choices and Their Justifications

• Three-tool API over single-tool: separates retrieval depth from context management, enabling deep caching while giving the agent explicit control over which evidence consumes context window space. Prior single-tool baselines conflated these decisions.

• BM25 parameters k1 = 25, b = 1 over defaults k1 = 0.9, b = 0.4: adapted for long-document retrieval where term frequency continues to provide signal at higher counts and full length normalization prevents very long documents from dominating rankings by chance. Selected via grid search on 100 queries.

• Retrieval depth 1000 over 5: maximizes the probability that relevant documents are somewhere in the cached ranking, even if they rank below the top-5 viewport. The cost of deep BM25 retrieval is near-zero; the cost of inspecting deep results is borne by the agent's context window, which the agent controls via browsing and reading decisions.

• Time-budget steering over fixed iteration cap: aligns termination with practical latency constraints rather than an arbitrary iteration count. The 70% threshold reserves 30% of the budget for answer synthesis after tool calls are blocked.

• Four-tier trajectory logging over single document set: enables decomposition of system failures into retrieval, inspection, and synthesis losses. Without this instrumentation, high surfaced recall with low answer accuracy would be uninterpretable.

• Structured response format with inline citations: enables automated evaluation (extracting answers, computing calibration) while also forcing the agent to document which evidence it used, creating the D_cited set for behavior recall computation.

4. Key Insights and Innovations

Innovation 1: Reframing the Lexical Retriever Question as One of Configuration, Not Capability

The paper's most distinctive intellectual move is not methodological but diagnostic: it reframes the field's central assumption about BM25 — that it is fundamentally insufficient for deep research — as an untested configuration hypothesis rather than an established fact. The paper does not claim BM25 is universally adequate or superior to dense retrievers. Instead, it identifies a specific experimental confound: prior comparisons between lexical and dense retrievers in search agents simultaneously varied three factors — retriever type (BM25 vs. neural), retriever configuration (default vs. tuned parameters), and retrieval depth (shallow top-5 vs. deep caching) — while attributing the entire performance gap to retriever type alone.

This reframing is significant because it changes where the research community looks for solutions. Before this paper, the dominant narrative — reinforced by released BrowseComp-Plus baselines showing BM25 at 50.8%–58.3% accuracy versus dense retrievers at 66.3%–73.0% (Chen et al., 2025, Table 1 rows a–d) — was that lexical matching cannot capture the semantic complexity of deep research queries, and that progress requires increasingly sophisticated neural retrieval models (reasoning-aware retrievers, instruction-tuned embeddings, multi-vector representations). The paper does not disprove this narrative outright, but it demonstrates that the evidence supporting it was contaminated: the same gpt-5 + BM25 pairing that scored 58.3% in the released baseline reaches 74.6% in PI-SERINI (Table 1, row i), exceeding the dense-retriever baseline of 73.0% (row d), purely through parameter tuning and increased retrieval depth — not through a new retrieval model.

The conceptual contribution is a confound decomposition applied to search agent evaluation. Prior work treated "BM25" as a fixed point — the default implementation in whatever search library was used. This paper shows that BM25 is a family of retrieval functions parameterized by k1 and b, and that the member of this family appropriate for long-document evidence search (k1 = 25, b = 1, tuned for the corpus) is qualitatively different from the member appropriate for passage retrieval (k1 = 0.9, b = 0.4, the ANSERINI default that prior baselines inherited). The 18.0 percentage point accuracy improvement from tuning alone (Table 3, 64.0% → 82.0% on the 100-query subset) is larger than many reported gains from switching to dense retrievers in the literature, yet it requires no model serving, no embedding computation, and no training data.

This is not an incremental refinement — it is a fundamental corrective. It echoes the pattern established by Yang et al. (2019), who showed that weak lexical baselines created an illusion of neural model superiority in ad-hoc retrieval. The paper extends this critique to the agentic deep research setting, where the stakes are higher because retriever choice cascades into agent design decisions (whether to support deep ranking inspection, how to structure tool interfaces, what prompt strategies to use). The lesson is not that BM25 is "good enough" in any absolute sense, but that evaluating retrievers for deep research requires controlling for configuration and retrieval depth — factors that prior work treated as incidental rather than central.

Innovation 2: Separating Retrieval Depth from Context Management via Cached Ranking with Selective Inspection

Prior search agents for deep research — including the BrowseComp-Plus baselines from Chen et al. (2025) and Meng et al. (2026), and the coding-agent approaches from Cao et al. (2026) and Li et al. (2026) — operate under an implicit model where retrieval depth and context consumption are tightly coupled. When an agent calls a search tool, the tool returns a fixed number of results (typically 5), and those results are inserted directly into the agent's context window. This design enforces a hard tradeoff: increase retrieval depth and you proportionally increase context consumption, rapidly hitting the LLM's context window limits. The practical consequence is that agents are forced into shallow retrieval — they see only the top handful of documents, and any relevant document that ranks below the viewport is invisible regardless of how the retriever scored it.

PI-SERINI's retrieval controller introduces a different model: retrieval depth is decoupled from context consumption through a session-local cache with paginated browsing. The search tool caches up to 1000 ranked documents but initially exposes only the top 5 excerpts. The agent can then page through the cached ranking using read_search_results, and selectively open specific documents using read_document. The context cost scales with what the agent chooses to inspect, not with what the retriever makes available.

This is a conceptual shift, not merely an engineering convenience. It transforms retrieval from a one-shot, capacity-limited operation into a browsable resource. The agent's interaction with the document corpus becomes analogous to a human researcher using a search engine: the search results page shows snippets; the researcher decides which links to click; the researcher can scroll deeper into results if the first page is unproductive. Prior search agents lacked this browsing capability — they were essentially forced to work with whatever fit in the first page of results.

The empirical significance is substantial: increasing retrieval depth from 5 to 1000 improves surfaced evidence recall from 70.5% to 95.8% (Figure 2), a 25.3 percentage point gain that is impossible in a shallow-retrieval architecture regardless of how good the retriever is. More importantly, the gap between surfaced recall (95.8%) and previewed recall (74.7%) reveals that the bottleneck is not retrieval but agent attention: the agent does not inspect most of the documents it has available. This diagnostic insight — distinguishing "the retriever didn't find it" from "the agent didn't look at it" — is only possible because the architecture separates these two stages. Prior systems could not make this distinction because their retrieval depth was too shallow for the question to arise.

This innovation is a system architecture contribution rather than an algorithmic one. It does not propose a new retrieval scoring function or a new ranking objective. Instead, it changes the interface between the agent and the retriever in a way that makes deep retrieval tractable and makes agent inspection behavior observable. The three-tool API (search, read_search_results, read_document) is deliberately minimal — it does not include query expansion, reranking, or any retrieval intelligence beyond what BM25 provides — precisely because the paper's goal is to isolate retriever quality from interface design. The fact that this minimal interface, combined with deep retrieval, enables BM25 to match dense-retriever performance suggests that interface design is an underappreciated lever in search agent development, potentially as important as retriever model quality.

Innovation 3: Four-Tier Trajectory Logging as a Diagnostic Decomposition of Search Agent Failures

The paper introduces a structured taxonomy of document access levels — surfaced, previewed, opened, and cited — that decomposes the end-to-end search agent pipeline into distinct, measurable stages. This is not just a logging convention; it is a diagnostic framework that answers a question that prior work could not address: when a search agent produces a wrong answer, is it because the retriever failed to find relevant documents, the agent failed to inspect documents it found, or the agent failed to use documents it inspected?

Prior evaluations of deep research systems report end-to-end answer accuracy and, in some cases, retrieval recall over whatever document set the agent's tools returned (analogous to D_surfaced). But they cannot distinguish retrieval failures from agent failures because they do not instrument the intermediate steps. The paper's Table 1 demonstrates the diagnostic value: PI-SERINI with gpt-5.5 achieves 94.7% surfaced evidence recall but only 73.6% previewed recall and 58.9% behavior recall. The drop from 94.7% to 73.6% means that roughly 21% of ground-truth evidence documents are in the cached rankings but the agent never browses to their position. The further drop to 58.9% means that even when the agent previews relevant documents, it does not always read or cite them. These are qualitatively different failure modes with different remedies: the previewed recall gap suggests the agent needs better browsing strategies or more time to inspect deep rankings; the behavior recall gap suggests the agent needs better evidence evaluation or synthesis capabilities.

This decomposition has implications beyond this paper. It provides a methodology for future deep research evaluations to report not just answer accuracy and cost, but also the intermediate document access metrics that reveal why a system succeeded or failed. A system that achieves 80% accuracy with 95% surfaced recall has very different properties from one that achieves 80% accuracy with 60% surfaced recall. The former can potentially improve by upgrading the agent's inspection strategy; the latter requires a better retriever. Without the four-tier logging, these systems would be indistinguishable.

The innovation is fundamentally measurement-oriented rather than method-oriented. It does not improve any component of the system; it makes the system's behavior legible in a way that enables better decisions about which components to improve. This is characteristic of mature engineering disciplines — think of profilers in software engineering or segment-level analytics in marketing — but is rare in LLM agent research, where evaluation typically treats the agent as a black box.

Innovation 4: Demonstrating That LLM Capability Growth Changes the Retriever Adequacy Calculus

The paper's research question is explicitly conditional: "Does a lexical retriever suffice as LLMs become more capable in an agentic loop?" The answer the paper provides — yes, under specific conditions — is not just an empirical finding but a conceptual reframing of the retriever selection problem as LLM-capability-dependent. This challenges the prevailing assumption that retriever quality is an absolute, task-fixed property: that for a given corpus and query distribution, better retrieval models will always produce better end-to-end system performance.

The paper's evidence for capability-dependence is most visible in the variation across LLMs in Table 1. With the same BM25 retriever, the same retrieval depth, and the same tool interface, answer accuracy ranges from 54.8% (claude-haiku-4.5) to 83.1% (gpt-5.5) — a 28.3 percentage point spread. This variation is not attributable to retrieval differences (surfaced recall is similarly high across all PI-SERINI variants, ranging from 81.2% to 95.8%) but to differences in how effectively each LLM navigates the cached rankings, decides which documents to read, and synthesizes evidence into answers. The retriever provides the same candidate set; the LLM determines what the agent does with it.

The implication is that investments in better retrievers and investments in more capable LLMs are partially substitutable for deep research tasks. When paired with a sufficiently capable LLM, a well-configured lexical retriever can achieve performance comparable to a dense retriever paired with a less capable LLM. The decision of where to allocate improvement effort — retriever quality vs. LLM capability vs. agent interface design — should depend on the relative costs and the current capability frontier, not on a fixed assumption that retriever quality is the binding constraint.

The paper's comparison between claude-opus-4.7 and gpt-5.5 (both high-cost frontier models, both paired with the same BM25 configuration, achieving 69.8% vs. 83.1% accuracy respectively) further refines this picture: even among frontier models, capability differences produce large end-to-end performance gaps that dwarf the gaps attributed to retriever choice in prior work. The Appendix C trajectory analysis identifies a specific failure mode — premature branch commitment, where claude-opus-4.7 pursues a weak hypothesis (e.g., searching vertically within the "Renato Paris" candidate) while gpt-5.5 keeps its probes reversible — that explains the performance gap. This is a purely agent behavior difference, not a retrieval quality difference, and it would not be visible in systems that conflate retrieval and agent performance.

This innovation is an empirical finding with theoretical implications. It suggests that the retriever adequacy threshold — the minimum retrieval quality needed for an agent to achieve a target accuracy — is not fixed but decreases as LLM capability increases. A more capable LLM can compensate for retrieval gaps through better query formulation, more strategic result inspection, and more robust evidence evaluation. This does not mean retrieval quality is unimportant; it means that retrieval quality interacts with agent capability in ways that prior evaluations, which typically fix the LLM and vary the retriever, cannot capture.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use BrowseComp-Plus (Chen et al., 2025), a deep research benchmark consisting of 830 queries and 100,195 documents. On average, each query is associated with 6.1 evidence documents and 2.9 gold documents, with documents averaging approximately 5,179 words and 32,296 characters. Evidence documents are those required to answer the query, while gold documents are a stricter subset that both support answering and semantically contain the final answer. The paper uses the full 830-query test set for all main experiments, with a 100-query subset used for BM25 parameter tuning and ablation studies.

• Base models. PI-SERINI is evaluated using frontier LLMs from three providers, selected to represent "the class of sufficiently capable LLMs considered in our research question" (Section 4.2). For OpenAI: gpt-5, gpt-5.2, gpt-5.4, gpt-5.4-mini, and gpt-5.5. For Anthropic: claude-haiku-4.5 and claude-opus-4.7. For DeepSeek: deepseek-v4-flash and deepseek-v4-pro. This diverse model set spans multiple capability levels and cost tiers, enabling analysis of how LLM capability interacts with retriever adequacy. The baseline comparisons use o3 and gpt-5 from Chen et al. (2025), gpt-5.2 from Meng et al. (2026), and AgentIR-4B (a reasoning-aware retriever) from Chen et al. (2026).

• Metrics. The paper reports both answer-quality metrics and retrieval-behavior metrics. For answer quality: Accuracy is the fraction of queries whose final answers are judged correct by an LLM judge (gpt-5.3-codex, configured per Appendix A.5) that determines whether the extracted final answer is semantically equivalent to the benchmark-provided correct answer, allowing harmless wording differences, equivalent formatting, and added correct detail. Calibration Error is the discrepancy between the model's self-reported confidence (0–100%) and empirical correctness; the paper reports this value without specifying the exact computation formula. For retrieval behavior: Surfaced Recall is recall computed over all documents returned by any search call during the trajectory (D_surfaced). Previewed Recall is recall over documents whose excerpts were shown through read_search_results calls (D_previewed). Behavior Recall is recall over the union of documents read via read_document and documents cited in the final answer (D_opened ∪ D_cited). All recall metrics are reported separately for evidence documents and gold documents. Cost is total USD computed from per-model token pricing (Appendix B, Table 4) accounting for input, output, and cache read tokens.

• Baselines. The paper compares PI-SERINI against several representative BrowseComp-Plus systems: (1) Released baselines from Chen et al. (2025) pairing o3 and gpt-5 with either qwen3-embed-8b (a dense retriever) or BM25, using a single retriever tool that directly returns top-5 results. (2) The system from Meng et al. (2026) pairing gpt-5.2 with qwen3-embed-8b, replacing the reasoning model but using the same dense retriever. (3) AgentIR-4B from Chen et al. (2026), a reasoning-intensive retriever specifically trained for deep research. Additionally, Figure 1 includes reference points from Cao et al. (2026) and Li et al. (2026) for coding agents that reformulate deep research as file-system navigation, but these are excluded from Table 1 due to differences in problem formulation (documents are pre-localized to a working set).

• Generation budget / compute accounting. The paper does not use a "generation budget" in the standard sense (e.g., number of sampled completions). Instead, computation is bounded by a wall-clock timeout of T = 300 seconds per query, with a two-stage steering policy: at 0.7T (210 seconds), a submission steer is injected and all tools are blocked; at T (300 seconds), the process is forcibly terminated. Cost is measured in USD, computed from actual token consumption using per-model pricing (Appendix B). Tool calls are unbounded within the time budget, reported as average tool calls per query in Table 2. For the 100-query ablation subset in Section 6, the paper additionally compares timeout-based termination against a fixed 100-iteration cap (matching prior work) and a 3600-second timeout (simulating absence of practical termination).

• Cross-validation / statistical protocol. The paper does not report formal statistical significance testing or confidence intervals for any metrics. For BM25 parameter tuning (Section 6, Figure 3), the paper samples 100 queries from BrowseComp-Plus and performs a grid search over k1 and b, measuring surfaced evidence document recall as the objective; the best configuration is then validated on the full 830-query test set. For the gold-answer LLM judge, each evaluation call runs with a 180-second timeout and outputs structured JSON including correctness (correct: boolean) and the judge's own confidence; however, no inter-judge reliability or judge calibration analysis is reported. The paper does not describe any cross-validation protocol for strategy selection (unlike the compute-optimal framework in the example paper, which uses two-fold cross-validation within difficulty bins).

Main Quantitative Results

The paper's main results are organized around four axes of comparison: answer quality across retriever types and LLMs, retrieval behavior across document access tiers, cost analysis, and tool usage patterns. I'll walk through each in turn.

Answer Quality: BM25 vs. Dense Retrievers When Controlled for Configuration

The central quantitative claim is that BM25-based PI-SERINI can match or exceed dense-retriever baselines when BM25 is well-configured and used with sufficient retrieval depth. Table 1 provides the evidence.

Headline numbers. PI-SERINI with gpt-5.5 and BM25 achieves 83.1% accuracy (row m) — the highest in the table. PI-SERINI with gpt-5 and BM25 achieves 74.6% (row i), compared to the released gpt-5 + qwen3-embed-8b dense-retriever baseline at 73.0% (row d). PI-SERINI with gpt-5.2 and BM25 achieves 70.5% (row j), while Meng et al. (2026) with the same gpt-5.2 LLM but qwen3-embed-8b dense retriever achieves only 45.1% (row e).

Within-LLM, across-retriever comparison (gpt-5). Under the same gpt-5 LLM, the comparison is:

• Released baseline gpt-5 + BM25 (Chen et al., 2025, row c): 58.3% accuracy
• Released baseline gpt-5 + qwen3-embed-8b (row d): 73.0% accuracy
• PI-SERINI gpt-5 + BM25 (row i): 74.6% accuracy

The 16.3 percentage point improvement from the released BM25 baseline (58.3%) to PI-SERINI's BM25 (74.6%) is achieved without changing the retriever type or the LLM — only BM25 parameters, retrieval depth, and the tool interface differ. This directly supports the paper's claim that "previously released baselines understate the potential of BM25" (Section 5). The PI-SERINI BM25 configuration also slightly exceeds the dense-retriever baseline (74.6% vs. 73.0%), which contradicts the narrative that dense retrievers are necessary for competitive deep research performance.

Across-LLM comparisons. The performance range across PI-SERINI configurations with BM25 spans from 54.8% (claude-haiku-4.5, row g) to 83.1% (gpt-5.5, row m) — a 28.3 percentage point gap. This variation occurs while surfaced recall remains high (81.2%–95.8% across all PI-SERINI variants), indicating that retrieval effectiveness is not the binding constraint. The LLM's reasoning and tool-use capabilities drive most of the answer quality variation.

Two comparisons are particularly striking: (1) gpt-5.5 (83.1%) vs. claude-opus-4.7 (69.8%) — both high-cost frontier models, yet a 13.3 percentage point accuracy gap. (2) deepseek-v4-flash (68.1%, row n) vs. claude-opus-4.7 (69.8%) — near-identical accuracy at dramatically different costs ($28.9 vs.$246.6), suggesting DeepSeek models offer a strong cost-performance trade-off.

Dense retrievers do not guarantee better end-to-end quality. The Meng et al. (2026) system (row e) pairs gpt-5.2 with qwen3-embed-8b and achieves 45.1% accuracy — far below PI-SERINI with the same LLM and BM25 at 70.5% (row j). AgentIR-4B (row f), a reasoning-aware retriever specifically trained for deep research, achieves 68.1% — below four PI-SERINI BM25 configurations. The paper interprets this as evidence that "final answer quality is not determined by retriever choice alone, but also by how the agent searches, manages context, and interacts with retrieved documents" (Section 5).

Calibration error. The paper reports calibration error alongside accuracy in Table 1 but does not analyze these numbers in the text. Values range from 6.2 (gpt-5.2) to 17.3 (claude-haiku-4.5). The paper does not specify whether these represent overconfidence or underconfidence, or how they correlate with accuracy. gpt-5.5 shows relatively high calibration error (15.7) despite highest accuracy, which could indicate overconfidence on its incorrect answers — but this is not discussed.

Retrieval Behavior: Surfaced, Previewed, and Behavior Recall

The retrieval behavior metrics (Table 1, right columns) provide the diagnostic decomposition that distinguishes retrieval failures from agent inspection failures.

Surfaced recall. All PI-SERINI variants substantially outperform the released baselines on surfaced recall. For evidence documents:

• PI-SERINI variants range from 81.2% (claude-opus-4.7) to 95.8% (deepseek-v4-flash)
• Released baselines range from 56.6% (o3 + BM25, row a) to 79.2% (AgentIR-4B, row f)
• The dense-retriever baseline gpt-5 + qwen3-embed-8b (row d) achieves 79.0%

The key comparison is between gpt-5 + BM25 in PI-SERINI (row i: 90.5% surfaced evidence recall) vs. gpt-5 + qwen3-embed-8b (row d: 79.0%). The BM25 configuration in PI-SERINI surfaces 11.5 percentage points more relevant documents than the dense retriever baseline. This is the opposite of what one would expect if dense retrievers were uniformly superior — the PI-SERINI BM25 configuration finds more evidence, likely because of the 1000-document retrieval depth rather than because BM25 ranks documents better.

For gold documents (the stricter subset), the pattern is similar: PI-SERINI with gpt-5.5 achieves 95.3% gold surfaced recall (row l) vs. 81.3% for the dense-retriever baseline (row d). PI-SERINI with gpt-5.5 achieves 94.4% gold surfaced recall (row m).

Previewed recall. Previewed recall measures what fraction of relevant documents the agent actually sees in search result pages:

• PI-SERINI with gpt-5.5 achieves the highest previewed recall: 73.6% evidence, 72.9% gold (row m)
• The gap between surfaced and previewed recall is substantial: for gpt-5.5, 94.7% surfaced vs. 73.6% previewed means ~21% of evidence documents are in cached rankings but never previewed
• Previewed recall varies considerably across LLMs (from 43.3% for claude-opus-4.7 to 73.6% for gpt-5.5), even though surfaced recall is consistently high

The paper notes that "because agents can often synthesize answers from previewed excerpts without calling read_document, higher previewed recall need not yield proportional gains in behavior recall" (Section 5). This is observable in the data: gpt-5.4 (row l) has high previewed recall (70.3% evidence) but behavior recall of only 51.8%, suggesting it extracts sufficient information from excerpts alone or fails to translate previews into citations.

Behavior recall. Behavior recall (D_opened ∪ D_cited) represents documents the agent actually read or cited:

• PI-SERINI with gpt-5.5: 58.9% evidence, 56.1% gold (row m) — the highest behavior recall
• PI-SERINI with deepseek-v4-flash: 55.2% evidence, 60.6% gold (row n) — notably higher gold behavior recall than evidence behavior recall
• The gap from surfaced recall (~95%) to behavior recall (~55–60%) is approximately 35–40 percentage points. This is the most informative number in the table: even with near-perfect surfaced recall, agents only actually use about 60% of the available evidence.

Interpreting the tiers. The three-tier recall breakdown reveals a clear bottleneck pattern. For gpt-5.5: 94.7% surfaced → 73.6% previewed → 58.9% behavior. The largest absolute drop (21.1 percentage points) occurs between surfaced and previewed — this is the "browsing bottleneck": relevant documents exist in the ranking but the agent never examines their excerpts. The next drop (14.7 percentage points) occurs between previewed and behavior — this is the "evaluation bottleneck": the agent sees excerpts but fails to read or cite the documents. This decomposition is not possible with single-tool baselines, which only measure surfaced recall (and at shallow retrieval depth, where the surfaced set is inherently small).

Cost Analysis

The paper claims PI-SERINI "substantially lowers evaluation cost while preserving competitive performance" (Section 5). Table 1 reports total cost in USD for each full 830-query benchmark run.

Headline cost comparisons. Under the same gpt-5 + BM25 setting, PI-SERINI reduces cost from $400.4** (released baseline, row c) to **$94.9 (row i) — a 4.2× reduction. Even PI-SERINI with the most expensive LLM (gpt-5.5, row m: $291.6) costs less than the released gpt-5 baselines with either BM25 ($400.4, row c) or qwen3-embed-8b ($360.7, row d).

The comparison to Meng et al. (2026) is particularly stark. Their gpt-5.2 + qwen3-embed-8b experiment (row e) costs "roughly $1000–$2000" per the paper's citation, while PI-SERINI with the same gpt-5.2 LLM costs $122.2 (row j) — an 8×–16× reduction. This is attributed partly to PI-SERINI using far fewer tool calls (17.2 average per query vs. 73.8 for Meng et al., per Table 2) and partly to the prefix-cache-friendly agentic loop.

Cost variation across LLMs. Within PI-SERINI configurations, cost varies substantially: $28.9 (deepseek-v4-flash) to$291.6 (gpt-5.5). The DeepSeek models offer particularly strong cost-efficiency: deepseek-v4-flash achieves 68.1% accuracy at $28.9, comparable to claude-opus-4.7's 69.8$246.6 — an 8.5× cost difference for similar accuracy. This is visible in the Pareto frontier visualization of Figure 1, where PI-SERINI (DeepSeek) points occupy the lower-left (low cost, moderate accuracy) region and PI-SERINI (OpenAI) points trace the upper-right toward higher accuracy and cost.

Cost drivers. The paper discusses three factors contributing to cost efficiency in Section 6: BM25 tuning, termination policy, and prefix caching. The prefix-cache-friendly loop is identified as "a major contributor to PI-SERINI's overall cost efficiency" based on the high cached-token ratios (around 82–90% of total tokens served from cache, Table 3). Given that cache read tokens cost 10× less than input tokens for most models (e.g., gpt-5.5: $5.00/1M input vs.$0.50/1M cache read, Appendix B Table 4), a 90% cache hit rate would reduce effective input cost by roughly 5× compared to no caching.

Figure 1 accuracy–cost trade-off. The paper's Figure 1 plots accuracy against cost for all systems, showing PI-SERINI configurations consistently on or near the Pareto frontier. The coding-agent baselines from Cao et al. (2026) and Li et al. (2026) occupy a region with comparable or higher accuracy at lower cost, but the paper argues these results are not directly comparable because those systems assume pre-localized document subsets (see Section 7 discussion).

Tool Usage Patterns

Table 2 reports average tool calls per query across systems, decomposed into search calls, read calls (read_document), and browse calls (read_search_results).

Total tool calls. PI-SERINI systems show substantial variation in total tool calls: from 9.0 (claude-opus-4.7) to 41.3 (claude-haiku-4.5) per query. This is directly comparable to the released baselines: o3 + BM25 uses 25.9, gpt-5 + BM25 uses 23.2, and gpt-5 + qwen3-embed-8b uses 21.7 (all from Chen et al., 2025). Most PI-SERINI configurations use similar or fewer total calls than the baselines — gpt-5 + BM25 in PI-SERINI uses 15.2 vs. 23.2 for the released baseline with the same LLM and retriever.

The Meng et al. (2026) system stands out: 73.8 tool calls per query on average, far more than any PI-SERINI configuration. Combined with lower accuracy (45.1%), this suggests the agent is spinning its wheels — issuing many searches without converging to a correct answer. The paper interprets this as evidence that "replacing dense retrievers with BM25 does not necessarily increase the number of tool calls, while still maintaining competitive retrieval effectiveness and answer accuracy" (Section 5).

Tool call composition. Within PI-SERINI, search calls dominate (68–79% of total calls across configurations), with read calls comprising most of the remainder (18–23%) and browse calls being rare (0.1–0.9 calls per query). The low browse usage is notable: despite the design emphasis on read_search_results as a mechanism for exploring deep rankings, agents rarely use it. This suggests that either (a) agents find what they need in the top-5 results of their search calls and don't need to browse deeper, or (b) agents prefer to refine their query and issue a new search rather than browse deeper into an existing ranking. The paper does not comment on this pattern.

LLM-level variation. The variation in tool calls across LLMs is informative about agent behavior: claude-opus-4.7 uses the fewest calls (9.0 total, 6.8 search) and achieves moderate accuracy (69.8%), while claude-haiku-4.5 uses the most calls (41.3 total, 31.8 search) and achieves the lowest accuracy (54.8%). This is not a simple "more calls = better" relationship — gpt-5.5 achieves the highest accuracy (83.1%) with moderate call volume (19.3 total, 13.5 search). The implication is that call efficiency (making the right calls) matters more than call volume.

Ablation Studies and Robustness Checks

All ablation studies are conducted on a 100-query subset of BrowseComp-Plus (Section 6), not the full 830-query test set. This is an important caveat: the ablation results may not generalize to the full benchmark with the same precision as the main results. The paper uses gpt-5.4 for these ablations (stated in Figure 2 caption and Table 3), which is a mid-tier model (73.3% accuracy on the full benchmark, row l) rather than the best-performing gpt-5.5.

Retrieval depth: Increasing k from 5 to 1000 improves surfaced evidence recall by 25.3 percentage points (70.5% → 95.8%), but previewed recall saturates around k = 50 at 74.7% (Figure 2). The paper varies the number of documents returned by search (parameter k) and measures both surfaced and previewed recall. Surfaced evidence recall increases from 70.5% at k = 5 to 86.2% at k = 100, then continues climbing more slowly to 95.8% at k = 1000. The gold-document surfaced recall follows a similar trajectory. Previewed recall increases from 70.5% at k = 5 (where all surfaced documents are also previewed because there are only 5) to 74.1% at k = 100, then essentially plateaus: 74.7% at k = 50 is the peak, with values declining slightly at k = 500 and k = 1000. This saturation is the critical finding: the agent does not browse proportionally deeper as more documents become available. The paper interprets this as showing "retrieval depth is a major driver of surfaced recall" but "the saturation of previewed recall suggests that larger cached rankings do not automatically lead the agent to inspect more relevant evidence" (Section 6). The implication is clear: deep retrieval is necessary to get evidence into the cache, but agent browsing behavior is the bottleneck for translating surfaced evidence into inspected evidence.

BM25 parameter tuning: Tuning k1 and b from ANSERINI defaults (0.9, 0.4) to tuned values (25, 1) improves answer accuracy by 18.0 percentage points (64.0% → 82.0%) on the 100-query subset (Table 3, first two columns). The paper conducts a grid search over k1 and b on 100 queries, visualized in Figure 3. ANSERINI's default setting (k1 = 0.9, b = 0.4) falls in a low-performing region (marked with ×). The best-performing configuration is near k1 = 16, b = 1.0 (marked with ⋆). The paper also tests k1 = 25, b = 1.0 (the configuration used in all main experiments) and notes similar downstream results (82.0% vs. 81.1% for the best grid-search setting). The downstream effects on the 100-query subset (Table 3, first two columns) are:

• Accuracy: 64.0% (default) → 82.0% (tuned), an 18.0 percentage point gain
• Surfaced evidence recall: 84.6% → 95.7%, an 11.1 percentage point gain
• Previewed recall: 50.7% → 70.4%, a 19.7 percentage point gain
• Behavior recall: 36.4% → 52.2%, a 15.8 percentage point gain
• Cost: $24.1 →$21.7, a modest reduction

The improvement in surfaced recall is expected (better ranking means more relevant documents in the top-1000). The larger improvement in previewed recall (19.7 points) is less obvious: better ranking also means relevant documents appear higher in the cached ranking, making them more likely to be in the pages the agent actually browses. The improvement in behavior recall (15.8 points) shows that the agent reads and cites more evidence when it's better-ranked. The cost reduction suggests that better retrieval leads to more efficient agent behavior (fewer wasted searches).

Termination policy: A 300-second timeout achieves similar or better accuracy at lower cost than a 100-iteration cap or a 3600-second timeout (Table 3, columns 2–4). The paper compares three termination conditions on the 100-query subset with tuned BM25 and gpt-5.4:

• Timeout 300s (standard PI-SERINI): 82.0% accuracy, $21.7 cost, 23.1 avg tool calls
• Max 100 iterations (matching prior work): 76.0% accuracy, $24.9 cost, 27.3 avg tool calls
• Timeout 3600s (essentially no termination constraint): 83.0% accuracy, $26.3 cost, 27.1 avg tool calls

The 100-iteration cap achieves lower accuracy (76.0% vs. 82.0%) at higher cost ($24.9 vs.$21.7) — this is a surprising result, since one might expect a generous iteration budget to improve accuracy. The paper does not explain this counterintuitive finding, but a possible mechanism is that agents hitting the iteration cap are forced to answer with incomplete reasoning, while the timeout-based agents have more flexibility to reason within their time budget. The 3600-second timeout achieves marginally higher accuracy (83.0% vs. 82.0%) at higher cost ($26.3 vs.$21.7), suggesting diminishing returns to additional time.

The paper also reports a column on the 100-query subset (Table 3, last column, "Tuned BM25, Timeout 3600s") with accuracy marked as "Acc.† 83.0%" and a second accuracy figure "Acc.† 92.7%" but appears to have a table formatting error — the "92.7%" value appears under "MaxIter100" column but relates to some other metric. Given the ambiguity, I'll note this as a potential error in the paper.

Prefix caching impact: The agent loop achieves 82–90% cached token ratios, making prefix caching a major cost efficiency driver (Table 3, "Med. Cached Tokens" rows). Table 3 reports median total tokens and median cached tokens across the four termination policy settings. Cached token ratios range from 82% (215,552/263,880.5 for MaxIter100) to 90% (228,736/281,324 for default BM25), with all settings showing high cache utilization. The paper does not run an ablation without prefix caching (e.g., by clearing the cache between turns), so the cost savings specifically attributable to caching are not isolated. However, the paper argues that "the prefix-cache-friendly loop is a major contributor to PI-SERINI's overall cost efficiency" (Section 6) based on the high cached-token ratios and the 10× price differential between cache read and input tokens (Appendix B, Table 4).

LLM capability variation (not a formal ablation, but a robustness check across models): The PI-SERINI architecture produces consistent high surfaced recall across all tested LLMs (81.2%–95.8%), but answer accuracy varies substantially (54.8%–83.1%), confirming that agent capability rather than retrieval quality drives performance variation (Table 1). The paper tests 9 LLM configurations with the same BM25 retriever, same retrieval depth, and same tool interface. Surfaced recall is uniformly high (all above 81%, most above 90%), indicating that the retrieval pipeline is not the source of performance variation. This is effectively a robustness check: PI-SERINI's retrieval design works across LLM families (OpenAI, Anthropic, DeepSeek) and scales (flash, pro, mini variants), producing consistent retrieval behavior.

Failure mode analysis (Appendix C): Premature branch commitment explains the gap between gpt-5.5 and claude-opus-4.7, showing that agent search strategy — not retrieval quality — drives performance differences. The paper includes a detailed trajectory comparison for Query 678 in Appendix C, which is not a controlled ablation but serves as qualitative evidence for the mechanism behind the 13.3 percentage point accuracy gap between gpt-5.5 (83.1%) and claude-opus-4.7 (69.8%). The trajectories show that both agents sometimes use internal knowledge or weakly related entities from retrieved documents to expand queries. The key behavioral difference is reversibility: gpt-5.5 "tends to keep candidate-specific probes reversible, returning to the original clues when they fail," while claude-opus-4.7 "often searches vertically within the same weak hypothesis" — continuing to issue queries about a candidate (Renato Paris) even after evidence fails to confirm the original constraints. This is a search strategy difference, not a retrieval quality difference, and it would affect any retriever paired with these models.

Negative result — ReSTEM revision model (not in this paper, but referenced as a cautionary example): The paper does not include any negative ablation results within its own experiments. All ablations (tuning, depth, termination policy) show improvements from the paper's design choices. The one cautionary note comes from prior work: Appendix K in the reference example discusses how attempting to optimize a revision model with ReSTEM (RL-style training) actually degraded performance. This paper does not replicate that finding, but the absence of negative results is itself worth noting — every design choice tested (deeper retrieval, tuned parameters, time-budget steering) improves the metrics.

Critical Assessment

Does the paper demonstrate that BM25 "can suffice" for deep research?

The paper's central claim is qualified carefully: "with more capable LLMs in an agentic loop, BM25 can be sufficient for effective deep research when it is well-configured, used at sufficient retrieval depth, and paired with a tool interface that helps the agent manage cached retrieval ranking" (Section 1). The experiments provide strong evidence for this qualified claim but do not support a stronger, unqualified interpretation.

What is demonstrated: On BrowseComp-Plus, a system using BM25 with tuned parameters (k1 = 25, b = 1), 1000-document retrieval depth, and the PI-SERINI three-tool interface achieves answer accuracy (83.1% with gpt-5.5) that exceeds all released baselines, including those using dense retrievers like qwen3-embed-8b (73.0% best dense-retriever accuracy from Chen et al., 2025, row d). The surfaced evidence recall of 94.7% (Table 1, row m) demonstrates that BM25, with sufficient depth, can find the vast majority of evidence documents needed to answer BrowseComp-Plus queries. This is a genuine empirical finding that contradicts the narrative — reinforced by the released baselines — that BM25 is fundamentally inadequate for this task.

What is NOT demonstrated: The paper does not show that BM25 is "sufficient" in an absolute sense, because the behavior recall gap (94.7% surfaced → 58.9% behavior) means that agents fail to use roughly 36% of available evidence documents even when they are in the cached rankings. The paper acknowledges this explicitly in Section 8: "It often surfaces the right evidence documents, but the agent... does not always browse, open, or use them effectively." This means that while BM25 can surface evidence, the end-to-end system is still losing substantial information in the previewed and behavior stages. A dense retriever that ranks relevant documents higher (so they appear in the initial top-5 view rather than requiring deep browsing) might improve this pipeline loss without requiring better agent browsing behavior. The paper does not test this hypothesis because it does not run PI-SERINI with a dense retriever — all PI-SERINI runs use BM25. The comparison to dense retrievers is only against baseline agents that use different tool interfaces and shallow retrieval depth, so retriever type is confounded with agent architecture.

The confound between retriever type and retrieval depth

The most significant limitation in the paper's experimental design is that retriever type and retrieval depth are confounded in all comparisons. PI-SERINI always uses BM25 with retrieval depth 1000. The dense-retriever baselines from Chen et al. (2025) use retrieval depth 5. The Meng et al. (2026) baseline uses retrieval depth that is not explicitly stated but appears to be 5 based on the single-tool design. When PI-SERINI's BM25 outperforms these baselines, we cannot determine whether the improvement comes from (a) BM25 being a better retriever than qwen3-embed-8b for this task, (b) retrieval depth 1000 being better than retrieval depth 5, or (c) the three-tool interface being better than the single-tool interface.

The paper's own Figure 2 shows that retrieval depth alone accounts for a 25.3 percentage point improvement in surfaced recall (70.5% at k = 5 to 95.8% at k = 1000). If the dense-retriever baselines were given the same 1000-document retrieval depth with a similar pagination interface, their surfaced recall might equal or exceed PI-SERINI's BM25. The paper does not run this experiment, so the claim that BM25 "matches or exceeds" dense retrievers is actually a claim that deep BM25 with a three-tool interface exceeds shallow dense retrieval with a single-tool interface — a much narrower and less surprising finding.

A clean experiment to isolate retriever type would run PI-SERINI with a dense retriever at the same retrieval depth (1000) and same tool interface, then compare surfaced recall, previewed recall, and answer accuracy directly. The paper does not do this, and it is the single most important missing experiment for supporting the central claim.

The missing counterfactual: PI-SERINI with a dense retriever

Throughout the paper, the comparison is always PI-SERINI + BM25 vs. other agents + dense retrievers. The paper never tests PI-SERINI + dense retriever to establish whether the BM25 choice is actually responsible for the competitive performance, or whether PI-SERINI's architecture (deep caching, three-tool interface, time-budget steering, prefix caching) would produce even better results with a dense retriever. This matters because if PI-SERINI + dense retriever achieved, say, 90% accuracy, then BM25 would not be "sufficient" in the sense of being non-inferior — it would be leaving performance on the table relative to an alternative that is equally feasible to deploy.

The paper's conclusion — "The lesson is not that BM25 is enough everywhere, but that under-configured baselines can hide weak agent–retriever interaction" (Section 8) — implicitly acknowledges this limitation. The paper is arguing for better baselines and better experimental hygiene, not for BM25's universal adequacy. But readers may miss this nuance when the abstract and introduction emphasize that PI-SERINI "outperforms released search agents that use dense retrievers."

The 100-query ablation subset and generalizability

All ablations (retrieval depth, BM25 tuning, termination policy) are conducted on a 100-query subset, not the full 830-query test set. The paper does not describe how this subset was sampled (random? stratified by difficulty? first 100 queries?) or report confidence intervals for the subset results. The tuned BM25 accuracy on the subset (82.0%, Table 3) is higher than the full-benchmark accuracy for the same model (gpt-5.4 achieves 73.3% on the full benchmark, Table 1 row l), suggesting the subset may not be representative. This means the 18.0 percentage point improvement from BM25 tuning (64.0% → 82.0% on the subset) may not translate to an 18.0 point improvement on the full benchmark — it could be larger or smaller depending on how the subset's difficulty distribution differs.

Single benchmark, single corpus

All experiments are on BrowseComp-Plus, a single benchmark with 100,195 English documents. The paper does not test on BrowseComp (the open-web version), BrowseComp-ZH (Chinese), or any other deep research benchmark. The BM25 tuning (k1 = 25, b = 1) is corpus-specific — these parameters were selected for BrowseComp-Plus's document length distribution, and would likely need re-tuning for other corpora. The paper's claim about BM25 sufficiency is therefore bounded by this specific corpus, and the generalizability to other document collections (shorter documents, multilingual, different domains) is unknown.

Evaluation judge reliability

The LLM judge (gpt-5.3-codex, Appendix A.5) is the sole arbiter of answer correctness. The paper does not report inter-judge agreement (e.g., comparing gpt-5.3-codex judgments against human evaluation or against a different judge model), judge calibration on this specific task, or any analysis of judge failures. If the judge has systematic biases — for example, being more lenient toward verbose answers or more strict about numerical precision — these biases would affect all systems but could differentially impact systems with different answer formats (PI-SERINI uses a structured Explanation/Exact Answer/Confidence format, while baselines may use different formats). The paper does not discuss this potential confound.

Statistical significance and variance

The paper reports no confidence intervals, no standard deviations, and no statistical significance tests for any metric. Given the 830-query test set, differences of a few percentage points may be within sampling error. For example, PI-SERINI gpt-5 + BM25 at 74.6% vs. gpt-5 + qwen3-embed-8b at 73.0% is a 1.6 percentage point difference — small enough that it could disappear with a different random seed or minor prompt variation. The paper's strongest claims (e.g., the 83.1% vs. 73.0% comparison) involve larger gaps that are less likely to be noise, but the absence of variance reporting makes it difficult to assess the reliability of smaller differences.

The calibration error metric is undefined and unanalyzed

Table 1 reports calibration error for all PI-SERINI configurations, but the paper never defines the computation, never discusses the results, and never explains why calibration error varies across models (6.2 for gpt-5.2 vs. 17.3 for claude-haiku-4.5). This metric seems included for completeness rather than to support any claim, and its presence without analysis is a minor weakness in an otherwise careful experimental design.

Coverage of the claim about "3.3×–10× cost reduction"

The abstract claims PI-SERINI "reduces evaluation cost by 3.3×–10×" compared to dense-retriever agents. Checking Table 1: the 3.3× figure appears to come from comparing PI-SERINI gpt-5 + BM25 ($94.9, row i) to the dense-retriever baseline gpt-5 + qwen3-embed-8b ($360.7, row d) — $360.7 /$94.9 ≈ 3.8×. The 10× figure likely comes from comparing PI-SERINI gpt-5.2 ($122.2, row j) to Meng et al.'s reported ~$1000–$2000 —$1000 / $122.2 ≈ 8.2×, and$2000 / $122.2 ≈ 16.4×, with 10× being an approximate midpoint. However, the Meng et al. cost is not precisely reported (it's a range), and the comparison is between different LLMs (gpt-5.2 vs. gpt-5.2, but with different agent architectures and different retrieval depths). The cost reduction claim conflates savings from the agent architecture (fewer tool calls, prefix caching) with savings from using BM25 instead of a dense retriever, since the dense-retriever baselines also used different agent designs. This is a rhetorical overstatement — the cost reduction is real, but attributing it to "BM25-based search agents" rather than to the PI-SERINI design as a whole is imprecise.

Summary of experimental support for key claims

1. "BM25 can be sufficient for effective deep research when well-configured": Supported with qualification. The experiments show that BM25 with tuned parameters and deep retrieval can surface evidence at high recall (94.7%) and support competitive answer accuracy (83.1%). However, the large gap between surfaced and behavior recall (94.7% → 58.9%) means the system is leaving substantial evidence unused, and the paper does not test whether a dense retriever with the same architecture would close this gap.

2. "Previously released BM25 baselines underperform due to configuration and retrieval depth, not inherent limitations": Strongly supported. The BM25 tuning ablation (18.0 point accuracy improvement, Table 3) and retrieval depth ablation (25.3 point surfaced recall improvement, Figure 2) provide direct evidence that these factors account for a large fraction of the baseline gap. The PI-SERINI gpt-5 + BM25 result (74.6%) exceeding the released gpt-5 + BM25 baseline (58.3%) with the same retriever and LLM is the cleanest demonstration.

3. "Lexical retrieval agents can match or outperform dense-retriever agents": Supported with the significant caveat that retriever type is confounded with retrieval depth and agent architecture in all comparisons. The paper demonstrates that PI-SERINI + BM25 outperforms shallow dense-retriever baselines; it does not demonstrate that BM25 outperforms dense retrievers under equivalent conditions.

4. "3.3×–10× cost reduction": Supported as an empirical observation, but the attribution to BM25 specifically (rather than to the agent architecture, prefix caching, and time-budget steering) is imprecise.

6. Limitations and Trade-offs

6.1 The Central Confound: Retriever Type Is Never Isolated from Retrieval Depth and Agent Architecture

The assumption or constraint. The paper's headline claim — that BM25-based PI-SERINI "outperforms released search agents that use dense retrievers" (Abstract) — rests on comparisons where retriever type is systematically confounded with retrieval depth and agent interface design. Every PI-SERINI configuration uses BM25 with retrieval depth 1000 and the three-tool API (search, read_search_results, read_document). Every dense-retriever baseline — from Chen et al. (2025), Meng et al. (2026), and Chen et al. (2026) — uses a single retriever tool that directly returns top-5 results. The paper never runs PI-SERINI with a dense retriever to determine whether the performance advantage comes from BM25, from deep retrieval, or from the agent architecture.

The consequence. The paper cannot distinguish between three competing explanations for PI-SERINI's performance relative to dense-retriever baselines: (a) BM25 with tuned parameters is genuinely a better retriever than qwen3-embed-8b for BrowseComp-Plus; (b) retrieval depth 1000 matters more than retriever type, and a dense retriever at depth 1000 would perform similarly or better; (c) the three-tool interface with cached rankings and selective inspection is more effective than the single-tool interface, regardless of retriever. Any of these could be the primary driver, and the paper's experimental design cannot adjudicate between them. A practitioner deciding whether to invest in BM25 tuning or dense retriever deployment for their own deep research system receives no guidance on this central question.

What evidence exists in the paper. The paper's own Figure 2 demonstrates that retrieval depth alone accounts for a 25.3 percentage point improvement in surfaced evidence recall (70.5% at k=5 to 95.8% at k=1000), using BM25. This is a larger effect than many reported differences between BM25 and dense retrievers in the literature. The paper also shows (Table 3) that BM25 tuning from default to optimized parameters produces an 18.0 percentage point accuracy improvement on the 100-query subset. Together, these results suggest that configuration and depth — not retriever type — may explain most of the gap between PI-SERINI and the dense-retriever baselines. But the paper never runs the clean counterfactual experiment: PI-SERINI + dense retriever at depth 1000, with the same three-tool interface and time-budget steering, compared directly against PI-SERINI + BM25 at the same settings. This experiment would be straightforward with the PI-SERINI architecture (swap the ANSERINI backend for a dense retriever serving embeddings with nearest-neighbor search) but is absent.

Mitigation status. The paper implicitly acknowledges this confound through its cautious language — "The lesson is not that BM25 is enough everywhere, but that under-configured baselines can hide weak agent–retriever interaction" (Section 8). The conclusion is framed as a methodological corrective rather than a claim of BM25 superiority per se. However, the abstract and introduction make stronger claims ("outperforms released search agents that use dense retrievers") that a casual reader would interpret as evidence for BM25's competitiveness against dense retrievers in absolute terms. The paper suggests no future work to disentangle these factors, such as running PI-SERINI with a dense retriever backend.

6.2 The Surfaced-to-Behavior Recall Gap: Surfacing Evidence Is Not Using Evidence

The assumption or constraint. PI-SERINI demonstrates that BM25 with deep retrieval can surface nearly all relevant evidence documents (94.7% surfaced recall for gpt-5.5, Table 1, row m), and the paper treats this as the primary evidence that BM25 "can suffice" for deep research. But the paper's own trajectory logging reveals that surfaced evidence does not translate into used evidence: for gpt-5.5, only 73.6% of evidence documents are previewed (a 21.1 percentage point drop from surfaced), and only 58.9% are either read or cited (a further 14.7 point drop). The worst-case gap is larger: claude-opus-4.7 has 81.2% surfaced evidence recall but only 30.4% behavior recall — meaning roughly 50% of available evidence documents are surfaced but never used.

The consequence. The gap between surfaced recall and behavior recall means that the end-to-end system is losing approximately 36% of available evidence (for gpt-5.5) between retrieval and answer synthesis. This loss occurs in the agent's browsing and evaluation decisions — the agent does not inspect deep enough into cached rankings, or it inspects documents but fails to extract or cite their evidence. The paper's argument that BM25 "can suffice" is therefore conditional on solving this agent inspection bottleneck. If a dense retriever could rank relevant documents higher (so they appear in the initial top-5 view rather than requiring deep browsing), it might close some of this gap simply by placing evidence where the agent is more likely to see it. The paper does not test this hypothesis because it does not measure where in the ranking the surfaced-but-not-previewed documents appear for BM25 vs. where they would appear for a dense retriever.

The practical implication for a deployer is that high surfaced recall does not guarantee high end-to-end accuracy. Even with a near-perfect retriever (in terms of recall), the system's answer quality is bounded by the agent's ability to navigate and use the evidence it has been given. The paper's failure mode analysis (Appendix C) and the variation in behavior recall across LLMs (30.4% for claude-opus-4.7 vs. 58.9% for gpt-5.5, from the same surfaced recall baseline) confirm that agent capability is the binding constraint, not retrieval. But this very finding undercuts the paper's implicit message that retriever improvement is overemphasized: if better retrieval ranking could reduce the browsing burden on the agent (by surfacing evidence in the top-5 instead of at rank 500), it might improve end-to-end performance without requiring better agent capabilities.

What evidence exists in the paper. The three-tier recall metrics in Table 1 and the retrieval depth ablation in Figure 2 provide direct evidence for this limitation. Figure 2 is particularly informative: previewed recall saturates at 74.7% around k=50, while surfaced recall continues climbing to 95.8% at k=1000. This means the agent stops browsing deeper into results long before the retrieval system runs out of relevant documents to surface. The paper acknowledges this explicitly (Section 8): "It often surfaces the right evidence documents, but the agent... does not always browse, open, or use them effectively."

Mitigation status. The paper identifies this as the key failure mode and suggests it as a direction for future work: "That failure mode shifts the question of system improvements from Can the retriever find the evidence? to Can the agent recognize and spend context on the evidence it has already been given? Future progress in deep research may therefore come less from pulling the retriever lever harder, and more from designing tools that help agents navigate evidence documents with better judgments" (Section 8). This is a candid acknowledgment, but the paper does not propose or test any concrete mechanism for improving agent browsing behavior. The implicit recommendation — invest in agent design rather than retriever design — is plausible but untested.

6.3 The Difficulty Estimation and Strategy Selection Costs Are Unmeasured

The assumption or constraint. Unlike the compute-optimal test-time scaling framework described in the reference example paper (which explicitly acknowledges and measures difficulty estimation cost), PI-SERINI's approach to achieving its headline performance relies on two pre-experiment optimization steps whose costs are never quantified or amortized: (1) the BM25 parameter grid search over 100 queries (Section 6, Figure 3), which required evaluating multiple (k1, b) configurations and measuring surfaced recall for each, and (2) the prompt engineering and tool interface design that encode assumptions about BM25's behavior (prefer short lexical queries, browse before re-querying, etc., Appendix A.1). These optimization costs are implicitly treated as free — the paper reports per-query costs of $28.9–$291.6 (Table 1) without including the cost of arriving at the configuration that achieves those numbers.

The consequence. The reported cost efficiency gains (3.3×–10× over dense-retriever baselines, per the abstract) are computed after the system has already been optimized for the specific corpus and task. A practitioner deploying PI-SERINI on a new corpus with different document lengths, different query distributions, or a different domain would need to re-tune BM25 parameters (k1, b) and possibly re-engineer the agent prompt to match the retriever's characteristics. These tuning costs could be substantial: the paper's grid search (Figure 3) sampled 100 queries and tested multiple parameter combinations, each requiring running the full agentic loop. For a production deployment where the corpus is updated regularly or where multiple corpora must be supported, the ongoing tuning cost could erode the efficiency advantage.

More subtly, the paper does not measure how sensitive PI-SERINI's performance is to the specific BM25 parameters. The grid search (Figure 3) shows a plateau rather than a sharp peak — the paper notes that k1=25, b=1 and k1=16, b=1 produce similar accuracy (82.0% vs. 81.1% on the 100-query subset) — suggesting some robustness. But the performance of ANSERINI's defaults (k1=0.9, b=0.4, with 64.0% accuracy on the subset) shows that an untuned deployment would lose most of the paper's reported gains. The paper does not characterize how far from optimal a deployment can be before performance degrades unacceptably, or whether the optimal parameters transfer across corpora with similar document length distributions.

What evidence exists in the paper. The paper provides a partial picture: the grid search (Figure 3) covers a range of k1 and b values and shows the default setting's poor performance, and Table 3 quantifies the downstream effect of tuning on the 100-query subset. However, the grid search cost itself (in GPU hours, API calls, or USD) is not reported. The prompt engineering cost is entirely unmeasured — the paper provides the final prompt (Appendix A.1) but does not describe the design process, the number of prompt iterations tested, or whether alternative prompts would produce different results. This is standard practice in LLM agent papers (prompt engineering is rarely cost-accounted), but it is a limitation for a paper whose core contribution is a cost-efficiency argument.

Mitigation status. The paper does not address these costs. The abstract and Section 1 frame cost efficiency as a key contribution, but the efficiency analysis (Table 1, Figure 1) only includes per-query inference costs, not the one-time optimization costs. The paper's discussion of future work (Section 8) does not mention corpus adaptation or parameter transfer as problems to solve. This is a practical limitation for deployers, especially those with limited resources for hyperparameter tuning or those operating in domains where ground-truth query-answer pairs are scarce (the grid search requires knowing which documents are relevant to compute surfaced recall).

6.4 Single Benchmark, Single Language, and the Long-Document Assumption

The assumption or constraint. All experiments are conducted on BrowseComp-Plus (Chen et al., 2025), a single benchmark with 100,195 English-language documents where the median document length is approximately 2,000 tokens and the 90th percentile is approximately 14,000 tokens. The BM25 parameter tuning (k1=25, b=1) was selected specifically for this document length distribution. The paper acknowledges this limitation explicitly in Section 9: "Our evaluation is limited to BrowseComp-Plus, leaving open the question of whether the current agent harness generalizes to other settings, including multilingual queries and domain-specific scenarios."

The consequence. The paper's central empirical finding — that BM25 with tuned parameters and deep retrieval can suffice for deep research — may not transfer to settings with different document characteristics. Several failure modes are plausible:

• Short-document corpora (e.g., passage retrieval, factoid QA). The tuned BM25 parameters (k1=25, b=1) are optimized for long documents where term frequency continues to provide signal at high counts and full length normalization prevents very long documents from dominating. On short-document corpora, these parameters would likely produce degraded ranking: with k1=25, term frequency differences between documents (which might each contain a query term at most 1–3 times) would be underweighted, and with b=1, short documents would be heavily penalized relative to slightly-longer documents even when they contain the same relevant content. The default parameters (k1=0.9, b=0.4) that the paper showed performed poorly on BrowseComp-Plus might actually be more appropriate for short-document tasks.

• Multilingual settings. BM25's effectiveness depends on language-specific tokenization and stemming. The paper's experiments use English only. For languages with different morphological complexity, different writing systems, or different word order patterns, BM25's lexical matching behavior changes. The paper does not test whether the tuned parameters transfer or whether the agent prompt (which instructs "short lexical searches") remains effective across languages.

• Domain-specific corpora with specialized vocabulary. BrowseComp-Plus contains diverse web documents, so the vocabulary distribution is broad. In narrow domains (legal, medical, scientific), lexical matching may be both more effective (specialized terms are distinctive) and more fragile (synonyms and paraphrases are common and cannot be matched lexically). The paper does not test whether the balance between BM25's strengths and weaknesses shifts in these settings.

• Corpora where evidence documents are sparse. BrowseComp-Plus queries have an average of 6.1 evidence documents and 2.9 gold documents per query. Deep retrieval (1000 documents per query) works because the evidence-to-noise ratio is high enough that relevant documents are likely to appear somewhere in the deep ranking. In settings where each query has only 1–2 relevant documents in a much larger corpus, deep retrieval may not help if BM25 cannot rank those documents within the top-1000. The paper does not characterize the minimum evidence density required for deep lexical retrieval to be effective.

What evidence exists in the paper. The paper provides no cross-benchmark or cross-domain evidence. The limitation is acknowledged in Section 9 but not explored experimentally. The BM25 tuning grid search (Figure 3) is entirely within BrowseComp-Plus. The agent prompt (Appendix A.1) was designed for this specific corpus and task. There is no ablation testing whether the prompt instructions (e.g., "prefer short lexical searches") transfer to other corpora or whether they are BrowseComp-Plus-specific heuristics.

Mitigation status. The paper does not mitigate this limitation. It acknowledges it in Section 9 as a direction for future work but does not even speculate about which findings might generalize. A practitioner considering PI-SERINI for a different corpus should expect to (a) re-tune BM25 parameters via grid search, (b) re-evaluate whether the three-tool API and browsing strategy remain appropriate, and (c) possibly re-engineer the agent prompt. The paper provides a methodology for doing this (the grid search, the ablation framework) but no evidence about how costly or effective this adaptation process would be.

6.5 The Missing LLM Judge Reliability Analysis

The assumption or constraint. All answer accuracy results in Table 1 are determined by a single LLM judge (gpt-5.3-codex, Appendix A.5) that compares the agent's extracted final answer against a benchmark-provided correct answer and returns a binary correct judgment. The paper provides no inter-judge reliability analysis (e.g., comparing gpt-5.3-codex judgments against a different judge model or against human evaluation), no calibration analysis of the judge's own confidence scores, and no analysis of judge failure modes. The judge prompt instructs it to "allow harmless wording differences, equivalent formatting, and added correct detail," which requires semantic equivalence judgments that may be inconsistent across model calls.

The consequence. The paper's headline accuracy numbers — particularly the small differences between systems — could be affected by judge bias or noise. For example, PI-SERINI gpt-5 + BM25 at 74.6% vs. gpt-5 + qwen3-embed-8b at 73.0% (Table 1, rows i and d) is a 1.6 percentage point difference. On an 830-query test set, this represents approximately 13 queries. If the judge makes systematic errors on even a small fraction of these — for example, accepting verbose answers that happen to contain the correct answer string without actually answering the question, or rejecting correct answers that use unusual formatting — the ordering of systems could change. Without a reliability analysis, the reader cannot assess whether a 1.6 point difference is signal or noise.

The structured response format that PI-SERINI uses (Explanation: ... Exact Answer: ... Confidence: ...) may also interact with the judge differently than the response formats of baseline systems. If PI-SERINI's structured format makes the exact answer easier for the judge to extract correctly, this could inflate PI-SERINI's accuracy relative to baselines with less structured outputs. Conversely, if the Exact Answer format sometimes produces overly terse answers that the judge rejects as incomplete, this could deflate PI-SERINI's accuracy. The paper does not analyze these interaction effects.

What evidence exists in the paper. The paper provides the judge configuration and prompt (Appendix A.5) and reports judge confidence as a field in the judge output schema, but never reports or analyzes judge confidence scores. The judge's own confidence metric is unused despite being collected. The paper does not describe any manual verification of judge outputs, any comparison against a second judge, or any analysis of judge disagreement patterns. The calibration error reported in Table 1 is for the agent's self-reported confidence against correctness, not the judge's confidence.

Mitigation status. Partially mitigated by standard practice. LLM-as-judge is widely used in agent evaluation and the paper's judge setup (separate model from the agents being evaluated, structured JSON output, explicit instructions) follows reasonable conventions. The paper also uses the same judge for all systems (both PI-SERINI and baselines), so systematic judge biases should affect all systems similarly — but this does not address noise or format-dependent judge behavior. The paper does not suggest improvements to the evaluation protocol as future work, which is a missed opportunity given that PI-SERINI's four-tier trajectory logging would naturally support a more detailed error analysis (e.g., for queries the agent got wrong despite high surfaced recall, did the judge correctly identify the error?).

6.6 The Time-Budget Steering Mechanism May Penalize Hard Queries and Reward Fast-but-Shallow Answers

The assumption or constraint. PI-SERINI uses a wall-clock time budget of 300 seconds per query with a two-stage steering policy: at 210 seconds (70% of budget), a submission steer is injected instructing the agent to stop using tools and produce its best answer; at 300 seconds, the process is forcibly terminated. This policy assumes that 300 seconds (and the 90-second window for answer synthesis after the steer) is sufficient for all queries in the benchmark. The paper does not characterize how query complexity varies across BrowseComp-Plus or whether the fixed time budget differentially affects easy vs. hard queries.

The consequence. The time-budget mechanism creates an implicit accuracy–completeness tradeoff that the paper does not analyze. Queries that require reading many long documents, synthesizing evidence from multiple sources, or iterating through many query reformulations may be cut off before the agent can find and process sufficient evidence. These queries would be marked as incorrect not because the retriever failed or the agent made bad decisions, but because the time budget was too short. Conversely, queries for which the agent quickly finds a plausible-but-incorrect answer within the time budget may be rewarded relative to an agent that would have found the correct answer given more time.

The 300-second budget is not calibrated to query difficulty — it is uniform across all 830 queries. The paper's ablation (Table 3) compares 300 seconds against 3600 seconds and finds only a marginal accuracy improvement (82.0% → 83.0% on the 100-query subset), suggesting that most queries are resolved within the budget. But this aggregate result might mask bimodal behavior: most queries finish quickly, while a small subset of complex queries times out prematurely, and the 3600-second budget allows some of these to succeed. If so, the 300-second budget would be a bottleneck specifically for the hardest queries — the ones where test-time compute is most valuable. The paper does not report accuracy broken out by query completion time, by whether the agent received the submission steer, or by query difficulty.

The steering mechanism itself (injecting a "submit now" instruction) may interact with the agent's reasoning quality in ways the paper does not measure. If an agent is in the middle of a complex reasoning chain when the steer arrives, the forced answer might be the best the agent can produce under pressure — but "best under pressure" may be substantially worse than "best after completing the reasoning chain." The paper does not compare the quality of answers produced after the submission steer against answers produced naturally within the time budget.

What evidence exists in the paper. The Table 3 ablation provides some evidence: the 3600-second budget achieves only marginally higher accuracy than the 300-second budget (83.0% vs. 82.0% on the 100-query subset), while the 100-iteration cap achieves lower accuracy (76.0%). This suggests the 300-second timeout is not severely constraining overall performance. However, the 100-query subset may not be representative — as noted in the prior sections analysis, the subset accuracy (82.0%) is higher than the full-benchmark accuracy for the same model (gpt-5.4 at 73.3%, Table 1, row l), suggesting the subset may contain disproportionately easy queries that finish well within the time budget. The full-benchmark effect of the timeout is unknown since the paper does not run the 3600-second condition on the full 830-query set.

Mitigation status. The paper does not analyze query-level time-budget effects. The Section 9 limitation statement acknowledges that "queries with different levels of complexity may require different amounts of research time" and that "a fixed time budget may therefore fail to balance effectiveness, latency, and monetary cost, leading either to insufficient exploration for difficult queries or unnecessary computation for simpler ones." This is a candid self-assessment, but the paper provides no experimental characterization of this failure mode, no measurement of what fraction of queries time out, and no proposal for adaptive time budgets (e.g., based on difficulty estimation, analogous to the compute-optimal allocation in the reference example paper). A practitioner deploying PI-SERINI in a setting where query difficulty varies widely would need to determine their own per-query timeout without guidance from this paper on how to set it.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new retrieval model, a new training objective, or even a new agent architecture in the conventional sense. Its contribution is methodological hygiene applied to search agent evaluation — and that makes it a corrective rather than a breakthrough. But in a field where retriever choice has been treated as the dominant factor in deep research system design, a well-executed corrective can redirect substantial research investment.

The paper's central reframing is to convert the question "Is BM25 sufficient for deep research?" from an absolute capability question (which the field had implicitly answered "no" based on released baselines) into a configuration and depth question (which the paper answers "yes, under specific conditions"). The 18.0 percentage point accuracy improvement from BM25 tuning alone (Table 3, 64.0% → 82.0% on the 100-query subset) and the 25.3 percentage point surfaced recall improvement from increasing retrieval depth from 5 to 1000 (Figure 2) are individually larger than many reported gains from switching to dense retrievers in prior work — yet they require no model serving, no embedding computation, and no training data. This shifts the burden of proof: any future paper claiming that a neural retriever outperforms BM25 for deep research must demonstrate that the BM25 baseline was properly configured for the corpus and used at retrieval depths comparable to the neural system. A baseline with default parameters and shallow depth is no longer admissible.

The paper also provides the field with a diagnostic framework that distinguishes retrieval failures from agent inspection failures. The four-tier trajectory logging (surfaced, previewed, opened, cited) is not just instrumentation for this paper — it is a generalizable methodology for any search agent evaluation. When the paper reports that gpt-5.5 achieves 94.7% surfaced evidence recall but only 58.9% behavior recall (Table 1, row m), it identifies a 35.8 percentage point gap that is entirely attributable to agent browsing and evaluation decisions, not retrieval quality. This decomposition was invisible in prior work, which treated the search agent as a black box producing an answer. Making it visible reframes the research agenda: the bottleneck is not finding evidence, but getting the agent to inspect and use evidence it already has. This suggests that the field's heavy investment in better retrievers (reasoning-aware retrievers, instruction-tuned embeddings, multi-vector representations) may be addressing a problem that is already largely solved for the deep research setting, while underinvesting in agent-side improvements (browsing strategies, evidence evaluation, context allocation).

The paper also reconciles contradictory signals in the literature about BM25's capabilities. Chen et al. (2025) reported gpt-5 + BM25 achieving 58.3% on BrowseComp-Plus while gpt-5 + qwen3-embed-8b achieved 73.0% — a 14.7 percentage point gap suggesting clear dense retriever superiority. This paper shows that the same gpt-5 + BM25 pairing, when properly configured, achieves 74.6% — slightly exceeding the dense retriever baseline (Table 1, rows c, d, i). The contradiction is resolved not by arguing that one result is wrong, but by identifying the specific configuration differences (BM25 parameters, retrieval depth, tool interface) that produced the gap. This is a methodological resolution, not an empirical one: the earlier finding was correct for the configuration tested, but that configuration was not representative of what BM25 can achieve.

Which research directions become more attractive after this paper?

• Agent browsing and evidence inspection strategies become the highest-priority area, since the surfaced-to-behavior recall gap (35.8 points for gpt-5.5) represents the largest known source of lost performance in deep research systems. Improving browsing strategies has higher potential marginal return than further improving retrieval recall from 94.7%.

• Retrieval depth as a first-class design parameter — not just for BM25 but for any retriever — becomes an essential consideration in search agent architecture. The paper demonstrates that shallow retrieval (k=5) discards information even when the retriever correctly ranks it, and that deep caching with paginated browsing can recover it.

• Cost-efficient agent architectures enabled by prefix caching, time-budget steering, and lexical retrieval become an alternative path to competitive deep research performance without the infrastructure complexity of dense retrieval.

Which directions become less attractive?

• Incremental retriever improvements for deep research marginally lose priority. If a tuned BM25 with deep retrieval can surface 94.7% of evidence, the ceiling for retriever improvement is at most 5.3 percentage points on surfaced recall — and that ceiling can only be reached if the agent actually inspects and uses the additional evidence, which the behavior recall numbers suggest it currently does not. Investing in better retrievers without simultaneously improving agent inspection behavior is like widening a pipe whose downstream end is already clogged.

• Reasoning-aware retrieval for BM25-style systems becomes less urgent if the bottleneck is not semantic matching but agent browsing. The paper's analysis suggests that even when BM25 surfaces relevant documents, the agent fails to inspect them — adding reasoning capabilities to the retriever would not fix this unless it also changes how the agent interacts with results.

Follow-Up Research This Work Enables

1. Run PI-SERINI with a dense retriever at matched retrieval depth to isolate retriever type from architecture. This is the single most important missing experiment in the paper. The cleanest follow-up would replicate the PI-SERINI architecture (three-tool API, 1000-document retrieval depth, time-budget steering, prefix caching) with a dense retriever backend — qwen3-embed-8b is the natural choice since it is used in the Chen et al. (2025) and Meng et al. (2026) baselines — and run the full 830-query BrowseComp-Plus benchmark. The comparison would answer: does BM25's high surfaced recall (94.7%) hold up against a dense retriever also given 1000-document depth? Does the dense retriever place relevant documents higher in the ranking (thus improving previewed recall by making evidence more likely to appear in the pages the agent actually browses)? Does the answer accuracy improve, and if so, is it because of better ranking or because the agent's browsing strategy interacts differently with the ranking? This experiment would directly isolate retriever quality from architecture, resolving the central confound that the current paper cannot address. A strong version of this follow-up would also test whether the agent prompt needs modification for dense retrieval (e.g., the "prefer short lexical searches" instruction may be suboptimal when the retriever handles natural language queries well).

2. Characterize and improve agent browsing behavior to close the surfaced-to-previewed recall gap. The paper identifies that previewed recall saturates at ~74.7% while surfaced recall reaches 95.8% (Figure 2), meaning agents leave roughly 21% of available evidence unexamined. A direct follow-up would instrument where in the cached ranking surfaced-but-not-previewed evidence documents appear, then design interventions: (a) adaptive initial page size — if the first search returns ranks 1–20 instead of 1–5, does previewed recall improve without degrading agent performance? (b) relevance-based re-ranking within the cache — can a lightweight reranker (even BM25 score-based re-ordering) move evidence into the pages the agent is likely to browse? (c) agent prompt modifications that explicitly encourage deeper browsing ("if the top-10 results are unhelpful, browse to at least rank 50 before reformulating"). The paper's trajectory logging makes all of these experiments tractable: each intervention's effect on previewed recall, behavior recall, tool call count, and cost can be measured directly. A negative result — showing that no browsing intervention improves previewed recall beyond 75% — would be equally valuable, suggesting that the bottleneck is not browsing depth but the agent's ability to recognize relevant excerpts, which is a fundamentally different research problem.

3. Test whether BM25 tuning transfers across corpora with different document length distributions. The paper's BM25 parameters (k1=25, b=1) were optimized for BrowseComp-Plus's long-document distribution. A direct transferability study would take the same PI-SERINI architecture and tuned BM25 parameters to other deep research benchmarks — BrowseComp (open-web browsing, likely different document lengths), BrowseComp-ZH (Chinese, different tokenization and morphology), or domain-specific corpora (legal documents, scientific papers) — and measure whether the tuned parameters remain effective or whether corpus-specific re-tuning is required. The experiment would also characterize the cost of re-tuning: how many queries and parameter configurations are needed to find a good setting for a new corpus? Is there a predictable relationship between corpus-level statistics (average document length, vocabulary size, term frequency distributions) and optimal BM25 parameters that could guide practitioners without requiring full grid search? A finding that tuned parameters transfer poorly would qualify the paper's practical deployability claim; a finding that they transfer well would substantially strengthen the paper's argument that BM25 can be a "drop-in" lexical retriever for deep research across settings.

4. Measure the interaction between time budget and query difficulty to develop adaptive budget allocation. The paper uses a fixed 300-second timeout for all queries and acknowledges (Section 9) that "queries with different levels of complexity may require different amounts of research time." A follow-up would first characterize BrowseComp-Plus queries by difficulty — using a proxy such as the number of evidence documents, the number of tool calls required to reach the correct answer, or the surfaced recall achieved within the first 60 seconds — then evaluate whether PI-SERINI's accuracy on the hardest quartile of queries degrades disproportionately under the 300-second budget compared to a 600-second or unbounded budget. If so, an adaptive policy could allocate more time to queries estimated as hard (e.g., those where the agent has issued many search calls without finding strong evidence). This directly parallels the compute-optimal test-time scaling framework from the reference example paper: just as that work used difficulty estimation to allocate inference compute across strategies, this follow-up would use difficulty estimation to allocate wall-clock time across queries. The paper's trajectory logging (tool calls over time) provides the raw material for building such a difficulty estimator without requiring additional benchmark infrastructure.

5. Combine PI-SERINI's deep caching architecture with a lightweight neural reranker to improve the ranking without full dense retrieval. The paper demonstrates that deep BM25 retrieval can surface 94.7% of evidence, but the agent's previewed recall is only 73.6% — meaning relevant documents are in the cache but the agent doesn't see them because they rank too low. A neural reranker applied to the cached top-1000 (not as a first-stage retriever, but as a second-stage refinement) could push relevant documents from rank 500 to rank 5, making them more likely to be inspected without requiring the infrastructure of a full dense retrieval pipeline. This hybrid approach — cheap lexical first-stage retrieval at high depth, cheap neural second-stage reranking — would test whether the paper's finding about BM25 sufficiency extends to a setting where BM25 is augmented with minimal neural components. The key metric would be previewed recall: does reranking the cached BM25 results increase the fraction of evidence the agent actually browses? If previewed recall improves substantially, the implication is that BM25's ranking quality (not just its recall ceiling) matters for end-to-end performance, and that a lightweight reranker provides most of the benefit of a full dense retriever at lower cost.

Practical Applications and Downstream Use Cases

1. Cost-constrained deep research deployments where GPU serving for dense retrieval is infeasible. Organizations running deep research systems on CPU-only infrastructure, in edge deployments, or under tight cost constraints can adopt the PI-SERINI architecture with tuned BM25 immediately. The 4.2× cost reduction from the released gpt-5 + BM25 baseline ($400.4, Table 1, row c) to PI-SERINI with the same LLM ($94.9, row i) — achieved without changing the retriever type — translates to real operational savings at scale. For a deployment processing 10,000 queries per day, this is the difference between $4,000/day and$949/day in inference costs alone, not counting the eliminated cost of embedding model serving. The DeepSeek-based PI-SERINI configurations ($28.9–$55.1 per full benchmark run, rows n–o) further demonstrate that capable but cost-efficient LLMs paired with BM25 can achieve competitive accuracy (68.1–71.4%) at an order of magnitude lower cost than frontier-model dense-retriever systems. For startups or research labs where compute budgets are the binding constraint, this architecture provides a path to meaningful deep research capability without the infrastructure overhead of dense retrieval.

2. Rapid prototyping and ablation studies for search agent research. The paper frames cost reduction as enabling more practical deep research evaluation: "By lowering the cost of each full benchmark run, PI-SERINI makes ablations and future studies of deep research more practical" (Section 5). This is not merely a rhetorical point — the Meng et al. (2026) experiment cost $1,000–$2,000 for a single run, making systematic ablation studies prohibitively expensive for most academic labs. PI-SERINI reduces a full BrowseComp-Plus run to $29–$292 (Table 1), meaning that a 10-condition ablation (varying prompts, tool configurations, time budgets) costs $290–$2,920 rather than $10,000–$20,000. This directly enables the kind of experimental thoroughness — testing multiple BM25 parameter configurations, comparing termination policies, measuring agent behavior at different retrieval depths — that the paper itself demonstrates. For researchers who want to study agent–retriever interaction but cannot afford the compute budgets of industrial labs, PI-SERINI's cost profile makes BrowseComp-Plus a tractable benchmark for the first time.

3. Self-hosted or air-gapped deep research systems for sensitive corpora. Organizations operating on classified, proprietary, or privacy-sensitive document collections cannot use cloud-hosted dense retrieval APIs, and deploying self-hosted dense retrievers requires maintaining embedding model infrastructure, vector databases, and GPU serving. BM25 via ANSERINI or Lucene runs entirely on CPU, requires no model serving, and can be deployed as a single binary. The paper shows that with tuned parameters and sufficient retrieval depth, this minimal infrastructure can surface evidence at 94.7% recall. For intelligence analysis, legal discovery, or internal corporate research where the corpus is large but static and queries are complex, the PI-SERINI architecture provides a path to deep research capability that can be deployed entirely within a secured environment. The only external dependency is the LLM API (or a self-hosted LLM), and the BM25 backend can be updated incrementally as documents are added without retraining.

4. Educational and training environments for search agent design. The paper's explicit documentation of its system prompt, tool API, retrieval configuration, and time-budget policy (Appendix A) — combined with the open-source code release — makes PI-SERINI a reproducible starting point for teaching search agent design. Students or practitioners can start with the paper's configuration, reproduce the Table 1 numbers on BrowseComp-Plus (at a cost of $29–$292 per run), then systematically modify components: swap BM25 for a dense retriever, change the tool interface, modify the agent prompt, adjust the time budget. The four-tier trajectory logging provides immediate feedback on whether a modification improved retrieval, improved inspection, or improved synthesis. This makes the paper effective as a curriculum component — not just as a result to study, but as a system to modify and extend.

When to Prefer This Method

The paper does not articulate an explicit decision rule for when to prefer lexical retrieval with deep caching over dense retrieval. Instead, its conclusions are carefully bounded: BM25 can suffice under specific conditions (well-configured, sufficient depth, capable LLM in an agentic loop), but "the lesson is not that BM25 is enough everywhere" (Section 8). The paper's results suggest — but the paper itself does not claim — the following decision heuristics, which should be understood as reader inferences rather than paper-endorsed guidance:

• Prefer BM25 with deep caching and the PI-SERINI three-tool interface when the corpus has long documents where term frequency continues to provide ranking signal (BM25 tuning improves surfaced recall from 84.6% to 95.7%, Table 3), when cost or infrastructure constraints make dense retrieval impractical (4.2× cost reduction vs. the shallow BM25 baseline, Table 1, rows c vs. i), and when the LLM is sufficiently capable to navigate cached rankings and formulate effective lexical queries (accuracy varies from 54.8% to 83.1% across LLMs with the same retriever, Table 1).

• Dense retrieval with deep caching and the same interface remains an untested counterfactual. The paper never runs this experiment, so there is no evidence-based reason to prefer BM25 over a dense retriever operating at the same retrieval depth with the same tool interface. The paper's results only support preferring PI-SERINI + BM25 over shallow dense-retriever baselines with different agent architectures — a comparison that confounds retriever type with retrieval depth and interface design.

• For hard queries where the agent struggles to find evidence, the paper provides no guidance on whether BM25 with increased retrieval depth or a dense retriever with better semantic matching would be more effective, because the paper does not characterize query-level difficulty or measure where in the ranking missed evidence documents appear.