Optimizing Test-Time Query Representations for Dense Retrieval

ArXiv: 2205.12680

🎯 Pitch

Dense retrieval encoders often misrepresent unseen test queries, hurting recall. TOUR fixes this by treating each query as its own optimization problem at test time—iteratively refining the query vector using signals from a re-ranker, yielding up to 10.7% QA accuracy gains and running 1.3–2.4× faster than direct re-ranking.

1. Executive Summary

This paper introduces TOUR (Test-Time Optimization of Query Representations), which optimizes instance-level query representations at test time for dense retrieval by iteratively updating each test query's vector using gradient descent guided by a cross-encoder re-ranker's fine-grained pseudo labels on top-k retrieval results. Evaluated on phrase retrieval with DensePhrases and passage retrieval with DPR across multiple open-domain QA benchmarks (Natural Questions, TriviaQA, WebQuestions, CuratedTrec, SQuAD, EntityQuestions), TOUR provides two instantiations — TOURhard (selecting pseudo-positive contexts as hard binary labels via thresholded cumulative probability) and TOURsoft (using the re-ranker's normalized scores as soft continuous labels via KL divergence minimization) — and is theoretically shown to generalize the classical Rocchio algorithm for pseudo relevance feedback, with the key difference that each pseudo-positive context receives a weight proportional to how much its retrieval ranking needs improvement rather than equal treatment. TOUR improves end-to-end open-domain QA accuracy by up to 10.7% and passage retrieval accuracy by up to 8.3% over baseline retrievers, while running 1.3–2.4× faster than direct re-ranking with a larger top-k (e.g., TOURhard with top-10 outperforms the re-ranker with top-40 by 1.0% while being 2.5× faster), establishing that test-time instance-level query optimization efficiently substitutes for expensive re-ranking only when the re-ranker can provide sufficiently discriminative signals on a small number of initial retrieval candidates.

2. Context and Motivation

The Core Problem: Trained Query Encoders Fail on Unseen Queries

The fundamental problem this paper addresses is straightforward yet pervasive: pre-trained query encoders in dense retrieval systems often fail to retrieve relevant contexts when applied to test queries, especially when those test queries differ from the training distribution. Despite the success of dense retrieval in overcoming the term mismatch problem that plagues sparse methods like BM25, the quality of retrieval depends entirely on the quality of the query representation produced by a fixed encoder at inference time. The paper states this plainly in Section 1:

"However, directly using trained query encoders often fails to retrieve the relevant context as many test queries are unseen during training."

This is not merely an out-of-distribution problem confined to domain transfer — it occurs even when the test queries are drawn from the same datasets used for training, because individual test queries may express information needs in ways that the encoder's training process did not adequately capture. A query encoder trained via contrastive learning on millions of query-context pairs learns a mapping that works well on average across the training distribution, but it has no mechanism to adapt to the specific phrasing or semantic nuances of a particular test query at inference time.

The practical manifestation of this failure is measurable and severe. Table 2 in the paper shows that DensePhrases trained only on Natural Questions achieves just 15.4% end-to-end QA accuracy on SQuAD and 33.4% on TriviaQA — both substantial drops from the in-domain performance of a model trained on all datasets (34.5% and 56.3% respectively, from Table 1). Table 3 shows that even the multi-dataset trained DPR achieves only 57.9% top-20 passage retrieval accuracy on EntityQuestions — a dataset specifically designed to challenge dense retrievers with "simple entity-centric questions" that are structurally different from typical open-domain QA training data. These are not edge cases; they represent the reality that test queries in the wild rarely conform neatly to the training distribution.

Why This Problem Matters: Practical and Theoretical Significance

The practical significance of this problem is immediate. Dense retrieval underpins virtually all modern open-domain question answering systems, enterprise search engines, and retrieval-augmented generation pipelines. In these deployments, the cost structure is asymmetric: context representations for the entire corpus (often millions of documents or phrases) are pre-computed once and stored, while query representations are computed on-the-fly for each incoming user query. The fixed cost of building the index is amortized over all queries, but the variable cost — and the quality bottleneck — lies entirely in how well the query encoder represents each individual test query.

If query representations are suboptimal, downstream components (readers, generators, re-rankers) inherit degraded input, and no amount of post-retrieval sophistication can fully recover missed relevant contexts that were never retrieved in the first place. The paper's focus on test-time optimization is therefore addressing the most fragile link in the dense retrieval pipeline: the point where a trained model meets a new query it has never seen.

The theoretical significance is equally compelling. The paper frames the problem as one of instance-level optimization — treating each test query as its own optimization problem rather than applying a one-size-fits-all encoder. This represents a departure from the dominant paradigm in neural information retrieval, where model parameters are learned during training and then frozen at inference. By introducing gradient-based optimization at test time for individual queries, the paper opens a conceptual bridge between classical information retrieval techniques (which have always allowed per-query adaptation through relevance feedback) and modern dense retrieval (which has largely abandoned query-side adaptation in favor of larger encoders and more training data). The connection to the 50-year-old Rocchio algorithm (Rocchio, 1971) — made explicit in Section 3.2 and Appendix C — is not merely a citation; it demonstrates that the proposed method is a principled generalization of a well-understood technique, adapted for the neural era.

Where Prior Approaches Fall Short

The paper identifies a landscape of existing solutions and systematically explains why each is insufficient:

1. Training larger or better encoders is a partial fix. The dominant approach to improving retrieval quality has been to scale up: larger pre-trained language models as query and context encoders (Karpukhin et al., 2020; Lee et al., 2021a), better contrastive learning objectives (Xiong et al., 2020), and more extensive training data. While effective in aggregate, this approach does not solve the instance-level problem. A larger encoder still produces a fixed representation for a given test query; it may be a better fixed representation on average, but it remains incapable of adapting to the specific retrieval landscape that query encounters. The paper's results bear this out: even the multi-dataset-trained DensePhrases model (Table 1) leaves substantial room for improvement on in-domain datasets (e.g., 41.6% on NQ, 34.5% on SQuAD), and more dramatically on out-of-domain queries (Table 2).

2. Query-side fine-tuning over training data helps domain transfer but not instance-level adaptation. Lee et al. (2021a) proposed fine-tuning the query encoder on retrieval results from training queries in a new domain, formulated as maximizing the marginal likelihood of relevant contexts in top-k results (Equation 3). This is effective for domain adaptation — adjusting the encoder's parameters so that its representations work better for a new query distribution. However, it operates at the distribution level, not the instance level. Once fine-tuning is complete, all test queries share the same frozen encoder. A test query that is atypical even within the target distribution receives the same encoder as a typical one. The paper positions TOUR as a natural next step: instead of fine-tuning the encoder's parameters using training queries, directly optimize the representation of each test query.

3. Re-ranking with cross-encoders is accurate but computationally expensive. Cross-encoder re-rankers (Nogueira and Cho, 2019; Fajcik et al., 2021) process query-context pairs jointly through a transformer, producing highly accurate relevance judgments by attending to fine-grained textual interactions that dual encoders miss. They are known to generalize better to unseen distributions compared to dual encoders (Rosa et al., 2022). However, re-rankers have a fundamental limitation: they can only rerank what has already been retrieved. If a relevant context is not in the top-k retrieval results, the re-ranker never sees it and cannot recover it. This is why Table 3 shows that re-ranking improves Acc@20 but never improves Acc@100 — it rearranges the order within the retrieved set but does not pull in new relevant documents from outside that set. Moreover, re-rankers scale poorly: to achieve good coverage, they require a large top-k, which means running the expensive cross-encoder on many candidates. Figure 2 shows that the re-ranker with top-40 runs 2.5× slower than TOUR with top-10 while achieving worse performance.

The paper explicitly contrasts TOUR with the re-ranker's role: the re-ranker provides signals (pseudo labels) that guide the query representation to retrieve new relevant contexts that were previously missed. TOUR uses the cross-encoder not as a final gatekeeper but as an oracle that teaches the query vector where to look:

"We leverage a cross-encoder re-ranker to provide fine-grained pseudo labels over retrieval results and iteratively optimize query representations with gradient descent."

This fundamentally changes the cross-encoder's role from selector (picking the best among already-retrieved candidates) to teacher (guiding the query representation toward unexplored regions of the embedding space where relevant contexts reside).

4. Traditional pseudo relevance feedback (PRF) is too coarse-grained for neural retrieval. PRF techniques (Rocchio, 1971; Lavrenko and Croft, 2001) share TOUR's core motivation: refine a query representation for a single test query using feedback from initial retrieval results. However, the classical Rocchio algorithm (Equation 6) makes a crude assumption: all top-k′ results are treated as equally relevant, and all top-(k−k′) results are treated as equally non-relevant. This works adequately for sparse retrieval where term overlap provides some signal, but it ignores the nuanced relevance gradations that a neural model can detect. Two contexts that both appear in the top-10 may have very different actual relevance; treating them identically discards information that could be used to make finer-grained updates.

The paper demonstrates this empirically through ablation (Table 5): replacing TOUR's cross-encoder-guided pseudo-positive selection (Cq_hard) with the classical uniform top-k′ selection (k′=3) reduces performance from 48.4% to 46.1% on Natural Questions. Figure 3 shows that PRF using the Rocchio algorithm gains almost nothing from additional iterations beyond t=1, while TOUR continues to improve through t=3 — evidence that the coarse-grained update rule saturates quickly because it cannot distinguish between genuinely helpful and only coincidentally top-ranked contexts.

5. Distillation-based approaches require full training set access at fine-tuning time. Methods like RocketQAv2 (Ren et al., 2021) and the distilled reader-to-retriever approach (Izacard and Grave, 2020) use cross-encoder signals to fine-tune the entire retriever on training data. This is powerful but requires a complete training pipeline with access to all training queries and a full re-ranking pass over the corpus. TOUR operates with a fundamentally different assumption: only the test query and its initially retrieved results are available at optimization time. This makes TOUR deployable in scenarios where re-training the retriever is infeasible — for instance, when the index is pre-built and immutable, or when the test queries arrive one at a time in a streaming fashion.

How This Paper Positions Itself

The paper situates TOUR at the intersection of three research threads: dense retrieval (which provides the base encoders and index), cross-encoder re-ranking (which provides the pseudo label signal), and pseudo relevance feedback (which provides the conceptual framework of per-query representation updates). The novelty is not in any one of these components individually but in their integration into an instance-level, gradient-based optimization framework that is both principled (with a theoretical connection to the Rocchio algorithm) and practical (with efficient implementation techniques that make it faster than direct re-ranking).

The paper's positioning is explicitly complementary rather than competitive with existing approaches. TOUR is designed to work with any pre-trained dual encoder retriever and any cross-encoder re-ranker — it takes them as fixed components and uses them to improve retrieval at test time. This modular design means that improvements to either the retriever or the re-ranker automatically benefit TOUR. The paper demonstrates this modularity by applying TOUR to two different retrieval paradigms (phrase retrieval with DensePhrases and passage retrieval with DPR) and two different re-ranker types (a custom phrase re-ranker developed in Section 3.4 and an off-the-shelf passage re-ranker from Fajcik et al., 2021).

Importantly, the paper positions TOUR not as a replacement for re-ranking but as a more efficient way to achieve what re-ranking with a large top-k accomplishes — and more, since TOUR can retrieve new relevant contexts that the initial retrieval missed, which re-ranking alone cannot do. The latency vs. performance trade-off in Figure 2 makes this positioning concrete: TOUR beats the re-ranker's best performance while running substantially faster, because it needs to process fewer candidates through the expensive cross-encoder.

The paper also positions its contribution relative to the train-test overlap concern raised by Lewis et al. (2021). Section 5.1 shows that TOUR's improvements are not merely an artifact of overlapping training and test data — it improves performance on the "None overlap" split (queries without any query or answer overlap with training data), demonstrating genuine generalization beyond memorized patterns.

3. Technical Approach

3.1 Reader Orientation

This paper proposes TOUR, a system that treats each test query as its own optimization problem, iteratively refining the query's dense vector representation using gradient descent guided by pseudo relevance labels from a cross-encoder re-ranker. The problem TOUR solves is that frozen query encoders produce suboptimal representations for individual test queries — especially those unseen during training or from different distributions — and the solution is an instance-level optimization loop where the query vector is updated to move closer to relevant contexts and away from non-relevant ones, where "relevant" is judged by the re-ranker on the fly rather than by ground-truth labels.

3.2 Big-Picture Architecture (Diagram in Words)

The TOUR system has five interconnected components:

1. Pre-trained dense retriever (e.g., DensePhrases or DPR): Provides the initial query encoder Eq(·) that maps a test query q to its starting vector q0, and the pre-computed context index over which retrieval is performed. The context encoder Ec(·) is frozen and all context vectors are pre-computed and indexed.

2. Cross-encoder re-ranker ϕ(·): A fine-tuned language model (e.g., RoBERTa-large) that takes a query-context pair as input and produces a scalar relevance score s = ϕ(q, c). This serves as the pseudo-labeling oracle — it provides the signal that guides query vector updates, replacing the ground-truth labels that are unavailable at test time.

3. Query optimizer: The iterative gradient-based update loop that takes the current query vector qt, retrieves top-k contexts using it, obtains re-ranker scores on those contexts, computes a loss based on pseudo labels, and updates qt via gradient descent to produce qt+1.

4. Pseudo label generator: A decision procedure that converts re-ranker scores into training signals. In TOURhard, this selects a subset of contexts as pseudo-positive via cumulative probability thresholding (Equation 8). In TOURsoft, this normalizes re-ranker scores into a probability distribution over the top-k (also Equation 8) and uses that directly as soft targets.

5. Score aggregator: At the final iteration (or after early stopping), the system linearly interpolates the dense retrieval similarity score with the re-ranker score for the final ranking: λ · si + (1 − λ) · sim(q, ci).

Information flows as follows: test query enters → initial encoder produces q0 → retriever returns top-k contexts → re-ranker scores them → pseudo labels are generated from scores → gradient descent updates q0 to q1 → retriever returns new top-k using q1 → process repeats up to 3 iterations (or stops early) → final ranking produced by score aggregation.

3.3 Roadmap for the Deep Dive

• First, the gradient descent optimization loop (Section 3.1): the core mechanism — how a query vector is iteratively updated, what retrieval and re-ranking happen at each step, and why this differs fundamentally from both distribution-level fine-tuning and classical PRF.
• Second, TOURhard (Section 3.2): the hard-label instantiation, including the pseudo-positive selection procedure (Equation 8), the marginal likelihood objective (Equation 9), and the gradient derivation that reveals the connection to the Rocchio algorithm (Equation 11) — this is essential because the Rocchio connection is the paper's main theoretical contribution.
• Third, TOURsoft (Section 3.3): the soft-label instantiation, the KL divergence objective (Equation 12), and its gradient (Equation 13) — understanding this contrast with TOURhard clarifies when fine-grained continuous signals matter.
• Fourth, the phrase re-ranker (Section 3.4): a new component developed specifically for phrase retrieval, including training data construction, architecture, pre-training strategy, and score aggregation — this is necessary because no off-the-shelf phrase re-ranker existed.
• Fifth, efficient implementation (Section 3.5): the early stopping conditions, caching strategy, and iteration budget that make TOUR faster than direct re-ranking — this is critical for understanding the paper's latency-vs-performance claims.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method paper whose core idea is that test-time gradient-based optimization of individual query representations, guided by a cross-encoder re-ranker's pseudo labels on intermediate retrieval results, can substantially improve dense retrieval quality while being more computationally efficient than applying the re-ranker directly over a larger candidate set.

The Optimization Loop: Instance-Level Query Refinement

The central mechanism of TOUR is an iterative optimization loop that treats a single test query q as its own training example and optimizes its representation q directly in the embedding space, rather than updating the parameters of the query encoder Eq(·). This is the key distinction from query-side fine-tuning (Lee et al., 2021a): query-side fine-tuning updates the encoder's weights so that future queries produce better representations; TOUR updates the representation itself for just this one query.

Initialization and iteration structure. Given a test query q ∈ Qtest, the initial query vector is obtained by passing q through the frozen query encoder:

$q_0 = E_q(q)$

where E_q is the pre-trained query encoder (e.g., the DensePhrases or DPR query encoder, fine-tuned via contrastive learning but now frozen).

At each iteration t (starting from t = 0), the following steps execute:

1. Retrieve top-k contexts using the current query vector qt. The retriever searches the pre-computed context index and returns the top-k contexts ranked by inner product similarity:

$C^{q_t}_{1:k} = [c_1, \ldots, c_k] = \text{top-k}_{c \in C} \; \text{sim}(q_t, c)$

where sim(q, c) = q^\top c is the inner product between the query and context vectors, and the contexts are sorted in descending order: sim(q_t, c_1) ≥ ··· ≥ sim(q_t, c_k).

What this step computes: a list of the k contexts from the entire corpus whose pre-computed vectors have the highest dot product with the current query vector. This is the standard dense retrieval operation; the set C^{q_t}_{1:k} represents the model's current best guess at which contexts are relevant to the query.

Why this form: inner product retrieval with pre-computed context vectors is the standard dual-encoder inference procedure, enabling sub-linear search via maximum inner product search (MIPS) over an index built once from the frozen context encoder. TOUR inherits this infrastructure unchanged — it only modifies the query vector that is used to query this index.

2. Score the retrieved contexts with the cross-encoder re-ranker ϕ. For each ci ∈ C^{q_t}_{1:k}, the re-ranker produces a scalar relevance score:

$s_i = ϕ(q, c_i)$

where ϕ(q, c_i) is typically computed by passing the concatenation of query and context through a pre-trained language model and extracting a scalar from the [CLS] token representation (Equation 14). These scores s_i represent the re-ranker's fine-grained judgment of how relevant each context is to the query, leveraging full cross-attention between query and context tokens that dual encoders cannot perform.

What this step computes: a relevance score for each of the k retrieved contexts, produced by a model that jointly encodes the query and context, attending to token-level interactions that the dual encoder's independent encoding missed. The result is a set of k scores {s_1, ..., s_k} that are typically more accurate than the retrieval similarity scores {sim(q_t, c_1), ..., sim(q_t, c_k)}.

Why this form: cross-encoders are known to produce more accurate relevance judgments than dual encoders because they can model exact term matches, paraphrases, negation, and other fine-grained linguistic phenomena through cross-attention (Nogueira and Cho, 2019). The paper leverages this accuracy not for final ranking but as a teaching signal — the re-ranker tells the query vector which contexts are actually relevant, and the query vector updates to retrieve more like them.

3. Generate pseudo labels from the re-ranker scores by converting them into a training signal. In TOURhard, this means selecting a subset C^q_hard ⊂ C^{q_t}_{1:k} of pseudo-positive contexts (detailed in §3.4.2). In TOURsoft, this means normalizing the scores into a probability distribution P(c_i | q_t, ϕ) over the top-k (detailed in §3.4.3). In both variants, the pseudo labels serve as the optimization target — the query vector will be updated to make the retriever's ranking closer to what the re-ranker "believes" is correct.

4. Compute the loss L that measures the discrepancy between the current retrieval ranking (under the query vector qt) and the pseudo labels (from the re-ranker). The specific loss function differs between TOURhard (marginal likelihood, Equation 9) and TOURsoft (KL divergence, Equation 12).

5. Update the query vector via a single gradient descent step:

$q_{t+1} \leftarrow q_t - \eta \frac{\partial \mathcal{L}(q_t, C^{q_t}_{1:k})}{\partial q_t}$

where η is the learning rate. The gradient flows through the retrieval scores (i.e., the inner products q_t^\top c_i) but the context vectors c_i themselves are treated as fixed — gradient computation only updates q_t, not the context embeddings.

What this step computes: a new query vector q_{t+1} that is the result of moving q_t in the direction that reduces the loss — that is, in the direction that makes the retriever's ranking (under the new query vector) more consistent with the re-ranker's pseudo labels. The step size is controlled by η.

Why this form: gradient descent on the query vector directly optimizes the retrieval objective for this specific query. Unlike updating the encoder parameters (which would affect all queries and require careful regularization to avoid catastrophic forgetting), updating only q_t is query-specific and carries no risk of degrading other queries' representations. The gradient naturally points q_t toward the pseudo-positive context vectors and away from the others, weighted by how much each context's ranking needs to change (as the gradient analysis in Equation 11 reveals).

6. Repeat steps 1–5 with the updated query vector q_{t+1} to obtain new retrieval results C^{q_{t+1}}_{1:k}. These new results may include contexts that were not in the previous top-k because the query vector has moved in the embedding space, potentially "discovering" relevant contexts that the initial query vector missed.

This iterative refinement is the key mechanism that differentiates TOUR from pure re-ranking: re-ranking only reorders the initially retrieved set, while TOUR can pull in new relevant contexts from outside the initial top-k because the query vector itself changes, enabling it to retrieve different contexts in subsequent iterations. Figure 5 in the paper provides a concrete example: at iteration t = 0, the correct answer is not in the top-10; at t = 1, three new contexts appear including the correct one; by t = 3, the correct answer is ranked first.

Optimization details. The paper uses gradient descent with momentum set to 0.99 and weight decay λ_decay = 0.01 (from Appendix D). Weight decay effectively multiplies qt by (1 - ηλ_decay) at each step, which — as noted in the Rocchio connection (§3.2) — dynamically sets the α parameter in the Rocchio analog, controlling how much of the original query representation is retained. A linear learning rate scheduling is applied per iteration. For open-domain QA with DensePhrases, the learning rate η = 1.2; for passage retrieval with DPR, η = 0.2 (Table 7). The maximum number of iterations is 3 for open-domain QA and 1 for passage retrieval, with early stopping conditions that can terminate earlier (§3.5).

What makes this different from distribution-level fine-tuning. Query-side fine-tuning (Lee et al., 2021a) optimizes the encoder parameters θ of Eq(·; θ) over all training queries by maximizing the marginal likelihood of relevant contexts (Equation 3). This changes how all future queries are encoded. TOUR, by contrast, takes the encoder as fixed and optimizes the query vector q itself, treating each test query as an independent optimization problem. The distinction is analogous to the difference between meta-learning (learning parameters that enable fast adaptation) and test-time training (adapting representations directly at test time): TOUR does not change the model; it changes the input representation for a single instance.

What makes this different from classical PRF. Classical PRF (Equation 6) updates the query vector as a fixed linear combination — α times the original query, plus β times the average of pseudo-relevant context vectors, minus γ times the average of pseudo-non-relevant context vectors. The weights are uniform within the pseudo-relevant and pseudo-non-relevant sets, and they are fixed hyperparameters. TOUR's gradient-based update, as derived in Equation 11 and Equation 13, dynamically weights each context based on two factors: (a) the re-ranker's confidence that it is relevant (via the pseudo label), and (b) how much its current retrieval ranking needs improvement (via the probability gap between its pseudo-label probability and its current retrieval probability). This dynamic weighting is what enables TOUR to benefit from multiple iterations while classical PRF saturates (Figure 3).

TOURhard: Hard Pseudo-Labels via Cumulative Thresholding

TOURhard is the first instantiation of the TOUR framework, using the cross-encoder re-ranker to make a binary hard decision about which contexts in the top-k are pseudo-relevant, and then optimizing the query vector to maximize the marginal likelihood of those pseudo-positive contexts.

Pseudo-positive selection via Nucleus Sampling-style thresholding. TOURhard does not simply treat the top-k′ contexts as relevant (which would be the classical PRF approach). Instead, it uses the re-ranker scores to select a variable-sized set C^q_hard of pseudo-positive contexts based on cumulative probability mass. The procedure works as follows:

First, the re-ranker scores {s_1, ..., s_k} are converted into a probability distribution over the top-k contexts by applying softmax with temperature τ:

$P(c_i = c^* | q, \phi) = \frac{\exp(\phi(q, c_i) / \tau)}{\sum_{j=1}^k \exp(\phi(q, c_j) / \tau)}$

where P(c_i = c^* | q, ϕ) is the re-ranker's estimated probability that context c_i is the truly relevant context (c*) for query q, ϕ(q, c_i) is the raw re-ranker score, τ is a temperature parameter (set to 0.5 in all experiments per Appendix D), and k is the number of retrieved contexts.

What this equation computes: a normalized probability distribution over the top-k retrieved contexts, where each context's probability represents the re-ranker's confidence that it is the single correct context for this query. The temperature τ controls sharpness: τ < 1 sharpens the distribution (making high-scoring contexts even more probable relative to low-scoring ones), while τ > 1 flattens it. At τ = 1, this is a standard softmax. At τ = 0.5 (the paper's setting), the distribution is sharper, meaning the re-ranker's top picks dominate the probability mass more strongly.

Why this form: using a softmax over the re-ranker scores converts the raw scalar outputs (which have arbitrary scale and are not calibrated probabilities) into a proper probability distribution that sums to 1 over the top-k set. The temperature provides a knob to control how much the optimization trusts the re-ranker's top predictions versus treating candidates more uniformly. A lower temperature means the optimization will be steered primarily by the contexts the re-ranker is most confident about, reducing noise from low-confidence judgments.

Next, the set C^q_hard is defined as the smallest set of contexts whose cumulative probability under this distribution meets or exceeds a threshold p:

$C^q_{\text{hard}} = \text{the smallest set such that } \sum_{\tilde{c} \in C^q_{\text{hard}}} P(\tilde{c} = c^* | q, \phi) \geq p$

where p is the cumulative probability threshold (set to 0.5 in all experiments per Appendix D). The set is constructed by adding contexts in descending order of P(c_i = c* | q, ϕ) until the cumulative sum reaches or exceeds p. This is directly analogous to Nucleus Sampling (Holtzman et al., 2020) for text generation, but applied to the problem of selecting pseudo-positive training examples rather than selecting next tokens.

What this procedure computes: a variable-sized subset of the top-k that the re-ranker collectively considers highly likely to contain the truly relevant context. If the re-ranker is highly confident about one or two contexts (their individual probabilities exceed p), then C^q_hard will be a singleton or pair. If the re-ranker's confidence is spread across many contexts, C^q_hard will be larger. This adaptivity is purposeful: when the re-ranker has a clear signal, TOURhard focuses the query update on a small number of high-confidence contexts; when the signal is ambiguous, it casts a wider net.

Why this form over fixed top-k′ selection: classical PRF (Equation 6) uses C^q_{1:k'} — the top k′ by retrieval rank, regardless of how the re-ranker scores them. This is problematic because retrieval rank and true relevance are imperfectly correlated: a context ranked 5th by the retriever might be highly relevant (the retriever just didn't rank it highly enough), while a context ranked 1st might be judged non-relevant by the re-ranker. Using cumulative probability over re-ranker scores adapts the selection to the re-ranker's actual confidence. The ablation in Table 5 validates this choice: replacing C^q_hard with fixed top-3 selection (C_{1:3}) reduces performance from 48.4% to 46.1% on Natural Questions, demonstrating that the thresholded selection based on re-ranker scores is substantially more effective than uniform top-k′ selection.

The optimization objective. Given the selected pseudo-positive set C^q_hard, TOURhard optimizes the query vector to maximize the marginal likelihood of these contexts under the retriever's ranking distribution. The loss is defined as the negative log marginal likelihood:

$\mathcal{L}_{\text{hard}}(q_t, C^{q_t}_{1:k}) = -\log \sum_{\tilde{c} \in C^q_{\text{hard}}} P_k(\tilde{c} | q_t)$

where

$P_k(\tilde{c} | q_t) = \frac{\exp(\text{sim}(q_t, \tilde{c}))}{\sum_{i=1}^k \exp(\text{sim}(q_t, c_i))}$

is the retriever's probability assigned to context \tilde{c} within the top-k set, computed by taking a softmax over the retrieval similarity scores for the k contexts.

What this loss computes: the negative log of the total probability mass that the retriever assigns to the pseudo-positive contexts. If the pseudo-positive contexts collectively receive high probability under the retriever's ranking (i.e., they are ranked highly), the loss is low. If they are buried at the bottom of the top-k, the loss is high. Minimizing this loss pushes the query vector q_t to increase the inner product with pseudo-positive context vectors, thereby moving them up in the ranking.

Why this form: the marginal likelihood objective aggregates evidence across all pseudo-positive contexts. This is important because in open-domain QA, multiple contexts might contain the correct answer (e.g., the same fact stated in different Wikipedia articles). Maximizing the sum of probabilities across these contexts encourages the query vector to retrieve any of them highly, rather than focusing on a single one. This is the same objective used by the query-side fine-tuning approach (Equation 3), but applied at the instance level and with pseudo-labels replacing ground-truth labels. The contrast with a standard cross-entropy loss over a single positive is instructive: if the re-ranker identifies three distinct contexts as likely relevant, a single-positive loss would force the model to choose one, potentially at the expense of the others. The marginal likelihood allows the optimization to benefit from all three simultaneously.

The gradient and the connection to the Rocchio algorithm. The gradient of L_hard with respect to q_t has a specific structure that reveals the relationship to classical PRF. The paper derives this gradient in Appendix A (reproduced in the paper). Taking the gradient and rearranging yields:

$\frac{\partial \mathcal{L}_{\text{hard}}(q_t, C^{q_t}_{1:k})}{\partial q_t} = -\sum_{\tilde{c} \in C^q_{\text{hard}}} P(\tilde{c} | q_t) \cdot (1 - P_k(\tilde{c} | q_t)) \cdot \tilde{c} + \sum_{\tilde{c} \in C^q_{\text{hard}}} \left[ P(\tilde{c} | q_t) \cdot \sum_{c \in C^{q_t}_{1:k}, c \neq \tilde{c}} P_k(c | q_t) \cdot c \right]$

where P(\tilde{c} | q_t) is the re-normalized probability of \tilde{c} among only the pseudo-positive contexts:

$P(\tilde{c} | q_t) = \frac{\exp(\text{sim}(q_t, \tilde{c}))}{\sum_{\tilde{c}' \in C^q_{\text{hard}}} \exp(\text{sim}(q_t, \tilde{c}'))}$

What this expression represents: the gradient decomposes into two opposing forces. The first term (negative) pulls q_t toward each pseudo-positive context \tilde{c}, with the pull strength equal to P(\tilde{c} | q_t) · (1 − P_k(\tilde{c} | q_t)). The factor P(\tilde{c} | q_t) gives more weight to pseudo-positive contexts that already have high retrieval similarity (relative to other pseudo-positives), since these are the ones the retriever is closest to "getting right" and just needs a small nudge. The factor (1 − P_k(\tilde{c} | q_t)) gives more weight to pseudo-positive contexts that have low probability within the full top-k — that is, contexts that are currently ranked poorly and need a larger update to improve their position. The product is maximized when a pseudo-positive context is the dominant one among pseudo-positives but is buried in the overall top-k ranking.

The second term (positive) pushes q_t away from all non-pseudo-positive contexts (and, implicitly, from pseudo-positive contexts relative to each other), weight-averaged by their current retrieval probabilities. This prevents the update from simply increasing all inner products uniformly, which would not change the ranking.

Why this form matters: substituting the gradient into the gradient descent update q_{t+1} ← q_t − η · ∂L/∂q_t gives the update rule:

$g_{\text{hard}}(q_t, C^{q_t}_{1:k}) = q_t + \eta \sum_{\tilde{c}} P(\tilde{c} | q_t)(1 - P_k(\tilde{c} | q_t)) \tilde{c} - \eta \sum_{\tilde{c}} \left[ P(\tilde{c} | q_t) \sum_{c \neq \tilde{c}} P_k(c | q_t) c \right]$

This is Equation 11 in the paper. Now compare this to the classical Rocchio algorithm (Equation 6):

$g_{\text{Rocchio}}(q_t, C^{q_t}_{1:k}) = q_t + \beta \frac{1}{k'} \sum_{i=1}^{k'} c_i - \gamma \frac{1}{k - k'} \sum_{i=k'+1}^k c_i$

The paper proves in Appendix C that these two update rules are equivalent when: (1) C^q_hard = C^q_{1:k'} (pseudo-positives are exactly the top-k′ by retrieval rank), (2) all k contexts have equal retrieval probability P_k(c | q_t) = 1/k, and (3) weights are set as β = γ = η(k − k′)/k. Under these conditions, the TOURhard gradient update reduces to exactly the Rocchio update.

In practice, these conditions never hold — the re-ranker selects C^q_hard differently from C^q_{1:k'}, the retrieval probabilities are not uniform, and the per-context weights are dynamic rather than fixed. TOURhard is thus a strict generalization of the Rocchio algorithm: it recovers Rocchio as a special case under the coarsest possible assumptions, but in realistic settings it applies instance-specific, dynamically-weighted updates that are informed by the re-ranker's fine-grained confidence rather than uniform treatment.

The intuitive meaning of the per-context weight. The factor P(\tilde{c} | q_t)(1 − P_k(\tilde{c} | q_t)) has an operational interpretation: a pseudo-positive context \tilde{c} exerts the strongest pull on the query vector when it is simultaneously (a) the one the retriever is most confident about among pseudo-positives (high P(\tilde{c} | q_t), meaning it dominates other pseudo-positives in similarity) and (b) ranked near the bottom of the top-k (low P_k(\tilde{c} | q_t), meaning it needs a large ranking improvement). The paper gives a concrete extreme case: if there is a single pseudo-positive context \tilde{c} (so P(\tilde{c} | q_t) = 1) and it is ranked last in the top-k with zero probability mass (so P_k(\tilde{c} | q_t) ≈ 0), then the product is 1 — maximum pull. This is exactly the case where the optimization is most needed and most impactful.

This adaptive weighting is the mechanism by which TOURhard outperforms classical PRF with multiple iterations (Figure 3). Classical PRF applies the same uniform weighting at every iteration, which means it keeps pushing the query vector in the same average direction and quickly saturates. TOURhard, by contrast, re-weights at each iteration based on the current retrieval probabilities — as a pseudo-positive context moves up in the ranking, its pull diminishes, allowing other contexts (or new contexts that enter the top-k) to receive more optimization attention.

TOURsoft: Soft Pseudo-Labels via KL Divergence Minimization

TOURsoft is the second instantiation of the TOUR framework. While TOURhard makes a binary hard decision about pseudo-relevance (contexts are either in C^q_hard or not, with equal treatment within the set for the marginal likelihood objective), TOURsoft uses the full continuous distribution of re-ranker scores as soft labels, optimizing the query vector to make the retriever's ranking distribution match the re-ranker's relevance distribution via KL divergence minimization.

The motivation for a soft-label variant. In TOURhard, once C^q_hard is selected, the relative confidence among pseudo-positive contexts is ignored in the loss (Equation 9 sums their probabilities equally). In TOURsoft, every context's re-ranker score matters, with weight proportional to that score. The paper notes this distinction in Equation 11 vs. Equation 13: in TOURhard, the update weight depends on P(\tilde{c} | q_t) (re-normalized retrieval probability among pseudo-positives), which does not directly involve the re-ranker score beyond the binary selection decision. In TOURsoft, the update weight directly uses P(c_i | q_t, ϕ) (the re-ranker's normalized score), bringing the re-ranker's fine-grained confidence into the gradient computation itself.

This matters when the re-ranker has calibrated confidence across the full top-k. If three contexts have re-ranker scores of 0.9, 0.7, and 0.5 (all fairly relevant but with different degrees), TOURhard might select all three as pseudo-positive and treat them equally, while TOURsoft would weight the 0.9 context most heavily in the update. If the re-ranker's scores are reliable, TOURsoft should make more efficient updates by prioritizing the most relevant contexts.

The KL divergence objective. TOURsoft uses the re-ranker's probability distribution P(c_i | q_t, ϕ) over the top-k (defined in Equation 8, same as for TOURhard's selection step) as a target distribution, and minimizes the KL divergence from this target to the retriever's ranking distribution P_k(c_i | q_t):

$\mathcal{L}_{\text{soft}}(q_t, C^{q_t}_{1:k}) = -\sum_{i=1}^k P(c_i | q_t, \phi) \log \frac{P_k(c_i | q_t)}{P(c_i | q_t, \phi)}$

where P(c_i | q_t, ϕ) is the re-ranker's estimated probability that c_i is the relevant context (the soft label), P_k(c_i | q_t) is the retriever's probability for c_i within the top-k (the model's current prediction), and the sum is over all k retrieved contexts.

What this equation computes: the KL divergence D_KL(P(· | q_t, ϕ) || P_k(· | q_t)), which measures how much information is lost when the retriever's ranking distribution is used to approximate the re-ranker's relevance distribution. When the two distributions match exactly (the retriever ranks contexts in proportion to the re-ranker's confidence), the KL divergence is zero. When they differ (e.g., the re-ranker thinks context 3 is highly relevant but the retriever ranks it 10th), the divergence is large.

Why this form: KL divergence is the standard objective for knowledge distillation — a teacher model (the re-ranker) provides a target distribution, and the student model (the retriever's ranking) is trained to match it. This is conceptually similar to the distillation approaches used in RocketQAv2 (Ren et al., 2021) and Izacard and Grave (2020), but with a crucial difference: those methods distill from the re-ranker into the retriever's parameters by fine-tuning on the full training set, while TOURsoft distills into a single query vector at test time. The KL divergence naturally handles the multi-class nature of the ranking problem (the retriever assigns probability to k candidates, not just one), and it uses the re-ranker's scores as a full distribution rather than collapsing them to binary labels.

The loss can be expanded to reveal its structure:

$\mathcal{L}_{\text{soft}}(q_t, C^{q_t}_{1:k}) = -\sum_{i=1}^k P(c_i | q_t, \phi) \cdot q_t^\top c_i + \sum_{i=1}^k P(c_i | q_t, \phi) \cdot \log\sum_{j=1}^k \exp(q_t^\top c_j) + \text{const}$

The first term encourages increasing inner products with contexts that have high re-ranker probability. The second term (the log-sum-exp over all top-k contexts, multiplied by the re-ranker probabilities) acts as a normalizer, preventing the trivial solution of increasing all inner products uniformly.

The gradient and its operational form. The gradient of L_soft with respect to q_t is derived in Appendix B and takes a remarkably clean form:

$\frac{\partial \mathcal{L}_{\text{soft}}(q_t, C^{q_t}_{1:k})}{\partial q_t} = -\sum_{i=1}^k P(c_i | q_t, \phi) \cdot c_i + \sum_{i=1}^k P_k(c_i | q_t) \cdot c_i$

which, when substituted into the gradient descent update, gives:

$g_{\text{soft}}(q_t, C^{q_t}_{1:k}) = q_t + \eta \sum_{i=1}^k P(c_i | q_t, \phi) \cdot c_i - \eta \sum_{i=1}^k P_k(c_i | q_t) \cdot c_i$

This is Equation 13 in the paper.

What this expression represents: the update to the query vector consists of two weighted averages of context vectors. The positive term +η Σ_i P(c_i | q_t, ϕ) · c_i moves q_t toward the re-ranker-weighted average of all top-k contexts — contexts the re-ranker thinks are relevant get more weight, pulling the query vector in their direction. The negative term −η Σ_i P_k(c_i | q_t) · c_i moves q_t away from the retriever-weighted average of all top-k contexts — contexts the retriever currently ranks highly get subtracted out.

The net effect is intuitive: the query vector moves toward contexts the re-ranker likes and away from where the retriever currently thinks relevance lies. If the retriever is already ranking contexts in the right order (i.e., P_k(c_i | q_t) and P(c_i | q_t, ϕ) are similar), the two averages nearly cancel and the update is small — the query vector is already good. If the retriever ranks a highly relevant context low (high P(c_i | q_t, ϕ) but low P_k(c_i | q_t)), the positive term dominates for that context and the query vector moves strongly toward it. If the retriever ranks a non-relevant context high (low P(c_i | q_t, ϕ) but high P_k(c_i | q_t)), the negative term dominates and the query vector moves away from it.

Why this form is different from TOURhard: in TOURhard (Equation 11), the update toward pseudo-positive contexts is modulated by P(\tilde{c} | q_t)(1 − P_k(\tilde{c} | q_t)), which depends only on the retriever's own probabilities — the re-ranker only influences which contexts get pulled toward (via the binary selection into C^q_hard), not how strongly they are pulled. In TOURsoft, the strength of the pull is P(c_i | q_t, ϕ) itself — the re-ranker's confidence directly determines how much each context influences the update. This makes TOURsoft more sensitive to the re-ranker's fine-grained confidence, but also potentially more vulnerable to re-ranker miscalibration (e.g., if the re-ranker assigns high probability to many contexts, the update becomes diffuse).

Additionally, TOURsoft's update includes all contexts (weighted by the re-ranker), including those the re-ranker thinks are non-relevant (with low but non-zero probability). TOURhard's update explicitly ignores non-pseudo-positive contexts in the positive term (summation only over \tilde{c} ∈ C^q_hard), though they still appear in the negative term. TOURsoft thus uses more of the available re-ranker signal, at the cost of potentially being influenced by low-confidence re-ranker judgments on clearly non-relevant contexts.

The empirical trade-off. The paper finds that TOURhard and TOURsoft perform similarly in most settings, with nuances. For in-domain open-domain QA (Table 1), they are comparable: on Natural Questions, TOURhard achieves 48.6% and TOURsoft achieves 47.9% (both with top-10). On TriviaQA, they are nearly identical (66.4% vs. 66.5%). Under query distribution shift (Table 2), TOURhard consistently outperforms TOURsoft (averaging about 0.5–1.0% higher across datasets). For passage retrieval (Table 3), TOURsoft slightly outperforms TOURhard (e.g., 84.2% vs. 84.0% Acc@20 on NQ with DensePhrases, and 84.2% vs. 84.0% on DPR). The paper does not provide a definitive explanation for these differences but notes that it "requires more investigation in the future" (Section 4.2, discussing the query distribution shift results).

The Phrase Re-ranker: A Cross-Encoder for Phrase Retrieval

TOUR requires a cross-encoder re-ranker ϕ to provide pseudo labels on retrieved contexts. For passage retrieval, off-the-shelf re-rankers exist (e.g., the one from Fajcik et al., 2021, used in the passage retrieval experiments). For phrase retrieval, no prior re-ranker existed because phrase retrievers are a relatively recent development (Seo et al., 2019; Lee et al., 2021a). The paper develops a custom phrase re-ranker in Section 3.4, which is a necessary contribution for applying TOUR to phrase retrieval and is also independently valuable as the first cross-encoder re-ranker designed for phrase-level retrieval.

Training data construction. The re-ranker is trained on the retrieval outputs of the phrase retriever (DensePhrases) over the training set Q_train. For each training query q ∈ Q_train:

1. Retrieve the top-k contexts C_{1:k} from DensePhrases (the paper uses the top-100 for training data construction).
2. Identify the positive context c^+_q: a context that contains the correct answer to the query. In open-domain QA, "contains the correct answer" is determined by string matching against the gold answer — if the retrieved phrase (or its surrounding sentence) contains an answer string that matches the ground truth, the context is considered positive. This follows the standard open-domain QA assumption that a context containing the answer is relevant.
3. Sample one negative context c^-_q: a context that does not contain the correct answer. The paper does not specify the sampling strategy in detail, but it's implied to be random sampling from among the top-100 contexts that do not contain the answer.
4. Form training triplets (q, c^+_q, c^-_q) for all training queries to create D_train = {(q, c^+_q, c^-_q) | q ∈ Q_train}.

This triplet construction follows the standard pairwise ranking loss paradigm: the re-ranker is trained to assign a higher score to c^+_q than to c^-_q for each query.

Re-ranker architecture. The re-ranker is built on RoBERTa-large (Liu et al., 2019), a 24-layer transformer with 355M parameters. Given a query q and a context c, the re-ranker produces a scalar relevance score:

$s = \phi(q, c) = \mathbf{w}^\top \mathbf{M}(q \oplus c)[\text{CLS}]$

where M is the RoBERTa-large model, q ⊕ c is the concatenation of the query and context text separated by a [SEP] token, M(q ⊕ c)[CLS] is the final hidden state of the [CLS] token (a d-dimensional vector where d = 1024 for RoBERTa-large), and w ∈ ℝ^d is a learned linear projection vector that maps the [CLS] representation to a scalar. The parameters {M, w} are trainable; in practice, RoBERTa-large is fine-tuned from pre-trained weights and w is randomly initialized.

What this computes: the standard cross-encoder architecture for passage re-ranking (Nogueira and Cho, 2019). The query and context are concatenated into a single sequence and processed through all transformer layers with full cross-attention, allowing the model to attend to token-level interactions between query and context at every layer. The [CLS] token aggregates the sequence-level representation, and the linear projection w extracts a relevance score from it.

Why this architecture: cross-encoders outperform dual encoders for relevance judgment because the cross-attention mechanism can model exact term matches, semantic paraphrases, negation scope, and syntactic dependencies between query and context — all of which are lost when the query and context are encoded independently. The cost is that inference must be performed for each query-context pair (no pre-computation of context representations), which is why the re-ranker is typically applied only to a small top-k rather than the full corpus. For TOUR's purposes, the re-ranker is only needed on the top-k (typically 10–40) retrieval results per iteration, making the cost manageable.

Phrase-specific input formatting. For phrase retrieval, each retrieved phrase is embedded within its surrounding sentence. The paper uses the sentence containing the phrase as the context, following Lee et al. (2021b). Additionally, the document title is prepended to the context to provide broader topical context. Crucially, special tokens [S] and [E] are inserted around the retrieved phrase within the context to mark its position. The full input to the re-ranker is:

[CLS] query_text [SEP] document_title : ... [S] phrase_text [E] ... [SEP]

The [S] and [E] markers enable the re-ranker to attend specifically to the retrieved phrase and its immediate textual neighborhood, which is critical for assessing whether the phrase correctly answers the query or is merely contextually related.

Context length choice. The paper uses three sentences as context: the sentence containing the retrieved phrase plus the two surrounding sentences. This is motivated by efficiency: using entire paragraphs as input doubles the query latency of re-ranking while providing only marginal performance improvement. Table 6 (Appendix D) shows that using paragraphs achieves 45.6% on NQ versus 45.4% with three sentences — a negligible 0.2% gain at the cost of 2× slower inference. Using only one sentence drops performance to 43.6%, indicating that some surrounding context is necessary for accurate relevance judgment but full paragraphs are excessive.

Training procedure. The re-ranker is trained using a pairwise binary cross-entropy loss. For each triplet (q, c^+_q, c^-_q), the model computes scores h+ = ϕ(q, c^+_q) and h- = ϕ(q, c^-_q). The probability that c^+_q is preferred over c^-_q is:

$P^+ = \frac{\exp(h^+)}{\exp(h^+) + \exp(h^-)}$

and the loss is the binary cross-entropy (treating the positive as the correct class with target 1):

$\mathcal{L}_{\text{reranker}} = -\log P^+$

What this loss computes: the standard pairwise ranking loss. When h+ ≫ h-, P+ approaches 1 and the loss approaches 0. When h+ ≈ h-, P+ approaches 0.5 and the loss is log(2). When h+ ≪ h-, P+ approaches 0 and the loss grows large. Minimizing this loss pushes the model to assign higher scores to positive contexts than to negative ones.

Why this form over pointwise or listwise losses: pairwise ranking is the standard approach for training re-rankers because it directly optimizes the relative ordering, which is what matters for re-ranking. Pointwise losses (e.g., binary classification of relevant vs. non-relevant) require setting a relevance threshold and do not directly optimize ranking; listwise losses (e.g., LambdaRank) can be more effective but are more complex to implement and train. Pairwise binary cross-entropy is a pragmatic, well-validated choice.

Training hyperparameters. The paper trains the re-ranker for 20 epochs on D_train. For each epoch, positive and negative contexts are freshly sampled from the top-100 retrieval results, providing data augmentation through varying negative examples. The best checkpoint is selected based on validation accuracy of the re-ranker (i.e., how often h+ > h- on a held-out validation set). The implementation uses the HuggingFace Transformers library (Wolf et al., 2020) with standard hyperparameters from the library's text classification examples, except for the number of training epochs (increased from the default).

Pre-training on reading comprehension. Before fine-tuning on the phrase retrieval training triplets, the re-ranker is pre-trained on reading comprehension datasets: SQuAD (Rajpurkar et al., 2016), TriviaQA (Joshi et al., 2017), and Natural Questions (Kwiatkowski et al., 2019). Pre-training on these datasets teaches the model to identify spans within contexts that answer questions, which transfers to the phrase re-ranking task where the model must judge whether a retrieved phrase (marked with [S] and [E]) correctly answers a query. Table 6 shows the impact: without any pre-training (training only on the phrase retrieval triplets from scratch), performance drops from 45.4% to 42.0% on NQ. Replacing reading comprehension pre-training with MNLI (a natural language inference dataset) pre-training drops performance to 43.8%, confirming that reading comprehension is the more effective pre-training task due to its closer alignment with the phrase-answer-verification objective.

Ablation details (Table 6). The paper reports a thorough ablation of the re-ranker design:

• Without prepending titles: 44.8% (down from 45.4%), showing that document-level context aids relevance judgment — knowing that a phrase comes from an article about "Cellular Networks" versus "Baking" helps disambiguate queries.
• Using RoBERTa-base instead of RoBERTa-large: 43.2% (down from 45.4%), confirming that model capacity matters for fine-grained relevance assessment.
• Using 1 sentence instead of 3 sentences as context: 43.6% (down from 45.4%), indicating that some surrounding context is necessary beyond the immediate sentence.
• Using entire paragraphs instead of 3 sentences: 45.6% (marginally up from 45.4%), but at the cost of doubled latency — the small accuracy gain does not justify the computational cost.

Score aggregation at test time. After TOUR completes its iterations, the final ranking is produced by linearly interpolating the retrieval similarity score with the re-ranker score:

\text{final_score}(q, c_i) = \lambda \cdot s_i + (1 - \lambda) \cdot \text{sim}(q, c_i)

where λ ∈ [0, 1] controls the relative weight of the re-ranker versus the retriever. For open-domain QA with DensePhrases, λ = 0.1 (Table 7), giving 10% weight to the re-ranker and 90% to the retriever's similarity. This is notable: the re-ranker's score receives relatively little weight in the final ranking, suggesting that TOUR's primary value is in changing which contexts appear in the top-k (via query vector updates) rather than in re-ranking them, and the re-ranker's direct score is used mainly as a tiebreaker on the final set. For passage retrieval, λ = 1.0, meaning only the re-ranker's scores are used for the final ranking.

The ablation in Table 5 confirms the value of score aggregation: setting λ = 0 (retriever-only final ranking) reduces performance from 48.4% to 48.1%, and setting λ = 1 (re-ranker-only final ranking) reduces it to 48.0%. While the differences are small (0.3–0.4%), they are consistent across experiments, and Figure 3 shows that without score aggregation, TOUR's performance initially drops below the baseline before recovering with more iterations — score aggregation provides an immediate improvement that gradient-based optimization then builds upon.

Efficient Implementation: Making TOUR Faster Than Direct Re-ranking

A naive implementation of TOUR would be computationally expensive: each iteration requires (a) running the cross-encoder re-ranker on all k retrieved contexts, which is the most expensive operation, and (b) performing a full retrieval pass over the index with the updated query vector. With multiple iterations, the total cost could exceed that of simply running the re-ranker on a larger top-k and being done. The paper introduces two techniques — early stopping and score caching — that together make TOUR faster than direct re-ranking while achieving better performance.

Early stopping conditions. TOUR terminates the iterative optimization before reaching the maximum iterations (set to 3 for open-domain QA, 1 for passage retrieval) when a convergence criterion is met. The stopping condition differs between the two variants:

• For TOURhard: stop when the top-1 retrieval result (the highest-ranked context by retrieval similarity) is classified as pseudo-positive by the re-ranker — i.e., when c_1 ∈ C^{q_t}_{hard}. This condition means: "the retriever's most confident answer is one that the re-ranker considers relevant, so further optimization is unnecessary."

• For TOURsoft: stop when the top-1 retrieval result has the highest re-ranker score among all top-k contexts — i.e., when c_1 = argmax_i ϕ(q, c_i). This condition means: "the retriever and re-ranker agree on the best context, so their ranking distributions are already aligned."

What these conditions achieve: they prevent unnecessary iterations when the query representation is already good (the retriever and re-ranker agree on the top result). In practice, this means that easy queries — where the initial encoder already produces a good representation — require only 1 iteration (or sometimes 0 if the condition is met before any optimization), while harder queries that require substantial query vector adjustment use the full budget of 3 iterations. This adaptivity is crucial for the latency-performance trade-off: easy queries are fast, hard queries get more compute.

Score caching. The re-ranker score ϕ(q, c_i) for a given query-context pair does not change across TOUR iterations (the re-ranker is frozen), but the set of top-k contexts changes as the query vector is updated. To avoid re-computing scores for contexts that appear in multiple iterations, the paper caches ϕ(q, c_i) for each query: the first time a context appears in the top-k, its score is computed and stored; if the same context appears again in a later iteration, the cached score is reused.

What caching achieves: it eliminates redundant cross-encoder computations. If the top-k set has high overlap across iterations (which is common when the query vector moves only slightly), caching saves substantial computation. The paper does not report the cache hit rate, but Figure 4 illustrates the latency reduction from caching alone.

Maximum iterations budget. The paper limits TOUR to at most t = 3 iterations for open-domain QA and t = 1 for passage retrieval (Table 7). The choice of 3 iterations for open-domain QA is motivated empirically by Figure 3, which shows that TOURhard's performance increases from iteration 1 to iteration 3 but not beyond — PRF with the Rocchio algorithm, by contrast, peaks at iteration 1 and degrades slightly thereafter. The single iteration for passage retrieval is a practical choice given the larger top-k (k = 100 for passage retrieval versus k = 10–20 for open-domain QA, making each iteration substantially more expensive).

Latency analysis (Figure 4). The paper measures query latency (in milliseconds) as a function of the number of iterations with and without the efficiency techniques:

• Without any optimization: latency increases linearly with iterations, from approximately 500ms at 1 iteration to over 1700ms at 5 iterations.
• With caching only: latency at 5 iterations drops from ~1700ms to ~1300ms — a substantial reduction, confirming that cross-encoder computation is a major bottleneck and that many contexts recur across iterations.
• With early stopping only: latency at 5 iterations drops to ~600ms — an even larger reduction, because most queries stop before reaching 5 iterations. The average number of iterations is far lower than the maximum.
• With both caching and early stopping: latency at 5 iterations is approximately 500ms — essentially flat across iterations, because early stopping ensures that few queries reach even 3 iterations, and caching handles the overlap for those that do.

Comparison to direct re-ranking (Figure 2). The latency-performance trade-off is visualized with k varied from 10 to 50:

• TOURhard with top-10 (0.44 seconds per query per Table 1, or ~440ms) achieves 48.6% EM on NQ.
• Direct re-ranker with top-40 (1.04 seconds per query) achieves 46.5% EM.
• TOURhard is thus 2.4× faster while achieving 2.1% higher accuracy.
• Even comparing at the same top-k, TOURhard is more efficient because it runs the re-ranker only on the top-10 (versus top-40 for the re-ranker) and because early stopping reduces the average number of re-ranker calls.

Why TOUR can be faster than direct re-ranking. The fundamental efficiency advantage comes from a virtuous cycle: TOUR uses a small top-k (typically 10) because the query vector updates pull new relevant contexts into the top-k from below, effectively achieving the coverage of a larger top-k with the computational cost of a small one. A re-ranker with a small top-k would miss many relevant contexts; TOUR's iterative optimization recovers them. This is visible in Figure 2: TOURhard with top-10 outperforms re-ranker with top-40, showing that TOUR's top-10 after optimization contains more relevant contexts than the re-ranker's static top-40.

Parameter summary for different tasks (from Table 7). The hyperparameters differ by task and model:

• Open-domain QA (DensePhrases): learning rate η = 1.2, max iterations = 3, retrieval top-k = 10, re-ranker top-k = 10, final score interpolation λ = 0.1.
• Passage retrieval (DensePhrases as passage retriever): learning rate η = 1.2, max iterations = 1, retrieval top-k = 100, re-ranker top-k = 100, λ = 1.0.
• Passage retrieval (DPR): learning rate η = 0.2, max iterations = 1, retrieval top-k = 100, re-ranker top-k = 100, λ = 1.0.

The larger learning rate for DensePhrases (1.2 vs. 0.2 for DPR) likely reflects differences in the embedding space geometry: DensePhrases uses phrase-level embeddings that may require larger steps to move meaningfully, while DPR's passage-level embeddings may be more sensitive to updates. The paper does not discuss this tuning in detail.

Momentum and weight decay. All TOUR variants use momentum = 0.99 and weight decay λ_decay = 0.01. The momentum provides inertia: the update direction at iteration t is a moving average of current and past gradients, which smooths optimization and can help escape shallow local optima. The weight decay provides an implicit α < 1 in the Rocchio analog (recall §3.4.2: α = 1 − ηλ_decay), ensuring that the query vector does not drift unboundedly far from its initial position and retains some of the original encoder's knowledge about the query.

Linear learning rate scheduling. The paper applies linear learning rate scheduling per iteration, though the specific schedule (e.g., whether it increases, decreases, or cycles) is not detailed. This is listed in Appendix D but not elaborated upon in the main text.

Hardware and infrastructure. The paper does not specify exact hardware (GPU type, memory, etc.) used for latency measurements, but the absolute latency numbers (0.09 seconds for PRF, 0.24 seconds for the phrase re-ranker with top-10, 0.44 seconds for TOURhard with top-10) are provided in Tables 1 and 2. These latencies include the DensePhrases retrieval time (the base retriever's latency is embedded in the numbers), so the TOUR-specific overhead can be approximately computed by subtraction.

Design Choices Summary and Justifications

The paper makes several non-obvious design choices whose rationale is instructive:

Why gradient descent rather than a simpler update rule (like Rocchio directly)? The gradient-based approach automatically derives instance-specific, dynamically-weighted updates from the loss function, eliminating the need to tune fixed hyperparameters (β, γ, k′ in Rocchio) that would need to be set differently for each dataset, retriever, and possibly each query. The derivation in Equation 11 shows that the gradient is a weighted form of the Rocchio update, but with weights that adapt based on the current retrieval probabilities — a property that would be extremely cumbersome to specify manually.

Why instance-level optimization over the query vector rather than fine-tuning the encoder? Fine-tuning the encoder (query-side fine-tuning, Lee et al., 2021a) changes the encoding of all queries, which can cause catastrophic forgetting on queries that the original encoder handled well. Instance-level optimization affects only one query, so there is no interference. Additionally, instance-level optimization requires only the test query and its retrieval results, not a full training set — it is deployable in zero-shot settings where no in-domain training data exists.

Why the cross-encoder re-ranker as the pseudo-labeling oracle rather than a simpler heuristic (e.g., BM25, embedding similarity)? Cross-encoders produce substantially more accurate relevance judgments than heuristics, and they generalize better to unseen query distributions compared to dual encoders (Rosa et al., 2022). The paper's ablation in Table 5 shows that using the re-ranker's fine-grained labels (C^q_hard) outperforms the simpler heuristic of taking the top-k′ by retrieval rank, confirming that label quality matters. The trade-off is computational cost, which the paper addresses through efficient implementation techniques.

Why the Nucleus Sampling-style cumulative probability selection for TOURhard rather than top-k′ or a score threshold? The cumulative probability approach adapts the number of pseudo-positive contexts to the re-ranker's confidence distribution. When the re-ranker is highly confident about a small number of contexts, the set is small, enabling focused optimization. When confidence is diffuse, the set is larger, reflecting genuine uncertainty. A fixed top-k′ cannot adapt this way; a score threshold requires careful calibration (what threshold value? does it transfer across datasets and queries?). The cumulative probability with temperature 0.5 and threshold 0.5 provides a robust, hyperparameter-light selection mechanism.

Why both hard and soft variants? The hard variant (TOURhard) is simpler and more robust when the re-ranker's relative confidence is less reliable — it needs only to identify a set of likely-relevant contexts, not to rank them precisely. The soft variant (TOURsoft) can extract more signal when the re-ranker's scores are well-calibrated, using fine-grained confidence differences to weight updates. The empirical results suggest that hard labels are slightly better under distribution shift (Table 2), possibly because re-ranker calibration degrades on out-of-distribution queries, making the fine-grained confidence less trustworthy.

4. Key Insights and Innovations

Innovation 1: Reframing Test-Time Retrieval Failure as an Instance-Level Optimization Problem, Not a Model Capacity Problem

The paper's most fundamental conceptual move is reframing the problem of retrieval failure on unseen queries. The dominant paradigm before TOUR treated poor retrieval as evidence that the encoder needed more capacity, more training data, or better contrastive objectives — in other words, that the model was the problem. Solutions followed naturally: scale up the encoder (Karpukhin et al., 2020), add more training datasets (Lee et al., 2021a), or distill from a stronger teacher (Izacard and Grave, 2020; Ren et al., 2021). All of these improve the encoder's average behavior across the query distribution, but they leave the fundamental asymmetry untouched: once trained, the encoder produces a fixed representation for any given query, regardless of whether that particular query is typical or anomalous.

TOUR shifts the locus of the problem from the model to the instance. The encoder is treated as good enough — it produces a reasonable starting point — but each query is understood to require its own bespoke refinement based on the specific retrieval landscape it encounters. This is a diagnostic reframing, not just a method: it says that retrieval failure is often not a capacity problem but an adaptation problem. The encoder has the right general neighborhood in embedding space, but the initial query vector is slightly misaligned with where the relevant contexts actually cluster for this specific query. A small instance-level adjustment — one that would be inappropriate to bake into the encoder's shared parameters because it might hurt other queries — can recover the correct retrieval.

This idea draws conceptual lineage from dynamic evaluation in language modeling (Krause et al., 2019), where test-time gradient updates adapt a language model to the local statistics of the evaluation corpus. But TOUR applies this principle to the fundamentally different setting of retrieval, where the optimization target is not token prediction but ranking quality as judged by a frozen oracle (the re-ranker). The connection to dynamic evaluation is acknowledged in Section 3.2, but the adaptation is non-trivial: in language modeling, the gradient comes from next-token prediction loss on ground-truth tokens; in TOUR, the gradient comes from aligning the retriever's ranking with a cross-encoder's judgments, with no ground-truth relevance labels available at test time.

The significance of this reframing extends beyond the method TOUR itself. It suggests that the field's relentless focus on scaling encoders — ever-larger models, ever-larger training sets — may be addressing the wrong bottleneck for a significant fraction of retrieval failures. If many failures are instance-level misalignments that a few gradient steps can fix, then a smaller encoder with test-time optimization may be more cost-effective than a much larger encoder without it. The FLOPs-matched comparisons between pretraining and inference compute are not conducted in this paper (unlike the reference example), but the implicit argument is similar: test-time compute applied to instance-level optimization can substitute for model capacity on queries where the base encoder is approximately correct but not precisely aligned. This is evidenced by the fact that TOUR on top of a DensePhrases model trained only on Natural Questions (Table 2) achieves performance on unseen query distributions that approaches models trained on all five datasets (Table 1) — instance-level optimization partially closes the domain gap without any additional training data.

Innovation 2: A Theoretical Unification of Gradient-Based Query Optimization with Classical Pseudo Relevance Feedback

Prior to TOUR, pseudo relevance feedback (PRF) for dense retrieval existed as a heuristic transplant from sparse retrieval: take the average of the top-k' retrieved context vectors and add it to the query vector, with fixed, manually-tuned weights (Yu et al., 2021; Wang et al., 2021). These methods treated PRF as a post-hoc correction step, disconnected from the training objective that produced the encoders. TOUR provides the first principled theoretical bridge showing that gradient-based optimization of a well-specified loss function over pseudo labels recovers — and strictly generalizes — the classical Rocchio algorithm.

This is not merely a citation to prior work; it is a diagnostic insight about why gradient-based optimization should work better. The derivation in Appendix A (and summarized in Section 3.2) reveals that the Rocchio algorithm makes three implicit assumptions: (1) pseudo-relevant contexts are exactly the top-k' by retrieval rank (not by actual relevance), (2) all contexts in the top-k have equal retrieval probability, and (3) the per-context update weights are uniform and fixed. All three assumptions are violated in practice, and each violation is a source of inefficiency that TOUR's gradient-based approach eliminates. Specifically:

• Assumption (1) fails because retrieval rank and true relevance are imperfectly correlated. A context ranked 8th may be highly relevant (just not retrieved highly enough), while a context ranked 2nd may be non-relevant (a near-miss in embedding space). TOUR uses the cross-encoder to select pseudo-positives based on judged relevance, not rank, as validated by the ablation in Table 5 where replacing C^q_hard with uniform top-3 selection drops performance by 2.3%.

• Assumption (2) fails because retrieval probabilities are typically concentrated on a few top-ranked contexts. TOUR's gradient weights each context by P(˜c | q_t)(1 − P_k(˜c | q_t)) (Equation 11), which automatically allocates more update budget to pseudo-positive contexts that are currently ranked poorly — the factor (1 − P_k(˜c | q_t)) is largest when the context has near-zero retrieval probability, meaning it needs the most improvement. Rocchio's uniform weights cannot make this distinction.

• Assumption (3) fails because the optimal update direction and magnitude depend on the current retrieval state, which changes at each iteration. TOUR recomputes weights at every iteration based on the updated retrieval probabilities, while Rocchio applies the same fixed weights regardless of progress.

The theoretical significance is that this derivation provides a constructive explanation for the empirical finding that classical PRF saturates after one iteration while TOUR continues to improve through three (Figure 3). Rocchio's fixed weights mean that after the first update, further iterations just keep pushing in the same average direction, eventually overshooting or oscillating. TOUR's dynamic weights automatically reduce the pull on contexts that have already improved their ranking (since 1 − P_k(˜c|q_t) shrinks as P_k grows), and shift attention to contexts that still need improvement. This is a fundamental algorithmic advance, not merely a better choice of hyperparameters.

The unification also reveals that the cross-encoder re-ranker serves a dual role that has no analog in classical PRF. In Rocchio, the only signal is "top-k' are relevant, the rest are not" — a binary, rank-based judgment. In TOUR, the re-ranker provides both the selection of pseudo-positives (in TOURhard) and the weighting of each context's contribution (in TOURsoft, where P(c_i|q_t, ϕ) directly determines the update magnitude). This richer signal is what enables TOUR to make finer-grained updates that avoid the saturation and oscillation problems of classical PRF. The theoretical contribution is thus the framework that explains why richer feedback signals should improve optimization — not just that they do.

Innovation 3: The Cross-Encoder as Teacher, Not Gatekeeper — Repurposing Re-rankers for Retrieval Improvement Rather Than Just Reordering

The standard role of a cross-encoder re-ranker in information retrieval is well-established: take a set of already-retrieved candidates, apply the expensive cross-encoder to score them, and reorder the results so that the truly relevant ones appear at the top. This is a selection role — the re-ranker acts as a gatekeeper that picks winners from a fixed pool. The fundamental limitation is that if the pool does not contain any relevant contexts (low recall in the top-k), the re-ranker cannot fix the problem — it can only reorder what it is given. This is visible in Table 3, where re-ranking consistently improves Acc@20 but never improves Acc@100, because the relevant contexts in the top-100 were already retrieved; re-ranking just moves them from positions 21–100 into the top 20.

TOUR fundamentally repurposes the cross-encoder from a selector to a teacher. The re-ranker's scores are not used directly for the final ranking (the score aggregation in TOUR gives them only 10% weight, λ = 0.1, for open-domain QA). Instead, they serve as a training signal that teaches the query vector where to look in subsequent retrieval rounds. This changes what the cross-encoder accomplishes: rather than picking the best among what was found, it guides the query to find new contexts that were previously missed. The re-ranker's output becomes a direction for exploration, not a final judgment.

This repurposing is what enables TOUR to improve Acc@100 in Table 3 — something pure re-ranking cannot do. When TOUR updates the query vector, the next retrieval round searches the entire corpus with a different query representation, potentially pulling in relevant contexts from outside the initial top-100. The re-ranker then scores these new contexts, provides updated pseudo labels, and the cycle continues. The cross-encoder is still doing what it does best — making fine-grained relevance judgments — but those judgments are fed back into the retrieval process rather than being the endpoint of it.

This innovation has a subtle but important implication for the design of dense retrieval systems: the cross-encoder re-ranker and the dual-encoder retriever are not competitors (as often framed in the accuracy-vs-efficiency trade-off) but complements in an iterative improvement loop. The dual encoder provides fast, approximate retrieval across the full corpus; the cross-encoder provides accurate, localized relevance judgments on a small subset; and TOUR closes the loop by using the cross-encoder's judgments to improve the dual encoder's next retrieval. This architecture — a fast proposer guided by a slow but accurate verifier — is reminiscent of search algorithms like MCTS where a cheap rollout policy is guided by an expensive value function, but TOUR applies it to the continuous optimization of a query representation rather than discrete tree search.

The empirical evidence for this innovation's significance is the latency-performance trade-off in Figure 2 and Table 1. TOUR with top-10 (0.44s/query) outperforms the direct re-ranker with top-40 (1.04s/query) by 2.1% on Natural Questions — it achieves better accuracy than the re-ranker's best effort while running more than 2× faster. This is only possible because TOUR uses the re-ranker more efficiently: it applies the expensive cross-encoder to a small number of candidates per iteration, uses the resulting signal to improve retrieval, and stops early when the top result is already good. A re-ranker that tried to match TOUR's accuracy by using a massive top-k would be prohibitively slow; TOUR achieves the same or better accuracy with a small top-k because the query vector does the work of finding new relevant contexts that the re-ranker would have needed a large top-k to cover.

Innovation 4: Nucleus Sampling-style Cumulative Probability Thresholding as a Robust Pseudo-Label Selection Mechanism

The use of cumulative probability thresholding (inspired by Nucleus Sampling; Holtzman et al., 2020) to select pseudo-positive contexts in TOURhard is a small but conceptually elegant innovation that solves a nuisance problem in pseudo relevance feedback: how many pseudo-positive contexts should be used, and which ones? Classical PRF uses a fixed k' — a hyperparameter that must be tuned per dataset and per retriever, and that cannot adapt to per-query variation in the re-ranker's confidence. A query where the re-ranker is highly confident about one context should focus the update on that single context; a query where confidence is spread across five should cast a wider net. A fixed k' gets this wrong in both directions.

The cumulative probability approach (Equation 8) answers both questions adaptively: the size of the pseudo-positive set is determined by how many contexts are needed to reach cumulative probability p (0.5 in the paper), and the membership is determined by the re-ranker's confidence ordering. When the re-ranker assigns 0.6 probability to the top-ranked context, the pseudo-positive set is a singleton — focused optimization on the most promising candidate. When the re-ranker assigns 0.2 to each of the top five, the set contains three contexts — appropriately broader optimization reflecting genuine uncertainty.

This is not a major theoretical contribution, but it is a practical innovation that eliminates a hyperparameter that would otherwise require extensive tuning and would likely not transfer across datasets, retrievers, or query distributions. The ablation in Table 5 validates its importance: replacing the adaptive selection with fixed top-3 drops performance by 2.3% (48.4% → 46.1%), a substantial degradation from a seemingly minor change. This suggests that the adaptivity of the selection — matching the pseudo-positive set size to the re-ranker's confidence distribution per query — is doing real work, not just the use of re-ranker scores in general.

The connection to Nucleus Sampling is more than a citation: it imports a technique developed for a very different problem (preventing degenerate text generation) and repurposes it for a retrieval training problem. The shared insight is that cumulative probability thresholding provides a principled, adaptive way to select a variable-sized subset from a ranked distribution, avoiding both the brittleness of fixed-k selection and the arbitrariness of score-threshold selection. For text generation, this prevents the model from falling into repetitive loops by sampling from the nucleus of high-probability tokens. For pseudo-label selection, it prevents the optimization from being derailed by low-confidence pseudo-positives by focusing on the nucleus of high-probability relevant contexts. The transfer of this technique across domains is an elegant example of cross-pollination in ML method design.

Innovation 5: The Construction and Validation of a Phrase-Level Re-Ranker, Extending Cross-Encoder Reranking to a New Retrieval Granularity

While not the central methodological contribution, the phrase re-ranker developed in Section 3.4 fills a genuine gap: prior to TOUR, no cross-encoder re-ranker existed for phrase retrieval systems like DensePhrases. Phrase retrieval represents a different granularity than passage or document retrieval — the retrieved unit is a short span (typically a few words) embedded within a sentence, and relevance depends on whether that specific span correctly answers the query, not just whether the surrounding passage is topically related. This requires the re-ranker to attend to the relationship between the query and the exact phrase, which standard passage re-rankers are not designed to do.

The design choices that make this re-ranker effective are instructive beyond this specific application. The use of special tokens [S] and [E] to mark phrase boundaries within the context — explicitly signaling to the model which span is the candidate answer — transforms the task from "is this passage relevant?" to "does this specific phrase answer the query?" The pre-training on reading comprehension datasets (SQuAD, TriviaQA, Natural Questions) rather than natural language inference (MNLI) aligns the model's pre-training objective (identifying answer spans) with its fine-tuning objective (judging phrase relevance). The finding that three sentences of surrounding context are sufficient (matching full-paragraph performance at half the latency, Table 6) provides a practical guideline for the context window needed for phrase-level judgment: more than the immediate sentence (1 sentence drops to 43.6%) but much less than the full paragraph (paragraph gives +0.2% for 2× cost).

The phrase re-ranker also demonstrates that cross-encoder architectures can be effectively adapted to retrieval tasks at finer granularity than passages, opening the door to phrase-level re-ranking as a general technique that TOUR enables by making re-ranking more computationally efficient (since TOUR needs only a small top-k). Without TOUR, applying this phrase re-ranker naively would require scoring a large top-k to achieve good recall (as the baseline re-ranker with top-40 does in Table 1), making it expensive. TOUR's efficiency advantage means that phrase re-rankers become practically deployable, which in turn enables the broader use of phrase retrieval systems in latency-sensitive applications.

The innovation here is incremental — adapting an existing architecture (cross-encoder re-ranking) to a new retrieval granularity (phrases) with careful input formatting and pre-training choices — but it is practically significant because it enables the entire TOUR pipeline for phrase retrieval, which accounts for the bulk of the paper's open-domain QA experiments. Without this component, TOUR on phrase retrieval would not be possible, since no alternative phrase re-ranker existed to provide the pseudo labels that TOUR requires.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use six open-domain QA datasets: Natural Questions (Kwiatkowski et al., 2019; 79,168 train / 8,757 dev / 3,610 test), TriviaQA (Joshi et al., 2017; 78,785 train / 8,837 dev / 11,313 test), WebQuestions (Berant et al., 2013; 3,417 train / 361 dev / 2,032 test), CuratedTrec (Baudiš and Šediv`y, 2015; 1,353 train / 133 dev / 694 test), SQuAD (Rajpurkar et al., 2016; 78,713 train / 8,886 dev / 10,570 test), and EntityQuestions (Sciavolino et al., 2021; 22,075 test only, used exclusively for out-of-distribution evaluation). Full statistics are listed in Appendix E (Table 8). The paper also reports preliminary BEIR benchmark results in the Limitations section (48.1% macro-averaged NDCG@10 vs. 47.8% for the re-ranking baseline), noting significant gains on BioASQ (+11.6% NDCG@10 over the baseline retriever) and TREC-COVID (+23.8%).

• Base model(s). For phrase retrieval and open-domain QA, the paper uses DensePhrases (Lee et al., 2021a) — specifically, the multi-dataset-trained DensePhrases_multi for in-domain evaluation (trained on all five training datasets) and DensePhrases_NQ (trained only on Natural Questions) for query distribution shift experiments. For passage retrieval, DPR (Karpukhin et al., 2020) is used — specifically DPR_multi (trained on multiple datasets) for in-domain and DPR_NQ for out-of-domain, alongside DensePhrases configured as a phrase-based passage retriever following Lee et al. (2021b). The paper describes PaLM 2-S* style comparisons in the reference example, but for this paper the models are RoBERTa-large-based dual encoders and cross-encoders; the phrase re-ranker uses RoBERTa-large (355M parameters) as its backbone. The models are chosen because DensePhrases represents state-of-the-art phrase retrieval and DPR represents the standard passage retrieval baseline — both are "representative of the capabilities of contemporary dense retrievers" and the paper aims to show TOUR works across retrieval paradigms.

• Metrics. For end-to-end open-domain QA, the metric is Exact Match (EM) — the fraction of test questions for which the retrieved phrase (for phrase retrieval) or the extracted answer (for retriever-reader baselines) exactly matches one of the ground-truth answers, graded using string matching (following standard open-domain QA evaluation). For passage retrieval, the metric is top-k retrieval accuracy (Acc@k) — the fraction of questions for which at least one retrieved passage in the top-k contains the correct answer (binary: 1 if the answer string appears in any of the top-k passages, 0 otherwise). The paper reports Acc@20 and Acc@100. Query latency is measured in seconds per query (s/q) and includes the base retriever's retrieval time; for TOUR, the additional latency of re-ranking and gradient updates is included in the reported numbers.

• Baselines. For open-domain QA, the paper compares against: (1) DPR_multi + extractive reader (Karpukhin et al., 2020) — the standard retriever-reader architecture where DPR retrieves passages and a BERT-based reader extracts answers; (2) DPR_multi + Re-ranker (Iyer et al., 2021) — the ReConsider re-ranker applied on top of DPR's retrieval results with top-5; (3) GAR (Mao et al., 2021) — generation-augmented retrieval using BM25 with query expansion; (4) ColBERT-QA_large (Khattab et al., 2021) — late interaction retrieval with a QA component; (5) UnitedQA-E_large — an ensemble model; (6) PRF_Rocchio — the classical Rocchio algorithm applied to dense retrieval, using uniform top-k′ selection with fixed weights (Equation 6); (7) DensePhrases_multi — the base phrase retriever without any test-time optimization; (8) Phrase re-ranker — the paper's own cross-encoder phrase re-ranker applied as a direct re-ranking baseline (scores final results without query vector updates). For passage retrieval, the baselines are: (1) the base retriever alone (DensePhrases or DPR), and (2) the off-the-shelf passage re-ranker from Fajcik et al. (2021) applied as direct re-ranking.

• Generation budget / compute accounting. Unlike the reference example paper which measured compute in "generations" for LLM inference, this paper measures compute in two dimensions: top-k (number of retrieval candidates processed, which determines re-ranker calls and gradient computation cost) and iterations (number of optimization rounds, each requiring a full retrieval pass and re-ranking). Top-k is swept from 10 to 50 for latency-performance trade-off analysis (Figure 2). For the main open-domain QA experiments (Tables 1 and 2), TOUR uses k = 10 or k = 20, while the direct re-ranker baseline uses k = 10 or k = 40. For passage retrieval (Table 3), all methods use k = 100 due to "limited resource budget." Maximum iterations are set to 3 for open-domain QA and 1 for passage retrieval (Table 7), though early stopping typically terminates earlier. Latency measurements (s/q) provide the practical compute accounting — they include the base retriever's time plus all TOUR/re-ranking overhead, measured in seconds per query on (unspecified) GPU hardware. The fair comparison point is: TOUR with top-10 vs. re-ranker with top-40 (TOUR achieves better accuracy with fewer re-ranker calls and faster total latency).

• Cross-validation / statistical protocol. The paper does not use cross-validation in the standard ML sense (no k-fold splitting for hyperparameter selection). Instead, hyperparameters are tuned using the in-domain development set for each dataset (e.g., the NQ development set for NQ test evaluation), following standard open-domain QA practice. Table 7 lists task-specific hyperparameters (learning rate, max iterations, top-k, λ) that were selected via development set tuning. The paper reports single-run results on test sets without confidence intervals or statistical significance tests. For the train-test overlap analysis (Section 5.1, Table 4), the paper uses the pre-defined splits from Lewis et al. (2021), which partition each test set into subsets based on whether test queries paraphrase training queries ("Query overlap"), share answers but not queries with training data ("Answeronly overlap"), or have neither query nor answer overlap with training data ("None overlap"). No multiple random seeds or bootstrap confidence intervals are reported, which is a limitation for assessing result stability given the relatively small test sets (e.g., CuratedTrec has only 694 test questions, WebQuestions has 2,032).

Main Quantitative Results

The experimental results are organized around three axes: (1) end-to-end open-domain QA with phrase retrieval under in-domain and distribution-shift settings, (2) passage retrieval accuracy, and (3) latency-performance trade-offs and efficiency comparisons. Each result section below addresses a distinct evaluation setting.

End-to-End Open-Domain QA: In-Domain Evaluation

Headline results (Table 1). TOUR substantially improves over all baselines on in-domain open-domain QA. On Natural Questions, TOURhard with k = 10 achieves 48.6% EM — an improvement of 7.0 percentage points over the base DensePhrases_multi (41.6%) and 1.6 points over the paper's own phrase re-ranker with k = 10 (47.0%). Against the stronger re-ranker with k = 40 (46.5%), TOURhard with k = 20 achieves 47.9% — a 1.4-point improvement while running 1.3× faster (0.78 s/q vs. 1.04 s/q). TOURsoft with k = 20 achieves parity with TOURhard at 47.6% (k = 10: 47.9%). The largest absolute improvements are on SQuAD (DensePhrases: 34.5% → TOURhard,k=20: 46.4%, +11.9 points) and TriviaQA (56.3% → TOURhard,k=20: 66.8%, +10.5 points).

Comparison against retriever-reader models (Table 1, top section). TOURhard with top-10 (0.44 s/q) outperforms DPR_multi + ReConsider re-ranker with top-5 (1.21 s/q) by 5.5 points on NQ (48.6% vs. 43.1%), 7.1 points on TriviaQA (66.4% vs. 59.3%), and 2.2 points on CuratedTrec (62.0% vs. 49.3% — though the re-ranker actually degrades from DPR's 49.4%). Against the much larger UnitedQA-E_large ensemble (51.8% on NQ, 68.9% on TriviaQA), TOURhard underperforms by 3.2 points on NQ but closes to within 2.5 points on TriviaQA — notable given that TOUR operates on a single retriever without a reader. ColBERT-QA_large achieves 47.8% on NQ (TOURhard: 48.6%) and 70.1% on TriviaQA (TOURhard: 66.4%) — TOUR wins on NQ but loses by 3.7 points on TriviaQA.

Comparison against PRF Rocchio baseline (Table 1). The classical Rocchio algorithm applied to dense retrieval (PRF_Rocchio with k = 10, 0.09 s/q) provides negligible improvement over DensePhrases_multi: +0.0% on NQ (41.6% → 41.6%), +0.2% on TriviaQA, +0.2% on WebQuestions, +0.1% on CuratedTrec, +0.4% on SQuAD. This is a critical negative result: directly transplanting the classical PRF update rule (Equation 6) to dense retrieval provides essentially no benefit, confirming the paper's argument that uniform-weight, rank-based pseudo-labeling is insufficient. TOURhard with comparable computational budget (k = 10, 0.44 s/q — 4.9× slower than PRF_Rocchio but still faster than the re-ranker with k = 40) provides gains of +7.0 to +11.9 points across datasets — the fine-grained, dynamically-weighted gradient updates are empirically necessary, not just theoretically elegant.

Iteration analysis (Figure 3). DensePhrases_multi starts at approximately 46.5% EM on the NQ development set (note: this differs from the test set result of 41.6% in Table 1, likely reflecting development vs. test set differences or the specific model checkpoint used). The re-ranker baseline (flat line, since re-ranking doesn't iterate) achieves approximately 48.5% — roughly a 2-point gain over the base retriever. PRF_Rocchio peaks at iteration 1 (~47.0%, +0.5 over baseline) but degrades at iteration 2 (~46.8%) and stabilizes — confirming the saturation problem. TOURhard without score aggregation starts at iteration 0 (base retrieval) at 46.5%, drops to ~46.7% at iteration 1 (below the re-ranker, because the query vector update loses the re-ranker's direct scoring benefit), but recovers to ~48.0% at iteration 3 and surpasses the re-ranker. TOURhard with score aggregation starts higher (since the λ = 0.1 interpolation already includes re-ranker signal) and improves from ~48.2% at iteration 0 to ~49.0% at iteration 3 — a continuous improvement trajectory, unlike PRF_Rocchio. This figure directly validates the paper's central claim: gradient-based optimization with fine-grained re-ranker feedback enables sustained improvement across multiple iterations, while uniform-weight PRF saturates immediately.

Latency-performance trade-off (Figure 2). On the NQ development set, when varying top-k from 10 to 50: DensePhrases achieves ~41.8% EM at ~100ms (fastest, lowest accuracy). The re-ranker baseline (blue line) traces a curve from ~47.2% at k = 10 (~240ms) to ~48.2% at k = 50 (~1200ms) — diminishing returns after k = 20. TOURhard (red line) achieves ~48.6% at k = 10 (~440ms) and improves to ~49.2% at k = 30 (~750ms), consistently above the re-ranker curve at all latency points. Critically, TOURhard at k = 10 (440ms) achieves better performance than the re-ranker at k = 40 (1040ms, ~47.8%) — TOUR is both 2.4× faster and more accurate. TOURhard without the efficient implementation techniques (§3.5) traces a parallel but slower curve (e.g., ~48.6% at ~800ms for k = 10), confirming that caching and early stopping provide approximately 1.8× speedup at k = 10. The figure visually demonstrates that TOUR shifts the entire latency-performance Pareto frontier upward and leftward — for any given latency budget, TOUR achieves higher accuracy than direct re-ranking.

End-to-End Open-Domain QA: Query Distribution Shift

Headline results (Table 2). When all models are trained only on Natural Questions (NQ) and evaluated on unseen query distributions, TOURhard with k = 20 provides the largest and most consistent improvements. Averaging across the five unseen datasets (TriviaQA, WebQuestions, CuratedTrec, SQuAD, EntityQuestions), TOURhard,k=20 improves DensePhrases_NQ by approximately 6.5 percentage points (from an average of ~28.8% to ~35.3%, computed across the five non-NQ columns — TriviaQA: +9.5, WebQuestions: +4.4, CuratedTrec: +6.2, SQuAD: +6.7, EntityQuestions: +5.9). The base DensePhrases_NQ suffers dramatically on SQuAD (15.4%) and EntityQuestions (22.4%), confirming severe distribution shift. TOURhard recovers substantially: SQuAD improves to 22.1% (+6.7) and EntityQuestions to 28.3% (+5.9).

Comparison against the re-ranker baseline under distribution shift. The paper's phrase re-ranker with k = 40 already provides large gains over DensePhrases_NQ (e.g., SQuAD: 15.4% → 21.4%, +6.0; EntityQuestions: 22.4% → 27.1%, +4.7). TOURhard with k = 20 further improves upon the re-ranker_k=40 by 0.7 points on SQuAD (22.1% vs. 21.4%) and 1.2 points on EntityQuestions (28.3% vs. 27.1%), while running 1.3× faster (0.78 s/q vs. 1.04 s/q). On TriviaQA, the re-ranker_k=40 achieves 41.7% and TOURhard,k=20 achieves 42.9% — a 1.2-point improvement. The consistent pattern is that TOUR matches or exceeds the best re-ranker configuration while using less compute, even under distribution shift where the re-ranker's generalization is stressed.

TOURhard vs. TOURsoft under distribution shift. Unlike the in-domain results where the two variants performed similarly (Table 1), TOURhard consistently outperforms TOURsoft by 0.5–1.5 points across all datasets in the distribution shift setting. For example, on WebQuestions: TOURhard,k=20 achieves 28.2% vs. TOURsoft,k=20 at 27.7% (gap: +0.5). On SQuAD: 22.1% vs. 21.7% (+0.4). On CuratedTrec: 39.8% vs. 39.8% (tie for k=20, but TOURhard,k=10 at 38.3% vs. TOURsoft,k=10 at 38.2%, a 0.1 gap). On EntityQuestions: 28.3% vs. 27.9% (+0.4). The paper explicitly notes this pattern in Section 4.2: "Interestingly, TOURhard consistently performs better than TOURsoft in this setting, which requires more investigation in the future." A plausible hypothesis (not stated in the paper) is that cross-encoder calibration degrades under distribution shift — the re-ranker's fine-grained confidence scores become less reliable, making the hard-label approach (which only needs the re-ranker to identify a rough set of relevant contexts) more robust than the soft-label approach (which relies on the exact probability distribution being accurate).

Comparison against DPR under distribution shift. The paper includes DPR_NQ as a baseline, showing 0.1% on SQuAD — essentially complete failure. DensePhrases_NQ achieves 15.4% on SQuAD, already much better than DPR_NQ, suggesting that phrase retrieval has better out-of-distribution generalization than passage retrieval with DPR. TOUR on top of DensePhrases_NQ pushes this to 22.1% — a meaningful improvement, but still far below in-domain performance (34.5% for DensePhrases_multi on SQuAD in Table 1), indicating that test-time optimization can partially but not fully close the distribution gap.

Passage Retrieval Results

Headline results (Table 3). TOUR consistently improves passage retrieval accuracy for both DensePhrases (used as a phrase-based passage retriever) and DPR. On in-domain evaluation (NQ, TriviaQA), improvements over the base retriever are moderate because the base models already achieve high accuracy (e.g., DensePhrases_multi achieves 79.8% Acc@20 on NQ, leaving limited headroom). On out-of-domain EntityQuestions, the improvements are substantially larger: DPR_multi + TOURsoft improves Acc@20 from 57.9% to 66.2% (+8.3 points), and Acc@100 from 70.8% to 72.5% (+1.7 points). For DensePhrases, TOURsoft improves EntityQuestions Acc@20 from 61.0% to 66.2% (+5.2 points) and Acc@100 from 71.2% to 72.4% (+1.2 points).

Re-ranking vs. TOUR on passage retrieval (Table 3). The off-the-shelf passage re-ranker (Fajcik et al., 2021) improves Acc@20 across all settings (e.g., DPR_multi: 79.4% → 83.6% on NQ, +4.2 points; DensePhrases_multi: 79.8% → 83.2% on NQ, +3.4 points) but never improves Acc@100 (all Acc@100 numbers are identical between the base retriever and the re-ranker — 86.5% for DPR on NQ, 85.8% for DensePhrases on TriviaQA, etc.). This is the expected behavior: re-ranking reorders the top-100 but cannot introduce new passages from outside that set. TOUR, by contrast, improves both Acc@20 and Acc@100 in every configuration. For DPR_multi on EntityQuestions: Acc@100 improves from 70.8% (baseline) to 71.9% (TOURhard, +1.1) and 72.5% (TOURsoft, +1.7). For DensePhrases_multi on NQ: Acc@100 improves from 86.0% to 86.9% (TOURhard, +0.9) and 87.0% (TOURsoft, +1.0). These improvements, while numerically smaller than the Acc@20 gains, are qualitatively important — they demonstrate that TOUR is pulling relevant passages into the top-100 that were previously outside it, which re-ranking structurally cannot do. This is direct evidence for the paper's central claim that query vector optimization enables retrieval of new relevant contexts, not just reordering of already-retrieved ones.

Why Acc@100 gains are small (Table 3). The Acc@100 numbers for the base retrievers are already high: 84.8–86.5% for in-domain evaluation. This means the base retrievers already find most relevant passages within the top-100; the problem is that they are not in the top-20 (hence the large Acc@20 improvements from re-ranking and TOUR). TOUR's additional Acc@100 gains, while modest in absolute terms (0.9–1.7 points), represent recovering some of the most difficult cases — the 1–2% of queries where the relevant passage is entirely outside the initial top-100. On EntityQuestions, where base Acc@100 is lower (70.8–71.2%), TOUR's gains are larger (+1.2–1.7 points), suggesting that query vector optimization is most valuable precisely when the base retriever is weakest — a finding that aligns with the open-domain QA results where TOUR provides the largest gains on datasets with the largest distribution shift.

TOURhard vs. TOURsoft on passage retrieval. In contrast to the open-domain QA results (where TOURhard was generally equal or slightly better), TOURsoft marginally outperforms TOURhard on passage retrieval across most configurations. For DPR_multi on NQ: TOURsoft achieves 84.2% Acc@20 vs. TOURhard at 84.0% (+0.2). On EntityQuestions with DPR_multi: both achieve 65.6% Acc@20 for the hard variant and 66.2% for the soft variant (+0.6). On DensePhrases_multi with EntityQuestions: 66.2% for both. The differences are small (≤0.6 points) and inconsistent, but the slight edge for TOURsoft in this setting may reflect the passage re-ranker having better-calibrated scores than the phrase re-ranker (the passage re-ranker is a mature off-the-shelf model, while the phrase re-ranker is newly developed), making the soft labels more reliable for the KL divergence objective.

Iteration budget for passage retrieval. Unlike open-domain QA where up to 3 iterations are used, passage retrieval uses only 1 iteration (Table 7). This is a pragmatic choice given the larger top-k (k = 100 vs. k = 10–20) — each iteration requires running the cross-encoder on 100 passage candidates and performing a full retrieval pass, making multiple iterations expensive. The paper does not report whether additional iterations would further improve passage retrieval accuracy; this is a missing ablation. Given that open-domain QA benefits from up to 3 iterations (Figure 3), it is plausible that passage retrieval with more iterations (and perhaps a smaller top-k per iteration, analogous to the open-domain QA configuration) could achieve additional gains, but this experiment is not conducted.

Performance on Train-Test Overlap Splits

Headline results (Table 4). Lewis et al. (2021) identified that standard open-domain QA test sets contain questions that paraphrase training questions ("Query overlap") or share answers with training data ("Answeronly overlap"), potentially inflating performance estimates. The paper evaluates on the pre-defined splits to verify that TOUR's gains are not merely from memorizing training patterns.

Across all three overlap categories on Natural Questions: TOURhard improves over DensePhrases_multi on "Query overlap" (63.3% → 70.1%, +6.8 points), "Answeronly overlap" (33.7% → 40.3%, +6.6 points), and "None overlap" (23.9% → 33.7%, +9.8 points). The largest relative improvement is on the "None overlap" split — queries with no training overlap at all — where TOURhard achieves a 41% relative improvement over the base retriever. This strongly supports the claim that TOUR's benefits come from genuine test-time adaptation, not from exploiting training set memorization. The re-ranker baseline also improves the "None" split (23.9% → 31.0%, +7.1 points), but TOURhard adds an additional 2.7 points on top of that.

On TriviaQA: improvements follow a similar pattern — "Query overlap" (76.5% → 83.6%, +7.1), "Answeronly overlap" (46.2% → 62.3%, +16.1 — the largest absolute gain), "None overlap" (32.6% → 42.2%, +9.6). On WebQuestions: "Query overlap" shows a slight regression from the re-ranker (70.8% → 73.4% for re-ranker, then TOURhard at 70.1% — 3.3 points below the re-ranker), which the paper attributes to "unnecessary exploration" on overlapping queries. However, TOURhard still improves over the base retriever on all other splits: "Answeronly overlap" (39.5% → 48.5%, +9.0) and "None overlap" (27.5% → 33.0%, +5.5). The paper's observation that "our finding on the effectiveness of query optimization is similar to that of Mao et al. (2021), while our approach often improves performance on query overlap cases" is partially true — TOURhard does improve "Query overlap" on NQ and TriviaQA, but on WebQuestions it underperforms the re-ranker on this split, suggesting that the adaptivity of TOUR can sometimes hurt when the initial retrieval is already correct and optimization moves the query vector away from the good representation.

Ablation Studies and Robustness Checks

Hard label selection method (Table 5): Replacing TOURhard's cumulative probability thresholding with uniform top-k′ selection (k′ = 3) reduces NQ development set performance from 48.4% to 46.1% — a 2.3-point drop that validates the importance of fine-grained, re-ranker-confidence-aware pseudo-positive selection over the classical PRF approach.

Optimization algorithm (Table 5): Replacing gradient descent (SGD with momentum) with direct interpolation between the query vector and the average of pseudo-positive contexts — analogous to the original Rocchio update rule with β = 0.3 — drops performance from 48.4% to 48.2%, a small but consistent 0.2-point decline that suggests gradient-based optimization provides marginal benefits over simpler weighted averaging for this task, at least at the tested configuration.

Score aggregation weight λ (Table 5): Setting λ = 0 (using only retrieval similarity for final ranking, no re-ranker score interpolation) reduces performance to 48.1% (−0.3). Setting λ = 1 (using only re-ranker scores, ignoring retrieval similarity) reduces performance to 48.0% (−0.4). The optimal λ = 0.1 provides a small but consistent benefit, indicating that both signals contribute complementary information — the re-ranker provides fine-grained relevance calibration, while the retrieval similarity provides robustness to re-ranker errors on out-of-domain contexts that the re-ranker hasn't seen during training.

TOURhard vs. TOURsoft (Table 5): Switching from hard to soft labels reduces NQ development set performance from 48.4% to 47.7% (−0.7), consistent with the test set pattern where TOURsoft slightly underperforms TOURhard on open-domain QA (Table 1: 47.9% vs. 48.6% for k = 10).

Multiple iterations (Figure 3): Already discussed in the main results — TOURhard with score aggregation improves from ~48.2% at iteration 0 to ~49.0% at iteration 3 on the NQ development set, while PRF_Rocchio peaks at iteration 1 (~47.0%) and degrades. TOURhard without score aggregation initially underperforms the re-ranker (dropping at iteration 1) but recovers and surpasses it by iteration 3, demonstrating that the query vector optimization itself provides benefits that compound with iterations, independent of the score aggregation boost. The fact that PRF_Rocchio degrades after iteration 1 while TOURhard continues to improve is the strongest evidence for the paper's theoretical claim that dynamic, gradient-based weighting prevents the saturation that plagues uniform-weight PRF.

Efficiency techniques (Figure 4): The ablation of caching and early stopping on query latency shows that without either optimization, latency increases linearly with iterations (from ~500ms at 1 iteration to ~1,700ms at 5 iterations). Adding caching alone reduces 5-iteration latency to ~1,300ms — a 24% reduction, confirming that many contexts recur across iterations and re-computing their re-ranker scores is wasteful. Adding early stopping alone reduces 5-iteration latency to ~600ms — a 65% reduction, indicating that most queries terminate well before 5 iterations because the top-1 retrieval result becomes pseudo-positive. Adding both techniques flattens the curve almost entirely (5-iteration latency ~500ms, similar to 1-iteration latency), meaning that the amortized cost per query is close to the single-iteration cost. This ablation directly supports the paper's claim that TOUR can run faster than direct re-ranking: without these techniques, TOUR would be substantially slower (5 iterations at 1,700ms vs. re-ranker at 1,040ms for k = 40), making the comparison unfavorable.

Phrase re-ranker design choices (Table 6, Appendix D): Removing document titles from the re-ranker input drops performance from 45.4% to 44.8% on NQ (−0.6) — titles provide useful document-level context for disambiguation. Switching from RoBERTa-large to RoBERTa-base drops performance to 43.2% (−2.2) — model capacity matters substantially for fine-grained phrase relevance judgment. Reducing context from 3 sentences to 1 sentence drops to 43.6% (−1.8) — the surrounding sentences provide necessary context beyond the immediate phrase-bearing sentence. Using full paragraphs instead of 3 sentences yields 45.6% (+0.2) but doubles latency — the marginal accuracy gain does not justify the cost. Replacing reading comprehension pre-training with MNLI pre-training drops to 43.8% (−1.6), and removing all pre-training drops to 42.0% (−3.4) — reading comprehension pre-training is crucial for the phrase re-ranker to learn answer-span identification before fine-tuning on phrase retrieval triplets. These ablations justify each design choice in the phrase re-ranker architecture and demonstrate that the re-ranker's quality — which TOUR depends on for pseudo labels — is sensitive to careful input formatting and pre-training alignment with the downstream task.

Relevance labeler quality for TOUR (Table 5, via the C^q_hard vs. C_{1:3} ablation): The experiment replacing TOURhard's cross-encoder-guided pseudo-positive selection with uniform top-3 selection (46.1% vs. 48.4%, −2.3 points) is not just an ablation of the selection mechanism — it quantifies the value of the re-ranker's fine-grained relevance signal. The 2.3-point gap represents the marginal benefit of using a trained cross-encoder to identify pseudo-relevant contexts rather than relying on retrieval rank as a proxy. This is a substantial fraction of TOUR's total improvement over the baseline (48.4% − 41.6% = 6.8 points on NQ test, or 48.4% − 41.3% ≈ 7.1 points on the NQ development set per Table 5's baseline), indicating that the re-ranker's quality is responsible for roughly one-third of TOUR's total gain. The remaining two-thirds come from the iterative gradient-based optimization framework itself — how the pseudo labels are used, not just their quality.

Critical Assessment

The experiments collectively provide strong evidence that TOUR improves dense retrieval performance across multiple datasets, retrieval paradigms (phrase and passage), and evaluation settings (in-domain, distribution shift, train-test overlap splits). However, the strength of evidence varies substantially across the paper's specific claims, and several experimental choices limit the generality of the conclusions.

Claim: TOUR improves end-to-end QA accuracy by up to 10.7% and passage retrieval by up to 8.3%. The claim is numerically accurate — SQuAD sees +10.7 points for open-domain QA (Table 1: 34.5% → 45.2% with TOURhard, though note this is +10.7 percentage points, not a 10.7% relative improvement). EntityQuestions passage retrieval with DPR sees +8.3 points Acc@20 (Table 3: 57.9% → 66.2%). However, the "up to" framing masks substantial variance: on in-domain NQ, the improvement is +7.0 points; on WebQuestions, +5.4 points. The largest gains occur on datasets where the base retriever performs worst (SQuAD, EntityQuestions), consistent with TOUR being most valuable when the initial query representation is poor. The claim is narrow in one important respect: it is only demonstrated for open-domain QA with Wikipedia as the corpus. The BEIR results mentioned in the Limitations section (48.1% macro-averaged NDCG@10) suggest TOUR's benefits generalize to other retrieval tasks, but these results are not presented in detail (no per-task breakdown, no comparison to state-of-the-art BEIR methods, no ablation). The paper would be stronger with a full BEIR evaluation in the main results rather than a one-sentence mention in Limitations.

Claim: TOUR runs 1.3–2.4× faster than direct re-ranking. The claim is supported by specific comparisons (Table 1: TOURhard,k=20 at 0.78 s/q vs. re-ranker_k=40 at 1.04 s/q = 1.3×; TOURhard,k=10 at 0.44 s/q vs. re-ranker_k=40 at 1.04 s/q = 2.4×; Figure 2 shows TOURhard,k=10 at 440ms vs. re-ranker_k=40 at 1,040ms). However, the speedup depends critically on the early stopping and caching mechanisms (Figure 4 shows 5-iteration latency drops from ~1,700ms to ~500ms with both optimizations), and the paper does not report the distribution of iterations across queries — what fraction stop at iteration 1 vs. 2 vs. 3? The speedup also depends on the specific top-k comparison. TOUR is faster than direct re-ranking with a larger top-k needed to match TOUR's accuracy. If the comparison were TOURhard,k=20 vs. re-ranker_k=20 (both using the same k), the latency gap would narrow since both run the cross-encoder on the same number of candidates — TOUR's advantage would come only from early stopping reducing the average number of re-ranker passes. The paper doesn't report this comparison. The latency measurements also lack hardware specification (GPU model, batch size, whether retrieval is on CPU or GPU), making it difficult to replicate or compare against other systems.

Claim: TOUR generalizes the Rocchio algorithm. The theoretical derivation (Appendix A, C) is mathematically sound — under the specified conditions, the TOURhard gradient reduces to the Rocchio update. The empirical evidence for the practical significance of this generalization is strong: PRF_Rocchio provides negligible improvement over the baseline (Table 1: +0.0 to +0.4 points), while TOUR provides large gains (+5.4 to +11.9 points). Figure 3 directly visualizes the mechanism: PRF_Rocchio saturates at iteration 1, TOURhard improves through iteration 3. This suggests the generalization matters — the dynamic weighting derived from the gradient of the marginal likelihood objective genuinely prevents the saturation problem. However, the paper does not isolate which aspect of the generalization is most responsible. Is it the use of re-ranker scores for pseudo-positive selection (C^q_hard vs. C_{1:k′})? The adaptive per-context weighting (the P(˜c|qt)(1 − P_k(˜c|qt)) factor)? The iterative re-weighting at each step? The ablation in Table 5 shows that replacing C^q_hard with C_{1:3} costs 2.3 points, suggesting the selection mechanism accounts for roughly a third of TOUR's improvement. The remaining contribution from dynamic weighting and iteration is unquantified — a missing ablation that would directly test the theoretical claim.

Claim: TOUR improves performance on unseen query distributions. Supported by Table 2, where TOURhard trained only on NQ improves DensePhrases_NQ by 5.5–9.5 points across multiple unseen datasets. This is genuine generalization — the re-ranker, the query encoder, and the context index were all trained on NQ data, yet the test-time optimization improves retrieval on structurally different queries from other datasets. The train-test overlap analysis (Table 4) further confirms that gains are not driven by memorization, with the largest relative improvements on the "None overlap" split. However, the generalization is within the same task (open-domain QA over Wikipedia) — the queries come from different datasets but all ask factoid questions answerable from Wikipedia. The paper does not test generalization to genuinely different tasks (e.g., biomedical retrieval, legal document retrieval, multilingual retrieval) except for the brief BEIR mention. The distribution shift experiment also uses a specific base model (DensePhrases_NQ) with a specific weakness (poor generalization); it is plausible that a base retriever with better inherent generalization (e.g., a model pre-trained on more diverse data) would leave less room for TOUR to improve, reducing the apparent benefit. The paper does not test TOUR on top of a strong zero-shot retriever.

Concerns about test set size and statistical reliability. The paper does not report confidence intervals, standard deviations, or statistical significance tests for any result. This is particularly concerning for the smaller test sets: CuratedTrec has only 694 test questions, WebQuestions has 2,032. A 1–2 point difference on CuratedTrec (e.g., TOURsoft_k=20 at 62.5% vs. TOURhard_k=20 at 62.5% in Table 2) could easily fall within sampling noise. The cross-validation protocol is minimal — hyperparameters are tuned on the development set, but no cross-validation is used for the final test set evaluation. The paper would benefit from bootstrap confidence intervals or multiple-run variance estimates, especially given the relatively small absolute differences between TOURhard and TOURsoft (often ≤1 point) and between different k values.

Missing comparison: TOUR vs. simply using a larger k for the re-ranker. The paper frames TOUR as more efficient than direct re-ranking because it achieves better accuracy with a smaller k (e.g., TOURhard,k=10 vs. re-ranker_k=40). But what if the baseline retriever's top-k is increased to, say, k = 100 or k = 200, and the re-ranker is applied directly? At some sufficiently large k, the re-ranker would achieve the recall that TOUR achieves through query vector optimization. The paper does not explore this saturation point — how large must k be for direct re-ranking to match TOUR's accuracy, and what is the latency at that k? If direct re-ranking with k = 60 already matches TOURhard,k=10 at comparable latency, the efficiency argument weakens. Figure 2 shows the re-ranker curve still rising at k = 50, suggesting it hasn't saturated, but the comparison stops there.

Missing comparison: TOUR vs. distilling the re-ranker into the retriever. Methods like RocketQAv2 (Ren et al., 2021) use cross-encoder signals to fine-tune the retriever on the full training set, improving the encoder's parameters for all future queries. TOUR operates at test time without modifying the encoder. A fair comparison would be: fine-tune DensePhrases on NQ training data using RocketQAv2-style distillation, then evaluate on the test sets, and compare against DensePhrases_NQ + TOUR. If distillation achieves similar or better gains, the case for test-time optimization (with its per-query latency cost) weakens. The paper acknowledges this distinction in Section 2.2 but does not empirically compare against distillation-based methods. This is a significant omission, as it leaves open the question of whether TOUR's test-time compute could be better spent on one-time training-time distillation.

Missing ablation: sensitivity to the number of difficulty bins (N/A for this paper — TOUR does not use difficulty estimation). Unlike the reference example paper which uses difficulty bins, TOUR applies the same optimization procedure to all queries (with the same hyperparameters). The paper does not analyze whether TOUR's gains are concentrated on certain query types (easy vs. hard, based on initial retrieval quality) or uniform across all queries. A per-query analysis — do queries where the initial top-1 is already correct benefit, stay the same, or degrade? — would reveal whether TOUR's optimization sometimes hurts already-good queries (as suggested by the WebQuestions "Query overlap" regression in Table 4). This is an important missing analysis for practical deployment: if TOUR improves hard queries but occasionally degrades easy ones, a selective application strategy (apply TOUR only when the re-ranker disagrees with the initial top-1) might be more robust than universal application.

Missing experiment: combining TOUR with query-side fine-tuning. The paper positions TOUR as an alternative to query-side fine-tuning (Lee et al., 2021a), but they are complementary: query-side fine-tuning adapts the encoder to a new domain, and TOUR further optimizes each test query's representation. The natural experiment — apply query-side fine-tuning on NQ, then apply TOUR at test time — is not conducted. It is plausible that the combination would yield additive gains, since query-side fine-tuning improves the starting point (q0) and TOUR refines it further. The absence of this experiment leaves the relationship between distribution-level and instance-level adaptation unexplored.

Concerns about hyperparameter sensitivity. The paper reports hyperparameters in Table 7 but does not ablate their sensitivity. The learning rate η = 1.2 for DensePhrases is notably high (typical learning rates for embedding-space optimization are in the 0.001–0.1 range). The temperature τ = 0.5 and threshold p = 0.5 are used throughout without sensitivity analysis. The momentum (0.99) and weight decay (0.01) are also fixed. Given that TOUR involves only 1–3 gradient steps per query, the optimization is extremely sensitive to learning rate — a single step that is too large could overshoot and degrade retrieval, while a step too small provides no benefit. The paper would benefit from a learning rate sensitivity sweep (e.g., 0.1, 0.5, 1.0, 1.2, 1.5, 2.0) to show that the chosen value is near-optimal and that performance doesn't collapse with small changes. The lack of this analysis is a practical concern for reproducibility.

Concerns about the phrase re-ranker's training data construction. The re-ranker is trained on triplets sampled from the top-100 retrieval results of DensePhrases on the training queries. This creates a potential distribution mismatch: at test time, TOUR may retrieve contexts that look different from the top-100 training contexts (because the query vector has been updated), and the re-ranker may be poorly calibrated on these "off-distribution" contexts. The paper does not analyze whether re-ranker scores degrade as TOUR iterates and the retrieved set shifts. This is a subtle but important point — TOUR relies on the re-ranker to provide reliable signals even on contexts that are far from the initial retrieval results, but the re-ranker was only trained on contexts retrieved by the original (unoptimized) query encoder. If the re-ranker's accuracy drops on contexts retrieved by the optimized query vector, TOUR's later iterations might be guided by noisy signals, potentially explaining why performance saturates at iteration 3 in Figure 3. An experiment evaluating re-ranker accuracy as a function of TOUR iteration would address this concern.

Overall assessment. The experiments convincingly demonstrate that TOUR improves dense retrieval performance across the tested settings, with the strongest evidence for open-domain QA with phrase retrieval (Tables 1, 2) and passage retrieval (Table 3). The comparison against classical PRF (Figure 3) directly validates the paper's central theoretical claim — gradient-based optimization with dynamic weighting outperforms uniform-weight Rocchio. The efficiency claim is supported by latency measurements (Figures 2, 4) but would be strengthened by hardware specification and a broader range of k comparisons. The main weaknesses are: (1) lack of statistical reliability measures for small test sets, (2) missing comparisons against distillation-based methods that also leverage cross-encoder signals, (3) absence of per-query analysis to characterize when TOUR helps vs. hurts, (4) no hyperparameter sensitivity analysis despite the optimization involving only 1–3 steps with a high learning rate, and (5) the BEIR results mentioned only briefly without detailed evaluation. The paper's claims are supported for the specific configurations tested, but the generality across retrieval tasks, base model architectures, and hyperparameter choices remains incompletely characterized.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted For — The Difficulty of Knowing When TOUR Helps vs. Hurts

The assumption or constraint. The paper identifies an intriguing failure mode that it does not systematically characterize or resolve: TOUR's optimization can degrade performance on queries where the initial retrieval is already correct. Section 5.1 reports that on the WebQuestions "Query overlap" split — test queries that paraphrase training queries — TOURhard underperforms the direct re-ranker (70.1% vs. 73.4%, a 3.3-point regression). The paper attributes this to "unnecessary exploration" on overlapping queries: when the initial query vector already retrieves the correct answer, gradient-based optimization may move the query vector away from that correct region of embedding space, chasing the re-ranker's pseudo labels onto a different but ultimately incorrect retrieval trajectory. This is a specific instance of a broader problem: TOUR applies the same optimization procedure to all queries, regardless of whether the initial retrieval is already good. There is no mechanism to detect that the initial query representation is adequate and skip optimization.

The consequence. In a deployment where the query distribution contains a substantial fraction of "easy" queries — those that the base retriever already handles correctly — TOUR will waste computation on unnecessary optimization at best, and actively degrade performance at worst. The WebQuestions result demonstrates that degradation is not merely a theoretical concern: on queries with training overlap, the optimization can be counterproductive. The paper does not report how often this occurs across datasets or quantify the trade-off (how many queries improve vs. degrade vs. stay the same). A practitioner deploying TOUR needs to know: for what fraction of queries does the initial top-1 retrieval change, and how often is that change from correct to incorrect versus incorrect to correct? Without this per-query analysis, the expected net benefit on a given query stream is difficult to estimate.

What evidence exists in the paper. Only Table 4 provides direct evidence of the regression, and only for WebQuestions "Query overlap." The paper does not report per-query transition matrices (correct → correct, correct → incorrect, incorrect → correct, incorrect → incorrect) for any dataset. The train-test overlap analysis (Table 4) partitions queries by training data overlap but does not analyze queries by initial retrieval quality — a more direct proxy for whether TOUR is needed. The iteration analysis (Figure 3) shows that TOURhard without score aggregation initially drops below the re-ranker and baseline before recovering, which is consistent with some initially-correct queries being degraded at early iterations, but this is aggregate behavior and does not isolate the degradation rate on initially-correct queries specifically.

Mitigation status. The early stopping conditions partially address this issue: TOURhard stops when the top-1 retrieval result is pseudo-positive (c_1 ∈ C^{q_t}_{hard}), which should terminate early on queries where the retriever and re-ranker agree that the top result is relevant. However, this only prevents additional iterations — it does not prevent a first iteration that moves the query vector away from an already-correct answer if the re-ranker's pseudo labels happen to prefer a different (incorrect) context. The condition tests whether the current top-1 is pseudo-positive, not whether the initial top-1 was correct. A more robust approach would involve detecting that the initial retrieval is likely correct (e.g., if the re-ranker assigns high confidence to the initial top-1 and it contains a plausible answer) and skipping optimization entirely. The paper does not develop or evaluate such a mechanism. The authors do not flag this as an explicit open problem, though the observation about WebQuestions in Section 5.1 implicitly acknowledges the issue.

The Cross-Encoder Training-Query Distribution Mismatch: The Re-Ranker May Provide Noisy Pseudo Labels on Contexts Retrieved by the Optimized Query Vector

The assumption or constraint. TOUR assumes that the cross-encoder re-ranker ϕ provides reliable relevance judgments on all contexts that appear in the top-k across all iterations. However, the re-ranker is trained on contexts retrieved by the original (unoptimized) query encoder from the training queries (Section 3.4). At test time, as TOUR iteratively updates the query vector, the set of retrieved contexts shifts — the optimized query vector may retrieve contexts that are substantially different in embedding space (and potentially in textual characteristics) from those the re-ranker encountered during training. There is no guarantee that the re-ranker's calibration transfers to these "off-distribution" contexts.

The consequence. If the re-ranker's accuracy degrades on contexts retrieved by the optimized query vector, later TOUR iterations would be guided by increasingly noisy pseudo labels. This would create a self-reinforcing problem: the query vector is updated based on unreliable signals, which leads it to retrieve even more off-distribution contexts, which receive even less reliable scores. This could explain why TOURhard's performance saturates at iteration 3 in Figure 3 — the re-ranker's signal quality may degrade as the retrieved set drifts from the training distribution, causing later iterations to provide diminishing or negative returns. The ablation in Table 5 showing that replacing C^q_hard with uniform top-3 selection drops performance by 2.3 points demonstrates that the re-ranker's pseudo-label quality matters substantially. If that quality degrades during optimization, TOUR's effectiveness at later iterations would be limited, and the optimal number of iterations would depend on the re-ranker's generalization robustness.

What evidence exists in the paper. The paper does not directly measure re-ranker accuracy as a function of TOUR iteration. There is no experiment that evaluates how often the re-ranker correctly identifies relevant contexts in the top-k retrieved by q_0 versus q_1 versus q_2 versus q_3. The performance saturation at iteration 3 (Figure 3) is consistent with this degradation hypothesis but could also be explained by the optimization converging to a local optimum. No analysis distinguishes between these explanations. The paper does not report re-ranker calibration metrics (e.g., expected calibration error) on initial vs. later retrieval sets.

Mitigation status. The paper partially mitigates this by training the re-ranker on the top-100 contexts from the base retriever, which provides some coverage of the embedding space neighborhood around each training query. However, TOUR's query vector can move arbitrarily far from q_0 (limited only by weight decay with λ_decay = 0.01), potentially retrieving contexts well outside the training distribution. The weight decay itself provides a soft constraint: the query vector is pulled back toward the origin, which limits drift but does not guarantee the retrieved set stays within the re-ranker's region of reliability. The paper does not evaluate whether re-ranker scores remain calibrated at different distances from q_0, nor does it explore re-ranker fine-tuning on the optimized query's retrieval outputs as a way to close the distribution gap. This is not flagged as a limitation in the paper, but the observed iteration saturation (Figure 3) and the difference between TOURhard and TOURsoft under distribution shift (Table 2, where TOURhard — less reliant on fine-grained score calibration — outperforms TOURsoft) are consistent with re-ranker degradation being a real effect.

Single Retriever Family, Single Corpus, Single Language: The Generality Across Retrieval Architectures, Domains, and Languages Is Uncharacterized

The assumption or constraint. All main experiments use DensePhrases (a phrase retrieval model) or DPR (a passage retrieval model) as the base retriever, with Wikipedia as the corpus and English as the language. The paper explicitly tests only open-domain QA over Wikipedia, with the exception of a brief mention in the Limitations section:

"We have also experimented on the BEIR benchmark (Thakur et al., 2021) to evaluate our method in the zero-shot document retrieval task. Overall, we obtained 48.1% macro-averaged NDCG@10 compared to 47.8% by the re-ranking method."

The BEIR results are mentioned only in passing — no per-task breakdown, no comparison to BEIR-specific state-of-the-art methods, no ablation of TOUR components on BEIR tasks. Two BEIR datasets are singled out (BioASQ: +11.6% NDCG@10 over baseline; TREC-COVID: +23.8%), but the paper explicitly states "We plan to better understand why TOUR performs better specifically on these tasks."

The consequence. A practitioner considering TOUR for a retrieval system outside the Wikipedia-based open-domain QA setting — biomedical literature search (BioASQ, TREC-COVID), legal document retrieval, product search, multilingual retrieval, or any corpus with different characteristics (length, structure, domain-specific terminology) — cannot rely on the paper's results to predict TOUR's effectiveness. The BEIR macro-average of 48.1% vs. 47.8% for re-ranking (+0.3 NDCG@10) suggests that TOUR's average benefit on diverse retrieval tasks may be substantially smaller than the open-domain QA results (where gains of +5.4 to +11.9 points are reported). The large gains on BioASQ and TREC-COVID (+11.6% and +23.8%) are promising but unexplained — are these tasks where the base retriever is particularly weak? Where the re-ranker is particularly strong? Where the query distribution is especially different from the retriever's training? Without understanding the mechanism, a practitioner cannot determine whether their specific task resembles these high-gain cases or the lower-gain average.

Additionally, TOUR's dependence on a cross-encoder re-ranker means that the approach is only applicable when a re-ranker can be obtained (trained or off-the-shelf) for the target domain. For low-resource domains or specialized corpora where no re-ranker exists and training one requires labeled relevance data, TOUR adds the burden of re-ranker development to the deployment pipeline. The paper provides a recipe for training a phrase re-ranker (Section 3.4), but this requires access to training queries with ground-truth relevance labels and sufficient computational resources to train a RoBERTa-large model — a non-trivial requirement for many practical settings.

What evidence exists in the paper. The BEIR results are mentioned only in the Limitations section, without tables, figures, or per-task breakdown. The open-domain QA experiments cover six datasets, all with similar characteristics (factoid questions answerable from Wikipedia). The passage retrieval experiments (Table 3) cover only three datasets (NQ, TriviaQA, EntityQuestions) and use only two retrievers (DPR, DensePhrases). The paper does not test colBERT-style late interaction models, sparse-dense hybrid retrievers, or models with different embedding dimensionalities. The phrase re-ranker is trained specifically for DensePhrases' output distribution; its applicability to other phrase retrieval models (e.g., different phrase index construction, different training objectives) is untested.

Mitigation status. The paper explicitly acknowledges this limitation in the Limitations section and mentions ongoing work on BEIR. The BEIR results are described as preliminary ("We plan to better understand why TOUR performs better specifically on these tasks and further improve it"). The re-ranker training recipe (Section 3.4) provides a template for adapting TOUR to new domains, but the paper does not demonstrate this adaptation process — for example, training a domain-specific phrase re-ranker from scratch and showing that TOUR's benefits transfer. The dual-evaluation on both phrase retrieval (DensePhrases) and passage retrieval (DPR) provides some evidence of architectural generality, but both are dual-encoder architectures with inner-product scoring. Extension to late-interaction models, learned sparse retrieval, or multi-vector models is not discussed.

Hyperparameter Sensitivity in an Extremely Low-Data Optimization Regime: 1–3 Gradient Steps with a High Learning Rate

The assumption or constraint. TOUR applies gradient-based optimization directly to the query vector with only 1–3 steps per query and a learning rate that is notably high (η = 1.2 for DensePhrases, η = 0.2 for DPR). This is a fundamentally different optimization regime from standard neural network training: there is no mini-batching, no averaging of gradients across examples, and no opportunity for the optimizer to "recover" from a bad step. A single gradient step that is too large could move the query vector into a region of embedding space where retrieval quality collapses entirely, and there is no mechanism to detect or correct this. The paper uses momentum (0.99) and weight decay (0.01) but does not ablate these choices or analyze sensitivity to the learning rate.

The consequence. A practitioner deploying TOUR on a new retriever, corpus, or embedding dimensionality would need to tune the learning rate from scratch. The learning rate that works for DensePhrases (η = 1.2) is 6× larger than what works for DPR (η = 0.2), suggesting the optimal value is sensitive to the embedding space geometry (dimensionality, normalization, training objective). The paper provides no guidance for how to set η for a new retriever — is it proportional to embedding dimensionality? To the typical inner product magnitude? To the number of top-k contexts? Without such guidance, deployment on a new system requires a grid search over learning rates on a validation set, with the risk that values far from optimal cause performance degradation. The paper also uses a temperature τ = 0.5 for the softmax in pseudo-label generation (Equation 8) and a cumulative probability threshold p = 0.5 for TOURhard, without any sensitivity analysis — both control how the re-ranker's scores are converted into optimization targets, and poor choices could lead to degenerate updates (e.g., τ close to zero would make the pseudo-label distribution nearly one-hot regardless of the re-ranker's confidence, losing the benefit of fine-grained signals).

What evidence exists in the paper. The learning rate values appear in Table 7 without justification or ablation. The paper does not report learning rate sensitivity experiments (e.g., performance as a function of η in {0.1, 0.5, 1.0, 1.2, 1.5, 2.0}) for any dataset. The temperature τ and threshold p are stated as 0.5 in Appendix D but never varied. The momentum and weight decay values are listed without discussing their role or sensitivity. This is a significant omission because TOUR's optimization involves only 1–3 gradient steps — the typical intuition from training neural networks for thousands of steps (where learning rate can be somewhat forgiving because the optimizer has many opportunities to correct course) does not apply. The paper does not report whether performance degrades catastrophically at η = 2.0 or η = 0.5, or whether the optimal η varies across datasets within the same retriever.

Mitigation status. The paper uses a linear learning rate scheduling per iteration (mentioned in Appendix D but not detailed), which may provide some robustness by reducing the step size in later iterations. Weight decay provides a soft constraint preventing unbounded drift. Momentum adds inertia, which can smooth optimization and prevent overreaction to a single noisy gradient. However, none of these mechanisms is evaluated for its effect on hyperparameter robustness. The paper does not provide a principled method for setting the learning rate (e.g., based on gradient norm statistics, a validation procedure, or a theoretical bound). The use of development set tuning for hyperparameter selection (standard in the paper) is noted, but the sensitivity of the results to these choices is not characterized. This is a practical barrier to adoption: without sensitivity analysis, a practitioner cannot assess the risk of performance degradation from suboptimal hyperparameter choices in their specific deployment context.

The Re-Ranker Is a Hard Dependency: TOUR Requires a High-Quality Cross-Encoder, Which May Not Be Available, and Its Quality Directly Determines TOUR's Effectiveness

The assumption or constraint. TOUR fundamentally depends on a cross-encoder re-ranker to provide pseudo labels for query vector optimization. The quality of these pseudo labels directly determines the quality of the optimization: if the re-ranker provides poor relevance judgments, TOUR will optimize the query vector toward non-relevant contexts, degrading retrieval rather than improving it. The paper demonstrates this dependency implicitly — the ablation replacing cross-encoder-guided pseudo-positive selection with uniform top-3 selection drops performance by 2.3 points (Table 5) — but never evaluates how TOUR performs with a weaker re-ranker (e.g., a smaller model, fewer training examples, or a re-ranker trained on a different domain). The paper's phrase re-ranker uses RoBERTa-large (355M parameters), pre-training on reading comprehension datasets, and training on in-domain retrieval triplets (Section 3.4, Appendix D). This is a substantial computational investment.

The consequence. In any deployment setting where a high-quality cross-encoder is not available — low-resource domains, specialized corpora, latency-constrained environments where even a small re-ranker is too expensive, or settings where labeled relevance data for re-ranker training is scarce — TOUR is not directly applicable. The paper's efficiency argument (TOUR is faster than direct re-ranking) implicitly assumes that a cross-encoder is available and would be used anyway; if re-ranking was never part of the deployment plan because of cost or unavailability, TOUR introduces a new dependency rather than optimizing an existing one. Moreover, the paper provides no guidance on the minimum re-ranker quality needed for TOUR to be beneficial. Can a lightweight re-ranker (e.g., a distilled 6-layer BERT, or even a simple BM25-based scorer) provide sufficient signal? At what point does re-ranker noise overwhelm the gradient-based optimization and cause TOUR to degrade performance? The ablation in Table 6 shows that using RoBERTa-base instead of RoBERTa-large for the phrase re-ranker drops its standalone accuracy by 2.2 points (45.4% → 43.2%), but the paper does not report how this weaker re-ranker affects TOUR's performance — would TOUR with a RoBERTa-base re-ranker still outperform direct re-ranking with RoBERTa-base? Or does the weaker pseudo-label signal cause TOUR's optimization to be ineffective?

What evidence exists in the paper. The re-ranker quality ablations in Table 6 are measured only for the re-ranker's standalone accuracy, not for TOUR's performance when using that re-ranker. The paper does not run TOUR with different re-ranker configurations (base vs. large, with vs. without pre-training, 1 sentence vs. 3 sentences of context) to determine how re-ranker quality propagates through to TOUR's final performance. The paper uses an off-the-shelf passage re-ranker (Fajcik et al., 2021) for the passage retrieval experiments (Table 3), but does not characterize its quality relative to the custom phrase re-ranker or vary it to test sensitivity. The TOURsoft variant, which relies more heavily on the re-ranker's fine-grained score calibration, underperforms TOURhard under distribution shift (Table 2) — consistent with the hypothesis that re-ranker quality degrades on out-of-distribution queries and that TOUR's performance tracks re-ranker quality. However, this is an observation, not a controlled experiment varying re-ranker quality.

Mitigation status. The paper does not address re-ranker dependency as a limitation or propose alternatives to cross-encoder-based pseudo-labeling. The connection to classical PRF (Section 3.2) suggests that simpler pseudo-labeling strategies could be used (e.g., uniform top-k′ as in Rocchio), but the ablation in Table 5 shows this is substantially less effective (−2.3 points). The paper does not explore intermediate approaches: using the retriever's own similarity scores as pseudo labels (self-training), using a lightweight scoring function (e.g., BM25 on the retrieved contexts), or using an ensemble of weak re-rankers. The current design ties TOUR's effectiveness tightly to the availability and quality of a cross-encoder, making it a method for improving retrieval when a re-ranker is already available rather than a standalone retrieval improvement technique. For practitioners without an existing re-ranker, the combined cost of training a re-ranker and running TOUR may exceed the cost of alternative approaches (e.g., training a larger retriever, using query-side fine-tuning, or applying distillation-based methods that amortize the re-ranker's cost over training).

The Computational Overhead of Gradient-Based Optimization vs. The Marginal Gain: Is TOUR More Cost-Effective Than Simply Using More Re-Ranker Candidates?

The assumption or constraint. The paper's central efficiency claim is that TOUR "runs 1.3–2.4× faster" than direct re-ranking while achieving better accuracy. This comparison is between TOUR with a small top-k (e.g., k=10) and direct re-ranking with a larger top-k (k=40) — the re-ranker needs the larger k to achieve comparable accuracy, while TOUR achieves that accuracy with a smaller k by updating the query vector to find relevant contexts that would otherwise require a larger initial retrieval set. The paper does not, however, compare TOUR against the simplest possible alternative: applying the re-ranker directly on a progressively larger top-k until the accuracy matches TOUR's, and then comparing latencies at that iso-accuracy point. The re-ranker curve in Figure 2 is still rising at k=50, suggesting that direct re-ranking with, say, k=60 or k=80 might match or exceed TOURhard,k=10 at some latency point. The paper truncates the comparison at k=50 and does not explore this iso-accuracy trade-off systematically.

The consequence. A practitioner choosing between TOUR and direct re-ranking needs to know: at the accuracy level I need, which approach is faster? The paper provides evidence that TOURhard,k=10 is both faster and more accurate than re-ranker_k=40, but it does not show whether re-ranker_k=60 is faster or slower than TOURhard,k=10 while matching its accuracy. If the re-ranker's latency scales sub-linearly with k (because the cross-encoder forward pass can be batched, and the dominant cost may be fixed overhead rather than per-candidate computation), then the efficiency advantage of TOUR may shrink or disappear at higher k. The paper's caching and early stopping mechanisms (Figure 4) reduce TOUR's latency substantially, but the per-iteration cost still includes a full retrieval pass over the index — an operation whose cost depends on the index size and implementation and may dominate the total latency for large corpora. The paper's latency measurements (Tables 1, 2) include the base retriever's retrieval time, but this is constant across methods (since the base retriever is always run at least once) and does not isolate TOUR's specific overhead.

What evidence exists in the paper. Figure 2 shows the latency-accuracy trade-off for re-ranker and TOUR up to k=50. The re-ranker curve is shallower than TOURhard's — accuracy increases slowly with k for the re-ranker (from ~47.2% at k=10 to ~48.2% at k=50, +1.0% for a 5× increase in k), while TOURhard improves more with k (from ~48.6% at k=10 to ~49.2% at k=30, +0.6% for a 3× increase in k). This suggests the re-ranker may need a very large k to match TOURhard,k=10, but the paper does not project what that k would be. The paper does not report how TOUR's latency scales with k — TOURhard,k=20 is 1.8× slower than TOURhard,k=10 (0.78 vs. 0.44 s/q, Table 1), approximately linear in k, while the re-ranker scales from 0.24 s/q at k=10 to 1.04 s/q at k=40 (Table 1), which is also approximately linear. Given these linear scaling trends, the efficiency comparison at iso-accuracy depends on the slopes and intercepts of these lines, which are not characterized beyond the measured points.

Mitigation status. The paper does not frame this as a limitation — the presented comparison (TOURhard,k=10 vs. re-ranker_k=40) is the standard way to demonstrate that TOUR achieves better accuracy with a smaller k. However, the paper does not provide the iso-accuracy analysis that would let a practitioner determine the optimal configuration for their accuracy target. The efficient implementation techniques (early stopping, caching) are well-ablated (Figure 4), but their interaction with k is not explored — does early stopping occur more or less frequently at larger k? Does caching become more or less effective? The paper's framing of TOUR as a replacement for re-ranking rather than a complement leaves this trade-off unresolved. For a deployment where accuracy requirements are fixed and the question is "what is the fastest configuration that achieves this accuracy?", the paper does not provide a decision procedure for choosing between TOUR with small k and direct re-ranking with large k.

7. Implications and Future Directions

How This Work Changes the Landscape

TOUR introduces a conceptual shift in how the field thinks about the relationship between training and inference for dense retrieval. The dominant paradigm has been to treat retrieval quality as a function of encoder capacity and training data — improve the encoder, and all queries benefit uniformly. TOUR demonstrates that this framing misses a crucial dimension: the quality of a query representation is not fully determined at training time, and a small amount of instance-level computation at inference can recover performance that would otherwise require substantially larger encoders or more training data. This is not a paradigm shift on the scale of the original dense retrieval breakthrough (Karpukhin et al., 2020), but it is a significant reframing of where the performance bottleneck lies: not in the encoder's average-case behavior, but in its inability to adapt to individual query idiosyncrasies.

The evidence for this reframing is concrete. DensePhrases trained on Natural Questions achieves 15.4% on SQuAD (Table 2) — a dramatic failure of generalization. TOUR, without any additional training data or model parameter updates, recovers to 22.1% (+6.7 points) purely through test-time optimization guided by a re-ranker trained on the same NQ data. This means the base encoder already contained representations that could succeed on SQuAD — it just needed a per-query nudge in the right direction. The implication is that a substantial fraction of what the field has attributed to "distribution shift" or "model capacity limitations" may actually be instance-level misalignment that test-time optimization can address without scaling the encoder. This is a diagnostic insight, not just a method: it suggests that the marginal dollar of compute may be more effectively spent on smarter inference than on larger pretraining, at least for queries within the base model's approximate capability range.

The paper also reconciles a tension between classical information retrieval and modern neural retrieval that has existed since dense retrievers displaced BM25. Classical IR has always embraced per-query adaptation through relevance feedback (Rocchio, 1971) and query expansion — the idea that a query is not a fixed entity but an iterative process of refinement. Neural IR largely abandoned this tradition in favor of fixed encoder representations, trading adaptivity for the efficiency of pre-computed indexes. TOUR shows that this trade-off was a false choice: gradient-based query vector optimization using a cross-encoder as a teacher brings the adaptivity of classical PRF into the neural era, and does so in a way that is both principled (the Rocchio generalization in Equation 11–13) and efficient (faster than direct re-ranking, Figure 2). This reconnects two research traditions that had diverged, and it suggests that future dense retrieval systems should treat the query representation as an optimizable variable rather than a fixed output of an encoder — a perspective that opens up connections to meta-learning, test-time training, and dynamic evaluation that were previously unexplored in the IR context.

The work also redirects research attention from cross-encoder re-rankers as selectors to cross-encoders as teachers. The standard view — re-rankers reorder a fixed candidate set — has structured the research agenda around improving re-ranker accuracy, scaling to larger candidate sets, and distilling re-rankers into retrievers (Ren et al., 2021; Izacard and Grave, 2020). TOUR demonstrates that a cross-encoder can be more valuable as a signal for iterative retrieval improvement than as a final gatekeeper. The finding that TOUR with λ = 0.1 (only 10% re-ranker weight in the final score) outperforms direct re-ranking (Table 5 ablation: λ = 1 gives 48.0% vs. λ = 0.1 giving 48.4%) indicates that the re-ranker's primary value in TOUR is guiding retrieval, not selecting answers. This suggests that research on re-ranker architectures should evaluate them not just on ranking accuracy (NDCG, MRR) but on their effectiveness as pseudo-label providers for query optimization — a different and currently unevaluated criterion. A re-ranker with slightly lower ranking accuracy but better-calibrated confidence scores might be far more useful for TOUR than a state-of-the-art ranker with peaked, overconfident predictions.

Perhaps most importantly, TOUR demonstrates that the latency-accuracy Pareto frontier for retrieval can be shifted by changing how computation is allocated, not just how much computation is performed. This is the same insight that has driven research on adaptive computation in other areas of ML (e.g., dynamic network depth, early exiting), but it had not been systematically applied to retrieval query representations before TOUR. The finding that TOURhard,k=10 (0.44 s/q) outperforms re-ranker_k=40 (1.04 s/q) by 1.0% while running 2.4× faster (Table 1) is not just an efficiency gain — it is evidence that spending computation on iterative query refinement with a small candidate set is fundamentally more compute-effective than spending it on scoring a larger candidate set once. This reframes the retrieval efficiency problem from "how do we make re-ranking faster?" to "how do we allocate a fixed inference budget across rounds of retrieval and refinement?"

A limitation of this landscape assessment is that the paper's evidence base is narrow — all main experiments are on open-domain QA over Wikipedia with two retriever families (DensePhrases, DPR) and one language (English). The BEIR results mentioned in Limitations (+0.3 NDCG@10 macro-average over re-ranking) suggest that the average benefit on diverse retrieval tasks is much smaller than the open-domain QA gains, and the paper does not explain why. The landscape shift described above is therefore most clearly established for the specific setting of factoid open-domain QA with dual-encoder retrievers — its generality to other retrieval tasks, modalities, and architectures remains to be demonstrated.

Follow-Up Research This Work Enables

1. Adaptive early stopping and selective application: learning when to optimize. The paper observes that TOUR can degrade performance on queries where the initial retrieval is already correct (Section 5.1: WebQuestions "Query overlap" regression from 73.4% to 70.1%). The current early stopping conditions (stop when top-1 is pseudo-positive in TOURhard, or when top-1 has highest re-ranker score in TOURsoft) are reactive — they terminate optimization after at least one iteration has already occurred. A more principled approach would be to predict, before any optimization, whether TOUR is likely to help or hurt a given query. This is directly analogous to the difficulty estimation problem in the reference example paper, but for retrieval rather than generation. A concrete experiment: train a lightweight binary classifier on the initial retrieval results (e.g., the re-ranker's confidence in the top-1, the entropy of the re-ranker score distribution, the retrieval similarity margin between top-1 and top-2) to predict whether TOUR will change the top-1 answer and whether that change will be correct. Apply TOUR only when the classifier predicts a benefit. Evaluate on the train-test overlap splits from Lewis et al. (2021) to measure whether selective application recovers the WebQuestions "Query overlap" regression while preserving gains on "None overlap." This would directly address the most important practical limitation identified in Section 6 — the inability to distinguish queries that need optimization from those that don't — and could substantially improve TOUR's average-case performance by avoiding detrimental optimization on already-good queries.

2. Re-ranker quality thresholds: how good does the teacher need to be? The paper demonstrates that TOUR relies on cross-encoder quality (the C^q_hard vs. C_{1:3} ablation in Table 5 shows a 2.3-point drop), but never characterizes the minimum re-ranker quality needed for TOUR to be beneficial. This is a critical practical question: in low-resource domains where a RoBERTa-large cross-encoder cannot be trained, can a smaller re-ranker (e.g., distilled 6-layer BERT, or even a BM25-based scorer) provide sufficient signal? A concrete experiment: systematically degrade the phrase re-ranker by (a) reducing model size (RoBERTa-large → base → small → tiny), (b) reducing training data (100% → 50% → 25% → 10% of training queries), and (c) ablating pre-training (reading comprehension → MNLI → no pre-training). For each degraded re-ranker, measure both its standalone ranking accuracy (Acc@1, Acc@5 on the top-10 retrieval results) and TOUR's end-to-end performance when using that re-ranker. Plot TOUR performance against re-ranker quality to identify the threshold below which TOUR degrades rather than improves retrieval. This would provide deployment guidance and could reveal whether the re-ranker's ranking accuracy or its calibration (expected calibration error) is more predictive of TOUR performance — an insight that would inform re-ranker design for test-time optimization specifically.

3. Multi-iteration passage retrieval and the k-iteration trade-off. The paper applies TOUR with only 1 iteration for passage retrieval (Table 7), motivated by computational budget since k = 100. But the open-domain QA results show that iterations 2–3 provide meaningful additional gains (Figure 3). A natural experiment: for passage retrieval, fix the total re-ranker call budget (e.g., 100 calls total) and vary the allocation across iterations — 1 iteration with k = 100, 2 iterations with k = 50 each, 3 iterations with k = 33 each, 4 iterations with k = 25 each. This tests whether the benefit of iterative refinement outweighs the cost of using a smaller k per iteration (which might miss relevant contexts that a larger initial k would capture). The hypothesis (from open-domain QA) is that 2–3 iterations with a moderate k should outperform a single iteration with a larger k, because the query vector improvement from early iterations pulls in new relevant contexts that the larger single-pass k might still miss if they are far from the initial query vector. This experiment would establish a compute-optimal iteration budget for passage retrieval and reveal whether the iteration-vs-k trade-off follows the same pattern across retrieval granularities.

4. Cross-encoder calibration under query vector drift: does the teacher's accuracy degrade as the student moves? Section 6 identifies a theoretical concern: the cross-encoder re-ranker is trained on contexts retrieved by the original query encoder, but TOUR's query vector updates may retrieve contexts from regions of embedding space where the re-ranker's calibration is poor. This has never been measured. A concrete experiment: for a set of test queries, run TOUR for 5 iterations. At each iteration, record the retrieved top-10 contexts and compute the re-ranker's accuracy on those contexts (using ground-truth relevance labels from the QA dataset). Plot re-ranker accuracy (e.g., P@1, NDCG@10) as a function of TOUR iteration. Additionally, measure the cosine distance between the current query vector q_t and the initial query vector q_0 and correlate this distance with re-ranker accuracy degradation. If re-ranker accuracy degrades substantially by iteration 3 (consistent with the performance saturation in Figure 3), this would confirm the distribution-shift hypothesis and motivate re-ranker fine-tuning on TOUR's own iterative outputs — that is, after training the initial re-ranker on base-retriever outputs, run TOUR on the training queries and collect the contexts retrieved at each iteration, then fine-tune the re-ranker on these expanded retrieval sets. This would create a re-ranker that is calibrated for the full trajectory of query vectors that TOUR explores, potentially enabling more iterations before saturation.

5. Instance-level optimization as a bridge to self-improvement: distilling TOUR's optimized query vectors back into the encoder. TOUR optimizes query vectors directly in embedding space, but the optimized vectors are discarded after the query is answered — they do not improve the encoder for future queries. This is a missed opportunity. A concrete follow-up: after running TOUR on a set of test queries, collect the initial query vectors q_0 (from the frozen encoder) and the optimized query vectors q_T (after TOUR's final iteration). Use these as (input, target) pairs to fine-tune the query encoder E_q, minimizing the MSE between E_q(query_text) and q_T. This distills the per-query optimization back into the encoder's parameters, so that future queries similar to the optimized ones start from a better initial representation. Evaluate whether this self-distillation loop improves the base retriever's performance on held-out queries, and whether the improvement compounds over multiple rounds (i.e., use the improved encoder to generate new q_0, run TOUR again, distill again). This connects TOUR to the self-improvement literature (e.g., STaR, ReST) and addresses a fundamental limitation of instance-level optimization: it treats each query independently and learns nothing transferable. The paper's modular design (TOUR works with any encoder) makes this experiment straightforward to implement.

6. TOUR for multi-vector and late-interaction retrievers: does test-time optimization generalize beyond dual encoders? All TOUR experiments use dual-encoder architectures with single-vector query representations (DensePhrases, DPR). ColBERT (Khattab and Zaharia, 2020) uses multiple query vectors (one per token) with late interaction against multiple context vectors. In this setting, "optimizing the query representation" is not a single-vector update — it involves updating an entire set of token-level embeddings. The paper's theoretical framework (the Rocchio generalization in Equation 11) is specific to single-vector inner-product retrieval. A concrete experiment: adapt TOUR to ColBERT by applying the gradient-based update to each token-level query vector independently (using the same pseudo labels from a cross-encoder, but computing the loss over the late-interaction similarity scores). Measure whether the token-level optimization provides gains comparable to those observed for DPR, and whether certain query tokens benefit more from optimization (e.g., content words vs. function words). Additionally, test whether the optimized multi-vector representation is interpretable — do the updated token vectors remain close to semantically related tokens, or do they drift into opaque regions of the embedding space? This experiment would determine whether TOUR's approach is fundamentally tied to single-vector dual encoders (a significant limitation) or extends to the broader class of dense retrieval architectures. A negative result (TOUR doesn't work for ColBERT) would be equally informative, as it would clarify the boundary conditions of test-time query optimization and motivate architecture-specific approaches for multi-vector models.

Practical Applications and Downstream Use Cases

1. Cost-efficient batch inference for open-domain QA over static corpora. Organizations that need to answer large volumes of factoid questions over a fixed document collection (e.g., customer support with a knowledge base, legal document QA, biomedical literature search) can deploy TOUR to improve accuracy without upgrading hardware or training larger models. The concrete benefit comes from two sources: (a) TOUR with a small top-k (10–20) achieves better accuracy than direct re-ranking with a much larger top-k (40+), reducing the number of expensive cross-encoder calls per query by 2–4× (Table 1: 0.44 s/q for TOURhard,k=10 vs. 1.04 s/q for re-ranker_k=40); (b) early stopping means that "easy" queries (those where the initial retrieval is already good) terminate after 1 iteration, while compute is automatically allocated to harder queries that need more refinement (Figure 4). The paper's latency numbers (0.44 s/q for TOURhard,k=10 on open-domain QA) are measured on unspecified hardware, but the relative speedups (1.3–2.4× over direct re-ranking) provide a concrete estimate of cost reduction. For a batch of 100,000 queries, this translates to 44,000 seconds of compute for TOUR versus 104,000 seconds for the re-ranker — approximately 2.4× cost savings with better accuracy. The deployment requirement is a trained cross-encoder re-ranker for the target domain, which the paper provides a recipe for (Section 3.4).

2. Improving zero-shot retrieval for specialized domains without in-domain training data. The query distribution shift experiments (Table 2) demonstrate that TOUR improves retrieval on unseen query distributions even when all components (retriever, re-ranker, index) are trained on a different domain (Natural Questions). For a practitioner deploying dense retrieval in a specialized domain (e.g., scientific literature, technical documentation, historical archives) where no in-domain training queries exist, TOUR provides a way to improve retrieval without any domain-specific training data — the cross-encoder re-ranker can be trained on a general-domain QA dataset (like NQ) and applied to the specialized corpus at test time. The gains on EntityQuestions (+5.9 points for TOURhard,k=20 over DensePhrases_NQ, Table 2) and the BEIR results mentioned in Limitations (+11.6% NDCG@10 on BioASQ, +23.8% on TREC-COVID) suggest that the largest benefits occur precisely in the zero-shot setting where the base retriever is weakest. The concrete deployment scenario: a biomedical researcher wants to search PubMed with natural language questions. They take DensePhrases trained on Wikipedia/NQ, build a phrase index over PubMed, and use a cross-encoder re-ranker trained on NQ. Without TOUR, retrieval accuracy is poor due to domain mismatch; with TOUR, the re-ranker's generalization ability (Rosa et al., 2022) provides pseudo labels that guide the query vector toward relevant biomedical contexts, improving retrieval without any biomedical training data.

3. Enabling phrase-level retrieval as a practical alternative to passage retrieval + reading comprehension. Phrase retrieval (DensePhrases) has theoretical advantages over passage retrieval: it provides direct answer spans rather than requiring a separate reader model, and it can be more efficient since answers are pre-indexed. However, phrase retrieval typically underperforms retriever-reader pipelines on accuracy (Table 1: DensePhrases_multi at 41.6% on NQ vs. UnitedQA-E_large at 51.8%). TOUR with the custom phrase re-ranker narrows this gap significantly (TOURhard,k=20 at 47.9% on NQ vs. DPR_multi + reader at 44.6% with the large variant). Combined with the latency advantage (TOUR at 0.44–0.78 s/q vs. DPR + ReConsider re-ranker at 1.21 s/q), this makes phrase retrieval with TOUR a competitive option for latency-sensitive open-domain QA deployments where a separate reader model would be too slow. The concrete deployment scenario: a voice assistant that needs to answer factoid questions within 500ms end-to-end. A passage-retrieval-then-reading pipeline would exceed the latency budget; DensePhrases + TOURhard,k=10 at 440ms fits within it while achieving accuracy competitive with much slower systems.

4. Improving retrieval in retrieval-augmented generation (RAG) pipelines. RAG systems use dense retrieval to provide relevant contexts to a language model for generation. The quality of generation depends critically on the quality of the retrieved contexts — if the retriever misses a key document, the generator cannot compensate. TOUR improves retrieval recall (Acc@100 improvements in Table 3: +0.9–1.7 points) and top-20 precision (Acc@20 improvements: +1.6–8.3 points), which directly translates to better context提供给 the generator. The concrete benefit: for a RAG system answering questions over a technical documentation corpus, replacing the base DPR retriever with DPR + TOUR increases the probability that the correct answer-containing passage appears in the top-20 retrieved contexts from 57.9% to 66.2% on EntityQuestions-like queries (Table 3, out-of-domain). This means the generator receives the correct context for 8.3% more queries — a substantial improvement in the information available for generation. Since TOUR is applied at query time and does not require modifying the index or the generator, it can be added to existing RAG pipelines as a drop-in improvement. The main deployment consideration is the added latency (TOUR takes 0.44–0.78 seconds per query for open-domain QA, Table 1), which may be acceptable for batch generation or asynchronous applications but not for real-time interactive use.

When to Prefer This Method

The paper implicitly positions TOUR against two alternatives: (1) direct cross-encoder re-ranking on a larger top-k, and (2) query-side fine-tuning of the encoder on in-domain training data. The trade-offs are clear from the experimental results, though the paper does not articulate them as explicit decision rules.

Prefer TOUR over direct re-ranking when:

• You already have a cross-encoder re-ranker (or can train one using the paper's recipe) and are currently applying it to a large top-k to achieve adequate recall. TOUR with a smaller top-k (10–20) will likely match or exceed the larger-k re-ranker's accuracy while running 1.3–2.4× faster (Table 1, Figure 2).
• The base retriever has reasonable but imperfect recall — enough that relevant contexts are in the top-100 but not consistently in the top-20 (Table 3: baseline Acc@100 is 71–87%, but Acc@20 is 58–80%). TOUR's query vector updates pull relevant contexts from positions 21–100 into the top-20, which direct re-ranking cannot do (it can only reorder within the retrieved set).
• Test queries come from a different distribution than the training data (Table 2), and you cannot afford to fine-tune the encoder on in-domain data. TOUR's per-query optimization adapts to distribution shift without any training data.

Prefer direct re-ranking when:

• Latency requirements are extremely tight (e.g., < 100ms per query) and the base retriever's top-k already contains the relevant context with high probability. TOUR adds 0.35–0.44 seconds of overhead on top of the base retriever (Table 1: TOURhard,k=10 at 0.44 s/q vs. PRF_Rocchio at 0.09 s/q, suggesting DensePhrases base retrieval takes approximately 0.09 s/q). If the base retriever's Acc@20 is already very high, direct re-ranking with a small k (and lower latency) may be sufficient.
• The cross-encoder re-ranker is poorly calibrated or trained on a significantly different domain than the test queries. Under distribution shift, TOURhard (which makes binary pseudo-relevance decisions) is more robust than TOURsoft (Table 2), but both rely on the re-ranker providing non-random signals. If the re-ranker is worse than random, TOUR will actively degrade retrieval.
• You are willing to invest in one-time training cost to improve all future queries. Methods like RocketQAv2 (Ren et al., 2021) that distill cross-encoder signals into the retriever's parameters amortize the re-ranker's cost over training and require no per-query optimization at test time. For high-volume deployments, distillation may be more cost-effective than TOUR's per-query overhead.

Prefer query-side fine-tuning when:

• You have access to in-domain training queries with relevance labels. Query-side fine-tuning (Lee et al., 2021a) adapts the encoder to the new domain and benefits all test queries at no per-query cost. TOUR can be applied on top of a query-side fine-tuned encoder for additional gains (this combination is not evaluated in the paper but is a natural extension).
• The distribution shift is systematic (e.g., all test queries share a different vocabulary or phrasing pattern compared to training). Query-side fine-tuning can learn the systematic mapping; TOUR addresses per-query idiosyncrasies. The two approaches are complementary and address different levels of the distribution-shift problem.

Prefer TOUR over query-side fine-tuning when:

• No in-domain training data is available (zero-shot setting, Table 2, EntityQuestions in Table 3). TOUR requires no training queries — only a cross-encoder re-ranker, which can be trained on a different domain's data.
• The retriever's index is pre-built and immutable, preventing encoder fine-tuning that would require re-encoding the corpus. TOUR modifies only the query vector, leaving the context index unchanged.
• Individual test queries vary widely in how well the encoder represents them, and per-query adaptation is needed even within a single domain. TOUR's instance-level optimization addresses query-level variance that distribution-level fine-tuning averages over.