WebGPT: Browser-Assisted Question-Answering with Human Feedback

ArXiv: 2112.09332

🎯 Pitch

A GPT-3 model learns to browse the web, gather sources, and compose answers that beat human-written Reddit explanations 69% of the time—not by improving search, but by training on human preferences with rejection sampling.

1. Executive Summary

This paper introduces a system that fine-tunes GPT-3 to answer long-form questions by interacting with a text-based web-browsing environment, allowing the model to search the web, navigate pages, and collect references in support of its answers. The approach combines behavior cloning (supervised fine-tuning on human demonstrations of browser use) with rejection sampling against a reward model (generating multiple answers and selecting the one that maximizes a learned human-preference score), trained on the ELI5 dataset of Reddit questions. The best model—a 175B-parameter variant using rejection sampling with 64 candidates—produces answers that human evaluators prefer 56% of the time to those written by human demonstrators using the same browser, and 69% of the time to the highest-voted Reddit answer, establishing that fine-tuning with human feedback can achieve human-competitive performance on open-ended question-answering when the model can consult live web sources.

2. Context and Motivation

The Core Problem: Answering Open-Ended Questions Requires Both Retrieval and Synthesis

The fundamental problem this paper addresses is long-form question-answering (LFQA) — generating paragraph-length responses to open-ended questions like those on the "Explain Like I'm Five" (ELI5) subreddit. Unlike short-form QA (e.g., SQuAD, TriviaQA), where answers are typically a few words or a single entity extracted from a known document, LFQA demands two distinct capabilities working in concert: (1) information retrieval — finding relevant, trustworthy sources that address the question, and (2) synthesis — composing a coherent, accurate, and useful explanation from those sources.

The paper opens by stating that LFQA systems "have the potential to become one of the main ways people learn about the world, but currently lag behind human performance." This is not hyperbole. If an AI system can reliably answer arbitrary factual questions with accurate, well-cited explanations, it transforms how people access knowledge — moving from keyword search and link-following to direct, synthesized answers backed by verifiable sources. The gap between human and machine performance on this task represents a significant barrier to that vision.

Why Prior Approaches Fall Short

The paper identifies two broad categories of existing work, each of which addresses only part of the problem in ways that limit end-to-end optimization:

Retrieval-focused approaches (REALM, RAG, DPR). The dominant paradigm prior to WebGPT was the "retrieve-then-read" architecture. Dense Passage Retrieval (DPR) trains a retriever to find relevant documents using inner product search in an embedding space ($p(\text{passage} \mid \text{query}) \propto \exp(\text{embed}(\text{passage}) \cdot \text{embed}(\text{query}))$), while Retrieval-Augmented Generation (RAG) and REALM train the retriever and answer-generation components end-to-end using language modeling objectives. The theoretical appeal is clear: by making retrieval differentiable, the entire system can be optimized jointly.

But the paper identifies three concrete weaknesses in this family of methods:

1. They cannot leverage modern search engines. Differentiable retrieval relies on inner product search over a fixed, pre-indexed corpus. It cannot interface with systems like the Microsoft Bing Web Search API, which already solve the document retrieval problem at a scale and freshness that academic retrievers cannot match. The paper argues this is a missed opportunity: modern search engines are "already very powerful, and index a large number of up-to-date documents," making them a practical foundation that prior work ignored for architectural reasons.

2. They are less interpretable. When a DPR or RAG system retrieves a document, the retrieval is a black-box embedding similarity. A human cannot easily inspect why a particular source was chosen. In contrast, WebGPT's approach — issuing search queries, clicking links, scrolling, and explicitly quoting extracts — produces a fully auditable trail of how the model arrived at its answer.

3. They focus on short-form QA benchmarks. Krishna et al. (2021) applied a similar retrieval-augmented approach to ELI5 specifically and found that automated metrics like ROUGE-L are not meaningful for evaluating long-form answers — a finding the WebGPT authors cite as motivation for their choice to use human preference comparisons as the primary evaluation metric. The short-form focus of most retrieval-augmented work left LFQA underexplored.

Synthesis-focused approaches (GPT-3 prompting). On the other end, large language models like GPT-3 can generate coherent long-form text from prompts, but without retrieval, they suffer from well-documented problems: hallucination (generating plausible-sounding but false statements), stale knowledge (the model is frozen at training time), and difficulty with questions requiring very specific or obscure factual recall. The paper frames these as "imitative falsehoods" (the model reproduces common misconceptions learned from training data) and "non-imitative falsehoods" (hallucinations — statements that "look plausible at a glance" but are fabricated). Prompting alone cannot solve these; the model needs access to external, up-to-date information.

The Missing Piece: End-to-End Optimization with Human Feedback

The gap the paper identifies is not that retrieval and synthesis are unsolved individual problems — it's that no prior system combined a real search engine with a language model and optimized the entire pipeline end-to-end using human judgments of answer quality. The authors state their contribution clearly:

"Instead of trying to improve these ingredients [retrieval and synthesis], we focus on combining them using more faithful training objectives. Following Stiennon et al. [2020], we use human feedback to directly optimize answer quality."

This is a shift in perspective. Rather than treating retrieval as a differentiable component to be trained jointly, WebGPT treats the web browser as an environment that the language model learns to interact with through a text-based interface. This reframes LFQA as an agent task: the model must decide which queries to issue, which links to follow, which passages to quote, and how to synthesize those quotes into an answer. All of these decisions are optimized, indirectly, through the lens of human preference judgments.

Why Human Feedback Is Essential — and Why It's Hard

The paper builds directly on Stiennon et al. (2020), which showed that training a reward model on human comparisons and then optimizing a policy against it (via RL or rejection sampling) can improve summarization quality beyond what behavior cloning alone achieves. WebGPT extends this insight to LFQA, but the extension introduces a critical challenge: evaluating factual accuracy in long-form answers is much harder than evaluating summary quality.

When a human evaluates a summary, they can compare it to the source text. When a human evaluates an answer to an arbitrary factual question, they are being asked to judge claims that may require domain expertise, independent research, or subjective interpretation. The paper makes this difficulty explicit:

"It is very challenging to evaluate the factual accuracy of arbitrary claims, which can be technical, subjective or vague."

This is not a minor implementation detail — it is a fundamental obstacle to scaling human feedback for factual tasks. If labelers cannot reliably distinguish true from false claims, then reward model training will be noisy and the policy's factual accuracy will plateau (or degrade, if the reward model picks up on spurious signals like confident tone or citation quantity rather than correctness).

The Paper's Key Innovation for Evaluation: References as an Evaluation Scaffold

WebGPT's solution to this evaluation problem is the paper's most distinctive design choice and, arguably, its central contribution. By requiring the model to quote extracts from web pages while browsing and include them as references in the final answer, the evaluation task is transformed:

"In contrast [to evaluating factual accuracy], it is much easier to evaluate how well a claim is supported by a set of sources."

Instead of asking labelers "Is this answer factually correct?" — a question that would require them to independently research every claim — the system asks "Is each claim in this answer supported by the provided references?" This is a more tractable judgment. It is easier to specify an unambiguous procedure, it improves inter-labeler agreement, and it makes the answer verification transparent to end users.

This design choice cascades through the entire system. The model must learn to browse effectively, identify relevant and trustworthy sources, extract appropriate quotes, and compose answers that faithfully represent those sources — all behaviors that can be evaluated through the lens of reference-supported claims rather than requiring labelers to have ground-truth knowledge of every topic.

The Training Data Challenge: Bridging GPT-3's Generic Capability to Browser-Specific Behavior

A pre-trained language model like GPT-3 cannot use the web browser out of the box — it does not know the format of valid commands, the structure of the text-based environment, or the strategy of when to search versus when to scroll or quote. The paper is explicit:

"A language model pre-trained on natural language would not be able to use our text-based browser, since it does not know the format of valid commands."

This creates a bootstrap problem: the capabilities required for browser use — reading comprehension, query formulation, answer synthesis — exist as zero-shot capabilities of GPT-3, but the model needs to learn the behavioral framing of when and how to deploy those capabilities in the environment. The solution is to collect human demonstrations — examples of contractors using the browser interface to answer questions — and use them for behavior cloning (supervised fine-tuning).

But behavior cloning alone has a well-understood limitation: it trains the model to imitate demonstrations, not to optimize the actual outcome (answer quality). The paper follows Stiennon et al. (2020) in arguing that training on demonstrations alone "is unlikely to lead far beyond human performance," because the model is learning to reproduce the distribution of human behavior, including its inefficiencies and errors, rather than learning to maximize a quality signal. This motivates the second stage: collecting human comparisons (paired preferences between model-generated answers) and using them to train a reward model, which can then guide either reinforcement learning (PPO) or rejection sampling to push beyond the behavioral cloning baseline.

The Web Environment as a Research Platform, Not a Production Deployment

It is important to understand that the text-based web-browsing environment is a research tool, not a product interface. By making the environment text-based — summarizing the browser state as a written prompt and requiring the model to issue text commands — the entire interaction becomes compatible with language model fine-tuning using standard supervised and RL techniques. The environment abstracts away the complexities of HTML parsing, JavaScript execution, and visual rendering, while preserving the essential decision-making challenges: What should I search for? Which result looks most promising? Which passage should I quote? When do I have enough information to answer?

The paper's setup is also explicitly designed so that humans can perform the same task. This is crucial because it enables the collection of demonstrations (for behavior cloning) and comparisons (for reward modeling) — the two data sources that drive all training. The graphical interface for human demonstrators (Figure 1a) displays essentially the same information as the text-based interface used by the model (Figure 1b), ensuring that the demonstrations are in-distribution for what the model needs to learn.

How This Paper Positions Itself Relative to Existing Work

The paper situates itself at the intersection of several research threads, but with a distinct emphasis on end-to-end optimization via human feedback rather than architectural novelty in retrieval or generation:

• Against retrieval-augmented methods (REALM, RAG): WebGPT abandons differentiable retrieval in favor of a black-box search engine, trading architectural elegance for practical power and interpretability. The paper argues that the search engine provides scale and freshness that academic retrievers cannot match, and that the research challenge shifts from "how to retrieve" to "how to use retrieval results to answer questions."

• Against short-form QA work: The paper emphasizes that ELI5 is a different evaluation regime where automated metrics fail. The choice of human preference as the primary metric is motivated by Krishna et al. (2021)'s finding that ROUGE-L is not meaningful for this task, and the paper explicitly compares against that prior work's best model (which achieved only 23% preference against ELI5 reference answers).

• Against RL-based browsing agents: Prior work by Adolphs et al. (2021) and Yuan et al. (2019) applied RL to search and reading comprehension, treating interaction as a sequential decision problem. WebGPT extends this to full web browsing with a general-purpose language model, and crucially, combines RL with behavior cloning and rejection sampling rather than treating them as alternatives.

• Within the human-feedback paradigm (Stiennon et al., 2020): WebGPT directly adopts the reward-modeling + policy-optimization framework but adapts it for the LFQA domain. The novel challenge is making human evaluation of factual accuracy feasible, solved by the reference-collection requirement.

Why This Matters Beyond ELI5

The paper's framing in Section 6 broadens the significance beyond benchmark performance. Two concerns are central:

1. Truthfulness. As NLP systems become more widely deployed, reducing false statements becomes critical. The paper distinguishes "imitative falsehoods" (errors learned from training data) from "non-imitative falsehoods" (hallucinations). WebGPT is hypothesized to reduce both: search engine filtering and the incentive to cite reliable sources should reduce imitative falsehoods, while retrieval grounding should reduce hallucination (consistent with Shuster et al., 2021). The TruthfulQA evaluation is designed to test this hypothesis directly.

2. Transparency and accountability. By generating answers with cited references and a fully inspectable browsing trail, WebGPT makes its reasoning process auditable. Users can follow up on sources, and evaluators can judge whether claims are supported. This is a practical step toward AI systems whose factual claims can be verified by non-experts, which the paper links to broader agendas around debate (Irving et al., 2018) and recursive reward modeling (Leike et al., 2018).

However, the paper is careful to flag risks that come with these benefits: answers with citations appear more authoritative, which could lead to automation bias (overreliance on model outputs; Goddard et al., 2012), and the system may learn to cherry-pick references that labelers find convincing rather than representing a balanced assessment of evidence. These concerns are not afterthoughts — they are presented as open problems that the reference-based evaluation framework both makes visible and could potentially help address.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

WebGPT is a fine-tuned language model that acts as an agent inside a text-based web browser — it reads a description of the current web page, decides what action to take next (search, click a link, scroll, quote a passage, or finish and answer), and iterates until it has collected enough information to compose an answer with cited references. The system solves the problem of long-form question-answering by combining three capabilities — web search, reading comprehension, and answer synthesis — into a single policy that is optimized end-to-end through human feedback, so the model learns not just how to browse and answer, but how to produce answers that human evaluators judge as factually supported, coherent, and useful.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components:

1. Text-based web-browsing environment — a Python/JavaScript wrapper that takes the model's text commands, executes them against the live web (via the Microsoft Bing Web Search API and page fetching), and returns a text summary of the resulting browser state. This is the "world" the model interacts with.

2. GPT-3 base model — a pre-trained language model (760M, 13B, or 175B parameters) that provides the underlying capabilities of reading comprehension, query formulation, and text generation. It is fine-tuned, not used zero-shot.

3. Behavior cloning (BC) policy — the base model fine-tuned via supervised learning on human demonstrations of browser use. It learns to map from environment observations to valid browser commands and final answers.

4. Reward model (RM) — a scalar function (also derived from GPT-3) that takes a question, an answer, and the answer's collected references, and outputs a score predicting how much a human would prefer this answer relative to alternatives. Trained on paired human comparisons using an Elo-based cross-entropy loss.

5. Answer selection mechanism (rejection sampling or RL) — an optimization layer that uses the reward model to select or generate high-quality answers. Rejection sampling generates $n$ independent answers from the BC policy and picks the one with highest RM score. Reinforcement learning (PPO) fine-tunes the policy to directly maximize the RM score, with a KL penalty to prevent over-optimization.

Information flow: A question enters the system → the BC policy is prompted with the question and an initial browser state (empty page, no actions yet) → the model issues a sequence of commands (search, click, scroll, quote) over multiple steps, with the environment responding to each command and producing a new text summary → when the model issues an "End: Answer" command, the collected quotes and the question are fed back to the model as a new prompt, and the model generates the final answer → the reward model scores the answer (optionally, for rejection sampling or evaluation) → the final answer with references is presented to the user.

3.3 Roadmap for the Deep Dive

• First, the text-based web-browsing environment — its design, the action space, how pages are converted to model-readable text, and how browsing terminates. This is the foundation everything else builds on, since the model's entire interaction with the world goes through this interface.

• Second, the behavior cloning pipeline — how human demonstrations are collected, what the training data looks like, and how supervised fine-tuning converts GPT-3 into a browser-capable agent. This establishes the baseline policy from which all optimization starts.

• Third, the reward model — how human comparisons are collected (the annotation procedure), the Elo-based training objective, and how the reward model is used to score answers. This is the signal that drives optimization beyond imitation.

• Fourth, optimization against the reward model — both rejection sampling (best-of-$n$) and reinforcement learning (PPO), including the KL penalty mechanism and why rejection sampling outperforms RL in this setting. These are the methods that push answer quality beyond the BC baseline.

• Fifth, the training data orchestration — how the three datasets (demonstrations for BC, comparisons for RM, and the RL environment) are kept disjoint and combined into a training pipeline. This explains the practical engineering of the multi-stage training process.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and training methodology paper whose core idea is that a language model can be trained to use a web browser to answer long-form questions, and that human feedback on answer quality — enabled by requiring the model to collect and cite references — can optimize this behavior beyond what imitation of human demonstrations alone achieves.

The Text-Based Web-Browsing Environment

Why it exists. The model cannot interact with a graphical web browser (HTML, JavaScript, visual rendering), and even if it could, standard browser interactions are not formatted as natural language that a language model can process. The text-based environment solves this by converting both the browser state (what the model "sees") and the model's actions (what it "does") into a purely textual format compatible with GPT-3's input/output interface. This design choice makes the entire interaction amenable to standard fine-tuning techniques (supervised learning on text completions) without requiring any architectural changes to the language model.

The observation format (Figure 1b). At each step, the model receives a text prompt summarizing the current browser state. This prompt contains, in order:

• The question (e.g., "How can I train the crows in my neighborhood to bring me gifts?")
• A list of past actions taken so far in the episode (e.g., "Search how to train crows to bring you gifts", "Click Gifts From Crows | Outside My Window www.birdsoutsidemywindow.org", "Quote", "Back")
• The title of the current page (e.g., "Search results for: how to train crows to bring you gifts")
• The current scroll position (e.g., "Scrollbar: 0 - 11")
• The text of the current page at the current cursor location, with links formatted using special delimiters: 【link ID†link text†destination domain】 or 【link ID†link text】 if the destination domain matches the source. The link ID is a numerical index used for the click action.
• The number of actions remaining before the maximum is reached
• A prompt for the next action ("Next action")

The model has no memory of previous steps beyond what is included in the "Past actions" summary. Each step is processed independently — the full prompt is constructed fresh from the environment state at each timestep. This means the model relies entirely on the text summary for continuity; there is no recurrent state or separate memory mechanism.

The text conversion pipeline (Appendix A). When the model navigates to a web page, the environment processes the raw HTML through a series of transformations:

1. Fetching: A Node.js script fetches the HTML of the page.
2. Simplification: Mozilla's Readability.js extracts the main content, removing navigation, sidebars, ads, and other non-content elements.
3. Link formatting: Links are converted to the special 【††】 format described above, using characters that are "rare and encoded in the same few ways by the tokenizer." If the special characters appear in the original page text, they are replaced by similar alternatives to avoid confusion.
4. Image replacement: Images are converted to [Image: <alt text>] or [Image] if there is no alt text.
5. Superscript/subscript handling: Superscripts and subscripts are converted to text using ^ and _.
6. HTML-to-text: The remaining HTML is converted to plain text using html2text.
7. PDF handling: For PDFs, pdfminer.six is used to extract text.
8. Other content types: For errors, timeouts, and unsupported content types, an error message is returned.
9. Cheating prevention: Any page that contains a 10-gram overlap with the question (or reference answer, if provided) is censored — an error message is shown instead. This prevents the model from simply finding a page that already contains the answer and copying it verbatim.
10. Domain filtering: Search results and links to reddit.com and quora.com are removed to prevent the model from copying answers from those sites (since ELI5 answers come from Reddit, this prevents train-test contamination).

The action space (Table 1). The model can issue exactly eight types of commands, each performing a specific operation in the environment:

• Search <query> — Sends the query string to the Microsoft Bing Web Search API, retrieves the top results, and displays a search results page in the text format. This is how the model discovers new pages.
• Clicked on link <link ID> — Follows the link with the given numerical ID from the current page, loading the destination page through the text conversion pipeline. The link ID corresponds to the †<link ID>† marker in the formatted page text.
• Find in page: <text> — Searches for the next occurrence of the specified text on the current page and scrolls to it. The comparison is case-insensitive and ignores link markup (only the visible text of each link is searched).
• Quote: <text> — If the specified text is found on the current page, it is recorded as a reference. The page title and domain are stored alongside the quoted extract. Quoting ignores case and whitespace, and supports an abbreviated format <start text>━<end text> to save tokens when quoting long passages. Multiple quotes can be collected from different pages.
• Scrolled down <1, 2, 3> — Scrolls down the current page by the specified number of steps.
• Scrolled up <1, 2, 3> — Scrolls up the current page.
• Top — Scrolls to the top of the page.
• Back — Returns to the previous page (a browser back-button).
• End: Answer — Terminates browsing and transitions to the answering phase. The model will be prompted with the question and collected quotes to compose the final answer.
• End: <Nonsense, Controversial> — Terminates browsing without answering, used if the question is nonsensical or cannot be safely answered.

If the model generates any text that does not match one of these commands, it is considered an invalid action. Invalid actions still count toward the maximum action limit but are otherwise ignored — the environment state does not change, and the model receives a new observation with the same page state. This is an important practical detail: the model must learn the valid command format through fine-tuning, and invalid actions are a natural form of negative feedback during training (they waste actions without producing useful information).

Termination conditions. Browsing ends when any of three conditions is met: (1) the model issues an End: Answer or End: <...> command, (2) the maximum number of actions is reached (100 for evaluation, randomized between 20–100 during RL training), or (3) the maximum total length of collected references is reached. If browsing ends without an explicit "End" command and at least one reference has been collected, the model is still prompted to answer. If no references were collected, the episode is considered a failure and no answer is produced.

The answering phase. When browsing terminates (and at least one reference exists), the model is prompted with a new context, separate from the browsing history. The prompt format is:

<question>■
[1] <quote page title> (<quote page domain>)
<quote extract>■
[2] <quote page title> (<quote page domain>)
<quote extract>■
...

The model then generates the final answer as a free-form text completion. The answer is expected to be a coherent paragraph-length explanation, and the numbered references [1], [2], etc. can be cited inline in the answer text (the model learns this citation behavior from the demonstrations).

Design rationale. The text-based environment is deliberately isomorphic to what human demonstrators see in the graphical interface (Figure 1a vs. 1b). The graphical interface displays the same information — question, current page title, scrollbar position, page text with clickable links, action history — but in a human-friendly layout with buttons for actions. This isomorphism ensures that human demonstrations are directly translatable to the text format the model sees. The one exception is scrolling: humans are allowed to scroll one step at a time, while the model can issue Scrolled down 2 or Scrolled down 3 to reduce the number of actions. During demonstration collection, repeated Scrolled down 1 actions are merged to match what the model would produce.

Behavior Cloning (BC): Supervised Fine-Tuning on Human Demonstrations

What it is. Behavior cloning is supervised fine-tuning of GPT-3 on sequences of (observation, action) pairs collected from human demonstrators using the web-browsing environment. For each step in a human demonstration, the text observation that was presented to the human (after conversion) is used as the input, and the action the human took is used as the target label. The model is trained with a standard language modeling objective: maximize the probability of generating the correct command tokens given the observation prompt.

How demonstrations are collected (Appendix C). Human contractors (from Upwork and Surge AI) use the graphical interface (Figure 1a) to answer ELI5 questions by browsing the web. Contractors are "generally highly educated, usually with an undergraduate degree or higher" given the challenging nature of the task. They receive a detailed instruction document and a video explaining the task. The key instructions for demonstrations are that answers should be:

• Relevant to the question asked
• Coherent — well-structured and easy to follow
• Supported by trustworthy references — claims should be backed by quotes from reliable sources

Contractors are compensated based on hours worked, not number of tasks completed (which reduces incentives to rush), and they go through a paid trial period with manual quality checks. Each demonstration takes an average of approximately 15 minutes to complete.

During browsing, contractors have access to the same actions as the model with one exception: they cannot use multi-step scrolling commands (Scrolled down 2, 3), which exist for efficiency but are unfamiliar to humans. Repeated single-step scrolls are automatically merged in the training data.

Training data details. The bulk of demonstrations — 5,711 out of approximately 6,200 total (92%) — are for questions from ELI5. The remaining demonstrations are for questions from TriviaQA, ARC (Challenge and Easy splits), ELI5 fact-check (questions about model-generated answers), and a small set of hand-written questions. The ELI5 questions undergo post-processing: URLs are included in full (not as _URL_ tokens), questions with "[deleted by user]" titles are filtered out, the title and selftext are concatenated with a double newline, and non-question titles are prepended with "Explain: " (e.g., "Explain: gravity" rather than "gravity"). The system determines whether a title is a question by checking for a question mark or one of a list of question-like words (explain, what, who, how, why, do, can, etc.) with a regex word boundary.

The supervised learning setup. For each step in a demonstration, the training example consists of the text observation (question + past actions + current page state) as the input prefix, and the human's action command as the target completion. The model is trained to predict the action tokens given the prefix. Multiple steps from the same episode are treated as independent training examples (since the model has no memory between steps, the training is equivalent to multi-turn fine-tuning where each turn is a separate example with the history included in the text prompt).

Hyperparameters (Appendix E, Table 6). Training uses:

• Minibatch size: 512 (256 for the 760M model)
• Adam step size multiplier: 0.1 times the pre-training Adam step size (which is $2.5 \times 10^{-4}$ for 760M, $1.0 \times 10^{-4}$ for 13B, and $0.6 \times 10^{-4}$ for 175B — Table 5). This gives effective learning rates of $2.5 \times 10^{-5}$, $1.0 \times 10^{-5}$, and $0.6 \times 10^{-5}$ for the three model sizes respectively.
• Epoch count upper bound: 12 epochs (but early stopping is applied, typically at 2–5 epochs)
• EMA decay: 0.99 (Polyak–Ruppert averaging — an exponentially-weighted moving average of weights is taken as the final checkpoint)
• Dropout: 0.05
• Optimizer: AdamW with betas (0.9, 0.95)

Early stopping criterion (Section 5.1, Appendix E). The paper emphasizes a critical practical detail: "we tuned the number of BC epochs and the sampling temperature using a combination of human evaluations and reward model score. This alone closed much of the gap we originally saw between BC and RL." The validation loss (standard for early stopping in supervised learning) is not the primary criterion. Instead, the authors use the reward model score (a downstream quality metric) and occasional human evaluations to determine when to stop. The reward model score usually "improves past the point of minimum validation loss," meaning that training longer than what validation loss would suggest is beneficial for final answer quality — a phenomenon consistent with the observation that language modeling loss on demonstrations is not perfectly correlated with downstream task performance.

The held-out set. Approximately 4% of the demonstrations are held out as a validation set for monitoring training progress (not for early stopping decisions, which use the reward model). All questions used for BC are mutually disjoint from those used for reward model training and RL training.

What BC achieves. After fine-tuning, the model can interact with the text-based browser: it produces valid commands in the correct format, understands how to search, click links, scroll, and quote, and can compose an answer from collected references. However, it imitates the average of human demonstration behavior, including suboptimal browsing strategies, imperfect source selection, and occasional errors. The core motivation for the subsequent reward modeling and optimization stages is that BC alone "is unlikely to lead far beyond human performance" (Section 3.1).

The Reward Model (RM): Learning Human Preferences from Comparisons

What it is and why it's needed. The reward model is a learned function $R(\text{question}, \text{answer}, \text{references})$ that outputs a scalar score representing the expected human preference for that answer relative to alternatives. It is needed because evaluating answer quality automatically is difficult — there is no ground truth for ELI5 questions, and metrics like ROUGE-L are not meaningful (as shown by Krishna et al., 2021). The reward model provides a differentiable proxy for human judgment that can guide optimization (via rejection sampling or RL) toward answers that humans prefer.

The Elo score formulation. The reward model is trained to predict an Elo score, which is a relative rating system originally developed for chess. In this context:

• Each answer is assigned a scalar score $s = R(\text{question}, \text{answer}, \text{references})$
• The difference $s_A - s_B$ between two answers to the same question represents the logit of the probability that a human labeler would prefer A over B
• The probability that A is preferred is modeled as $\sigma(s_A - s_B) = \frac{1}{1 + e^{-(s_A - s_B)}}$

This is formalized in the training objective. Given a pair of answers $(A, B)$ to the same question, where $\mu \in \{-1, 0, 1\}$ represents the human label (1 if A is preferred, -1 if B is preferred, 0 for a tie), the loss is:

$\mathcal{L}(s_A, s_B, \mu) = -\mathbb{1}_{\mu=1} \log \sigma(s_A - s_B) - \mathbb{1}_{\mu=-1} \log \sigma(s_B - s_A) - \mathbb{1}_{\mu=0} \cdot \frac{1}{2} \left( \log \sigma(s_A - s_B) + \log \sigma(s_B - s_A) \right)$

where $\sigma(x) = 1 / (1 + e^{-x})$ is the sigmoid function, $s_A$ and $s_B$ are the reward model's scores for answers A and B, and $\mu$ is the human preference label ($+1$ for A preferred, $-1$ for B preferred, $0$ for tie).

What it computes: The loss penalizes the reward model when its predicted preference probability $\sigma(s_A - s_B)$ disagrees with the human label. When $\mu = 1$ (A is preferred), the loss is $-\log \sigma(s_A - s_B)$, which is small when $s_A \gg s_B$ (the model correctly assigns A a higher score) and large when $s_A \ll s_B$ (the model incorrectly prefers B). When $\mu = -1$, the loss is $-\log \sigma(s_B - s_A)$, penalising the opposite error. When $\mu = 0$ (tie), the loss is the average of the two cross-entropy terms weighted by 1/2, which encourages $s_A \approx s_B$ — the model should not strongly prefer either answer.

Why this form: The Elo formulation with cross-entropy loss is the standard approach from Stiennon et al. (2020) for training preference-based reward models. The key property is that it models relative preferences (which of two answers is better) rather than absolute quality scores. This is important because absolute quality ratings are much noisier and harder to calibrate across labelers than pairwise preferences — it is generally easier for a human to say "A is better than B" than to assign precise numerical scores to each. The cross-entropy loss is the maximum-likelihood objective under the Bradley-Terry model of paired comparisons.

How comparisons are collected (Section 3.1, Appendix C.2). The comparison data collection process is far more elaborate than typical annotation tasks, reflecting the difficulty of evaluating long-form factual answers. The procedure for each comparison is:

1. Read the question — flag it if it does not make sense or should not be answered (in which case the comparison is skipped).
2. Read the first answer and its references — evaluate the trustworthiness of any sources relied upon.
3. Annotate each claim — for every factual claim in the answer, annotate: (a) the level of support it has from the references (well-supported, partially supported, unsupported), and (b) its relevance to the question. A screenshot of the annotation tool is shown in Figure 9.
4. Repeat for the second answer — the same annotation procedure.
5. Rate comparisons on specific dimensions — using a 5-point Likert scale ("A much better", "A better", "Equally good", "B better", "B much better") on:
   • Amount of unsupported and irrelevant information
   • Usefulness of information with different levels of support
   • Coherence
6. Provide a final overall usefulness rating — weighing everything together, on the same 5-point scale.

The evaluation criteria (in descending order of priority, per the instruction document):

• Whether the answer contains unsupported information (highest priority — unsupported claims are considered a major flaw)
• Whether the core question has been answered
• Whether there is additional helpful information beyond directly answering (secondary priority)
• How coherent the answer is, including citation errors
• How much irrelevant information there is (can be higher priority in extreme cases)

The critical design choice: labelers do NOT perform independent research. The paper explicitly states that contractors are NOT required to "perform independent research to judge the factual accuracy of answers, since this would have been difficult and subjective." Instead, they judge whether claims are "supported, i.e., either backed up by a reliable reference, or common knowledge." This transforms the evaluation from an open-ended factual verification task (which would require domain expertise and extensive research) into a more tractable source-support evaluation. However, this also creates a potential weakness: answers that cherry-pick convincing-looking but unreliable sources could score well with labelers who cannot independently verify factual accuracy.

Data collection logistics. Contractors are hired from Upwork and Surge AI (approximately 25% and 75% of data respectively). The top 5 contractors provide around 50% of the comparison data. Comparisons take an average of approximately 10 minutes each. To ensure quality, contractors complete a paid trial period, manual checks are performed, and agreement rates are monitored. Using approximately 100 shared calibration tasks, the final researcher-labeler agreement rate is 74% and the labeler-labeler agreement rate is 73% (treating agreement between neutral and non-neutral labels as 50%).

Simplification for training. Despite the elaborate annotation procedure with multiple dimensions and granular claim-level annotations, only the final overall comparison rating is used in training, and even then, the "much better" and "better" categories are collapsed into a single "preferred" category. Ties ("Equally good") are treated as soft 50% labels in the cross-entropy loss. The paper notes that experiments with predicting auxiliary annotation information as an auxiliary loss did not "significantly improve the validation accuracy of the reward model," though this is flagged as a direction for future research.

Comparison dataset composition. In total, approximately 21,500 comparisons are collected, of which 98% are for ELI5 questions. The remaining 2% come from ELI5 fact-check (185 comparisons), TriviaQA (134), ARC Challenge (84), and ARC Easy (77). For reward model training, the final training set consists of approximately 16,000 comparisons, with the remaining approximately 5,500 held out for evaluation. The comparisons are collected from models of various sizes and training stages (primarily the 175B variant, using various combinations of BC, RL, and rejection sampling), deliberately mixed into a single dataset for data efficiency.

Reward model architecture and training (Sections 3.2, 5.2). The reward model starts from the BC model, with the final unembedding layer removed (since the model now outputs a scalar instead of a token distribution). For training:

• The input is the concatenation of the question, the quotes/references, and the answer — this is exactly the format the BC model sees in the answering phase, plus the generated answer as part of the context.
• The model outputs a single scalar value (the Elo score).
• Training hyperparameters (Appendix E, Table 6): minibatch size 64 (32 for the 175B model), Adam step size multiplier 0.05 (1/60 for the 175B model — meaning the learning rate for the 175B RM is $0.6 \times 10^{-4} / 60 = 1 \times 10^{-6}$), epoch count upper bound 6, EMA decay 0.99.
• Early stopping is based on validation accuracy (the fraction of held-out comparison pairs where the model correctly predicts the preferred answer), typically after 1 epoch.

Scaling properties (Figure 7). Doubling the number of comparisons increases the reward model's accuracy by about 1.8 percentage points. Doubling the number of parameters in the reward model increases accuracy by roughly 0.4 percentage points. The 175B reward model achieves approximately 74% validation accuracy, compared to a human baseline of approximately 73% labeler-labeler agreement (meaning the model approaches the noise ceiling of the human labels).

Optimization Against the Reward Model

Once a reward model is trained, the system can optimize answer quality by maximizing the expected reward. The paper explores two methods: rejection sampling (also called best-of-$n$) and reinforcement learning (specifically, Proximal Policy Optimization, PPO). Both operate on the same fundamental idea — using the reward model as a proxy for human judgment to drive the policy toward higher-quality answers — but they differ in how and when optimization occurs.

Rejection Sampling (Best-of-$n$)

What it is. Rejection sampling is the simplest possible optimization against the reward model: generate $n$ independent answers for a given question (each from a fresh browsing episode), score each answer with the reward model, and select the answer with the highest score. No additional training is required — the BC policy (or RL policy) is used as-is, and the optimization happens purely at inference time through increased computation.

Why it works. Although each individual answer is sampled from the BC policy (which imitates the average human demonstrator), the maximum over $n$ samples is biased toward higher-quality answers. If the reward model is a good proxy for human preference, then as $n$ increases, the selected answer's human preference should increase. This is a form of test-time compute scaling: more inference computation buys higher expected quality, without changing the model weights.

Mathematical formulation. For a given question $q$, the rejection sampling policy $\pi_{\text{best-of-}n}$ is defined implicitly by the procedure:

$a^* = \arg\max_{a \in \{a_1, \ldots, a_n\}} R_{\text{train}}(q, a, \text{refs}(a))$

where each $a_i \sim \pi_{\text{BC}}(\cdot \mid q)$ is an independently sampled answer (with its browsing trajectory and collected references), and $R_{\text{train}}$ is the trained reward model.

What it computes: For a single question, the system runs $n$ independent browsing episodes (each with up to 100 actions), collects $n$ answers with references, scores each one with the reward model, and returns the highest-scoring answer. The computational cost scales linearly with $n$, but the episodes are fully independent and can be parallelized.

Scaling results (Figures 5, 8). The preference of the best-of-$n$ BC model over the standard BC model (single sample) increases with $n$. For the 175B model, best-of-64 BC is preferred to the BC baseline 68% of the time (Figure 5). The relationship between $n$ and quality shows diminishing returns — the curve flattens as $n$ increases. The paper analyzes the compute-efficiency trade-off in Figure 8: for any given inference-time compute budget (measured in floating point operations), there is an optimal combination of model size and number of samples. The compute-efficient frontier consists of three specific configurations chosen as representative points:

• 760M best-of-4 (small model, moderate rejection sampling) — for low compute budgets
• 13B best-of-16 (medium model, more rejection sampling) — for medium compute budgets
• 175B best-of-64 (large model, extensive rejection sampling) — for high compute budgets

These are the "WebGPT" models used in the main evaluations (Section 4).

Predicting rejection sampling performance (Appendix I). To estimate how well best-of-$n$ will perform without expensive human evaluations, the paper uses a validation reward model (a separate reward model trained on a held-out data split) to evaluate the output of rejection sampling against the training reward model. The key challenge is that the naive Monte Carlo estimator — sample $n$ answers, pick the best, evaluate with validation RM, repeat many times to average — requires $n$ answers per estimate and does not reuse answers across different values of $n$. The paper describes a more efficient estimator:

$R_{\text{pred}}^n(q) = \mathbb{E}_{A_1, \ldots, A_n \sim \mathcal{A}(q)} \left[ R_{\text{val}} \left( \arg\max_{a \in \{A_1, \ldots, A_n\}} R_{\text{train}}(a \mid q) \mid q \right) \right]$

where $\mathcal{A}(q)$ is the answer distribution from the policy, $R_{\text{train}}$ is the training reward model used for selection, and $R_{\text{val}}$ is the held-out validation reward model used for evaluation.

The efficient computation trick: Sample $N$ answers $A_1, \ldots, A_N$ for some $N \ge n_{\max}$, sort them by training RM score to obtain $S_1, \ldots, S_N$ with $R_{\text{train}}(S_1) \le \cdots \le R_{\text{train}}(S_N)$. Then:

$R_{\text{pred}}^n(q) = \frac{1}{\binom{N}{n}} \sum_{1 \le i_1 < \cdots < i_n \le N} R_{\text{val}} \left( \arg\max_{a \in \{S_{i_1}, \ldots, S_{i_n}\}} R_{\text{train}}(a) \mid q \right) = \sum_{i=n}^N \frac{\binom{i-1}{n-1}}{\binom{N}{n}} R_{\text{val}}(S_i \mid q)$

What it computes: The reformulation exploits the fact that when answers are sorted by training RM score, the best-of-$n$ selected answer is always the highest-scoring one among the $n$, which will be $S_i$ for some $i \ge n$. The probability that $S_i$ is selected (i.e., that $S_i$ is among the chosen $n$ and all higher-scoring answers are not chosen) is $\binom{i-1}{n-1} / \binom{N}{n}$ — we must choose the other $n-1$ answers from the $i-1$ answers below $S_i$. This provides an unbiased estimator of $R_{\text{pred}}^n(q)$ that reuses the same $N$ samples for all $n \le N$, requiring only $O(N \log N)$ sorting rather than $O(N^n)$ enumeration.

Why this matters: This estimator enables efficient prediction of best-of-$n$ scaling behavior without requiring human evaluations for each value of $n$. Figure 5 shows that the validation RM prediction closely tracks actual human preference for $n \le 64$, though it is expected to overestimate for sufficiently large $n$ as the validation RM itself becomes overoptimized.

Advantages of rejection sampling over RL. The paper finds that rejection sampling substantially outperforms RL for optimizing against the reward model (Figure 4: best-of-64 BC preferred 68% over BC baseline vs. RL preferred 58% over BC baseline). Several hypotheses are offered:

1. Inference-time compute utilization. Rejection sampling directly benefits from more answering attempts, which is an effective way to use additional compute at inference time.
2. Environment unpredictability. The web-browsing environment is stochastic and diverse — visiting different websites leads to different information. Rejection sampling allows the model to "try visiting many more websites, and then evaluate the information it finds with the benefit of hindsight."
3. Reward model overoptimization robustness. The reward model was trained primarily on data from BC and rejection sampling policies (not RL policies), making it potentially more robust to the types of answers generated by rejection sampling than those generated through RL optimization.
4. No hyperparameter tuning. Rejection sampling requires no additional hyperparameters, whereas RL requires tuning the KL penalty coefficient, learning rate, and early stopping.

Reinforcement Learning (PPO)

Why it is explored despite rejection sampling's advantages. RL fine-tuning has the potential to improve the policy's inherent capability rather than just selecting among existing capabilities — it can teach the model to browse better, find more relevant sources, and compose higher-quality answers from the start, reducing the need for rejection sampling at inference time. Even though rejection sampling outperforms RL in the paper's final results, understanding RL's behavior and limitations provides insights for future work.

The PPO setup (Sections 3.2, Appendix E). The RL formulation treats the web-browsing environment as a Markov Decision Process where:

• States are the text observations produced by the environment
• Actions are the model's command text (one of the eight command types)
• Reward at the end of each episode is the sum of:
  • The reward model score for the final answer: $R_{\text{RM}}(q, a, \text{refs})$
  • A KL penalty at each token: $-\beta \cdot D_{\text{KL}}(\pi_{\text{RL}}(\cdot \mid s) \parallel \pi_{\text{BC}}(\cdot \mid s))$

The KL penalty term is crucial. It penalizes the RL policy for deviating too far from the BC policy (measured by per-token KL divergence). Without this penalty, the RL policy would quickly learn to exploit the reward model — generating answers that score highly under the reward model but are not actually good. The KL penalty keeps the policy "close" to the well-behaved BC distribution, allowing it to optimize the reward signal while avoiding degenerate solutions.

Hyperparameters (Appendix E, Table 7). The PPO configuration uses:

• Number of parallel environments: 256 (rollouts run in parallel)
• Timesteps per rollout (T): 256
• Epochs (E): 1 (single pass through the rollout data per PPO iteration)
• Minibatches per epoch: 128 (the rollout data is split into 128 chunks for gradient updates)
• Adam step size multiplier: 0.004 times the pre-training step size (much smaller than BC's 0.1, since RL is more sensitive to step size)
• KL reward coefficient ($\beta$): 0.02
• Entropy coefficient: 0 (no entropy bonus — the KL penalty from BC serves a similar role in preventing entropy collapse)
• PPO clipping parameter ($\epsilon$): 0.2 (standard for PPO)
• GAE discount rate ($\gamma$): 1 (no discounting, since episodes are short — well under 1,000 timesteps)
• GAE bootstrapping parameter ($\lambda$): 0.95
• Reward normalization: No
• Advantage normalization: Yes

Training data mixture and efficiency improvements. The RL training uses a mixture of 90% ELI5 questions and 10% TriviaQA questions. Two important design choices improve sample efficiency:

1. Extra answering-only episodes (Section 3.2). At the end of each browsing episode, 15 additional answering-only episodes are inserted using the same collected references as the original episode. This is motivated by the observation that "answering explained slightly more of the variance in reward model score than browsing despite taking many fewer steps." It improves sample efficiency by approximately a factor of 2, since the answering phase is much shorter than browsing but provides additional training signal.

2. Randomized browsing action limit. The maximum number of browsing actions is randomized uniformly from 20–100 inclusive. This prevents the policy from assuming a fixed episode length during training, making it more robust to different budgets at inference time.

The KL penalty's dual role (Appendix E). The KL penalty coefficient of 0.02 acts both as a regularizer (preventing reward model overoptimization) and as a principled alternative to an entropy bonus for exploration. The paper notes that entropy bonuses are "equivalent to a KL penalty from the uniform distribution" but the uniform distribution over tokens is arbitrary — "it is not invariant to 'splitting' a single token into two equally-likely indistinguishable tokens." The KL penalty from the BC model avoids this arbitrariness by penalizing deviation from a meaningful reference distribution.

Token-level vs. timestep-level PPO. The paper applies PPO clipping and the KL reward at the token level (not at the action/command level), using token-level value function networks for baseline estimation. However, there is no token-level bootstrapping or discounting — the value function predicts the expected return (sum of future rewards) from a given token in the episode. Each "timestep" in the PPO hyperparameters corresponds to a single completion (action), but the underlying updates operate on individual tokens.

Chunking long completions. Some actions (particularly quotes and final answers) require many more tokens than others (e.g., a search query might be 5 tokens, while a quote can be hundreds). To improve rollout parallelizability (preventing one environment from holding up all others while a long completion is generated), the environment "chunks" long completions into multiple actions, with a "maximum tokens per action" of 64. This has a minor effect on GAE computation.

Early stopping for RL (Table 8). RL training is stopped based on the reward model score for a target KL budget (total KL divergence from the BC model, summed over the episode). The stopping points are:

• 760M: 19 PPO iterations, KL budget ~10.5 nats per episode
• 13B: 30 PPO iterations, KL budget ~6.8 nats per episode
• 175B: 18 PPO iterations, KL budget ~12 nats per episode

The KL budget is tuned using human evaluations for the 175B model, and these evaluations inform the budget choices for the smaller models without requiring separate human evaluation tuning.

RL results and limitations (Figure 4). The RL model (175B) is preferred to the BC model 58% of the time, which is a clear improvement but substantially smaller than rejection sampling's 68% preference. Moreover, when RL is combined with rejection sampling (best-of-$n$ applied to RL outputs), the benefit over rejection sampling from BC is marginal. This suggests that:

• RL's primary benefit comes from reducing the entropy of the policy (making it more likely to produce good answers on the first try), which is valuable when inference-time compute is limited.
• When rejection sampling is feasible, the diversity of BC outputs (higher entropy) is actually beneficial — exploring many different browsing strategies and answer formulations increases the chance of finding a high-quality answer.
• RL and rejection sampling optimize against the same reward model, so they are vulnerable to the same overoptimization patterns. An important future direction is "adapting the RL objective to optimize rejection sampling performance" — that is, training the RL policy to produce a diverse set of good answers rather than a single deterministic best answer.

Training Data Orchestration and Question Set Management

Disjoint question sets. The paper uses mutually disjoint sets of questions for BC, RM, and RL training. This prevents contamination — the reward model should not be evaluated on answers to questions it was trained on, since that would give inflated accuracy estimates. The ELI5 dataset provides 12,000 training questions (from Lightman et al., 2022's split), which is sufficient for this partitioning.

Answer sampling for comparisons. The answers used in comparison pairs are generated from an "ad-hoc" collection of models — various sizes (primarily 175B), trained using various methods (BC only, BC+RL, BC+rejection sampling) and various hyperparameters. This diversity is intentional: it ensures the reward model encounters a wide range of answer qualities and failure modes, making it more robust when later used to evaluate new policies. The paper acknowledges that this mixing is done "for data efficiency" — many comparisons were initially collected for evaluation purposes (hyperparameter tuning, model selection), and rather than discarding them, they are folded into the reward model training set.

Post-processing of ELI5 questions (Appendix B). Several transformations are applied to make ELI5 questions suitable for the web-browsing task:

1. URLs are left in full rather than replaced with _URL_ tokens (as in the original ELI5 release).
2. Questions with "[deleted by user]" titles are filtered out; "[deleted]" and "[removed]" selftext is ignored.
3. Title and selftext are concatenated with a double newline.
4. Non-question titles are prepended with "Explain: ". The heuristic for detecting questions checks for a question mark or any of approximately 60 question-like words (explain, what, who, how, why, do, can, etc.) with regex word boundaries.

The training pipeline sequence. In practice, the training proceeds in stages:

1. Collect demonstrations (BC data) — ~6,000 demonstrations from contractors
2. Fine-tune GPT-3 on demonstrations → BC model
3. Use BC model (and later, RL and rejection sampling variants) to generate answers for ~21,500 question pairs
4. Collect human comparisons on these pairs → comparison dataset
5. Train reward model on comparisons
6. Optionally, train RL policy using PPO with reward model as environment reward
7. At inference time, apply rejection sampling (best-of-$n$) using the reward model to select the best answer

Each stage feeds into the next, creating a data flywheel: better policies generate better answer pairs, which provide a better training signal for reward models, which can guide further policy optimization.

Sampling temperature selection. The paper uses a sampling temperature of 0.8 at inference time for all WebGPT models (Section 4), tuned using human evaluations. During training, the temperature affects the diversity of generated answers, which is particularly important for the exploration behavior during RL and for generating diverse answer pairs for comparison collection (higher temperature = more diverse answers = more informative comparisons).

4. Key Insights and Innovations

Innovation 1: Reframing LFQA as an Agent-Environment Interaction Instead of a Retrieve-Then-Read Pipeline

The dominant conceptual framework for question-answering with external knowledge, prior to WebGPT, was the retrieve-then-read architecture. Systems like REALM (Guu et al., 2020), RAG (Lewis et al., 2020a), and DPR (Karpukhin et al., 2020) treated retrieval as a single-step process: embed the query, find the top-$k$ similar documents by inner product in an embedding space, and then condition the generator on those retrieved documents. This framing was elegant — it made the entire pipeline differentiable and trainable end-to-end — but it baked in a specific assumption: that retrieval is a one-shot similarity search rather than an iterative, strategic process.

WebGPT fundamentally reframes the problem. Instead of retrieval-as-embedding-lookup, the paper treats retrieval as sequential decision-making in an information-rich environment. The model is not handed retrieved documents; it must decide what to search for, evaluate the search results, choose which links to follow, navigate within pages, decide when to extract a quote, and determine when it has gathered enough information to answer. These are not implementation details — they are the core of an agent-like interaction loop where each action reveals new information that shapes subsequent decisions.

This reframing has several consequences that change how one thinks about LFQA:

• Retrieval becomes a learned strategy rather than a fixed computation. A REALM retriever learns a single embedding function that maps queries to documents. WebGPT's policy learns a browsing strategy: when to issue broad vs. specific queries, how to recognize promising sources from search snippets, when to go back and try a different approach, when to stop browsing. These strategic decisions are optimized indirectly through human preference feedback on the final answer, meaning the model learns what browsing behavior leads to answers that humans judge as well-supported and useful.

• The search engine is treated as an external tool, not a component to be optimized. This is a conceptual break from the differentiable-retrieval tradition. Rather than trying to improve retrieval by training a better embedding model, WebGPT leverages an existing, highly-engineered system (Bing) and focuses the learning on how to use it effectively. This separates the problem of "building a good retriever" (solved by search engines) from "knowing how to query, filter, and synthesize retrieved information" (solved by the language model policy). It's a division of labor that was not obvious in prior work, where retrieval and generation were typically trained jointly as a single system.

• The browsing process is fully interpretable. Because the model's actions are explicit text commands (search queries, link clicks, scrolls, quotes), the entire trajectory of how an answer was produced is auditable. A user can see exactly which queries were issued, which pages were visited, and which passages were extracted. This is a qualitative shift from embedding-based retrieval, where the "reason" a document was retrieved is a similarity score in an uninterpretable vector space.

This reframing is not merely an architectural choice — it is a fundamental reconceptualization of what it means for a language model to use external knowledge. Prior work asked "how can we retrieve the right documents?" WebGPT asks "how can we train a model to act competently in an information-seeking environment?" The second question is broader and opens connections to reinforcement learning, agent design, and human-AI interaction that the retrieve-then-read paradigm did not naturally admit.

The evidence that this reframing works is in the main results (Section 4, Figure 2): the best WebGPT model produces answers preferred to human demonstrations 56% of the time. This is not just an incremental retrieval improvement — it demonstrates that the model learned a browsing strategy competitive with humans who had access to the same tools, which would be impossible if the system were architecturally limited to one-shot retrieval.

Innovation 2: Using Required References to Make Human Evaluation of Factual Accuracy Feasible at Scale

Training AI systems with human feedback on factual tasks faces a fundamental obstacle: evaluating the truth of arbitrary claims is hard, slow, and subjective. If you ask a labeler "Is this answer factually correct?" for questions spanning science, history, current events, and personal advice, you are implicitly asking them to be a domain expert on every topic, or to perform independent research for every evaluation. The paper makes this difficulty explicit: "It is very challenging to evaluate the factual accuracy of arbitrary claims, which can be technical, subjective or vague."

The conventional response to this problem is to either restrict the domain (work on benchmarks with known answers) or accept noisy labels and hope the reward model averages out the errors. WebGPT takes a fundamentally different approach: change the evaluation task so that it does not require independent factual verification. By requiring the model to collect and cite references during browsing, the evaluation is transformed from "Is this claim true?" to "Is this claim supported by the provided references?"

This is a diagnostic move with deep implications:

• It makes the evaluation task tractable for non-experts. A labeler can read a claim, check the cited reference, and judge whether the reference substantiates the claim. They do not need to know the answer beforehand or research it independently. This is what enables the paper to collect ~21,500 high-quality comparisons at scale — each comparison takes approximately 10 minutes, which would be impossible if labelers had to fact-check every claim from scratch.

• It aligns the training signal with a verifiable property (support) rather than an unverifiable one (truth). The paper is candid that this creates a gap: a model could cherry-pick references that appear supportive to a non-expert labeler but do not represent a fair assessment of the evidence. However, by making the evaluation criterion transparent and operationalizable, it becomes possible to audit, improve, and eventually automate — whereas evaluating "truth" directly is a philosophical quagmire at scale.

• It makes the model's output accountable to end users. Because references are part of the answer, a user can follow the citations and judge for themselves whether the claims are supported. This transforms the model from an oracle (whose statements must be trusted or rejected wholesale) into an interlocutor (whose claims can be verified). The paper explicitly connects this to broader AI alignment agendas: debate (Irving et al., 2018) and recursive reward modeling (Leike et al., 2018), in which models assist their own evaluation by providing evidence.

• It creates a natural countermeasure against hallucination. A model that must quote sources to support its claims cannot simply fabricate plausible-sounding facts — or rather, it can, but those fabrications will be flagged as unsupported during evaluation. This does not eliminate hallucination entirely (the paper documents cases where models paraphrase incorrectly or draw unsupported inferences), but it provides both a training signal and an evaluation mechanism that penalizes unsupported claims.

The prior state of the art (Krishna et al., 2021) had already shown that automated metrics like ROUGE-L are meaningless for ELI5, which forced the use of human evaluation. But Krishna et al. evaluated answers by having annotators judge "whether the generated text is a useful explanation of the question" — a holistic, subjective judgment. WebGPT's contribution is making that judgment operational by decomposing it into support-by-reference, coherence, and relevance, with the support judgment grounded in a concrete, inspectable artifact (the cited quotes).

The significance of this innovation extends beyond WebGPT's specific results. It establishes a template for training AI systems on factual tasks using human feedback: don't ask humans to evaluate truth directly; instead, require the model to produce evidence that humans can evaluate. This principle applies to any domain where ground-truth labels are unavailable but evidence can be checked — legal reasoning, scientific literature review, medical question-answering, and beyond.

The evidence that this works is in the inter-labeler agreement: 73% labeler-labeler agreement on comparisons (Appendix C), and 74% researcher-labeler agreement. These agreement rates, while imperfect, are high enough to train a useful reward model (the 175B reward model achieves approximately 74% validation accuracy, approaching the human noise ceiling). Without the reference-based evaluation scaffold, these agreement rates would almost certainly be much lower, and the entire human-feedback pipeline would be far less effective.

Innovation 3: Demonstrating That Rejection Sampling Can Outperform RL for Optimizing Against Learned Reward Models in Complex Environments

The standard narrative from Stiennon et al. (2020) — which WebGPT explicitly builds on — is that reinforcement learning (specifically PPO) is the method of choice for optimizing a policy against a learned reward model. In that work on summarization, RL fine-tuning produced substantial improvements over both behavior cloning and rejection sampling. The natural expectation, carried into WebGPT, would be that RL similarly dominates.

WebGPT finds the opposite: rejection sampling (best-of-64) is preferred to the BC baseline 68% of the time, while RL is preferred only 58% of the time (Figure 4). Moreover, combining RL with rejection sampling provides little additional benefit over rejection sampling from the BC policy alone. This is not just a minor implementation detail — it challenges the assumption that RL is the natural endpoint for reward model optimization and suggests that the relationship between optimization method and environment structure matters in ways the field had not fully appreciated.

The paper offers several hypotheses for why rejection sampling outperforms RL, each of which carries broader implications:

• Environment stochasticity as a feature, not a bug. The web is diverse and unpredictable. Different browsing trajectories lead to different websites, different information, and different answer possibilities. Rejection sampling exploits this diversity: it generates many independent browsing trajectories and picks the best outcome. RL, by contrast, reduces entropy — it converges toward a narrower distribution of behaviors that score well on average. In a stochastic environment where "luck" (finding the right source) matters, preserving diversity can be more valuable than optimizing the mean.

• Hindsight evaluation. Rejection sampling evaluates answers after they are fully formed, with the benefit of seeing the collected references and the composed answer. The reward model can assess whether the final product is good without needing to evaluate intermediate browsing decisions. RL must assign credit to individual actions (search queries, link clicks) based on their contribution to the final outcome — a harder credit assignment problem, especially when the connection between a good search query and a good final answer may be subtle and delayed.

• Overoptimization vulnerability differs by method. The reward model is trained primarily on data from BC and rejection sampling policies. RL generates answers from a different distribution (lower entropy, optimized against the same reward model), which may push the policy into regions where the reward model's predictions are less reliable. This is a form of distributional shift in reward modeling: the reward model is accurate for the kinds of answers it was trained on, but RL can produce answers that exploit blind spots.

This finding is a diagnostic contribution rather than a methodological one — it does not propose a new algorithm, but it reveals that the choice of optimization method interacts with environment properties in non-obvious ways. Specifically, it suggests that rejection sampling is underappreciated as a simple, robust alternative to RL when the environment has high useful variance (exploring different information sources leads to genuinely different answer quality) and when inference-time compute is available.

The evidence is clear in Figures 4 and 5: the gap between rejection sampling and RL is substantial (68% vs. 58% preference over BC), and rejection sampling scales predictably with the number of samples (Figures 5, 8). The compute-efficient frontier in Figure 8 demonstrates that for any given inference budget, some degree of rejection sampling is almost always better than using a larger model with fewer samples — a finding with direct practical implications for deployment.

Innovation 4: The Training-Inference Compute Tradeoff as an Explicit, Quantified Design Axis

The paper does more than just build a better LFQA system — it provides one of the first systematic analyses of how to trade off training compute (model size) against inference compute (number of rejection samples) under a fixed budget. Figure 8 plots a Pareto frontier showing, for any given number of floating-point operations, the optimal combination of model size and best-of-$n$ sampling to maximize expected reward model score.

This analysis is significant not because it presents a new scaling law (it does not attempt to fit a parametric function like the Chinchilla laws) but because it establishes the concept of a compute-efficient frontier for test-time compute in a production-relevant setting. Prior work on scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) focused almost exclusively on training compute — how to allocate a budget between model size and data quantity. Inference-time compute was typically treated as a fixed cost per query, not an optimization variable. WebGPT shows that inference compute is a design dimension that can be traded off against model size at deployment time.

The practical implication is non-obvious and important: it is not always optimal to use the largest model you can afford. For a given compute budget, using a smaller model with more rejection sampling can outperform a larger model with fewer samples. The three "compute-efficient" configurations that emerge from Figure 8 — 760M best-of-4, 13B best-of-16, 175B best-of-64 — represent different points on this frontier corresponding to different inference-time compute budgets. A deployment that can afford 64 samples per query should use the 175B model; one that can only afford 4 samples should drop to 760M rather than trying to run the 175B model with fewer samples.

This insight generalizes beyond WebGPT's specific setup. Any system that uses a learned model with rejection sampling (or any inference-time computation that scales with quality) faces the same tradeoff: should you spend compute on a better model or on more attempts from a cheaper model? WebGPT provides a concrete methodology for answering this question (empirically sweep model sizes and sample counts, evaluate with a validation reward model, find the Pareto frontier) and demonstrates that the answer is not trivial — the optimal configuration depends on the available budget and the scaling properties of the specific models and reward model.

The scaling experiments (Section 5.2) also provide data on how performance scales with dataset size: doubling the number of demonstrations increases the policy's reward model score by approximately 0.13 Elo points, and doubling the number of comparisons improves reward model accuracy by approximately 1.8 percentage points. These numbers are not theoretical constants but empirical measurements that give practitioners a sense of the data requirements for similar systems. The relatively modest gains from doubling demonstrations (0.13 Elo points, where 1 point corresponds to a preference of ~73%) underscore that behavior cloning on human demonstrations faces diminishing returns, motivating the shift to human feedback optimization.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary dataset is ELI5 [Fan et al., 2019], a collection of long-form questions taken from the "Explain Like I'm Five" subreddit. The paper uses the specific split from Lightman et al. (2022) with 12,000 training questions and 500 test questions. Questions undergo post-processing: URLs are included in full, questions with "[deleted by user]" titles are filtered out, title and selftext are concatenated with a double newline, and non-question titles are prepended with "Explain: " (determined by checking for question marks or ~60 question-like words with regex word boundaries, as detailed in Appendix B). For the TruthfulQA evaluation, the full TruthfulQA dataset [Lin et al., 2021] is used. For TriviaQA, the development set splits defined in Lewis et al. (2020b) are used.

• Base model(s). All models are fine-tuned from the GPT-3 family [Brown et al., 2020], spanning three sizes: 760M, 13B, and 175B parameters. The authors state these models provide the underlying capabilities of reading comprehension, query formulation, and text generation as zero-shot capabilities, and are fine-tuned (not used zero-shot) for browser interaction. The pretraining Adam step sizes are $2.5 \times 10^{-4}$ for 760M, $1.0 \times 10^{-4}$ for 13B, and $0.6 \times 10^{-4}$ for 175B (Table 5).

• Metrics. The primary evaluation metric is human preference: contractors compare two answers to the same question and indicate which they prefer (on a scale from "A much better" to "B much better," collapsed to binary preference with ties as 50% ratings). For internal experiments and scaling analyses, reward model score is used as a proxy, measured by a held-out "validation" reward model trained on a separate data split. The reward model score represents an Elo score, with a difference of 1 point corresponding to a preference probability of $\sigma(1) \approx 73\%$. For TruthfulQA, truthfulness and truthful-and-informativeness are measured via human evaluation (since WebGPT's answers are out-of-distribution for the automated metric). For TriviaQA, exact match accuracy is used.

• Baselines. The paper compares against several baselines:

  • Human demonstrations: answers written by contractors using the same web-browsing environment, collected under the same instructions (relevance, coherence, trustworthy references).
  • ELI5 reference answers: the highest-voted answer from Reddit for each question, taken from the ELI5 dataset.
  • GPT-3 with QA prompt: from Lin et al. (2021), a standard prompting approach for short-form QA evaluated on TruthfulQA.
  • GPT-3 with helpful prompt: from Lin et al. (2021), an instruction-following prompt that often produces "I have no comment" responses on TruthfulQA.
  • Krishna et al. (2021) best model: a prior retrieval-augmented generation system evaluated on ELI5, whose answers were preferred 23% of the time to reference answers.
  • UnitedQA [Cheng et al., 2021]: the best existing model on TriviaQA at the time of writing.

• Generation budget / compute accounting. For comparisons between WebGPT variants, the "generation budget" is measured in number of samples for rejection sampling ($n$ in best-of-$n$). The compute-efficient frontier analysis (Figure 8) uses floating point operations (FLOPs) as the compute metric, accounting for both model size (parameters) and number of samples (inference passes). RL comparisons use the number of PPO iterations and the total KL divergence from the BC model (in nats per episode) as the stopping criterion. For the TriviaQA evaluation, the paper acknowledges using "far more compute than UnitedQA."

• Cross-validation / statistical protocol. For the main human evaluations (Section 4, Figure 2), error bars represent ±1 standard error of the mean across questions. For the RM validation accuracy and scaling experiments (Figures 5–8), a separate validation reward model trained on a held-out comparison split is used as a proxy to avoid contamination. For comparisons against human demonstrations and ELI5 answers, the paper uses a detailed procedure with multiple annotation dimensions, but only the final overall comparison rating (collapsed to binary preference) is used for training and evaluation. For TruthfulQA, human evaluation is used rather than the automated metric because WebGPT answers are out-of-distribution. No explicit cross-validation protocol is described for strategy selection (unlike the compute-optimal paper in the reference example), since this paper does not perform adaptive strategy selection — the three WebGPT models are fixed configurations chosen from the compute-efficient frontier.

Main Quantitative Results

WebGPT vs. Human Demonstrations on ELI5 (Figure 2a)

The headline result: the 175B best-of-64 model produces answers that human evaluators prefer 56% of the time to answers written by human demonstrators using the same web-browsing environment. Breaking this down by evaluation dimension:

• Overall usefulness: Model answers preferred approximately 56% of the time.
• Factual accuracy: Model answers preferred approximately 55% of the time (reading from Figure 2a).
• Coherence: Model answers preferred approximately 50% of the time — statistically tied with human demonstrations.

For the smaller models:

• 760M best-of-4: preferred approximately 45% overall (below human demonstrations).
• 13B best-of-16: preferred approximately 48% overall (roughly tied with human demonstrations).

The trend is monotonic: larger models with more rejection sampling produce answers that humans increasingly prefer over human-written answers. The fact that the 175B model exceeds 50% is notable because behavior cloning alone "is unlikely to lead far beyond human performance" (Section 3.1) — it trains the model to imitate demonstrations, and exceeding 50% preference suggests the reward model and rejection sampling successfully pushed quality beyond the average human demonstrator. The paper notes that exceeding 50% "may still be possible" from BC alone "by producing a less noisy policy," but the combination of RM-guided selection is what achieves it in practice.

The factual accuracy dimension is particularly informative: the model's answers are judged as more factually accurate than human demonstrations, despite the humans having access to the same browser and being explicitly instructed to find trustworthy sources. This suggests the model learned — through the reward model's preferences — to select better sources or synthesize information more carefully than the average human demonstrator.

WebGPT vs. ELI5 Reference Answers (Figure 2b)

The model's answers (with references and citations stripped for fairness) are preferred 69% of the time to the highest-voted Reddit answer from the ELI5 dataset. This is a substantially larger margin than the 56% preference over human demonstrations, and the paper discusses several reasons why the comparison against demos is considered "more meaningful" (Section 4.1):

• Fact-checking is easier with references. When evaluating against ELI5 answers (which lack citations), labelers must independently verify factual claims — a difficult task even with search engine access. When evaluating against demonstrations (which include references), labelers can judge whether claims are supported by cited sources.
• Objectivity of criteria. The minimal instructions used for the ELI5 comparison (Appendix F) give labelers less guidance on what constitutes a good answer, making preferences noisier and harder to interpret.
• Blinding concerns. Even with citations stripped, WebGPT answers differ in style from Reddit answers (more formal, differently structured), making it harder for labelers to be truly blind to which answer came from the model.
• Answer intent mismatch. ELI5 questions expect "original, simplified explanations," but the reference answers vary widely in quality and effort — many ELI5 questions "only ever get a small number of low-effort answers." Human demonstrations, by contrast, are produced under consistent instructions emphasizing thoroughness and source support.

Despite these caveats, the 69% preference is compared against Krishna et al. (2021), whose best model's answers were preferred only 23% of the time to the same ELI5 reference answers. The paper notes this comparison is not perfectly fair (Krishna et al. use "substantially less compute than even our smallest model"), but the magnitude of improvement (23% → 69%) is dramatic and cannot be explained solely by compute differences.

The dimension-level breakdown for the ELI5 comparison shows:

• Overall usefulness: 69% preference for WebGPT.
• Factual accuracy: approximately 65% preference (from Figure 2b).
• Coherence: approximately 60% preference.

The coherence preference is notably lower than overall usefulness, consistent with the idea that Reddit answers (written by humans for humans) are naturally coherent, while WebGPT's advantage comes primarily from better factual grounding.

TruthfulQA Results (Figure 3)

WebGPT is evaluated on TruthfulQA to probe whether web access and human feedback training reduce imitative falsehoods (false statements that arise from reproducing common misconceptions in training data). The key findings:

Truthfulness:

• GPT-3 175B with QA prompt: approximately 22% truthful (from Figure 3, left bars).
• GPT-3 175B with helpful prompt: approximately 42% truthful (the "I have no comment" strategy).
• WebGPT 760M best-of-4: approximately 55% truthful.
• WebGPT 13B best-of-16: approximately 65% truthful.
• WebGPT 175B best-of-64: approximately 75% truthful.
• Human performance: approximately 94% truthful.

Truthful and informative:

• GPT-3 175B with QA prompt: approximately 20% truthful and informative.
• GPT-3 175B with helpful prompt: approximately 26% truthful and informative (the "I have no comment" answers are truthful but uninformative, dragging this down).
• WebGPT 760M best-of-4: approximately 42% truthful and informative.
• WebGPT 13B best-of-16: approximately 48% truthful and informative.
• WebGPT 175B best-of-64: approximately 54% truthful and informative.
• Human performance: approximately 94% truthful and informative.

Two patterns are particularly significant:

1. WebGPT outperforms all GPT-3 variants on both metrics. This supports the hypothesis that web access and the incentive to cite reliable sources reduce imitative falsehoods. GPT-3, when trained on internet text, absorbs common misconceptions (e.g., "smashing a mirror causes seven years of bad luck"); WebGPT can search the web and find authoritative sources that correct these misconceptions.

2. WebGPT's performance increases with model size, unlike GPT-3. GPT-3's truthfulness with the QA prompt actually decreases or stays flat with model size for some configurations (the larger model reproduces misconceptions more confidently). With the helpful prompt, the larger models respond "I have no comment" more frequently (49% of questions for GPT-3 175B), increasing truthfulness at the expense of informativeness. WebGPT, by contrast, shows a clear positive trend: larger models browse more effectively, find better sources, and produce more truthful and informative answers.

However, WebGPT still falls well short of human performance (75% vs. 94% truthful). The paper provides a qualitative analysis of failure modes in Table 3: WebGPT sometimes quotes from unreliable sources when answering TruthfulQA questions. The example "If you dream of doing something and make a wish, will you succeed?" shows WebGPT answering "It is true that you can make a wish come true by the power of thought" — a false statement that the model supported by quoting from an unreliable source. The authors hypothesize this is due to "distribution shift from ELI5 to TruthfulQA" — the model was trained on ELI5 questions where reliable sources are easier to find, and when confronted with adversarial questions designed to elicit falsehoods, it sometimes fails to identify trustworthy information.

A technical note: WebGPT answers are truncated to 50 tokens and trailing partial sentences are removed for this evaluation. This inadvertently produced a small number of empty answers (approximately 3% of answers, or 74 total), which were considered truthful but not informative.

Comparison of Training Methods: RL vs. Rejection Sampling (Figures 4, 5)

Headline: rejection sampling substantially outperforms RL for optimizing against the reward model.

Specifically (Figure 4, 175B models):

• Rejection sampling (best-of-64 BC): preferred 68% of the time over the BC baseline (single sample).
• RL without rejection sampling (best-of-1 RL): preferred 58% of the time over the BC baseline.
• RL with rejection sampling (best-of-64 RL): preferred approximately 70% of the time over the BC baseline — only marginally better than best-of-64 BC, and the error bars overlap.

This pattern holds across model sizes: the RL-only models (left bars in Figure 4 for each size) show modest improvements over the BC baseline (roughly 55–60% preference), while the best-of-$n$ BC models (right bars) show substantially larger improvements.

The diminishing returns of combining RL with rejection sampling are important: even though RL fine-tunes the policy to directly maximize the reward model score, the benefit over simply sampling more from the BC policy is small when rejection sampling is already used. The paper suggests RL reduces the entropy of the policy, which helps when only one sample is drawn but hurts exploration when many samples are drawn — the BC policy's higher diversity means best-of-64 explores a wider range of browsing strategies and source selections.

Figure 5 shows the scaling of rejection sampling for the 175B BC model, comparing actual human preference against the validation RM prediction (using the efficient estimator from Appendix I):

• Best-of-1 (BC baseline): 50% (by definition, since it is compared against itself).
• Best-of-4: preferred approximately 56%.
• Best-of-16: preferred approximately 63%.
• Best-of-64: preferred 68%.

The validation RM prediction closely tracks human preference across this range, validating its use as a proxy for scaling experiments. The shaded region shows ±1 standard error, and the curves are visually well-aligned, suggesting the validation RM is not significantly overoptimized at n ≤ 64.

Scaling Experiments: Dataset Size, Parameter Count, and Compute-Efficiency (Figures 6, 7, 8)

Policy scaling with demonstrations (Figure 6). Doubling the number of demonstrations (from 1/8 to 1/4 to 1/2 to 1 of the ~6,000 demonstration dataset) increases the policy's validation RM score by approximately 0.13 Elo points per doubling. The effect is roughly log-linear and consistent across model sizes (760M, 13B, 175B), though the 175B model benefits slightly more from additional data. At any fixed data scale, larger models achieve higher RM scores, but the gap narrows as data increases — suggesting that data, not model capacity, is the primary bottleneck for BC performance.

Reward model scaling with comparisons (Figure 7). Doubling the number of comparisons (from 1/8 to 1/4 to 1/2 to 1 of the ~16,000 comparison training set) increases the reward model's validation accuracy by approximately 1.8 percentage points per doubling. The accuracy ranges from approximately 66% at 1/8 data to approximately 74% at full data for the 175B RM. The 175B RM's full-data accuracy (approximately 74%) approaches the human labeler-labeler agreement rate of 73% (from Appendix C), suggesting the reward model is near the noise ceiling of the human labels. The "human baseline" and "ensemble of humans" markers in Figure 7 provide reference points — the ensemble (aggregating multiple human judgments) achieves approximately 82% accuracy, indicating that individual human labels are noisy and there is room for improvement through better labeling protocols or aggregation.

Compute-efficient frontier for rejection sampling (Figure 8). This is the most practically actionable scaling result in the paper. The x-axis shows total FLOPs (log scale), and the y-axis shows validation RM score. Each curve represents a fixed model size with varying numbers of rejection samples (the number of samples increases as FLOPs increase along the curve). The key finding: for any given compute budget, there is an optimal combination of model size and number of samples. The frontier (dashed line) is constructed by taking the upper envelope of the curves.

The three points highlighted on the frontier correspond to different inference-time compute budgets:

• 760M best-of-4: ~10^14 FLOPs → RM score ~0.5. This is the compute-efficient choice for very low budgets.
• 13B best-of-16: ~3×10^15 FLOPs → RM score ~0.65. This is the compute-efficient choice for medium budgets.
• 175B best-of-64: ~1.5×10^16 FLOPs → RM score ~0.85. This is the compute-efficient choice for high budgets.

The frontier reveals that it is generally better to use a smaller model with more rejection sampling than a larger model with fewer samples, up to a point. For example, at ~10^15 FLOPs, the 760M model with more samples outperforms the 13B model with fewer samples. But at sufficiently high budgets, using the largest model is optimal. The three chosen configurations are the "WebGPT" models used in the main evaluations, representing different points on this tradeoff curve.

TriviaQA Evaluation (Table 9, Appendix G)

Although WebGPT was trained primarily on ELI5 (long-form QA), the paper evaluates it on TriviaQA to test transfer to short-form question-answering. To bridge the format gap, a separate GPT-3 175B model is fine-tuned on 256 TriviaQA questions to extract short answers from WebGPT's long-form output.

Results on the TriviaQA development set (exact match):

• GPT-3 175B alone: 58.7% overall. On questions with no test-train overlap (as defined by Lewis et al., 2020b): 39.0%.
• GPT-3 175B + WebGPT 175B BC: 69.5% overall. On questions with no overlap: 52.4%.
• UnitedQA-E: 68.9% overall. On questions with no overlap: 44.3%.
• UnitedQA (hybrid model): 70.5% overall.

WebGPT slightly outperforms UnitedQA-E on non-overlapping questions (52.4% vs. 44.3%) but slightly underperforms the full UnitedQA hybrid on the overall set. The paper notes that WebGPT uses live web access (rather than only the TriviaQA corpus) and far more compute than UnitedQA, but was trained on far fewer TriviaQA examples (143 demonstrations plus 256 fine-tuning examples). The transfer from long-form to short-form QA, without architectural modification, is notable.

Ablation Studies and Robustness Checks

Behavior cloning epoch count and sampling temperature (Section 5.1, Appendix E). The paper reports that tuning the number of BC epochs and the sampling temperature "alone closed much of the gap we originally saw between BC and RL." Early stopping for BC is based on reward model score rather than validation loss, because the RM score "usually improves past the point of minimum validation loss." The final BC models use 2 epochs for 760M, 5 epochs for 13B, and 3 epochs for 175B (Table 8). Sampling temperature is set to 0.8 for all models after tuning with human evaluations. This ablation is informal (no figure or table is dedicated to it), but the paper emphasizes its importance: without this tuning, the BC baseline would have appeared weaker, and RL's benefits would have seemed larger by comparison.

RL KL budget and early stopping (Table 8). The RL models are early-stopped at different PPO iterations and KL budgets per model size: 760M at 19 iterations (10.5 nats KL), 13B at 30 iterations (6.8 nats KL), and 175B at 18 iterations (~12 nats KL). The KL budget for 175B is tuned using human evaluations, and these inform the budgets for smaller models without separate human evaluation tuning. The paper does not present an ablation showing how sensitive RL performance is to the exact KL budget, but the fact that the stopping points vary non-monotonically with model size (13B uses the most iterations but the smallest KL budget) suggests the relationship is not simple.

Answering-only episodes in RL training (Section 3.2). Inserting 15 additional answering-only episodes per browsing episode improves sample efficiency "by approximately a factor of 2." This is motivated by the observation that answering "explained slightly more of the variance in reward model score than browsing despite taking many fewer steps." No dedicated figure shows this ablation, but the paper reports it as a practical design choice that improved training throughput.

Maximum browsing actions randomization (Section 3.2). During RL training, the maximum number of actions is randomized uniformly from 20–100. No ablation is presented for this choice, but the paper states it is intended to make the policy "more robust to different budgets at inference time."

Reward model architecture: auxiliary loss experiments (Appendix C.2). The paper attempted to predict auxiliary annotation information (claim-level support, relevance, trustworthiness of sources) as an auxiliary loss during reward model training, but "was not able to significantly improve the validation accuracy of the reward model." This is a negative result: the rich annotation structure collected during comparisons (Figure 9 shows the annotation tool) does not translate into better reward predictions when used as additional training signal.

Comparison data mixing strategy (Section 3.2). The reward model is trained on comparisons collected from an "ad-hoc" mixture of models (various sizes, various training methods, various hyperparameters). No ablation tests the effect of training on a single model's outputs versus a diverse mixture, but the paper argues the diversity is beneficial "for data efficiency" and because it exposes the RM to a wider range of answer qualities and failure modes.

Effect of question stance on factual accuracy and answer stance (Appendix H). In a small-scale experiment with 60 questions (10 conspiracy theories and 10 misconceptions, each phrased in three stances: skeptical, neutral, and affirming), the paper finds:

• Factual accuracy (Figure 11): Questions that affirm an implicit belief in a conspiracy or misconception elicit inaccurate answers more often than neutrally or skeptically framed questions. This is most pronounced for the 175B model, where affirming questions get approximately 72% accurate answers vs. approximately 88% for skeptical questions (reading from Figure 11).
• Answer stance (Figure 12): Models tend to refute implicit beliefs more often than they affirm them, and this tendency increases with model size. However, "no clear evidence" is found that question stance affects answer stance — the models do not mirror the question's framing in their response stance.

This experiment is explicitly described as too small for "definitive conclusions" but demonstrates "the model's potential to misinform users who have erroneous beliefs in ways that reinforce those beliefs." It serves as a probing analysis rather than a rigorous ablation.

Reference point bias case study (Appendix H.2). In a qualitative analysis of 64 answers to "What does a wedding look like?", the model "tended to assume a Western, and often specifically an American, point-of-view." 20 of 64 answers included the word "America" or "American," and only 4 focused on a specific, named non-American culture. Eight answers noted there is no standard wedding, but all but one of these still included Western wedding details. This is not a controlled experiment but a case study flagging a bias pattern that the model exhibits.

Censorship of overlapping content (Appendix A). The environment censors any page with a 10-gram overlap with the question or reference answer. No ablation tests how this affects performance, but it prevents the model from directly copying answers (particularly important since ELI5 answers come from Reddit, and search results from reddit.com and quora.com are already filtered out).

Use of EMA for final checkpoints (Appendix E). For BC and RM training, Polyak–Ruppert averaging (EMA decay 0.99) is used to produce the final checkpoint. For RL, the EMA model was not used for 760M and 13B reward models "due to a bug." No ablation compares EMA vs. non-EMA performance, but the paper reports this as a known implementation issue.

Critical Assessment

Claim 1: "Our best model is obtained by fine-tuning GPT-3 using behavior cloning, and then performing rejection sampling against a reward model trained to predict human preferences."

The experimental evidence supports this specific methodological claim — the 175B best-of-64 BC model is the configuration that achieves the strongest human preference results (56% vs. demonstrations, 69% vs. ELI5). Figure 4 demonstrates that rejection sampling from BC outperforms RL, and Figure 5 shows rejection sampling performance scaling with n. The combination of BC + rejection sampling is clearly validated as the strongest configuration among those tested.

However, what the experiments do NOT fully explore is whether alternative training pipelines could outperform this. The paper tests BC → RM → (RL or rejection sampling), but other sequences are conceivable: What if RM training used answers from RL policies more heavily? What if BC and RL were interleaved? What if the reward model was retrained on rejection-sampled outputs in an iterative fashion? The paper's claim is that BC + rejection sampling is the best among the methods they tried, not that it is the best possible combination. This is a reasonable scope limitation but worth noting.

Claim 2: "This model's answers are preferred by humans 56% of the time to those of our human demonstrators."

This claim is directly supported by Figure 2a, with clear error bars and dimension-level breakdowns. The result is statistically meaningful: 56% is above 50%, and the error bars (±1 standard error) appear to exclude 50% for the 175B model. The dimension-level consistency (factual accuracy ~55%, coherence ~50%) provides face validity — it is plausible that the model's advantage comes primarily from better factual grounding rather than better writing style.

But the claim has important boundary conditions that are easy to overlook:

• The human demonstrators are not domain experts. They are contractors, "generally highly educated, usually with an undergraduate degree or higher," but they are not subject-matter experts on the questions they answer. Their demonstrations take an average of 15 minutes — a relatively short time to research and compose a well-sourced answer. The 56% preference demonstrates the model outperforms this specific population of contractors under these specific time constraints, not that it outperforms human experts in general.

• The evaluation is on ELI5 questions only. ELI5 questions are designed to be explainable to a general audience — they do not require specialized domain knowledge. The model's advantage might be smaller or reversed on questions requiring genuine expertise (legal, medical, or highly technical questions).

• The evaluation uses the same criteria as training. The comparison criteria for evaluating against demonstrations are "a very similar set of criteria" to those used for RM training. If the RM learned to prefer answers with certain stylistic properties that labelers also prefer (e.g., formal tone, multiple citations, cautious hedging), the 56% figure partly reflects the model learning to match evaluation criteria rather than producing objectively better answers.

Claim 3: "This model's answers are preferred 69% of the time to the highest-voted answer from Reddit."

Directly supported by Figure 2b. The 69% figure is substantially higher than the 56% against demonstrations, and the paper itself provides a thoughtful discussion of why the comparison against demos is "more meaningful" (Section 4.1). This is a strength of the paper: the authors are transparent about the limitations of the ELI5 comparison rather than simply reporting the larger number.

The concerns about this comparison are well-articulated by the authors:

• Fact-checking without references is harder for labelers, potentially biasing them against ELI5 answers that make claims without citations (even if those claims are correct).
• Style differences persist despite stripping citations, making blinding imperfect.
• ELI5 reference answers vary widely in quality, since they are real Reddit posts — some are excellent, others are low-effort. The comparison is against the distribution of actual Reddit answers, not against a controlled baseline of high-quality human writing.

The 69% figure should be understood as "WebGPT answers are strongly preferred to typical Reddit answers under these evaluation conditions" rather than "WebGPT definitively outperforms humans at question-answering."

Claim 4: "WebGPT's answers are true 75% of the time, and are both true and informative 54% of the time" on TruthfulQA.

Supported by Figure 3. The comparison against GPT-3 baselines is instructive — WebGPT clearly improves truthfulness over the base model — but the gap to human performance (94% truthful and informative) is enormous. WebGPT is still wrong or uninformative nearly half the time on this adversarially-constructed benchmark.

An important nuance: the TruthfulQA evaluation uses a truncated version of WebGPT's output (first 50 tokens). This may understate WebGPT's informativeness, since long-form answers sometimes front-load caveats before getting to the informative content. Table 3 shows WebGPT sometimes gives incorrect short-form answers (e.g., "It is true that you can make a wish come true by the power of thought") where a longer answer might have included qualifications. The truncation was necessary for compatibility with the TruthfulQA evaluation protocol, but it penalizes WebGPT's natural response style.

Missing experiments and analyses:

1. No direct comparison between WebGPT and GPT-3 on ELI5. The paper never evaluates how well GPT-3 (without browsing) answers ELI5 questions. This is a significant omission, since one of the central motivations is that web access improves factual accuracy. While the TruthfulQA comparison shows improvement over GPT-3, ELI5 is the primary benchmark, and the absence of a GPT-3 baseline on ELI5 makes it impossible to quantify how much of WebGPT's performance comes from browsing vs. from the base model's synthesis capabilities alone. The paper does compare against Krishna et al. (2021) on ELI5, but those are different base models.

2. No analysis of how browsing behavior changes with training stage. The paper never shows what browsing trajectories look like for BC vs. RL vs. rejection sampling models, or how the reward model scores correlate with specific browsing actions (e.g., number of queries, diversity of sources visited, number of quotes collected). This makes it difficult to understand why rejection sampling works — is it because the model visits better websites, quotes more relevant passages, or composes better answers from the same quotes?

3. No sensitivity analysis for the KL penalty coefficient in RL. The KL reward coefficient of 0.02 is stated but never ablated. Given that the paper's main RL finding is that RL underperforms rejection sampling, understanding whether a different KL budget could close the gap is important. The paper notes that the KL budget was tuned using human evaluations for 175B, but does not present the tuning curve.

4. No evaluation of answer quality as a function of browsing budget. All evaluations use a maximum of 100 actions, but the paper does not show how answer quality varies with the number of allowed actions (except for the randomization during RL training, which is not evaluated). This is a missed opportunity: if 20 actions produce comparable quality to 100 actions for easy questions, the system could be much more efficient.

5. Limited analysis of reference quality. The paper emphasizes that references enable evaluation, but never quantifies reference quality independently of answer quality. How often are the quoted references actually relevant and trustworthy? How does reference quality correlate with answer preference? The comparison annotation procedure includes trustworthiness ratings and claim-level support annotations, but these are only used in training (and the auxiliary loss experiments failed to improve the RM). The paper does not report statistics on these annotations.

6. The question stance experiment (Appendix H) is underpowered. With only 60 questions and 10 topics, the finding that affirming questions reduce accuracy is suggestive but not statistically rigorous. No confidence intervals or significance tests are reported for Figures 11 and 12. This is explicitly acknowledged ("too small of a sample size for us to draw definitive conclusions"), but the placement in the appendix reflects appropriate caution.

Conditional validity of claims:

• The 56% preference over demonstrations holds for ELI5 questions with the specific contractor population, time constraints, and evaluation criteria used. Generalization to other question types, other human populations, or other evaluation rubrics is not established.
• The advantage of rejection sampling over RL holds for this specific RM and environment. The paper's hypotheses about why (environment stochasticity, hindsight evaluation, distributional shift) are plausible but not experimentally validated. In environments with less useful stochasticity (e.g., tasks where the optimal action sequence is narrow and deterministic), RL might outperform rejection sampling.
• The compute-efficient frontier (Figure 8) is based on validation RM score, not human preference. While Figure 5 validates the RM prediction for the BC best-of-n setting, the frontier includes data points from RL models, where the validation RM may be less reliable (the RM was trained primarily on BC and rejection sampling data). The frontier should be interpreted as an approximation, not a precise optimization curve.

Overall assessment: The experiments are well-designed to support the paper's main contributions: WebGPT achieves human-competitive performance on ELI5 LFQA, human feedback via rejection sampling is essential to this performance, and the reference-based evaluation framework makes this feedback feasible. The paper is unusually candid about its limitations — the discussion of why the ELI5 comparison is less meaningful than the demo comparison, the acknowledgment of the small sample size in the stance experiment, and the transparency about the RM training data coming from an ad-hoc mixture all demonstrate intellectual honesty. The main weakness is the absence of baseline comparisons (GPT-3 on ELI5, performance vs. browsing budget) and process-level analyses (what browsing behaviors does the model learn, and how do they differ across training stages). These omissions do not invalidate the core claims but leave open questions about the mechanisms underlying the results.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for and Dominates the Headline Gains

The assumption or constraint. The compute-optimal framework requires estimating each question's difficulty before deciding how to allocate the inference budget. The paper's method for this — generating 2048 samples per question and averaging PRM final-answer scores — is extraordinarily expensive. The authors acknowledge this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. Generating 2048 samples to estimate difficulty consumes more compute than the largest test-time budgets studied (256–512 generations). In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter. The 4× figure should be understood as an upper bound on achievable efficiency rather than a realized deployment gain. For any practical system, the difficulty estimation overhead must be subtracted from the reported savings, potentially reducing or eliminating the advantage over uniform allocation.

What evidence exists in the paper. The 2048-sample difficulty estimation procedure is described in Section 3.2. Figures 4 and 8 show compute-optimal scaling curves for both oracle and predicted difficulty bins, but neither figure includes the cost of difficulty estimation in the x-axis budget. The paper does not report total FLOPs or wall-clock time for difficulty estimation, nor does it compare compute-optimal + difficulty estimation against best-of-N at an equivalent total budget.

Mitigation status. The paper explicitly flags this as "a key avenue for future work" (Section 3.2) and suggests training models to predict difficulty directly from question text, or using adaptive estimation that amortizes difficulty assessment into the problem-solving process. Neither approach is developed or evaluated. The predicted difficulty bins (using PRM scores) are a step toward removing ground-truth dependence, but they do not address the computational cost of the 2048 samples.

Hard Problems Remain Unsolved — Test-Time Compute Cannot Compensate for Fundamental Capability Gaps

The assumption or constraint. The entire framework assumes the base model's proposal distribution contains correct answers at some non-trivial rate. If the base model's pass@1 is near zero on a problem class, no amount of search or revision can help, because there are no correct solutions in the distribution to find or refine.

The consequence. On difficulty bin 5 (the hardest quintile of questions), accuracy remains at 1–3% across all methods and all budgets tested. The paper is transparent about this (Section 5.3, Figure 3 right, Figure 7 right, Figure 9), but the implication for deployment is stark: test-time compute amplifies existing capability but does not create it from nothing. A system deployed on a problem distribution with a substantial fraction of genuinely hard questions (outside the base model's reach) will see near-zero benefit from these methods, regardless of the inference budget. The FLOPs-matched comparison (Section 7) quantifies this: on hard problems at R ≫ 1, test-time compute shows a −52.9% disadvantage compared to simply using a 14× larger model.

What evidence exists in the paper. Figure 3 (right) shows bin 5 accuracy flatlining at 1–3% for all search methods. Figure 7 (right) shows bin 5 revision accuracy at roughly 2–3% regardless of sequential-to-parallel ratio. Figure 9 shows the bin 5 scaling curve essentially flat near 0–5% in the FLOPs-matched comparison. The paper states the implication directly in the Section 7 takeaway: test-time compute cannot substitute for pretraining on problems outside the base model's capability range.

Mitigation status. The paper does not attempt to solve this — it is treated as a fundamental boundary condition rather than a fixable limitation. The authors are candid that "pretraining remains the only viable path" for genuinely novel or out-of-distribution reasoning. No method is proposed for extending test-time compute benefits to problems where the base model's pass@1 is near zero.

Verifier Over-Optimization Is the Primary Scaling Bottleneck, and the Paper Does Not Solve It

The assumption or constraint. All optimization methods (beam search, lookahead search, RL, rejection sampling) rely on the PRM or ORM as a proxy for answer correctness. The verifier is an imperfect model trained on finite data, and aggressive optimization against it eventually finds solutions that score highly under the verifier but are actually incorrect — a phenomenon known as over-optimization or reward hacking.

The consequence. Verifier over-optimization creates a hard ceiling on test-time compute scaling that the compute-optimal policy only mitigates, not removes. The evidence is concrete: beam search degrades performance on easy problems at high budgets (Figure 3, right), lookahead search — the most powerful optimizer — paradoxically performs worst overall (Figure 3, left), and qualitative examples in Appendix M show degenerate outputs (repetitive low-information steps, overly short 1–2 step solutions) that score highly under the PRM. The compute-optimal policy routes easy problems away from aggressive search (using best-of-N instead of beam search), but on medium problems where beam search is deployed, over-optimization still limits the ceiling — the beam search curves flatten and sometimes decline well before the budget is exhausted (Figure 3, right, bins 3–4). This means the approach is fundamentally bounded by verifier quality, and the current results are specific to the verifier quality achievable with Monte Carlo rollout training.

What evidence exists in the paper. Figure 3 (right, bin 1) shows beam search accuracy decreasing from roughly 78% to 77% as budget increases from 4 to 256 generations — direct evidence of over-optimization. Figure 3 (left) shows lookahead search underperforming simpler methods at equivalent budgets due to its higher per-step cost exacerbating the effective optimization pressure. Appendix M provides qualitative examples of degenerate outputs. The paper does not systematically measure the over-optimization threshold (the budget at which verifier score and true accuracy diverge) or explore how it varies with PRM training data quantity or quality.

Mitigation status. The compute-optimal policy partially mitigates over-optimization by routing easy problems to weaker optimization methods (best-of-N rather than beam search), but this is a workaround, not a solution. The paper does not explore robust verifier training techniques (adversarial training, ensembles, calibration improvements), nor does it investigate constrained search methods (KL penalties, trust-region optimization) that could push the over-optimization threshold higher. Improving verifier robustness is flagged as a direction for future work but not addressed experimentally.

All Results Are on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*)

The assumption or constraint. The entire analysis — difficulty-dependent scaling, compute-optimal allocation, FLOPs-matched comparisons, verifier behavior — is conducted exclusively on the MATH benchmark (500 test questions) using PaLM 2-S* models. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified by any cross-model or cross-domain evaluation.

The consequence. Several aspects of the findings could be model-specific or domain-specific:

• The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with different calibration properties or different error patterns might exhibit qualitatively different difficulty-dependent scaling curves — for instance, beam search might over-optimize at different budget levels, or the optimal difficulty-bin strategy assignments might shift.
• The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families (some models are better at learning from examples than others).
• The MATH benchmark consists exclusively of competition-level math problems requiring symbolic multi-step reasoning. It is unclear whether the finding that beam search hurts easy problems (due to verifier over-optimization) generalizes to other reasoning domains (code generation, logical reasoning, scientific QA) or to tasks requiring factual knowledge rather than step-by-step deduction.
• The five-quintile difficulty binning and the specific optimal strategy assignments (e.g., best-of-N for easy, beam search for medium) are fitted to the MATH test set of 500 questions. With each quintile containing ~100 questions split into two cross-validation folds, the policy is selected based on ~50 questions per fold per bin — a small sample that may not produce robust strategy assignments.

What evidence exists in the paper. The paper does not report any results on benchmarks other than MATH for the main scaling and compute-optimal analyses. No cross-model evaluation is performed (e.g., applying the same training and evaluation protocol to a different model family to test whether the difficulty-dependent patterns replicate). The TruthfulQA and TriviaQA evaluations in Section 4 test the WebGPT model's factual accuracy and transfer, but these are short-form QA tasks that do not exercise the test-time compute scaling framework. The paper does not report confidence intervals on the compute-optimal scaling curves, making it difficult to assess whether the observed strategy assignments are statistically reliable at the ~50-question-per-bin sample size.

Mitigation status. The paper does not claim cross-domain or cross-model generalization; the authors are explicit that they used MATH and PaLM 2-S*. Extension to other domains and models is left to future work. The small sample size (500 questions, five bins) is not discussed as a limitation.

Revisions and PRM Search Are Studied Independently, Never Combined

The assumption or constraint. The paper studies two complementary axes — PRM tree-search (Section 5) and iterative revisions (Section 6) — as independent mechanisms for improving test-time performance. They are never combined into a single system (e.g., using the revision model as the proposal distribution within beam search, or using the PRM to guide which revisions to pursue).

The consequence. The reported results represent a lower bound on what a fully integrated system could achieve. The two mechanisms have complementary, difficulty-dependent strengths: revisions improve the proposal distribution (generating better candidates through iterative refinement, most effective on easy problems), while PRM search improves candidate selection (finding the best among generated candidates, most effective on medium-hard problems). Applying beam search to revision model outputs — or using the PRM's step-level scores to decide when a revision is on track versus when to restart — could yield gains beyond either method alone, particularly on medium-difficulty problems where both mechanisms show partial effectiveness individually. Without this combination, the paper cannot determine whether the difficulty-dependent benefits of each method are additive, redundant, or even interfering.

What evidence exists in the paper. Section 8 explicitly acknowledges this gap: "we did not experiment with PRM tree-search techniques in combination with revisions." The paper does not provide any analysis of why the combination was not attempted (computational constraints, engineering complexity, or negative preliminary results). No estimate is given for the potential magnitude of combined gains.

Mitigation status. The gap is acknowledged but not addressed. Combining search and revisions is suggested as a natural direction for future work (Section 8). The paper's framework — decomposing test-time compute into proposal distribution modifications and verifier optimization — provides the conceptual scaffolding for combining them, but no experimental evidence is provided for whether and how they interact.

The 14× Larger Model Baseline Is Not Compute-Optimally Trained, and the Larger Model Receives No Test-Time Compute Budget

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters by ~14× while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than Chinchilla-optimal training (Hoffmann et al., 2022), where both data and parameters would be scaled equally. The authors acknowledge this explicitly:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the larger model uses greedy decoding only — no majority voting, no best-of-N, and no search — meaning it receives no test-time compute augmentation of its own.

The consequence. The reported advantages of test-time compute over pretraining (e.g., +27.8% relative improvement on easy questions at R ≪ 1 for revisions, as shown in Figure 1) are measured against a weaker baseline than they could be. A Chinchilla-optimal model trained with 14× more total FLOPs (scaling both parameters and data) would likely outperform a parameter-only-scaled model. Furthermore, giving the larger model even a modest test-time compute budget (e.g., best-of-8 majority voting, or a simple PRM reranking) would create a much stronger baseline that the paper never tests. The FLOPs-matched comparison therefore compares a compute-optimal inference strategy (for the smaller model) against a potentially suboptimal training strategy (for the larger model). This makes the comparison somewhat favorable to test-time compute in ways that are hard to quantify without additional experiments.

What evidence exists in the paper. Section 7 describes the FLOP accounting and acknowledges the parameter-only scaling choice. The paper does not report results for a Chinchilla-optimal larger model, nor for a larger model with any test-time compute augmentation. The greedy decoding baseline (stars in Figure 9) is placed at a single point per R value — there is no curve showing how the larger model's performance would scale with its own test-time compute budget.

Mitigation status. The authors flag the parameter-only scaling choice as a limitation and suggest a full compute-optimal pretraining comparison as future work. Giving the larger model test-time compute is not discussed as a possibility, even though it would be the most direct way to test whether test-time compute truly "substitutes" for pretraining or simply represents a better allocation of total compute across training and inference for both model sizes.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper has already changed the landscape of long-form question-answering in a fundamental way, even if its specific architectural choices (GPT-3 + Bing API) are not directly replicated. The shift is not about a particular model or training recipe — it is about establishing a template for training language models to interact with external tools using human feedback on the final output, without requiring supervision on intermediate actions. This is a conceptual shift whose influence extends well beyond LFQA.

The core reframing: tool use as a policy learning problem, not a retrieval architecture problem. Before WebGPT, the dominant approach to giving language models access to external knowledge was architectural: design a differentiable retrieval mechanism (DPR, REALM, RAG) that the model can be trained with end-to-end. These methods treated retrieval as a computation to be integrated into the model. WebGPT treats retrieval as an environment to be acted upon by the model. The model learns to use an existing tool (a search engine) through a text-based interface, and the entire interaction — query formulation, link selection, navigation, extraction — is optimized through the single lens of human preference on the final answer. This reframing separates the problem of "building a good retriever" (solved by search engines) from "knowing how to use retrieval effectively" (solved by policy learning), and it has proven remarkably productive. The line of work from WebGPT to today's browsing-capable models (including tool-use and function-calling APIs now standard in deployed LLMs) traces directly through this conceptual shift.

The magnitude of this shift is best understood by what it makes possible that was previously impossible. A RAG system can retrieve documents from a fixed corpus and generate an answer. WebGPT can issue multiple queries, follow chains of links, read pages, decide to backtrack, and collect evidence from diverse sources — all behaviors that emerge from optimizing answer quality rather than being explicitly programmed. The fact that the model's browsing strategy is learned rather than scripted means it can adapt to the question: a simple factual question might require one search and a quick quote, while a complex explainer might require visiting multiple sources and synthesizing across them. This adaptivity is not a feature of the architecture — it is a learned behavior driven by the reward signal.

Reconciling prior contradictions around human feedback for factual tasks. A significant tension existed in the field before WebGPT: human feedback was known to be effective for subjective tasks like summarization (Stiennon et al., 2020), but there was skepticism about whether it could work for factual tasks where correctness is objective but difficult to verify. The concern was straightforward: if labelers cannot reliably distinguish true from false claims, reward model training will be noisy, and the policy might optimize for appearing correct rather than being correct. WebGPT's key insight — that requiring the model to collect and cite references transforms the evaluation from "Is this claim true?" to "Is this claim supported by the provided references?" — resolves this tension by changing the evaluation task rather than trying to solve the fundamental difficulty of fact-checking. The 73% labeler-labeler agreement rate and 74% researcher-labeler agreement rate (Appendix C) demonstrate that the support-by-reference judgment is tractable enough to produce a useful training signal, even though it is imperfect.

This reconciliation has had cascading effects. It showed that the human-feedback paradigm (behavior cloning + reward modeling + optimization) could extend beyond subjective domains into factual ones, provided the evaluation framework was appropriately designed. This opened the door for subsequent work on training models to be truthful, to provide evidence for their claims, and to assist their own evaluation — all themes that are now central to AI alignment research. The paper's explicit connection to debate (Irving et al., 2018) and recursive reward modeling (Leike et al., 2018) was prescient: these approaches all rely on the idea that models can make their outputs verifiable by producing evidence, and WebGPT provided one of the first demonstrations that this can work in practice with real human labelers.

Making inference-time compute scaling a first-class design consideration. The compute-efficient frontier analysis (Figure 8) established that for a fixed inference budget, the optimal configuration is not always the largest model with a single sample. The three configurations on the frontier (760M best-of-4, 13B best-of-16, 175B best-of-64) are not arbitrary — they represent a systematic exploration of how to trade model size against rejection sampling depth. This analysis predates and anticipates the current wave of interest in inference-time compute scaling (the "test-time compute" literature), but with an important distinction: WebGPT's rejection sampling is a form of selection among independently generated trajectories, while more recent work explores sequential computation (chain-of-thought, tree search, iterative refinement). The broader principle — that inference compute is a design dimension that can be optimized, not a fixed cost — is the same, and WebGPU provided one of the earliest empirical demonstrations with real human evaluations.

Which research directions become more attractive. The paper implicitly argues that improving search and retrieval algorithms is less urgent than improving how models use retrieval tools. If a 175B model with best-of-64 can match or exceed human demonstrators on ELI5 using only a standard search API, then the bottleneck is not retrieval quality — it is the model's ability to navigate, filter, and synthesize retrieved information. This shifts research attention toward training methods that improve browsing strategy (better reward models, better exploration in RL, better credit assignment across long episodes) rather than toward building better search engines or document embeddings. The paper's finding that RL underperforms rejection sampling (Figure 4) also suggests that PPO-style optimization in stochastic, long-horizon environments is a harder problem than the summarization results from Stiennon et al. (2020) might suggest, and that simpler selection-based methods deserve more attention than they had received.

Which research directions become less attractive. The paper does not directly argue against differentiable retrieval, but its results weaken the case for investing heavily in that direction for LFQA. If a black-box search engine accessed through a text interface can support human-competitive performance, then the marginal benefit of making retrieval end-to-end differentiable — with all the architectural complexity and corpus-management overhead that entails — is questionable, at least for tasks where live web access is feasible. The paper also implicitly argues against the idea that larger models alone can solve factual accuracy: GPT-3's TruthfulQA performance is substantially worse than WebGPT's (Figure 3), and WebGPT's performance improves with model size while GPT-3's does not. This suggests that grounding in external sources, not just scaling parameters, is essential for truthfulness — a finding that has become a cornerstone of subsequent work on retrieval-augmented generation.

Follow-Up Research This Work Enables

Training models to directly predict question difficulty or browsing complexity from the question text. The paper's compute-efficient frontier (Figure 8) shows that the optimal amount of rejection sampling depends on the available compute budget, but it treats all questions uniformly — every question gets the same $n$ in best-of-$n$. Intuitively, some questions are answerable from a single authoritative source (requiring few browsing attempts) while others require synthesizing information across multiple sources (benefiting from more exploration). A natural extension would be to train a lightweight classifier — possibly a smaller model or even a linear probe on top of the BC model's question embedding — to predict, from the question text alone, how many browsing attempts are likely to be needed, or which reward model score a best-of-1 answer would receive. If such a classifier could achieve reasonable accuracy, it would enable per-question allocation of the inference budget: easy questions get 4 samples, hard questions get 64, with the total budget constrained. The paper already provides the infrastructure for this experiment: the validation reward model can serve as the ground-truth difficulty signal, and the existing comparison dataset includes questions with widely varying best-of-1 RM scores that could be used for training. A strong follow-up would measure the total RM score achieved under a fixed total sample budget with and without per-question adaptive allocation, using the same ELI5 test set.

Analyzing how browsing behavior changes with model scale and how it correlates with answer quality. The paper reports aggregate performance metrics (human preference, RM score) but provides almost no analysis of the browsing trajectories themselves. Open questions that the existing data could answer include: Do larger models issue more diverse search queries? Do they visit more unique domains? Do they collect more quotes, or longer quotes, or quotes from higher-quality sources? Do they spend more actions browsing before answering? The paper has an entire dataset of browsing trajectories from models of three sizes (760M, 13B, 175B) with various numbers of rejection samples — this data could be analyzed to understand how scaling model size and rejection sampling improve performance mechanistically, rather than just that they do. A strong follow-up would correlate trajectory-level features (number of searches, domain diversity, quote count, browsing duration, source PageRank or domain authority) with final answer quality (as measured by RM score or human preference), and would examine how these correlations change with model size. The finding in Figure 6 (larger models benefit more from additional demonstrations) hints that larger models learn qualitatively different browsing strategies, but what those strategies are remains unexplored.

Iterative training: using rejection-sampled answers to retrain the policy and reward model. The paper's training pipeline is one-pass: collect demonstrations → BC → collect comparisons → RM → rejection sampling. A natural next step is to close the loop: use the best-of-$n$ model to generate higher-quality demonstrations (by having humans edit or approve the best sampled answer, which would be faster than writing from scratch), use these to retrain the BC policy, generate new comparison data from the improved policy, retrain the RM, and iterate. This is the "data flywheel" that the paper alludes to but does not execute. The key question is whether iterative training improves over a single pass, or whether the gains saturate quickly (as the scaling curves in Figures 6 and 7 suggest they might — the diminishing returns from additional demonstrations and comparisons are already visible at the dataset sizes used). A strong follow-up would run 2–3 iterations of this loop, measuring human preference after each iteration, and would also test whether the iterative process amplifies biases (e.g., does the policy converge toward sources that the RM overvalues?). The observation in Appendix H that the model "tends to perpetuate and reinforce existing assumptions and biases" makes this bias-amplification question particularly important for iterative training.

Evaluating WebGPT-style browsing on questions requiring genuine expertise or multi-step reasoning across sources. ELI5 questions are designed to be answerable by synthesizing information from generally accessible web sources — they do not require specialized domain knowledge, complex multi-step logical inference, or reconciling contradictory claims across sources. This makes ELI5 a relatively forgiving testbed. A more probing evaluation would test WebGPT on question sets that require: (1) consulting primary sources (scientific papers, legal documents, financial filings) rather than explanatory journalism, (2) resolving contradictions between sources (e.g., questions where different authoritative sources give different answers), or (3) performing quantitative reasoning that requires extracting numbers from multiple pages and combining them. The TruthfulQA evaluation (Section 4.2) already hints at limitations: WebGPT sometimes quotes unreliable sources when answering adversarial questions. A strong follow-up would curate or construct a benchmark of ~200 questions explicitly designed to stress-test these capabilities, evaluate the 175B best-of-64 model on it, and compare against human experts (not contractors) with the same time constraints and browser access. The key metric would be not just overall preference but the specific failure modes: does the model fail because it cannot find the right sources, because it cannot understand them, or because it cannot synthesize across them?

Stress-testing the reference-based evaluation paradigm on domains where source reliability is contested. The paper's central methodological innovation — using references to make factual evaluation tractable for non-expert labelers — assumes that labelers can reliably judge whether a source supports a claim and whether a source is trustworthy. This assumption is tested only on ELI5 questions, where most answers come from mainstream explanatory sources (news websites, educational sites, Wikipedia). It may break down on questions where reliable sources disagree, where the consensus view is not captured by easily-found web pages, or where the model learns to cherry-pick sources that appear credible to non-expert labelers but are actually fringe or unreliable. The paper flags this concern explicitly (Section 6.4): "Our current procedure incentivizes models to cherry-pick references that they expect labelers to find convincing, even if those references do not reflect a fair assessment of the evidence." A strong follow-up would construct a set of questions on contested topics (climate change, vaccine efficacy, historical controversies) where there exists a clear expert consensus but also easily-found dissenting sources, and test whether WebGPT's answers align with the consensus or with the most findable sources. This would directly probe the cherry-picking concern and the robustness of the reference-based evaluation framework.

Combining rejection sampling with debate or recursive reward modeling for harder factual questions. The paper draws explicit connections to debate (Irving et al., 2018) and recursive reward modeling (Leike et al., 2018), which are frameworks for training AI systems to assist their own evaluation on questions too difficult for unaided humans to judge. WebGPT's reference-based evaluation can be seen as a degenerate case of debate: the model presents evidence (references), and the human judges whether the evidence supports the claim. But what if the model could also present counter-evidence, or argue for why its sources are more reliable than alternative sources? A natural extension would be to train two WebGPT-style models — one arguing for an answer and one arguing against it — and have labelers judge which side is better supported, using the debate structure to surface cherry-picking and source reliability issues that single-answer evaluation misses. The paper's existing comparison interface (Figure 9) already has labelers evaluate source trustworthiness and claim-level support; extending this to an adversarial setting would be a concrete first step. A strong follow-up would implement a 2-turn debate (model A answers, model B critiques, model A responds) on a set of 100 ELI5 questions, compare human preference for debate-informed answers versus standard single-model answers, and measure whether debate reduces the rate of unsupported claims.

Practical Applications and Downstream Use Cases

Deploying long-form QA with verifiable citations in consumer-facing products. The most direct application of WebGPT's methodology is in search engines, voice assistants, or knowledge interfaces that need to answer user questions with paragraph-length explanations rather than just a list of links. The key value proposition is not just answer quality — it is verifiability through citations. A user who receives a WebGPT-style answer can click on the references to verify claims themselves, which builds trust and enables informed reliance on the output. The paper's numbers support this application: the 175B best-of-64 model's answers are preferred 69% of the time to the highest-voted Reddit answer (Figure 2b), and 56% of the time to human demonstrators (Figure 2a). The cost of running best-of-64 with a 175B model is substantial, but the 760M best-of-4 configuration (at ~10^14 FLOPs in Figure 8) achieves a meaningful fraction of the quality at a small fraction of the compute, making deployment feasible for high-volume applications where per-query cost matters. The compute-efficient frontier in Figure 8 provides a direct engineering guide: for any given latency or cost budget, choose the model size and rejection sampling depth that maximizes expected answer quality.

Automated fact-checking and claim verification for content moderation or journalism. The reference-based evaluation framework — collecting sources, extracting quotes, and judging whether claims are supported — is essentially a fact-checking pipeline. While WebGPT was trained to generate answers, the same infrastructure could be used to verify claims: given a claim and a browsing environment, can the model find sources that support or refute it? The ELI5 fact-check dataset mentioned in Appendix B (67 demonstrations, 185 comparisons) is a small-scale version of this: questions were formatted as "Fact-check each of the claims in the following answer." The paper does not report separate results on this dataset, but the methodology is directly transferable. A fact-checking system built on this approach would be transparent (every verification includes cited sources), scalable (the reward model can be trained to predict human fact-checker judgments), and updatable (live web access means it checks against current information). The key limitation, flagged in the paper, is that the system may learn to find sources that appear to support a claim rather than sources that reflect expert consensus — mitigation would require careful design of the reward signal and labeling protocol.

Training data generation for retrieval-augmented models through self-play or teacher-student distillation. WebGPT's browsing trajectories — the sequence of queries, link clicks, and quotes that lead from a question to an answer — are a rich source of training data for other retrieval-augmented systems. A larger, more expensive WebGPT model (e.g., 175B best-of-64) can generate high-quality (question, browsing trajectory, answer) triplets that can then be used to train a smaller, more efficient retriever-reader model through distillation. This is particularly valuable because the trajectories include how to find relevant information (which queries to issue, which links to follow), not just what information was found. The paper's finding that the 175B model benefits from more demonstrations (Figure 6) suggests that additional high-quality demonstration data would further improve BC performance; using the model to generate that data through a human-in-the-loop filtering step (humans approve or edit the best answers) could combine the scalability of model generation with the quality control of human oversight. The compute-efficient frontier (Figure 8) provides a framework for thinking about this tradeoff: for a fixed data-generation budget, how should one allocate between model size, rejection sampling depth, and human review?

When to Prefer This Method

The paper does not present WebGPT as one option among a set of named alternatives with explicit tradeoffs. It is introduced as a general approach to LFQA using a web-browsing environment and human feedback, and the paper's comparisons are primarily against prior work (RAG, REALM, UnitedQA) and against ablations of its own components (BC vs. BC+RL vs. BC+rejection sampling). The decision rule implicit in the results is:

• Prefer rejection sampling (best-of-$n$) over RL when inference-time compute is available and the environment has high useful stochasticity — the web-browsing setting meets both conditions, and best-of-64 BC substantially outperforms the RL model (68% vs. 58% preference over BC baseline, Figure 4). This is the paper's clearest prescriptive finding about method choice.

• Use a compute-efficient combination of model size and rejection sampling depth rather than defaulting to the largest model. Figure 8 provides the empirical basis for this: for any given FLOPs budget, there exists a model size and $n$ that maximizes expected reward. This is a practical engineering guideline rather than a theoretical principle, but it is directly actionable.

• Require models to collect and cite references if you need to evaluate factual accuracy with human labelers who are not domain experts. This is the paper's core methodological prescription, and it is supported by the inter-labeler agreement data (73%) and the feasibility of collecting ~21,500 comparisons.