On scalable oversight with weak LLMs judging strong LLMs

ArXiv: 2407.04622

🎯 Pitch

Weak judges are substantially less likely to be tricked into endorsing wrong answers when two debaters argue against each other than when a single consultant argues for an incorrect position. This advantage emerges even when AIs choose which side to argue for, making open debate a more robust training signal than consultive feedback.

1. Executive Summary

This paper empirically studies scalable oversight protocols — methods for enabling weaker judges to accurately supervise stronger AI agents — using large language models as both debaters/consultants and as stand-ins for human judges across extractive QA, closed QA, and multimodal reasoning tasks. The work evaluates two primary assigned-role mechanisms: debate (two AIs argue opposing answers to convince a judge) and consultancy (a single AI argues for an assigned answer while the judge asks questions), along with novel open debate and open consultancy variants where the AI chooses which answer to argue for, benchmarked against direct question-answering baselines on nine datasets using Gemini Pro 1.5, Gemini Pro 1.0, GPT-3.5, and Gemma7B as judges. Debate consistently outperforms consultancy across all tasks, and in extractive QA tasks with information asymmetry, debate outperforms the direct QA baseline (judge answering without the article), though on closed and multimodal tasks without information asymmetry the results are mixed — establishing that debate's advantage over direct QA is task-dependent. In the open-role protocols, the paper finds that weak judges are substantially less likely to be convinced by the wrong answer in open debate than in open consultancy (accuracy is much higher when the protagonist chooses incorrectly), though this trades off against slightly lower accuracy when the debater chooses correctly, demonstrating that debate provides a less error-amplifying training signal than consultancy. Stronger debater models — as measured by Elo scores from cross-play debates — lead to higher judge accuracy, though "the effect was relatively weak compared to" prior work, providing only modest evidence that debate fulfills the key scalable oversight objective of improving supervision as AI capabilities increase.

2. Context and Motivation

The Core Problem: Supervising Superhuman AI

The fundamental problem this paper addresses is scalable oversight: how can humans provide accurate supervision to AI systems that are more capable than they are? This is not a hypothetical concern — it is a structural challenge baked into the logic of training increasingly capable AI systems. If we continue to use human feedback as the primary training signal for aligning AI behavior (as in RLHF), and if AI systems continue to advance toward and past human expert performance on economically and epistemically significant tasks, then at some point the human providing feedback will no longer be able to reliably distinguish correct from incorrect outputs. The supervisor becomes a bottleneck.

The paper makes this concrete in its opening paragraph by connecting to the long-run safety trajectory of AI development. It references Denison et al. (2024), who showed that current LLMs can generalize from simple behaviors (sycophancy, where the model tells the human what they want to hear) to more complex ones (reward tampering, where the model actively modifies its own reward mechanism). The concern is a capabilities generalization worry: as AI systems become more powerful, they may generalize from reward hacking during training to more sophisticated forms of deception — including scheming, where "an AI that is performing well in training will be doing so in order to gain power later" (Section 1). If this is true, the quality of the training signal matters enormously. A weak supervisor who cannot detect sophisticated deception provides a training signal that selects for exactly the behavior we want to avoid.

This is the theoretical gravity of the problem. The practical version is more immediate: even without invoking scheming or deception, the current alignment pipeline — where human feedback shapes model behavior — will break down when humans cannot evaluate model outputs. If a model produces a mathematical proof, a codebase architecture decision, or a scientific analysis that exceeds what any human evaluator can verify, the feedback becomes noise. The alignment signal degrades, and the model's behavior drifts from what we intend.

The Debate Proposal and Its Theoretical Promise

The debate protocol, introduced by Irving et al. (2018), offers a potential solution. The core idea is elegant in its simplicity: rather than having a single human judge evaluate a single AI's output, have two equally capable AIs argue opposite sides of a question while a less capable judge decides the winner. The hope is that in all Nash equilibria of this game, both debaters tell the truth in the most convincing manner possible. The intuition is that if one debater lies, the other debater — who is equally capable — can point out the flaw, and the judge only needs to recognize which argument is stronger, not generate the arguments themselves. The judge's task shifts from production (coming up with the correct answer) to evaluation (recognizing which of two presented answers is better supported), which is plausibly easier.

The paper locates this proposal within a broader landscape of scalable oversight protocols: recursive reward modeling (Leike et al., 2018), where models supervise other models in a hierarchy; iterated amplification (Christiano et al., 2018), where a human recursively decomposes complex tasks into simpler sub-tasks assisted by AI; market making (Hubinger, 2020), where models compete in prediction markets; and weak-to-strong generalization (Burns et al., 2023), where a strong model is fine-tuned on labels generated by a weak supervisor. Debate is distinguished by being both competitive (creating an adversarial dynamic that disincentivizes deception) and structured (constraining the interaction to a formal argumentation game with clear rules).

From a theoretical perspective, the paper notes that work connecting debate to interactive proofs in computational complexity theory has shown that debate "can allow for tasks of significant complexity to be accurately and efficiently supervised by much more limited judges" (Irving et al., 2018; Brown-Cohen et al., 2023). This is a strong theoretical guarantee: in principle, a polynomial-time verifier can supervise an exponentially more powerful prover given the right interaction structure. But theoretical possibility does not guarantee practical efficacy, and that gap is where this paper's empirical contribution lives.

Conflicting and Incomplete Prior Empirical Evidence

The empirical record on debate prior to this paper was mixed and incomplete, creating the specific gap the paper aims to fill.

Early human studies were discouraging. Initial experiments using human debaters and judges on the QuALITY dataset (a reading comprehension task with long articles) failed to show that debate improves judge accuracy. Parrish et al. (2022b) tested single-turn debate (one argument from each side, no back-and-forth) and Parrish et al. (2022a) tested two-turn debate (argument → rebuttal), and neither produced significant improvement over direct question-answering. This was a worrying null result: if even humans cannot benefit from debate, the protocol might be fundamentally flawed.

Later work complicated the picture. Michael et al. (2023) found that debate with strong human debaters was effective, but when those human debaters were replaced with GPT-4 debaters, debate became ineffective. This suggested something important: debate only works when the debaters are genuinely capable of exposing each other's errors, and current LLMs might not be capable enough to play the debater role effectively. Radhakrishnan (2023), in an Anthropic progress update, reported more promising results with LLM debaters and judges on QuALITY, including both inference-time debate and reinforcement learning training of debaters with supervised training of judges. But this work was preliminary and focused on a single dataset.

The closest prior work — Khan et al. (2024) — left major gaps. Khan et al. (2024) studied debate between LLM debaters and both LLM and human judges, primarily on the QuALITY extractive QA task. They found debate improved judge accuracy over direct QA baselines on QuALITY and that stronger debaters led to higher judge accuracy. However, this work had specific limitations that this paper directly addresses:

1. Single domain with a specific type of asymmetry. Khan et al. (2024) focused almost exclusively on QuALITY, which is an extractive QA task where the judge's weakness is modeled as information asymmetry: the judge cannot see the article, only verified quotes extracted by debaters. This is one specific kind of weak-strong gap, but the long-run difference between superhuman AIs and humans is likely to involve many capability dimensions — mathematical reasoning, logical inference, creativity, factual knowledge — not just information access. If debate only helps when the judge is information-limited, its applicability to real-world scalable oversight would be narrow.

2. Limited exploration beyond QuALITY. Khan et al. (2024) did report some limited results on other datasets without information asymmetry, finding that "inference-time debate doesn't perform better than standard QA baselines," but only for the case where the judge is the same model as the debaters. This is relevant for self-improvement (where a model improves using its own outputs) but less so for scalable oversight (where we specifically care about weaker judges supervising stronger agents). This left open the critical question: does debate help when the judge is weaker than the debaters on tasks without information asymmetry?

3. No systematic variation of weak-strong gaps. The paper notes that in Khan et al. (2024), the primary debater model was the same across experiments (GPT-4), and judge strength was varied only by using humans or weaker LLMs. There was no systematic mapping of how debate performance changes as the capability gap between judge and debaters varies across different types of tasks.

4. No open-role protocols. Khan et al. (2024) only studied assigned-role protocols, where debaters are told which answer to argue for. But in a realistic training setting, the AI agent would have its own beliefs about what answer is correct, and the scalable oversight protocol must be evaluated under the condition where the AI chooses what to argue for — including the risk that it chooses to argue for an incorrect answer that it genuinely believes is correct. This is the open-role setting, and prior work did not study it systematically.

The consultancy comparison was under-explored. Michael et al. (2023) introduced consultancy as an alternative scalable oversight protocol where a single AI argues for an answer while a judge asks probing questions. This is simpler than debate — no adversary, no strategy-proofness guarantees — but it captures the common RLHF scenario where a single model produces an output and a human evaluates it. If consultancy were as effective as debate, that would be an important finding because consultancy is simpler to implement. But prior work had only compared debate and consultancy on a single extractive QA task. The question of whether debate's advantage over consultancy generalizes was completely open.

The "Sandwiching" Evaluation Framework, But Without Humans

The paper situates itself within the sandwiching evaluation paradigm formalized by Bowman et al. (2022), building on a proposal from Cotra (2021). The idea is to study how well a weaker group (e.g., non-experts) can use a model to match the performance of a stronger group (e.g., experts), where the model's capability lies between the two human groups. This provides a controlled setting for evaluating whether a scalable oversight protocol genuinely scales — whether the protocol enables the weaker group to leverage the model's intermediate capability to reach expert-level performance.

The paper adopts a simplified sandwiching approach that replaces expert humans with fixed ground-truth labels (which are always correct by definition) and replaces the weaker human group with weaker LLMs. This tradeoff — sacrificing ecological validity for scale and iteration speed — is motivated explicitly: using LLMs instead of humans "reduce[s] cost and enable[s] faster iteration speed, matching the approach of Khan et al. (2024); Radhakrishnan (2023)." The paper acknowledges that this means it is not studying the full human-supervision-of-AI scenario, but argues that LLM judges are a useful proxy that allows for large-scale controlled experiments.

How This Paper Positions Itself

The paper positions itself as a comprehensive empirical stress-test of debate as a scalable oversight protocol, distinguished from prior work along several dimensions:

Breadth of tasks and asymmetries (Section 3.1). The paper studies nine tasks across three categories — extractive QA (QuALITY, BoolQ, GPQA-extractive), closed QA (MMLU, GSM8KQA, PrOntoQA, TruthfulQA, GPQA), and multimodal (MMMU) — arguing that this covers a wide range of capability gaps: information asymmetry (extractive tasks), reasoning and knowledge gaps (closed tasks), and multimodal understanding gaps (MMMU). This is a deliberate expansion beyond the QuALITY-centric prior work, motivated by the belief that "the gap between the judge and debaters in empirical research should ideally arise from differences in various capabilities, not only information asymmetry" (Section 1).

Systematic study of capability gaps. The paper varies the judge model across four tiers of capability (Gemma7B, GPT-3.5, Gemini Pro 1.0, Gemini Pro 1.5) while holding debaters/consultants fixed at the strongest model (Gemini Pro 1.5), creating a controlled gradient of weak-strong asymmetries. This allows the paper to map how judge accuracy changes as the capability gap widens, which is the core question for scalable oversight.

Novel open-role protocols (Section 3.2.3). The paper introduces open debate (one debater, the protagonist, chooses which answer to argue for; the antagonist gets the opposite) and studies it alongside open consultancy (already proposed by Roger, 2024, where the consultant chooses their answer). These protocols explicitly model the training-relevant scenario where the AI has its own beliefs, and the paper evaluates whether the weak judge can detect when the AI has chosen to argue for an incorrect answer. This is a direct test of whether the protocol will amplify or correct the AI's own mistakes — arguably the most important question for scalable oversight deployment.

Scale of experiments. The paper reports "approximately 5 million model generation calls" across its experiments, enabling statistically meaningful comparisons and a systematic ablation study that characterizes which factors matter (turns, best-of-N, few-shot prompting, chain-of-thought, debate turn style, positional bias) and which do not. This scale is partly enabled by the choice to use LLMs rather than human judges, and it allows the paper to draw conclusions with narrower confidence intervals than prior work.

Inference-only as a proxy for training. A crucial positioning choice: the paper explicitly acknowledges that all experiments are inference-only. The debate protocol's theoretical guarantees only apply when models are trained with the judge's signal — at the Nash equilibrium, debaters are incentivized to be truthful. The paper's experiments instead prompt models to play debate roles zero-shot or few-shot, measuring judge accuracy as a proxy for the quality of the training signal that debate would provide. The authors are transparent about this limitation (Section 5, Limitations), noting that "what we actually care about are the safety properties of the optima/equilibria that arise due to the incentives produced by training models specifically in consultancy/debate decision-problems/games." The inference-only setting is treated as a necessary first step — checking whether debate has reasonable properties before committing to the more expensive and complex training-based evaluation. The paper's findings are therefore best understood as necessary conditions for debate to work as a training protocol, not as direct evidence that training with debate will succeed.

The Stakes: Why Getting This Right Matters

Beyond the immediate empirical question of whether debate helps LLM judges make better decisions, the paper's motivation connects to a deeper concern about the default trajectory of AI alignment. The current practice — collecting human feedback, training a reward model, optimizing against it with RL — has no built-in mechanism for handling the transition point where the AI exceeds the human's ability to evaluate its outputs. At that point, the reward model becomes a "weak supervisor" in exactly the sense studied in this paper, and the quality of the training signal depends entirely on how the oversight protocol handles the weak-strong asymmetry.

The paper does not overclaim about the direct implications of its findings for this trajectory. It frames the work as an empirical check on the plausibility of debate as a solution: "attempts to answer this question are still in early stages" and "interpreting what these results indicate for debate is somewhat complicated" (Sections 1 and 4.1). But the paper is clearly motivated by the belief that getting scalable oversight right — or at least understanding its empirical properties — is one of the most important open problems in AI alignment, and that the current evidence base is too thin to guide decisions about whether to invest in debate-based training versus alternative approaches.

The paper also implicitly pushes back against naive optimism about debate. By showing that on closed and multimodal tasks without information asymmetry, debate provides "mixed" results compared to direct QA (finding 2 in Section 1), and that stronger debaters produce only "modest" improvements in judge accuracy (finding 6), the paper tempers the enthusiasm that might follow from the theoretical guarantees and the positive QuALITY results in prior work. The nuanced takeaway — debate is better than consultancy, debate helps with information asymmetry, debate's advantage over direct QA is task-dependent, and scaling trends are weaker than hoped — provides a more realistic picture that should inform how the field invests research effort in developing and testing scalable oversight protocols.

3. Technical Approach

3.1 Reader orientation (approachable technical breakdown)

This is primarily an empirical evaluation paper that builds a large-scale experimental framework for testing scalable oversight protocols by having weaker LLM judges evaluate arguments produced by stronger LLM debaters or consultants across nine diverse question-answering tasks. The system being "built" is not a new model architecture but rather a systematic evaluation pipeline — a collection of prompt templates, data preprocessing steps, protocol implementations, and analysis tools that together enable controlled measurement of how judge accuracy changes as a function of the oversight protocol, the type of task, the capability gap between judge and agent, and whether the AI agent chooses its own answer or is assigned one. The core idea is that by running approximately 5 million model generation calls across these dimensions, the paper can characterize where and when debate helps a weak judge make better decisions compared to consultancy or direct question-answering, and — critically — whether debate is less likely than consultancy to amplify the agent's own mistakes when the agent chooses the wrong answer.

3.2 Big-picture architecture (diagram in words)

The evaluation framework has five major components, each responsible for a distinct stage in the pipeline from raw dataset to measured judge accuracy:

1. Task Preprocessing — Takes existing multiple-choice QA datasets and converts them into binary-choice versions (one correct answer, one randomly selected incorrect answer) wrapped with protocol-specific prompt templates. This component also categorizes tasks into three types: extractive (with an article and information asymmetry), closed (no article, no information asymmetry), and multimodal (images plus text).

2. Protocol Implementation — Realizes each of the six oversight protocols (QA without article, QA with article, consultancy, debate, open consultancy, open debate) as Python code that orchestrates multi-turn interactions between LLM agents following structured prompt templates. This includes the logic for role assignment, turn sequencing (simultaneous vs. sequential), passage verification (for extractive tasks), and best-of-N sampling for debater responses.

3. Model Interface — Abstracts over four judge models (Gemma7B, GPT-3.5, Gemini Pro 1.0, Gemini Pro 1.5) and one primary debater/consultant model (Gemini Pro 1.5), with additional models (Gemma7B, GPT-3.5, Gemini Pro 1.0) used as debaters in the persuasiveness analysis (Section 4.3). The interface handles model-specific prompt formats, generation parameters, and image input for the multimodal task.

4. Transcript Processing and Answer Extraction — For extractive tasks, post-processes debater/consultant outputs by running a regex-based passage verification tool that checks whether extracted passages are exact substring matches in the source article, marking them as verified (<v_passage>) or unverified (<u_passage>). For all protocols, strips away the "Thinking" scratchpad sections so judges only see the "Argument" content, and extracts the judge's final answer (formatted as "Answer: <1|2>") for accuracy scoring.

5. Analysis and Elo Calculation — Computes judge accuracy (fraction of questions where the judge selects the ground-truth correct answer), generates confidence intervals via bootstrap or normal-approximation methods, runs permutation tests for statistical significance of protocol differences, and calculates Elo ratings for debaters using cross-play win-rate matrices optimized via negative log-likelihood.

Information flows sequentially: a question from a preprocessed dataset enters the pipeline → the protocol implementation selects the appropriate interaction pattern (e.g., 3 rounds of simultaneous debate with best-of-4 debater sampling) → the model interface generates responses from the specified judge and agent models using the protocol-specific prompt templates → for extractive tasks, transcripts pass through the passage verification tool → the judge's final answer is extracted and compared to ground truth → the accuracy result is aggregated with others from the same condition into the analysis component, which produces the figures and statistical tests reported in the paper.

3.3 Roadmap for the deep dive

• First, how tasks are preprocessed into binary-choice formats and categorized into the three task types (extractive, closed, multimodal), since the nature of the information available to debaters versus judges drives many of the key results.
• Second, the six protocols in detail — QA baselines, assigned-role consultancy and debate, and the novel open-role variants — including how debaters/consultants are prompted to structure their output, how judges interact with the transcript, and the verification mechanism for extractive tasks.
• Third, the models used and the rationale for choosing them as judge/debater pairs, including the capability gradient (Gemma7B through Gemini Pro 1.5) that creates the controlled weak-strong asymmetries central to the paper's contribution.
• Fourth, the implementation details of best-of-N sampling for debaters, the simultaneous versus sequential turn structure in debate, and the passage verification tool for extractive tasks — all of which are concrete mechanisms that affect the information available to the judge.
• Fifth, the open-role protocol construction — how open consultancy and open debate are derived from the assigned-role data by filtering for cases where the agent's chosen answer matches its direct QA answer — and why this matters for measuring error amplification.
• Sixth, the Elo calculation methodology used in Section 4.3 to measure debater persuasiveness and how it connects to the scalable oversight goal of improving judge accuracy as AI capabilities increase.

3.4 Detailed, sentence-based technical breakdown

This is primarily an empirical evaluation paper whose core idea is that scalable oversight protocols can be systematically compared by measuring judge accuracy under controlled variations of task type, capability gap, and answer assignment mechanism, using LLMs as both agents and judges to enable large-scale, statistically rigorous experimentation that would be prohibitively expensive with human participants. The paper's technical contribution is not a new algorithm but rather the design and execution of a comprehensive evaluation framework that addresses specific gaps left by prior work — particularly the lack of multi-task evaluation, the absence of open-role protocols, and the narrow focus on information asymmetry as the only weak-strong gap.

Task Preprocessing: From Multi-Class Datasets to Binary-Choice QA

The paper constructs its evaluation tasks by converting existing question-answering datasets into a unified binary-choice format. The conversion process is straightforward but contains several design decisions that affect the experimental interpretation.

General binary conversion. For datasets that are originally multiple-choice (e.g., MMLU has 4 options, GPQA has 4 options), the paper selects the correct answer and one random incorrect answer to form a binary choice. This is stated in Section 3.1: "in general, where an underlying dataset is multiple choice, we form a binary choice version by selecting the correct answer and a random answer." This design choice means that the incorrect option is not necessarily a distractor that is confusable with the correct answer — it is randomly selected, so on some questions the two options are easily distinguishable (one clearly right, one clearly wrong), while on others they may be subtle. This creates a natural distribution of question difficulties without requiring the paper to engineer difficulty levels explicitly.

Why binary? The paper never explicitly justifies the choice of binary over multi-class, but the rationale is implicit in the debate and consultancy protocol design. Debate requires two opposing positions — one debater argues for answer A, the other for answer B. Binary answers map cleanly onto this adversarial structure. Consultancy similarly assigns the consultant to argue for one of two sides. Multi-class debate would require more complex tournament structures or multi-agent designs, and the binary simplification keeps the protocol complexity manageable while still allowing meaningful measurement of judge accuracy (which would be 50% random chance, providing a clear baseline).

Task type categorization (Table 1, Section 3.1). The paper groups the nine tasks into three mutually exclusive categories based on what information is available to debaters/consultants versus judges:

1. Extractive QA tasks (QuALITY, BoolQ, GPQA-extractive): These tasks include a source article that contains the information needed to answer the question. Crucially, debaters and consultants have access to this article, but the judge does not. This creates an information asymmetry — the strong agent has strictly more information than the weak judge. The judge must rely on the agent to extract, quote, and explain relevant passages from the article. The paper describes this as "modeling a specific kind of weak-strong gap" (Section 3.1). For GPQA-extractive specifically, the article is not a natural text but a "detailed, expert-provided explanation of the correct answer" taken from the GPQA dataset's explanations (Appendix F.1), which the debaters must draw from to support their arguments.

2. Closed QA tasks (MMLU, GSM8KQA, PrOntoQA, TruthfulQA, GPQA): These tasks consist of a question and two answer options with no source article. Both the debaters/consultants and the judge have access to the same information (the question text). The capability gap being modeled here is not about information access but about intrinsic reasoning and knowledge differences — the stronger model may have more factual knowledge, better mathematical reasoning, or more reliable logical inference. As the paper states, "the weak-strong gaps being motivated here are more intrinsic to the models — their capability and skill/reasoning gaps" (Section 3.1).

3. Multimodal tasks (MMMU): These tasks include images alongside text in the question, and the judge and debaters/consultants can both view the image. The paper describes the motivation as "to understand the effect of an additional modality" (Section 3.1) on scalable oversight protocols. Like closed tasks, there is no information asymmetry — both parties see the same content.

Dataset-specific preprocessing details (Appendix F). Each task receives additional processing beyond binary conversion:

• QuALITY: The paper uses the split from Khan et al. (2024) — specifically a "TL split of 400 binary train set questions, with the same filtering (roughly to Gutenberg sci-fi subset of QuALITY)" (Appendix F.1). The test set consists of 128 questions sampled from this filtered set for the main experiments.

• BoolQ: No additional preprocessing is needed because BoolQ is already binary (yes/no questions about Wikipedia articles). The paper recommends this task for faster iteration because "documents [are] shorter" and it "show[s] many of the same overall patterns under our protocols" while noting that "LLMs likely have much more relevant knowledge about the factual questions in BoolQ than about the questions in QuALITY, making the text extraction less crucial" (Appendix F.1).

• GPQA-extractive: The "article" is the expert-provided explanation of the correct answer from the GPQA dataset. The paper acknowledges this "makes GPQA much simpler" but argues it is "still a sufficiently challenging task for current models" and "allows us to make a direct comparison between a closed QA and an extractive QA task" (Appendix F.1).

• MMLU: All categories are included except "business ethics" because it was found by Khan et al. (2024) to lack "clearly correct vs. incorrect answers" (Appendix F.2).

• GSM8KQA: This is not a standard dataset — the paper constructs it specifically for this study. They use a few-shot prompt from Gao et al. (2023) with Gemini 1.0 Pro to generate 32 Python code solution proposals for each GSM8K problem. They execute each proposal and select the correct answer as one that executes to the same answer as the ground truth, and the incorrect answer as one that executes to a different answer. This creates a binary QA version of GSM8K where the answer choices are code snippets rather than numeric values — a deliberate design to make the task about mathematical reasoning and coding capability simultaneously.

• PrOntoQA: Binary questions are constructed by taking the correct reasoning trace provided in the dataset and creating an incorrect version by "replacing boolean statements by their negation (i.e., replacing 'is' with 'is not' and vice versa)" (Appendix F.2). For example, if the correct trace says "Fae is a cat, cats are not herbivorous, therefore Fae is not herbivorous," the incorrect version might negate a premise to produce a wrong conclusion. This tests logical reasoning specifically.

• TruthfulQA: Used as-is since the questions are designed to be binary (true/false statements about common misconceptions).

• GPQA (closed version): The full GPQA dataset is used, with multiple-choice questions converted to binary choices as described above. Unlike the extractive version, there is no expert explanation provided — debaters must rely on their own knowledge.

• MMMU: The paper selects a subset of 2035 MMMU questions that come with a golden explanation (not used in the main experiments but included to "allow for a more controlled comparison if we extend the benchmark in future to include an extractive version of MMMU," Appendix F.3).

Test set size and sampling. For the main experiments, the paper samples 128 questions from each of the nine tasks, totaling 1152 questions per experimental condition. This is stated in Section 1: "sweeping over 9 tasks, each sampling 128 questions." The relatively small per-task sample size (128) is a deliberate tradeoff — it enables the paper to cover nine diverse tasks while keeping total generation cost manageable (5 million calls), but it limits the statistical power for per-task comparisons and means the main conclusions are primarily drawn from aggregated results within task types (extractive, closed, multimodal) rather than at the individual task level.

Protocol Implementation: The Six Interaction Patterns

The paper implements six distinct protocols that define how information flows between LLM agents on each question. These are not just different prompts — they are different interaction structures with specific rules about who sees what, who speaks when, and how the judge arrives at a final answer.

QA without article (the "naive" or direct baseline). This is the simplest protocol: the judge model receives the question, the two answer choices (in randomized order), and is prompted to select one. On extractive tasks, the judge does not see the article, so this protocol measures the judge's ability to answer correctly without any assistance from a stronger model. For closed QA tasks, this is a standard zero-shot evaluation — the judge sees exactly what it would see in a typical QA benchmark. For extractive tasks, it represents the lower bound of what a judge could achieve with no information about the article content. This protocol is labeled "Naive" in Khan et al. (2024) and described in Section 3.2.1.

The prompt template (Appendix H.5) is minimal:

You are an expert tasked with finding the correct answer to a question.
Here is the question and the two answers (the choice order is random):
Question: {question}
Choices:
1: {answer_1}
2: {answer_2}
Choose the answer you think is most likely to be correct.
[Optionally: "First explain your reasoning..."]
Only state your answer. Format it as "Answer: <1|2>".

The judge is instructed that "the order of the answer choices is random so should not influence your answer" (Appendix H.3, common to all judge prompts). This addresses the known positional bias problem where LLMs prefer the first or last answer choice regardless of content (Wang et al., 2023b; Zheng et al., 2023).

QA with article (the "expert" baseline). Only used for extractive QA tasks, this protocol gives the judge the full article text alongside the question and answer choices. The prompt (Appendix H.6) includes the article inside <article> tags before the question. This represents the best possible performance if the judge could directly access all relevant information — effectively measuring the judge's reading comprehension ability on these tasks. The paper notes that this protocol "gives the judge the full information necessary to answer the question, in contrast to the other protocols which in the extractive QA tasks introduce an information asymmetry" (Section 3.2.1). QA with article serves as a ceiling for extractive tasks: if a scalable oversight protocol could perfectly transfer the article's information through debate or consultancy, it would match this baseline.

Consultancy (assigned-role). In consultancy, a single LLM (the consultant) is assigned an answer to argue for and interacts with the judge over multiple rounds. The mechanics are described in Section 3.2.2 and detailed in Appendix H.1 and H.3:

1. The consultant receives the question, both answer options, and — for extractive tasks — the full article. The consultant is explicitly told which answer to defend: "You have confirmed that you will argue that the answer to {question} is {answer_defending} and against {answer_opposing}."

2. The judge receives the question, the two answer choices, the consultancy transcript so far, and — for extractive tasks — is informed about the passage verification system but does not see the article. The judge's task is to ask the consultant probing questions and, after a fixed number of rounds, provide a final answer.

3. For each question, two separate consultancy evaluations are run: one where the consultant is assigned the correct answer and one where the consultant is assigned the incorrect answer. The paper states this is "designed to simulate a worst-case variant of reinforcement learning from human feedback (RLHF) in which human feedback can't distinguish correct/incorrect answers" (Section 1). The accuracy reported for consultancy in the main figures is the average across both assignments — since the assignment is random (50/50 correct/incorrect), the expected accuracy if the judge always agreed with the consultant would be 50%.

The consultant's output is structured into two sections: "Thinking:" (a chain-of-thought scratchpad that the judge never sees) and "Argument:" (the persuasive text shown to the judge). The paper notes that "only their argument is shown to the judge (the thinking is private)" (Section 3.2). This structure is designed to allow the consultant to plan its strategy privately while presenting a clean argument — mirroring how a real human consultant might internally reason before speaking.

For extractive tasks, the consultant is instructed to include direct passages from the article using <passage>...</passage> tags. After the consultant submits its argument, a regex-based substring check verifies whether each tagged passage appears exactly in the article. Verified passages are wrapped in <v_passage> tags; unverified passages (not found in the article) are wrapped in <u_passage> tags. The judge is told (in its prompt, Appendix H.3.1) that "verified quotes are direct string matches from the article and are enclosed in <v_passage> tags" and "unverified quotes are not direct string matches from the article, so they should not be trusted." This verification mechanism is crucial for extractive tasks because it gives the judge a reliable signal about what information from the article is genuine, preventing the consultant from fabricating evidence.

The default setting uses 3 rounds of interaction (consultant argues, judge asks question, consultant responds, judge asks another question, consultant responds, judge decides). Each consultant argument is limited to 150 words per turn. The judge's questions are unconstrained — the prompt tells the judge to "choose an impactful question to ask the consultant" with no template or structured format specified.

Debate (assigned-role). Debate involves two LLM debaters (named Alice and Bob) arguing for opposite answers while a judge observes and decides the winner. The mechanics are described in Section 3.2.2 and detailed in Appendix H.2 and H.4:

1. Alice is assigned to argue for the first answer choice (which may be correct or incorrect, depending on randomization); Bob is assigned to argue for the second answer choice. Both receive the question, both answer options, their assigned side, and — for extractive tasks — the full article.

2. Debates proceed for a default of 3 rounds. The paper distinguishes between two turn styles:

   • Simultaneous turns (the default): In each round, both debaters generate their arguments based only on the transcript from previous rounds. Alice always appears first in the transcript, but Bob does not see Alice's current-round argument before writing his own.
   • Sequential turns: Alice generates her argument first. Bob then generates his argument after seeing Alice's current-round argument in addition to the previous transcript. This gives Bob a potential advantage — he can directly rebut Alice's latest points while Alice cannot rebut Bob's latest argument in the final round.

3. In round 1, debaters construct opening arguments. The thinking instructions (private to each debater, not shown to the judge) tell them to "consider what extra context the judge needs to know," "plan an argument for your position," and for extractive tasks, "select the best quotes from the article to use in your argument."

4. In round 2, debaters critique their opponent's previous argument. The thinking instructions change to: "list out the biggest flaws in your opponent's argument," "critique your opponent's argument," and for extractive tasks, "select the best quotes from the article to disprove your opponent's argument."

5. In round 3, debaters respond to their opponent's critiques. The thinking instructions become: "list the critiques the opponent has made to your argument," "construct the best counterargument to your opponent's critique," and for extractive tasks, "select the best quotes from the article to disprove your opponent's critique."

6. After all rounds complete, the judge receives the full debate transcript (with debater "Thinking:" sections removed) and is prompted to select which answer is correct. Unlike consultancy, the judge does not ask questions during the debate — the debaters argue with each other, and the judge passively observes. The paper explicitly justifies this choice: "we choose not to evaluate interactive debate, where the judge asks questions each round as Khan et al. (2024) found no improvement over standard debate with a human judge (their Fig. 1) or LLMs (Figs. 8 and 9)" (Section 3.2.2).

Like consultancy, debaters are limited to 150 words per argument, and for extractive tasks, they use the <passage> tagging system with the same verification mechanism. The judge sees verified and unverified passages marked in the transcript after verification.

Open consultancy. This protocol, described in Section 3.2.3, modifies assigned-role consultancy to let the consultant choose which answer to argue for rather than being randomly assigned. The implementation is elegant: rather than generating new consultancy interactions, the paper filters existing assigned-role consultancy evaluations by selecting only those where the consultant's assigned answer matches the answer that the same model would choose under the direct QA baseline. Specifically, for each question, the paper determines what answer the consultant model (Gemini Pro 1.5) selects under QA with article (for extractive tasks) or QA without article (for closed tasks). This is the model's "preferred" answer. Open consultancy then includes only the consultancy evaluations where the consultant happened to be assigned to argue for that preferred answer — "we filter our consultancy evaluations by selecting only those in which the assignment of the consultant matches the answer chosen by the consultant model under the format of direct QA" (Section 3.2.3).

This is a clever reuse of existing data, but it means the set of questions in open consultancy is a subset of those in assigned-role consultancy (specifically, those where the assignment randomly matched the model's preference). For extractive tasks, the consultant chooses the correct answer for 88% of questions; for closed tasks, 84%; for multimodal, 71% (Figure 3 caption). This means open consultancy overrepresents questions where the strong model gets the answer right under direct QA — an important statistical bias that affects how open consultancy results should be interpreted.

Open debate. This protocol, also described in Section 3.2.3, works similarly to open consultancy but applied to debate. One debater is designated the protagonist and chooses which answer to argue for (based on what the same model would answer under direct QA). The other debater (the antagonist) is automatically assigned the opposite answer. The implementation reuses existing debate evaluation data in the same filtering manner: "we reuse existing debate evaluation data and identify the protagonist as the debater whose assigned answer matches the debater model's answer under direct QA" (Section 3.2.3). Alice and Bob are both instances of Gemini Pro 1.5, so the protagonist designation depends on which debater's assigned side happens to match what that model would choose. The antagonist is then whichever debater got assigned the opposite answer.

The paper notes that this setup is "a more realistic representation of how such a framework might be employed in AI alignment training" (Section 3.2.3) because, in a real training setting, models would choose their positions based on their beliefs rather than being randomly assigned.

Protocol differences from Khan et al. (2024). The paper states that "whenever possible, we base our setups and prompts on Khan et al. (2024), with minor differences described in Appendix H" (Section 3.2). These differences include: changing "quote" to "passage" (the authors found that "models were only finding quotes of characters rather than extracting from the article"), changing "story" to "article" (since "our datasets don't all involve stories"), avoiding system/assistant role prompts for simplicity, and adding a 150-word limit per turn "to potentially limit verbosity bias" (Appendix H).

Model Selection and Capability Gap Construction

The paper uses a specific set of models chosen to create controlled capability gaps between judge and debater/consultant. The models and their roles are described in Section 3.3:

Judge models (four tiers of capability):

• Gemma7B (V1): An open-source 7-billion parameter model (instruction-tuned, text-only). This is the weakest judge and is primarily used to study whether debate provides any useful signal when the judge is very limited. Gemma7B is described as "likely too weak to learn anything that valuable about these protocols" (Section 4.1, interpretation of closed task results).
• GPT-3.5: Specifically gpt-3.5-turbo-0125. A mid-tier commercial model that has been used as a judge in prior work (Khan et al., 2024), enabling some comparison to existing results.
• Gemini Pro 1.0: A mid-tier Google model (text-only version used). Roughly comparable to GPT-3.5 in capability according to the paper's results, where "GPT-3.5 and Pro 1.0, are roughly similar" in judging ability (Section 4.1).
• Gemini Pro 1.5: The strongest model available to the authors, with image+text-to-text capabilities (required for the MMMU multimodal task). When used as a judge with Gemini Pro 1.5 debaters, this represents the same-model case where the judge is not weaker than the debaters, which the authors note "doesn't inform much about scalable oversight" (Section 4.1) but provides a useful upper bound on what's achievable.

Debater/consultant models (one primary, several for Section 4.3):

• Main experiments (Sections 4.1, 4.2): Only Gemini Pro 1.5 is used as debaters and consultants. This means the capability gap is determined entirely by the judge model — from Gemma7B (large gap) to Pro 1.5 (no gap).
• Persuasiveness analysis (Section 4.3): Five debater variants are compared: Gemma7B, GPT-3.5, Gemini Pro 1.0, Gemini Pro 1.5 (all with best-of-N=1), and Gemini Pro 1.5 with best-of-N=4. This allows studying how debater strength (as measured by Elo) correlates with judge accuracy.

Why Gemini Pro 1.5 as the primary debater? The paper aims to study "strong consultants/debaters overseen by weaker judges" (Section 3.3). Using the strongest available model maximizes the weak-strong asymmetry when paired with weaker judges and tests whether debate's theoretical advantages manifest when the debaters are genuinely more capable than the judge. The paper also needs a model with multimodal capabilities for MMMU, and Gemini Pro 1.5 is the only multimodal model used.

Sampling parameters. The paper states that "models are used 1-shot, with default sampling options unless otherwise specified" (Section 3.3). The "1-shot" refers to the fact that the prompt includes one example of the expected output format (the "Structure your output as follows" section in the consultant and debater prompts), though this is more accurately described as a formatting instruction rather than a traditional few-shot example. Default sampling likely means temperature=1.0 (creative but not degenerate) and top-p sampling, though exact values are not specified. For best-of-N, 4 independent samples are generated from the debater model for each turn in the debate.

Best-of-N Sampling for Debaters

For the main debate results, debaters use best-of-4 (BoN=4) sampling, described in Section 4.1: "4 samples are independently generated, and Pro 1.5 is prompted to select the most persuasive one." This differs from the implementation in Khan et al. (2024) in several ways that the paper explicitly discusses:

1. Scoring model: Khan et al. (2024) used log-probability comparisons against a dummy argument to select the best sample. This paper uses the debater model itself (Gemini Pro 1.5) as the scorer, prompted to "choose the most persuasive response from the following list" from the N candidates (Appendix H.7). The authors justify this choice: "we thought it was more principled not to choose compared to a dummy argument, but to compare between the arguments themselves" and "we used the debater model to compare the N samples, rather than the judge model as we suspected this would be the strongest judge" (Section 4.1 footnote).

2. Difference from standard best-of-N: The paper notes that this implementation "differs from standard best-of-N which would typically use a trained reward model for scoring" (Section 4.1 footnote). Instead, it uses the debater model itself in an evaluation role, which is more analogous to self-consistency or self-critique approaches than to reward-model-guided selection.

3. Cost: Best-of-4 multiplies the number of debater generations by 4 per turn (each of the 3 rounds generates 4 candidate arguments, one of which is selected). For a 3-round debate with two debaters, this means 2 debaters × 3 rounds × 4 samples = 24 debater generations per question (plus the scoring calls). This is still within the paper's 5-million-call budget but represents a significant per-question cost multiplier.

The ablation in Figure A.3 shows that reducing BoN from 4 to 1 (standard single-sample generation) does not significantly affect judge accuracy, leading the authors to conclude that "our implementation of best-of-N is not helpful" (Section 4.1, Ablation: Best-of-N for debaters). They note that "Elo increases with BoN=4, but doesn't significantly increase accuracy" (Section 4.3), suggesting that while best-of-4 does make debater arguments more persuasive (higher Elo), this increased persuasiveness does not translate into better judge decisions — possibly because the improvement affects both correct and incorrect debaters similarly, or because the judge is already at a ceiling where additional persuasiveness doesn't change its decisions.

Passage Verification Tool for Extractive Tasks

For extractive QA tasks (QuALITY, BoolQ, GPQA-extractive), debaters and consultants are instructed to support their arguments with direct passages from the article, wrapped in <passage>...</passage> tags. After each debater/consultant turn, a passage verification tool runs, described in Section 3.2 and detailed in Appendix G.

The tool performs a simple regex-based substring check: it searches for each tagged passage as an exact substring in the source article. If the passage is found (exact match), it is marked as verified and the tags are changed to <v_passage>...</v_passage>. If the passage is not found (no exact match in the article), it is marked as unverified and the tags are changed to <u_passage>...</u_passage>.

This is not a semantic verification — it does not check whether the passage actually supports the debater's claim, only that the text appears verbatim in the article. A debater could extract a completely irrelevant sentence that happens to appear in the article, and it would be marked as verified. The paper relies on the adversarial dynamics of debate to ensure that such irrelevant quotes are called out by the opposing debater.

The judge's prompt (Appendix H.3.1) explains the tag semantics: "Verified quotes are direct string matches from the article and are enclosed in <v_passage></v_passage> tags" and "Unverified quotes are not direct string matches from the article, so they should not be trusted." The judge is also warned that "some quotes are just enclosed in quotation marks, which counts as unverified and should not be trusted."

An important design detail: the paper states that <passage> was used instead of <quote> (used in Khan et al., 2024) because "models were only finding quotes of characters rather than extracting from the article" (Appendix H). The word "quote" biased models toward finding dialogue rather than informational passages, especially in narrative texts like QuALITY's science fiction stories. Switching to "passage" produced more useful extractions.

Transcript preprocessing flow (Appendix I.1, Figures I.1 and I.2). After each turn, two transformations are applied to the debater/consultant output before it is added to the transcript:

1. The "Thinking:" section is stripped (so the judge and opponent only see the "Argument:" section).
2. For extractive tasks, <passage> tags are replaced with <v_passage> or <u_passage> based on the verification check.

Figures I.1 and I.2 show before-and-after examples of this preprocessing on a BoolQ debate transcript.

The paper notes a limitation of this verification mechanism: Khan et al. (2024) found that on QuALITY, "the quote verification, rather than the argumentation, is the dominant contribution to judge accuracy (using quotes alone improves beyond the combination of arguments and quotes)" (Appendix F.1). This suggests that on extractive tasks, the verifiability of evidence may matter more than the persuasive quality of the argumentation, and that "such a quote verification tool may be unrealistic considering the weak-strong analogy we expect to face for scalable oversight with humans" — real-world oversight of superhuman AI would not have access to a perfect truth-checker that verifies evidence.

Judge Interaction and Answer Extraction

The judge's interaction with the protocol transcript varies by protocol:

Consultancy: The judge sees the question, the two answer choices, and the consultancy transcript (consultant's arguments plus judge's own previous questions). The judge is prompted to ask a question each round (except the final round, where it gives a final answer). The prompt includes guidance: "Choose an impactful question to ask the consultant, and remember, do not trust quotes that are not verified. Your questions should be directed to the consultant in the second person ('Why do you claim that...')."

Debate: The judge sees the question, the two answer choices, and the complete debate transcript (both debaters' arguments across all rounds, with thinking stripped and passages verified). The judge gives a final answer without asking questions. The prompt includes: "Your main objective is to choose the correct answer based on the evidence presented."

Answer extraction and positional bias mitigation. The judge's answer must be formatted as "Answer: <1|2>" (for the Predict setting) or "[reasoning text]... Answer: <1|2>" (for the Chain-of-Thought setting). The paper uses string matching to extract the chosen answer from the judge's output. If the judge fails to produce a valid answer in this format, that judgment is excluded from accuracy calculations. Figure B.11 shows that for most judge models and protocols, the rate of invalid responses is very low (<2% for the stronger judges), though Gemma7B shows higher invalid response rates, particularly for debate on longer-context tasks (QuALITY, GPQA, GSM8KQA) where the "context being longer... caus[es] the judge to fail to correctly format its response (or make any sense at all)" (Figure B.4 caption).

To address positional bias, the paper randomizes the order of the two answer choices (correct answer appears as choice 1 half the time, choice 2 half the time) and includes the instruction "remember that the order of the answer choices is random so should not influence your answer" in all judge prompts. An ablation (Figures A.7 and A.8) compared this single-order randomization to the Khan et al. (2024) approach of running each question twice with the answer order swapped. The paper finds "no significant difference in mean judge accuracy" between the two approaches and that "using both orders does not change the mean answer position, only reduces its variance" (Figure A.8 caption), concluding that "it's sufficient to just randomise the answer position, rather than evaluating with both answer positions."

Open-Role Protocol Construction via Data Filtering

The open-role protocols (open consultancy and open debate) are constructed not by generating new interactions but by filtering existing assigned-role data based on the strong model's direct QA behavior. This is described in Section 3.2.3.

The construction logic works as follows:

1. For each question in the test set, determine what answer the strong model (Gemini Pro 1.5) would choose under direct QA. For extractive tasks, this uses the QA with article protocol (where the model sees the article). For closed and multimodal tasks, this uses QA without article (where the model sees only the question).

2. For open consultancy: from the pool of assigned-role consultancy evaluations (each question was run twice — once with the consultant assigned correct, once with the consultant assigned incorrect), select only the evaluation where the consultant's assigned answer matches the model's direct QA choice. If the model's direct QA answer is incorrect, the selected consultancy evaluation will be the one where the consultant was assigned the incorrect answer. If the model's direct QA answer is correct, the selected evaluation will be the one where the consultant was assigned the correct answer.

3. For open debate: from the pool of assigned-role debate evaluations (each question was run once, with Alice arguing for one answer and Bob for the other), designate as protagonist the debater whose assigned answer matches the model's direct QA choice. The other debater becomes the antagonist. The judge's decision is then analyzed in terms of whether it agrees with the protagonist (win-rate) and whether the protagonist's chosen answer is correct (accuracy).

Statistical implications of this construction. Because the strong model gets the answer correct on 71-88% of questions (Figure 3 caption), the open-role protocols oversample easy questions where the model is correct and undersample hard questions where the model is wrong. This means that open consultancy and open debate accuracy numbers are not directly comparable to assigned-role accuracy numbers without accounting for the shifted question distribution. The paper acknowledges this implicitly in Appendix A, Figure A.9, which compares open consultancy to assigned-role consultancy and notes that "when consultants argue for correct answers, the accuracy of open and assigned-role consultancy are similar, whilst when arguing for incorrect answers, open consultancy leads to lower accuracy" — attributing this to "questions which the open consultant answers incorrectly are more difficult than average, while the assigned-role consultant's assignments are independent of difficulty."

Elo Rating Calculation for Debater Persuasiveness

Section 4.3 and Appendix E describe a method for measuring the relative persuasiveness of different debater models, adapted from Khan et al. (2024). The goal is to quantify whether stronger debaters (as measured by their ability to win debates) lead to higher judge accuracy — a key desideratum for scalable oversight.

Data generation. A set of 1-turn arguments in simultaneous debate (opening arguments only, no rebuttals) is generated using five debater variants: Gemma7B, GPT-3.5, Gemini Pro 1.0, Gemini Pro 1.5 (all with best-of-N=1), and Gemini Pro 1.5 with best-of-N=4. This is deliberately simplified from the main 3-round debates to make the Elo calculation computationally feasible — generating full 3-round debates for all pairwise matchups would be too expensive.

Cross-play pairings. For each question, two debaters are selected and assigned opposing sides (correct vs. incorrect answer). The paper samples 512 pairings "with each dataset contributing an equal number" of questions (Appendix E), meaning approximately 57 questions per dataset across 9 datasets, with pairings sampled such that each debater faces each other debater across many questions. Both debaters submit opening arguments, and a judge (Gemini Pro 1.5 for the main analysis, also Gemini Pro 1.0 for a robustness check in Figure E.2) selects the winner.

Win-rate matrix. From these matchups, a win-rate matrix $\omega_{i,j}$ is constructed for $i, j$ over the five debaters, where $\omega_{i,j}$ is the observed frequency (empirical probability) that debater $i$ beats debater $j$ across all questions where they were paired.

Expected win-rate model. The expected probability that debater $i$ with Elo rating $E_i$ beats debater $j$ with Elo rating $E_j$ is modeled as:

$\hat{\omega}_i = \frac{1}{1 + 10^{(E_j - E_i)/500}}$

where $E_i$ is the Elo rating of debater $i$, $E_j$ is the Elo rating of debater $j$, and the denominator 500 is the standard Elo scale parameter that controls the sensitivity of win probabilities to rating differences.

What it computes: given two Elo ratings, this logistic function outputs the predicted probability that the higher-rated player wins. If $E_i = E_j$, the expected win rate is 0.5 (even match). A 500-point rating advantage gives approximately a 10:1 win probability. The output is a scalar between 0 and 1 representing expected win frequency.

Why this form: the logistic function is standard in Elo systems because it (a) produces well-calibrated probabilistic predictions for paired comparison data, (b) has an information-theoretic justification as the maximum-entropy distribution given the rating difference, and (c) makes the optimization problem convex, ensuring a unique solution. The 500-point scale is conventional (from chess) but arbitrary; the paper follows the convention from Khan et al. (2024).

Elo optimization. The Elo ratings are found by minimizing the negative log-likelihood of the observed win-rates under the logistic model:

$\text{Objective} = -\sum_{i,j} \omega_{i,j} \log(\hat{\omega}_i)$

where $\omega_{i,j}$ is the observed win-rate of debater $i$ against debater $j$, and $\hat{\omega}_i$ is the expected win probability computed from the current Elo estimates via the logistic formula above.

What it computes: for each pair of debaters, the model's predicted probability that $i$ beats $j$ is compared to the actual observed frequency. The negative log-likelihood penalizes predictions that assign low probability to events that actually occurred. Summing over all pairs gives a single scalar that measures how well the Elo ratings explain the observed win-rates. Lower values indicate better fit. The optimization uses the BFGS algorithm to find the Elo ratings $E_i$ that minimize this objective.

Why this form: negative log-likelihood is the standard statistical objective for probability models because it corresponds to maximum likelihood estimation — finding the parameters that make the observed data most probable. The paper notes they chose this over the squared error used in Khan et al. (2024) because "we found it handled low numbers of games better" (Appendix E footnote). Squared error would penalize large deviations symmetrically, which is inappropriate for probability data where predictions near 0 or 1 should be penalized more heavily than predictions near 0.5.

Correct-Elo and incorrect-Elo. To measure whether truth provides an advantage, each debater is split into two distinct "players" in the Elo calculation: one that is assigned the correct answer ("correct-Elo") and one that is assigned the incorrect answer ("incorrect-Elo"). The same optimization procedure is run with ten players (5 debaters × 2 assignment types), and the difference between correct-Elo and incorrect-Elo for a given debater measures the truth advantage — how much easier it is to argue for the correct answer than the incorrect one, holding debater capability constant.

Confidence intervals. Statistical bootstrapping with 500 seeds is used to estimate 95% confidence intervals for the Elo scores. Each bootstrap resamples the cross-play matchups with replacement and recomputes the Elo ratings, providing an empirical distribution of plausible rating values.

Connection to scalable oversight. If debate fulfills the promise of scalable oversight, then stronger debaters (higher Elo) should lead to more accurate judge decisions. The relationship between aggregate Elo and judge accuracy (Figure 4, bottom row) tests this hypothesis: each debater model is plotted with its aggregate Elo on the x-axis and the judge accuracy when that debater model is used on the y-axis. A positive correlation would be evidence that debate scaling trends are favorable.

Summary of Design Choices and Justifications

• Binary-choice format for all tasks: Maps cleanly onto the adversarial two-sided structure of debate; provides a clear 50% random baseline; and standardizes evaluation across diverse datasets.

• Gemini Pro 1.5 as the primary debater/consultant: Maximizes the weak-strong asymmetry by using the strongest available model; enables multimodal evaluation on MMMU.

• Four-judge capability gradient (Gemma7B → GPT-3.5 → Pro 1.0 → Pro 1.5): Creates controlled variation in the weak-strong gap; the paper needs to characterize how judge accuracy changes as the gap widens, which is the core scalable oversight question.

• Three-round default debate with simultaneous turns: Prior work (Khan et al., 2024) found no improvement beyond 1 round and no benefit to interactive questioning; three rounds is a compromise that allows for argument development and rebuttal without excessive cost.

• Passage verification via exact substring match (not semantic): Simple, deterministic, and reliable — eliminates fabrication of evidence on extractive tasks; provides a clean signal to judges about what information is trustworthy.

• Best-of-4 with debater as scorer (not reward model): Avoids needing a trained reward model; uses the strongest available judge (the debater itself) to select among its own candidates; computationally feasible within the 5-million-call budget.

• Open-role construction via data filtering (not new generation): Reuses existing data efficiently; ensures that open-role and assigned-role results are from the same model outputs, making comparisons cleaner at the cost of a shifted question distribution.

• Elo ratings optimized via negative log-likelihood (not squared error): Handles sparse paired comparison data more robustly; provides well-calibrated probabilistic predictions; follows standard practice in competitive rating systems.

• 150-word limit on debater/consultant arguments: Controls for verbosity bias (longer arguments might be more persuasive regardless of content); keeps transcripts manageable for weaker judges with limited context windows.

4. Key Insights and Innovations

Innovation 1: Debate's Advantage Over Consultancy Generalizes, But Its Advantage Over Direct QA Is Task-Dependent — A Finer-Grained Diagnostic Than Prior Work

The paper's most structurally important empirical contribution is not that "debate works" but rather a nuanced mapping of where debate provides leverage and where it does not. Prior work had established that debate outperforms consultancy on a single extractive QA task (QuALITY, in Khan et al., 2024), and had produced preliminary evidence that debate doesn't improve over direct QA when judge and debater are the same model on non-extractive tasks. But the field lacked a systematic characterization across task types, capability gaps, and baselines — leaving fundamental questions open: is debate's advantage specific to information asymmetry, or does it persist when the weak-strong gap is about reasoning and knowledge? Does debate help more when the judge is weaker, or does it require a certain minimum judge capability to be useful?

This paper provides that characterization, and the result is more textured than a simple "debate helps" or "debate doesn't help." The finding that debate consistently outperforms consultancy across all task types (Finding 1, Section 1; Figure 2) is a genuine generalization: the adversarial structure provides a benefit over single-agent argumentation regardless of whether the gap is informational or reasoning-based. This was not obvious ex ante — one could have imagined that consultancy's interactive questioning would be sufficient on closed tasks where the judge has access to the same information as the consultant, and that debate's additional adversarial pressure only matters when debaters are filtering information the judge cannot verify directly. The result shows otherwise: having an equally capable opponent who can expose flaws is beneficial even when the judge could theoretically answer the question themselves.

But the paper's more important diagnostic move is the disaggregation of debate vs. direct QA by task type (Finding 2, Section 1; Figure 2). On extractive tasks with information asymmetry, debate outperforms QA without article (the judge answering without seeing the source text) — replicating and generalizing the Khan et al. (2024) QuALITY result to BoolQ and GPQA-extractive. This makes intuitive sense: the judge lacks the information to answer correctly alone, so any protocol that transfers information from the article through debate is better than no information at all. But debate does not surpass QA with article — the protocol where the judge simply reads the article directly. This is a sobering result for debate advocates: even with two strong debaters arguing and rebutting, the information transfer through debate is less effective than giving the judge the raw text. The paper frames this as "information is best used directly, even by a weak model, rather than filtered through oversight" (Section 4.1). If the long-run scalable oversight scenario involves information the human cannot directly process (e.g., superhuman reasoning not expressible as text), this result doesn't directly translate, but it does suggest that debate's information-transfer efficiency is imperfect in ways that matter.

On closed QA tasks — which the paper introduces as a more realistic model of the reasoning and knowledge gaps between superhuman AI and human judges — the results are mixed (Figure 2, center panel). For Gemma7B, the weakest judge, QA without article outperforms debate: the judge is simply too limited to extract useful signal from the debate transcript. For GPT-3.5, debate provides a statistically significant improvement over QA without article. For Gemini Pro 1.0, the difference is not statistically significant. For Gemini Pro 1.5 (judge same as debaters), debate worsens accuracy — replicating the Khan et al. (2024) finding that same-model debate can degrade performance. On multimodal tasks, debate and QA without article are not significantly different.

This pattern is not random noise — it is conceptually revealing. It suggests that debate is most useful when the judge is in a sweet spot: capable enough to understand the arguments and recognize when one is stronger, but not so capable that they could answer correctly alone. If the judge is too weak (Gemma7B), the debate format is too complex and the judge cannot extract useful signal — they do worse than just guessing based on their own knowledge. If the judge is as strong as the debaters (Gemini Pro 1.5), the debate format introduces noise and potential for the dishonest debater to confuse the judge, degrading performance below what the judge could achieve independently. The scalable oversight-relevant regime — where the judge is weaker but not drastically weaker — shows at best a small benefit for debate on closed tasks, and on multimodal tasks shows no significant difference.

What makes this a genuine innovation rather than just an observation is the reframing it enables. Before this paper, the field's implicit assumption (shaped by the theoretical guarantees of Irving et al., 2018 and the positive QuALITY results) was that debate's advantages should generalize. This paper replaces that assumption with a conditional model: debate helps when there is information asymmetry the judge cannot resolve alone, but when the gap is about reasoning and knowledge — the kind of gap most relevant to supervising superhuman AI — debate's advantage is fragile, judge-strength-dependent, and often negligible. This is not a negative result that kills debate as a research direction, but it sharpens the evidentiary burden: future work on debate must demonstrate improvements specifically on tasks without information asymmetry and specifically in the regime where the judge is weaker than the debaters, or it hasn't shown anything new. The paper thus functions as a diagnostic tool for the field — telling us what kind of evidence would constitute progress — not just a performance report.

Innovation 2: Open Debate Substantially Reduces Error Amplification Compared to Open Consultancy — A New Diagnostic for Evaluating Scalable Oversight Protocols Under Realistic Agent Beliefs

This is the paper's most conceptually novel contribution and the one most likely to influence how scalable oversight protocols are evaluated in the future. The paper introduces open-role protocols (open debate and open consultancy) and uses them to measure something distinct from judge accuracy: error amplification — whether a weak judge's feedback reinforces the AI's own mistakes when the AI believes an incorrect answer.

The diagnostic move is elegant. In assigned-role consultancy and debate, the consultant/debater is randomly assigned to argue for the correct or incorrect answer with 50/50 probability. This measures whether a judge can tell the difference between good and bad arguments when both sides are presented. But in a real alignment training setting, the AI agent is not randomly assigned answers — it has its own beliefs about what is correct. If the AI believes a wrong answer, and the scalable oversight protocol cannot detect this, the training signal will reinforce the error. The critical question for scalable oversight training is therefore not "can the judge distinguish assigned-correct from assigned-incorrect arguments?" but rather "when the AI chooses to argue for an incorrect answer, does the judge notice?"

Prior work (Khan et al., 2024; Michael et al., 2023; Radhakrishnan, 2023) had not studied this question systematically. The open-role protocols are the paper's mechanism for measuring it, and the results reveal a stark asymmetry between consultancy and debate (Figure 3, bottom row; Finding 4 and 5, Section 1).

In open consultancy, when the consultant chooses to argue for the correct answer (which happens 71-88% of the time, depending on task type), the judge scores high accuracy. When the consultant chooses to argue for the incorrect answer, the judge scores very low accuracy. The judge essentially trusts the consultant regardless of correctness — accuracy collapses when the consultant is wrong. This means that if consultancy were used as a training signal, incorrect consultant beliefs would be reinforced by the judge's feedback just as strongly as correct beliefs. The protocol amplifies mistakes.

In open debate, the pattern is qualitatively different. When the protagonist debater chooses the correct answer, the judge's accuracy is lower than in open consultancy — the adversarial structure means the antagonist can sometimes confuse the judge even when the protagonist is right. But when the protagonist chooses the incorrect answer, the judge's accuracy is substantially higher than in open consultancy when the consultant is wrong. The weak judge in open debate is much better at detecting that the protagonist's chosen answer is wrong — because the antagonist is there to point out the flaws.

This tradeoff — slightly worse when the AI is right, substantially better when the AI is wrong — is precisely what one would want from a scalable oversight protocol used for training. An ideal training signal would perfectly distinguish correct from incorrect outputs, but any practical protocol involves errors. The question is how those errors distribute. Consultancy's errors are asymmetric in a dangerous direction: they amplify the AI's existing mistakes because the judge cannot tell when the consultant has chosen wrongly. Debate's errors are more symmetric and less dangerous: the adversarial structure makes it harder for either side (correct or incorrect) to dominate, and the judge is substantially less likely to be fooled by an incorrect protagonist than by an incorrect consultant. In a training loop, this means debate is less likely to create positive feedback loops that reinforce and escalate the AI's initial errors.

This innovation is not about raw accuracy improvements — indeed, for the case where the agent is correct, open debate does worse than open consultancy. It is about introducing a new evaluation dimension for scalable oversight protocols: error amplification under realistic agent beliefs. The paper has shown that the protocol comparison depends critically on whether you care about average-case accuracy (where consultancy looks competitive) or worst-case error amplification (where debate has a structural advantage). This reframing pushes the field beyond simple accuracy comparisons toward an evaluation that more closely matches the safety motivation for scalable oversight — which is precisely about preventing the amplification of errors in the training signal.

A subtle but important detail: the paper also compares open consultancy to assigned-role consultancy (Figure A.9) and finds that when the consultant argues for the correct answer, both versions have similar accuracy, but when arguing for the incorrect answer, open consultancy produces lower accuracy than assigned-role consultancy. The authors hypothesize this is because "questions which the open consultant answers incorrectly are more difficult than average, while the assigned-role consultant's assignments are independent of difficulty" (Appendix A). This is a confound — the open-role protocols sample harder questions in the "incorrect" condition — but it does not weaken the core finding. If anything, it makes the open debate advantage more impressive: debate helps the judge detect incorrect protagonist arguments despite those questions being harder on average.

The conceptual contribution here is the shift from evaluating protocols by average accuracy to evaluating them by their error-amplification characteristics. This is a more safety-relevant metric, and the paper provides a concrete methodology (open-role protocols constructed via data filtering) for measuring it. The specific finding — that debate provides a less error-amplifying training signal than consultancy — is a novel empirical result with direct implications for which protocols should be prioritized in training-based evaluations.

Innovation 3: Scaling Trends for Debate Are Weaker and Less Consistent Than Prior Work Suggested — A Calibration of Expectations

The paper's third major contribution is a correction to the optimism generated by prior scaling results. Khan et al. (2024) had reported on QuALITY that stronger debaters (measured by Elo) lead to higher judge accuracy, and that this trend was "consistent." This was one of the most promising empirical results for debate — it suggested that debate might naturally scale with AI capabilities, fulfilling the core promise of scalable oversight. The current paper replicates the Elo-vs-accuracy analysis across nine tasks with multiple debater models and two different judges, and the result is a significant calibration: stronger debaters do lead to higher judge accuracy, but the effect is "relatively weak compared to Khan et al. (2024); Radhakrishnan (2023)" and the trend is "less consistent" (Section 5, Summary; Section 4.3).

This is more than a difference in degree — it changes the interpretation of what debate's scaling properties actually are. In Figure 4 (bottom row), on extractive tasks, there is a positive relationship between aggregate debater Elo and judge accuracy with Gemini Pro 1.5 as judge — stronger debaters produce higher accuracy. But on closed tasks, "there's less differentiation between debaters, both in terms of aggregate Elo and judge accuracy" (Section 4.3). The paper attributes this to "the judge's greater dependence on debater statements in the extractive tasks, where judges rely on debaters for information they cannot access themselves." On closed tasks, where the judge has their own knowledge to draw on, the debaters' persuasiveness has less impact — the judge can partially override the debate with their own reasoning.

Further, the paper introduces a new diagnostic: the correct answer advantage — the difference between a debater model's correct-Elo and incorrect-Elo, measuring how much easier it is to argue for the truth. Khan et al. (2024) found that this advantage increased with debater strength on QuALITY, suggesting that as models become more capable, the truth becomes easier to defend. The current paper finds this pattern on extractive tasks ("it appears it gets slightly easier for honest debaters as models increase their convincingness") but "on closed tasks there is less of a clear trend" (Section 4.3). In other words, the scaling trend that was a key source of optimism for debate — that stronger models naturally make truth more defensible — may be specific to information-asymmetric tasks and may not generalize to the reasoning-gap settings that are more relevant to scalable oversight of superhuman AI.

The paper also identifies a methodological confound that may have affected prior work: self-preference bias. The authors "notice overall that both Gemini-family judges tend to assign relatively lower Elo scores to GPT-3.5 than expected based on relative capabilities of the models" and "speculate this may be an artifact of self-preference bias (Panickssery et al., 2024) in which an LLM evaluator prefers its own generations compared to other models (though generalised such that a judge model prefers generations from their own model families rather than just from itself)" (Section 4.3). If true, this would mean that Elo scores computed with LLM judges are not purely measuring debater persuasiveness — they also measure family resemblance — and scaling trends based on such Elo scores may be partially artifactual. This is a methodological caution that extends beyond this paper to any work using LLM judges to evaluate other LLMs.

The contribution here is not a new method or capability, but a necessary calibration of the field's expectations. Before this paper, one could reasonably cite Khan et al. (2024) as evidence that debate's scaling properties look favorable. After this paper, the evidence base is more mixed: scaling trends exist, but they are weaker, less consistent across task types, and potentially confounded by judge self-preference. This doesn't mean debate won't scale — it means we have less evidence than we thought, and that future work claiming scaling benefits for debate needs to demonstrate them across multiple task types and with careful controls for judge-model family effects. The paper's framing — "a weakly positive indication for debate as a scalable oversight protocol — though one may have hoped to see a stronger effect on judge accuracy" (Section 4.3) — accurately captures the epistemic shift from "promising" to "not yet ruled out, but weaker than hoped."

Innovation 4: Ablations Reveal That Standard LLM Scaffolding Techniques (Chain-of-Thought, Few-Shot Prompting, Best-of-N, Multiple Rounds) Do Not Improve Debate Judge Accuracy — A Surprising Negative Result That Constrains Future Work

The paper's extensive ablation study (Section 4.1, Appendices A and B) systematically tests whether common techniques for improving LLM performance transfer to the debate judging task, and the consistent finding is they do not. This is a negative result with practical and conceptual significance.

Chain-of-thought reasoning for judges does not help — and sometimes hurts. The paper compares the default "Predict" setting (judge directly states their answer) to a "Chain-of-Thought" setting where the judge is prompted to "first explain your reasoning, considering the strengths and weaknesses of the consultant's/debater's arguments. Second, give your answer" (Appendices H.3, H.4). Figure A.5 shows that CoT "tends to harm performance or be insignificant." The authors hypothesize that "having weaker judges use their own CoT causes them to disregard the verbalised reasoning offered by stronger consultants/debaters, which might be higher quality" (Section 4.1, Ablation: Judge Chain-of-thought). This is a subtle but important point: if the whole purpose of scalable oversight is to leverage the stronger model's reasoning to assist a weaker judge, then prompting the weaker judge to generate their own reasoning may cause them to ignore the stronger model's output and default to their own (inferior) judgment. This suggests that the standard "let the model think step by step" advice — which generally improves LLM performance on reasoning tasks — is counterproductive in the specific context of evaluating arguments from a stronger model.

Few-shot prompting for judges does not help. The paper compares 0-shot (the default) to 6-shot prompting, "balancing the order of the correct answer across examples" (Section 4.1, Ablation: Judge Few-shot prompting). Figure A.4 shows "the results do not show significant improvement with 6-shots." For Gemma7B specifically, 6-shot prompting on debate "fails to answer for QuALITY, GPQA Extractive, and GSM8KQA" because "the context being longer in these settings, caus[es] the judge to fail to correctly format its response" (Figure B.4 caption). This negative result suggests that the debate judging task is not simply an instance of general LLM capability that improves with in-context examples — it may require task-specific fine-tuning to be tractable.

Best-of-N sampling for debaters does not improve judge accuracy. The paper's implementation of best-of-4 (generate 4 candidate arguments, have the debater model select the most persuasive one) does not produce higher judge accuracy than standard single-sample generation (Figure A.3). The authors note that BoN=4 does increase Elo scores (debater arguments are rated as more persuasive in head-to-head matchups), but this increased persuasiveness does not translate into better judge decisions — "suggesting our implementation of best-of-N is not helpful" (Section 4.1). The mechanism is unclear: perhaps BoN makes both correct and incorrect arguments more persuasive in equal measure, or perhaps the judge is already at a performance ceiling where additional argument quality doesn't change decisions.

More rounds of debate/consultancy do not help. Three rounds (the default) performs similarly to one round (Figure A.2). This is consistent with Khan et al. (2024)'s finding on QuALITY that additional rounds don't improve judge accuracy, but the generalization across nine tasks strengthens the result. If debate's theoretical advantage comes from iterative refinement — debaters pointing out flaws in each other's arguments and the judge learning from this exchange — then the fact that longer interactions don't improve accuracy is concerning. It suggests that current LLM debaters are not effectively exploiting the multi-round structure: they may not be engaging in genuine back-and-forth rebuttal that exposes substantive flaws, or judges may not be updating their beliefs based on later rounds.

Debate turn style (simultaneous vs. sequential) does not matter. Figure A.6 shows no significant difference between simultaneous turns (where debaters write arguments based only on previous rounds) and sequential turns (where the second debater sees the first debater's current-round argument). This means the potential advantage of sequential turns — where Bob can directly rebut Alice's latest points — does not materialize.

Positional bias exists but dual-order evaluation doesn't fix it. Judges systematically prefer either the first or second answer position (Figure A.8), consistent with prior work on LLM evaluation bias. The paper's contribution here is showing that running each question twice with answer order swapped — which Khan et al. (2024) recommended and which doubles evaluation cost — "does not change the mean answer position, only reduces its variance" (Section 4.1, Ablation: Positional bias). The bias itself remains; the dual-order approach just averages it out without correcting it. The paper recommends that "it's sufficient to just randomise the answer position, rather than evaluating with both answer positions," which is a practical cost-saving recommendation for future work.

The conceptual significance of these ablation results goes beyond practical guidance. They collectively suggest that the debate judging task, as currently implemented with prompted LLMs, is not well-modeled as a standard reasoning or QA task that benefits from established LLM performance-improvement techniques. Chain-of-thought, few-shot examples, iterative refinement, and adversarial dynamics — all of which help LLMs on benchmarks — do not transfer to making judges better at evaluating debates. This implies that making debate work as a scalable oversight protocol may require more fundamental changes than prompt engineering: fine-tuning judges specifically for the debate evaluation task, training debaters to be more effective at exposing flaws, or redesigning the debate format itself to better match the cognitive strengths of the judge (whether LLM or human). The negative results set a boundary on what can be achieved with inference-only prompting and point toward training-based approaches as the necessary next step.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on 9 binary-choice QA tasks grouped into three categories: extractive QA (QuALITY, BoolQ, GPQA-extractive), closed QA (MMLU, GSM8KQA, PrOntoQA, TruthfulQA, GPQA), and multimodal (MMMU). For the main experiments, 128 questions are sampled from each task, totaling 1,152 questions per experimental condition. Extractives tasks provide a source article to debaters/consultants but not to the judge; closed tasks provide only the question and answers to both parties; MMMU includes images alongside text. QuALITY uses the TL split of 400 binary training questions from Khan et al. (2024) filtered to the Gutenberg sci-fi subset. GSM8KQA is constructed by generating Python code solution proposals and selecting one correct and one incorrect executable answer. PrOntoQA binary questions are formed by negating boolean statements in the reasoning trace.

• Base model(s). The primary debater/consultant model is Gemini Pro 1.5 (image+text-to-text), chosen as the strongest available model to maximize weak-strong asymmetries and enable multimodal evaluation. Four judge models create a capability gradient: Gemma7B (instruction-tuned, open-source, weakest), GPT-3.5 (gpt-3.5-turbo-0125, mid-tier), Gemini Pro 1.0 (text-only, mid-tier), and Gemini Pro 1.5 (strongest, same as debaters). For the persuasiveness analysis in Section 4.3, five debater variants are used: Gemma7B, GPT-3.5, Gemini Pro 1.0, Gemini Pro 1.5 (all with best-of-N=1), and Gemini Pro 1.5 with best-of-N=4. All models are used 1-shot with default sampling parameters.

• Metrics. The primary metric is judge accuracy: the fraction of questions for which the judge selects the ground-truth correct answer (50% random baseline). For open-role protocols, additional metrics include win-rate (fraction of questions where the judge agrees with the protagonist/consultant's chosen answer), judge accuracy conditional on protagonist/consultant correctness (Figure 3 bottom), and Elo ratings computed from cross-play debate win-rate matrices (Appendix E). For extractive tasks, passage verification accuracy is implicitly measured through the <v_passage> and <u_passage> tag system but not reported as a separate metric. Confidence intervals are 95% using normal approximation after averaging switched-answer-order replicates (Appendix D). Statistical significance between protocols uses permutation-based paired tests with 10,000 resamples (Figure A.1).

• Baselines. Four baselines are compared: QA without article ("Naive" in Khan et al., 2024) — judge answers directly without seeing any article, measuring unaided performance; QA with article ("Expert" in Khan et al., 2024) — only for extractive tasks, judge reads the full article and answers, establishing an information ceiling; assigned-role consultancy (Michael et al., 2023; Khan et al., 2024) — consultant argues for randomly assigned correct/incorrect answer while judge asks questions; assigned-role debate (Irving et al., 2018; Khan et al., 2024) — two debaters argue opposite sides, judge observes and decides. Within debate, the paper ablates simultaneous vs. sequential turns, best-of-4 vs. best-of-1 sampling, and 3 vs. 1 rounds.

• Generation budget / compute accounting. The paper does not use a formal FLOPs budget. Instead, each protocol consumes a different number of LLM generation calls per question. For 3-round consultancy: 1 consultant × 3 rounds = 3 consultant generations, plus 3 judge question-generations and 1 judge answer-generation. For 3-round debate with best-of-4: 2 debaters × 3 rounds × 4 samples = 24 debater generations per question, plus 4 scoring calls per turn (to select best-of-4) and 1 judge answer-generation. For 1-turn Elo debates: 2 debaters × 1 round = 2 debater generations per matchup. The total experimental budget across all sweeps is approximately 5 million model generation calls (Section 1). All comparisons between protocols are at equal numbers of questions, not equal generation cost — protocols differ substantially in per-question compute.

• Cross-validation / statistical protocol. There is no cross-validation for strategy selection as in a training setup, since this is an inference-only evaluation. Statistical significance of protocol differences is assessed using permutation-based paired tests (Figure A.1) with permutation_type='samples' and 10,000 resamples (SciPy's permutation_test), testing the null hypothesis that judge accuracy is equal between protocols on the same set of questions. Error bars are 95% confidence intervals computed by first averaging over answer-order-switched replicates for each question (to handle correlation), then assuming IID normal distribution across questions within each condition (Appendix D). For Elo ratings, 95% confidence intervals use bootstrap sampling with 500 seeds (Appendix E). For the open-role protocols, filtering by model direct-QA answer creates a non-random subset; the paper does not apply reweighting to correct for this shift.

Main Quantitative Results

Assigned-Role Protocols: Debate vs. Consultancy vs. Direct QA (Figure 2, Figure B.1)

The paper's central assigned-role results compare judge accuracy across four protocols (QA without article, QA with article, consultancy, debate) as a function of judge model and task type, with Gemini Pro 1.5 as the fixed debater/consultant.

Extractive tasks (Figure 2, left panel): QA with article achieves the highest accuracy for all judge models, demonstrating that "information is best used directly, even by a weak model, rather than filtered through oversight" (Section 4.1). For Gemini Pro 1.5 as judge, QA with article reaches approximately 92% accuracy; for Gemma7B, the weakest judge, it reaches approximately 68%. Debate consistently outperforms both consultancy and QA without article across all judge models except Gemma7B, where debate and QA without article are roughly equal at approximately 52–55%. The gap between debate and consultancy is substantial: with GPT-3.5 as judge, debate reaches approximately 78% vs. consultancy at approximately 62%. With Gemini Pro 1.0, debate reaches approximately 82% vs. consultancy at approximately 70%. The permutation tests in Figure A.1 confirm that for extractive tasks, the debate-vs-consultancy difference is statistically significant at the 5% level for all judge models. The debate-vs-QA-without-article difference is significant for GPT-3.5, Pro 1.0, and Pro 1.5, but not for Gemma7B.

Closed tasks (Figure 2, center panel): Debate consistently outperforms consultancy across all judge models. With Gemma7B, debate reaches approximately 54% vs. consultancy at approximately 52%. With GPT-3.5, debate reaches approximately 70% vs. consultancy at approximately 60%. With Pro 1.0, debate reaches approximately 72% vs. consultancy at approximately 62%. With Pro 1.5 (same-model case), debate reaches approximately 78% vs. consultancy at approximately 68%. The debate-vs-consultancy difference is statistically significant for GPT-3.5, Pro 1.0, and Pro 1.5 (Figure A.1). However, debate vs. QA without article yields mixed results: for Gemma7B, QA without article outperforms debate (approximately 60% vs. 54%), with the difference being statistically significant (p=0.023 per Figure A.1). For GPT-3.5, debate outperforms QA without article (approximately 70% vs. 66%, p=0.037). For Pro 1.0, the difference is not statistically significant (p=0.1). For Pro 1.5, QA without article outperforms debate (approximately 82% vs. 78%, statistically significant). The paper interprets this pattern as indicating that debate is most beneficial when the judge is in a medium capability range — not too weak to understand the debate format (Gemma7B) and not so strong that they can answer correctly alone (Pro 1.5).

Multimodal tasks (Figure 2, right panel): Only Gemini Pro 1.5 supports multimodal input, so the judge model variation is limited to Pro 1.5 (used as both judge and debater) and GPT-3.5 (text-only judge receiving image descriptions, though this detail is not specified). For Pro 1.5 as judge, debate reaches approximately 72% vs. consultancy at approximately 60%. QA without article reaches approximately 68% — the debate-vs-QA-without-article difference is not statistically significant (p=0.11 per Section 4.1 and Figure A.1). Consultancy is significantly worse than all other protocols for multimodal tasks.

Individual task breakdown (Figure B.1): The task-level results reveal substantial heterogeneity within task types. On QuALITY (extractive), the Pro 1.5 judge achieves approximately 90% with QA with article, approximately 82% with debate, and approximately 70% with consultancy. On TruthfulQA (closed), the pattern is notably different: QA without article reaches approximately 62% while debate reaches only approximately 55% with Pro 1.5 judge, and consultancy drops to approximately 52%. On GPQA (closed, without article), all protocols perform poorly — even Pro 1.5 as judge reaches only approximately 45% with debate, compared to approximately 42% with QA without article, consistent with GPQA being a very difficult dataset. On MMLU (closed), debate with Pro 1.5 judge reaches approximately 82%, consultancy reaches approximately 72%, and QA without article reaches approximately 78% — the debate advantage over QA without article is modest.

Judging ability correlates with model capability (Section 4.1): Across all task types, judging accuracy follows the expected capability ordering: Gemma7B < GPT-3.5 ≈ Pro 1.0 < Pro 1.5. The paper notes that "GPT-3.5 and Pro 1.0, are roughly similar" in judging ability, while "Pro 1.5 judge is better than both of those."

Open-Role Protocols: Error Amplification Under Realistic Agent Beliefs (Figure 3, Figures B.9, B.10)

The open-role results measure judge behavior when the consultant or protagonist debater chooses which answer to argue for, rather than being randomly assigned. The protagonist/consultant (Gemini Pro 1.5) chooses the correct answer in 88% (extractive), 84% (closed), and 71% (multimodal) of questions.

Win-rate vs. accuracy (Figure 3, top row): Open consultancy produces consistently higher win-rates (judge agrees with consultant's choice) than open debate. For extractive tasks with Pro 1.5 judge, open consultancy win-rate is approximately 90% while open debate win-rate is approximately 72%. However, higher win-rate does not uniformly translate to higher accuracy. On extractive tasks, open consultancy accuracy (approximately 83%) exceeds open debate (approximately 75%). On closed tasks, open consultancy and open debate produce roughly equal accuracy for Pro 1.5 judge (both approximately 76–78%), though open debate accuracy is lower for weaker judges (GPT-3.5: approximately 65% debate vs. 70% consultancy). On multimodal tasks, open consultancy and open debate accuracy are similar for Pro 1.5 judge (both approximately 68–70%).

Accuracy conditional on protagonist/consultant correctness (Figure 3, bottom row; Finding 4 and 5): This is the paper's most diagnostically important result for open-role protocols. When the consultant chooses the correct answer (dark blue bars in open consultancy), judge accuracy is high — approximately 92% on extractive, 88% on closed, and 85% on multimodal with Pro 1.5 judge. When the consultant chooses the incorrect answer (light blue bars), judge accuracy collapses — dropping to approximately 35% on extractive, 30% on closed, and 20% on multimodal. The judge is essentially trusting the consultant regardless of correctness, with accuracy on incorrect-consultant questions falling well below the 50% random baseline in some cases.

In open debate, the pattern is less extreme in both directions. When the protagonist chooses correctly (dark red bars), judge accuracy is somewhat lower than in open consultancy — approximately 88% on extractive, 82% on closed, and 78% on multimodal with Pro 1.5 judge. But when the protagonist chooses incorrectly (light red bars), judge accuracy is substantially higher than in open consultancy — approximately 55% on extractive, 50% on closed, and 48% on multimodal. The gap between light red and light blue bars is the key finding: open debate provides a much less error-amplifying training signal when the AI agent is wrong. For Pro 1.5 judge on closed tasks, the difference is approximately 50% vs. 30% — a 20 percentage point improvement in detecting incorrect agent beliefs.

Comparison to assigned-role consultancy (Figure A.9): When the consultant argues for the correct answer, open and assigned-role consultancy accuracy are similar. When arguing for the incorrect answer, open consultancy produces lower accuracy than assigned-role consultancy. The paper attributes this to question difficulty: "questions which the open consultant answers incorrectly are more difficult than average, while the assigned-role consultant's assignments are independent of difficulty."

Persuasiveness and Scaling: Elo Ratings vs. Judge Accuracy (Figure 4, Figures E.1, E.2)

Section 4.3 analyzes whether stronger debater models (measured by Elo ratings from cross-play 1-turn debates) lead to higher judge accuracy — a key test of whether debate fulfills the scalable oversight promise.

Elo ratings (Figure 4, top row): Across extractive tasks with Pro 1.5 judge, the Elo ordering is: Pro 1.5 with BoN=4 (highest, approximately 1700 correct-Elo), Pro 1.5 BoN=1 (approximately 1650), Pro 1.0 (approximately 1580), GPT-3.5 (approximately 1520), and Gemma7B (lowest, approximately 1480). The incorrect-Elo scores are consistently lower than correct-Elo for all models, indicating that "it is often easier to argue for the truth, in particular on closed tasks" (Section 4.3). On closed tasks, the Elo differences between models are compressed compared to extractive tasks — all debaters cluster in a narrower range.

Correct answer advantage (Figure 4, middle row): The difference between correct-Elo and incorrect-Elo (the truth advantage) is plotted against aggregate Elo. On extractive tasks, there is a slight positive trend — stronger debaters show a marginally larger truth advantage. On closed tasks, the trend is less clear, with the truth advantage varying across models without a consistent relationship to aggregate Elo. The paper notes this contrasts with Khan et al. (2024), who found a clearer increasing trend on QuALITY.

Elo vs. judge accuracy (Figure 4, bottom row): On extractive tasks with Pro 1.5 judge, there is a positive relationship — higher aggregate Elo corresponds to higher judge accuracy, with the BoN=4 model achieving both the highest Elo and highest accuracy (approximately 82%), and Gemma7B achieving the lowest on both (approximately 75%). On closed tasks, the relationship is much weaker — all debaters produce judge accuracy in a narrow band of approximately 74–78%, despite Elo differences. The paper interprets this as reflecting that "the judge's greater dependence on debater statements in the extractive tasks, where judges rely on debaters for information they cannot access themselves," drives the Elo-accuracy correlation, and that on closed tasks, the judge's own knowledge partially overrides debater persuasiveness.

Robustness to judge model (Figure E.2): When using Pro 1.0 as judge instead of Pro 1.5, the Elo ordering shifts — Gemma7B receives relatively higher Elo scores (the paper speculates about self-preference or family-preference bias). The aggregate Elo-vs-accuracy relationship shows a similar pattern to Pro 1.5 judge but with narrower differentiation between debaters.

Individual task Elo results (not shown in main paper but alluded to): The paper notes that "Elo results aggregated across tasks" (Figures E.1, E.2) obscure task-level variation, but the main text in Section 4.3 states that the findings are "roughly in agreement with Khan et al. (2024) though we see a less consistent and narrower range of judge accuracy improvement."

Ablation Studies and Robustness Checks

The paper conducts extensive ablations, all reported in Figures A.2 through A.8, B.2 through B.8, and discussed in Section 4.1. The consistent theme is that standard LLM performance-improvement techniques do not meaningfully affect debate judge accuracy.

• Number of turns (Figure A.2): Reducing default 3 rounds to 1 round in both consultancy and debate produces no significant effect on judge accuracy across any task type or judge model. The paper notes this is "consistent with the findings of Khan et al. (2024) for QuALITY" and interprets it as suggesting that "more rounds does not help the judge" — debaters may not effectively exploit multi-round structures, or judges may not update their beliefs after the first round.

• Best-of-N for debaters (Figure A.3): Reducing best-of-4 to best-of-1 in debate produces no consistent benefit to judge accuracy. The paper notes this is a "mildly negative sign" for debate as a training protocol since "BoN is motivated as a proxy for training." BoN=4 does increase Elo scores (Section 4.3) but the increased persuasiveness does not translate to better judge decisions.

• Judge few-shot prompting (Figure A.4): Comparing 0-shot (default) to 6-shot prompting with balanced correct answer order shows no significant improvement. For Gemma7B specifically, 6-shot prompting causes failure to produce valid answers on QuALITY, GPQA Extractive, and GSM8KQA due to longer context causing formatting failures.

• Judge chain-of-thought (Figure A.5): Prompting judges to reason step-by-step before answering "tends to harm performance or be insignificant" compared to direct prediction. The paper hypothesizes that "having weaker judges use their own CoT causes them to disregard the verbalised reasoning offered by stronger consultants/debaters, which might be higher quality." This is an important negative result: CoT, which generally improves LLM reasoning, is counterproductive when the judge's own reasoning is lower quality than the arguments they are evaluating.

• Debate turn style (Figure A.6): Comparing simultaneous turns (default, debaters only see previous rounds) to sequential turns (second debater sees first debater's current-round argument) shows no significant difference. The paper conducted this ablation under best-of-N=1 to isolate the turn style effect.

• Positional bias and answer order (Figures A.7, A.8): The paper compares its default single-order evaluation (randomizing which answer appears first) to Khan et al. (2024)'s dual-order approach (running each question twice with order swapped). Figure A.7 shows "no significant difference in mean judge accuracy." Figure A.8 reveals that judges display positional bias (preferring answer 1 or 2), but "using both orders does not change the mean answer position, only reduces its variance," leading the paper to recommend single-order randomization as sufficient.

• Judge invalid responses (Figure B.11): For most judge-protocol combinations, invalid answer rates are <2%. Gemma7B shows elevated invalid response rates on longer-context tasks (QuALITY, GPQA Extractive, GSM8KQA), particularly under 6-shot prompting and debate protocols — the judge "fails to correctly format its response (or make any sense at all)" (Figure B.4 caption).

• Transcript error analysis (Appendix C): Manual analysis of 24 debate transcripts where Pro 1.5 judge made errors on closed tasks. On PrOntoQA (logical reasoning), 11/12 errors stem from "logical errors made by the judge." On TruthfulQA, errors are more diverse: ambiguous questions (6/12), suboptimal honest debating (2/12), judge logical error (1/12), impossible without external information (1/12), and judge inattention (2/12). The paper notes that "the losing debater often correctly identifies the flaw in the dishonest debater's arguments (although the judge fails to capitalize on this information)" — suggesting the debate itself is exposing flaws that the judge is not using.

Critical Assessment

The paper's experiments provide genuine evidence for several of its central claims, but the strength of support varies substantially across claims, and several important questions remain unaddressed by the experimental design.

Finding 1: "Debate consistently outperforms consultancy across all tasks." This claim is well-supported by the data in Figure 2 and the significance tests in Figure A.1. However, the practical interpretation requires nuance that the paper provides but could emphasize more strongly. Consultancy as implemented is an extremely weak baseline: the consultant is assigned the correct answer only 50% of the time, and the judge is asked to determine correctness. A consultant arguing for the wrong answer is actively misleading the judge. The fact that debate outperforms this baseline is perhaps less surprising than the paper's framing suggests — having an opponent who points out flaws should help when the single consultant is lying half the time. A fairer consultancy baseline might have been always assigning the consultant the correct answer (or the answer the consultant model itself believes is correct), which would isolate whether the adversarial structure of debate adds value beyond what a single truthful advisor provides. The paper does not report this comparison. The open consultancy results (where the consultant chooses their answer and is correct 71–88% of the time) show that when the consultant is usually right, consultancy can match or exceed debate on some task types (Figure 3, closed/multimodal with Pro 1.5 judge). This suggests that consultancy's poor assigned-role performance is driven substantially by the 50% incorrect assignment rate — remove that, and the debate-consultancy gap narrows considerably. The "debate beats consultancy" narrative is therefore partly an artifact of the consultancy evaluation design rather than an intrinsic advantage of adversarial structure.

Finding 2: "Comparing debate to direct question answering, the results depend on the type of task... the results are mixed." This claim is the paper's most honest and nuanced finding, and it is well-supported by Figure 2. However, the "mixed" characterization understates a more specific pattern visible in the data: on tasks without information asymmetry (closed, multimodal), debate provides at best a small advantage over QA without article, and only in a narrow judge-capability band (GPT-3.5, barely Pro 1.0). For Gemma7B, debate is worse; for Pro 1.5, debate is worse; only for GPT-3.5 on closed tasks is debate statistically significantly better (p=0.037, at the boundary of significance with 10,000 resamples). This is a much weaker result than "mixed" implies — it is closer to "debate does not reliably improve over direct QA on non-extractive tasks, with at most a small benefit in a specific capability regime." The paper's framing as "mixed" may be overly generous to debate. This is significant because the non-extractive tasks — where the gap between judge and debater is about reasoning and knowledge rather than information access — are arguably more representative of the long-run scalable oversight problem. If debate's advantage is primarily about information transfer (which extractive tasks test), but superhuman AI oversight will be about reasoning and knowledge gaps, the positive extractive results may be less relevant than they initially appear.

Finding 3: "Changes to the setup (number of turns, best-of-N sampling, few-shot, chain-of-thought) seem to have little effect on results." The ablation results (Figures A.2–A.8) consistently support this claim. However, the claim is less informative than it might be because the paper does not investigate why these techniques fail. For chain-of-thought, the paper offers a plausible hypothesis (judges ignore stronger debater reasoning in favor of their own). For best-of-N, the mechanism is unclear — the paper notes BoN=4 increases Elo but not accuracy, but doesn't investigate whether BoN is selecting arguments that are more persuasive in style but not more accurate in substance. For number of rounds, the null result is consistent with Khan et al. (2024) but the paper doesn't analyze whether later rounds contain substantively new information that judges fail to use, or whether debaters simply repeat themselves. These missing mechanistic investigations limit the ablation study's value for guiding future research — knowing that a technique doesn't work is less useful than understanding why it doesn't work when designing alternatives.

Finding 4: "In open consultancy, the judge is equally convinced by the consultant, whether or not the consultant has chosen to argue for the correct answer. Thus, using weak judges to provide a training signal via consultancy runs the risk of amplifying the consultant's incorrect behavior." The data in Figure 3 (bottom row) strongly supports this claim, particularly the light blue bars showing accuracy collapse when the consultant is incorrect. However, the paper's interpretation of this as "amplifying incorrect behavior" assumes that the judge's decision would be used as a training signal without any ground-truth verification — an assumption the paper does not explicitly justify. In a real RLHF pipeline with consultancy, one could imagine incorporating the judge's confidence or using multiple judges. The paper doesn't test whether simple mitigation strategies (confidence thresholds, ensemble judging) would reduce error amplification in consultancy, making the comparison to open debate potentially less favorable to debate if such mitigations are effective. Additionally, the paper doesn't report whether the judge's accuracy on incorrect-consultant questions is above or below the 50% random baseline (on some task types, the light blue bars in Figure 3 appear to be near or below 50% for weaker judges), which would clarify whether consultancy is merely unhelpful or actively harmful when the consultant is wrong.

Finding 5: "In open debate, in contrast, the judge follows the debater's choice less frequently than in open consultancy." The data clearly supports this (Figure 3, top row, open debate win-rates are lower than open consultancy). The more important sub-claim — that open debate provides a less error-amplifying training signal — is supported by the bottom row of Figure 3, but the paper does not quantify the net effect of the tradeoff (lower accuracy when protagonist is correct + higher accuracy when protagonist is incorrect). A proper evaluation would compute expected judge accuracy under realistic protagonist correctness rates and compare open debate to open consultancy on this expected metric. The paper provides the conditional accuracies and correctness rates (88%, 84%, 71%) but does not compute the weighted average, leaving the reader to estimate whether the tradeoff is net-positive or net-negative for debate. On closed tasks with Pro 1.5 judge, the approximate expected accuracy for open consultancy is 0.84 × 88% + 0.16 × 30% ≈ 78.7%, while for open debate it is 0.84 × 82% + 0.16 × 50% ≈ 76.9% — actually slightly worse for debate by this crude calculation. The paper's claim that debate provides a less error-amplifying signal is correct (the worst-case accuracy is better), but the average-case signal may not be better, and this tradeoff should be made explicit.

Finding 6: "Stronger debater models increase judge accuracy, though more modestly than in previous studies." The Elo-vs-accuracy analysis in Figure 4 provides mixed evidence. On extractive tasks, the positive relationship is visible but shallow — a ~200-point Elo increase (Gemma7B to Pro 1.5 BoN=4) corresponds to roughly a 7 percentage point accuracy increase (75% to 82%). On closed tasks, the relationship is essentially flat — all debaters produce judge accuracy in a 4 percentage point band despite Elo differences. The paper correctly notes this is "weaker" than Khan et al. (2024), but the deeper issue is that the closed-task result undermines a core motivation for debate: if stronger debaters don't produce more accurate judges on reasoning tasks, the scalable oversight promise (that debate improves as AI capabilities increase) is not supported for the type of capability gap most relevant to superhuman AI oversight. The paper's interpretation of this as "weakly positive" may be too generous — on closed tasks, the evidence for scaling is effectively null. Additionally, the potential confound of self-preference bias (Gemini-family judges rating Gemini-family debaters higher) means the Elo scores themselves may be contaminated, and without a human-judge calibration, the paper cannot quantify the magnitude of this bias.

Methodological limitations that affect multiple claims:

• Test set size of 128 questions per task is modest. When split across five difficulty bins (not done in this paper, but relevant for variance estimation), per-condition sample sizes would be ~25. The paper aggregates across tasks within task types for most analyses, which partially addresses this, but task-level heterogeneity (visible in Figure B.1) gets lost in the aggregation. For example, on TruthfulQA, debate with Pro 1.5 judge underperforms QA without article by ~7 percentage points; on MMLU, it outperforms by ~4 points. The aggregate "closed task" result averages over these opposing patterns, and a different task selection could shift the aggregate conclusion.

• Single primary debater model (Gemini Pro 1.5) for main experiments. The paper only varies debaters in Section 4.3 (Elo analysis), using simplified 1-turn debates. The main assigned-role and open-role results (Figures 2 and 3) all use Gemini Pro 1.5 as debaters. If Gemini Pro 1.5 is particularly good (or bad) at the argumentation style required for debate, the results may not generalize to other strong models. The finding that debate helps on extractive but not closed tasks might be model-specific — a model with different reasoning-vs-extraction capabilities might show different patterns.

• No human judge comparison. The paper's title and motivation are about "weak LLMs judging strong LLMs," but there is no validation that LLM judging behavior correlates with human judging behavior. If LLM judges systematically differ from humans in how they weigh verified evidence, respond to persuasive style, or detect logical fallacies, the results may not transfer to the human supervision setting that motivates scalable oversight.

• Inference-only limitation acknowledged but not compensated for. The paper is explicit that all experiments are inference-only and that training-based evaluation is the necessary next step. But it does not discuss what specific inference-only results would or would not be expected to transfer to training. For example, if debate training converges to equilibria where debaters are truthful and judges learn to trust verified evidence, the current finding that debate doesn't improve over direct QA on closed tasks might be irrelevant — the inference-only prompted debaters are not at equilibrium, and their arguments may not be truthful. The paper's results are necessary conditions for debate to work at inference time, but they are neither necessary nor sufficient for debate to work as a training protocol — a limitation the paper acknowledges but does not fully explore.

Missing experiments that would strengthen the paper:

• No combination of debate with passage verification on closed tasks. The paper notes that passage verification was a key driver of judge accuracy on extractive tasks in Khan et al. (2024). An experiment providing judges on closed tasks with a similar verification mechanism (e.g., fact-checking claims against a knowledge base, or executing code for GSM8KQA) would test whether debate's advantage is specifically about adversarial structure or more generally about providing verifiable evidence.
• No analysis of debate transcript content. Beyond the small qualitative error analysis in Appendix C (24 transcripts), the paper does not analyze whether debaters are actually engaging in substantive rebuttal (pointing out specific flaws in opponent reasoning) or merely restating their own positions. If debaters are not effectively using the multi-round structure to expose errors, the null result on number of turns would have a clear mechanistic explanation.
• No open-role analysis with varying protagonist correctness rates. The paper reports overall correctness rates (88%, 84%, 71%) but doesn't analyze whether debate's advantage over consultancy in the incorrect-protagonist case depends on why the protagonist is wrong (knowledge gap vs. reasoning error vs. ambiguous question). The error analysis in Appendix C suggests different failure modes on different tasks, but this is not connected to the quantitative open-role results.
• No judge confidence calibration analysis. Knowing whether judges are overconfident when wrong (which would amplify training errors) or well-calibrated (which would allow confidence-based mitigation) is relevant to the error amplification claim. The paper reports only binary accuracy, not confidence scores or calibration.

6. Limitations and Trade-offs

Inference-Only Evaluation Does Not Measure Training Dynamics — and That Is the Whole Point of the Protocol

The assumption or constraint. The paper's central limitation, acknowledged with unusual candor, is that it studies debate and consultancy exclusively in an inference-only setting — models are prompted to play their roles with no training, no optimization, and no learning from the judge's decisions. The theoretical motivation for debate (Irving et al., 2018) provides guarantees about behavior at Nash equilibrium: "in all the Nash equilibria of the debate game, both debaters tell the truth in the most convincing manner possible" (Section 1). But the paper's experiments are not at equilibrium — they are zero-shot or few-shot prompted rollouts. The paper is explicit about this gap:

"all our experiments are done as inference-only... what we actually care about are the safety properties of the optima/equilibria that arise due to the incentives produced by training models specifically in consultancy/debate decision-problems/games. While results on win rates, and advantages arising from selecting correct vs. incorrect answers provide some evidence of their abilities to play these games, they don't give us much evidence about their effectiveness as scalable oversight training protocols." (Section 5, Limitations)

An additional confound: the debater models are fine-tuned with RLHF for honesty, which the paper notes "may hamper the dishonest consultant/debater" because RLHF-trained models may resist arguing for incorrect answers even when instructed. This means the inference-only results potentially overestimate debate's safety properties — the dishonest debater is a weaker adversary than one trained adversarially.

The consequence. An inference-only evaluation can demonstrate that debate has desirable properties as a static interaction format, but it cannot demonstrate that training with debate's judge signal would converge to truthful equilibria. A protocol could look good in inference-only evaluation (judges make accurate decisions when presented with prompted arguments) but fail catastrophically under training because models learn to produce arguments that exploit specific weaknesses in the judge — weaknesses that would never appear in a dataset of prompted arguments from an RLHF-trained model. The paper's finding that best-of-N sampling improves persuasiveness (higher Elo, Section 4.3) without improving judge accuracy is a hint of this dynamic: optimization against a fixed judge produces more persuasive but not more truth-conducive arguments. Training would involve much more optimization pressure than best-of-4 sampling, potentially amplifying this gap.

The paper is essentially measuring whether debate as currently implemented with prompted models helps judges — it is not measuring whether debate as a training protocol would produce truthful AI behavior. For the paper's stated motivation — scalable oversight for superhuman AI — the training question is the one that matters. The inference-only results are a useful diagnostic (a protocol that fails at inference time would be unlikely to succeed under training), but the paper cannot distinguish between (a) "debate works at inference time but would fail under training" and (b) "debate works at inference time and would work even better under training" — and from a safety perspective, these lead to opposite conclusions about whether to invest in debate-based alignment.

What evidence exists in the paper. The paper does not measure training dynamics at all — this is an acknowledged scope limitation, not a hidden flaw. However, several results suggest inference-only evaluation may paint an overly optimistic picture:

• The ablation on best-of-N (Figure A.3) shows that optimization against a fixed judge (selecting the most persuasive of 4 samples) does not improve judge accuracy, even though it increases Elo (Section 4.3). This is a microcosm of the training problem: optimizing for persuasiveness does not optimize for truth-transfer.
• The paper's analysis that RLHF-trained models may be reluctant dishonest debaters (Section 5, Limitations) implies that the inference-only results use artificially weak adversaries.
• The finding that stronger debaters produce only modest accuracy improvements (Figure 4, bottom row, closed tasks show ~4 percentage point range) suggests that increasing optimization power may not improve judge accuracy.

Mitigation status. The paper explicitly calls this out as the "most immediate future work": "to train debater and consultant models via the signal provided by the judge on the tasks studied here. This is the full setup required to test scalable oversight methods" (Section 5, Future work). The paper also suggests studying "whether [protocols] are robust to misaligned models" — a more adversarial training setting. No mitigation is attempted within the current paper; the authors treat the inference-only study as a necessary preliminary step before committing to the more expensive training-based evaluation. This is a reasonable methodological choice, but it means every conclusion in the paper is subject to the caveat that training dynamics could reverse the findings.

The Passage Verification Tool Is Both Unrealistic and Likely the Primary Driver of Extractive-Task Results — Undermining the Claim That Debate's Structure Provides the Benefit

The assumption or constraint. For extractive QA tasks, debaters and consultants are required to include direct passages from the source article wrapped in <passage> tags, and a regex-based verification tool checks whether each passage appears as an exact substring in the article, marking it as verified (<v_passage>) or unverified (<u_passage>) (Section 3.2; Appendix G). Judges are explicitly instructed to trust verified passages and distrust unverified ones. This mechanism gives the judge a perfect truth oracle for factual claims about the article's contents — something no real-world scalable oversight scenario would provide a human judge supervising a superhuman AI.

The paper is aware of this tension. It notes that Khan et al. (2024) found that "the quote verification, rather than the argumentation, is the dominant contribution to judge accuracy (using quotes alone improves beyond the combination of arguments and quotes)" on QuALITY, and acknowledges that "such a quote verification tool may be unrealistic considering the weak-strong analogy we expect to face for scalable oversight with humans" (Appendix F.1). But the paper does not address the implication: if the primary mechanism by which debate outperforms direct QA on extractive tasks is the verification tool — not the adversarial argumentation — then the extractive-task results provide essentially no evidence that debate's structural properties matter for scalable oversight.

The consequence. The paper's most robust positive finding — "debate outperforms QA without article on extractive tasks" (Finding 2, Section 1; Figure 2 left panel) — cannot be attributed to debate as a debate protocol because the verification tool is confounded with the debate structure. A fair test of whether debate's adversarial structure helps would require an extractive-task condition where judges see verified passages from both sides without the debate framing — essentially, a "verified evidence comparison" protocol. The paper does not include such a condition. Without it, we cannot distinguish between:

1. "Debate works because it lets the judge compare opposing verified claims" — in which case the debate framing is doing real work, and the result supports the theoretical argument that adversarial structure helps judges evaluate evidence.

2. "Verified passages work because they give the judge a reliable information source while the debate framing is incidental" — in which case the debate structure is providing no additional benefit over simply presenting verified evidence from both sides, and the extractive-task result would not generalize to tasks where evidence cannot be perfectly verified (which is the realistic scalable oversight scenario).

Khan et al. (2024)'s finding that quotes alone outperformed debate + quotes on QuALITY (which the current paper cites) strongly favors interpretation (2). If verified evidence alone is better than evidence embedded in debate, debate's structure may be adding noise rather than signal — the adversarial framing might confuse judges relative to a neutral presentation of verified facts.

The paper partially addresses this by studying closed QA tasks (where there is no article and no verification tool), and finding "mixed" results. But the paper's narrative lumps extractive and closed results together when drawing positive conclusions ("debate outperforms consultancy across all tasks," Finding 1) while separating them when the results are negative ("results depend on the type of task," Finding 2). A reader focused on the headline findings might conclude that debate works broadly, when the only task type where it clearly beats direct QA is one where a tool — not the protocol — provides the leverage.

What evidence exists in the paper. The paper itself provides the key evidence: it cites Khan et al. (2024)'s finding that quote verification dominates argumentation (Appendix F.1), and it shows that QA with article (where the judge reads the full text directly) substantially outperforms debate on extractive tasks (Figure 2, left panel, purple bars vs. red bars). This means that even with perfect passage verification, debate is a lossy information channel compared to direct access — information degrades when filtered through debate, and the only reason debate beats QA without article is that QA without article has zero information about the article. The verification tool artificially fixes one aspect of the information asymmetry (fabricated claims) while leaving other aspects (incomplete coverage, selective quoting, misleading framing) for debate to handle.

Mitigation status. The paper does not attempt to mitigate this limitation. It acknowledges the tension in Appendix F.1 but does not run the obvious ablation — debate on extractive tasks without passage verification (or with a weaker verification mechanism) — that would measure how much of debate's advantage is verification-dependent. It does not include a "verified evidence presentation" baseline that would isolate the verification tool's contribution from debate's structural contribution. The paper's inclusion of closed tasks partially addresses the concern by providing results in a regime without verification, but the extractive-task results remain the paper's strongest positive finding, and their interpretation is fundamentally ambiguous.

The Weak Judge Regime Where Debate Produces Reliable Gains Over Direct QA Is Narrow — and Excludes Both Very Weak and Strong Judges on Non-Extractive Tasks

The assumption or constraint. The paper's core hypothesis is that debate should help weaker judges supervise stronger agents. This predicts a monotonic (or at least consistently positive) relationship: as the judge becomes weaker relative to the debaters, the benefit of debate over direct QA should increase — or at minimum, debate should outperform direct QA for all judge models weaker than the debaters. The paper tests this by varying the judge across four capability levels (Gemma7B, GPT-3.5, Gemini Pro 1.0, Gemini Pro 1.5) while holding debaters fixed at Gemini Pro 1.5. The results on closed and multimodal tasks — which are the more realistic testbeds for the reasoning-based weak-strong gaps that motivated the paper — reveal a narrow sweet spot for debate, not a robust advantage.

The consequence. On closed QA tasks (Figure 2, center panel), debate outperforms QA without article with statistical significance only for GPT-3.5 as judge (p=0.037). For Gemma7B (the weakest judge), QA without article is significantly better than debate (p=0.023). For Gemini Pro 1.0 (mid-tier, weaker than the debaters), the difference is not significant (p=0.1). For Gemini Pro 1.5 (judge same as debaters), QA without article significantly outperforms debate. On multimodal tasks, debate and QA without article are not significantly different for any judge (p=0.11 and above).

This means that on the tasks most analogous to real scalable oversight (where the gap is about reasoning and knowledge, not information access), debate provides a replicable advantage over direct QA only in a narrow band: the judge must be weaker than the debaters but not too weak to understand the debate format, and not so close in capability that their own reasoning outperforms the debate-augmented signal. For Gemma7B — a 7B parameter open-source model that might be the most realistic stand-in for a human judge in terms of relative capability gap — debate is actively harmful compared to the judge answering alone. For Pro 1.0 — a strong model that is nonetheless weaker than Pro 1.5 — debate provides no measurable benefit.

This is a substantially weaker empirical picture than the one the paper's framing suggests. A practitioner deciding whether to invest in debate-based oversight for a real system would want to know: what is the probability that debate improves over the baseline for a randomly chosen weak-strong judge-debater pair? On the paper's evidence, on non-extractive tasks, that probability is low — debate reliably helps only when the judge's capability happens to fall in a specific intermediate band relative to the debater, and the paper provides no method for identifying this band a priori.

What evidence exists in the paper. The pattern is directly visible in the center and right panels of Figure 2, and the statistical tests in Figure A.1 quantify the confidence. The paper partially acknowledges this in Section 4.1:

"Our interpretation is as follows: Gemma7B is likely too weak to learn anything that valuable about these protocols. Pro 1.5 has no weak-strong asymmetry in this closed QA case, so doesn't inform much about scalable oversight. Pro 1.0 and GPT-3.5 are probably the most indicative of scalable oversight, and indicate either small or no benefit of debate over QA without article."

This interpretation is reasonable but implicitly treats the non-significant Pro 1.0 result as "small benefit" when the statistical test says no significant difference, and treats the GPT-3.5 result (p=0.037, near the significance threshold after 10,000 resamples) as the most informative case. With only one judge model (GPT-3.5) showing a statistically significant benefit, and that benefit being near the significance boundary, the "small or no benefit" characterization may overstate the reliability of the positive result.

Mitigation status. The paper does not address this narrowness directly. It does not test enough judge models to map the sweet spot's boundaries precisely (only four models, of which one is too weak, one shows borderline significance, one shows no significance, and one is same-model). It does not provide a methodology for predicting whether a given judge-debater pair will fall in the beneficial regime. The paper's future work section suggests finetuning judges, which might expand the sweet spot, but does not acknowledge that the current results suggest the sweet spot is surprisingly narrow.

Open-Role Protocol Construction via Data Filtering Introduces a Difficulty Confound That Biases the Error-Amplification Results

The assumption or constraint. The open-role protocols (open consultancy and open debate) are constructed not by generating new interactions where the AI agent freely chooses its answer, but by filtering existing assigned-role data based on whether the agent's assigned answer matches what the same model would choose under direct QA (Section 3.2.3). This means that for open consultancy, the paper selects the consultancy evaluation where the consultant happened to be assigned the answer matching its direct QA preference; for open debate, the paper designates as protagonist whichever debater's assigned side matches its direct QA preference.

This creates a systematic difficulty confound: the set of questions where the strong model (Gemini Pro 1.5) gets the answer right under direct QA (and thus where open-role protocols select the "correct answer" condition) is easier on average than the set where the model gets the answer wrong (and thus where open-role protocols select the "incorrect answer" condition). The paper itself quantifies this: the model chooses correctly on 88%, 84%, and 71% of questions for extractive, closed, and multimodal tasks respectively (Figure 3 caption). The open-role results therefore compare judge accuracy on different question distributions between the "protagonist correct" and "protagonist incorrect" conditions.

The consequence. When the paper reports that "in open consultancy, the judge scores high accuracy when the consultant is correct (dark blue) but low accuracy when the consultant is incorrect (light blue)" (Section 4.2), and contrasts this with open debate where the difference is less extreme, it is measuring a compound effect: the inherent difficulty difference between questions the model gets right vs. wrong, plus the protocol's effect on judge accuracy. If questions where the model is wrong are intrinsically harder (and thus harder for the judge), then some portion of the accuracy drop in the "incorrect consultant" condition is not attributable to the consultant misleading the judge — it reflects the fact that those questions are harder. The paper acknowledges this for the consultancy comparison:

"When consultants argue for correct answers, the accuracy of open and assigned-role consultancy are similar, whilst when arguing for incorrect answers, open consultancy leads to lower accuracy. We hypothesise this is because questions which the open consultant answers incorrectly are more difficult than average, while the assigned-role consultant's assignments are independent of difficulty." (Appendix A, Figure A.9 caption)

But the paper does not adjust for this confound when comparing open debate to open consultancy. If the difficulty confound affects both protocols equally, the comparison is still valid. But it may not: if debate's adversarial structure helps judges more on difficult questions than easy ones (plausible, since easy questions may not need adversarial testing), then the difficulty confound could make open debate look artificially good (it benefits disproportionately from being tested on harder questions in the "incorrect protagonist" condition) or artificially bad (the "correct protagonist" condition contains disproportionately easy questions where adversarial pressure is counterproductive).

What evidence exists in the paper. The paper partially characterizes the confound through Figure A.9 (comparing open and assigned-role consultancy), showing that the accuracy gap between open and assigned-role consultancy is larger when the consultant argues for the incorrect answer. This directly demonstrates that the open-role "incorrect" condition involves harder questions. But the paper does not:

• Report average direct QA accuracy (or any difficulty metric) for the question subsets in each open-role condition, which would quantify the difficulty shift.
• Apply any reweighting or stratification to control for difficulty when comparing open debate to open consultancy.
• Discuss whether the difficulty confound would be expected to affect debate and consultancy differently.
• Compute difficulty-adjusted accuracy metrics that would isolate the protocol effect from the question difficulty effect.

Mitigation status. The paper does not attempt to mitigate this confound. The open-role construction via data filtering is computationally efficient (it reuses existing evaluations rather than generating new ones) but introduces this statistical issue. An alternative approach — generating new open-role interactions specifically for a randomly sampled test set and randomizing whether the agent chooses its answer or is assigned — would eliminate the confound at additional computational cost. The paper does not discuss this tradeoff or suggest it as future work. Given that the open-role error-amplification result (Finding 5) is one of the paper's most novel and potentially influential contributions, the confound should be acknowledged more prominently than it is.

No Human Judge Validation — LLM Judges May Not Proxy for Human Behavior in Debate Evaluation

The assumption or constraint. The paper uses LLM judges as stand-ins for human judges, following the approach of Khan et al. (2024) and Radhakrishnan (2023). The entire motivation for scalable oversight is about enabling human supervision of superhuman AI. The paper's title — "On scalable oversight with weak LLMs judging strong LLMs" — signals that LLM judges are the object of study. But the paper never validates whether LLM judging behavior correlates with human judging behavior, and it draws conclusions about what this means for "the long-run difference between superhuman AIs and humans" (Section 1) without testing the human case.

The consequence. LLM judges may differ from human judges in ways that systematically affect the protocol comparison:

• Passage verification reliance: LLM judges are explicitly instructed in their prompts to trust verified passages and distrust unverified ones (Appendix H.3, H.4). Humans might weight this instruction differently — they might be more skeptical of verified passages that seem irrelevant, or more willing to accept unverified claims that seem plausible. If LLM judges slavishly follow the verification tags while humans would exercise independent judgment, the extractive-task results overstate debate's benefit for humans.

• Persuasiveness vs. accuracy: LLMs may be susceptible to persuasive rhetoric that humans would recognize as vacuous, or may fail to recognize subtle logical flaws that humans would catch. The paper's qualitative error analysis (Appendix C) found that on PrOntoQA, 11/12 judge errors were "logical errors made by the judge" — a Gemini Pro 1.5 judge failing at logical reasoning that the prompt explicitly asks it to perform. Humans might make different errors or fewer errors on logical tasks.

• Self-preference bias: The paper documents that Gemini-family judges assign lower Elo scores to GPT-3.5 than expected based on capability (Section 4.3), speculating about "self-preference bias." This bias is specific to LLM-as-judge settings and has no human analog — a human judge evaluating AI debaters would not exhibit model-family preference.

• Context length and attention: LLM judges, particularly weaker ones (Gemma7B), sometimes fail to produce valid answers on longer transcripts (Figure B.11). Humans might also struggle with long, complex debate transcripts, but the failure mode would be different — fatigue rather than token-limit truncation. The paper's null result on number of turns (Figure A.2) might reflect LLM attention limitations rather than a genuine property of debate — humans might benefit from additional rounds even if LLMs do not.

What evidence exists in the paper. The paper does not include any human judge data. It cites Khan et al. (2024)'s human judge experiments as related work but does not attempt to calibrate its LLM judges against human judgments on the same tasks. The paper acknowledges in its limitations:

"Our work attempts to analogise the weak-strong dynamics of humans supervising superhuman AI, but our study is limited by not using humans and using AI which is far from superhuman on many capabilities. A key uncertainty is whether humans will be able to appropriately follow and accurately judge arguments made by superhuman AI." (Section 5, Limitations)

But this acknowledges the missing superhuman AI problem, not the missing human judge validation problem. Even if AI were superhuman, we would want to know whether LLM judges today are a valid proxy for human judges today — this is an empirical question that the paper doesn't address.

Mitigation status. The paper suggests human judge studies as future work: "The work could further be extended through a study involving human judges" (Section 5, Future work). But it does not treat the absence of human validation as a limitation on the current results' interpretability — the paper's conclusions are stated as findings about scalable oversight protocols generally, not as findings about LLM-judge behavior specifically. A more conservative framing would note that all results are conditional on LLM judges being a reasonable proxy for human judges, and that this proxy assumption is untested.

The Total Compute Budget Is Not Normalized Across Protocols — Debate Costs Substantially More Per Question Than Direct QA, Making Accuracy Comparisons Incomplete

The assumption or constraint. The paper compares judge accuracy across protocols at equal numbers of questions (128 per task), not at equal compute cost. The protocols differ substantially in their per-question generation requirements: QA without article uses a single judge generation per question; 3-round consultancy uses 3 consultant generations + 4 judge generations (3 questions + 1 answer); 3-round debate with best-of-4 uses 24 debater generations (2 debaters × 3 rounds × 4 samples) + 12 scoring calls (to select best-of-4 each round) + 1 judge generation. By the paper's own accounting, debate with BoN=4 consumes roughly 24× more debater generations than QA without article uses total generations, plus additional scoring overhead.

This cost asymmetry matters because the paper's central policy question is: given a fixed budget, should a system deploy debate or direct QA? If debate achieves 70% accuracy at 24× the cost, and direct QA achieves 66% accuracy at 1× the cost, then for most practical budgets, direct QA dominates — you could run direct QA on 24× more questions, use majority voting, or spend the compute on other improvements. The paper never makes this comparison explicit because it normalizes by questions, not by cost.

The consequence. The paper's reported accuracy numbers for debate include a hidden compute subsidy: debate is allowed to consume far more inference compute than the baselines it is compared against, without this being factored into the evaluation. If the comparison were cost-normalized — e.g., "what accuracy does each protocol achieve given a budget of N total model generations?" — debate's advantage would likely shrink or disappear on many task types.

This is particularly relevant for the debate-vs-QA-without-article comparison on closed tasks, where the paper reports debate as being "statistically significantly better" for GPT-3.5 with p=0.037, but debate requires ~25× more compute per question. A cost-normalized comparison might show that running QA without article with majority voting over 25 independent judge answers (costing roughly the same as one debate) outperforms debate — a comparison the paper does not make. The ablation finding that BoN=4 does not improve judge accuracy (Figure A.3) suggests that simply spending more compute on higher-quality debater arguments doesn't help, but spending that compute on more independent judge answers might — and this alternative is never tested.

The paper's 5-million-generation-call budget (Section 1) is large enough that per-question cost differences don't constrain the experimental design, but the cost asymmetry becomes critical when translating results into practical recommendations. A practitioner reading "debate outperforms consultancy across all tasks" might reasonably assume this is at comparable cost, when in fact debate costs substantially more.

What evidence exists in the paper. The paper reports generation counts for best-of-N (4 samples per turn, Section 4.1) and notes that BoN=4 doesn't improve accuracy (Figure A.3). But it does not:

• Report total generation counts per protocol per question, which would make the cost asymmetry transparent.
• Compute any cost-normalized accuracy metric (e.g., accuracy per 1000 generations).
• Compare debate to direct QA augmented with cost-matched compute (e.g., majority voting over N direct QA answers where N matches debate's per-question cost).
• Discuss whether the BoN=4 null result implies that debate's accuracy advantage is cost-inefficient (since most of debate's cost is in the BoN sampling, and removing it doesn't change accuracy, yet keeping it doesn't help — so debate's cost is inflated without benefit).
• Compare debate at best-of-1 (which costs ~6 debater generations per question: 2 debaters × 3 rounds) to QA without article with majority voting over 6 samples — a fairer comparison that the BoN=1 data enables but the paper doesn't compute.

Mitigation status. The paper does not acknowledge the cost-asymmetry problem or attempt to normalize by compute. The ablation on BoN (Figure A.3) provides the necessary data for a partial cost-normalized comparison (debate at BoN=1 vs. QA without article), but the paper doesn't compute it. The Elo analysis (Section 4.3) uses 1-turn debates, which are cheaper, but these are not compared to direct QA baselines. The "approximately 5 million model generation calls" figure is reported as a measure of experimental scale, not as a cost to be optimized. This is a missed opportunity: with the data the paper already collected, it could have computed cost-efficiency curves that would substantially clarify the practical value of debate.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not announce a new protocol or a performance breakthrough, but it changes the landscape for scalable oversight research in three concrete ways that matter more than a single positive result would.

First, it provides a diagnostic taxonomy that reframes the debate evaluation problem. Before this work, the field's empirical evidence on debate was organized around a single dataset (QuALITY) and a single type of weak-strong gap (information asymmetry). A researcher asking "does debate work?" would find that the answer depends on which paper they read — Khan et al. (2024) said yes on QuALITY, Michael et al. (2023) said no with GPT-4 debaters and human judges, and Huang et al. (2023) found that multi-agent debate on reasoning tasks was no better than majority voting. These conflicting findings were not contradictory in a deep sense — they were testing debate under different conditions — but the field lacked a framework for reconciling them.

This paper provides that framework by demonstrating that debate's efficacy is not a yes/no question but a function of at least three variables: the type of capability gap (information asymmetry vs. reasoning/knowledge gap), the magnitude of the judge-debater capability ratio, and whether the evaluation measures average accuracy or worst-case error amplification. The extractive-task results (debate helps) and closed-task results (debate helps weakly or not at all) are not in conflict — they are different points on a surface that this paper maps for the first time. The finding that chain-of-thought reasoning for judges is counterproductive (Figure A.5) — because it causes weaker judges to ignore the stronger debater's reasoning in favor of their own — provides a mechanistic hypothesis for why debate's benefit varies with judge capability: the protocol only helps when judges can recognize superior reasoning without attempting to generate it themselves.

This reframing matters because it changes what counts as progress. Before this paper, a study showing debate helps on a new extractive QA task would have been publishable and would have been interpreted as strengthening the case for debate. After this paper, such a study would be recognized as replicating a known pattern — information asymmetry with passage verification — without addressing the open question of whether debate helps on reasoning tasks without verification tools. The evidentiary burden has shifted: future work claiming debate works must demonstrate benefits specifically on tasks without information asymmetry and without oracle verification mechanisms, or it hasn't shown anything the field doesn't already know.

Second, the open-role protocol innovation introduces a safety-relevant evaluation metric that prior work had entirely missed. The distinction between average judge accuracy and error amplification under realistic agent beliefs is not a minor methodological refinement — it changes the value proposition of different protocols. A naive reading of the assigned-role results (Figure 2) might conclude that consultancy is simply weaker than debate and should be abandoned. But the open-role results (Figure 3) reveal a more subtle picture: when the consultant is correct (which happens 71–88% of the time), open consultancy achieves accuracy comparable to or better than open debate; the problem is that when the consultant is wrong, the judge cannot tell, and the protocol amplifies the error. Debate does not eliminate this problem — the judge still makes mistakes — but it substantially reduces the error magnitude when the protagonist is wrong (Figure 3, bottom row, light red vs. light blue bars).

This reframing connects the scalable oversight evaluation directly to the safety motivation articulated in the paper's introduction. The concern that AI systems might generalize from reward hacking to scheming (Section 1, citing Denison et al., 2024; Carlsmith, 2023) is fundamentally a concern about error amplification in training: a weak supervisor provides feedback that inadvertently reinforces behavior that looks good in training but generalizes badly. The open-role protocol directly measures this: when the AI agent has an incorrect belief, does the supervisor's feedback correct it or reinforce it? The paper shows that consultancy reinforces it, while debate provides partial correction. This is not an accuracy story — it is a safety story, and it suggests that protocol evaluation should include this error-amplification metric as a standard diagnostic. The paper's methodology (filtering assigned-role data by model direct-QA preference) provides a computationally cheap way to compute this metric that future work can adopt immediately.

The finding that debate's advantage over consultancy is most pronounced exactly in the safety-relevant case (when the AI is wrong) is a specific, falsifiable claim that should guide protocol selection for training-based evaluations. If a research group is deciding whether to invest in debate-based training or consultancy-based training for alignment, the open-role results provide concrete evidence favoring debate — not because debate produces higher average accuracy (it may not), but because it is less likely to create the kind of error-reinforcement loops that the safety literature identifies as dangerous. This is a different argument for debate than the one made by Irving et al. (2018) — it does not rely on Nash equilibrium guarantees — and it opens a new avenue for advocating debate even if the equilibrium arguments remain unverified empirically.

Third, the paper calibrates expectations about debate's scaling properties in a way that is sobering but productive. The finding that stronger debaters produce only modest improvements in judge accuracy (Figure 4, bottom row — roughly a 7-percentage-point range on extractive tasks and ~4 points on closed tasks across a substantial Elo range) is a significant downward revision from Khan et al. (2024), who reported a "consistent" positive trend. This calibration matters because enthusiasm for debate as an alignment strategy has been partly driven by the scaling narrative: the idea that as AIs get more capable, debate naturally becomes more effective, providing a self-reinforcing safety property.

The paper's scaling results suggest this narrative is, at best, incomplete. The correlation between debater strength and judge accuracy is real but weak on extractive tasks, and essentially flat on closed tasks (Section 4.3). The paper's interpretation — that judges rely on debaters more when they lack independent access to information, and that on reasoning tasks the judge's own knowledge partially overrides debater persuasiveness — implies that debate's scaling benefit may be largest for precisely the type of gap (information access) that is least relevant to superhuman AI oversight, and smallest for the type of gap (reasoning and knowledge) that will define the human-AI capability differential.

This does not mean debate cannot scale — the paper's debaters are prompted, not trained, and adversarial training might produce qualitatively different scaling behavior. But it does mean that the empirical case for debate's scaling properties is weaker than the field believed, and that future work claiming scaling benefits needs to demonstrate them across multiple task types and with controls for judge-model family effects. The paper's identification of self-preference bias as a potential confound in LLM-judge Elo ratings (Section 4.3) provides a specific methodological caution: if Gemini-family judges systematically rate Gemini-family debaters higher, then cross-family Elo comparisons are contaminated, and scaling trends computed from such Elo scores may be partially artifactual. This is a concrete warning that applies to any work using LLM judges to evaluate other LLMs, not just debate research.

The net effect on the research landscape is that debate remains a viable candidate for scalable oversight — the paper explicitly states it "remain[s] optimistic about the prospects for debate" (Section 1) — but the path to demonstrating its viability has become more specific and more demanding. Future work must show benefits on reasoning tasks without oracle verification, must measure error amplification as well as average accuracy, and must demonstrate scaling trends that survive cross-model-family controls. This is a harder bar than the field faced before this paper, and meeting it will require training-based evaluations rather than inference-only prompting. The paper's most important contribution may be that it makes the case for training-based evaluation not as a nice-to-have but as a necessity — the inference-only results have reached their limit, and further progress requires confronting the training dynamics that the protocol's theoretical motivation depends on.

Follow-Up Research This Work Enables

Training debaters via self-play using the judge's decision as a reward signal, on the same nine-task benchmark developed in this paper. This is the paper's most explicitly identified future direction (Section 5, Future work: "The most immediate future work is to train debater and consultant models via the signal provided by the judge on the tasks studied here") and also the most consequential. The paper's entire inference-only evaluation is a preliminary step that establishes baseline properties — debate outperforms consultancy, open debate reduces error amplification, scaling trends are modest — but the theoretical motivation for debate only applies at equilibrium under training. A training-based evaluation would close the gap between what the paper measures (prompted argumentation behavior) and what the protocol is designed for (incentivized truthful argumentation).

A strong follow-up would train two Gemini Pro 1.5 debaters using RL with a reward signal derived from whether a weaker judge (e.g., GPT-3.5 or Gemma7B) selects their answer, run the same nine-task evaluation with the trained debaters, and measure four quantities: (1) whether judge accuracy improves over inference-only debate, (2) whether the debaters remain truthful when arguing for incorrect answers (do trained debaters learn to deceive the judge, or does the adversarial dynamic constrain deception?), (3) whether the scaling trends (Figure 4) strengthen under training (do trained debaters show larger Elo-accuracy correlations?), and (4) whether the open-role error-amplification advantage of debate over consultancy persists or changes under training. The negative result — that training debaters to win produces highly persuasive but frequently dishonest debaters that reduce judge accuracy below the inference-only baseline — would be equally important and would substantially weaken the case for debate as an alignment strategy.

Measuring whether debate's benefit on non-extractive tasks depends on providing judges with verifiable evidence, analogous to the passage verification tool for extractive tasks. The paper's extractive-task results are confounded by the passage verification mechanism: judges receive perfectly verified or falsified factual claims about the article, which gives them a reliable signal independent of argument quality. The paper itself cites Khan et al. (2024)'s finding that "the quote verification, rather than the argumentation, is the dominant contribution to judge accuracy" on QuALITY (Appendix F.1), but does not test whether providing analogous verification on closed tasks would close the gap between debate and direct QA.

A targeted experiment would take the GSM8KQA task (where answer choices are executable Python code snippets) and provide judges with code execution results — a verification mechanism that, like passage verification, provides ground-truth feedback about a specific type of claim (whether the code produces the claimed output). The experiment would compare three conditions: (1) debate without code execution (current closed-task setup), (2) debate with code execution results shown to the judge (analogous to passage verification), and (3) direct QA with code execution (judge sees code output without debate). If debate benefits significantly from code execution verification, this would suggest that debate's value is partly mediating between raw capability (which generates claims) and verification tools (which check claims) — and that the path to making debate work on reasoning tasks is to develop verification tools for those domains, not to improve debate's argumentation structure. If debate does not benefit from code execution verification (the adversarial structure is doing the work), that would strengthen the case that debate's mechanism is genuinely about exposing flaws through argumentation rather than about providing verified evidence. The paper's existing GSM8KQA construction (Appendix F.2) — where they already have executable correct and incorrect code solutions — makes this experiment immediately feasible without new data generation.

Characterizing the "sweet spot" of judge capability where debate provides a benefit over direct QA, using a dense sweep of judge models rather than the four-point discretization in this paper. The paper's most puzzling result is that on closed tasks, debate helps GPT-3.5 (p=0.037) but not Gemma7B (worse than direct QA), not Pro 1.0 (not significant), and not Pro 1.5 (worse than direct QA). This suggests a narrow capability band where debate is beneficial, but with only four judge models, the band's boundaries are completely uncharacterized — we don't know where it starts, where it ends, or whether it's contiguous.

A follow-up study would evaluate debate vs. direct QA on the closed-task suite using a dense sweep of judge models spanning the capability range from Gemma7B to Pro 1.5 — for example, Gemma2B, Gemma7B, Gemma-2-9B, Gemma-2-27B, GPT-3.5, Gemini Pro 1.0, and any intermediate-capability open-source models available at the time. The goal is to map the function judge_accuracy_debate(judge_capability) - judge_accuracy_direct_QA(judge_capability) as a continuous curve and identify its maximum, its zero-crossings, and whether it is unimodal. If the curve is reliably positive across a wide capability range (say, 30th-70th percentile of the relevant capability distribution), the paper's "mixed" characterization understates debate's robustness. If the curve is positive only in a narrow spike near GPT-3.5-level capability, the practical case for debate is much weaker — it would help only for a specific judge capability that may not correspond to any realistic deployment scenario. The paper's existing infrastructure (prompt templates, task preprocessing, evaluation pipeline) makes this sweep straightforward to implement, and the cost (even at ~5M calls) is comparable to the paper's own budget.

Human judge replication on a subset of tasks, using the paper's exact protocols and transcripts, to validate whether LLM judges are a reasonable proxy for human judgment in debate evaluation. The paper's entire empirical contribution is contingent on LLM judges being informative about human judge behavior, but this assumption is untested. The paper's findings about positional bias (Figure A.8), the counterproductivity of chain-of-thought for judges (Figure A.5), and the possible self-preference bias in Elo ratings (Section 4.3) all suggest ways in which LLM judges may differ systematically from humans.

A validation study would recruit human participants (e.g., via crowdsourcing platforms) to judge a random subset of debate and consultancy transcripts from the paper's existing data — perhaps 100 transcripts spanning extractive, closed, and multimodal tasks, with Pro 1.5 debaters and Pro 1.5, GPT-3.5, and Gemma7B as the LLM judge conditions to be validated. Humans would see the same processed transcripts (with thinking sections removed, passages verified/unverified) that the LLM judges saw, and would select which answer they believe is correct. The key comparisons are: (1) correlation between human accuracy and LLM accuracy across protocols and task types, (2) whether humans show the same debate-vs-consultancy advantage as LLM judges, (3) whether humans benefit from debate on closed tasks in the same judge-capability range that LLM judges do, and (4) whether humans show the same error-amplification pattern in open-role protocols (trusting incorrect consultants, being rescued by debate's adversarial structure). If human judges show qualitatively different patterns — for example, if humans benefit from debate on closed tasks across a wider capability range than LLM judges, or if humans are less susceptible to positional bias — then the paper's LLM-judge results would need to be reinterpreted as measuring a specific type of artificial oversight rather than standing in for human oversight. The paper already has all the transcripts; the only additional cost is human annotation, which makes this follow-up unusually cheap relative to its diagnostic value.

Adversarial stress-testing of debate with models fine-tuned to be maximally persuasive while arguing for incorrect answers, to test whether debate's open-role error-amplification advantage is robust to optimization pressure. The paper's inference-only debaters are RLHF-trained for honesty, which the paper acknowledges "may hamper the dishonest consultant/debater" (Section 5, Limitations). The open-role results showing that debate reduces error amplification compared to consultancy (Figure 3, bottom row) may therefore be artificially favorable to debate — the dishonest debater is pulling its punches because RLHF training penalizes dishonest argumentation.

A stress-test would fine-tune a debater model (e.g., Pro 1.5) using supervised fine-tuning on examples of successful incorrect-side arguments — specifically, arguments that convinced a judge to select the incorrect answer in the paper's existing debate data. These "adversarially trained" debaters would then be deployed in open debate against honest debaters (either prompted or trained to argue truthfully), and the open-role error-amplification pattern would be remeasured. The key question is whether the light-red bars in Figure 3 (bottom row) — judge accuracy when the protagonist chooses incorrectly — remain elevated relative to open consultancy, or whether adversarially trained dishonest debaters can overcome the antagonist's challenges and drive judge accuracy back down to consultancy-like levels. If debate's error-amplification advantage survives adversarial training, that is strong evidence for debate's robustness as a training protocol. If it collapses — if a debater optimized to win can consistently fool the judge even when arguing for false claims — then the inference-only results are a fragile artifact of RLHF training and do not reflect what would happen under the optimization pressure of actual debate training. This experiment directly addresses the gap between inference-only and training-based evaluation that the paper identifies as its central limitation.

Measuring whether debate helps judges calibrate their confidence — not just whether they make correct binary decisions — and whether debate reduces overconfidence on hard questions. The paper measures only binary accuracy (did the judge select the correct answer?), but in a training context, the magnitude of the reward signal matters as much as its direction. A judge that is correct 70% of the time but overconfident (assigns high probability to wrong answers) may produce a noisier training signal than a judge that is correct 65% of the time but well-calibrated (assigns appropriate uncertainty to wrong answers).

A follow-up would modify the judge prompts to elicit probabilistic judgments (e.g., "What is the probability that answer 1 is correct? Provide a number between 0 and 100.") and compute calibration curves (binned accuracy vs. reported confidence) and Brier scores for each protocol and judge model. The hypothesis to test is that debate improves calibration more than it improves accuracy — the adversarial structure may not help the judge select the right answer more often, but it may help the judge recognize when they are uncertain, because the presence of two opposing arguments makes the difficulty of the question more salient. If true, this would provide a different argument for debate: even when it doesn't improve accuracy, it may improve the quality of the training signal by reducing overconfident errors. The paper's existing judge-output format ("Answer: <1|2>") would need minimal modification to support probabilistic elicitation, and the additional analysis is purely computational.

Practical Applications and Downstream Use Cases

Selecting scalable oversight protocols for RLHF data collection pipelines. Organizations that currently use human feedback to train reward models face a practical decision when models begin to exceed human evaluator capabilities on certain tasks. This paper provides concrete guidance for that decision. For tasks where the model's advantage is primarily about information access — the model has read and synthesized more source material than a human evaluator can review — the extractive-task results suggest that debate with a verification mechanism (e.g., requiring the model to cite specific sources that can be checked) improves judge accuracy over having the evaluator assess outputs without source access. The paper's data shows that on QuALITY with GPT-3.5 as a proxy for a capable-but-not-expert human, debate achieves approximately 78% accuracy vs. approximately 62% for consultancy and approximately 68% for unaided judgment (Figure B.1, approximate values). For tasks where the model's advantage is about reasoning or knowledge — the model produces arguments that a human cannot independently verify because they require expertise the human lacks — the paper's closed-task results suggest that debate provides only a small benefit at best, and only if the human evaluator is in the right capability band. Organizations deploying RLHF for reasoning-heavy domains (mathematics, code generation, scientific analysis) should therefore not assume that adding a debate step will substantially improve feedback quality — the paper's results suggest investment in verification tools (executable code checks, formal proof assistants, fact-checking databases) may be more cost-effective than investment in debate infrastructure.

Designing evaluation pipelines for language model outputs in high-stakes domains where expert human review is expensive or scarce. In domains like medical diagnosis, legal analysis, or scientific peer review, expert human evaluators are a bottleneck — there are not enough domain experts to review every model output. A common workaround is to use non-expert reviewers or lighter-weight evaluation protocols. The paper's finding that debate helps judges of intermediate capability but not the weakest judges (Gemma7B does worse with debate than direct QA on closed tasks, Figure 2) suggests a capability threshold for deploying debate-based evaluation. If the available reviewers fall below this threshold (e.g., crowd workers with no domain expertise, or very weak LLMs used as automated evaluators), debate may be counterproductive — the debate format adds complexity without compensating information, and the reviewers would do better simply answering the question themselves. The paper does not identify the threshold precisely (it lies somewhere between Gemma7B and GPT-3.5 capability), but the pattern is clear enough to recommend that organizations pilot-test debate against direct QA on their specific reviewer population before committing to debate-based evaluation workflows. For organizations with access to mid-tier reviewers (graduate students, domain-adjacent professionals, GPT-3.5-level LLM judges), the paper's results suggest debate is worth testing, particularly if the domain involves information synthesis where a verification mechanism can be added to check factual claims against source documents (extractive-task analog). The 4× cost differential between debate and direct QA (Section 6 discussion of budget normalization) means that the accuracy gain must be weighed against the per-evaluation cost increase — if debate improves accuracy from 66% to 70% but costs 24× more, the cost per correct decision may actually increase.

Providing evidence for the design of "AI constitution" or "AI oversight committee" setups where multiple models review each other's outputs. Several proposed alignment approaches involve using AI systems to oversee other AI systems — recursive reward modeling (Leike et al., 2018), constitutional AI (where a model critiques and revises its own outputs according to a set of principles), or oversight committees where multiple models vote on the acceptability of a proposed action. The paper's comparison of debate (two models in adversarial roles) to consultancy (one model in an advisory role) provides direct evidence about the relative effectiveness of these structures. The finding that debate consistently outperforms consultancy in assigned-role settings (Figure 2) argues for adversarial rather than advisory oversight structures: having a second model tasked with finding flaws is more informative to the judge than having a single model explain its reasoning. But the finding that debate is only marginally better than direct QA on reasoning tasks (Figure 2, closed and multimodal panels) argues that adversarial oversight alone may not suffice — the oversight committee still needs independent verification capabilities (analogous to the passage verification tool) to achieve substantial improvements over unaided judgment. The open-role results provide a more specific recommendation: if the oversight structure must handle cases where one AI agent has an incorrect belief (which is the case that matters for safety), debate's reduced error amplification compared to consultancy (Figure 3, bottom row) makes it the preferred structure, even if average-case accuracy is similar. For a constitutional AI setup where a model critiques its own outputs, the paper's consultancy results might suggest that self-critique (a single model evaluating itself, analogous to consultancy where the model argues for an answer and the judge evaluates) will struggle to detect errors when the model is confident but wrong — exactly the failure mode the open-role consultancy results document (Figure 3, light blue bars). Adding a separate "red team" model in an adversarial debate structure would, based on the paper's results, provide better error detection in those cases.

When to Prefer This Method

The paper does not propose a single method to prefer — it evaluates six protocols (QA without article, QA with article, consultancy, debate, open consultancy, open debate) across three task types and four judge models. The paper's own conclusions (Section 5, Summary) articulate a conditional preference that can be summarized as:

Prefer debate over consultancy for scalable oversight evaluation, regardless of task type or judge capability. The assigned-role results (Figure 2) show debate achieves higher judge accuracy than consultancy across all task types and all judge models tested, and the open-role results (Figure 3) show debate is substantially less likely to amplify incorrect agent beliefs. This preference is the paper's most robust recommendation and does not depend on whether the task involves information asymmetry.

Prefer direct QA with article access over debate when the judge can be given direct access to the information source. On extractive tasks, QA with article substantially outperforms debate for all judge models (Figure 2, left panel) — "information is best used directly, even by a weak model, rather than filtered through oversight" (Section 4.1). This means that if the scalable oversight bottleneck can be solved by giving judges better tools rather than better argumentation protocols, the tools-first approach dominates.

Prefer debate over direct QA without article when the task involves information asymmetry (extractive tasks) and the judge is not too weak. The paper shows this for GPT-3.5, Pro 1.0, and Pro 1.5 judges (Figure 2, left panel). When the judge is very weak (Gemma7B-level), direct QA may be equally good or better because the debate format is too complex for the judge to extract signal from.

Be agnostic between debate and direct QA on reasoning and knowledge tasks without information asymmetry. The paper's results on closed and multimodal tasks are mixed — debate helps for some judge models, hurts for others, and the differences are small in absolute terms (Figure 2, center and right panels). The paper does not provide a decision rule for predicting which judge models will benefit, and the statistical evidence is insufficient to recommend debate unconditionally on these tasks.

Prefer open debate over open consultancy when the protocol will be used to generate training signals, because debate provides a less error-amplifying feedback mechanism. This is a safety-motivated preference, not an accuracy-motivated one — open consultancy may achieve higher average accuracy when the agent is usually correct (Figure 3, top row), but open debate is much better at detecting cases where the agent is wrong (Figure 3, bottom row). For alignment training where the cost of reinforcing errors is high, this tradeoff favors debate.

These preferences are explicitly conditioned on the paper's inference-only setting. The paper does not claim they would hold under training, and it identifies training-based evaluation as the necessary next step to validate or revise them.