Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models

🎯 Pitch

A panel of smaller, diverse LLM judges beats a single giant like GPT-4 at evaluating free-form answers—achieving higher human correlation, lower bias toward their own model families, and over 7× cost reduction. This jury approach works for factoid QA and preference ranking but may not transfer to reasoning or math benchmarks.

1. Executive Summary

This paper proposes evaluating LLM generations using a Panel of LLm evaluators (PoLL) — a jury of smaller, heterogeneous judge models drawn from disjoint model families — rather than relying on a single large judge like GPT-4. Across three judge settings (single-hop QA, multi-hop QA, and Chatbot Arena) spanning six datasets, a PoLL composed of Command R, Haiku, and GPT-3.5 correlates better with human judgments than GPT-4 alone while being over seven times less expensive, and exhibits reduced intra-model scoring bias — establishing that a diverse panel of weaker models can outperform a monolithic strong evaluator, but only in the evaluation settings tested (factoid QA and preference ranking), with applicability to reasoning or math benchmarks remaining an open question.

2. Context and Motivation

The Core Problem: Evaluating Free-Form LLM Outputs Is Fundamentally Broken

The paper addresses a problem that sits at the heart of modern NLP research and deployment: we lack reliable, scalable ways to evaluate what LLMs produce. As language models have become more capable, they increasingly generate free-form text — long-form answers, explanations, multi-turn dialogues, creative writing — for which traditional automatic metrics are profoundly inadequate. The authors frame this as a two-part evaluation crisis (Section 1): not only is it difficult to find meaningful data to probe specific model properties, but evaluating the correctness or quality of a generated response is itself a difficult, unsolved challenge.

This matters for three concrete reasons the paper implicitly raises:

Research progress is gated on measurement. If we cannot accurately assess whether model A outperforms model B on open-ended generation tasks, we lack the scientific instrument to determine which architectural innovations, training recipes, or scaling strategies actually improve models. The paper's Figure 1 (top) dramatizes this: rankings of the same set of models change drastically depending on which LLM serves as the judge. A researcher relying on GPT-4 as judge might conclude one model is best, while a researcher using GPT-3.5 might reach the opposite conclusion — meaning that the field could be optimizing models against measurement noise rather than genuine capability improvements.
Deployment decisions carry real consequences. Organizations selecting which model to integrate into production systems (customer support, medical QA, legal document analysis) need accurate evaluations of output quality to make informed procurement decisions. A biased evaluator that systematically overrates its own model family (intra-model bias, documented in Section 4.4 and prior work by Panickssery et al., 2024) can steer adoption toward models that perform worse for end users.
The dominant evaluation paradigm is expensive and centralized. GPT-4 is the default judge for a large fraction of LLM evaluation research (Zheng et al., 2024; Li et al., 2024a; and many others). This creates a practical barrier: GPT-4 API access is costly ( $10/input +$ 30/output per million tokens, per Section 4.5), and its availability, pricing, and behavior are controlled by a single organization. A cheaper, open alternative that matches or exceeds GPT-4's judging quality would democratize LLM evaluation and reduce dependency on a single proprietary system.

Where Prior Approaches Fall Short

The paper identifies a progression of evaluation methods, each with specific documented limitations that motivate the transition toward model-based judging — and, ultimately, toward panel-based evaluation.

String-matching metrics (BLEU, ROUGE, Exact Match, F1) fail on open-ended generation. These metrics — originally developed for machine translation (BLEU; Papineni et al., 2002), summarization (ROUGE; Lin, 2004), and extractive QA (EM and token-level F1; Rajpurkar et al., 2016) — remain widely used because they are cheap, deterministic, and reproducible. However, the paper highlights specific failure modes (Section 1):

False negatives from incomplete reference sets. QA datasets typically contain a small number of human-written reference answers ("gold answers"), but a language model's free-form generation might express the correct answer in a semantically equivalent yet lexically different way — different date formats, inclusion of a middle initial in a person's name, British versus American spelling. The authors note a concrete example: "EM unjustly penalized Command models for a tendency to write in Canadian or British English as QA dataset annotations typically format dates in American MM-DD-YYYY format." A model that writes "7 March, 1797" is marked wrong when the reference says "March 7, 1797," even though the answer is factually correct.
False positives from surface-level token overlap. A generated answer that contains many words from the reference but is semantically wrong can receive a high ROUGE or F1 score, while a concise, accurate answer using different vocabulary scores poorly. These metrics measure lexical similarity, not semantic correctness, and the gap between the two widens as model outputs become more verbose and stylistically varied.

Trained evaluation models (e.g., BLEURT) are task-specific and data-hungry. BLEURT (Sellam et al., 2020) and similar learned metrics address some weaknesses of heuristic metrics by training neural models to predict human quality judgments. However, they require substantial training data of human annotations for each task and domain, limiting their generalizability. The paper cites this lineage but does not invest heavily in criticizing it, as the field has largely moved toward prompted LLM evaluators for their greater flexibility and zero-shot capability.

Single-model LLM judges introduce intra-model bias. The paper's most thoroughly documented criticism of the status quo concerns bias. Multiple prior works (Zheng et al., 2024; Panickssery et al., 2024) have demonstrated that when an LLM is used as a judge — particularly when evaluating outputs from its own model family — it tends to recognize and prefer its own generations over those of other models. The paper replicates this finding in its own experiments (Section 4.4, Figures 3 and 4): "the highest positive delta for each individual model being scored occurs when it is judged by itself." On Chatbot Arena Hard (Figure 2), a GPT-4 judge ranks another GPT-4 variant at position 2 when its actual position (per crowd-sourced human ELO rankings) is position 4. This isn't just a fairness concern — it systematically distorts model comparisons, inflating the apparent performance of whichever model family the judge belongs to.

Beyond bias, single-model judges exhibit high variance with minor prompt changes (Section 4.3). The paper shows that GPT-4's agreement with human judgments on Natural Questions varies substantially depending on prompt formatting: zero-shot (lowest), few-shot standard, few-shot with the instruction moved to a system prompt, few-shot with chat-formatted shots, and few-shot with an explicit "don't overthink" instruction (highest). This sensitivity means that GPT-4 is not a stable measurement instrument — two researchers using slightly different prompts may obtain different rankings of the same models, making results difficult to reproduce and compare across studies.

Single-model judges are expensive and slow. GPT-4's pricing makes large-scale evaluation costly, particularly for research groups with limited budgets or for use cases requiring evaluation of many model outputs (e.g., running evaluations across many model checkpoints during development, or evaluating on large test sets). The paper quantifies this: at the time of writing, GPT-4 Turbo costs $10/input +$ 30/output per million tokens, while the entire three-model PoLL (Command R + Haiku + GPT-3.5) costs $1.25/input +$ 4.25/output — a seven to eight-fold cost reduction (Section 4.5). For a research group evaluating thousands of model outputs, this difference translates to hundreds or thousands of dollars saved per experiment.

The gap in prior panel-based approaches. The paper acknowledges a direct intellectual precursor: Li et al. (2023) proposed Peer Rank and Discussion (PRD), an evaluation method using multiple judges to reduce bias. However, that work only examined large models in pair-wise evaluation settings. The current paper's contribution is to demonstrate that a panel of smaller models can be effective, and to extend the panel approach across three distinct evaluation settings (single-point, reference-based, and pair-wise scoring) with rigorous human correlation validation.

How This Paper Positions Itself

The paper situates itself not as proposing a fundamentally new evaluation paradigm, but rather as challenging a specific, increasingly entrenched assumption in the field: that the best judge is the largest, most capable model available. This assumption is so widespread that "GPT-4 as judge" has become a de facto standard, with relatively little scrutiny of whether a single large evaluator is truly optimal. The paper argues that this assumption is incorrect for two interrelated reasons.

First, the optimal judge is not a single model — it's a diverse panel. The paper draws a parallel to human annotation practice (citing Voorhees, 1998): pooling judgments across multiple annotators reduces variance, normalizes individual biases, and yields more reliable assessments than any single annotator. The same logic, the paper argues, should apply to LLM judges. The mechanism is straightforward: each judge model has idiosyncratic biases (preferences for certain answering styles, sensitivities to particular lexical patterns, systematic errors on certain question types). By drawing judges from disjoint model families — explicitly, Command R (Cohere), Haiku (Anthropic), and GPT-3.5 (OpenAI) — the panel's biases are decorrelated. An error or bias that causes one judge to score incorrectly is unlikely to be shared by judges from different training distributions with different architectures and training data.

Second, the large-model-as-judge assumption ignores a cost-quality Pareto frontier. The paper's headline result is that a PoLL of three smaller models (GPT-3.5, not GPT-4; Haiku, not Sonnet or Opus; Command R) outperforms GPT-4 while being over seven times cheaper. This is a concrete challenge to the "bigger is better" assumption: on the specific tasks studied, the diversity gain from a panel outweighs the individual capability gap between the panel members and GPT-4. The paper is careful not to claim this holds universally — Section 5 explicitly restricts the claim to factoid QA and preference ranking, flagging math and reasoning evaluations as unverified — but the existence of this cost-quality frontier is the paper's central conceptual contribution.

The paper's position is empirical, not theoretical. It does not derive conditions under which panel diversity dominates single-judge capability. Instead, it conducts a systematic comparison across three evaluation settings and six datasets, measuring correlation with human judgments (Cohen's $\kappa$ and rank correlation coefficients) as the ground-truth signal. The paper treats human evaluators as the gold standard throughout, and the claim that PoLL is "better" is specifically the claim that PoLL agrees more closely with human judgments than GPT-4 does — not that PoLL is intrinsically more correct in some abstract sense. This framing is important: the paper sidesteps the philosophical question of what constitutes a "correct" evaluation of free-form text by grounding everything in human inter-annotator agreement.

The paper's relationship to the LLM-as-judge literature is complementary, not adversarial. It builds directly on the frameworks established by Zheng et al. (2024) (MT-Bench and Chatbot Arena, including the Arena Hard subset used in Section 3.5) and adopts their methodology for head-to-head model comparison. It also builds on Panickssery et al. (2024)'s documentation of intra-model bias. The contribution is not to argue that LLM-based evaluation is flawed — the paper explicitly endorses it as the best available approach — but rather to argue that the implementation of LLM-based evaluation (single large judge vs. diverse panel) can be dramatically improved along multiple dimensions simultaneously: accuracy, bias reduction, and cost.

3. Technical Approach

3.1 Reader Orientation

The system being built is a procedure for scoring the quality of LLM-generated text that replaces a single large judge model with a panel of smaller, diverse judge models whose individual judgments are aggregated into a consensus score. It solves the problem that single-model evaluation (especially GPT-4) introduces systematic bias, high cost, and sensitivity to prompt phrasing, all of which distort the measurement of which LLMs actually produce better outputs. The shape of the solution is deceptively simple: instead of asking one powerful model "is this answer correct?", ask multiple weaker models from different model families, and combine their answers through a voting function—the diversity across judges decorrelates their errors, and the aggregation washes out individual biases, yielding scores that correlate more strongly with human judgments than any single judge alone.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components connected in a linear pipeline:

Test Model — any LLM whose outputs need to be evaluated (e.g., Command R, GPT-4, Mistral Large, Claude 3 Sonnet). It receives a prompt (a question, optionally plus retrieved evidence documents, optionally plus a comparison output) and produces a generated answer. This model is the subject of evaluation, not part of the evaluation machinery itself.
Panel of Judge Models (PoLL) — a fixed set of three LLM evaluators drawn from three disjoint model families: Command R (Cohere), Claude 3 Haiku (Anthropic), and GPT-3.5 (OpenAI). Each judge independently receives the same scoring prompt (containing the question, the generated answer, and depending on the evaluation setting, a reference answer or a comparison answer) and produces an individual judgment. The judges operate independently—they do not see each other's outputs or communicate.
Voting/Aggregation Function — a deterministic rule that pools the three individual judgments into a single PoLL score. The function is task-dependent: for binary correctness judgments in QA, it uses max voting (the answer is considered correct if any judge says so); for graded preference judgments in Chatbot Arena, it uses average pooling (the three scalar scores are averaged into one continuous score).
Correlation/Evaluation Backend — the PoLL scores are compared against ground-truth human judgments (from professional annotators for QA tasks; from crowd-sourced Chatbot Arena ELO rankings for the Arena task) using correlation metrics (Cohen's $\kappa$ for binary judgments; Kendall's $\tau$ and Pearson correlation for rankings).

Information flows strictly forward: a prompt enters the test model → the generated answer enters each of the three judges independently → the individual judgments enter the voting function → the aggregated score is compared against human ground truth to measure the quality of the evaluation. The PoLL itself is static: the panel composition does not adapt per-question or per-test-model.

3.3 Roadmap for the Deep Dive

First, the three evaluation settings (single-point, reference-based, pair-wise scoring) and their formal interfaces, since the PoLL is a drop-in replacement for any single judge and understanding the scoring contract is prerequisite to everything else.
Second, the panel composition logic—which models are selected, why those specific model families, and why disjoint families matters for bias decorrelation—since panel diversity is the central mechanism underlying the method.
Third, the voting/aggregation functions (max voting for QA, average pooling for Arena) and their operational semantics, since aggregation is what converts three individual judgments into a single evaluation signal and the choice of function materially affects the behavior.
Fourth, the judge prompting infrastructure—the prompt templates, few-shot example selection, and prompt sensitivity experiments—since the evaluation quality depends critically on how judges are instructed and prompted.
Fifth, the human annotation pipeline that provides ground truth, since all claims about PoLL "outperforming" GPT-4 are specifically claims about stronger correlation with these human judgments, and the annotation methodology therefore determines what is being measured.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical methods paper whose core idea is that for LLM-based evaluation, diversity across multiple smaller judges from disjoint model families provides better correlation with human judgments than any single large judge, while also reducing cost and intra-model bias. The paper proposes no new model architecture or training procedure; instead, it reconfigures existing off-the-shelf LLMs into a panel evaluation framework and rigorously measures the resulting evaluation quality.

The Three Evaluation Settings and Their Formal Interfaces

The paper formalizes three distinct evaluation scenarios (Section 2.1), each with a different scoring interface. The PoLL is designed to be a drop-in replacement for any single judge in any of these settings—it accepts the same inputs and produces the same type of output, just with higher reliability. Understanding these interfaces is essential because they determine what information the judges see and what form their judgments take.

Reference-Based Scoring (used in Sections 3.3 and 3.4 for QA tasks). Given a generated answer $a$ from a test model and a human-written reference answer $r$ (the "gold" answer), the judge $J$ produces a score:

$\text{score} = J(a, r)$

where $a$ is the free-form text produced by the test model, and $r$ is the short reference answer from the dataset annotation.

In the paper's QA experiments, this score is binary: either "True" (the generated answer semantically contains the reference answer) or "False" (it does not). The prompt template (Tables 10–13 in the Appendix) instructs the judge to output exactly the string "True" or "False." The paper explicitly notes that this is not string-matching—judges are instructed that "differently formatted dates, people with missing middle names, and alternative spellings should all be considered the same," making this a semantic containment judgment rather than a lexical overlap test.

Why this form: reference-based scoring gives the judge explicit grounding in what constitutes a correct answer. Unlike single-point scoring (which relies entirely on the judge's internal world knowledge), reference-based scoring reduces the evaluation problem to a comparably simpler sub-task: determining whether the generated answer conveys the same information as the reference, without requiring the judge to independently know the correct answer. This is appropriate for QA tasks where correctness is objective and reference answers exist. An alternative would be to ask the judge to score answer quality on a Likert scale (1–5), but binary True/False is simpler, more reproducible, and maps directly to the accuracy metrics the paper reports (Cohen's $\kappa$ , which operates on categorical agreement).

Pair-wise Scoring (used in Section 3.5 for Chatbot Arena Hard). Given two generated answers $a$ and $b$ (produced by two different test models in response to the same prompt), the judge produces a preference score:

$\text{score} = J(a, b)$

In the Chatbot Arena Hard setting, this score is a value on a five-point scale: $a \gg b$ , $a > b$ , $a \approx b$ , $a < b$ , $a \gg b$ (where $\gg$ indicates a clear win and $>$ indicates a marginal win). To avoid ordering bias—the well-documented tendency of LLMs to prefer whichever answer appears first—the paper follows the standard practice from Zheng et al. (2024) and Li et al. (2024a) of evaluating both $(a, b)$ and $(b, a)$ orderings and combining the results. The paper uses the exact evaluation codebase from the Arena Hard repository (https://github.com/lm-sys/arena-hard) without modifying the judge prompts.

Why this form: pair-wise scoring is the standard for head-to-head model comparison because absolute quality scores are difficult to calibrate across models, while relative preferences are more stable. The five-point ordinal scale captures both the direction of preference (which model wins) and the magnitude (how decisively), enabling finer-grained rankings than binary win/loss. The ordering-bias mitigation (evaluating both orders and combining) is a standard technique in the LLM-as-judge literature and is essential for fair comparison—without it, a judge that systematically favors the first- or second-positioned answer would distort all rankings.

Single-Point Scoring (described but not the focus of experiments). Given only a generated answer $a$ and no reference or comparison point, the judge produces a quality score based solely on its internal model of what constitutes a good output:

$\text{score} = J(a)$

The paper mentions this setting in Section 2.1 (citing Kocmi and Federmann, 2023b on translation quality evaluation) but focuses its experiments on reference-based and pair-wise scoring. This setting is the most demanding because it requires the judge to independently assess quality without any anchoring information. The paper's PoLL framework would apply here by having each panel member produce an independent quality score and then pooling them, but no experiments test this setting.

Why this is mentioned but not tested: single-point scoring introduces the most degrees of freedom—each judge's internal quality standards, factual knowledge, and stylistic preferences all influence the score—making it the setting where intra-model bias is likely strongest and where a diverse panel would theoretically provide the largest benefit. However, it is also the hardest setting to obtain ground-truth human judgments for (humans themselves disagree on open-ended quality), so the paper focuses on the more tractable settings where ground truth is well-defined.

PoLL Composition and the Diversity Principle

The paper constructs its Panel of LLm evaluators from three models drawn from three disjoint model families (Section 3.1). The specific composition is:

Judge Model	Model Family	Developer	Relative Size
Command R	Command R family	Cohere	Medium
Claude 3 Haiku	Claude 3 family	Anthropic	Small (the smallest in the Claude 3 line: Haiku < Sonnet < Opus)
GPT-3.5	GPT family	OpenAI	Medium (predecessor to GPT-4)

The choice of these three specific models is driven by several considerations:

Disjoint model families. This is the central design principle. Each judge comes from a different organization with independent training data, architecture decisions, fine-tuning recipes, and RLHF reward models. The paper's hypothesis is that errors and biases are correlated within a model family (models trained by the same organization share similar inductive biases, training data distributions, and RLHF reward signals) but decorrelated across families. By selecting judges from three different families, the panel maximizes the independence of its error sources. When two judges from different families agree that an answer is correct, that agreement is stronger evidence than agreement between two models from the same family (which might both share the same blind spot).

The paper explicitly does not include multiple models from the same family in the panel (e.g., it uses GPT-3.5 but not GPT-4, Command R but not Command R+, Haiku but not Sonnet or Opus). This is a deliberate choice to avoid correlated biases. Including GPT-4 alongside GPT-3.5 would increase the panel's average capability but reduce its effective diversity, since the two GPT-family models likely share similar evaluation tendencies.

Smaller rather than larger models. The panel deliberately avoids using the largest, most capable model from each family. GPT-3.5 is used instead of GPT-4; Haiku instead of Sonnet or Opus. This is both an economic choice (the three-model PoLL is over seven times cheaper than GPT-4 alone, per Section 4.5) and a conceptual one: the paper aims to demonstrate that a panel of weaker models can outperform a single stronger model, challenging the assumption that judge quality is monotonic in model capability.

No Mistral judge in the panel. The paper explicitly notes (Section 3.2, footnote 6) that it evaluates Mistral Large and Mistral Medium as test models (to have a model "unaffiliated with any judges") but does not include a Mistral model in the PoLL. This is a practical limitation—the panel was fixed at three members—but it also serves an analytical purpose: having a test model that shares no family membership with any judge allows the paper to measure whether the PoLL's bias reduction extends to models outside the judges' families. The results (Section 4.4, Figures 3–4) show that Mistral-LG's scores are not systematically inflated or deflated by the PoLL relative to human ground truth, suggesting the bias reduction generalizes.

The panel size of three. The paper provides no ablation over panel size (e.g., comparing panels of 2, 3, 5, or 7 judges). Three is chosen as a practical number that provides enough diversity for majority or average aggregation while keeping cost manageable. With two judges, a disagreement produces a tie with no resolution; with three, there is always a majority (or, for continuous scores, a central tendency). The paper acknowledges in Section 5 that "panel selection"—identifying the optimal models to include in terms of quality and cost—is left to future work.

Voting and Aggregation Functions

The individual judgments from the three panel members must be combined into a single PoLL score. The paper uses two different aggregation functions depending on the evaluation setting (Section 3.1), and the choice is driven by the type of judgment being aggregated.

Max Voting for QA (binary judgments). In the reference-based QA setting, each judge produces a binary output: "True" (the generated answer semantically contains the reference answer) or "False" (it does not). The PoLL score is computed as:

$\text{PoLL}_{\text{score}}(a, r) = \max_{j \in P}\ j(a, r)$

where $P = \{\text{Command R}, \text{Haiku}, \text{GPT-3.5}\}$ is the panel, $j(a, r) \in \{0, 1\}$ is judge $j$ 's binary judgment, and $\max$ returns 1 if any judge says the answer is correct, and 0 only if all three judges agree it is incorrect.

What it computes: an answer is considered correct by the PoLL if at least one judge considers it correct. This is the most lenient possible aggregation rule—an answer is given the benefit of the doubt, being marked correct if any panel member recognizes the generated answer as containing the reference information.

Why max voting: the paper's rationale is implicit in the QA task structure. In reference-based scoring, the judgment task is to identify whether a semantic relationship holds between the generated answer and the short reference. False negatives (incorrectly marking a correct answer as wrong) are more damaging than false positives (incorrectly marking a wrong answer as correct), because false negatives systematically undercount model capability and are the primary failure mode of string-matching metrics like EM (as the paper extensively argues in Section 1). Max voting is biased toward reducing false negatives: if any judge correctly recognizes the semantic match, the answer is scored as correct. The alternative—majority voting—would require at least two judges to agree, which would increase false negatives whenever two judges miss a valid semantic match. The paper notes (Section 3.1, footnote 4) that for Chatbot Arena, "a three judge panel often does not produce a clear majority decision," which also applies to binary judgments where judges frequently disagree. Max voting resolves all disagreements in favor of the most generous judge.

A subtle implication: max voting makes the PoLL's aggregate behavior more lenient than any individual judge. If Judge A is correct 80% of the time, Judge B 75%, and Judge C 70%, max voting can (in principle) achieve higher recall than any individual judge, at the potential cost of lower precision. The paper does not formally decompose this precision-recall tradeoff, but the empirical human-correlation results (Table 1) show that max-voted PoLL judgments do in fact correlate more strongly with human judgments than any single judge, suggesting the reduction in false negatives outweighs any increase in false positives for these tasks.

Average Pooling for Chatbot Arena (continuous judgments). In the pair-wise preference setting, each judge produces a score on a scale that is mapped to a numeric value (the paper follows the Arena Hard codebase's scoring convention, which translates the five-point preference scale into numeric scores). The PoLL score is computed as:

$\text{PoLL}_{\text{score}}(a, b) = \frac{1}{|P|} \sum_{j \in P} j(a, b)$

where $|P| = 3$ is the panel size and $j(a, b)$ is the numeric score from judge $j$ .

What it computes: the arithmetic mean of the three judges' preference scores. This produces a single continuous value that represents the panel's average preference between the two generated answers.

Why average pooling: unlike the binary QA setting where judgments are coarse (correct/incorrect), Chatbot Arena judgments are graded (clear win, marginal win, tie, marginal loss, clear loss). Averaging these graded preferences preserves more information than a majority vote would—a judge's strong preference (clear win) contributes more to the average than a weak preference (marginal win), capturing the intensity of preferences. The paper notes (Section 3.1, footnote 4) that it experimented with "max voting with average voting fallback which yielded similar results with an identical overall ranking," suggesting the choice of aggregation function is not highly sensitive for the Arena setting, likely because the preference distributions are sufficiently consistent across judges that different aggregation rules produce similar final rankings.

Judge Prompting Infrastructure

The way judges are prompted is critical to evaluation quality, and the paper invests substantial effort in prompt design and prompt sensitivity analysis. The prompting infrastructure differs between the QA and Chatbot Arena settings.

QA Judge Prompts: Few-Shot with Reference-Based Scoring. For the single-hop QA (KILT) and multi-hop QA (Bamboogle, HotPotQA) experiments, each judge receives a structured prompt containing:

A natural language instruction that defines the task: "You will be given a Question and a Provided Answer. Judge whether the Provided Answer is correct by comparing it to the Reference Answer. Differently formatted dates, people with missing middle names, and alternative spellings should all be considered the same. If the Provided Answer is correct say exactly 'True', otherwise say 'False'."
A set of few-shot in-context examples of (question, provided_answer, reference_answer, correct) tuples, where correct is either "True" or "False." These examples are drawn from human-annotated data and are selected through a systematic procedure (Section A.1.1): for each KILT dataset, the authors randomly sample examples subject to constraints—at least one positive and one negative example from each model family's generations, and at least one example where human annotators disagree with the exact match (EM) metric. They sweep over n_shots in {8, 10, 16} and select the prompt that produces the strongest average agreement across all judges on a small held-out portion of human labels.
The test instance to be judged, with the question and generated answer substituted into the template, and the reference answer provided for comparison.

The full prompts are reproduced in Appendix Tables 10 (multihop), 11 (KILT NQ), 12 (KILT HotPotQA), and 13 (KILT TriviaQA). The few-shot examples are critical—the paper's prompt ablation (Section 4.3, Table 3) shows that zero-shot prompts produce substantially lower agreement with human judgments than few-shot prompts for GPT-4. The few-shot examples teach the judge the desired behavior through demonstration: what constitutes semantic containment, how to handle formatting differences, when to say "False" even when the generated answer is verbose and plausible.

Why few-shot with constrained example selection: The constraint that examples must include cases where all model families are represented and where EM disagrees with human judgment is deliberate. If few-shot examples were randomly sampled, they might over-represent easy cases where all judges would agree anyway, failing to teach the model how to handle the boundary cases that actually differentiate judges. By including examples where EM is wrong (false positives or false negatives), the few-shot prompt teaches the judge to go beyond surface-level string matching—the exact behavior that distinguishes LLM judges from EM in the first place.

Chatbot Arena Judge Prompts: Unmodified from Prior Work. For the Arena Hard experiments, the paper uses the exact prompt templates and evaluation code from the original Arena Hard repository (https://github.com/lm-sys/arena-hard), which follows the pair-wise scoring framework of Zheng et al. (2024). The paper does not modify these prompts in any way. This design choice enables direct comparison with the original Arena Hard results (where GPT-4 was used as the single judge) and isolates the effect of replacing the single judge with a PoLL while holding all other factors constant.

GPT-4 Prompt Sensitivity Analysis. Section 4.3 presents a detailed ablation of how GPT-4's judging performance varies with prompt modifications on KILT Natural Questions. The variants tested are:

Zero-shot: The natural language instruction is supplied as a system call, with no few-shot examples. This yields the lowest agreement with human judgments.
Few-shot Standard: The same prompt used for all other judge models (Table 11), with 10 few-shot examples. This is the baseline against which other variants are compared.
No instruction line: The natural language instruction is removed from the few-shot standard prompt. This decreases agreement $(-0.03\ \Delta\kappa)$ , contrary to the hypothesis that the instruction might be confusing the model.
Move instruction line: The instruction is moved from the user message into a separate system call. This yields a small improvement $(+0.01\ \Delta\kappa)$ .
Chat-formatted shots: Each few-shot example is formatted as a conversational turn between user and assistant. This reduces performance $(-0.07\ \Delta\kappa)$ , suggesting that the flat prompt format is more effective for this task than a dialogue format.
"Don't overthink" prompt: The instruction line is replaced with a wording that explicitly tells the model to avoid over-reasoning: "Don't worry about factuality with respect to the real world, just judge the example based on what you see. No need to overthink this task, it really comes down to just soft matching." This yields the largest improvement for GPT-4 $(+0.07\ \Delta\kappa)$ . Additional small surface-level changes and moving the instruction to a system call add another $+0.03\ \Delta\kappa$ . The final optimized prompt is in Appendix Table 14.

The optimized prompt brings GPT-4's agreement up to the level of GPT-3.5 with the standard few-shot prompt, though still below Command R, Haiku, and PoLL. Crucially, the "don't overthink" prompt has different effects on different models (Section A.1.1, footnote 10): it has a negligible positive effect on Haiku, a minor negative effect on Command R, and a severe negative impact on GPT-3.5. This heterogeneity is itself evidence for the paper's central argument—prompt sensitivity is model-specific, and a single optimization (tuning prompts for GPT-4) can degrade other judges, whereas a diverse panel is more robust to suboptimal prompting for any individual member.

Why prompt sensitivity matters for the PoLL argument: The paper's finding that GPT-4 requires specific prompt engineering (a "don't overthink" instruction) to match the performance of smaller models with standard prompts undermines the convenience of the "just use GPT-4" paradigm. A researcher who uses GPT-4 with a suboptimal prompt (e.g., zero-shot, or standard few-shot without the "don't overthink" qualifier) will obtain worse evaluations than a PoLL using standard prompts, not because GPT-4 is incapable of the task, but because its behavior is more sensitive to prompt framing. PoLL's relative robustness to prompt variation across its diverse panel members is an understated practical advantage.

Human Annotation Pipeline for Ground Truth

All claims about PoLL "outperforming" GPT-4 are specifically claims about stronger correlation with human judgments. The quality of the human annotation pipeline therefore determines what the evaluation metrics actually measure. Section 3.7 and Appendix A.3 describe this pipeline.

Annotation task. Annotators are shown a single anonymized model-generated answer, along with the original question and a short reference answer. They are asked to judge whether the reference answer is "semantically contained" inside the generated answer. The authors explicitly state (Section 3.7, footnote 9): "By asking for answer correctness alone, we hope to minimize any potential annotator bias or preference towards a particular model's style." This is a deliberate design choice—annotators are not asked which answer they "prefer" or to rate "quality," but simply to verify whether the factual information in the reference appears somewhere in the generation. This reduces the influence of stylistic factors (verbosity, politeness, formatting) on the human judgment, making the annotation closer to an objective correctness check.

Annotator workforce. The paper uses "Cohere's internal highly-qualified annotation workforce," described as "professional annotators with diverse demographic and professional backgrounds including novelists, copywriters, copy editors, and journalists" (Appendix A.3). The use of professional annotators rather than crowd-sourced workers (e.g., Amazon Mechanical Turk) is significant: professional annotators are typically more consistent, have higher inter-annotator agreement, and produce fewer noisy or adversarial labels. This raises the quality floor for the ground truth but may also limit the generalizability of the findings—professionally annotated QA judgments might be easier for LLM judges to match than more subjective or noisy crowd judgments.

Annotation redundancy. Different datasets received different annotation protocols (Appendix A.3, Table 8):

Triple-annotated with majority voting: Natural Questions (NQ) and single-hop HotPotQA. Each instance is judged by three independent annotators, and the final ground-truth label is determined by majority vote. If two annotators say "correct" and one says "incorrect," the label is "correct." This protocol increases label reliability by averaging out individual annotator errors and idiosyncrasies.
Singly annotated: TriviaQA, Bamboogle, and multi-hop HotPotQA. Each instance is judged by a single annotator. This is less reliable (no redundancy to catch errors) but more cost-effective for larger datasets. The paper does not provide inter-annotator agreement statistics for the triple-annotated datasets, which would quantify the difficulty of the annotation task and the ceiling for LLM judge performance (no automated judge can exceed human inter-annotator agreement, since the human labels are the ground truth).

Scale. The total number of annotations is reported in Appendix Table 8, though the paper text does not quote the exact figures. The magnitude spans thousands of individual judgments across the six datasets, providing sufficient statistical power for the correlation analyses in Section 4.

How Information Flows Through the PoLL System End-to-End

To make the complete pipeline concrete, here is the flow for a single KILT Natural Questions instance being evaluated by the PoLL:

Test model generation: A test model receives a question (e.g., "when did the nfl start playing in london") and a set of retrieved Wikipedia documents. It produces a free-form answer (e.g., "According to the provided search results, the NFL started playing regular season games in London as part of the NFL International Series in 2007...").
Human annotation (ground truth): A professional annotator sees the question, the generated answer, and the reference answer ("since 2007"). They judge whether the reference is semantically contained in the generation. If yes, the ground-truth label is "True." This step happens offline and is not part of the PoLL inference pipeline—it only serves to measure how well the PoLL correlates with humans.
PoLL inference (evaluation): The question, generated answer, and reference answer are formatted into the few-shot judge prompt (Table 11). This identical prompt is sent to each of the three judge models—Command R, Haiku, and GPT-3.5—independently and in parallel. Each judge returns either "True" or "False."
Aggregation: The max-voting function is applied. If any of the three judges returned "True," the PoLL score is "True." If all three returned "False," the PoLL score is "False."
Correlation computation: Across all instances in the dataset, the PoLL's binary judgments are compared against the human ground-truth labels. Cohen's $\kappa$ is computed as:

$\kappa = \frac{p_o - p_e}{1 - p_e}$

where $p_o$ is the observed agreement proportion between PoLL and human judgments, and $p_e$ is the expected agreement proportion by chance (computed from the marginal frequencies of "True" and "False" labels in each set of judgments).

This $\kappa$ value is what appears in Table 1, and it is the metric on which PoLL is compared against individual judges (GPT-4, GPT-3.5, Command R, Haiku) and against EM.

Why This Approach Over Alternatives

The paper's design choices can be understood as optimizing a multi-objective function over evaluation quality, cost, bias reduction, and robustness. Several alternative approaches were explicitly or implicitly rejected:

Why a panel rather than a single stronger model? The single-model alternative (GPT-4) exhibits three documented failure modes: intra-model bias (Sections 4.4, Figures 3–4), high prompt sensitivity (Section 4.3, Table 3), and high cost (Section 4.5). A single stronger model might have higher average capability but still concentrates all bias in one system—if that system has a blind spot (e.g., systematically underrating verbose answers, or overrating answers that match its own generation style), that blind spot affects every judgment. A panel decorrelates these errors: a blind spot in one judge is unlikely to be shared by judges from different training distributions. The paper's empirical claim is that this decorrelation benefit outweighs the per-judge capability gap relative to GPT-4.

Why smaller rather than larger panel members? The paper could have constructed a panel of three "best" models (e.g., GPT-4 + Claude 3 Opus + Command R+), which would likely outperform the smaller panel. The choice of smaller models is partly pragmatic (cost—the paper's headline includes "over seven times less expensive") and partly conceptual: the paper aims to demonstrate that diversity compensates for individual weakness, which is a stronger claim than "a panel of the best models is better than any single one." If a panel of weaker models can beat a single strong model, the diversity benefit is robust; if only a panel of the strongest models can beat a single strong model, diversity is a marginal improvement over capability scaling.

Why panel size of exactly three? The paper does not formally justify three, but the logic is implicit. Two judges provide no tie-breaking mechanism for binary decisions. Three provides a natural majority for both max voting (any "True" → PoLL "True") and majority voting (at least two "True" → PoLL "True"), and allows average pooling for continuous scores with a meaningful central tendency. Beyond three, diminishing returns likely set in: additional judges add cost proportionally but provide less incremental diversity benefit, and the paper explicitly leaves optimal panel selection to future work (Section 5).

Why max voting for QA rather than majority voting? The paper tested both (Section 3.1, footnote 4 notes that for Chatbot Arena they "experimented with max voting with average voting fallback which yielded similar results with an identical overall ranking"). For QA, max voting is chosen without explicit comparison to alternatives. The implicit rationale is that false negatives are the primary concern in QA evaluation (as argued in Section 1 regarding EM's systematic false negatives), and max voting minimizes false negatives by design. The paper does not report precision and recall separately for different aggregation functions, which would directly test this rationale—this is a minor methodological gap.

Why average pooling for Chatbot Arena rather than majority voting? Continuous preference scores carry more information than discretized win/loss/tie decisions. Averaging preserves the intensity of preferences (a judge's strong preference influences the average more than a weak preference), whereas majority voting would discard this information. The paper's footnote 4 suggests this choice is not critical—different aggregation functions produce identical final rankings—but average pooling is the default in the Arena Hard codebase the paper uses.

4. Key Insights and Innovations

Innovation 1: Diversity Across Weaker Judges Systematically Outperforms Monolithic Capability — Reframing Evaluation as a Bias-Decorrelation Problem

The paper's deepest conceptual move is not proposing a panel per se (Li et al., 2023 already used multiple judges), but rather reframing the LLM evaluation problem from a capability-maximization problem (find the most capable judge) to a bias-decorrelation problem (find a set of judges whose errors are maximally independent). This is a genuinely non-obvious reframing because the dominant intuition in the field has been that evaluation quality scales with judge capability: GPT-4 is better than GPT-3.5, which is better than smaller open models, so just use the best model you can afford. The paper's empirical demonstration that a panel of weaker models — GPT-3.5 not GPT-4, Haiku not Sonnet or Opus — outperforms a stronger single judge on human-correlation metrics (Tables 1, 2; Cohen's $\kappa$ and rank correlation) constitutes a falsification of that monotonic-capability intuition.

What makes this more than a "ensembling helps" result is the disjoint model families principle. The paper doesn't just throw multiple judges together; it explicitly selects judges from independent model families with distinct training data, architectures, and RLHF reward models — Command R (Cohere), Haiku (Anthropic), GPT-3.5 (OpenAI). This is a deliberate bias-decorrelation strategy analogous to the statistical principle that averaging independent estimators reduces variance proportionally to the number of estimators, but only if their errors are uncorrelated. The paper operationalizes this by avoiding within-family redundancy (no GPT-3.5 + GPT-4, no Haiku + Sonnet) — a design choice that would be counterintuitive under a pure capability-maximization framework but is essential under a bias-decorrelation one.

The significance of this reframing extends beyond the specific PoLL instantiation. It implies that the marginal value of a judge model for panel evaluation depends not only on its individual accuracy, but on the correlation structure of its errors with existing panel members. A highly capable judge that makes the same mistakes as the panel (high error correlation) adds less value than a weaker judge with orthogonal error patterns. This opens a new axis for judge selection — what the paper calls "panel selection" (Section 5) — that is fundamentally different from simply picking the highest-accuracy models. It also provides a principled explanation for why the single-judge paradigm is fragile beyond just cost: a single judge concentrates all systematic bias into one measurement instrument, making evaluation quality brittle to that instrument's specific failure modes (prompt sensitivity, self-preference, stylistic biases), whereas a diverse panel's aggregated judgment is robust to any individual judge's idiosyncrasies so long as those idiosyncrasies are not shared across the panel.

The contrast with prior work is instructive. Li et al. (2023)'s Peer Rank and Discussion (PRD) used multiple judges but only examined large models in pair-wise settings, without the explicit diversity-over-capability tradeoff or the disjoint-families design principle. Panickssery et al. (2024) documented intra-model bias but treated it as a problem to be aware of, not as the central axis around which evaluation should be reorganized. Zheng et al. (2024) established LLM-as-judge as a paradigm using single GPT-4 evaluations. This paper synthesizes these threads into a coherent alternative framework where diversity is the primary design criterion, not a secondary consideration.

This is a fundamental reframing, not an incremental improvement. It changes what a researcher evaluating LLMs should optimize for — from "which judge model is most accurate?" to "which set of judge models have the most independent failure modes?" — and provides the empirical evidence (Table 1, Figure 1 bottom) that this reframing yields better evaluations at lower cost.

Innovation 2: Intra-Model Bias as a Quantifiable, Reproducible Failure Mode That Panel Evaluation Directly Addresses — Not Just Documented But Diagnostically Measured

While prior work (Panickssery et al., 2024; Zheng et al., 2024) had established that LLM judges prefer their own outputs, this paper makes two advances that elevate intra-model bias from an anecdotal concern to a diagnostically measured, systematically quantified failure mode with a direct remediation.

First, the paper provides clean, multi-setting replication of self-preference bias with precise quantification. On Chatbot Arena Hard (Figure 2), the GPT-4 judge ranks a GPT-4 variant at position 2 when its ground-truth position (per crowd-sourced human ELO) is position 4 — a two-position inflation. On multi-hop QA (Figures 3 and 4), the paper shows that "the highest positive delta for each individual model being scored occurs when it is judged by itself" — in other words, every judge model in the study (Command R, Haiku, GPT-3.5) exhibits self-preference when evaluating its own outputs relative to human ground truth. The scatter plots in Figures 3 and 4 visualize this systematically: for each test model, the delta relative to human judgment is plotted against different judges, and the self-judgment point is consistently the highest (most inflated) across all test models. This is not a single-model finding — it's a cross-model regularity that suggests self-preference is a structural property of LLM evaluation, not a quirk of GPT-4.

Second, the paper demonstrates that pooling across a heterogeneous panel directly reduces this bias without requiring explicit bias correction. The PoLL's scores (Figures 3 and 4, the rightmost column in the scatter plots) show the smallest standard deviation relative to human ground truth across all evaluated test models (standard deviation of 2.2, compared to 6.1 for GPT-3.5). The mechanism is straightforward but the demonstration is novel: when one judge inflates its own model's score, the other two judges — from different model families — have no reason to share that inflation, so max voting or average pooling attenuates the bias. The paper doesn't need to identify which judgments are biased or model why self-preference occurs; the panel structure handles it implicitly through decorrelation.

The significance here is that this converts intra-model bias from a "known issue to be aware of" into an actionable, solvable problem. Prior work documented the bias but left the field with no clear remediation beyond "be careful when interpreting GPT-4 evaluations of GPT-family models." This paper shows that the remediation is not more careful interpretation but a structural change to the evaluation protocol — replace the single judge with a diverse panel. The bias reduction emerges automatically from the panel architecture, not from any explicit debiasing procedure.

This is an incremental advance in mechanism (panels averaging out bias is not conceptually new — it's why human annotation uses multiple annotators) but a fundamental advance in evidence and operationalization for the LLM evaluation context. Before this paper, a researcher could reasonably ask: "Is intra-model bias large enough to matter in practice, and if so, what do I do about it?" After this paper, the answer is: "Yes — it systematically inflates self-family scores by multiple rank positions — and the solution is to replace your single judge with a panel of models from at least three disjoint families."

Innovation 3: Prompt Sensitivity as a First-Class Failure Mode That GPT-4 Is Uniquely Vulnerable To — and That PoLL Is Robust Against

The paper's prompt sensitivity experiments (Section 4.3, Table 3) reveal an underappreciated vulnerability in the single-large-judge paradigm: GPT-4's evaluation quality varies substantially with prompt phrasing, while smaller models are more robust to prompt variation — and prompt optimizations that help GPT-4 can actively harm other judges. This finding matters independently because it challenges the assumption that a "more capable" model is necessarily a more reliable measurement instrument. In measurement science, reliability (consistency under minor procedural variations) is as important as accuracy (correlation with ground truth). The paper shows that GPT-4 achieves high accuracy only after careful prompt engineering, and that the prompts that work best for GPT-4 are model-specific.

The evidence is in the prompt ablation (Table 3, Appendix Section A.1.1). GPT-4's Cohen's $\kappa$ on Natural Questions varies dramatically across prompt variants: zero-shot (lowest, exact value not reported but described as weakest), few-shot standard (baseline), few-shot minus instruction line ( $-0.03\ \Delta\kappa$ ), few-shot with chat-formatted examples ( $-0.07\ \Delta\kappa$ ), and few-shot with explicit "don't overthink" instruction ( $+0.07\ \Delta\kappa$ , plus an additional $+0.03\ \Delta\kappa$ for surface-level optimizations, totaling roughly $+0.10\ \Delta\kappa$ improvement over the standard prompt). A $\sim0.10\ \kappa$ swing from prompt engineering alone is substantial — it can change whether GPT-4 appears better or worse than alternative judges — and it means that two research groups using GPT-4 with slightly different prompts could reach materially different conclusions about which models perform best.

The finding that prompt optimizations are not transferable across models is the deeper insight. The paper's footnote 10 (Appendix A.1.1) reports that the "don't overthink" prompt — which substantially improved GPT-4 — had a "negligible positive effect on Haiku, minor negative effect to Command R and a severe negative impact on GPT-3.5." This heterogeneity means that prompt engineering for a single judge (GPT-4) is a local optimization that can degrade the very judges you might combine into a panel. A researcher who carefully tunes prompts for GPT-4 and then decides to compare against (or combine with) other judges faces a cross-model prompt incompatibility: the optimal prompt for GPT-4 may be suboptimal for GPT-3.5, making fair comparison difficult.

PoLL sidesteps this entirely. Because the panel's strength comes from diversity rather than per-judge optimization, the paper can use a single standard prompt across all judges without being hostage to any individual model's prompt sensitivity. The PoLL's aggregate judgment inherits robustness from the diversity of its members: even if one judge's performance degrades due to suboptimal prompting (for that specific model), the other two judges' judgments are unaffected. This is not a theoretical claim — it's empirically visible in the fact that PoLL achieves the strongest human correlation (Table 1) using standard prompts that are not optimized for any individual panel member.

This insight is a fundamental contribution to evaluation methodology because it identifies prompt brittleness as a specific, quantifiable failure mode of the single-judge paradigm that is structurally addressed by panel evaluation. It adds a dimension to the case against single-judge evaluation that goes beyond bias and cost: even if a single judge were unbiased and free, its sensitivity to prompt phrasing makes it an unreliable measurement instrument. This is a diagnostic contribution — it identifies what makes single-judge evaluation fragile — not just a performance contribution.

Innovation 4: The Absence of a Universal "Best Judge" — and the Practical Implication That Panel Evaluation Sidesteps the Judge-Selection Problem Altogether

A finding that runs through the paper's results but is never explicitly highlighted as a headline insight is the systematic absence of a single "best judge" across settings. This is empirically visible but the paper doesn't frame it as a contribution — and it deserves to be elevated because it has significant practical implications.

The evidence: across the different datasets and evaluation settings, no individual judge consistently ranks first or even second in human correlation. On KILT Natural Questions (Table 1), Command R achieves the highest individual-judge $\kappa$ , while GPT-4 is among the weakest. On KILT TriviaQA (Table 1), the ordering is different — GPT-3.5 and Haiku may outperform Command R (the exact values aren't reproduced in the main text but the pattern of rank instability is evident from Table 1 having different "best" and "second-best" entries across datasets). On Chatbot Arena (Table 2), GPT-4 has a specific ordering behavior (inflating GPT-4 variants, Figure 2) that makes it a poor rank-correlator at the top of the list, while PoLL is consistently the best or second-best.

The implication is that a researcher starting an evaluation project faces an impossible choice: which single judge should I use? The answer depends on the dataset, the task format, the models being evaluated, and the specific prompt — and the paper provides no predictive model for making this choice a priori. The PoLL framework dissolves this problem entirely. By using a fixed panel of three diverse judges and a standard aggregation rule, the researcher doesn't need to solve the judge-selection problem — the panel's aggregate performance is consistently at or near the top across settings, even though no individual panel member is universally best. This is a form of robustness through composition: the panel inherits the strengths of each judge across different settings without requiring the researcher to know in advance which judge will perform best on a particular evaluation.

This insight is a practical reframing with theoretical implications. Practically, it means that adopting PoLL eliminates a difficult model-selection decision that currently every researcher using LLM-as-judge must make implicitly or explicitly. Theoretically, it suggests that the evaluation landscape is sufficiently heterogeneous — different models have different strengths as judges on different tasks — that the optimal single-judge strategy is inherently fragile, and only a composite approach can achieve consistency. This parallels findings in other domains (algorithm selection, portfolio theory, ensemble methods) where no single model dominates across all instances, but is a novel observation for the LLM evaluation context.

This is an incremental finding in terms of the evidence required (it follows directly from the rank instability in Tables 1 and 2) but a conceptually important one because it provides an additional, independent justification for panel evaluation that does not depend on bias or cost arguments. Even if single judges were unbiased and free, the fact that judge quality is non-stationary across evaluation settings would still favor a panel approach for reliability.

Innovation 5: The 7× Cost Reduction Is Not Just an Economic Footnote — It Democratizes Access to High-Quality LLM Evaluation

The paper's cost analysis (Section 4.5) reports that the three-model PoLL costs $1.25 per million input tokens plus$ 4.25 per million output tokens, compared to GPT-4 Turbo's $10/input +$ 30/output — roughly a 7–8× reduction. While cost comparisons can feel like minor footnotes in methodology papers, this one carries genuine intellectual weight because it changes who can participate in rigorous LLM evaluation.

The significance is not the dollar amount per se, but the democratization of evaluation infrastructure. Prior to this work, the de facto standard for high-quality LLM evaluation required access to GPT-4, which is: (a) expensive enough to be prohibitive for academic labs and researchers in low-resource settings running large-scale evaluations; (b) controlled by a single company, meaning its availability, pricing, and behavior can change unpredictably; and (c) subject to rate limits and API constraints that can bottleneck evaluation throughput. A panel composed of smaller models — including models that may be available through cheaper APIs, open-weight releases, or even local deployment — lowers the barrier to entry for rigorous LLM evaluation without sacrificing (and in fact improving) evaluation quality.

This is a practical contribution with methodological consequences. It means that evaluation doesn't have to be centralized around a single proprietary model — a diverse panel can be assembled from whatever models are available, and the paper's diversity-over-capability principle suggests that even a panel of modest models from different families can outperform a single expensive model. For a graduate student evaluating their new fine-tuning method across multiple checkpoints, or a startup comparing model providers for a production system, the difference between spending $100 on GPT-4 API calls versus$ 14 on a PoLL is the difference between running a thorough evaluation and cutting corners.

The paper does not overclaim this point (it's presented straightforwardly as a cost comparison), but in the context of the broader argument — that a diverse panel beats a single large judge — the cost result transforms the paper from "here's a better way to evaluate" to "here's a better and dramatically cheaper way to evaluate, which means you can now afford to do evaluations properly." This is a significant practical reframing of the cost-quality tradeoff in LLM evaluation, and it makes the paper's recommendations actionable for the research community at large, not just well-resourced industry labs.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper spans six datasets across three evaluation settings. For single-hop QA: KILT Natural Questions (NQ), KILT TriviaQA (TQA), and KILT HotpotQA (HPQA) — all drawn from the KILT benchmark (Petroni et al., 2021) using standard test splits. For multi-hop QA: Bamboogle (125 adversarially constructed questions requiring multiple internet searches; Press et al., 2023) and a subset of HotPotQA used in Shinn et al. (2024). For pair-wise preference: Chatbot Arena Hard (Li et al., 2024a), a curated subset of 500 prompts from the crowd-sourced Chatbot Arena benchmark (Zheng et al., 2024) designed to be challenging and to correlate well with human ELO rankings. The paper does not report exact test set sizes for the KILT datasets in the main text, but these are standard benchmarks with test sets typically in the range of hundreds to low thousands of instances.
Base model(s). The "base models" in this paper are not models being trained or fine-tuned — they are the LLMs serving as judges in the evaluation panel and the LLMs serving as test subjects being evaluated. The default PoLL composition is Command R (Cohere), Claude 3 Haiku (Anthropic), and GPT-3.5 (OpenAI) — three models from three disjoint model families, deliberately using the smaller models from each family rather than their largest variants (GPT-4, Claude 3 Opus, Command R+). The test models being evaluated include: Command R, Command R+, GPT-3.5, GPT-4, Claude 3 Haiku, Claude 3 Sonnet, Claude 3 Opus, Mistral Medium, and Mistral Large. Mistral models are included specifically as test subjects "unaffiliated" with any judge models (Section 3.2). Single-judge baselines use GPT-4, GPT-3.5, Command R, and Haiku individually.
Metrics. The primary evaluation metric for QA tasks (single-hop and multi-hop) is Cohen's κ (kappa), measuring inter-rater agreement between each judge's binary True/False correctness judgments and human annotator ground-truth labels. The formula is κ = (p_o − p_e) / (1 − p_e), where p_o is observed agreement proportion and p_e is expected agreement by chance. The paper interprets κ > 0.8 as strong correlation and κ > 0.6 as moderate correlation (Section 4.1). For Chatbot Arena Hard, the metrics are Kendall's τ (tau) for rank correlation and Pearson correlation, both comparing the ranked list of models produced by each judge method against the ground-truth crowd-sourced Chatbot Arena ELO rankings (Section 4.2). The paper does not report standard accuracy or F1 metrics for the judges themselves — the only criterion is agreement with human judgments.
Baselines. The paper compares against five categories of baselines: (1) Individual LLM judges — GPT-4, GPT-3.5, Command R, and Haiku each used as single judges, with GPT-4 being the de facto standard in the field and the primary comparison point; (2) Exact Match (EM) — the "containment" version from Liu et al. (2024) that checks whether a normalized reference answer string appears anywhere within the generated model response, representing traditional heuristic metrics; (3) Majority voting — not evaluated as a standalone baseline but referenced as an alternative aggregation function in Chatbot Arena (Section 3.1, footnote 4, noting similar results); (4) The original Arena Hard GPT-4 judge — the single GPT-4 judge from Li et al. (2024a)'s original implementation, used as the comparison point for the Chatbot Arena experiments (Section 4.2, Figure 2, Table 2); (5) Different prompt variants for GPT-4 — zero-shot, few-shot standard, few-shot without instruction, chat-formatted few-shot, and "don't overthink" optimized prompt, used in the prompt sensitivity ablation (Section 4.3, Table 3).
Generation budget / compute accounting. The paper measures cost in dollars per million tokens rather than FLOPs or generations, since the judges are API-accessed models with known pricing. At the time of writing (Section 4.5): the three-model PoLL costs $1.25 per million input tokens plus$ 4.25 per million output tokens; GPT-4 Turbo costs $10 per million input tokens plus$ 30 per million output tokens. The paper states this is a "seven to eight times" cost reduction depending on the input-to-output token ratio. Latency is discussed qualitatively — "running a collection of smaller models in parallel (as in PoLL) is faster than a single big model" — but no explicit latency measurements or wall-clock time comparisons are reported (Section 4.5).
Cross-validation / statistical protocol. For the QA prompt construction, the paper uses a selection procedure on a held-out portion of human labels: they sweep over n_shots in {8, 10, 16} and pick the prompt with the strongest average agreement across all judges (Appendix A.1.1). For the GPT-4 prompt ablation (Section 4.3, Table 3), the paper reports Δκ relative to the standard few-shot prompt, though statistical significance tests (confidence intervals, p-values) are not reported for any of the κ comparisons. For human annotations, some datasets are triple-annotated with ground truth determined by majority voting (NQ, single-hop HPQA), while others are singly annotated (TriviaQA, Bamboogle, multi-hop HPQA) per Appendix A.3, Table 8. The paper does not report inter-annotator agreement statistics for the triple-annotated datasets, which would establish the noise ceiling for automated judge performance. No formal cross-validation is used for the Chatbot Arena experiments — the Arena Hard benchmark has a fixed test set, and the ground-truth rankings come from the independently collected crowd-sourced Chatbot Arena ELO scores.

Main Quantitative Results

Correlation with Human Judgments on Single-Hop QA

The central result for reference-based QA evaluation is Table 1. Across three KILT datasets (NQ, TQA, HPQA), the PoLL — using max voting over its three constituent judges — achieves the highest Cohen's κ on every dataset. The paper does not reproduce the exact numeric values in the main text body (they appear only in Table 1 and Appendix Tables 5–6), but the pattern is clear from the table's bold/underline formatting convention (best results bold, second-best underlined). The PoLL entries are consistently bolded, indicating first-place performance.

The ranking of individual judges is dataset-dependent. On KILT Natural Questions, Command R achieves the highest individual-judge κ (making it the second-best overall, underlined), followed by other individual judges, with GPT-4 being one of the weaker evaluators — the paper states that "GPT-4 is one of the weaker evaluators on this particular task setup" (Section 4.1). On KILT TriviaQA and HotPotQA, the individual judge rankings shift, with different models occupying the second-best position — evidence that no single model is a universally strong judge.

The magnitude of the improvement that PoLL provides over GPT-4 specifically is not isolated in a single headline number. The paper emphasizes the ranking (PoLL is best) rather than reporting the exact κ differences. From Table 1's layout, GPT-4 does not appear as the underlined (second-best) entry on any dataset, meaning it trails not only PoLL but also at least one smaller model on each QA dataset tested.

The Exact Match (EM) baseline serves as a lower bound — it is included in the judge comparison tables and consistently underperforms all LLM-based judges (individual and panel), validating the paper's motivation that heuristic string-matching is inadequate for semantic correctness evaluation. The specific EM κ values are visible in Tables 5 and 6 (Appendix) for the multi-hop datasets, confirming this pattern.

Correlation with Human Judgments on Multi-Hop QA

The multi-hop QA results appear in Appendix Tables 5 (Bamboogle) and 6 (multi-hop HotPotQA). The pattern from single-hop QA extends to these more complex tasks: PoLL achieves the highest κ on both datasets. The paper does not discuss these tables in detail in the main text (they are relegated to the Appendix), but they provide replication across task complexity — from single-hop factoid questions requiring one retrieval step to multi-hop questions requiring compositional reasoning across multiple retrieved documents. The consistency of PoLL's advantage across both single-hop and multi-hop settings strengthens the claim that panel evaluation generalizes beyond any individual task format.

Rank Correlation on Chatbot Arena Hard

For the pair-wise preference setting, Table 2 reports both Kendall's τ and Pearson correlation between each judge method's model ranking and the ground-truth Chatbot Arena crowd-sourced ELO rankings. The PoLL achieves the best performance on both metrics. GPT-4 (the single judge used in the original Arena Hard) achieves lower correlation, particularly for Kendall's τ.

Figure 2 provides the qualitative visualization that makes this result interpretable. The figure plots model rankings: the diagonal represents perfect agreement with ground truth. The PoLL's ranking points cluster more tightly around the diagonal than GPT-4's, especially at the top of the ranked list (positions 1–4). The most visually striking evidence of GPT-4's intra-model bias is directly visible: the GPT-4 judge places a GPT-4 variant at position 2, whereas its ground-truth position is position 4 — an inflation of two rank positions. The PoLL does not exhibit this distortion for GPT-family models, placing them closer to their ground-truth ranks.

The paper reports the raw Arena Hard scores from the PoLL in Appendix Table 7, using average pooling across the three judges. These scores are the per-model aggregates that produce the rankings visualized in Figure 2. The exact numeric scores matter less than their rank ordering, since the evaluation is comparative (which model is better, not what absolute score a model achieves).

Bias Reduction Quantified on Multi-Hop QA

Figures 3 (HotPotQA multi-hop) and 4 (Bamboogle) visualize the systematic bias in individual judges and the bias reduction achieved by PoLL. Each figure is a scatter plot where:

X-axis: test models being evaluated (Command R, Command R+, GPT-3.5, GPT-4, Claude 3 Haiku, Claude 3 Sonnet, Claude 3 Opus, Mistral Medium, Mistral Large).
Y-axis: the delta between the judge's accuracy score and human annotator accuracy — that is, how much the judge overestimates (+) or underestimates (−) each test model's performance relative to ground truth. The dotted horizontal line at y = 0 represents perfect agreement with human judgments.
Color/symbol: different judge models (EM, Command R, Haiku, GPT-3.5, GPT-4, PoLL).

The key empirical findings from these figures:

Self-preference is systematic. For each individual judge model (Command R, Haiku, GPT-3.5), the highest positive delta — the largest overestimation of performance — occurs when that judge evaluates its own outputs. For example, when Command R is the test model, the Command R judge shows the most inflated accuracy score relative to human ground truth. When GPT-3.5 is the test model, the GPT-3.5 judge shows the most inflated score. This pattern holds across all three judges and both datasets, providing clean within-experiment replication of the intra-model bias phenomenon documented by Panickssery et al. (2024).
Bias is not restricted to exact self-matches. The bias extends to within-family models. GPT-3.5's scores for GPT-4 outputs (and vice versa) show some inflation relative to human ground truth, though less extreme than for exact self-matches. This is consistent with the paper's rationale for requiring disjoint model families in the panel — if within-family bias exists, a panel that includes both GPT-3.5 and GPT-4 would not fully decorrelate their errors.
PoLL has the smallest spread. The PoLL column (rightmost position in the scatter plots) consistently shows deltas closest to zero across all test models. The paper reports a standard deviation of 2.2 for PoLL's deltas, compared to 6.1 for GPT-3.5 (the highest) and intermediate values for other individual judges (Section 4.4). This is a concrete quantification of bias reduction: PoLL's scores are not only closer to human judgments on average, but more consistently close across all evaluated models, meaning it does not systematically favor or penalize any particular model or model family.
EM (Exact Match) has a distinct failure mode. EM's deltas are consistently negative — it systematically under-estimates model performance relative to human judgments, particularly for models that produce verbose or stylistically varied outputs. This validates the paper's motivation (Section 1) that string-matching metrics suffer from false negatives on free-form generation tasks.

A noteworthy detail: the Mistral models (Mistral Medium and Mistral Large) are test models with no family relationship to any judge model in the panel (none of Command R, Haiku, or GPT-3.5 are Mistral-family models). The scatter plots show that PoLL's deltas for Mistral models are close to zero, suggesting that the bias reduction benefits extend to models completely outside the judges' training families — the PoLL does not simply eliminate self-preference but provides generally fairer evaluations across all tested models.

Prompt Sensitivity: GPT-4 Is Uniquely Volatile

Table 3 reports the prompt ablation experiment on KILT Natural Questions with GPT-4 as the judge. The baseline is GPT-4 with the standard few-shot prompt. The results (reported as Δκ relative to this baseline) are:

Zero-shot (no few-shot examples): lowest performance (exact κ not reported, but described as weakest).
Few-shot standard (Table 11 prompt, 10 examples): baseline.
No instruction line (instruction removed): −0.03 Δκ — the instruction is helpful, contrary to the hypothesis that it might confuse the model.
Move instruction line (instruction moved to separate system call): +0.01 Δκ — negligible improvement.
Chat-formatted shots (examples formatted as conversational turns): −0.07 Δκ — substantially worse, suggesting flat formatting is better for this evaluation task.
"Don't overthink" prompt (instruction replaced with explicit guidance to avoid reasoning about external factuality): +0.07 Δκ — the largest positive change, bringing GPT-4 up to the level of GPT-3.5 with the standard prompt.
Additional surface-level optimizations on top of "don't overthink": +0.03 Δκ additional improvement, for a total of approximately +0.10 Δκ over the standard prompt.

The paper compares this optimized GPT-4 prompt against GPT-3.5 with the standard few-shot prompt (which achieves κ = 0.726 on NQ, per Table 1). The optimized prompt brings GPT-4's performance up to roughly that level, but still below Command R (the best individual judge on NQ) and PoLL.

Appendix Table 4 reports the cross-model effects of the GPT-4-optimized prompt: when the "don't overthink" prompt (designed for GPT-4) is applied to the other judges, it has a negligible positive effect on Haiku, a minor negative effect on Command R, and a "severe negative impact" on GPT-3.5 (footnote 10). This heterogeneity — a prompt that helps one model substantially harms another — is the key evidence that prompt optimization cannot be done uniformly across a panel and that PoLL's strategy of using a single standard prompt across all judges avoids this cross-model incompatibility.

Ablation Studies and Robustness Checks

Panel composition across model families (implicit ablation): The paper's choice of three judges from three disjoint families is not ablated against alternative compositions (e.g., three judges from the same family, two from one family plus one from another, a panel of five, or a panel with different size models from the same families such as GPT-4 instead of GPT-3.5). The paper explicitly acknowledges this gap: "We also leave the task of 'panel selection,' or identifying the best models to include in PoLL in terms of quality and cost, to future work" (Section 5). This means we cannot determine from the reported experiments whether the key driver of PoLL's advantage is: (a) the number of judges (three > one, regardless of diversity), (b) the diversity of model families (disjoint families > overlapping families), (c) the specific choice of models (these three models happen to be well-suited to these tasks), or some interaction of these factors. This is the most significant ablation gap in the paper.

Aggregation function (footnote 4): For Chatbot Arena, the paper notes that it "experimented with max voting with average voting fallback which yielded similar results with an identical overall ranking" compared to average pooling alone. This suggests the ranking is robust to the choice of aggregation function for the pair-wise setting, though the paper does not provide the full comparison data or test other aggregation functions (e.g., median, weighted voting based on judge confidence, or learned aggregation). For QA, only max voting is used, and no comparison against majority voting, average pooling of binary scores, or weighted voting is reported. The justification for max voting over alternatives is implicit (max voting minimizes false negatives) but not empirically validated.

Judge model scaling (implicit comparison): The paper compares PoLL (GPT-3.5 + Haiku + Command R) against GPT-4 as a single judge, but does not compare against: (a) a panel of the best individual judges (e.g., GPT-4 + Claude 3 Opus + Command R+), which would test whether diversity still matters when all panel members are highly capable; (b) a single judge of intermediate capability between GPT-3.5 and GPT-4 (e.g., GPT-4 with lower temperature or a different prompt), which would test whether the diversity benefit is specific to the capability gap; or (c) a single large model with majority voting across multiple samples (querying GPT-4 multiple times and aggregating), which would test a different form of "panel" that preserves within-family homogeneity.

Dataset difficulty and task generalization: The paper spans six datasets but all fall within two broad categories: factoid QA (with reference answers) and preference ranking (Chatbot Arena). The paper does not test on: math reasoning (where correctness is less subjective but harder to verify via semantic containment), code generation (where functional correctness via test cases is possible but not covered), summarization (where quality is multi-dimensional and reference-based scoring is less straightforward), or open-ended creative generation (where human judgments are highly subjective). The paper explicitly acknowledges this limitation: "further work is needed to see how broadly applicable the method is, for example, in math or reasoning evaluations, where language models often struggle" (Section 5).

Few-shot example count sweep: For the QA prompts, the paper sweeps over n_shots in {8, 10, 16} and selects the prompt with the strongest average agreement across all judges on a held-out portion of human labels (Appendix A.1.1). This sweep provides some evidence that the results are not brittle to the exact number of few-shot examples in the 8–16 range, though performance at the extremes (zero-shot, very many shots) is not fully characterized beyond the zero-shot ablation for GPT-4 specifically.

Human annotation quality (missing inter-annotator agreement): The paper does not report Cohen's κ or other inter-annotator agreement metrics for the triple-annotated datasets (NQ and single-hop HPQA). This is a meaningful gap because human inter-annotator agreement establishes a theoretical ceiling for automated judge performance: no LLM judge can achieve κ higher than human annotators achieve with each other, since the human majority vote is the ground truth. If human inter-annotator κ is, say, 0.85, then PoLL's κ of 0.75 leaves a gap of 0.10 attributable to judge error. If human κ is only 0.80, PoLL at 0.75 is much closer to the ceiling. Without this number, we cannot assess how close PoLL is to optimal performance.

Answer order bias in pair-wise scoring: The paper follows the standard practice of evaluating both (a, b) and (b, a) orderings in Chatbot Arena to mitigate position bias, but does not report the magnitude of the ordering effect or test whether the PoLL reduces ordering bias compared to a single judge. If individual judges exhibit position bias (preferring the first or second answer regardless of quality), and if this bias differs across judges from different model families, the PoLL's aggregation should attenuate it — but this is not tested or reported.

Test model coverage: The paper evaluates nine test models spanning four families (Command R/Command R+, GPT-3.5/GPT-4, Claude 3 Haiku/Sonnet/Opus, Mistral Medium/Large). This provides reasonable coverage, but the inclusion of Mistral models as "unaffiliated" test subjects serves as an informal robustness check: since no Mistral model sits on the PoLL, any bias the PoLL shows toward Mistral models cannot be self-preference and must reflect a more general evaluation tendency. The results (Figures 3–4) show PoLL's Mistral deltas close to zero, which is reassuring but limited to two Mistral models on two datasets.

Prompt robustness across judges: The finding that the GPT-4-optimized "don't overthink" prompt degrades GPT-3.5 performance (Appendix footnote 10) is a negative result that doubles as a robustness check for the PoLL's design. It demonstrates that prompt engineering is model-specific and that a panel using a single uniform prompt avoids the pitfall of optimizing for one judge at the expense of others. The paper does not systematically test whether alternative standard prompts (different wording, different few-shot selection) would change the PoLL's relative performance, so we cannot fully assess how robust the PoLL advantage is to the specific prompt templates used.

Critical Assessment

The experiments support the paper's central claims, but with important scope limitations that the paper itself largely acknowledges. The claims should be understood as holding specifically for the evaluation settings and model combinations tested, not as a universal demonstration that panel evaluation is always superior.

Claim from the executive summary: "PoLL correlates better with human judgments compared to a single large judge (GPT-4), while being over seven times cheaper." This claim is supported by the reported data, but the support is narrow in two ways.

What was tested: PoLL (Command R + Haiku + GPT-3.5) vs. GPT-4 alone, on six datasets across three evaluation settings (reference-based QA and pair-wise preference), using human correlation metrics (Cohen's κ, Kendall's τ, Pearson r). Tables 1 and 2 show PoLL outperforming GPT-4 on every dataset. The cost comparison (Section 4.5) uses publicly available API pricing at the time of writing.

What was not tested:

Only one panel composition was evaluated. The paper does not demonstrate that the specific composition (these three models from these three families) is necessary or sufficient for the improvement. A panel of GPT-4 + Haiku + Command R (replacing GPT-3.5 with GPT-4) might perform even better, which would narrow the cost advantage but strengthen the correlation advantage. A panel of GPT-3.5 alone with three different prompts or three different temperature settings might capture some diversity benefit at even lower cost. The paper's claims about "diversity across model families" being the mechanism are plausible but not directly tested against alternative explanations.
Only one "single large judge" comparison point (GPT-4) was used. Claude 3 Opus, Mistral Large, or Command R+ as single judges are not compared. It is possible that some other single large judge would match or exceed PoLL on these tasks. The paper's finding that no single model is universally best across datasets (Section 4.1) suggests this is unlikely but does not rule it out for the specific datasets where GPT-4 was the weakest.
The seven-times cost figure is based on list API prices that may vary over time, across usage tiers, and across providers. The paper acknowledges this implicitly by noting "at the time of writing" in Section 4.5.

Claim from the executive summary: "PoLL exhibits less intra-model bias due to its composition of disjoint model families." This claim is supported by Figures 3 and 4, which provide clear visual and quantitative evidence of bias reduction. The mechanism (decorrelation across disjoint families) is not directly tested — the paper never compares a "disjoint families" panel against a "same-family" panel of equal size (e.g., three GPT-family models or three Claude-family models). The evidence is consistent with the diversity mechanism but does not rule out alternative explanations: perhaps any panel of three models (even from the same family) would reduce bias simply through averaging; perhaps the specific models chosen happen to have complementary error patterns that are not specifically due to their family membership; or perhaps max voting alone (which is more lenient) is responsible for the bias reduction rather than diversity per se. The paper's interpretation of the mechanism is reasonable but not uniquely established by the reported experiments.

A strength of the bias analysis that deserves explicit recognition: the scatter plots in Figures 3 and 4 test a within-experiment prediction of the self-preference hypothesis — specifically, that the highest positive delta for each test model will be when that model judges itself. This prediction is confirmed for all three judges on both datasets, which is a clean, falsifiable pattern that strengthens the case for intra-model bias as a real and systematic phenomenon, not a statistical artifact.

Claim from Section 4.3: "GPT-4 is a relatively weak judge, exhibiting high variance with minor changes to the prompt." This claim is supported by Table 3, which shows substantial κ variation across prompt variants for GPT-4 on Natural Questions. However, the claim's framing — "GPT-4 is a relatively weak judge" — is dataset-specific. GPT-4 performs weakly on the specific KILT QA tasks tested, but the paper's own data show GPT-4 as a reasonable judge on Chatbot Arena (Table 2, though still below PoLL). The prompt sensitivity finding is robust: the optimized "don't overthink" prompt improves GPT-4's κ by approximately +0.10 on NQ, which is a large effect relative to the differences between judge models in Table 1. The finding that prompt optimizations are not transferable across models (footnote 10, Appendix Table 4) is an important negative result that is under-emphasized in the main text but has significant implications for the single-judge paradigm.

Missing experiments that would strengthen the paper:

Panel composition ablation. Comparing PoLL (3 diverse models) against: (a) the same three models but all from the same family (e.g., GPT-3.5 + GPT-4 + GPT-4-turbo at different temperatures — though this is not possible since GPT-4-turbo is just GPT-4); (b) panels of size 2 (every pair from the three models) to test whether three is meaningfully better than two; (c) a single model queried three times with varying prompts or temperature to capture within-model diversity. These would clarify whether diversity or simply multiplicity drives the benefit.
Ground-truth quality characterization. Reporting human inter-annotator κ on the triple-annotated datasets would establish the performance ceiling and contextualize the reported κ values. If human κ is 0.82 and PoLL achieves 0.78, PoLL is near-optimal; if human κ is 0.95 and PoLL achieves 0.78, there is substantial room for improvement.
Per-category or per-difficulty breakdown. The paper does not analyze whether PoLL's advantage is uniform across question types, difficulty levels, or answer formats. If PoLL's advantage is concentrated on ambiguous or edge-case instances (where individual judges disagree) and PoLL is equivalent to GPT-4 on clear-cut instances, that would have practical implications for when the panel is worth its (still modest) cost and complexity.
Calibration analysis. The paper evaluates judges on correlation with humans (ranking and agreement) but not on calibration — whether a judge's confidence in its judgments matches its actual accuracy. If PoLL systematically overestimates or underestimates model performance (even if it ranks models correctly), that has practical consequences for interpreting absolute accuracy scores.
Statistical significance. No confidence intervals, standard errors, or significance tests are reported for any κ or correlation comparisons. With test sets typically in the hundreds to low thousands, differences in κ of 0.02–0.05 may or may not be statistically significant. The paper relies on consistent patterns across datasets rather than formal inference, which is reasonable for an empirical study but limits the strength of individual pairwise claims.

Where the claims hold and where they are conditional:

PoLL outperforms GPT-4: holds on all six datasets tested, but these are exclusively factoid QA and preference ranking tasks. The paper explicitly does not claim this extends to math, reasoning, or open-ended generation evaluation (Section 5).
PoLL is over seven times cheaper than GPT-4: holds at the reported API prices and for the specific panel composition. If GPT-4 prices decrease or if a panel with more expensive models were used, the multiple would change. Cost is not an invariant property of the method — it is a contingent economic fact at a particular point in time.
PoLL reduces intra-model bias: holds for the tested judges and test models. The mechanism (diversity across families) is not directly tested, so the claim is empirically supported but mechanistically underspecified. Whether a panel of two diverse models would achieve similar bias reduction, or whether diversity along dimensions other than model family (e.g., model size, training data recency, architecture type) matters, is unknown.
GPT-4 shows high prompt sensitivity: holds on the NQ dataset with the specific prompt variants tested. Whether GPT-4 is equally sensitive on other tasks, and whether other large models (Claude 3 Opus, Mistral Large) share this sensitivity, is not tested. The finding that prompt optimizations are model-specific is supported by the cross-model prompt application data in Appendix Table 4.

Overall assessment: The experiments provide credible, multi-dataset evidence that a diverse panel of smaller models can outperform a single GPT-4 judge on factoid QA and preference ranking tasks, while reducing cost and intra-model bias. The evidence for the mechanism (diversity across model families as the driver of improvement) is suggestive but not rigorously isolated — the paper demonstrates that a specific diverse panel works well without systematically testing which aspects of that panel (size, diversity, specific model choices, aggregation function) are necessary for the benefit. The paper is transparent about these scope limitations (Section 5), which strengthens its credibility. The findings are sufficient to motivate adoption of panel-based evaluation in the tested settings and to motivate further research on panel selection and broader task coverage, but they are not sufficient to conclude that any diverse panel will outperform any single large judge on any evaluation task.

6. Limitations and Trade-offs

Limitation 1: A Single, Untested Panel Composition — No Evidence That the Specific Choice of Judges Generalizes

The assumption or constraint. The paper evaluates exactly one panel composition: Command R + Claude 3 Haiku + GPT-3.5. The paper does not ablate against alternative panel sizes, different model choices within the same families (e.g., GPT-4 instead of GPT-3.5, Claude 3 Sonnet instead of Haiku), or panels with overlapping family membership. The authors explicitly acknowledge this gap in Section 5:

"We also leave the task of 'panel selection,' or identifying the best models to include in PoLL in terms of quality and cost, to future work."

This means there is no empirical evidence that the paper's central mechanism — diversity across disjoint model families — is what drives PoLL's advantage, rather than simply having any three judges, having these particular three judges, or having a panel of this specific size. The mechanism is hypothesized but not isolated.

The consequence. A practitioner attempting to deploy PoLL faces an immediate, unresolved question: which models should go in my panel? If they cannot access Command R (a Cohere model), or if they want to use open-weight models, or if they want to include a model from a fourth family, the paper provides no guidance. The reported results are tied to a specific combination of three proprietary models from three specific companies at a specific point in time — and the paper does not demonstrate that alternative compositions would preserve the benefits. Worse, if the advantage comes primarily from the specific error-correlation structure of these three models rather than from a general diversity principle, swapping any panel member could degrade performance unpredictably. A panel of GPT-3.5 + Claude 3 Haiku + Mistral Small (substituting Mistral for Cohere) might perform better, worse, or equivalently — the paper provides no way to predict this.

Relatedly, the paper does not test whether panel size (three) is optimal. A panel of two might achieve most of the benefit at even lower cost; a panel of five might provide diminishing returns at increasing cost. The absence of a size sweep means practitioners cannot make an informed tradeoff between panel breadth and evaluation budget.

What evidence exists in the paper. No direct evidence. The paper demonstrates that one specific diverse panel works well, but provides zero ablation experiments varying panel composition, panel size, or diversity degree. Figures 3 and 4 show that individual judges exhibit different per-model biases (the self-preference deltas vary in magnitude across Command R, Haiku, and GPT-3.5), which is consistent with the diversity mechanism but does not test whether diversity of family matters specifically versus, say, diversity of model size, diversity of training data recency, or simply multiplicity (three independent judgments, regardless of source). The cross-model prompt sensitivity data (Appendix footnote 10, showing that a prompt helping GPT-4 harms GPT-3.5) provides indirect evidence that these models behave differently, but does not show that this behavioral difference is what drives PoLL's evaluation quality.

Mitigation status. Not addressed. The paper explicitly defers panel selection to future work (Section 5). This is the single largest gap between the paper's empirical demonstration and its deployability as a practical method.

Limitation 2: Max Voting Is Unvalidated — No Evidence That the Leniency Bias It Introduces Is Calibrated or Appropriate

The assumption or constraint. For all QA experiments, the PoLL uses max voting: an answer is considered correct if any of the three judges says it is correct. This is the most lenient possible aggregation rule — it systematically biases the panel toward higher accuracy estimates by converting any single judge's "True" into a PoLL "True." The paper provides no comparison against alternative aggregation functions for QA (majority voting, average pooling of binary scores, weighted voting based on judge confidence, or requiring unanimity). The choice of max voting is justified implicitly by the task structure (false negatives are more damaging than false positives for QA evaluation, per the paper's critique of EM in Section 1) but is never empirically validated against alternatives.

The consequence. Max voting introduces a structural leniency bias into the PoLL scores. A test model's accuracy as measured by PoLL will be systematically higher than its accuracy as measured by any individual judge, because PoLL counts an answer as correct if the most generous judge says so. This has two practical consequences:

First, absolute accuracy numbers are inflated. A researcher reporting "Model X achieves 75% accuracy on KILT NQ as measured by PoLL" is reporting a number that overestimates the model's performance relative to any single-judge evaluation. If downstream decisions (model selection, deployment thresholds) use absolute accuracy cutoffs, the leniency bias matters. The paper does not provide a calibration curve or correction factor to map PoLL scores to individual-judge or human-annotator scores.

Second, the bias may not be uniform across models. If one judge (say, Command R) is consistently more lenient than the others, and if that leniency interacts with test model characteristics (e.g., Command R is especially generous toward verbose answers), then PoLL's inflation is not a constant offset — it varies per test model, potentially distorting relative rankings even as it improves correlation with human judgments on average. The paper does not analyze whether max voting introduces differential inflation across the nine test models.

The paper's justification for max voting — that false negatives are the primary concern in QA evaluation — is reasonable given the documented failure mode of EM (systematic false negatives from string mismatches), but this argument establishes that max voting is better than string matching, not that max voting is better than alternative aggregations of the same panel. Majority voting would also reduce false negatives relative to EM (since it uses semantic judges rather than string matching) while being less susceptible to inflation from a single overly-generous judge. The paper provides no data to choose between these options.

What evidence exists in the paper. None. For Chatbot Arena, footnote 4 (Section 3.1) notes that the paper "experimented with max voting with average voting fallback which yielded similar results with an identical overall ranking," suggesting robustness to aggregation function for the pair-wise setting. But this evidence is for a different task (graded preference, not binary correctness) and the footnote provides no quantitative comparison. For the QA setting where max voting carries the most risk (since it inflates binary accuracy), no aggregation ablation is reported at all. We do not know whether PoLL's advantage over GPT-4 in Table 1 is driven by the diversity of the panel or by the leniency of max voting — a single judge using a more lenient decision threshold might achieve similar gains without requiring a panel.

Mitigation status. Not addressed. The paper treats max voting as a fixed design choice without empirical justification or comparison to alternatives. A practitioner cannot determine from the reported results whether max voting, majority voting, or some intermediate rule (e.g., at least two judges must agree) is optimal for their use case.

Limitation 3: Task Scope Is Narrow — Only Factoid QA and Preference Ranking; No Evidence for Reasoning, Math, Code, or Open-Ended Tasks

The assumption or constraint. Every experiment in the paper falls into one of two evaluation paradigms: (1) reference-based binary correctness on factoid question answering (single-hop and multi-hop), where the judge compares a generated answer against a short human-written reference answer and outputs True/False; (2) pair-wise preference ranking on Chatbot Arena Hard, where the judge compares two generated outputs on arbitrary user prompts and outputs a graded preference. The paper explicitly acknowledges the scope limitation in Section 5:

"further work is needed to see how broadly applicable the method is, for example, in math or reasoning evaluations, where language models often struggle"

But this acknowledgment understates the narrowness of the tested task distribution. The paper does not test PoLL on any of the following common evaluation settings: math problem grading (where correctness is objective but verification requires multi-step reasoning, not just semantic containment); code generation evaluation (where functional correctness can be checked via test cases, but LLM judges are also used for style, efficiency, and partial credit); summarization quality assessment (where quality is multi-dimensional — factual consistency, coherence, relevance, conciseness — and reference-based scoring is notoriously unreliable); open-ended creative generation (dialogue, story writing, poetry, where human judgments are highly subjective and inter-annotator agreement is low); or safety/toxicity evaluation (where the judge must detect subtle harmful content).

The consequence. The paper's core findings — that PoLL outperforms GPT-4, reduces bias, and costs 7× less — are proven only for tasks that reduce evaluation to a relatively simple subtask: checking whether a short reference answer is semantically contained within a longer generation, or expressing a preference between two outputs. These subtasks are well within the capability range of the smaller models used in PoLL. For math or reasoning evaluation, the subtask the judge must perform is much harder — it requires verifying multi-step logical deductions, detecting subtle calculation errors, or determining whether an unconventional solution path is valid. The paper's own finding that GPT-4 struggles with the (relatively simple) QA judgment task without careful prompt engineering (Section 4.3) raises the concern that all models, including PoLL panel members, may struggle with harder judgment tasks. If the difficulty of judging exceeds the capability of the panel members, the diversity benefit may not compensate — the panel would simply produce consistently wrong judgments rather than decorrelated errors.

Specifically, a practitioner evaluating a math tutoring system or a code generation model cannot safely assume that PoLL will outperform GPT-4. The paper provides zero evidence either way. The mechanism the paper proposes (diversity across model families decorrelates errors) should in principle extend to harder tasks — if the panel members make independent errors, aggregation still helps — but the paper does not test whether the errors remain independent when the task difficulty increases to the point where all panel members are near their capability ceiling. At the capability ceiling, models may converge on similar failure modes (e.g., all smaller models might fail to detect the same type of logical flaw), reducing the effective diversity that PoLL relies on.

What evidence exists in the paper. Explicitly none — the paper states this is future work. The six datasets tested (NQ, TQA, single-hop HPQA, Bamboogle, multi-hop HPQA, Chatbot Arena Hard) all fall within the factoid QA and preference ranking paradigms. The paper does not include any task where the judging itself requires complex reasoning beyond semantic comparison or preference expression. This is not a hidden limitation — the authors are transparent — but it significantly bounds the practical applicability of the results. A researcher or practitioner working on non-factoid evaluation cannot extrapolate from this paper's results and must treat PoLL as unvalidated for their use case.

Mitigation status. Acknowledged but not addressed. The paper flags this as future work but does not provide even a small-scale pilot experiment on a reasoning or math task, which would provide some signal about whether the findings are likely to transfer. This leaves the scope of applicability as a binary unknown rather than a characterized boundary.

Limitation 4: Human Inter-Annotator Agreement Is Not Reported — The Performance Ceiling Is Unknown

The assumption or constraint. The paper's entire evaluation framework rests on the assumption that human judgments are the gold standard: PoLL is "better" than GPT-4 specifically because it agrees more closely with human annotators. However, human annotators are not a perfect measurement instrument — they make errors, disagree with each other, and exhibit their own biases. The paper uses triple annotation with majority voting for two datasets (NQ and single-hop HotPotQA, per Appendix A.3) but does not report the inter-annotator agreement (e.g., Cohen's κ between annotators, or the proportion of instances where all three annotators agreed vs. 2-1 splits). For the singly-annotated datasets (TriviaQA, Bamboogle, multi-hop HotPotQA), there is no redundancy at all, meaning individual annotator errors are indistinguishable from ground truth.

The consequence. Without inter-annotator agreement statistics, we cannot assess how close the PoLL's performance is to the theoretical ceiling. If human annotators achieve κ = 0.85 with each other on NQ, and PoLL achieves κ = 0.75 with the human majority vote, then PoLL is capturing 0.75/0.85 ≈ 88% of the achievable agreement — reasonably close to optimal, with a gap of 0.10 attributable to judge error. If human annotators achieve κ = 0.95, PoLL at 0.75 is much further from the ceiling, with substantial room for improvement through better judges, better prompts, or different aggregation functions. If human κ is only 0.70 — meaning the annotation task is genuinely difficult and humans disagree substantially — then PoLL's κ of 0.75 is suspicious (it exceeds human-human agreement, which should not be possible if human judgments are the ground truth) and may indicate overfitting or systematic bias in a favorable direction.

This gap matters practically because it contextualizes how much improvement is possible. If PoLL is already at 90% of the ceiling, further investment in panel optimization (more judges, different models, better aggregation) will yield marginal gains. If PoLL is at 70% of the ceiling, there is substantial headroom and the panel selection problem (Limitation 1) becomes more urgent.

Additionally, the singly-annotated datasets introduce noise into the ground truth. If an annotator makes an error on a particular instance, that error becomes the ground-truth label against which all judges are evaluated. Both PoLL and GPT-4 may disagree with this label because the label is wrong, not because the judges are wrong — but the paper treats all disagreements as judge errors. If annotation errors are randomly distributed, they add noise that reduces all κ values uniformly, preserving the relative ranking of judges. But if annotation errors are systematic (e.g., annotators are more likely to err on ambiguous or edge-case instances), and if different judges handle ambiguity differently, the noise could differentially affect the measured κ of different judges.

What evidence exists in the paper. None. The paper describes the annotation protocol (professional annotators, triple annotation with majority voting for some datasets) in Appendix A.3 and Table 8, but does not report any inter-annotator agreement metrics. This is a standard practice in many NLP papers (human annotation is treated as ground truth without reliability quantification), but it is a meaningful gap for a paper whose entire contribution is about improving agreement with human judgments. The quality of the target signal directly bounds the interpretability of the improvement.

Mitigation status. Not addressed. The paper provides no inter-annotator agreement statistics and does not discuss the implications of annotation noise for its results. A simple inter-annotator κ for the triple-annotated datasets would substantially strengthen the paper by establishing the performance ceiling and validating that the annotation task is sufficiently reliable to serve as ground truth.

Limitation 5: The Cost Comparison Is Incomplete — It Excludes Prompt Engineering, Infrastructure, and Latency Costs

The assumption or constraint. The paper's headline cost comparison (Section 4.5) compares only the per-token API pricing of the three PoLL judges ( $1.25/input +$ 4.25/output per million tokens) against GPT-4 Turbo ( $10/input +$ 30/output). This is a 7–8× reduction in inference API cost. However, the total cost of deploying and maintaining a panel-based evaluation system includes several components that the paper does not account for, and the comparison makes implicit assumptions that may not hold for all practitioners.

The consequence. The 7× cost figure understates the true cost difference in at least three ways:

Prompt engineering and maintenance cost. The paper demonstrates (Section 4.3) that judge performance is sensitive to prompt design, and that prompts optimized for one model can degrade another (Appendix footnote 10: the "don't overthink" prompt that helps GPT-4 severely harms GPT-3.5). Deploying a PoLL therefore requires either: (a) using a single standard prompt across all judges and accepting that some judges will underperform relative to their potential, or (b) maintaining model-specific prompts for each judge, which multiplies prompt engineering effort and creates an ongoing maintenance burden as model versions change. GPT-4 as a single judge requires optimizing prompts for only one model. The paper's experiments use a standard prompt across all judges, which is pragmatically sensible but means the PoLL's performance may be lower than what could be achieved with model-specific optimization — and the cost of discovering and maintaining model-specific prompts is unaccounted for.

Infrastructure and operational complexity. Running a panel of three models requires: managing API keys, authentication, and rate limits for three different providers (Cohere, Anthropic, OpenAI); handling partial failures (one provider experiences an outage); monitoring usage and costs across three billing systems; and maintaining the panel if one provider deprecates a model or changes its API. A single GPT-4 judge requires managing one provider relationship. For a small research team or an individual researcher, this operational overhead is real and the paper's cost comparison does not factor it in.

Latency and throughput (partially addressed but not measured). The paper claims qualitatively that "running a collection of smaller models in parallel (as in PoLL) is faster than a single big model" (Section 4.5). This may be true for wall-clock latency if judges are queried in parallel, but it ignores: (a) the slowest judge determines the panel's latency if judgments are needed synchronously for aggregation — if Haiku responds in 0.5s but GPT-3.5 takes 2s, PoLL latency is 2s regardless of parallelism; (b) throughput may be limited by the most restrictive rate limit across the three providers; (c) parallel queries to three providers require three times the concurrent connection handling. The paper performs no latency measurements, so the claim that PoLL is "faster" is unsubstantiated and may not hold in all deployment configurations.

What evidence exists in the paper. The API pricing comparison (Section 4.5) is precise and transparent for the inference-cost component. Everything else — prompt engineering cost, operational overhead, latency characterization — is absent. The paper does not report wall-clock timing for any experiment, does not discuss rate-limit constraints, and does not cost the human effort of prompt development (which was substantial, per the prompt ablation in Section 4.3 and Appendix A.1.1).

Mitigation status. Partially acknowledged. The paper notes that "many factors such as model choice, serving platform, and more can impact speed" (Section 4.5) but does not measure or model these factors. The cost comparison is transparent about what it includes (per-token API pricing) and precise in its computation, which is appropriate — the limitation is not in the accuracy of the reported 7× figure but in its completeness as a picture of total cost of ownership. A practitioner should treat the 7× figure as an API cost ratio, not a total cost ratio.

Limitation 6: No Statistical Significance Testing — Rank Orderings and κ Differences May Be Within Sampling Noise

The assumption or constraint. The paper reports all results as point estimates: Cohen's κ values (Table 1, Appendix Tables 5–6), Kendall's τ and Pearson r values (Table 2), and accuracy deltas (Figures 3–4). No confidence intervals, standard errors, bootstrap estimates, or hypothesis tests are reported for any pairwise comparison. This is particularly consequential because the test sets are relatively small: the KILT datasets typically contain hundreds to low thousands of test instances, Bamboogle contains only 125 questions (Section A.1.2), and Chatbot Arena Hard contains 500 prompts (Section 3.5). Differences in κ of 0.02–0.05 between judges on datasets with a few hundred instances may be within sampling error.

The consequence. The paper's central ranking claims — "PoLL has the strongest correlation," "GPT-4 is one of the weaker evaluators" — are based on point estimates whose statistical reliability is unquantified. A practitioner considering whether to switch from GPT-4 to PoLL cannot assess whether the observed improvement (e.g., a κ difference of 0.03 on NQ) is likely to replicate on a different test set or is a sampling artifact of the specific test instances used. The paper relies on pattern consistency across datasets (PoLL wins on all six) rather than statistical significance to support its claims. This is a reasonable argument — the probability of PoLL being the top performer on six independent datasets by chance alone is low if judges are equally good — but it does not replace formal inference for individual comparisons.

The small test set sizes create particular vulnerability for the per-dataset breakdowns. Bamboogle (125 questions) and Chatbot Arena Hard (500 prompts) provide limited statistical power. The difference between a Kendall's τ of 0.85 and 0.80 on 500 pairs may not be significant, making the claim that "PoLL is best correlated with the gold rankings" (Section 4.2) less definitive than the point estimates suggest.

Additionally, the paper reports no uncertainty around the bias deltas in Figures 3 and 4. The scatter plots show per-model delta values, but without error bars, a practitioner cannot tell whether the differences between PoLL and individual judges for a specific test model are reliable or within the noise. The reported standard deviation of 2.2 for PoLL's deltas vs. 6.1 for GPT-3.5 (Section 4.4) is a useful summary statistic, but the standard deviation itself has sampling uncertainty that is not reported.

What evidence exists in the paper. The paper provides point estimates with consistent directional patterns across datasets, which strengthens informal confidence but does not constitute formal statistical evidence. The aggregation function robustness check (footnote 4, identical rankings with different voting methods) provides some evidence that the Chatbot Arena rankings are stable to that specific variation, but this is not a significance test. No bootstrap, permutation test, or Bayesian analysis is reported anywhere.

Mitigation status. Not addressed. The paper does not discuss statistical uncertainty, does not provide confidence intervals, and does not justify the omission. In fairness, this is common practice in the LLM-as-judge literature — Zheng et al. (2024) and Li et al. (2024a) similarly report point estimates without formal inference — but it is a limitation nonetheless, particularly for a paper whose contribution is a comparative claim (PoLL > GPT-4) rather than an absolute benchmark. A practitioner making a costly decision to switch evaluation infrastructure from a single judge to a multi-model panel would reasonably want to know whether the observed improvement is statistically reliable at their evaluation scale.

The consistent pattern across six datasets provides some informal protection against the worst-case interpretation (that all observed differences are noise), but without significance testing, the paper cannot distinguish between "PoLL is meaningfully better than GPT-4" and "PoLL and GPT-4 are roughly equivalent, and the observed differences reflect finite-sample variation." The paper's evidence favors the former interpretation, but the distinction is unquantified.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the LLM evaluation conversation from a capability-maximization paradigm (find the most capable judge model you can afford) to a bias-decorrelation paradigm (find a set of judges whose errors are maximally independent). This is a conceptual reframing, not a new model architecture or training procedure, but its practical implications are substantial: the field's default assumption — that GPT-4 is the best available evaluator and that evaluation quality scales monotonically with judge capability — is empirically falsified on the tested tasks. The paper demonstrates that diversity across disjoint model families compensates for individual judge weakness, producing evaluations that correlate more strongly with human judgments than any single larger model tested.

The magnitude of this shift depends on how the finding generalizes. Within the tested scope (factoid QA and preference ranking), the shift is actionable immediately: a researcher evaluating LLMs on these task types can replace GPT-4 with a three-model PoLL today and expect better human correlation at roughly one-seventh the API cost. The paper's results in Tables 1 and 2 — where PoLL achieves the highest Cohen's κ on every QA dataset and the highest rank correlation on Chatbot Arena — provide the empirical warrant for this switch. The fact that no single judge model is universally best across datasets (evident from the shifting "second-best" entries in Table 1) means that single-judge evaluation is inherently fragile to task and dataset variation, while the PoLL provides consistent top-tier performance without requiring the researcher to solve the judge-selection problem.

Beyond the immediate practical shift, the paper reconciles a tension in the LLM-as-judge literature that was previously unresolved. Panickssery et al. (2024) demonstrated that LLM evaluators exhibit systematic self-preference bias, and Zheng et al. (2024) showed that LLM judges can be effective evaluators, but the field lacked a framework for obtaining the benefits of model-based evaluation without the documented biases. This paper shows that the solution is not to fix individual judges (through better prompts, fine-tuning, or bias-correction techniques) but to structural change the evaluation protocol: pool judgments across diverse models whose biases are decorrelated by construction. The bias reduction demonstrated in Figures 3 and 4 — where the PoLL achieves the smallest standard deviation of accuracy deltas relative to human ground truth (2.2 vs. 6.1 for GPT-3.5) — provides quantitative validation that this structural approach works. This shifts the research agenda away from improving individual judge models and toward understanding and optimizing panel composition.

The paper also changes the economics of who can participate in rigorous LLM evaluation. Prior to this work, high-quality model-based evaluation (the kind that correlates well with human judgments and avoids the systematic errors of string-matching metrics) was gated on access to GPT-4, which is expensive and controlled by a single provider. The demonstration that a panel of smaller, cheaper models — including models from multiple providers — can outperform GPT-4 means that resource-constrained academic labs, independent researchers, and organizations without OpenAI API access can now deploy evaluation infrastructure that is both cheaper and better. This is a democratization effect: evaluation quality is no longer a function of how much you can spend on a single proprietary API, but of how thoughtfully you can compose a diverse panel from available models.

A research direction that becomes more attractive in light of this work is understanding error correlation structures across LLMs. If the mechanism driving PoLL's advantage is indeed decorrelated errors across model families, then the next step is to characterize those error correlations directly — both within families and across families — to enable optimal panel construction. This requires large-scale annotation of judge model judgments across many tasks, followed by correlation analysis. A direction that becomes less attractive is prompt-engineering individual judges to eke out marginal improvements in evaluation quality. The paper's prompt sensitivity experiments (Section 4.3) show that prompt optimizations are model-specific (what helps GPT-4 harms GPT-3.5), and that even with extensive prompt engineering, GPT-4 only reaches parity with smaller models on the tested tasks. The return on investment for single-judge prompt engineering is low compared to the return on investment for panel diversification, which requires no per-model optimization and provides larger gains.

Follow-Up Research This Work Enables

Panel composition ablation: what drives the benefit — diversity, multiplicity, or these specific models? The paper evaluates exactly one panel (Command R + Haiku + GPT-3.5) and provides no evidence about which properties of that panel are necessary for its advantage. A direct follow-up would compare panels varying along three axes: (a) panel size (1, 2, 3, 5 judges) to measure the diversity-vs-cost tradeoff curve; (b) within-family vs. across-family composition (e.g., a panel of GPT-3.5 + GPT-4 + GPT-4-0314 vs. the default cross-family panel) to isolate whether model family diversity drives the benefit or whether any three models would work; (c) substituting individual panel members (e.g., replacing Command R with Mistral Small or an open-weight model) to test sensitivity to specific model choice. Running this on the same six datasets with the same human correlation metrics would produce a practical guide for panel construction. The paper makes this tractable by establishing the evaluation protocol and baseline numbers — a follow-up needs only to run alternative panel compositions through the same pipeline.

Stress-test on hard evaluation tasks: math, reasoning, and code generation. The paper explicitly limits its claims to factoid QA and preference ranking (Section 5), noting that math and reasoning evaluations are unverified. A direct stress-test would replicate the PoLL protocol on evaluation tasks where the judging itself requires complex reasoning: grading math proofs on MATH (Hendrycks et al., 2021), verifying code correctness on HumanEval or MBPP (where unit tests provide ground truth but LLM judges are also used), or evaluating multi-step logical reasoning on datasets like FOLIO or ARC. The key question is whether the diversity benefit persists when the judging task is near or beyond the capability ceiling of the panel members — if all three smaller models fail in correlated ways on hard reasoning tasks, the PoLL will not outperform a single stronger model that can actually perform the reasoning. This experiment would characterize the capability ceiling for panel evaluation and determine whether panel diversity or individual capability dominates as judging difficulty increases. A strong follow-up would measure both human correlation and judge accuracy (since ground truth is known for math/code problems) to separately assess whether PoLL improves evaluation reliability or merely exchanges one set of errors for another.

Aggregation function comparison for binary judgments: max voting vs. majority voting vs. precision-recall tradeoffs. The paper uses max voting for QA (any judge's "True" makes the answer correct) without empirical comparison to alternatives. A systematic follow-up would compare max voting, majority voting (at least two "True"), unanimity (all three "True"), and soft aggregation (continuous scores from each judge, thresholded) on the same QA datasets with the same human ground truth. For each aggregation function, the study should report precision, recall, F1, and Cohen's κ relative to human judgments, separately analyzing whether the observed κ differences in Table 1 are driven by leniency (max voting inflating recall at the expense of precision) or by genuine improvement in semantic judgment quality. This experiment is enabled by the paper's human annotation data and judge outputs — a researcher needs only to apply alternative aggregation functions to the same individual-judge decisions, requiring no new model inference or annotation.

Dynamic panel composition: does a fixed panel work, or should the panel adapt per-question? The paper treats the PoLL as a static panel — the same three judges evaluate every instance. But the paper's finding that no judge is universally best (Table 1 shows different judges performing best on different datasets) hints that question-level variation exists. A follow-up experiment would add a difficulty estimation or judge-selection step: for each test instance, use a lightweight classifier (or the judges' own initial outputs) to predict which judge is most reliable for that instance, then weight that judge more heavily in aggregation. The experiment would compare adaptive weighting against the static max-voting PoLL on the same datasets, measuring whether instance-level adaptation provides gains beyond the already-strong panel baseline. The paper makes this tractable by providing per-instance judgments from all three judges across all datasets — a researcher can analyze the correlation structure of per-instance judge errors to determine whether predictable patterns exist that an adaptive policy could exploit.

Open-weight panel construction: can PoLL work with fully open models? The paper's PoLL uses three proprietary models, which limits replicability and accessibility for researchers without API access to all three providers. A practical follow-up would construct a PoLL from open-weight models (e.g., Llama 3, Mixtral, Gemma, Qwen) from different model families, replicate the same evaluation protocol on the same datasets, and measure whether the diversity benefit transfers to the open-weight ecosystem. This experiment is newly significant because it would test whether the "disjoint model families" principle extends to models where training data, architecture, and fine-tuning recipes are partially transparent — if open-weight models from different origins show sufficient diversity to replicate PoLL's advantage, the evaluation infrastructure becomes fully reproducible and cost-free beyond compute. A strong follow-up would compare multiple open-weight panels varying by model family composition, producing a practical recipe for open-source LLM evaluation.

Human inter-annotator agreement and the performance ceiling. The paper does not report human inter-annotator κ for its triple-annotated datasets (NQ and single-hop HPQA), leaving the performance ceiling unknown. A direct follow-up would compute inter-annotator κ from the individual annotator judgments (which the authors presumably have from the triple-annotation protocol described in Appendix A.3), report this number alongside the PoLL and judge κ values, and analyze whether PoLL is at 80%, 90%, or 95% of the human ceiling. If PoLL is near the ceiling, further investment in panel optimization yields diminishing returns; if a substantial gap exists, the ceiling analysis would indicate how much headroom remains and whether closing it requires more judges, different judges, or better aggregation. This is a straightforward analysis requiring only the annotation data the paper already collected, and it would substantially contextualize the reported results.

Practical Applications and Downstream Use Cases

Cost-efficient batch model evaluation for research and development. A research group developing or fine-tuning LLMs needs to evaluate model quality across many checkpoints, datasets, and ablations. Replacing GPT-4-based evaluation with a PoLL reduces per-evaluation API cost by approximately 7× (Section 4.5: $5.50 total per million tokens for PoLL vs.$ 40 for GPT-4 Turbo). For an evaluation pipeline that processes 10 million tokens across all test sets (a modest number for multi-dataset evaluation of several model variants), the cost drops from roughly $400 to$ 55 per run. Over the course of a project involving dozens of evaluation runs, this saves thousands of dollars while simultaneously improving evaluation quality — a rare case where the cheaper option is also the better one. The PoLL's parallel architecture (three models queried simultaneously) also reduces wall-clock latency relative to a single large model, accelerating the evaluation feedback loop during iterative development.

Vendor-neutral model selection for production deployments. An organization evaluating which LLM to integrate into a customer-facing product needs unbiased, human-aligned quality assessments to make informed procurement decisions. Using a single vendor's model as the judge (e.g., GPT-4 evaluating GPT-family models alongside competitors) introduces self-preference bias that can systematically inflate the apparent performance of that vendor's models — the paper quantifies this as a two-position rank inflation on Chatbot Arena Hard (Figure 2). Deploying a PoLL with judges from three disjoint model families eliminates this conflict of interest: since no single judge's family has majority control over the evaluation, self-preference bias in any individual judge is washed out by the other two judges from different families. For a company spending hundreds of thousands of dollars on API inference, an unbiased evaluation that correctly identifies the best-performing model — rather than the model whose family happens to provide the judge — has direct financial consequences on operational costs and user satisfaction.

Automated quality monitoring in production LLM systems. An organization serving an LLM-powered product needs ongoing monitoring of output quality to detect regressions, prompt drift, or model version degradation. Running a full human annotation pipeline continuously is prohibitively expensive; using a single GPT-4 judge for monitoring is cheaper but concentrates trust in one system that may itself change behavior over time (model updates, prompt sensitivity, API version changes). A PoLL provides a more robust monitoring signal because its aggregation across three independent judges from three providers is less sensitive to any single judge's drift. If one provider updates their model in a way that changes its judging behavior, the other two judges anchor the evaluation, making the monitoring system more stable. The 7× cost reduction relative to GPT-4 means continuous monitoring at meaningful scale (thousands of outputs per day) becomes economically viable for mid-size organizations that could not previously afford LLM-based evaluation.

When to Prefer This Method

The paper positions PoLL as a replacement for single large judge models, specifically GPT-4, in LLM-based evaluation. The decision matrix is straightforward based on the paper's empirical findings and acknowledged limitations:

Prefer PoLL over a single GPT-4 judge when: (a) the evaluation task is factoid QA with reference answers or pair-wise preference ranking — the only settings with empirical validation (Tables 1, 2); (b) the evaluation budget is constrained and API cost matters — PoLL is approximately 7× cheaper (Section 4.5); (c) the test models include models from the same family as any candidate judge — PoLL's cross-family composition reduces intra-model bias that a single GPT-4 judge would exhibit when evaluating GPT-family models (Figures 3–4, Figure 2); (d) the evaluation needs to be robust to prompt sensitivity — PoLL's diverse panel is less vulnerable to the prompt brittleness that causes substantial κ swings for GPT-4 alone (Section 4.3, Table 3); (e) consistent performance across multiple datasets without per-dataset judge selection is desired — PoLL is consistently best or second-best across all six tested datasets, while individual judge rankings shift (Table 1).
Prefer a single large judge (GPT-4 or equivalent) when: (a) the evaluation task involves math, reasoning, code generation, or other judgments requiring complex multi-step verification — the paper provides zero evidence that PoLL works in these settings and explicitly flags them as unvalidated (Section 5); (b) operational simplicity is paramount and managing API access, rate limits, and prompt templates for three different model providers is unacceptable overhead — a single judge requires maintaining one provider relationship (Section 6 limitation on infrastructure cost); (c) the evaluation requires fine-grained calibrated confidence scores rather than binary or ranked judgments — max voting introduces a leniency bias whose calibration properties are unknown and unmeasured (Section 6 limitation on max voting).
Open question requiring judgment: when evaluating open-ended generation tasks (summarization, dialogue, creative writing) where reference answers are unavailable and quality is inherently multi-dimensional, it is unknown whether PoLL or a single large judge is preferable. The paper's experiments all involve either reference-based correctness or comparative preference, and neither directly transfers to single-point absolute quality scoring. A practitioner evaluating open-ended generation should treat PoLL as an untested hypothesis in their domain.