Assessing Large Language Models on Climate Information

🎯 Pitch

LLMs appear fluent when answering climate questions, but raters found serious epistemological flaws—like omissions and unqualified certainty—in most responses. A new AI-assisted oversight protocol nearly doubled issue detection, revealing risks that surface quality hides.

1. Executive Summary

This paper introduces a comprehensive evaluation framework, grounded in science communication research, for assessing how LLMs respond to climate change questions. It evaluates several recent LLMs—including GPT-4, ChatGPT, InstructGPT variants, PaLM-2, and Falcon-180B-Chat—on a diverse set of 300 climate questions using human raters with relevant educational background, supported by AI Assistance (a scalable oversight protocol where a model-generated critique, grounded in verbatim Wikipedia evidence for epistemological dimensions, is shown to raters to help them identify issues). The framework spans 8 dimensions: presentational adequacy—style, clarity, linguistic correctness, and tone—and epistemological adequacy—accuracy, specificity, completeness, and uncertainty. The central empirical finding is that while presentational quality is high across models, epistemological quality is substantially lower, and introducing AI Assistance increases recall of detected issues—raters with assistance detected 77% of planted issues by majority vote versus 60% without—establishing that reliable assessment of LLM climate communication requires grounded oversight protocols, not surface-level evaluation alone.

2. Context and Motivation

The Core Problem: We Cannot Reliably Assess Whether LLMs Communicate Climate Science Adequately

As LLMs are increasingly deployed as information sources—through chatbots, search interfaces, and content generation pipelines—a critical question arises: how do we evaluate whether their outputs are adequate for domains where factual accuracy and responsible communication carry high stakes? Climate change is one such domain. The paper identifies a fundamental evaluation gap: existing LLM evaluation paradigms are not designed to assess the multi-dimensional quality of science communication, particularly the interplay between surface-level fluency and deeper epistemological soundness.

This gap is consequential for several reasons the paper establishes in Section 1:

Rising reliance on digital media for climate information: "more and more people turn to digital media as their primary source of information," yet "in spite of ubiquitous access to information, there remains a considerable gap in climate literacy, exacerbated by the spread of mis- and disinformation" (Section 1). If LLMs are to serve as information gateways—answering user questions about climate change—their outputs must be evaluated on dimensions that go beyond simple factual accuracy to encompass how scientific uncertainty is communicated, whether answers are complete enough to inform rather than mislead, and whether the tone avoids undermining credibility.
The challenge of scientific communication itself: Climate science is "laden with specialized knowledge, complexity, and inherent uncertainties" (Section 1). Communicating it effectively requires balancing comprehensibility with precision, acknowledging what is known versus what remains uncertain, and avoiding both oversimplification and jargon. An LLM that produces grammatically flawless but epistemologically inadequate answers—e.g., failing to communicate scientific consensus, omitting crucial caveats, or presenting contested findings as settled fact—can misinform users who trust its fluent output. The paper cites Chiesurin et al. (2023): "Eloquence and advanced dialogue behaviors are trusted by users, even in the absence of trustworthy information," which makes surface-level evaluation dangerously insufficient.
Scalable oversight becomes necessary: The paper frames this as an instance of the scalable oversight problem (Amodei et al., 2016): as models become more capable, evaluating their outputs becomes harder, potentially exceeding what unaided human raters can reliably do, especially in domains requiring specialized knowledge. The climate domain exemplifies this: raters need enough scientific literacy to catch subtle epistemological failures (e.g., a statement that is factually true but misleadingly incomplete, or one that omits critical uncertainty qualifiers), but must also resist being swayed by superficial fluency. The paper notes that "research on evaluating systems that may achieve or exceed human abilities... is so far mostly theoretical... with some recent more practical advances" (Section 1), positioning this work as a practical contribution to scalable oversight in a high-stakes domain.

Why Existing Evaluation Paradigms Fall Short

The paper identifies three categories of prior work and explains why each is insufficient for the problem at hand.

Factuality-focused evaluation is necessary but not sufficient. Standard LLM evaluation approaches emphasize factuality—whether outputs contain incorrect or hallucinated information (Weidinger et al., 2021; Ji et al., 2023). While accuracy is one dimension of the paper's framework (under epistemological adequacy), the authors argue that factual correctness alone misses critical dimensions of science communication quality. Consider an LLM that answers "Are climate change solutions too expensive?" with a factually accurate statement about the costs of renewable energy investment but fails to mention the costs of inaction, or that presents the economic argument without acknowledging uncertainty ranges in cost projections. Such an answer might pass a factuality check but would score poorly on completeness and specificity—dimensions that the paper's framework explicitly captures. The authors also note that "the digital media landscape, characterized by soaring amounts of AI-generated content... limited attention spans and adversarial dynamics, further compounds these challenges" (Section 1), suggesting that evaluation must account for how information is presented, not just whether it is technically correct.

Attribution-based evaluation (AIS) is orthogonal, not a substitute. A prominent approach to evaluating LLM factuality is Attribution to Identified Source (AIS) (Rashkin et al., 2022, 2023; Bohnet et al., 2023), which assesses whether generated statements can be grounded in retrieved documents. The paper runs a dedicated AIS experiment (Section 4.6) and finds that "AIS and epistemological assessments are mostly orthogonal"—the Spearman's coefficient between AIS scores and epistemological dimension ratings are near zero (0.03, −0.06, 0.002, −0.02 for accuracy, specificity, uncertainty, and completeness, respectively, with p-values all above 0.30). The paper provides concrete examples (Table 19) of answers that are fully attributable yet score poorly on epistemological dimensions: an answer about when the public became aware of CO2's effects scores only 1.6/5 on completeness despite being fully attributable, because it omits key milestones. Conversely, an answer that cannot be attributed to the retrieved evidence can still score high on accuracy if the evidence retrieval was simply insufficient. This finding is significant because it challenges the implicit assumption in some RALM (Retrieval Augmented Language Model) work that attributability is a sufficient proxy for response quality. The paper demonstrates that attribution evaluation "can only consider what is included in the answers, and not what important information is missing" (Section 4.6).

Domain-specific evaluation frameworks exist but don't cover climate communication. The paper acknowledges that expert-level evaluation has been developed for the medical domain, notably the clinical evaluation framework introduced with Med-PaLM (Singhal et al., 2023), which covers criteria like alignment with scientific consensus, potential harm, and comprehension. This work demonstrates that domain-specific evaluation is both feasible and necessary—medical information requires different quality criteria than general text generation. However, the paper notes that "prior work [on climate and NLP] mainly focused on text classification tasks" (Section 6, Related Work)—e.g., detecting climate claims (Diggelmann et al., 2020), classifying climate-related text (Varini et al., 2020; Webersinke et al., 2022), or assessing corporate climate disclosures (Bingler et al., 2022). There is no equivalent comprehensive evaluation framework for generative climate information, despite the growing deployment of LLMs in this role. The paper explicitly states that it "aims to fill this gap by providing a comprehensive evaluation framework for generative climate information" (Section 6).

Scalable oversight has theoretical proposals but limited practical validation. The paper situates its AI Assistance protocol within the scalable oversight literature, which asks how human evaluators can supervise AI systems that may match or exceed their own capabilities. Prior work in this area has been mostly theoretical—proposals for debate (Irving et al., 2018), amplification (Christiano et al., 2018), and reward modeling (Leike et al., 2018)—with limited practical validation in realistic settings. The paper cites some recent practical advances: Bowman et al. (2022) showed non-expert raters perform better on a difficult multiple-choice task when interacting with a dialogue agent, and Saunders et al. (2022) found that human raters of summarization tasks produce more critiques when given the opportunity to accept or edit model-written critiques. However, the paper notes that "several studies also evaluated short debates in this setting with mixed results" (Parrish et al., 2022a;b; Michael et al., 2023), indicating that scalable oversight protocols are not yet reliably effective across domains. The paper's contribution is a practical protocol—AI Assistance grounded in verbatim evidence—whose effectiveness is empirically tested in a domain (climate science communication) where raters need specialized knowledge and the evaluation dimensions are multi-faceted.

How This Paper Positions Itself

The paper frames its contribution not as a new model or training technique, but as an evaluation methodology informed by a body of scholarship—science communication research—that the NLP community has not systematically engaged with. Section 2 is entirely devoted to developing the evaluative dimensions from this literature, citing work from communication science, sociology, psychology, and education (Trench & Bucchi, 2021; Nisbet et al., 2018; Jamieson et al., 2017).

The key intellectual move is the distinction between presentational and epistemological adequacy. This distinction is not merely taxonomic; it directly motivates the paper's experimental design and its central finding that these two categories of quality are not correlated—models can excel at presentation while failing on epistemological soundness, and the paper even finds preliminary evidence of a trade-off: "Better performance on 'knowledge' comes at the cost of worse quality in the presentational dimensions" when prompts are augmented with evaluation criteria (Section 4.1, Table 1). This trade-off would be invisible to evaluation frameworks that either focus only on factuality or conflate surface quality with overall quality.

The paper also positions its AI Assistance protocol carefully: it is not claiming that AI can replace human raters (in fact, Section 4.5 shows that raters can "think critically about the assistance and do not follow it blindly"), nor is it claiming that the assistance is unbiased (the paper explicitly acknowledges that "the assistance may also influence the raters beyond enhancing discovery" and that "there is definitely a need to better understand these issues and to identify mitigation strategies," Section 5). Rather, the protocol is presented as a practical, empirically validated method for increasing the recall of detected issues—a necessary component of scalable oversight in domains where unaided raters miss important problems.

Finally, the paper is careful about scope. It does not claim to define a single quality metric that combines all dimensions ("we don't address the important question of how individual dimensions should be combined in a single metric... This is a complex topic which requires assigning a value to each individual dimension. We also believe that the combination of these scores will vary by application," Section 2.3). This is a deliberate choice: the framework is designed to surface the multi-dimensional nature of science communication quality, not to collapse it into a single number that might obscure important trade-offs. This positions the paper as a foundation for future work on climate-aware LLM evaluation, rather than a final benchmark.

3. Technical Approach

3.1 Reader Orientation

This paper builds an evaluation framework—a structured protocol for human raters to assess the quality of LLM-generated answers to climate change questions—rather than building a new model or training procedure. The core problem it solves is that standard evaluation methods (factuality checks, attribution to sources) are insufficient for science communication: they miss critical dimensions like how uncertainty is conveyed, whether answers are complete enough to inform rather than mislead, and whether the tone undermines credibility—all of which matter when users trust fluent LLM outputs as information sources.

3.2 Big-Picture Architecture

The evaluation system has five major components:

Question Set — a diverse collection of 300 climate change questions drawn from three sources (Google Trends, Skeptical Science, and GPT-4-generated questions from Wikipedia paragraphs) to cover real information needs, debated claims, and context-specific topics.
LLM-Generated Answers — responses from seven recent LLMs (GPT-4, ChatGPT, InstructGPT variants, PaLM-2, Falcon-180B-Chat) produced with a standardized prompt ("Answer each question in a 3-4 sentence paragraph"), providing a controlled baseline for comparison.
AI Assistance Pipeline — an automated system using GPT-4 to extract keypoints from each answer, retrieve relevant Wikipedia evidence, and generate dimension-specific critiques that help human raters identify issues they might otherwise miss.
Evaluative Dimensions — eight quality criteria (style, clarity, correctness, tone, accuracy, specificity, completeness, uncertainty) grounded in science communication research, each operationalized as a specific Likert-scale statement with a defined list of possible issues.
Human Rating Protocol — raters with climate-relevant educational backgrounds, admitted via a tutorial and examination, who assess each answer on all dimensions (on a 1–5 scale, identifying specific issues for scores ≤2), with each answer rated by three raters, both with and without AI Assistance.

Information flows as follows: a question from the question set enters the system → the base LLM generates a 3–4 sentence answer → GPT-4 extracts keypoints and retrieves supporting Wikipedia evidence → GPT-4 generates dimension-specific critiques (the AI Assistance) → human raters, who may or may not see this assistance, rate each of the eight dimensions → ratings are aggregated across three raters per answer to produce per-model, per-dimension mean scores.

3.3 Roadmap for the Deep Dive

First, the evaluative dimensions and their operationalization—how each of the eight criteria is defined from science communication literature and translated into a concrete rating statement with specific issue types that raters can identify. This is the conceptual core of the framework.
Second, the question data pipeline—how 300 diverse climate questions were sourced, filtered, and sampled. Understanding question diversity is essential because evaluation quality depends on covering the spectrum of real information needs.
Third, the AI Assistance pipeline—how keypoints are extracted, evidence is retrieved from Wikipedia, and dimension-specific critiques are generated. This is the novel scalable oversight protocol, and its mechanics directly affect what raters see and how they perform.
Fourth, the human rating protocol—the rater qualification process (tutorial + admission test), the rating interface, the aggregation of three raters per answer, and the experimental design for measuring the effect of AI Assistance.
Fifth, the attribution (AIS) comparison experiment—how the framework relates to established source-attribution evaluation.
Sixth, the validation methodology—how the framework was stress-tested with synthetic examples containing known issues, providing evidence that raters, especially with assistance, can detect genuine problems.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodology and empirical evaluation paper whose core idea is that evaluating LLM climate communication requires: (1) a multi-dimensional framework grounded in science communication research that separates presentational from epistemological adequacy, (2) human raters with relevant education, and (3) a grounded AI Assistance protocol that increases the recall of detected issues without fully replacing human judgment.

The Eight Evaluative Dimensions: Definitions and Operationalization

The framework organizes quality into two broad categories—presentational adequacy (how the message is conveyed) and epistemological adequacy (how well the message represents scientific knowledge)—with four dimensions in each category. Each dimension is grounded in specific findings from the science communication literature, operationalized as a Likert-scale agreement statement (1 = disagree completely, 5 = agree completely), and paired with a predefined menu of specific issues that raters can flag when they disagree (scores ≤2). The full statements and issue lists appear in Table 11.

Presentational Adequacy Dimensions

Style (Statement: "The information is presented well for a general audience.") This dimension assesses overall presentation appropriateness, drawing on findings that language that is too informal or colloquial can undermine credibility (Mazer & Hunt, 2008; Scharrer et al., 2012). It captures several distinct sub-concerns: answer length (too short risks creating a "feeling of knowing" where users falsely believe they understand, per Leonhard et al., 2020; too long requires cognitive resources readers may not invest, per Lang, 2000), repetition, and informality of language. The specific issues raters can flag are: too informal/colloquial, answer too long, answer too short, inconsistent language/style/terminology, repetitive, and other (free-text). The paper also borrows stylistic categories from the Multidimensional Quality Metrics (MQM) framework for translation evaluation (Lommel et al., 2013), linking the evaluation to an established quality assessment tradition.

Clarity (Statement: "The answer is clear and easy to understand.") This dimension captures whether the response is comprehensible to a general audience, drawing on the science communication principle that jargon and difficult technical content should be avoided (Baram-Tsabari & Lewenstein, 2013; Baram-Tsabari et al., 2020). It also incorporates guidance that responses should be concise and clearly formulated (Maibach et al., 2023). The specific issues are: sentences too long, language too technical, numbers/formulae hard to understand, and other. This dimension is conceptually distinct from style: a response can be stylistically appropriate (right length, consistent) but unclear due to jargon or convoluted syntax.

Linguistic Correctness (Statement: "The language in the answer does not contain mistakes.") Following MQM conventions, this dimension assesses adherence to linguistic conventions—grammar, spelling, punctuation, and sentence completeness—because violations can damage perceived credibility (Berger, 2020; Mollick, 2014). The specific issues are: sentence is incomplete, spelling mistakes, punctuation mistakes, grammatical errors, and other. The paper's shorthand "correctness" refers exclusively to linguistic correctness, not factual correctness (which falls under the epistemological dimension "accuracy"). This distinction is important for understanding the paper's key finding that presentational correctness is high while epistemological accuracy is substantially lower.

Tone (Statement: "The tone of the answer is neutral and unbiased.") This dimension captures the perceived neutrality, persuasiveness, and valence of the language, drawing on research suggesting that science communication—especially climate-related—is more effective when it avoids leaning toward a particular valence, worldview, or ideological conviction (Blanton & Ikizer, 2019; Yuan & Lu, 2020; Kerr et al., 2022; Munoz-Carrier et al., 2020). Messages should not use overly positive or negatively valenced language when conveying factual information (Palm et al., 2020). The specific issues are: the answer is biased, tries to convince me of an opinion/belief, the tone is too negative, and other. This dimension is notably the lowest-scoring presentational dimension across all models (Figure 2, left), suggesting that even fluent LLMs struggle with the subtler aspects of neutral science communication.

Epistemological Adequacy Dimensions

Accuracy (Statement: "The answer is accurate.") This is the most fundamental epistemological dimension: the information should be factually correct (Kelesidou & Chabrol, 2021). However, the paper's operationalization goes beyond simple fact-checking to capture common failure modes of LLM-generated science information: taking scientific findings out of context (Hinnant et al., 2016), self-contradiction within the answer, reliance on anecdotal evidence rather than scientific consensus, and misuse of key terms or scientific terminology. The specific issues are: incorrect, takes scientific findings out of context, self-contradictory, anecdotal, wrong use of key terms/scientific terminology, and other. The inclusion of "anecdotal" and "science out of context" as distinct issues is notable: an answer can be factually correct in isolation but misleading because it omits the broader scientific context, and the framework explicitly captures this.

Specificity (Statement: "The answer addresses only what the question asks for, without adding irrelevant information.") This dimension captures whether the answer is targeted versus including unnecessary or vague content. The underlying principle from the literature: information should be relevant to the audience, with spatial and temporal context addressed specifically (Lee et al., 2015; Leiserowitz & Smith, 2017; Holmes et al., 2020). The specific issues are: includes irrelevant parts, too vague/unspecific, and other. An answer that is accurate but padded with loosely related generalities about climate change (what the paper identifies as a common LLM behavior in Section 4.4, calling it "somewhat relevant but unspecific, cautionary and sometimes preachy reminders") would score poorly on this dimension even if nothing said is factually wrong.

Completeness (Statement: "The answer addresses everything the question asks for.") While specificity penalizes extra information, completeness penalizes missing information. The dimension draws on the principle that answers should address all aspects of the question in full (Leiserowitz & Smith, 2017; Bergquist et al., 2022) and reflect the depth and breadth of relevant scientific knowledge (Kelesidou & Chabrol, 2021). The specific issues are: misses important parts of the answer, does not address the region the question asks about, does not address time or time range the question asks about, does not give enough detail (e.g., numbers, statistics, details), ignores relevant scientific knowledge, and other. This fine-grained breakdown—region, time, detail, scientific knowledge—means that raters must check four distinct types of omission, which requires both careful reading of the question and sufficient domain knowledge to recognize what is missing.

Uncertainty (Statement: "The answer appropriately conveys the uncertainty involved.") This dimension captures a critical aspect of science communication that is often overlooked in standard NLP evaluation: whether the answer communicates the level of agreement, confidence, and supporting evidence appropriately (Howe et al., 2019; Budescu et al., 2012; Keohane et al., 2014). This is particularly important in climate communication because communicating scientific consensus has been found to function as a "gateway belief" that motivates public action (van der Linden et al., 2015), and failing to acknowledge uncertainty where it exists can mislead audiences about the state of scientific knowledge (Maertens et al., 2020; Chinn & Hart, 2021; Goldberg et al., 2022). The specific issues are: degree of (un)certainty not given when it should be, agreement in the scientific community not given when important, contradicting evidence (if existing) not mentioned, and other. Note the qualification "when it should be" and "if existing"—raters are asked to judge not just whether uncertainty is mentioned, but whether the answer's treatment of uncertainty is appropriate given the scientific context, which requires domain knowledge.

Why this structure over alternatives. The two-category, eight-dimension structure is not arbitrary. The presentational/epistemological distinction is motivated by the science communication literature's identification of two fundamental communication functions: comprehensibility (does the audience understand the message?) and representation of knowledge (does the message accurately reflect what is scientifically known?). The paper cites Lang (2000) for the former and Fähnrich et al. (2023) for the latter as anchoring references. By evaluating these separately, the framework can detect cases where they diverge—a fluent but misleading answer, or an accurate but poorly communicated one—which a single aggregated quality score would mask. The specific issues within each dimension provide a structured way for raters to justify low scores, which both improves consistency (raters are prompted to consider specific failure modes) and yields diagnostic information (researchers can see which types of issues specific models produce, as in Table 7).

The paper explicitly declines to combine dimensions into a single metric (Section 2.3): "we don't address the important question of how individual dimensions should be combined in a single metric... This is a complex topic which requires assigning a value to each individual dimension. We also believe that the combination of these scores will vary by application." This is a deliberate methodological choice reflecting the view that climate communication quality is inherently multi-dimensional and application-specific—a conversational agent may prioritize different dimensions than a static information page.

The Question Data Pipeline: Sourcing, Filtering, and Sampling 300 Climate Questions

The evaluation requires questions that represent genuine information needs about climate change. The paper constructs a 300-question dataset drawn equally (100 each) from three sources designed to capture different types of information needs.

Source 1: Google Trends (real-world search queries). The authors use Google Trends to collect the most popular questions by search volume from the United States for the topics "Climate Change" and "Global Warming" during 2020–2022. This source captures what people actually search for—the revealed information needs of the public—including both basic factual questions and potentially misinformed queries. Using search volume data anchors the evaluation to real-world usage patterns rather than researcher-specified test questions.

Source 2: Skeptical Science (debated claims). Skeptical Science is a website that publishes authoritative information about climate science and maintains a list of debated myths about climate change. The authors take this list of myths and manually rephrase them as questions. This source captures claims that are contested in public discourse—precisely the type of questions where high-quality communication is most important for countering misinformation. The manual rephrasing ensures the questions are natural rather than artifacts of the debate format.

Source 3: Synthetic questions from Wikipedia (context-specific topics). This source is generated through a multi-step pipeline designed to produce diverse, contextually grounded questions:

Article selection. The authors gather Wikipedia articles related to climate change using three strategies:
- REF: Articles referenced in the main "Climate Change" Wikipedia article (35 articles yielding 858 paragraphs).
- CAT: Articles directly listed in the Wikipedia climate change category (46 articles yielding 434 paragraphs).
- REG: Manually curated articles with titles "Climate Change in [country/region]" (48 articles yielding 677 paragraphs). This strategy specifically targets regional information needs, which are important because "specific, local information leads to higher perceived relevance" (Lee et al., 2015; Leiserowitz & Smith, 2017; Holmes et al., 2020).
The total corpus comprises 129 articles broken into 1,969 paragraphs (only paragraphs exceeding 500 characters are retained).
Question generation. Each selected paragraph is input to GPT-4 with the instruction to generate as many questions as possible that can be answered using that paragraph, with the constraint that questions must be "salient and related to climate change." This process generates 15,265 initial questions.
Multi-filter post-processing. The initial questions undergo four successive filters to remove undesirable ones:

Climate Change Filter. Questions not related to climate change are removed using the climate-bert classifier (Webersinke et al., 2022), which labels each question as climate-related or not. This removes 2,647 questions.

Duplicate Filter. Near-duplicate questions are removed by embedding all questions with a universal sentence encoder and treating pairs with cosine similarity above 0.85 as duplicates. This removes 1,188 questions.

Context Dependent Filter. Questions that are nonsensical without their accompanying Wikipedia paragraph (e.g., "What are the two classes of climate engineering discussed in the study?" without identifying the study) are removed. The authors build a dedicated classifier for this: they manually annotate 100 questions as context dependent or not, contextualize each question with the instruction "Write Yes if the query is taken out of context, write No otherwise," extract last-layer representations from a flan-xxl encoder (Chung et al., 2022) using in-context probing (Amini & Ciaramita, 2023), and train a logistic regression probe on these representations. The resulting classifier achieves 97% accuracy on a 100-question validation set and removes 552 questions.

Specificity Filter. Questions that ask about very specific, narrow details (e.g., "What was the reason for shutting down reactor number one of the Fessenheim Nuclear Power Plant on 4 August 2018?") are removed because they are typically easy to answer and do not represent the "challenging and multifaceted questions that target information needs of users." Again, a dedicated in-context probing classifier is built on flan-xxl representations, using the instruction "Write Yes if the following query is asking about a specific subject, write No otherwise," achieving 84% accuracy on a 100-question validation set. This filter removes 5,472 questions.

After all four filters, 5,404 questions remain.
Stratified sampling for representation. To ensure the final 100 selected questions cover diverse topics and question types, the authors label each question with two features:

Topic classification. An in-context probing classifier assigns questions to one of nine IPCC-inspired topics: Energy, Emissions-Pollutants, Policies-Mitigation-Adaptation, Weather-Temperature, Land-Ocean-Food-Water, Society-Livelihoods-Economy, Health-Nutrition, Biodiversity, and Cities-Settlements-Infra. The classifier achieves 75% accuracy on a 100-question validation set (the moderate accuracy reflects the inherent overlap between some climate topics).

Causal/Prediction classification. A second classifier identifies whether the question asks about causes, effects, or predictions about the future, using the instruction: "Write Yes if the following query is asking about causes or effects of something, or is asking about predictions about the future. Write No otherwise." This classifier achieves 93% accuracy on a 100-question validation set.

The 100 synthetic Wikipedia questions are sampled by taking 6 questions from each of the 18 possible combinations of topic (9) and causal-prediction feature (2), then manually inspecting all selected questions. The remaining two sources (Skeptical Science, GTrends) each contribute 100 questions for a total of 300. The paper reports that a manual inspection of all questions from all sources was performed, and that post-processing removes any remaining duplicates, non-climate-related questions, or context-dependent questions.

Why this question construction approach? The paper explicitly notes that "no standardized tests exist to assess climate-related knowledge; in contrast to e.g., the medical domain" (Section 3.1.1). The three-source design is a pragmatic response to this gap: GTrends captures real user queries, Skeptical Science captures contested claims where high-quality communication matters most, and Wikipedia-generated questions capture diverse, specific, and regional topics that might not arise in search trends but represent important climate knowledge domains. The stratified Wikipedia sampling ensures that the evaluation is not biased toward a particular subtopic or question type. Manual inspection of all selected questions provides a final quality check that automated filters might miss.

The paper reports in Appendix A.13 that comparing rating outcomes by question source reveals "no major differences, except for slightly lower scores for Wikipedia questions," which may be because "these questions ask for more details and very specific info compared to GTrends and Skeptical Sciences, and thus are harder to answer." This heterogeneity is actually desirable: it means the question set captures a realistic range of difficulty.

The AI Assistance Pipeline: Keypoint Extraction, Evidence Retrieval, and Critique Generation

This is the paper's central methodological innovation—a scalable oversight protocol that uses an LLM (GPT-4, for consistency) to generate dimension-specific critiques that help human raters identify issues they might otherwise miss. For epistemological dimensions, the critiques are grounded in verbatim evidence from Wikipedia passages, which is crucial because it gives raters something concrete to verify rather than having to trust the AI's judgment.

The pipeline (illustrated in Figure 1) proceeds through four stages:

Stage 1: Keypoint Extraction. Given a question-answer pair, GPT-4 is instructed to identify "1 to 3 key statements that are made to answer the question" and to "copy the statements verbatim from the answer" (Table 8, "Extract Keypoints" prompt). The paper argues that this provides "better signal to retrieve evidence than either using the whole answer or all sentences individually" (Section 3.2), citing Liu et al. (2023). The keypoints serve as focused queries for the evidence retrieval step: rather than retrieving passages relevant to the entire answer (which may include tangential content), the system retrieves passages relevant to each specific claim, increasing the precision of the evidence grounding. If no keypoints can be extracted, the model outputs "No Keypoints."

Stage 2: Evidence Retrieval from Wikipedia. For each keypoint, the system retrieves relevant Wikipedia evidence through a process that avoids off-the-shelf retrieval systems in favor of an LLM-driven approach the authors find more effective:

URL generation. Given the question and answer, GPT-4 is asked to "provide a Wikipedia article that supports your answer. Just state the url, do not include additional text. If there is no Wikipedia url supporting the answer just say 'No URL'" (Table 8, "Obtain URL" prompt). The paper justifies Wikipedia as the evidence source by noting that "GPT-4 is fairly consistent in generating relevant, valid Wikipedia URLs, while the quality is lower for the unrestricted web" and that "Wikipedia is uniform in style and quality as it adheres to established guidelines" (Section 3.2). For 87.7% of questions, GPT-4 produces a valid Wikipedia URL from which evidence passages can be extracted.
Paragraph scoring. The identified Wikipedia article is broken into paragraphs. For each keypoint, GPT-4 scores each paragraph on a scale from 0 to 100 based on "how useful the passage is for evaluating the statement," with the instruction to "rate the passage high only if it supports or contradicts the statement" (Table 8, "Rate Passages" prompt). The top 3 highest-scoring paragraphs are retained as evidence for that keypoint.

The paper explicitly notes that this LLM-driven approach "works better than off-the-shelf sparse or dense retrieval (e.g., using BM25/GTR (Ni et al., 2022)) over Wikipedia passages" (Section 3.2), though the details of this comparison are not reported.

For answers where GPT-4 fails to produce a valid Wikipedia URL, the system cannot generate grounded epistemological critiques. The paper does not specify the fallback behavior for the 12.3% of cases without valid URLs, but the rater templates (Figure 10) indicate that AI Assistance may simply not be shown when unavailable.

Stage 3: Critique Generation. For each of the eight evaluative dimensions, GPT-4 generates a critique expressing agreement or disagreement that the answer meets the dimension's standard. The exact instruction is: "Given the following question and answer, express your disagreement with the statement in a concise sentence in a single line. You may be provided with relevant paragraphs from Wikipedia, if so, you must use those verbatim to support your critique. If you fully agree with the statement, state 'No Critique'" (Table 8, "Presentational/E epistemological AI Assistance" prompts).

Critically, the critique generation differs between presentational and epistemological dimensions:

For presentational dimensions (style, clarity, correctness, tone), the model receives the question, the answer, and the dimension statement—but no Wikipedia evidence. This is because presentational qualities are about the answer's form, not its factual grounding, so external evidence is irrelevant. The paper notes that "presentational dimensions are evaluated without the additional paragraphs" (Figure 1 caption).
For epistemological dimensions (accuracy, specificity, completeness, uncertainty), the model receives the question, the answer, the dimension statement, and the retrieved Wikipedia evidence. It is instructed to "use those verbatim to support your critique," meaning that when the model claims the answer is inaccurate, incomplete, or missing uncertainty, it must quote specific Wikipedia passages that demonstrate the problem. This grounding is the key mechanism intended to make the assistance trustworthy and verifiable by raters.

The dimension statements themselves (Table 11) encode the specific sub-issues that the dimension covers. For example, the Accuracy statement is: "The answer is accurate. In particular, it does not take scientific findings out of context, does not contradict itself, does not rely on anecdotal evidence, and does not misuse key terms or scientific terminology." By prompting GPT-4 to evaluate this statement, the critique naturally addresses the specific failure modes the framework cares about.

Stage 4: Presentation to Raters. The AI Assistance critique, if available, is presented to human raters alongside the answer during the rating task. For epistemological dimensions, the retrieved Wikipedia paragraphs supporting the critique are also shown (as illustrated in Figure 10). Raters are also asked to rate the helpfulness of the assistance, providing data on whether the assistance is perceived as useful and whether its influence is uniform across dimensions.

Why this pipeline design? The key design choices reflect specific challenges in evaluating science communication:

Keypoint extraction before evidence retrieval rather than retrieving passages for the whole answer: The paper finds this provides "better signal" because the whole answer may mix specific claims with general commentary, and precise retrieval requires focusing on the claims themselves. This is consistent with findings from the attributed question answering literature (Bohnet et al., 2023) that statement-level grounding is more reliable than document-level.
GPT-4-generated Wikipedia URLs rather than web search: The paper reports that GPT-4's Wikipedia URLs are more consistent and higher quality than unrestricted web URLs, and Wikipedia's style and quality guidelines provide uniformity. This is a pragmatic choice that trades retrieval recall (the unrestricted web would have more evidence) for retrieval precision and verifiability (Wikipedia passages are easier for raters to evaluate than arbitrary web pages of varying quality).
Verbatim quoting requirement for epistemological critiques: This is the mechanism that makes the assistance "grounded"—the model cannot simply assert that an answer is inaccurate; it must point to specific Wikipedia text that demonstrates the inaccuracy. This gives human raters the ability to verify the AI's critique by checking whether the quote actually supports the claim, which is essential for the protocol to function as scalable oversight rather than as a proxy for judgment that might introduce its own errors.

The Human Rating Protocol: Rater Qualification, Rating Interface, and Experimental Design

Rater Selection and Qualification. The paper employs 32 raters, all fluent in English and all holding at least an undergraduate degree in a climate-related field. This includes "environmental disciplines (e.g., environmental science, earth science, atmospheric physics, ecology, environmental policy, climate economics), and also other disciplines (including the behavioral and social sciences) as long as their academic work (coursework, project work, or otherwise) involves work on climate or environmental studies" (Appendix A.7). The demographic data (Table 10) shows diversity in age (43.75% under 25, 34.38% 25–35), geography (10 countries, with UK at 25%, South Africa and Portugal each at 12.5%), and sex (56.25% female).

To be admitted to the rating pool, candidates must complete a tutorial followed by an admission test:

Tutorial: Four examples of increasing difficulty are presented in the rating interface, each exhibiting a particular main issue along one dimension. Raters must identify the issue correctly; if they select a wrong answer, a hint is given and they can only proceed if they eventually select an acceptable answer. Positive feedback explains why the correct answer is correct. No data is collected during the tutorial—it is purely for familiarization with the interface and task expectations.
Admission test: Three realistic examples exhibiting at least one major flaw each are presented using the full template (all eight dimensions). Responses are scored on a point system that awards or deducts points for detected, undetected, and over-detected (false positive) issues. The point scheme was calibrated by the authors using "an early group of raters with known performance on the task" (Appendix A.9), and a threshold is set to admit raters above that score.

This qualification process is important because the evaluation requires both domain knowledge (to recognize when answers are missing key information or misrepresenting uncertainty) and familiarity with the specific dimensions and issue types defined by the framework. The paper argues that the tutorial and admission test were "effective in ensuring that raters were both familiar with the interface as well as the type of assessment we expect from them."

The Rating Interface and Procedure. Each answer is rated by three independent raters. The rating interface (screenshots in Figures 9 and 10) presents:

A screening question to confirm the rater can judge the example (e.g., whether the question is comprehensible, whether the rater has sufficient background knowledge). If any screening question is answered "No," the example is skipped.
The question and answer to be evaluated.
For each dimension (presented one at a time), the dimension statement and the question: "To what extent do you agree with the statement below?" on a 5-point Likert scale (1 = disagree completely, 2 = disagree, 3 = neither, 4 = agree, 5 = agree completely). When a rater selects 1 or 2 ("disagree completely" or "disagree"), the interface expands to show a list of specific issues (the predefined issues for that dimension, plus a free-text "other" field), and raters must select at least one. For ratings of 3 or above, the issue list is hidden, meaning raters are not asked to justify positive or neutral ratings—only negative ones elicit diagnostic detail.
AI Assistance, if available: a model-generated critique, and for epistemological dimensions, the supporting Wikipedia evidence. Raters are also asked to rate the helpfulness of the assistance (on a scale from 1–5, Figure 10 shows this).

Experimental design for measuring AI Assistance effects. The paper runs two key comparisons to isolate the effect of AI Assistance:

"GPT-4, no assistance": A specific pool of raters is never exposed to AI Assistance during their rating work. This is the baseline for what raters can detect without any AI support.
"GPT-4, no assistance, previous exposure": Raters work without assistance on a given set of examples, but they have previously worked on studies that included assistance (though never on the same examples). This tests whether exposure to assistance has a training effect—whether raters learn to be more critical from seeing assistance in prior tasks.
"GPT-4" (with assistance): The standard setting where AI Assistance is shown whenever available. This is compared against the two no-assistance conditions to measure the effect.

Additionally, in Section 4.5, the paper reports a dedicated experiment on GPT-4 answers examining the number of issues detected per dimension across these three conditions (Figure 3). The results show raters with assistance detect more issues than those without, and raters with previous exposure to assistance detect more than assistance-naive raters but fewer than currently-assisted raters—evidence of both the immediate effect of assistance and a learning effect from prior exposure.

Why three raters per answer? Triplicate rating is a standard approach for subjective evaluation tasks where individual raters may disagree. The paper analyzes agreement metrics extensively (Appendix A.11), reporting both average pairwise distance between Likert ratings (Table 12) and Krippendorff's alpha (Table 13). The agreement is medium overall, with higher agreement on presentational dimensions (style, clarity, correctness) than epistemological ones and notably low agreement on tone. This is consistent with the inherently subjective nature of dimensions like tone and uncertainty. Three raters provide enough redundancy to identify systematic model-level differences even when individual ratings are noisy—the paper reports bootstrapped 95% confidence intervals on mean ratings (Figures 2, Tables 3–4) which are relatively tight for most dimension-model pairs, justifying the 3-rater design.

Why no gold ratings? The paper explicitly acknowledges that "as we do not have access to gold ratings, calibration of raters remains an open issue" (Section 5). This is a fundamental constraint of the domain—there are no objectively correct ratings for whether an answer adequately communicates uncertainty or is appropriately specific. The qualification process (tutorial + admission test) substitutes for gold-standard calibration by ensuring all raters share a common understanding of the dimensions and can identify clear issues. The validation experiment (Appendix A.14, with 30 synthetic examples each exhibiting a known issue) provides a partial calibration check by verifying that raters can detect planted problems.

The AIS (Attribution to Identified Source) Comparison Experiment

To evaluate the relationship between the framework's epistemological dimensions and source attribution—a prominent alternative evaluation paradigm—the paper runs a dedicated AIS experiment on GPT-4 answers (Section 3.2, Section 4.6). The goal is to determine whether assessing whether an answer is attributable to retrieved evidence is sufficient to capture the epistemological qualities the framework measures.

Procedure. The AIS evaluation leverages the same keypoints and evidence pipeline used for AI Assistance:

For each GPT-4 answer, the keypoints (1–3 verbatim statements extracted in Stage 1 of the AI Assistance pipeline) are presented to human raters along with the top 3 retrieved Wikipedia passages for that keypoint.
Raters judge whether each keypoint is fully, partially, or not supported by the evidence, and whether the evidence contradicts the keypoint. The paper also asks whether the top 3 passages jointly provide support for the keypoint.
An answer is considered fully attributed if all its keypoints are supported, not attributed if all keypoints are not supported, and partially attributed otherwise.

Results at the answer level (Table 17): 6.95% of answers are fully supported, 39.13% partially supported, and 53.91% not supported. At the keypoint level, 12% of keypoints are fully supported, 54.79% partially supported, 32.81% not supported, and 0.38% contradicted by the evidence.

Relationship to epistemological ratings (Figure 5). The paper plots the distribution of average epistemological ratings against AIS support categories (unsupported, partial, full). Key findings:

Answers that are fully attributed tend to have higher minimum ratings on accuracy and specificity—attributable answers are rarely rated as very inaccurate or very unspecific. This makes sense: if an answer's claims can all be traced to evidence, they are less likely to be fabricated or irrelevant.
However, answers that are fully attributed have lower minimum ratings on completeness than partially attributed or unsupported answers. The paper's interpretation: "AIS can only consider what is included in the answers, and not what important information is missing." A well-attributed answer may still omit crucial details, and in fact, an answer that stays close to retrieved evidence might be more likely to be incomplete because it only covers what was in the retrieved passages rather than the full scope of the question.
There is no clear pattern for uncertainty: unsupported answers have more low-uncertainty ratings than supported ones, but the relationship is not monotonic.
Spearman's correlation between AIS (treated as an ordinal variable: unsupported < partial < full) and the mean epistemological dimension ratings is near zero for all dimensions: 0.03 (accuracy), −0.06 (specificity), 0.002 (uncertainty), −0.02 (completeness), with p-values all above 0.30.

Interpretation. The paper concludes that "AIS and epistemological assessments are mostly orthogonal" (Section 4.6). This is not a negative finding but an important methodological result: it means that attribution-based evaluation—even when it works perfectly (fully attributable answers)—does not substitute for direct assessment of completeness, specificity, and uncertainty communication. Evaluation frameworks that rely solely on attributability to quantify response quality will systematically miss the dimensions captured by this paper's framework.

The paper provides concrete examples in Table 19. For instance, an answer about when the public became aware of CO2's effects is fully attributable (all its keypoints can be traced to Wikipedia evidence) yet receives an average completeness rating of only 1.6/5 because it omits key milestones like Glenn T. Seaborg's 1966 warnings and John Sawyer's 1972 study—omissions that would be invisible to an AIS evaluation that only checks whether what was said is supported.

This finding has direct implications for the Retrieval Augmented Language Model (RALM) literature (Lewis et al., 2020; Guu et al., 2020): while grounding generation in retrieved documents can improve factuality, it does not guarantee that generated answers will be complete, appropriately specific, or correctly calibrated about uncertainty—qualities that require separate evaluation.

The Validation Experiment: Synthetic Examples with Known Issues

To calibrate the framework's ability to detect real problems, the authors manually created 30 questions and answers, each exhibiting a specific issue from the full list of possible issues across all dimensions (Appendix A.14). This serves as a partial substitute for the gold-standard ratings the framework lacks—the authors know which issues are present because they planted them deliberately.

Procedure. The 30 synthetic examples were presented to raters in two conditions:

Without AI Assistance (three independent ratings per example—the same "3 raters, no assistance" setup as the main experiment).
With AI Assistance (three independent ratings per example, with the standard AI Assistance protocol).

The metric is whether the planted issue is detected by at least one of the three raters ("Any"), by a majority of the three raters ("Majority", meaning at least 2 of 3), or by all three raters ("All").

Results (Table 20):

Detection level	Without AI Assistance	With AI Assistance
Any (at least 1 of 3 raters)	83.33%	93.33%
Majority (at least 2 of 3 raters)	60.00%	76.67%
All (3 of 3 raters)	33.33%	43.33%

Interpretation. AI Assistance substantially increases detection rates, particularly at the "Majority" level (from 60% to 76.67%—a relative improvement of 27.8%). This is direct evidence that the assistance protocol helps raters find genuine issues they would otherwise miss. The paper also notes that "there was just one issue not detected by any rater: anecdotal evidence" (all three raters rated the dimension low but highlighted only the 'incorrect' issue rather than identifying the anecdotal nature of the evidence), suggesting that some specific issues may be inherently difficult to identify even with assistance.

The validation experiment is important not because it proves the framework is perfect—76.67% majority detection still means nearly a quarter of planted issues are missed even with assistance—but because it provides a lower bound on detection capability. In the main evaluation, the framework is detecting issues in real LLM outputs where the ground truth is unknown; the validation experiment shows that, at least for the types of issues the authors could construct, the framework catches the majority. The improvement from 60% to 76.67% with assistance justifies the paper's central methodological claim: grounded AI Assistance is a necessary component of reliable evaluation in this domain.

Why only 30 examples? The paper does not justify this sample size explicitly, but constructing examples that exhibit specific, well-defined issues while remaining realistic (not caricatures) is time-intensive. The 30 examples serve as a calibration check, not as a comprehensive benchmark—the main evaluation's robustness depends on the three-rater design and the inter-rater agreement metrics discussed in Appendix A.11.

Summary of Key Design Choices and Their Justifications

Two-category, eight-dimension framework grounded in science communication research rather than NLP evaluation traditions: ensures the evaluation captures what science communication scholars have identified as critical quality dimensions, not just what is easy to measure automatically.
No single aggregate metric (Section 2.3): reflects the view that quality is multi-dimensional and application-specific; prevents the framework from masking trade-offs (like the presentational-epistemological trade-off observed in Section 4.1).
Likert-scale rating with structured issue lists for low scores: provides both quantitative comparability (mean scores per model per dimension) and diagnostic detail (which specific issues each model tends to produce, as in Table 7).
Three-source question construction (GTrends, Skeptical Science, Wikipedia) with multi-filter quality control and stratified sampling: captures diverse information needs (real user queries, debated claims, regional/context-specific topics) while avoiding biases toward easy or hard questions.
GPT-4-generated Wikipedia evidence rather than web search or dense retrieval: prioritizes retrieval precision and source uniformity over recall; makes the evidence verifiable by human raters.
Keypoint-level evidence retrieval rather than whole-answer retrieval: improves the precision of evidence grounding by focusing on specific claims rather than mixing signal with tangential content.
Verbatim quoting requirement for epistemological AI Assistance: ensures the assistance is grounded in retrievable facts rather than being pure model judgment; enables raters to verify critiques independently.
Three raters per answer with admission testing and tutorial: balances the inherent subjectivity of the task with the need for reliable system-level comparisons; the qualification process ensures a shared understanding of dimensions without access to gold-standard reference ratings.
Synthetic validation set (30 examples) with known planted issues: provides a partial calibration of the framework's issue detection capability in the absence of gold ratings for real answers.
AIS comparison experiment: directly tests whether attribution-based evaluation (a prominent alternative) can substitute for the framework's epistemological dimensions, providing empirical evidence that it cannot—a finding that justifies the framework's additional complexity.

4. Key Insights and Innovations

Innovation 1: The Presentational-Epistemological Distinction as a Diagnostic for LLM Science Communication

The paper's foundational conceptual move is the clean separation of response quality into presentational adequacy (how well the message is conveyed—style, clarity, linguistic correctness, tone) and epistemological adequacy (how well the message represents scientific knowledge—accuracy, specificity, completeness, uncertainty). This is not merely a taxonomy borrowed from science communication research (Jamieson et al., 2017; Fähnrich et al., 2023); it is a diagnostic instrument that reveals a fundamental dissociation in LLM behavior that standard evaluation paradigms would miss.

Prior to this work, the dominant evaluation paradigms for LLMs in factual domains fell into two camps. The first measured surface quality—fluency, coherence, helpfulness—often through Likert ratings that conflated presentation with correctness (e.g., the Elo-based chatbot arena evaluations, or single-score helpfulness ratings common in RLHF reward modeling). The second measured factuality directly—accuracy against a knowledge base, hallucination rates, or attribution to retrieved sources (Rashkin et al., 2022; Bohnet et al., 2023; Liu et al., 2023). Neither paradigm could detect a case where a model produced a fluent, well-attributed, but epistemologically misleading answer—one that is factually correct in its individual claims yet incomplete in a way that misleads, or specific about the wrong things, or silent on critical uncertainties. Such answers are not "hallucinations" in the standard sense; they are failures of communication adequacy, not factual accuracy.

The paper's framework makes this dissociation visible. Figure 2 shows the pattern starkly: across seven LLMs, presentational dimensions (style, clarity, correctness) cluster in the 4.0–4.6 range (out of 5), while epistemological dimensions (accuracy, specificity, completeness, uncertainty) cluster in the 2.0–3.8 range, with a gap of 1–2 full Likert points between presentation and epistemology for every model. The temporal trend within the OpenAI model family is revealing: InstructGPT (davinci-002) → (davinci-003) → (turbo) → ChatGPT → GPT-4 shows a monotonic improvement in presentation, but epistemological scores do not follow this trajectory—ChatGPT, which achieves the highest presentational scores of any model (4.54–4.58 across style, clarity, and correctness), ranks between fourth and last across the epistemological dimensions. This is not a failure of RLHF or instruction tuning to improve epistemology; it is evidence that these training paradigms primarily optimize for presentation, and that epistemological quality does not automatically follow from surface fluency.

The significance of this finding extends beyond the specific models and domain studied. The presentational-epistemological dissociation challenges a tacit assumption underlying much of the RLHF and conversational AI literature: that human preference judgments (which drive reward model training) reflect overall quality in a way that includes factual soundness. If—as this paper suggests—human raters give high scores to fluent, well-structured answers even when those answers are epistemologically weak, then optimizing for human preference may systematically underweight the epistemological dimensions that matter most for high-stakes science communication. The paper does not make this argument about RLHF directly, but the data supports it: the "no assistance" condition for GPT-4 (Table 4, bottom row) shows epistemological ratings artificially inflated by 0.7–1.7 points compared to the assisted condition, meaning that unaided raters—who drive reward model training—substantially overestimate epistemological quality.

This is a fundamental reframing, not a metric improvement. It does not propose a new number to optimize; it argues that the field's current optimization targets are measuring the wrong thing, or at least measuring an insufficient subset of what matters.

Innovation 2: AI Assistance as an Empirically Validated Oversight Protocol—with Documented Biases and Boundaries

The concept of using AI to help humans evaluate AI outputs is not new. The scalable oversight literature (Amodei et al., 2016) has proposed various mechanisms—debate (Irving et al., 2018), amplification (Christiano et al., 2018), reward modeling (Leike et al., 2018)—and a small number of studies have empirically tested these ideas. Bowman et al. (2022) showed non-expert raters improved on a multiple-choice task with dialogue agent assistance. Saunders et al. (2022) found that raters of summarization produced more critiques when they could accept or edit model-written critiques. But these proof-of-concept studies operated in simplified settings—artificial multiple-choice tasks, summarization with accepted ground-truth quality criteria—and did not grapple with the core challenge that makes scalable oversight difficult in practice: AI assistance can help raters find real issues, but it can also mislead them, bias them toward the assistant's judgment, or help with some issue types while being useless for others.

The paper's contribution is not the invention of AI Assistance as an idea, but rather its empirical characterization as a real-world oversight protocol with measured strengths, measured limitations, and documented side effects. The key findings form a nuanced picture:

Assistance demonstrably increases recall of real issues (Figure 3, Table 20). The validation experiment with 30 synthetic examples containing known planted issues shows that majority detection improves from 60% to 76.67% with assistance. In the main experiment, assisted raters detect substantially more issues than unassisted raters across all epistemological dimensions (the "no assistance" GPT-4 ratings in Table 4 are inflated across the board). This is the "it works" story.

But assistance also influences raters beyond issue discovery. The paper explicitly acknowledges: "its influence on raters extends beyond this enhancement" and "there may also be errors caused by models falsely pointing out issues and wrongly convincing the raters" (Section 5). The relationship between rater helpfulness ratings and actual ratings (Figure 4) provides indirect evidence of this: when raters find assistance unhelpful, they tend to give higher ratings to the answer—suggesting that raters do not blindly follow the assistant's critique but sometimes push back against it. This is simultaneously good (raters are not passive recipients of AI judgment) and concerning (the interaction between rater judgment and AI critique is complex and not fully characterized).

Previous exposure to assistance has a training effect (Figure 3). Raters who have worked with assistance on prior studies but receive none on the current task detect more issues than raters who have never seen assistance—evidence that the protocol teaches raters to be more critical readers. This is a finding about rater learning, not about the assistance protocol per se, and it suggests that long-term deployment of assisted evaluation could produce progressively more reliable raters even when assistance is intermittently unavailable.

The protocol is domain-specific in its construction. The grounded evidence retrieval (verbatim Wikipedia passages) is essential for epistemological dimensions but irrelevant for presentational ones, and the paper's architecture reflects this (presentational AI Assistance uses no evidence). This distinguishes it from generic "ask an LLM to critique" approaches: the grounding mechanism provides raters with something to verify rather than having to trust the AI's judgment, which is what makes the protocol a scalable oversight system rather than simply replacing human raters with AI raters.

This contribution is practical rather than theoretical—it does not advance the theory of scalable oversight, but it provides one of the most thoroughly characterized empirical deployments of an oversight protocol in a realistic domain. The paper is unusually candid about the protocol's limitations and open questions, which itself represents a methodological contribution: it provides a template for how to report AI-assisted evaluation results, including the measurement of assistance effects, the validation against known issues, and the analysis of assistance helpfulness ratings as a diagnostic signal.

Innovation 3: Empirical Demonstration That Attributability (AIS) Is Orthogonal to Epistemological Adequacy

The Retrieval Augmented Language Model (RALM) paradigm (Lewis et al., 2020; Guu et al., 2020) and the attributed question answering literature (Bohnet et al., 2023; Rashkin et al., 2022, 2023) rest on a plausible and increasingly influential premise: if an LLM's generated statements can be traced to retrieved source documents, this provides a reliable signal of response quality. Under this premise, improving attribution—through training models to cite sources, or through retrieval-augmented generation—should improve overall response quality, and evaluating attribution should serve as a sufficient proxy for evaluating factuality.

The paper provides the first direct empirical test of this premise against a multi-dimensional epistemological evaluation framework—and the results challenge it fundamentally. The AIS experiment (Section 4.6) finds that Spearman's correlation between AIS support categories (unsupported, partial, full) and mean epistemological ratings is effectively zero: 0.03 for accuracy, −0.06 for specificity, 0.002 for uncertainty, −0.02 for completeness, with all p-values above 0.30. An answer's attributability tells you essentially nothing about its completeness, specificity, or uncertainty calibration.

The dis-aggregated analysis in Figure 5 reveals why this orthogonality arises, and the reasons are structural rather than contingent on this particular retrieval system or model. Fully attributed answers have higher minimum accuracy and specificity ratings (attributable answers are rarely completely inaccurate or completely irrelevant), which is the expected benefit of grounding. But fully attributed answers have lower minimum completeness ratings than unsupported answers. The paper's explanation is precise: "AIS can only consider what is included in the answers, and not what important information is missing." An answer that carefully grounds all its claims in retrieved passages may be more incomplete than one that does not, precisely because it restricts itself to what the retrieval system returned rather than covering the full scope of the question. Attribution optimizes for faithfulness to sources, which is not the same as comprehensiveness of answer.

This is not a negative result that invalidates attribution-based evaluation—the paper is careful to characterize AIS as "practical and complementary." But it is a finding that fundamentally limits the scope of attribution as a quality signal. Attribution tells you whether what was said can be verified; it does not tell you whether what should have been said was actually said, whether what was said is specific enough to be useful, or whether the answer communicates the appropriate level of scientific confidence. For high-stakes domains like climate communication, these missing dimensions are not optional—an answer that is correct in its individual claims but misleadingly incomplete or missing critical uncertainty qualifiers can be more harmful than an answer that is simply wrong, because users trust fluent, attributable text.

This contribution is a negative result with diagnostic implications—it identifies a specific blind spot in an increasingly popular evaluation paradigm, explains why the blind spot exists structurally, and provides concrete examples (Table 19) that make the failure mode tangible. It reframes the relationship between attribution and quality from one of sufficiency ("attribution is enough") to one of complementarity ("attribution is necessary but not sufficient"), which has direct implications for how future evaluation frameworks should be designed.

Innovation 4: The Science Communication Perspective as a Methodology for AI Evaluation

The paper's most distinctive intellectual move is methodological rather than technical: it imports an entire body of scholarship—science communication research, spanning communication science, sociology, psychology, and education (Trench & Bucchi, 2021; Nisbet et al., 2018; Jamieson et al., 2017)—into the AI evaluation domain, and demonstrates that this scholarship generates evaluative dimensions that are both empirically measurable and diagnostically revealing about LLM behavior.

This matters because AI evaluation has historically been dominated by perspectives internal to computer science and NLP: accuracy against held-out test sets, human preference judgments, automated metrics like BLEU or ROUGE, and more recently, LLM-as-judge frameworks. These perspectives have produced useful evaluative instruments, but they are not informed by decades of research on what makes science communication effective—research that has identified specific dimensions (like the communication of uncertainty as a "gateway belief" that motivates action, per van der Linden et al., 2015) that have no natural counterpart in standard NLP evaluation.

The paper demonstrates the value of this interdisciplinary methodology in two ways. First, the dimensions derived from the science communication literature produce results that would be invisible to standard evaluation: the finding that LLMs reliably fail on tone and uncertainty communication despite excelling at clarity and correctness, the identification of a pragmatic style that "fill[s] parts of the answer with somewhat relevant but unspecific, cautionary and sometimes preachy reminders" (Section 4.4), and the suggestion of a trade-off between presentational and epistemological quality when prompts are augmented with evaluation criteria (Table 1). Second, the paper shows that AI raters (GPT-4 prompted to evaluate along the same dimensions) benefit from assistance in the same way humans do and agree with human raters on major patterns (Appendix A.15)—suggesting that the dimensions are sufficiently well-operationalized to be applied by both humans and, in principle, automated systems.

This is a methodological contribution, not a technical one. It does not propose new metrics or new models; it proposes a new source for evaluation criteria—the relevant domain scholarship rather than only the ML evaluation tradition—and demonstrates that this source yields criteria that are both rigorous (each dimension is grounded in specific citations from the literature) and practically discriminating (they reveal systematic differences between models and systematic gaps between presentation and epistemology). The implication for the field is that evaluating LLMs for domain-specific communication tasks (not just climate, but any domain with a body of communication research—health, law, finance) should begin by engaging with the scholarship of that domain rather than by adapting generic NLP evaluation templates.

This contribution is arguably fundamental rather than incremental because it changes where evaluative criteria come from, not just which criteria are used. An NLP team developing a medical QA evaluation framework might independently arrive at criteria like accuracy and completeness, but they are unlikely to independently arrive at "communicating scientific consensus as a gateway belief" or "avoiding an argument-from-authority style"—criteria that the science communication literature identifies as critical and that this paper operationalizes. The framework makes this domain knowledge portable to AI evaluation.

5. Experimental Analysis

Evaluation Methodology

Dataset. The evaluation uses 300 climate change questions drawn equally from three sources: 100 from Google Trends (the most popular U.S. search queries for "Climate Change" and "Global Warming" during 2020–2022), 100 manually rephrased from the Skeptical Science list of debated climate myths, and 100 synthetically generated by GPT-4 from Wikipedia paragraphs about climate change topics, selected via stratified sampling after multi-filter post-processing (Section 3.1.1). The questions are designed to span real-world information needs, contested claims, and diverse climate subtopics (energy, emissions, policy, weather, land/ocean, society, health, biodiversity, cities/infrastructure) and include both causal/predictive and non-causal question types (Appendix A.3.1).
Base model(s). Seven LLMs are evaluated: GPT-4 (OpenAI, 2023), ChatGPT-3.5, three InstructGPT variants (turbo, text-davinci-003, text-davinci-002), PaLM-2 (text-bison) (Anil et al., 2023), and Falcon-180B-Chat. The models are chosen to span a range of capabilities, sizes (from the earlier davinci-002 to the 180B-parameter Falcon), and developers (OpenAI, Google, open-source). The paper notes that data was collected in September–October 2023, so model versions reflect that timeframe (Section 4). Answers are generated with a standardized prompt: "You are an expert on climate change communication. Answer each question in a 3-4 sentence paragraph" (Table 8). A follow-up experiment on GPT-4 in November 2023 tests a "dimension-aware" prompt that includes explicit descriptions of the evaluation criteria (Section 4.1, Table 1).
Metrics. The primary metrics are mean Likert ratings (1–5) per dimension per model, computed by averaging across three raters per answer and all 300 answers, with bootstrapped 95% confidence intervals (Tables 3 and 4, Figure 2). Agreement metrics include average pairwise distance between Likert ratings (Table 12) and Krippendorff's alpha (Table 13) for both ratings and issue selections. For the AIS comparison (Section 4.6), the metrics are the percentage of answers/keypoints that are fully, partially, or not supported by retrieved evidence (Table 17). The validation experiment (Appendix A.14) uses detection rate (percentage of 30 planted issues detected by any rater, majority of raters, or all raters).
Baselines. The primary comparison is between models on the same evaluation dimensions—there is no single "baseline model" in the traditional sense. However, several experimental conditions serve as baselines: (1) "GPT-4, no assistance" — raters who have never been exposed to AI Assistance, providing a baseline for what unaided raters can detect (Tables 3 and 4); (2) "GPT-4, no assistance, previous exposure" — raters without assistance on the current task but with prior experience working with assistance, testing for learning effects (Tables 3 and 4, Figure 3); (3) Majority voting is not applicable here since each answer is rated independently by three raters and scores are averaged, not voted. The AIS experiment (Section 4.6) compares the framework's epistemological dimensions against the Attribution to Identified Source (AIS) evaluation paradigm (Rashkin et al., 2022).
Generation budget / compute accounting. This is a human evaluation study, not a model training or inference scaling study, so "compute" is measured in terms of human rater effort rather than FLOPs. Each of the 300 answers from each of the 7 models is rated by 3 raters on 8 dimensions, yielding 300 × 7 × 3 × 8 = 50,400 individual dimension ratings (plus additional ratings for the AIS experiment on GPT-4 answers and the validation experiment). The AI Assistance pipeline (keypoint extraction, URL generation, passage scoring, critique generation) uses GPT-4 for all steps, but the cost of this automated pipeline is not quantified or compared to the human rating cost. The paper notes that rating times are longer for epistemological dimensions than presentational ones and longer for lower ratings (which require issue selection) than higher ones (Appendix A.16, Figures 13–15).
Cross-validation / statistical protocol. There is no cross-validation in the traditional ML sense, since this is an evaluation study, not model training. The statistical protocol includes: (1) three independent raters per answer to enable agreement measurement and variance estimation; (2) bootstrapped 95% confidence intervals on mean ratings (Tables 3 and 4, Figure 2); (3) pairwise t-tests between model pairs on the three-rater mean ratings to assess whether differences are statistically significant (Tables 5 and 6, using p < 0.01 and p < 0.05 thresholds); (4) rater qualification via tutorial and admission test (Appendix A.9) to ensure a shared understanding of the dimensions before participation; (5) a validation experiment with 30 synthetic examples containing known planted issues to calibrate detection capability with and without AI Assistance (Appendix A.14, Table 20); (6) helpfulness ratings from raters on the AI Assistance to detect whether raters are blindly following the assistant's critiques (Figure 4, Section 4.5).

Main Quantitative Results

Presentational vs. Epistemological Quality

The central finding of the study is visible immediately in Figure 2 and Tables 3–4: presentational dimensions (style, clarity, correctness) score consistently high across models, while epistemological dimensions (accuracy, specificity, completeness, uncertainty) score substantially lower, revealing a systematic gap between how well LLMs present information and how well they actually convey scientific knowledge.

For presentational dimensions (Table 3), the top-performing models achieve mean ratings in the 4.3–4.6 range (out of 5): ChatGPT scores 4.54 on style [4.50, 4.58 CI], 4.56 on clarity [4.52, 4.60 CI], 4.58 on correctness [4.54, 4.61 CI]; PaLM-2 scores 4.34, 4.48, 4.57; GPT-4 scores 4.35, 4.34, 4.38. Even the weakest model on presentation—InstructGPT (davinci-002)—still achieves 3.22, 3.63, 3.82 on these three dimensions, well above the midpoint. Only the tone dimension shows a marked drop: the best model (InstructGPT turbo) scores only 3.41 [3.33, 3.48 CI], and ChatGPT—the best on all other presentational dimensions—scores the lowest of all models on tone at 3.06 [2.99, 3.13 CI].

For epistemological dimensions (Table 4), the range is 2.0–3.8, a full 1–2 Likert points lower than presentational quality for every model. Accuracy is the highest epistemological dimension: Falcon-180B-Chat leads at 3.81 [3.74, 3.87 CI], GPT-4 at 3.67 [3.61, 3.73 CI], while InstructGPT (davinci-002) scores only 2.81 [2.73, 2.88 CI]. The remaining three epistemological dimensions—specificity, completeness, and uncertainty—are substantially lower. For GPT-4, specificity is 3.13 [3.05, 3.21 CI], completeness is 2.61 [2.53, 2.68 CI], and uncertainty is 2.21 [2.15, 2.27 CI]—below the midpoint of 3.0 in all cases. ChatGPT, despite its presentational excellence, ranks between fourth and last on epistemological scores: 3.48 accuracy, 2.71 specificity, 2.26 completeness, 2.05 uncertainty.

The gap between the best presentational dimension (correctness, with models scoring 4.4–4.6) and the worst epistemological dimension (uncertainty, with models scoring 2.0–2.6) is approximately 2 full Likert points—a large effect on a 5-point scale. This is the core empirical finding that motivates the paper's claim that "surface and epistemological qualities of LLMs in the realm of climate communication" diverge significantly (Abstract).

Model Comparisons and Rankings

The pairwise t-tests (Tables 5 and 6) reveal which model differences are statistically significant. For presentational dimensions:

InstructGPT (davinci-002) is significantly worse (p < 0.01) than every other model on style, clarity, and correctness—a clear outlier.
ChatGPT is significantly better than all other models on style (p < 0.01 vs. all except PaLM-2 and Falcon where p > 0.05) and better than most on clarity (p < 0.01 vs. davinci-002, davinci-003; p < 0.05 vs. turbo) but not significantly different from GPT-4, Falcon, or PaLM-2 on clarity.
On correctness, the pattern is similar: ChatGPT, PaLM-2, InstructGPT (davinci-003), GPT-4, Falcon, and InstructGPT (turbo) all score in the 4.4–4.6 range with no significant differences among themselves, but all are significantly better than davinci-002 (p < 0.01).
Tone is the noisiest presentational dimension: few pairwise differences reach significance, with Falcon-180B-Chat and InstructGPT (davinci-003) showing some significant differences from other models.

For epistemological dimensions, the rankings are qualitatively different from the presentational rankings:

Falcon-180B-Chat is the overall leader on epistemology: it ranks first on accuracy (3.81), specificity (3.15, tied with GPT-4), completeness (2.73), and uncertainty (2.55). It is significantly better than all other models on accuracy and uncertainty (p < 0.01 vs. all), and significantly better than most on completeness (p < 0.01 vs. davinci-002, davinci-003, turbo, ChatGPT; p < 0.05 vs. GPT-4).
GPT-4 is second on accuracy (3.67) and significantly better than davinci-002, davinci-003, and ChatGPT (p < 0.01). It ties with Falcon on specificity (3.13 vs. 3.15, not significantly different).
ChatGPT, despite its presentational dominance, is in the lower tier epistemologically: its accuracy (3.48) is not significantly different from davinci-003 or PaLM-2, and its specificity (2.71), completeness (2.26), and uncertainty (2.05) are among the lowest—significantly worse than Falcon on all four epistemological dimensions (p < 0.01).
InstructGPT (davinci-002) is the worst on accuracy (2.81) and specificity (2.49), significantly below all other models on both (p < 0.01), though it performs relatively better on uncertainty (2.35, not significantly different from most models) than on other epistemological dimensions.
Uncertainty is the most uniformly poor dimension: all models score in the 2.0–2.6 range, and even the best model (Falcon at 2.55) is below the midpoint of the scale.

This pattern—ChatGPT best on presentation, Falcon best on epistemology, GPT-4 strong on epistemology but weaker on presentation—documents what the paper calls "complex dependencies between presentational and epistemological properties" (Section 4.2) and provides evidence that there is no single model that excels across all dimensions.

The Effect of AI Assistance on Issue Detection

Figure 3 and the corresponding analysis in Section 4.5 present the number of issues detected across three conditions on GPT-4 answers: without AI Assistance, without AI Assistance but with previous exposure, and with AI Assistance. The bars in Figure 3 show that for epistemological dimensions (accuracy, specificity, completeness, uncertainty), the number of detected issues approximately doubles or triples when assistance is present compared to when it is absent, with the "previous exposure" condition falling between the two extremes.

The validation experiment (Appendix A.14, Table 20) provides controlled evidence for this effect. Of 30 synthetic examples with known planted issues:

Without AI Assistance: 83.33% detected by at least one rater, 60.00% detected by majority of raters (≥2 of 3), 33.33% detected by all three raters.
With AI Assistance: 93.33% detected by at least one rater, 76.67% detected by majority of raters, 43.33% detected by all three raters.

The majority detection rate improves from 60% to 76.67%, a relative improvement of 27.8%. The paper notes that "there was just one issue not detected by any rater: anecdotal evidence" (in both conditions), suggesting that detecting anecdotal reasoning remains challenging even with assistance.

The "GPT-4, no assistance" condition in Tables 3 and 4 provides a crucial baseline: epistemological ratings without assistance are dramatically inflated. For GPT-4, the "no assistance" ratings (Table 4, bottom row) are 4.49 accuracy [4.44, 4.55 CI], 4.41 specificity [4.35, 4.48 CI], 4.32 completeness [4.25, 4.39 CI], 3.38 uncertainty [3.29, 3.46 CI]—compared to the assisted ratings of 3.67, 3.13, 2.61, and 2.21, respectively. That is an inflation of 0.82 to 1.71 Likert points. Unassisted raters systematically overestimate epistemological quality, effectively misclassifying poor-to-moderate answers as good-to-excellent.

The "previous exposure" condition produces ratings intermediate between assisted and unassisted: 3.86 accuracy, 3.43 specificity, 3.30 completeness, 2.78 uncertainty. These are lower than the fully unassisted condition but higher than the fully assisted condition, suggesting that raters who have learned from prior assistance are more critical than naive raters but still less critical than when assistance is actively present.

Influence of AI Assistance on Rater Behavior

The helpfulness feedback data (Figure 4) shows that "when raters do not find assistance helpful, they give higher ratings." This is a non-trivial finding: it suggests that raters are not passive recipients of AI critiques, but actively evaluate the quality of the assistance and adjust their ratings accordingly. When they judge the critique to be unhelpful, they tend to push back by rating the answer more favorably. This provides some evidence against the concern that AI Assistance simply biases raters toward the assistant's judgment—though the paper is careful to note that the broader influence of assistance requires further study (Section 5).

The timing analysis (Appendix A.16, Figures 13–15) reveals that epistemological dimensions take longer to rate than presentational ones (consistent with the higher cognitive demand of evaluating factual accuracy, completeness, and uncertainty communication), and that lower ratings take longer than higher ratings (likely because low ratings require selecting specific issues from the predefined lists). The average time per epistemological dimension per example is roughly 10–15 seconds for high ratings and 15–20 seconds for low ratings (Figure 15).

Dimension-Aware Prompts: A Preliminary Trade-off

Table 1 reports a follow-up experiment on GPT-4 (conducted in November 2023 after an OpenAI model update) comparing the standard "basic" prompt with a "dimension-aware" prompt that includes explicit descriptions of the evaluation criteria: the answer should be "concise and tailored for a general audience... neutral, unbiased tone... factually accurate... specific to the question... comprehensively address all aspects... where scientific uncertainty exists, the answer should appropriately reflect this" (Table 8, "Dimension-aware main prompt"). The results show:

Epistemological dimensions improve: accuracy rises from 3.77 to 3.92 (+0.15), specificity from 3.29 to 3.46 (+0.17), completeness from 2.72 to 2.97 (+0.25), and uncertainty from 2.73 to 2.92 (+0.19).
Presentational dimensions mostly decline: style drops from 4.33 to 4.10 (−0.23), clarity from 4.34 to 4.24 (−0.10), correctness from 4.43 to 4.29 (−0.14).
Tone is the exception: it improves from 3.34 to 3.59 (+0.25), moving in the same direction as the epistemological dimensions.

The paper interprets this as "additional evidence for the existence of intrinsic tradeoffs between surface and epistemological quality" (Section 4.1). The trade-off is not necessarily a fundamental law—it could be an artifact of the specific prompt formulation—but it demonstrates that optimizing for epistemological completeness does not automatically improve presentation, and may even degrade it. This finding complicates the straightforward goal of making LLMs "better" at climate communication: improvements on one dimension may come at the expense of others.

Relationship to Question Source and Topic

The breakdown of ratings by question source and topic (Appendix A.13, Figures 11 and 12) shows relatively stable performance across most categories, with two notable patterns:

Synthetic Wikipedia questions receive slightly lower epistemological adequacy ratings than Google Trends or Skeptical Science questions (Figure 12). The paper speculates this is because Wikipedia-generated questions "ask for more details and very specific info compared to GTrends and Skeptical Sciences, and thus are harder to answer."
Policy-Mitigation-Adaptation questions receive lower ratings on most epistemological dimensions and particularly on tone (Figure 11). This may reflect the politically charged nature of climate policy questions—LLMs may struggle to maintain neutral tone when discussing mitigation strategies, or may lack sufficient detail on policy specifics.
Overall, "the source of the question does not affect the ratings significantly" and "the topic of the question does not show a strong correlation with answer quality" (Section 4.3), suggesting that the main results are not artifacts of particular question types.

The AIS Comparison: Attribution Is Orthogonal to Epistemological Quality

Section 4.6 and Figure 5 present the comparison between AIS attribution scores and epistemological dimension ratings for GPT-4 answers. Only 6.95% of answers are fully attributed (all keypoints supported by retrieved evidence), while 39.13% are partially attributed and 53.91% are not attributed (Table 17). At the keypoint level, 12% are fully supported, 54.79% partially supported, and 32.81% not supported.

The relationship between AIS categories and epistemological ratings (Figure 5) reveals:

Accuracy and specificity: Fully attributed answers have higher minimum ratings—unsupported answers can receive very low accuracy/specificity scores (down to ~1), while fully attributed answers rarely drop below ~2.5 for accuracy or ~2 for specificity. However, fully attributed answers can still receive low-to-moderate ratings (e.g., accuracy of 2.5–3.5) if the evidence itself is incomplete or the answer makes correct but poorly contextualized claims.
Completeness: Fully attributed answers have lower minimum ratings than partially attributed or unsupported answers—counter to what would be expected if attribution were a sufficient quality signal. The paper's explanation is that AIS "can only consider what is included in the answers, and not what important information is missing." An answer that grounds all its claims in retrieved passages may be systematically incomplete because it restricts itself to what was retrieved.
Uncertainty: Unsupported answers have more low-uncertainty ratings than supported ones, but the pattern is not monotonic—partially attributed answers look similar to fully attributed ones.

The Spearman's correlations confirm the absence of a monotonic relationship: 0.03 for accuracy (p = 0.65), −0.06 for specificity (p = 0.31), 0.002 for uncertainty (p = 0.97), −0.02 for completeness (p = 0.78). All are effectively zero and non-significant.

Issue-Level Diagnostics

Table 7 reports the percentage of specific issues identified by raters across all models, providing a fine-grained diagnostic of which types of failures each model tends to produce. Key patterns:

Style: InstructGPT (davinci-002) stands out with 20.15% of answers flagged as "repetitive" and 10.14% as "too short"—consistent with its shorter average answer length (Table 9, 4.99 sentences averaging only 14.3 words per sentence). Other models have repetition rates below 3%.
Tone: The dominant issue is "biased," flagged in 23.72% (Falcon) to 42.38% (InstructGPT davinci-003) of answers—making it by far the most common tone issue across all models. "Persuasive" is the second most common, ranging from 2.57% (davinci-002) to 10.69% (PaLM-2).
Accuracy: "Incorrect" is the most common accuracy issue, ranging from 3.23% (InstructGPT turbo) to 20.92% (davinci-002). "Science out of context" ranges from 2.56% (Falcon) to 9.37% (davinci-002). "Anecdotal" is notably high for InstructGPT (davinci-003) at 19.24% but much lower for other models (1.33–5.90%).
Specificity: "Vague" dominates, with 35.86% (Falcon) to 58.40% (InstructGPT davinci-003) of answers flagged—confirming that vagueness is a pervasive LLM failure mode across all models. "Irrelevant info" ranges from 3.79% (InstructGPT davinci-003) to 15.15% (davinci-002).
Completeness: "Not enough detail" is by far the most common issue, flagged in 47.57% (Falcon) to 79.53% (InstructGPT davinci-003) of answers—indicating that even when LLMs produce correct information, they rarely provide sufficient quantitative detail. "Does not address main parts" ranges from 8.78% (GPT-4) to 29.91% (davinci-002).
Uncertainty: "Uncertainty missing" is the dominant issue, flagged in 50.57% (Falcon) to 87.65% (InstructGPT davinci-003) of answers—showing that the failure to communicate uncertainty is near-universal. "Consensus missing" ranges from 9.34% (InstructGPT davinci-003) to 21.99% (davinci-002).

These issue-level results provide actionable diagnostics for model developers: they identify which specific aspects of climate communication each model fails on, going beyond aggregate scores to pinpoint systematic weaknesses (e.g., Falcon-180B-Chat's relatively low rate of "vague" and "uncertainty missing" issues compared to other models may explain its epistemic leadership).

Agreement Analysis

Inter-rater agreement is comprehensively analyzed in Appendix A.11 through multiple metrics. For Likert ratings:

Average pairwise distance (Table 12): For presentational dimensions (except tone), distances are typically 0.6–0.9, meaning raters rarely differ by more than one point on the 5-point scale. For epistemological dimensions, distances are slightly higher (0.9–1.3), reflecting the greater subjectivity of judgments about accuracy, specificity, completeness, and uncertainty. Tone shows relatively high distances (1.16–1.36), consistent with it being the most subjective dimension.
Krippendorff's alpha (Table 13): Similar pattern—higher for presentational dimensions (0.45–0.85) than epistemological ones (0.25–0.75), with tone notably low (0.25–0.48). The paper characterizes this as "medium agreement" overall, which is sufficient for system-level comparisons (distinguishing models) but not high enough for reliable answer-level evaluation.

For issue selection:

Pairwise agreement (Table 14): High (>90%) for most issues, but lower for "biased" (48–67%), "vague" (48–61%), "not enough detail" (44–69%), and "uncertainty missing" (49–80%)—the issues that are most subject to interpretation.
Krippendorff's alpha for low ratings only (Table 16): When raters agree that an answer deserves a low score (≤2) and must select specific issues, agreement improves substantially—e.g., "vague" Krippendorff's alpha rises from 0.05–0.21 (Table 15, all ratings) to 0.37–0.83 when computed only on low ratings. This suggests that the framework's core diagnostic signal (identifying the type of issue in a clearly problematic answer) is much more reliable than aggregate agreement metrics suggest, because much of the disagreement comes from marginal cases (ratings of 3–5 where issues are not required to be selected).

Ablation Studies and Robustness Checks

AI Assistance presence vs. absence: The comparison of assisted vs. unassisted GPT-4 ratings (Tables 3 and 4, bottom two rows) serves as the primary ablation confirming that assistance is essential for detecting epistemological issues. The inflation of unassisted epistemological ratings by 0.82–1.71 Likert points (Section 4.5, Figure 3) demonstrates that raters without assistance substantially overestimate epistemological quality. This is the most critical ablation: it validates the paper's central methodological claim that scalable oversight is necessary because unaided raters miss many epistemological problems.

Previous exposure to AI Assistance as a partial substitute: The "GPT-4, no assistance, previous exposure" condition (Tables 3 and 4, Figure 3) tests whether raters can learn from prior assisted rating and apply that learning when assistance is absent. The result—intermediate ratings between fully assisted and fully unassisted—is non-obvious: it suggests that AI Assistance has both an immediate effect (providing specific critiques that help detect issues in the current answer) and a training effect (teaching raters to be more critical readers in general). The paper does not quantify how long this training effect persists or how many assisted examples are needed to achieve it, but the finding is methodologically important for designing evaluation pipelines where assistance might be intermittently available.

Helpfulness feedback as a diagnostic for rater independence: Figure 4 demonstrates a negative correlation between the helpfulness raters report for the AI Assistance (rated on the same 1–5 scale) and the ratings they assign to answers. When raters find assistance unhelpful, they tend to give higher ratings—suggesting that they are not blindly following the assistant's critique but are instead pushing back when they judge the critique to be invalid. This is not an ablation in the traditional sense, but it serves as a robustness check against the concern that AI Assistance simply biases raters toward lower scores. The paper does not quantify the strength of this correlation or report whether it differs by dimension, limiting the interpretability of this finding.

PRM vs. ORM for revisions: Not applicable to this paper (no PRM, ORM, or revision models are used).

Difficulty bin granularity: Not applicable in the traditional sense (no difficulty bins), but the breakdown by question source and topic (Figures 11 and 12) serves an analogous function. The finding that ratings are relatively stable across question sources and topics, with only minor differences (lower epistemological ratings for Wikipedia-generated questions and policy topics), suggests that the main results are not artifacts of a particular question type. However, the paper does not systematically compare "easy" vs. "hard" questions or bin questions by model-specific difficulty (as in the compute-optimal scaling literature).

LLM rater as an alternative to human raters: Appendix A.15 reports a preliminary experiment in which GPT-4 itself rates the answers using the same prompts and assistance protocol as human raters (Table 21). The LLM rater shows several interesting patterns (Table 22): (1) it also benefits from assistance—ratings are lower with assistance than without, consistent with the human rater finding; (2) it agrees with human raters on large quality differences—e.g., it correctly identifies InstructGPT (davinci-002) as the worst model across all dimensions; (3) it exhibits an "egocentric bias"—it rates OpenAI models (GPT-4, ChatGPT, InstructGPT variants) more generously than humans do relative to non-OpenAI models (PaLM-2, Falcon). Specifically, the LLM rater gives GPT-4 a 4.13 accuracy vs. humans' 3.67, while giving Falcon a 4.03 vs. humans' 3.81 (humans rank Falcon higher on accuracy, while the LLM rater ranks GPT-4 higher). This contrast between human and LLM raters on relative model rankings is a methodological finding: it suggests that LLM raters, even with the same assistance protocol, are not a drop-in replacement for human raters because they exhibit systematic biases that humans do not share.

ReST $^{EM}$ revision model: Not applicable to this paper (no self-improvement or revision model training is performed).

Attribution (AIS) as a proxy for epistemological quality: The AIS experiment (Section 4.6, Figure 5, Tables 17 and 19) is perhaps the most important robustness check: it tests whether an established, more automatable evaluation paradigm (attribution to identified sources) can substitute for the paper's multi-dimensional epistemological evaluation. The near-zero Spearman correlations decisively show that AIS is orthogonal to epistemological quality, validating the paper's claim that the additional dimensions (completeness, specificity, uncertainty) capture something that attribution misses. Table 19 provides qualitative examples: an answer about public awareness of CO2's effects is fully attributable yet scores only 1.6/5 on completeness (it omits key milestones), and an answer about climate sensitivity that is not attributable still scores 4.7/5 on accuracy (it is correct but the evidence retrieval system failed to find supporting passages). These examples concretize the statistical finding, making the case that attribution is complementary rather than sufficient.

Critical Assessment

The paper makes three central claims through its experimental design: (1) LLMs exhibit a significant gap between presentational and epistemological adequacy on climate communication; (2) AI Assistance is necessary for reliable epistemological evaluation because unaided raters miss many issues; (3) the multi-dimensional framework captures quality dimensions that attribution-based evaluation misses. The experiments provide strong evidence for each claim, but with specific limitations that constrain the generality of the conclusions.

Claim 1: Presentational-epistemological gap. The evidence for this claim is robust within the studied setup. Figure 2 shows a consistent gap of 1–2 Likert points across all seven models, with the pattern holding across question sources and topics (Figures 11, 12). The pairwise t-tests confirm that epistemological differences between models are often significant, and the issue-level analysis (Table 7) provides granular diagnostic evidence for which epistemological failures are most common (vagueness in 35–58% of answers, missing detail in 48–80%, missing uncertainty in 51–88%).

However, the answer format—a standardized 3–4 sentence paragraph—is an artificial constraint that likely amplifies the epistemological gap, particularly on completeness. The paper acknowledges this: "these dimensions may be difficult to satisfy in short 3-4 sentence answers" (Section 4.1), and "being comprehensive in such a short space may be harder than being accurate." This is not a fatal confound—the prompt length constraint is realistic for many conversational AI applications—but it means the absolute epistemological scores should not be interpreted as the ceiling of what these models can achieve. A model prompted to produce longer, more detailed answers might close part of the gap, particularly on completeness. The paper notes (Section 4.4) that "LLMs don't seem to make a good use of space," pointing to the "pragmatic style" of padding answers with "somewhat relevant but unspecific, cautionary and sometimes preachy reminders," which suggests that the gap is at least partly a genuine failure of content selection rather than purely a space constraint. But the experiment does not directly test whether removing the length constraint improves epistemological quality, which would have been a useful ablation.

The "dimension-aware prompt" experiment (Table 1) partially addresses this by showing that explicitly instructing the model about the evaluation criteria can improve epistemological scores (accuracy +0.15, completeness +0.25, uncertainty +0.19) at the cost of presentational scores (style −0.23, clarity −0.10). This is a preliminary result (only GPT-4, only one prompt formulation, and conducted after a model update that prevents direct comparison with the main results), but it demonstrates that the gap is not immutable—prompt engineering can shift the presentation-epistemology trade-off. A more thorough prompt engineering study across all models and prompt formulations would have strengthened this finding.

Claim 2: AI Assistance is necessary for reliable epistemological evaluation. The evidence for this claim comes from three experimental conditions (Figure 3, Tables 3–4, Table 20). The most compelling piece is the validation experiment: majority detection of planted issues improves from 60% to 76.67% with assistance, confirming that assistance helps raters find real issues they would otherwise miss. The large inflation of unassisted epistemological ratings (0.82–1.71 points) further confirms that raters without assistance systematically miss problems.

However, several constraints limit how strongly this claim can be interpreted:

The validation set is small (30 examples) and constructed by the authors, not sampled from the actual evaluation data. It demonstrates that assistance helps for the types of issues the authors could construct, but it does not guarantee that assistance is equally helpful for all issue types that occur in real LLM outputs. The fact that "anecdotal evidence" was the one issue type not detected by any rater, even with assistance, illustrates this limitation—assistance helps, but not uniformly.
The rater population is narrow: all raters have at least undergraduate degrees in climate-related fields. The paper does not test whether AI Assistance is more or less beneficial for raters with different expertise levels. It is plausible that the benefit of assistance would be even larger for lay raters (who lack the domain knowledge to independently identify epistemological issues) and smaller for true climate science experts (who might already catch most issues). This limits the claim about "necessity" to the specific rater population studied.
The assistance itself is generated by GPT-4, which introduces a circularity: the same model family that produces the answers also produces the critiques. The paper does not test whether assistance generated by a different model (or a committee of models) would be equally effective, or whether GPT-4's assistance is biased toward or against certain models. The LLM rater experiment (Table 22) suggests that GPT-4 exhibits an egocentric bias when rating—it rates OpenAI models higher than humans do relative to non-OpenAI models—which raises the possibility that GPT-4-generated assistance may also exhibit systematic biases. The paper acknowledges this: "the assistance may also influence the raters beyond enhancing discovery. It may only help in the discovery of some issues but not others" (Section 5), but it does not quantify these biases.
The helpfulness analysis (Figure 4) is incomplete. The paper reports that when raters find assistance unhelpful, they give higher ratings, interpreting this as evidence that raters "can think critically about the assistance and do not follow it blindly." But the converse pattern is equally interpretable: when raters find assistance helpful, they may be more likely to lower their ratings, which could reflect either genuine issue detection or undue influence. The paper does not report whether the issues detected with assistance are more likely to be "real" (as validated against the synthetic examples) than issues detected without assistance, beyond the 30-example validation set.

Claim 3: The framework captures dimensions that attribution (AIS) misses. The evidence for this claim is strong and well-structured. The orthogonality finding—Spearman's correlations near zero across all four epistemological dimensions—is statistically unambiguous. The qualitative analysis (Figure 5, Table 19) provides concrete examples of the divergence, and the structural explanation (AIS cannot detect omissions) is logically compelling.

A limitation of this analysis is that the AIS evaluation uses the same Wikipedia retrieval pipeline as the AI Assistance, and that pipeline only successfully retrieves evidence for 87.7% of questions. The AIS result therefore conflates two failure modes: (1) the answer makes claims that are not attributable to Wikipedia (genuine attribution failure), and (2) the Wikipedia retrieval pipeline failed to find the relevant article (retrieval failure). The paper does not disentangle these, so the finding that 53.91% of answers are "not attributed" overstates the rate at which answers make unverifiable claims. This does not affect the orthogonality finding—the correlation between AIS and epistemological ratings is computed relative to the same retrieval pipeline, so any retrieval noise affects both—but it means the absolute AIS scores should not be interpreted as definitive attribution assessments.

Missing experiments that would have strengthened the paper:

Difficulty calibration: The paper does not bin questions by difficulty (e.g., by unaided rater agreement, or by model agreement on correctness). Without difficulty bins, it is impossible to tell whether the presentational-epistemological gap is uniform across easy and hard questions, or whether epistemological failures are concentrated on questions that are inherently challenging. In the compute-optimal scaling literature, difficulty-conditioned analysis reveals that different strategies work best on different difficulty levels—an analogous analysis here could reveal whether some models handle hard questions better than others or whether AI Assistance is more valuable for certain question types.
Multiple assistance sources: The AI Assistance is always GPT-4. Testing assistance from a different model (e.g., Falcon or PaLM-2) would reveal whether the assistance effect is model-specific or generalizable. The LLM rater experiment hints at model-specific biases; testing whether those biases propagate into the assistance critiques would be valuable for understanding the protocol's robustness.
Rater expertise gradient: Including raters with varying levels of climate expertise (e.g., lay raters, undergraduate-level, graduate-level, domain experts) would allow the paper to characterize how the benefit of AI Assistance varies with rater knowledge. This is directly relevant to the scalable oversight framing: if lay raters with assistance can match the performance of domain experts without assistance, then AI Assistance enables scaling oversight to raters who lack specialized training. If lay raters with assistance still underperform experts without assistance, then the protocol is augmenting but not replacing expertise.
Longitudinal study of the training effect: The "previous exposure" condition provides a snapshot of rater learning, but the paper does not track individual raters over time to quantify how the training effect accumulates, whether it plateaus, or whether it generalizes to new dimensions and models. This is understandable given the scale of the study, but it limits the claim that AI Assistance has a lasting training benefit.
Cross-domain validation: All questions are about climate change. Testing the framework on a different science communication domain (e.g., vaccine information, nutrition science, or another domain with established science communication research) would demonstrate that the dimensions and the AI Assistance protocol transfer beyond climate.

Overall assessment. The experiments support the paper's claims within the studied constraints: seven recent LLMs on 300 climate questions rated by three climate-educated raters per answer with GPT-4-generated grounded assistance. The presentational-epistemological gap is real and substantial. AI Assistance demonstrably increases issue detection. The AIS orthogonality finding is clean and well-supported. The primary limitations are the single domain (climate), the single assistance model (GPT-4), the relatively narrow rater population, the artificial answer length constraint, and the small (30-example) validation set. The paper is transparent about most of these limitations, explicitly calling out the need for better understanding of "undesired influence on the raters" (Section 7), the small validation experiment (Appendix A.14), and the preliminary nature of the LLM rater and dimension-aware prompt experiments. The main results justify the paper's core methodological contribution—that evaluating LLM science communication requires multi-dimensional frameworks with grounded assistance—but the absolute scores and model rankings should be interpreted as specific to this evaluation protocol, not as universal assessments of model capability.

6. Limitations and Trade-offs

The Difficulty Estimation Bottleneck: AI Assistance Is Costly and Model-Specific

The assumption or constraint. The AI Assistance pipeline—which the paper demonstrates is essential for detecting epistemological issues (unaided raters miss many problems, inflating ratings by 0.82–1.71 Likert points; Tables 3–4)—requires a complex, multi-stage automated system: keypoint extraction, Wikipedia URL generation, paragraph scoring, and dimension-specific critique generation, all using GPT-4. The paper does not quantify the computational or monetary cost of this pipeline, nor does it analyze whether the pipeline generalizes across different AI assistance models. The protocol relies on GPT-4 for consistency (Section 3.2: "For consistency, we always use GPT-4 for this purpose"), but the paper explicitly notes a circular dependency: "GPT-4 is also used to produce the auxiliary data" (Section 4.1, footnote), and the LLM rater experiment (Appendix A.15, Table 22) reveals that GPT-4 exhibits an egocentric bias—it rates OpenAI models more favorably relative to non-OpenAI models than human raters do. This raises the possibility that GPT-4-generated assistance critiques may be systematically biased toward or against specific model families.

The consequence. If the AI Assistance pipeline produces critiques that are not model-agnostic—i.e., if it is more likely to identify issues in PaLM-2 or Falcon answers than in GPT-4 or ChatGPT answers, controlling for actual quality—then the main evaluation results are partially an artifact of the evaluation apparatus rather than genuine quality differences. The finding that Falcon-180B-Chat leads all models on epistemological dimensions (Table 4) could be distorted if GPT-4-generated assistance is systematically harsher on non-OpenAI outputs. Even if the bias is small, the pipeline cost creates a practical deployment barrier: any organization wanting to use this framework must have API access to a model of GPT-4's capability level, must pay for the keypoint extraction, URL generation, passage scoring, and critique generation for every evaluated answer, and must maintain the pipeline as the underlying models change. The paper does not provide a cost analysis, but the described pipeline (extracting keypoints, generating URLs, scoring multiple paragraphs per keypoint, generating eight separate critiques per answer) involves many LLM calls per evaluated answer. For a realistic deployment evaluating thousands of answers across multiple models, this cost may be prohibitive.

What evidence exists in the paper. The paper's own data provides indirect evidence of the model-dependence issue. The LLM rater experiment (Table 22) shows that when GPT-4 acts as a rater, it assigns higher accuracy ratings to OpenAI models relative to human raters: GPT-4 rates itself at 4.13 accuracy vs. humans' 3.67, ChatGPT at 4.18 vs. humans' 3.48, while rating Falcon at 4.03 vs. humans' 3.81. If the AI Assistance critiques are generated by GPT-4, which shows this self-preference pattern when rating, it is plausible that the critiques inherit a similar bias. The paper acknowledges this risk explicitly: "the assistance may also influence the raters beyond enhancing discovery. It may only help in the discovery of some issues but not others. There may also be errors caused by models falsely pointing out issues and wrongly convincing the raters. The issues identified will likely vary by model" (Section 5). However, the paper does not run the critical ablation: generating AI Assistance using a non-OpenAI model (e.g., Falcon or PaLM-2) and comparing whether the resulting ratings and model rankings shift.

Mitigation status. The paper partially mitigates this concern through indirect evidence: (1) the helpfulness analysis (Figure 4) shows that raters push back against unhelpful assistance, suggesting they are not fully passive; (2) the validation experiment (Table 20) shows that assistance increases detection of known planted issues by majority vote from 60% to 76.67%, confirming that at least some of the additional issues detected are genuine, not artifacts of model bias; (3) the paper is transparent about this limitation, calling for future work to "understand and mitigate undesired influence on the raters" (Section 7) and to "develop a framework to understand and mitigate undesired influence" (Section 5). But the model-dependence of the assistance pipeline itself—the fact that the evaluation apparatus uses the same model family it evaluates—is a structural limitation that the current study design cannot fully address and that future work must resolve before the framework can claim to produce model-agnostic rankings.

Hard Problems: The Framework Provides No Path for Evaluation Where Ground Truth Is Fundamentally Unavailable

The assumption or constraint. The framework's epistemological dimensions—accuracy, specificity, completeness, uncertainty—depend on raters' ability to compare the answer against what is scientifically known. For questions where the science is genuinely unsettled, where multiple legitimate interpretations exist, or where the relevant knowledge is not captured in Wikipedia (the sole evidence source for AI Assistance), the framework provides no mechanism for evaluation. The paper explicitly limits evidence to Wikipedia because "GPT-4 is fairly consistent in generating relevant, valid Wikipedia URLs, while the quality is lower for the unrestricted web" (Section 3.2). However, Wikipedia is an incomplete and time-lagged representation of climate science knowledge, and for 12.3% of questions, GPT-4 cannot even produce a valid Wikipedia URL (Section 3.2). For these questions, the epistemological AI Assistance has no grounding evidence, and the paper does not specify what raters see or how they should evaluate epistemological dimensions in the absence of retrievable evidence.

The consequence. The framework effectively restricts its epistemological evaluation to questions whose answers can be verified against Wikipedia. This excludes an important class of climate information needs: questions about cutting-edge research, regionally specific impacts not documented in Wikipedia articles, emerging climate phenomena that post-date the Wikipedia articles used, or genuinely contested aspects of climate science where no single authoritative account exists. The paper acknowledges this indirectly through the Wikipedia URL success rate (87.7%—Section 3.2), but does not analyze how epistemological ratings differ between questions with and without retrievable Wikipedia evidence. If questions without evidence systematically receive different ratings (e.g., raters are unable to assess them and default to higher scores), this would bias the results in unknown ways. More fundamentally, the framework cannot evaluate whether LLMs appropriately handle genuine scientific uncertainty—the uncertainty dimension is about communicating the uncertainty that exists in the scientific community, but when the science itself is unsettled and Wikipedia does not capture that nuance, neither raters nor AI Assistance have a reliable reference point for judging whether the communication was adequate.

What evidence exists in the paper. The paper reports that for 87.7% of questions, GPT-4 produces a valid Wikipedia URL from which evidence can be extracted (Section 3.2), implying that 12.3% of the 300 questions (approximately 37 questions) have no grounded epistemological AI Assistance. The paper does not report whether epistemological ratings differ between these 37 questions and the other 263. The AIS experiment (Section 4.6) provides a related finding: 53.91% of answers are "not attributed" to the retrieved evidence, and only 6.95% are fully attributed (Table 17). If half of all answers cannot be supported by the Wikipedia evidence pipeline, it is unclear how raters are evaluating epistemological adequacy for those answers—they may be relying on their own domain knowledge, the AI Assistance (which itself cannot ground its critique for those answers), or some combination. The paper does not analyze this boundary case.

Mitigation status. The paper acknowledges this limitation partially, noting that "evaluation of systems to be deployed in critical contexts requires additional grounding and expert verification" (Impact Statement) and that "the evaluation is limited to the evaluated context, and we make no claims that models can be trusted and deployed outside of that context." The restriction to Wikipedia is presented as a pragmatic choice for source quality and uniformity, not as a claim that Wikipedia is comprehensive. However, the paper does not propose an alternative evidence pipeline for the 12.3% of questions without valid URLs, does not analyze whether these questions differ systematically from those with URLs (e.g., are they harder? More specific? About more recent events?), and does not suggest how the framework could be extended to domains where Wikipedia is insufficient. Future work could incorporate multiple evidence sources (peer-reviewed literature databases, IPCC reports, expert-curated knowledge bases) and explicitly model the cases where evidence is unavailable or contested as a distinct evaluation regime.

The Answer Format Constraint Amplifies the Epistemological Gap and Limits Generalizability

The assumption or constraint. All evaluated answers are generated with a standardized prompt instructing the model to "Answer each question in a 3-4 sentence paragraph" (Section 3.1.2, Table 8). The paper justifies this with the argument that "a direct question is the most common way for users to get answers from LLMs" and that this "provides a valuable baseline, reducing variance due to individual LLM's skills and optimization effort, and limiting confounding factors." However, this format constraint directly penalizes the epistemological dimensions: completeness requires that answers "address everything the question asks for" and "provide enough necessary details, e.g., numbers, statistics, and details" (Table 11), and uncertainty communication often requires qualifying statements and discussion of confidence intervals that consume space. The paper acknowledges this tension: "these dimensions may be difficult to satisfy in short 3-4 sentence answers. Being comprehensive in such a short space may be harder than being accurate" (Section 4.1).

The consequence. The absolute epistemological scores—particularly completeness (2.0–2.7 across models, Table 4), specificity (2.5–3.2), and uncertainty (2.0–2.6)—cannot be interpreted as the ceiling of what these models are capable of. A model that is simply obeying the length constraint cannot simultaneously be fully complete on a complex climate question; the low completeness scores partly reflect a format-imposed trade-off rather than a fundamental capability limitation. This matters for two reasons. First, it means the headline finding of a "significant gap between surface and epistemological qualities" (Abstract) may overstate the gap that would exist in realistic deployment settings where models are allowed (or prompted) to produce longer, more detailed answers. Second, the model rankings on epistemological dimensions might differ under a less restrictive format, because models differ in how they allocate the limited space—some may prioritize factual claims over hedging, or breadth over depth—and these allocation strategies interact with the length constraint in model-specific ways. The paper's finding that "LLMs don't seem to make a good use of space" (Section 4.4) and that they "fill parts of the answer with somewhat relevant but unspecific, cautionary and sometimes preachy reminders" suggests that space allocation strategy is an independent variable affecting epistemological scores, but the paper cannot disentangle poor space use from genuine inability to provide complete answers because the format constraint is fixed.

What evidence exists in the paper. The dimension-aware prompt experiment (Table 1) provides partial evidence: when GPT-4 is prompted with explicit evaluation criteria (the augmented prompt), epistemological scores improve (completeness +0.25, specificity +0.17, uncertainty +0.19), while presentational scores decline (style −0.23, clarity −0.10). This was done with the same length constraint, suggesting that the models can improve epistemological quality within the 3–4 sentence format if prompted appropriately. But this is only a single prompt variant on a single (post-update) model, and it does not test whether removing the length constraint entirely would produce substantially larger improvements. The paper notes that "this experiment was carried out in November 2023, after a major release from OpenAI, on Nov 6. GPT-4's performance cannot be directly compared with the previous results" (Table 1 footnote), further limiting what can be concluded from this ablation. The answer statistics in Table 9 show that most models actually produce 4–5 sentences (not 3–4), with average sentence lengths ranging from 14.3 (davinci-002) to 22.03 (Falcon-180B-Chat) words, so the effective answer length varies across models—but the degree of this variation is not extreme enough to test whether longer answers systematically improve epistemological scores.

Mitigation status. The paper partially acknowledges this limitation by noting that "space constraints alone do not seem sufficient to explain the result" (Section 4.1), pointing to the observation that LLMs waste space on generic content. The analysis of "pragmatic style" (Section 4.4) with its concrete example (Table 2: the answer about heading toward an ice age that pads with loosely relevant reminders about extreme weather) supports this argument—the gap is at least partly a genuine failure of content selection, not just a space shortage. But the paper does not run the necessary ablation: evaluating the same models on the same questions with no length constraint (or a much longer one), measuring whether epistemological scores improve and whether the presentational-epistemological gap narrows. Without this ablation, the claim that the gap is real and substantial is supported, but the magnitude of the gap is confounded with the format constraint.

Single Domain, Single Model Family for Assistance, Narrow Rater Population: Transferability Is Unknown

The assumption or constraint. The entire evaluation framework—the eight dimensions, the AI Assistance pipeline, the rater qualification process, and the validation against synthetic examples—is developed and tested exclusively on climate change questions, using GPT-4 for all components of the AI Assistance pipeline, with raters who all hold undergraduate degrees in climate-related fields. The paper makes no claims about generalizability: "the evaluation is limited to the evaluated context, and we make no claims that models can be trusted and deployed outside of that context" (Impact Statement). However, the underlying ambition of the framework—to provide a "comprehensive evaluation framework" (Abstract) for LLM science communication—implies an expectation that the methodology transfers to other domains.

The consequence. A practitioner wanting to adapt this framework to a different science communication domain (health, vaccine information, environmental policy, nutrition science) faces three unknown factors. First, the evaluative dimensions—derived from climate communication scholarship—may not transfer cleanly. For example, the "uncertainty" dimension draws on findings that communicating scientific consensus about climate change functions as a "gateway belief" that motivates public action (van der Linden et al., 2015; Section 2.2). It is not obvious whether consensus communication has the same psychological function in other scientific domains, and the operationalization of the dimension (the specific issue types: "uncertainty missing," "consensus missing," "contradicting evidence missing") may need domain-specific adaptation. Second, the AI Assistance pipeline's reliance on Wikipedia as the sole evidence source may be more or less appropriate depending on the domain—climate science has relatively good Wikipedia coverage, but other science domains may have sparser, more outdated, or more contested Wikipedia articles. Third, the rater population's domain expertise (climate-educated undergraduates) was selected to match the domain—a similar framework in health communication would need raters with health education—but the paper provides no guidance on what minimum expertise level is necessary, whether the AI Assistance can compensate for lower expertise, or how to calibrate the admission test for a new domain.

What evidence exists in the paper. The paper provides no cross-domain evidence. The only domain-internal variation comes from the question source analysis (Appendix A.13, Figures 11–12), which shows that Wikipedia-generated questions (which are more specific and detailed) receive slightly lower epistemological ratings than Google Trends or Skeptical Science questions, but the differences are small. This is evidence of robustness within the climate domain, but it does not speak to cross-domain transferability. The paper's demonstration that the AIS framework (which is domain-agnostic) is orthogonal to the epistemological dimensions (Section 4.6, Figure 5) actually strengthens the case that the framework captures something domain-specific that attribution misses—but this also means that the dimensions' relevance to other domains is not guaranteed by their climate-specific grounding.

The LLM rater experiment (Appendix A.15, Table 22) shows that GPT-4, when prompted with the same rating statements and provided with the same AI Assistance, can produce ratings that agree with human raters on broad patterns (correctly identifying InstructGPT davinci-002 as the worst model), but it also exhibits systematic biases (favoring OpenAI models). This is partial evidence that the rating dimensions are sufficiently well-operationalized to be applied by non-human raters, but it does not address whether the dimensions would be meaningful in a different domain.

Mitigation status. The paper makes no attempt to address this limitation beyond the explicit scope disclaimer. Unlike some other papers that test their evaluation frameworks on multiple benchmarks or domains, this paper is a deep single-domain study. This is a legitimate research choice—the climate domain is sufficiently complex and high-stakes to warrant dedicated evaluation methodology—but it means the framework's claim to being a general-purpose "science communication evaluation framework" is aspirational rather than demonstrated. The paper suggests future work on extending the framework "to evaluate references in a principled, systematic way" (Section 4.6), which would be one step toward broader applicability, but does not propose cross-domain validation.

The Validation Set Is Small: The Framework's Issue Detection Capability Is Only Partially Calibrated

The assumption or constraint. The validation experiment (Appendix A.14) tests whether raters can detect known issues using 30 synthetic questions and answers constructed by the authors, each exhibiting a specific issue from the full list of possible issues. The experiment shows that with AI Assistance, majority detection improves from 60% to 76.67% (Table 20), confirming that assistance helps raters find real problems. However, 30 examples constructed by the researchers to exhibit clear, unambiguous issues is a fundamentally different distribution from the actual evaluation data, where issues may be subtle, ambiguous, or co-occurring. The paper acknowledges this limitation implicitly by noting that "as we do not have access to gold ratings, calibration of raters remains an open issue" (Section 5), but the validation experiment is the only quantitative evidence that the framework's issue detection is accurate.

The consequence. The 76.67% majority detection rate with assistance is a lower bound on the framework's ability to detect clear, unambiguous, single issues—not an estimate of its ability to detect the full spectrum of issues in real LLM outputs. For real answers, issues may be subtle (e.g., a statement that is technically correct but misleadingly incomplete), may co-occur (an answer may be simultaneously vague, missing detail, and failing to communicate uncertainty), or may involve judgment calls that different raters could reasonably disagree about. The validation experiment cannot calibrate detection for these cases because the authors cannot construct a gold-standard set of subtle, ambiguous, or co-occurring issues without introducing their own judgment biases. This means that the absolute detection rates in the main evaluation—the percentages of answers flagged with specific issues in Table 7—should be interpreted as what the framework detects given its current calibration, not as the true prevalence of those issues. The framework may systematically under-detect certain issue types (the fact that "anecdotal" was the one issue type not detected by any rater in the validation experiment, even with assistance, is a concrete example) and over-detect others (the AI Assistance may consistently flag a particular type of issue that human raters accept without independent verification).

Furthermore, the small size of the validation set (30) means that detection rates are estimated with high variance. If a future deployment of the framework wanted to claim a specific detection rate for a specific issue type (e.g., "the framework detects 90% of factual inaccuracies"), the 30-example set does not support that precision—and the paper does not report per-issue-type detection rates anyway.

What evidence exists in the paper. The validation experiment results (Table 20) show the detection rates for all 30 examples combined, with no breakdown by issue type, dimension, or question source. The paper reports that "there was just one issue not detected by any rater: anecdotal evidence" and notes that "all raters did correctly rate the dimension ('accuracy' as 1 or 2), but highlighted only the 'incorrect' issue. The answer was incorrect, based on anecdotal evidence" (Appendix A.14). This is an interesting failure mode—raters correctly identified the answer as inaccurate but misclassified the type of inaccuracy—but the paper does not report how often this type of misclassification occurs in the main evaluation, where issue types are used to compute the percentages in Table 7.

The inter-rater agreement analysis on issue selection (Tables 14–16) provides indirect evidence about calibration. The observation that Krippendorff's alpha for issues improves substantially when computed only on low ratings (Table 16 vs. Table 15)—e.g., "vague" alpha rises from 0.05–0.21 on all ratings to 0.37–0.83 on low ratings—suggests that raters agree reasonably well on issue types when an answer is clearly problematic, but that agreement is lower in borderline cases. This is consistent with the validation experiment's design: the synthetic examples exhibited clear issues, and raters detected them at reasonable rates with assistance. But the main evaluation includes many marginal answers (recall that epistemological ratings cluster in the 2–3 range for most models), where issue classification is more ambiguous and the validation experiment provides no calibration.

Mitigation status. The paper is transparent about the absence of gold ratings and the resulting calibration challenge (Section 5). The validation experiment is presented as partial mitigation—it demonstrates that the framework can detect clear issues—but the paper does not claim that the 30 examples are representative of the main evaluation distribution. Future work would need a substantially larger validation set, ideally sampled from the actual evaluation data and annotated by multiple domain experts to establish a richer gold standard, to properly calibrate the framework's detection capability for subtle and co-occurring issues. The paper's suggestion that future work should "consider explicitly addressing this subjectivity in the data collection process" (Section 5, citing Rottger et al., 2022) acknowledges this gap, but the current study does not address it.

The Framework Decouples Evaluation Dimensions but Does Not Resolve Their Application-Specific Trade-off

The assumption or constraint. The paper deliberately does not combine the eight evaluation dimensions into a single aggregated metric. Section 2.3 states: "We don't address the important question of how individual dimensions should be combined in a single metric, e.g., for model selection and benchmarking. This is a complex topic which requires assigning a value to each individual dimension. We also believe that the combination of these scores will vary by application." This is presented as a principled design choice—the framework is designed to surface multi-dimensional quality, not to collapse it into a scalar that might obscure important trade-offs—and the paper provides evidence for such trade-offs: the dimension-aware prompt experiment (Table 1) shows that improving epistemological dimensions comes at the cost of presentational ones (style drops 0.23, clarity drops 0.10 when epistemological criteria are added to the prompt), and the model rankings across dimensions are inconsistent (ChatGPT leads on presentation but trails on epistemology; Falcon leads on epistemology but is mid-tier on presentation).

The consequence. While the decision not to aggregate is methodologically defensible for a research study, it leaves a critical gap for practitioners who need to make deployment decisions. If a product team is choosing between two models for a climate information chatbot—one with high presentational scores but low epistemological scores, and one with the reverse profile—the framework provides no guidance on which trade-off to prefer. This is not a minor issue: the paper's own results show that the best-presenting model (ChatGPT) has some of the worst epistemological scores (2.05 uncertainty, 2.26 completeness), while the best-epistemology model (Falcon-180B-Chat) has good but not leading presentational scores. Which model should be deployed? The answer depends on the application context—an educational tool for students might prioritize completeness and accuracy, while a public-facing chatbot might prioritize clarity and tone—but the framework provides no method for encoding these context-specific preferences into a selection criterion.

The paper argues that "the combination of these scores will vary by application," which is true but incomplete. A complete evaluation framework would provide not just the scores but also a methodology for combining them given application-specific weights. Without this, the framework risks becoming a benchmarking exercise that produces eight numbers per model without a clear decision rule for model selection. The danger is that practitioners will either (a) default to averaging all dimensions (which the paper's own data shows would obscure the presentation-epistemology trade-off), (b) cherry-pick the dimensions that favor their preferred model, or (c) ignore the framework entirely because it does not answer their deployment question.

What evidence exists in the paper. The dimension-aware prompt experiment (Table 1) is the paper's clearest evidence that the trade-off is real and quantifiable: epistemological improvements of +0.15 to +0.25 come at a cost of −0.10 to −0.23 on presentation. The per-model rankings (Section 4.2, Tables 3–6) show that no single model dominates across all dimensions. The pairwise t-tests (Tables 5–6) confirm that the ranking orders differ between presentational and epistemological dimensions. The analysis of "pragmatic style" (Section 4.4) provides a qualitative characterization of one specific trade-off mechanism: LLMs "fill parts of the answer with somewhat relevant but unspecific, cautionary and sometimes preachy reminders," which may improve the perception of completeness (for unaided raters) while actually reducing specificity and wasting space that could be used for genuine detail. The AIS comparison (Section 4.6) provides another angle: answers that are fully attributable can still be incomplete, and answers that are not attributable can still be accurate, showing that no single dimension suffices as a proxy for overall quality.

Mitigation status. The paper acknowledges this as a deliberate scope limitation rather than a failure. The statement in Section 2.3 is explicit about the choice, and the justification—that application-specific value judgments are required—is reasonable. However, the paper does not even sketch what a combination methodology would look like (e.g., Pareto optimization across dimensions, application-specific weighting schemes, or interactive model selection tools that let stakeholders explore trade-offs). The dimension-aware prompt experiment gestures toward a resolution: if models can be prompted to shift their position on the presentation-epistemology frontier, then the optimal model might not be a fixed choice but a prompt-engineered variant selected per-application. But the paper does not explore this systematically across models or prompt variants. The Impact Statement notes that "the evaluation of systems to be deployed in critical contexts requires additional grounding and expert verification," which implicitly acknowledges that the framework as presented is insufficient for deployment decisions—it is an analysis tool, not a certification process—but the gap between analysis and deployment remains unfilled.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose a new model, training objective, or inference strategy. It proposes a new source of authority for AI evaluation: domain scholarship. The climate communication literature—spanning communication science, sociology, psychology, and education—has spent decades studying what makes science communication effective (Jamieson et al., 2017; Trench & Bucchi, 2021), identifying dimensions like the communication of scientific consensus as a "gateway belief" (van der Linden et al., 2015) and documenting how tone, specificity, and uncertainty affect audience trust and understanding. Before this work, that body of knowledge had no systematic interface with LLM evaluation. The paper constructs that interface, translating eight evaluative dimensions from the literature into operationalized Likert-scale statements with structured issue menus and a grounded scalable oversight protocol, and demonstrates that these dimensions reveal a fundamental dissociation—the presentational-epistemological gap—that standard NLP evaluation paradigms would not detect.

This is a methodological reframing, not an incremental improvement. It changes where evaluative criteria are expected to come from when assessing LLMs for domain-specific communication tasks. The NLP evaluation tradition—accuracy on held-out test sets, BLEU/ROUGE, human preference judgments, attribution to sources—derives its criteria from ML engineering concerns: can the model reproduce the reference? Is the output fluent? Can it be traced to a document? These criteria are not wrong, but they are incomplete for communication quality. A model can be fluent, attributable, and factually accurate while still failing as a science communicator—producing answers that are vague, incomplete in ways that mislead, or silent about critical uncertainties. The paper's core empirical finding, that strong presentational quality coexists with weak epistemological quality across seven diverse LLMs (Figure 2, Tables 3–4), is not a indictment of those models; it is a demonstration that the criteria used to train and evaluate them prioritize presentation over epistemology.

This has specific consequences for the research landscape:

It makes the RLHF optimization target a visible problem rather than an invisible assumption. The paper shows that ChatGPT—the model family most aggressively optimized for human preference—achieves the highest presentational scores of any model (4.54–4.58 on style, clarity, and correctness; Table 3) while ranking between fourth and last on epistemological scores (2.05–3.48; Table 4). Meanwhile, the unassisted human raters (the "GPT-4, no assistance" condition in Table 4) give epistemological ratings inflated by 0.82–1.71 Likert points compared to assisted raters, meaning that the human judgments used to train reward models substantially overestimate epistemological quality. This creates a structural problem for RLHF as an alignment strategy: if human raters cannot reliably distinguish presentation from epistemology, then optimizing for their preferences will produce models that are eloquent and trusted but epistemologically weak—exactly the pattern the paper documents. This finding does not show that RLHF is harmful to epistemology (the temporal trend from davinci-002 to GPT-4 in Table 4 shows modest epistemological improvement alongside dramatic presentational improvement), but it does show that RLHF is insufficient for epistemological quality and that augmenting human feedback with grounded assistance (as the paper's protocol does) would be necessary for reward models to capture what human raters miss.

It resolves a latent tension between "LLMs produce fluent, helpful text" and "LLMs cannot be trusted for science information." The paper provides a mechanism for this tension: the skills that make LLMs fluent (smooth syntax, appropriate sentence length, varied vocabulary, coherent structure) are largely orthogonal to the skills that make them epistemologically adequate (understanding when uncertainty is important, recognizing which details are essential versus optional, avoiding the "argument from authority" style that the paper documents in Section 4.4). The two skill sets do not naturally co-vary. This explains why prior NLP evaluations focused on fluency and factuality have shown steady progress while domain experts remain skeptical of LLM-generated science communication—they are evaluating different things. The contribution is not to resolve the tension but to make it measurable. The eight-dimension framework is a diagnostic instrument that can track whether future models are closing the presentation-epistemology gap or only widening their lead on presentation.

It changes the status of attribution-based evaluation (AIS) from a sufficiency claim to a complementarity claim. The finding that AIS scores are orthogonal to epistemological ratings (Spearman's correlations of 0.03, −0.06, 0.002, −0.02; all p > 0.30; Section 4.6, Figure 5) is not a rejection of attribution—the paper calls it "practical and complementary"—but it is a falsification of the sufficiency hypothesis. The Retrieval Augmented Language Model literature (Lewis et al., 2020; Guu et al., 2020) and the attributed QA literature (Bohnet et al., 2023; Rashkin et al., 2022) have operated on an implicit premise that grounding generation in retrieved documents addresses the core factuality problem. This paper demonstrates that grounding does not address completeness (fully attributable answers have lower minimum completeness ratings; Figure 5, bottom-left panel), specificity, or uncertainty communication. This finding redirects the RALM research agenda: attribution should be viewed as one necessary component of quality alongside separate mechanisms for ensuring completeness, appropriate specificity, and uncertainty calibration. A RALM system that only optimizes attribution may actually reduce completeness by restricting answers to what was retrieved.

It provides a template for domain-informed AI evaluation beyond climate. Every domain with a body of communication scholarship—health (how to communicate risk, benefit, and uncertainty to patients), law (how to present legal information to non-lawyers), finance (how to explain financial products and risks to consumers), education (how to scaffold explanations for learners at different levels)—has established findings about what makes communication effective in that domain. The paper's methodology—deriving evaluative dimensions from that scholarship, operationalizing them as Likert-scale statements with structured issue menus, and validating with grounded AI Assistance—is portable. The specific dimensions will differ per domain (patient communication may prioritize empathy and risk calibration; legal communication may prioritize completeness and caveat clarity), but the process of constructing the framework from domain scholarship rather than from NLP evaluation conventions is the methodological contribution that transfers.

Follow-Up Research This Work Enables

Cross-domain replication of the presentational-epistemological gap. The paper's central empirical finding—the dissociation between surface quality and epistemological quality—is documented only for climate change. A natural and high-impact follow-up would replicate the framework in another science communication domain with comparable stakes: vaccine information, public health guidance (e.g., nutrition, pandemic response), or environmental policy. A study using the same methodology (eight dimensions adapted from domain scholarship, educated raters, grounded AI Assistance, three-rater design, 300 questions drawn from real search queries and domain-specific sources) on a new domain would answer two questions: (1) Is the presentational-epistemological gap a general property of LLM science communication, or is it specific to climate (perhaps because climate communication involves politically charged topics that amplify certain failure modes)? (2) Which dimensions transfer cleanly across domains, and which require substantial re-operationalization? A finding that the gap replicates across domains would establish it as a fundamental limitation of current LLM training paradigms. A finding that it does not replicate in a less politicized domain would implicate climate-specific factors (the prevalence of contested claims, the complexity of climate science) and narrow the scope of the framework's claims.

Training reward models with assisted human judgments. The paper's finding that unassisted raters inflate epistemological ratings by 0.82–1.71 Likert points (Tables 3–4) has a direct practical implication: human preference data used to train RLHF reward models is systematically biased toward overestimating epistemological quality. A strong follow-up would test whether training a reward model on assisted human judgments (using the paper's AI Assistance protocol to generate critiques before raters assign preference scores) produces a different ranking of model outputs than a reward model trained on unassisted judgments. The experiment would involve: (1) collecting preference judgments on the same set of LLM-generated climate answers from two rater pools, one with and one without AI Assistance; (2) training separate reward models on each pool; (3) comparing the reward models' rankings of held-out answers against the assisted human ratings as a reference. If the assisted reward model better correlates with the paper's epistemological dimensions (particularly accuracy, completeness, and uncertainty), this would provide evidence that assisted preference data can partially close the presentation-epistemology gap in RLHF training. The paper's helpfulness analysis (Figure 4, showing raters push back against unhelpful assistance) suggests that assisted judgments are not simply a proxy for AI judgment—raters still exercise independent judgment—which makes this experiment methodologically viable.

Difficulty-conditioned analysis of epistemological failures. The paper does not analyze whether epistemological failures are concentrated on hard questions (those requiring technical detail, nuanced uncertainty communication, or synthesis across multiple scientific findings) or distributed uniformly. A follow-up study would bin the 300 questions by difficulty, operationalized not by dataset labels (the questions come from three heterogeneous sources) but by the inter-rater agreement on epistemological dimensions for the best-performing model (Falcon-180B-Chat). Questions where all three raters agree on low epistemological scores would be "hard"; questions where raters agree on high scores would be "easy"; and questions with disagreement would be "ambiguous." The analysis would test: (1) whether the presentational-epistemological gap is larger for hard questions (suggesting that models fail specifically when precise knowledge is required) or uniform (suggesting a general failure to prioritize epistemology); (2) whether AI Assistance provides larger detection benefits for hard questions (where raters may lack the specific domain knowledge to independently identify issues) or for ambiguous questions (where the assistance may help arbitrate disagreement); (3) whether different models have different difficulty profiles—e.g., Falcon-180B-Chat may outperform GPT-4 on hard questions while GPT-4 excels on easy ones, which would have implications for model routing in deployment. The paper already has the necessary data—300 questions × 3 raters × 7 models × 8 dimensions—to perform this analysis without collecting new ratings, making it an immediate follow-up.

Stress-testing the AI Assistance protocol against model-specific bias. The paper explicitly acknowledges that AI Assistance is generated by GPT-4 and that "the assistance may also influence the raters beyond enhancing discovery" (Section 5), and the LLM rater experiment (Table 22) shows that GPT-4 rates OpenAI models more favorably than human raters do relative to non-OpenAI models. A critical stress-test would generate AI Assistance using a non-OpenAI model—ideally Falcon-180B-Chat, which leads on epistemological dimensions and is open-source—and compare the resulting human ratings against the GPT-4-generated assistance condition. The experiment would require re-running the AI Assistance pipeline (keypoint extraction, URL generation, passage scoring, critique generation) with Falcon (or another non-OpenAI model of sufficient capability) and having a new set of raters evaluate the same answers using Falcon-generated assistance. The key comparison: do the model rankings shift? If GPT-4-generated assistance systematically penalizes non-OpenAI models (by being more likely to generate critiques for their answers), Falcon-generated assistance might produce higher epistemological ratings for PaLM-2 and Falcon-180B-Chat and lower ratings for GPT-4 and ChatGPT. The direction and magnitude of any shift would quantify the upper bound on model-specific bias in the current protocol. A null result (no ranking shifts) would strengthen confidence that the framework produces model-agnostic evaluations despite using GPT-4 for assistance.

Measuring the persistence and transfer of the AI Assistance training effect. The "previous exposure" condition (Figure 3, Tables 3–4) shows that raters who have worked with assistance on prior studies but receive none on the current task detect more issues than assistance-naive raters—a training effect. This finding is presented as a single snapshot, but it raises questions with practical implications for evaluation pipeline design: How many assisted examples are needed to achieve the training effect? Does it persist across sessions separated by days or weeks? Does it transfer to new dimensions (e.g., does learning to detect specificity failures with assistance make raters better at detecting completeness failures without assistance) and to new domains (e.g., does climate-focused assistance training improve vaccine information ratings)? A longitudinal study tracking individual raters across multiple sessions, with controlled variation in the number and domain of assisted examples between sessions, would characterize the learning curve and its decay. If the training effect is durable and transfers across domains, then AI Assistance could be used for rater training (a fixed upfront investment) rather than continuous augmentation (an ongoing per-evaluation cost), substantially reducing the pipeline's computational cost in deployment. If the effect decays rapidly or does not transfer, then continuous assistance remains necessary, and the cost of the pipeline becomes a more binding constraint on the framework's adoption.

Human vs. LLM raters under matched assistance: quantifying the substitution frontier. The LLM rater experiment (Appendix A.15, Table 22) is preliminary—a single model (GPT-4), a single temperature setting (0.6, three samples), and no systematic comparison of how LLM raters' performance varies with assistance quality, rater expertise, or question difficulty. A systematic follow-up would compare human raters at three expertise levels (lay, climate-educated undergraduate, climate science graduate researcher) against LLM raters at three capability levels (GPT-4, a mid-tier model like PaLM-2, and a smaller model) under three assistance conditions (no assistance, ungrounded assistance without Wikipedia evidence, grounded assistance with Wikipedia evidence), all on the same 300-question set. The analysis would map a substitution frontier: at what (rater expertise, assistance quality) combination do LLM raters achieve agreement with expert human raters (the highest-expertise, fully-assisted condition) comparable to the agreement among the expert human raters themselves? If LLM raters with grounded assistance can match educated human raters with assistance on detecting epistemological issues, then the framework becomes substantially more scalable—LLMs could pre-screen large volumes of outputs, with human raters reserved for ambiguous or high-stakes cases. If LLM raters systematically miss certain issue types or exhibit model-family biases (as the Table 22 egocentric bias pattern suggests), then the substitution frontier is further away, and human raters remain essential.

Practical Applications and Downstream Use Cases

Pre-deployment auditing of LLM-based climate information systems. An organization deploying an LLM-powered chatbot, search interface, or content generation system for climate information—whether a weather service, an environmental NGO, a government climate portal, or a news organization—needs to know whether the system's outputs are epistemologically adequate before they reach users. The paper's framework provides a structured, validated audit protocol: generate answers to a representative set of user questions (the 300-question set can serve as a template, or the organization can construct a domain-specific set using the paper's three-source methodology), have climate-educated raters with grounded AI Assistance evaluate all eight dimensions, and use the dimension-level and issue-level results (as in Tables 3, 4, and 7) to identify specific failure modes. The finding that unassisted raters miss 60% of planted issues (Table 20, without assistance, majority vote) means that an audit conducted without grounded assistance—which would be the default for an organization unfamiliar with this framework—would substantially overestimate epistemological quality, potentially leading to the deployment of a system that presents fluent but misleading climate information. The audit results could also inform prompt engineering: if the system scores poorly on uncertainty but well on accuracy, the deployment prompt could be augmented with explicit uncertainty-communication instructions (as in the dimension-aware prompt in Table 1, which improved uncertainty by +0.19), and the audit re-run to verify improvement.

Calibrating user trust in LLM-generated climate information. The paper cites Chiesurin et al. (2023): "Eloquence and advanced dialogue behaviors are trusted by users, even in the absence of trustworthy information." This is a known risk for any domain where LLMs are deployed as information sources, but it is particularly acute for climate change, where misinformation is prevalent and the psychological mechanisms of trust are well-studied (van der Linden et al., 2015; Maibach et al., 2023). The framework's dimensions provide a language for communicating to users what the system is and is not good at. A climate chatbot could disclose: "This system is highly fluent and clear (clarity: 4.3/5, style: 4.4/5), but it sometimes misses important details (completeness: 2.6/5) and may not adequately communicate scientific uncertainty (uncertainty: 2.2/5). For the most complete and calibrated information, we recommend consulting [specific expert sources]." This kind of calibrated disclosure is not standard practice in LLM deployment, but the paper's framework makes it technically feasible—the dimension-level scores provide the quantitative basis for specific, honest statements about system limitations rather than generic disclaimers. The paper's finding that the presentational-epistemological gap is consistent across models (Figure 2) means that simply switching to a different model does not eliminate the gap; user-facing transparency about which dimensions are weak is more honest than claims of overall improvement.

Informing the design of retrieval-augmented climate QA systems. The paper's AIS comparison (Section 4.6, Figure 5) shows that attribution to sources does not guarantee completeness, specificity, or uncertainty communication—fully attributable answers can still be incomplete and vague. For teams building RALM systems for climate QA, this finding has a direct design implication: the retrieval and generation pipeline should be augmented with explicit mechanisms for each epistemological dimension, not just attribution. A retrieval system that only returns passages relevant to the query may produce attributable but incomplete answers; a system that also retrieves passages about what is missing (e.g., "the following climate impacts are also relevant to this region but were not covered in the retrieved passages") could improve completeness. A generation system that only conditions on retrieved passages may produce specific-but-partial answers; a system that also conditions on explicit uncertainty metadata (e.g., confidence intervals, scientific consensus levels from the IPCC reports) could improve uncertainty communication. The paper's finding that 87.7% of questions can be mapped to a valid Wikipedia URL by GPT-4 (Section 3.2) provides a baseline for retrieval coverage; the finding that only 6.95% of answers are fully attributable (Table 17) despite 87.7% URL coverage suggests that the gap between retrieval and attribution is substantial—improving the passage selection and generation stages could close this gap, but the paper's framework provides the multi-dimensional metrics to verify whether closing the attribution gap also closes the epistemological gaps.

Rater training and calibration for high-stakes LLM evaluation. The paper's AI Assistance protocol functions as both an evaluation aid and a training tool, as evidenced by the "previous exposure" condition (Figure 3, Tables 3–4) showing that raters who have seen assistance become more critical readers of unassisted answers. Organizations that need to evaluate LLM outputs in any high-stakes science communication domain—not just climate—can use this protocol for rater training: expose new raters to a set of assisted evaluations on a diverse question set, have them rate a separate set without assistance, and compare their performance against assisted benchmarks. The paper's tutorial and admission test structure (Appendix A.9), combined with the validation set methodology (Appendix A.14, constructing synthetic examples with known planted issues), provides a template for building domain-specific rater qualification pipelines. The finding that majority detection of planted issues improves from 60% to 76.67% with assistance (Table 20) quantifies the value of the training—a 27.8% relative improvement in detection—and provides a benchmark that other domains can use to calibrate their own training protocols. The finding that raters do not blindly follow assistance (Figure 4, higher ratings when assistance is perceived as unhelpful) suggests that trained raters maintain critical independence, which is essential for the protocol to function as genuine oversight rather than as a rubber stamp for AI judgment.