Fine-tuning Language Models for Factuality

🎯 Pitch

Language models can be trained to halve their factual error rate on open-ended generation—without a single human fact-checker. By simply ranking their own claims by confidence and fine-tuning on those preferences, models learn to self-censor false statements.

1. Executive Summary

This paper proposes a method for fine-tuning language models to improve their factuality in open-ended generation without requiring human labels. Using Llama-1-7b and Llama-2-7b on biography generation and medical question-answering benchmarks, the authors construct factuality preference rankings through two mechanisms — reference-based truthfulness estimation (checking atomic claims against Wikipedia via FactScore) and a novel reference-free confidence-based truthfulness estimation (converting claims into questions and measuring the model's own answer confidence) — and then optimize these preferences with Direct Preference Optimization (DPO). The resulting models, FactTune-FS and FactTune-MC, reduce the factual error rate by 58% on biographies and 40% on medical questions relative to Llama-2-Chat at the 7B scale, with FactTune-FS simultaneously increasing the number of correct facts generated while decreasing incorrect ones. The reference-free approach eliminates the need for external knowledge bases or human annotation during training, establishing that language models can learn to leverage their own internal uncertainty signals to avoid making untrue statements in long-form text, though the method operates only on topics where the base model already possesses non-trivial knowledge.

2. Context and Motivation

The Core Problem: Language Models Lie Convincingly, and We Don't Know How to Stop Them

The fundamental problem this paper addresses is deceptively simple: large language models generate confident-sounding but factually incorrect statements, and we lack scalable, human-free methods to train them to stop doing this. The authors frame this not as a minor nuisance but as a critical barrier to deploying LLMs in high-stakes settings. When a model generates a biography or medical advice, factual errors can "inadvertently spread misinformation or harmfully perpetuate misconceptions" (Section 1). The paper's opening example crystallizes the issue: GPT-3.5 (ChatGPT) "produces false citations more often than not when asked to provide the authors of a given study" (Agrawal et al., 2023). This is not a rare edge case — it is the modal behavior on certain knowledge-intensive tasks.

The problem manifests most acutely in long-form, open-ended generation. In short-form question answering, the model produces a single answer that can be checked against a gold label. But when asked to "Write a biography of Yo-Yo Ma," the model generates a paragraph containing many discrete factual claims — birth dates, education, career milestones, family details. Each claim is a potential failure point. The paper's core question (Section 1) is:

"Can language models be fine-tuned to leverage this internal awareness, to avoid making untrue statements in the first place?"

This question is important for several practical reasons the authors make explicit:

Misinformation at scale. LLMs are increasingly deployed as information retrieval tools, sometimes "even as a replacement for traditional search engines" (Section 1). A model that confidently fabricates biographical details or medical symptoms can cause real harm — consider a patient receiving incorrect information about stroke symptoms or a student learning false historical facts.
Human fact-checking is prohibitively expensive. The authors cite Min et al. (2023), who report that "professional fact-checkers took approximately 9 minutes to fact-check a single model-generated biography of a well-known individual; it cost about $2,000 to annotate 505 biographies" (Section 1). Extrapolating: fact-checking even a modest corpus of 10,000 model outputs at this rate would cost approximately$ 40,000 and require roughly 1,500 hours of expert labor. This makes human-supervised training for factuality economically infeasible at scale.
The pretraining objective actively works against factuality. Section 1 provides a detailed analysis of why maximum likelihood training — the standard objective for pretraining and supervised fine-tuning — fails to incentivize factual outputs. Consider the question "Where was Yo-Yo Ma born?" A model that near-deterministically outputs "idk, probably Paris?" (mostly correct, expressing appropriate uncertainty) receives extremely high loss if the training data contains any other response to that question. Meanwhile, a model that "hedges probability mass over many possible phrasings and many possible locations (including incorrect ones, like Antarctica)" achieves lower loss because it assigns non-trivial probability to every response seen in training. The authors call this the probability smearing problem: the objective rewards spreading probability mass broadly rather than concentrating it on the correct answer, particularly when the model encounters questions requiring knowledge at the boundary of its training distribution.

This last point is subtle but crucial. It means that even with perfect training data, the standard training paradigm does not necessarily produce factual models. The problem is not just about data quality — it is about objective misalignment. The pretraining loss cares about token-level likelihood, not truth. A model that outputs "Yo-Yo Ma was born in Paris" 10% of the time and "Yo-Yo Ma was born in Antarctica" 10% of the time may have lower loss on a diverse corpus than a model that outputs the correct birthplace 95% of the time and refuses to answer 5% of the time. The loss function has no mechanism to distinguish between these behaviors.

The Gap: We Can Detect Hallucinations, But We Can't Train Models Not to Produce Them

The paper situates itself at a specific inflection point in the literature. Prior work has developed increasingly sophisticated methods for detecting factual errors: sensitivity to prompt perturbations (Xu et al., 2023), high output diversity under resampling (Kadavath et al., 2022; Kuhn et al., 2023), inconsistency with external knowledge sources (Min et al., 2023; Chern et al., 2023), and properties of internal model activations (Azaria & Mitchell, 2023). Other work goes further and attempts to correct errors after generation, typically by retrieving relevant documents and using another LLM to verify consistency (Peng et al., 2023; Gao et al., 2023; Dhuliawala et al., 2023).

But between detection and post-hoc correction lies a gap: what if we could train the model not to hallucinate in the first place? The paper identifies a crucial asymmetry in the literature. Detection methods operate at inference time — they flag errors after they occur. Correction methods add system complexity by requiring retrieval pipelines, external verifiers, or multi-turn revision. Both approaches treat the symptom rather than the cause. The underlying language model remains unchanged, continuing to produce errors that downstream systems must catch.

Why Existing Training Approaches Fall Short

The paper identifies several limitations in prior approaches to training for factuality:

Reinforcement learning from human feedback (RLHF) helps but is insufficient. Models like Llama-2-Chat are trained with RLHF to be helpful, harmless, and honest (Touvron et al., 2023b). Yet the paper's own experiments (Table 2) show that Llama-2-Chat still generates 6.41 incorrect facts per biography on average — an error rate of approximately 25%. RLHF reduces some undesirable behaviors, but the human preference labels used in RLHF are typically collected for general helpfulness and safety, not specifically for fine-grained factual accuracy. Human annotators may not catch subtle factual errors, particularly for obscure claims. Moreover, the cost of collecting preference labels specifically targeting factuality would be astronomical at scale, as the $2,000/500 biographies figure demonstrates.

Decoding-time interventions show mixed results. Methods like Inference-Time Intervention (ITI; Li et al., 2023) and Decoding by Contrasting Layers (DOLA; Chuang et al., 2023) modify model behavior at inference time by shifting internal activations or altering the decoding procedure. The paper evaluates both as baselines (Table 2) and finds that while they can improve factuality, they do so inconsistently. For Llama-2 on biographies, ITI achieves 5.75 incorrect facts (vs. 6.41 for Chat), while DOLA achieves 5.84. Critically, the paper's own evaluation (Figure 3) shows that both ITI and DOLA lie outside the "strict improvement" region — they either increase correct facts at the cost of more errors, or reduce errors at the cost of fewer correct facts. They do not simultaneously improve both dimensions, which is what a genuinely more factual model should do.

Retrieval-augmented approaches add complexity and face fundamental limitations. Methods that ground generation in retrieved documents (Lewis et al., 2020) require maintaining a knowledge base, implementing a retrieval system, and resolving conflicts between parametric knowledge (what the model knows) and retrieved knowledge (what the documents say). Longpre et al. (2022) and Chen et al. (2022) document the difficulty of reliable conflict resolution. Furthermore, Mallen et al. (2023) show that the benefits of retrieval diminish as model size increases — larger models are more capable of generating factual content parametrically, but also more likely to confidently override retrieved information with incorrect parametric knowledge. The paper notes (Section 5) that "the most common open-source consumer language models thus use purely parametric models" (e.g., LLaMA family; Touvron et al., 2023a), making retrieval-dependence a practical limitation.

Prompting strategies are fragile and task-specific. Si et al. (2023) show that careful prompting can improve factuality, but the gains are typically small and inconsistent across domains. Prompting shifts behavior at the surface level without changing the underlying model's knowledge or its propensity to generate unsupported claims.

The self-supervision opportunity remains largely unexploited. The paper identifies a critical piece of evidence that motivates their approach: "large language models do exhibit systematic markers of uncertainty that indicate their factually unreliable statements" (Kadavath et al., 2022; Tian et al., 2023). In other words, the model knows (in some internal sense) when it is likely to be wrong — its confidence scores correlate with correctness. Yet no prior work had used this signal as a training objective for improving factuality. The detection literature uses uncertainty to flag errors; the paper's insight is to use uncertainty as a reward signal for preference learning. This transforms an inference-time diagnostic into a training-time optimization target.

How This Paper Positions Itself

The paper frames its contribution through a specific lens: automated factuality preference construction without human labels, using either external knowledge (reference-based) or the model's own internal uncertainty (reference-free) as the truthfulness signal, combined with DPO for efficient preference learning. This positioning addresses multiple limitations of prior work simultaneously:

No human labels required. By using FactScore's automated claim verification (reference-based) or the model's own confidence scores (reference-free), the preference dataset construction is fully automatic. This eliminates the $2,000/500 biographies cost cited for human annotation.
Prevention rather than detection. Unlike inference-time methods that flag or correct errors after generation, DPO trains the model's weights to produce factual outputs by default. The model learns to avoid making unsupported claims rather than relying on an external system to catch them.
No inference-time complexity. The reference-free approach (FactTune-MC) requires no retrieval system, no external knowledge base, and no additional verifier at inference time — only the fine-tuned model itself. Even the reference-based approach (FactTune-FS) uses Wikipedia retrieval only during training data construction, not during deployment.
Compatible with RLHF. The paper shows (Table 3) that factuality tuning can be applied on top of an already-RLHF-trained chat model (Llama-2-Chat), further improving its factuality. This positions the method as an additional fine-tuning stage rather than a replacement for existing alignment pipelines.
Targeting long-form generation. While most prior work on factuality focused on short-form QA (Kadavath et al., 2022), the paper specifically designs its truthfulness estimation procedures to handle open-ended paragraphs containing multiple atomic claims. This is a harder setting where defining and measuring factuality is less straightforward.

The paper also draws an explicit connection to the calibration literature. The reference-free approach is inspired by Kuhn et al. (2023)'s semantic uncertainty work, but repurposes it from an evaluation metric to a training objective. The key adaptation is the two-stage pipeline: (1) extract atomic claims from long-form text and convert them to unambiguous questions, then (2) measure the model's confidence in answering those questions correctly. This avoids the problem that raw sequence probability conflates factual content with stylistic choices — a confident but factually incorrect statement like "Yo-Yo Ma was born in Antarctica, a continent known for its harsh climate and diverse penguin populations" would receive high probability because the phrasing is fluent, even though the core fact is wrong.

The paper's overarching position is that the gap between detecting hallucinations and preventing them can be closed by converting uncertainty signals — whether from external knowledge or internal model confidence — into preference data that DPO can learn from. This is a conceptual bridge between two previously separate research threads: the hallucination detection literature and the preference-based fine-tuning literature. The paper's technical contribution is not a new detection method or a new RL algorithm, but rather a specific pipeline for connecting these two threads in a way that demonstrably reduces factual errors in open-ended generation.

3. Technical Approach

3.1 Reader Orientation

This paper builds a fine-tuning pipeline that makes a language model generate fewer false claims in long-form text without ever needing a human to label what is true or false. The core problem is that standard training objectives (maximum likelihood) reward the model for spreading probability mass over many possible outputs regardless of factual accuracy, and human fact-checking is too expensive ($2,000 per 500 biographies) to use as a direct training signal. The solution is to automatically construct preference pairs where one model-generated response is preferred over another because it contains fewer factual errors, then train the model to prefer the more factual response using Direct Preference Optimization (DPO). The key insight is that factuality preferences can be derived either from external knowledge (checking claims against Wikipedia) or from the model's own internal uncertainty (its confidence in its answers), eliminating the human bottleneck entirely.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components connected in a fixed pipeline:

Untuned Base Language Model — a pre-trained or instruction-tuned LLM (Llama-1-7b, Llama-2-7b, or Llama-2-7b-Chat) that generates candidate responses to open-ended prompts but produces factual errors at an unacceptably high rate.
Truthfulness Estimator — a scoring function that takes a long-form text response and returns a single scalar truthfulness score (fraction of atomic claims that are correct). This component has two variants: (a) FactScore (reference-based) which checks each atomic claim against a Wikipedia article using a fine-tuned NLI model, and (b) Model Confidence (reference-free) which converts each claim to a question, resamples answers from the base model, and measures the model's confidence in its most likely answer.
Preference Dataset Constructor — takes $n$ candidate responses per prompt, scores each with the truthfulness estimator, and creates $\binom{n}{2}$ pairwise preferences where the response with the higher truthfulness score is marked as preferred ( $y_w$ ) and the lower-scoring response as dispreferred ( $y_l$ ). Pairs with equal scores are discarded.
DPO Fine-Tuning Module — takes the constructed preference dataset and optimizes the language model's weights using the Direct Preference Optimization algorithm, which increases the relative log-probability of preferred responses while keeping the model close to a reference initialization via a KL-divergence penalty.

Information flows as follows: unlabeled prompts enter the system → the base model generates $n$ candidate responses per prompt via temperature sampling → the truthfulness estimator scores each response → the constructor creates preference pairs → DPO updates the model weights to favor more factual generations → the resulting model (FactTune-FS or FactTune-MC) generates responses with fewer factual errors at inference time, with no additional retrieval, verification, or human involvement required.

3.3 Roadmap for the Deep Dive

First, the formal RL objective (Equation 1) and the DPO algorithm (Equation 3), since all factuality tuning builds on this preference-learning framework. Understanding why DPO is chosen over PPO and how the KL penalty works is foundational.
Second, the reference-based truthfulness estimator (FactScore), since it is the established method that the paper adopts as one preference source and uses as the primary evaluation metric. This covers claim extraction, evidence retrieval, and NLI-based verification.
Third, the novel reference-free truthfulness estimator (model confidence), since it is the paper's key methodological innovation. This covers claim-to-question conversion, answer resampling, semantic binning, and confidence calculation.
Fourth, the preference dataset construction procedure, which is shared across both truthfulness estimators and determines how the DPO training data is assembled from scored responses.
Fifth, the design choices and their justifications — why atomic claim extraction rather than raw sequence probability, why heuristic string matching rather than GPT-3.5 equivalence checking, why maximum confidence over entropy, and why the specific prompt formats and sampling parameters.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodology paper whose core idea is that automated truthfulness scoring — whether reference-based or reference-free — can serve as a reward signal for preference-based fine-tuning that demonstrably reduces factual errors in long-form generation, and that the resulting improvements are not the result of reward overoptimization but reflect genuine gains in factual accuracy as validated by both human evaluators and independent LLM judges.

The Reinforcement Learning Objective for Language Models

The paper situates its approach within the standard framework of KL-regularized reinforcement learning for language model fine-tuning. The starting point is a pre-trained language model treated as a policy $\pi_\theta$ , which for any input prompt $x$ (a text sequence) produces a conditional distribution $\pi_\theta(y \mid x)$ over possible output responses $y$ (also a text sequence). The goal is to find parameters $\theta$ that maximize the expected reward of generated outputs while not deviating too far from a reference model $\pi_{\text{ref}}$ that represents the initialization:

$\max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D}_p, y \sim \pi_\theta(y \mid x)} \left[ r(x, y) - \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)} \right]$

where $\mathcal{D}_p$ is a dataset of prompts (e.g., "Write a biography of Yo-Yo Ma"), $r(x, y)$ is a scalar reward function that assigns higher values to more desirable outputs, $\pi_{\text{ref}}$ is the reference policy (typically the result of supervised fine-tuning on demonstration data), and $\beta$ is a hyperparameter controlling the trade-off between reward maximization and divergence from the reference model.

What it computes: The objective takes an expectation over two sources of randomness — which prompt is sampled from the prompt dataset $\mathcal{D}_p$ , and which response is sampled from the model's own policy $\pi_\theta(\cdot \mid x)$ — and for each such $(x, y)$ pair, computes a composite score. This score is the reward $r(x, y)$ (higher for factual, helpful responses) minus a penalty term $\beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}$ . The penalty term is zero when the current policy and reference policy assign identical probability to the response, positive when the current policy assigns higher probability than the reference (meaning it has moved away from the initialization), and negative when it assigns lower probability. The $\beta$ coefficient scales this penalty: when $\beta$ is large, the model stays close to $\pi_{\text{ref}}$ and changes little; when $\beta$ is small, the model can move far from the initialization to chase reward.

Why this form: Without the KL penalty (i.e., $\beta = 0$ ), the objective reduces to unconstrained reward maximization, which Gao et al. (2022) showed leads to overoptimization — the policy exploits idiosyncrasies of the reward function that correlate with high scores but do not correspond to the intended behavior. For example, a factuality reward based on FactScore might be gamed by generating text that contains few claims (making errors less likely) or that uses vague language that the NLI verifier classifies as "supported" by default. The KL penalty keeps the model near its initialization, which has been pre-trained on diverse data and therefore produces broadly reasonable text, preventing it from collapsing to degenerate strategies. This is the same objective used in RLHF (Ouyang et al., 2022; Bai et al., 2022), with the crucial difference that the reward function $r(x, y)$ in this work is derived from automated factuality estimation rather than human preference labels.

The standard algorithm for optimizing this objective is Proximal Policy Optimization (PPO; Schulman et al., 2017), which requires sampling from the policy during training, fitting a reward model, and performing multiple gradient updates per batch of sampled data. The authors note that PPO is "quite complex to implement and tune" (Section 2), motivating their choice of a simpler alternative.

Direct Preference Optimization (DPO)

The DPO algorithm (Rafailov et al., 2023) simplifies RL fine-tuning for the special case where the reward function is learned from a dataset of pairwise preferences over model outputs. Rather than fitting an explicit reward model and then optimizing it with PPO, DPO directly optimizes the policy from the preference data using a classification-style loss.

The foundation is the Bradley-Terry model of pairwise preferences (Bradley & Terry, 1952). Given a prompt $x$ and two candidate responses $y_w$ (preferred, or "winner") and $y_l$ (dispreferred, or "loser"), the probability that $y_w$ is preferred over $y_l$ is assumed to follow:

$p(y_w \succ y_l) = \sigma(r(x, y_w) - r(x, y_l))$

where $y_w \succ y_l$ denotes that $y_w$ is preferred to $y_l$ , $\sigma(\cdot) = \frac{1}{1 + e^{-(\cdot)}}$ is the sigmoid function mapping real values to $(0, 1)$ , and $r(x, y)$ is an unobserved scalar reward function.

What it computes: The right-hand side is the sigmoid of the reward difference between the two responses. When $r(x, y_w) \gg r(x, y_l)$ , the difference is large and positive, so $\sigma(\cdot) \approx 1$ , meaning the model is nearly certain that $y_w$ is preferred. When the rewards are equal, the difference is zero, so $\sigma(0) = 0.5$ , meaning the preference is a coin flip. When $y_l$ has higher reward, the sigmoid output is less than 0.5. The left-hand side $p(y_w \succ y_l)$ is the observed preference probability — in practice, this is always 1.0 in the constructed dataset because the higher-scoring response is deterministically labeled as preferred.

Why this form: The Bradley-Terry model is the standard probabilistic model for pairwise comparison data. It assumes that each item (response) has a latent quality score (reward), and the probability of preferring one item over another depends only on the difference in their quality scores. The sigmoid function ensures the output is a valid probability between 0 and 1. This choice connects preference learning to binary classification: predicting which of two responses is preferred given their latent rewards is equivalent to logistic regression on the reward difference. Alternative models like the Thurstone model (which uses a normal CDF rather than sigmoid) or the Plackett-Luce model (for rankings of more than two items) exist but would complicate the mathematical connection to the policy optimization objective.

Rafailov et al. (2023) proved a key theoretical result: the optimal policy $\pi^*$ for the KL-regularized RL objective in Equation 1, when the reward function $r(x, y)$ is derived from the Bradley-Terry preference model, can be found by directly optimizing the following loss on the preference data, without ever explicitly representing or fitting the reward function:

$\mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right]$

where $\mathcal{D} = \{x^{(i)}, y_w^{(i)}, y_l^{(i)}\}_{i=1}^N$ is a dataset of $N$ preference triples (prompt, preferred response, dispreferred response), $\pi_\theta$ is the policy being optimized, $\pi_{\text{ref}}$ is the reference policy (identical to the reference in Equation 1), and $\beta$ is the same KL penalty coefficient as in Equation 1.

What it computes: For each preference triple in the dataset, the loss does the following. First, it computes the log-ratio $\log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)}$ for the preferred response — this is the difference in log-probability between the current policy and the reference policy for generating $y_w$ . A positive value means the current policy assigns higher probability to $y_w$ than the reference did; a negative value means it assigns lower probability. The same computation is done for the dispreferred response $y_l$ . These log-ratios are multiplied by $\beta$ (the same KL coefficient) and subtracted: $\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}$ . This difference represents how much more the current policy has shifted toward the preferred response relative to how much it has shifted toward the dispreferred response, compared to the reference. The sigmoid converts this difference to a probability. The log of this probability is taken and negated, so minimizing this loss maximizes the probability that the model's implicit reward ranks $y_w$ above $y_l$ .

Why this form: The DPO loss is mathematically equivalent to the optimal solution of the PPO objective under the Bradley-Terry preference model, meaning that optimizing this simple classification loss yields the same policy as running the full PPO algorithm with a learned reward model — but without the complexity of reward model fitting, online sampling, and PPO's clipping and value function estimation. The loss is simply a binary cross-entropy where the logit is the scaled difference in log-ratios between the preferred and dispreferred responses. This makes DPO dramatically simpler to implement: it requires only a fixed dataset of preference pairs, standard supervised learning infrastructure (no online sampling loop), and tuning of a single hyperparameter $\beta$ . The paper exploits this simplicity to rapidly iterate on different sources of preference data (FactScore vs. model confidence) without re-implementing a full RL pipeline.

The practical consequence is that the entire factuality tuning procedure reduces to two steps: (1) constructing a preference dataset $\mathcal{D}$ where $y_w$ is more factual than $y_l$ according to some automated truthfulness estimator, and (2) running DPO with that dataset. The bulk of the paper's technical contribution is in Step 1 — how to build an automatic truthfulness estimator that produces preference pairs correlated with genuine factuality.

Reference-Based Truthfulness Estimation (FactScore)

The first method for constructing factuality preferences uses the FactScore pipeline (Min et al., 2023) to estimate the truthfulness of a long-form generation by checking its atomic claims against a trusted reference text — in this case, Wikipedia.

Claim extraction. Given a model-generated response (e.g., a biography of Yo-Yo Ma), the first step is to decompose it into a list of atomic claims — individual factual statements that can be independently verified. The paper uses GPT-3.5 (specifically the gpt-3.5-turbo model) to perform this decomposition. The output is a list of self-contained sentences, each expressing exactly one factual assertion. For example, from a biography of Yo-Yo Ma, the model might extract claims like "Yo-Yo Ma was born in Paris," "Yo-Yo Ma plays the cello," and "Yo-Yo Ma graduated from Harvard University." The extraction process is designed to isolate facts from stylistic or connective text — a sentence like "Yo-Yo Ma, a renowned cellist who was born in Paris, graduated from Harvard" would be split into two separate claims, each testing a single piece of knowledge.

Evidence retrieval. For each atomic claim, the system retrieves relevant evidence from Wikipedia. The paper leverages the FactScore implementation, which uses the specific Wikipedia article corresponding to the entity being discussed. For biographies, this is the Wikipedia page of the individual in question. For medical conditions, this is the Wikipedia page for that condition. The retrieval is therefore targeted: the system does not perform open-ended web search but rather retrieves from a known, curated reference document. This design choice means the method is only applicable to entities and topics that have corresponding Wikipedia pages — a limitation the paper acknowledges but accepts as the cost of using a high-quality, relatively consistent knowledge source.

Natural language inference for verification. With the atomic claim and the relevant Wikipedia text in hand, the system must determine whether the claim is supported by the evidence. FactScore uses a fine-tuned language model — specifically, a Llama-1-7B model that has been fine-tuned for the fact-checking task — to perform natural language inference (NLI; MacCartney & Manning, 2008). The model takes the Wikipedia text as premise and the atomic claim as hypothesis, and classifies whether the premise supports, refutes, or is neutral toward the hypothesis. A claim classified as "supported" is counted as correct; claims classified as "refuted" or "not supported" are counted as incorrect.

Truthfulness score. The final truthfulness score for a generated response is simply the fraction of extracted atomic claims that are classified as supported:

$\text{truthfulness} = \frac{\# \text{supported claims}}{\# \text{total claims extracted}}$

A response with 10 claims where 8 are supported by Wikipedia receives a score of 0.8; a response where only 3 of 10 claims are supported receives 0.3.

Why this form: This fraction-of-claims metric treats each atomic fact equally, which is a reasonable default when there is no a priori reason to weight some facts more heavily than others. It also produces a score in $[0, 1]$ that is directly interpretable as the proportion of factual claims. An alternative would be to use a weighted sum where more important or central claims count more, but this would introduce additional complexity and subjectivity in defining importance. The binary classification per claim is a simplification — real factuality is more nuanced (claims can be partially correct, misleading, or true but incomplete) — but the NLI framework provides a well-studied, reproducible approximation that has been shown to correlate well with human factuality judgments (Min et al., 2023).

Key limitation. The FactScore approach requires access to relevant, high-quality reference texts and a reliable NLI model. For topics without Wikipedia pages, or for claims where Wikipedia is outdated or incomplete, the truthfulness estimate can be inaccurate. Furthermore, the NLI model itself can make errors — it might classify a true claim as unsupported because the Wikipedia article phrases the fact differently, or it might classify a false claim as supported because of superficial lexical overlap. The paper acknowledges these limitations but notes that FactScore's correlation with human judgments (established in Min et al., 2023) makes it a reasonable proxy for truth. For the biography task, Wikipedia is a particularly good fit because biographical facts about well-known individuals tend to be well-documented and relatively stable.

Reference-Free Confidence-Based Truthfulness Estimation

The paper's second method for truthfulness estimation eliminates the need for external knowledge entirely by leveraging the observation that language model confidence is correlated with correctness (Kadavath et al., 2022). If a model is uncertain about a fact — generating different answers when resampled, or assigning low probability to its most likely answer — that fact is more likely to be incorrect. The challenge is to isolate the model's confidence on individual factual claims within a long-form passage, since the total probability the model assigns to a passage conflates factual content with stylistic and structural choices.

The reference-free pipeline has four stages:

Stage 1: Atomic claim extraction. As in the FactScore pipeline, the generated response is first decomposed into atomic claims using GPT-3.5. Each claim is a self-contained factual statement. This step is identical to the reference-based approach and serves the same purpose: isolating individual facts so that confidence can be measured on each one independently.

Stage 2: Claim-to-question conversion. Each atomic claim is converted into a specific, unambiguous question whose answer reveals whether the model knows the underlying fact. This step is crucial because directly measuring the model's probability of the claim text itself is misleading. A claim like "Yo-Yo Ma plays the cello" might receive high probability not because the model is confident in the fact, but because the phrasing is natural and fluent given the surrounding context. Conversely, a claim with low probability might be factually correct but expressed in an unusual way.

The paper uses GPT-3.5 with carefully designed few-shot prompts to perform this conversion, providing examples that emphasize specificity and unambiguity. For instance, the claim "Yo-Yo Ma plays the cello" should be converted to "What instrument does Yo-Yo Ma play?" rather than "What does Yo-Yo Ma play?" because the latter question admits answers of the wrong type (e.g., "basketball" or "chess"). The prompt (Table 7 in the appendix) provides entity-specific examples: for biographies, it shows conversions like "Hillary Clinton was born in 1947 → In what year was Hillary Clinton born?" and "She married Bill Clinton → Who did Hillary Clinton marry?"; for medical conditions, it shows conversions like "Menopause is a time in a woman's life → Menopause is a time in whose life?" and "Varicose veins occur when the veins under the skin become enlarged → Varicose veins occur when what happens to the veins under the skin?".

The prompt also includes a placeholder [NAME] or [CONDITION] that is filled with the specific entity or medical condition in the claim, along with a [STATEMENT] placeholder for the claim text. The model is shown three examples before the actual claim, establishing the pattern of converting declarative facts into targeted questions.

Why convert to questions rather than directly resampling the claim: If the model were simply asked to regenerate the claim text under resampling, the variation in outputs would reflect both factual uncertainty and paraphrasing diversity. Two responses might express the same fact in different words, which would be incorrectly interpreted as uncertainty. By converting to a question-answer format, the model is forced to produce a specific answer token or short phrase, making it easier to compare responses and determine whether the model consistently produces the same factual content.

Stage 3: Answer resampling and confidence estimation. For each generated question, the model resamples an answer 20 times from the base model — typically Llama-1-7B, the same model whose outputs are being evaluated. The resampling uses a few-shot prompt to encourage well-formed, concise answers. The 20 answers are then grouped into semantic equivalence bins.

The paper explores two methods for determining equivalence:

Heuristic string matching: Words in the answer are compared, excluding stop words. If the remaining content words are the same across two answers, they are binned together. For example, "the cello" and "cello" would match because the stop word "the" is excluded; "cello" and "violin" would not match because the content words differ. This method is computationally cheap and deterministic.
LLM-based equivalence checking: GPT-3.5 is prompted to assess whether two answers are semantically equivalent, inspired by Kuhn et al. (2023). This can capture cases like "cello" and "violoncello" that a string match would miss, but introduces additional cost, latency, and potential noise from the GPT-3.5 model's own errors.

The confidence score for the fact is the fraction of the 20 answers that fall into the largest equivalence bin:

$\text{confidence} = \frac{\text{count of answers in largest bin}}{20}$

If all 20 answers produce the same response (e.g., all say "cello"), the confidence is 1.0. If the answers are split across many different bins (e.g., 7 say "cello," 6 say "violin," 4 say "piano," 3 say "guitar"), the largest bin contains only 7, and the confidence is 0.35 — indicating high uncertainty. If there are ties for the largest bin, the paper uses the count of one of the tied bins (the procedure for tie-breaking is implicit; the fraction captures the model's concentration of probability mass on its single best guess).

The paper also evaluates two metrics for summarizing the distribution across bins:

Maximum confidence: the fraction in the largest bin, as described above.
Entropy over bins: the Shannon entropy of the normalized bin counts, $H = -\sum_i p_i \log p_i$ where $p_i$ is the fraction of answers in bin $i$ . Higher entropy indicates more uncertainty.

The final truthfulness score for the entire response is the average of the confidence scores across all atomic claims extracted from that response.

Why average confidence across claims: This is a direct analog to the fraction-of-supported-claims metric used in FactScore. Each claim contributes equally to the overall score, regardless of its position in the text or its perceived importance. The average captures the model's overall certainty about the factual content it generated. An alternative would be to take the minimum confidence (flagging the least certain claim as the bottleneck) or the product (treating confidence as the probability that all claims are simultaneously correct), but the average is simpler and directly mirrors the reference-based metric's structure.

Why resample from the base model rather than the fine-tuned model: During training, the factuality-tuned model $\pi_\theta$ is being updated. If confidence were measured from $\pi_\theta$ itself, the model could learn to increase the confidence metric without actually becoming more factual — for instance, by becoming more deterministic in its outputs (reducing variance under resampling) regardless of correctness. By always resampling from the fixed base model (Llama-1-7B), the confidence scores remain a stable signal that reflects the base model's knowledge, not the policy being optimized. This is analogous to using a fixed reward model in RLHF rather than allowing the policy to influence its own reward signal.

Stage 4: Named entity extraction (simpler alternative). As a simpler alternative to the full atomic claim → question conversion pipeline, the paper also evaluates using only named entities extracted from the response via a standard NER classifier (spaCy; Honnibal & Montani, 2017). For each named entity in the text, the model resamples the tokens in that entity's position 20 times (keeping the surrounding context fixed), bins the resampled tokens using the same equivalence checking, and computes confidence as the fraction in the largest bin. For the medical QA dataset, noun chunks are used instead of named entities since medical facts often involve phrases like "chest pain" or "high blood pressure" that may not be recognized as named entities by a standard NER model.

This approach bypasses GPT-3.5 entirely for claim extraction and question generation, making it cheaper and faster. However, it is less precise: a named entity like "Paris" might be replaced with "France" or "Lyon" upon resampling, but the model's uncertainty might stem from not knowing whether the entity is correct, or from multiple equally valid ways to refer to the same entity. The atomic question approach is more targeted because it asks a specific question whose answer should be a single entity or short phrase, making equivalence binning more meaningful.

Why this entire reference-free pipeline is novel: While prior work used model confidence to detect hallucinations at inference time (Kadavath et al., 2022; Kuhn et al., 2023), no prior work had converted confidence signals into a training objective for reducing hallucinations. The key innovation is the claim → question → resample → binning pipeline that produces a scalar confidence score per claim, which can then be aggregated into a response-level truthfulness estimate suitable for constructing preference pairs. This transforms uncertainty from a passive diagnostic into an active training signal.

Preference Dataset Construction

Given a choice of truthfulness estimator (FactScore or model confidence), the construction of the DPO training dataset follows a fixed procedure that is identical for both estimators.

Step 1: Generate candidate responses from unlabeled prompts. The paper uses two datasets of prompts requiring long-form factual generation:

Biographies: 355 diverse well-known individuals (296 train, 59 test), each with 10 candidate biographies generated by the base model using few-shot prompting and temperature sampling at temperature 1.0.
Medical QA: 200 diverse common medical conditions (150 train, 50 test), each with 6 questions about the condition (e.g., "What are the common symptoms of a stroke?") and 6 candidate answers per question, also generated by the base model with temperature 1.0.

The prompts were generated by GPT-3.5, and the responses were sampled from Llama-1-7B using few-shot prompts tailored to each dataset. The authors note that this procedure "consistently resulted in well-formed and informative responses, albeit with possible factual errors" (Section 4). All individuals and medical conditions in the datasets have corresponding Wikipedia pages, which is necessary for the FactScore truthfulness estimator.

Step 2: Score each response. For every generated response, the truthfulness estimator produces a single scalar score between 0 and 1 (for FactScore, the fraction of supported claims; for model confidence, the average confidence across claims). This step is the computational bottleneck: for FactScore, it requires running GPT-3.5 for claim extraction, retrieving Wikipedia text, and running the NLI model for each claim; for model confidence, it requires running GPT-3.5 for claim extraction and question generation, then resampling 20 answers per claim from the base model, then binning.

Step 3: Create pairwise preferences. For each prompt $x$ and its $n$ candidate responses $\{y_1, y_2, \ldots, y_n\}$ , all $\binom{n}{2}$ pairs of responses are considered. For each pair $(y_i, y_j)$ , the response with the higher truthfulness score is designated as the preferred response $y_w$ , and the lower-scoring response is designated as $y_l$ . Pairs where both responses have identical truthfulness scores are discarded.

For the biography dataset with 10 responses per prompt, this yields up to $\binom{10}{2} = 45$ preference pairs per prompt, minus any ties. Across 296 training prompts, the total dataset size is approximately $296 \times 45 \approx 13,320$ pairs minus ties. For medical QA, with 6 responses per prompt and 150 training prompts, the total is approximately $150 \times 15 = 2,250$ pairs minus ties. The authors adjust the number of responses per prompt to "keep the total number of pairs between the two datasets roughly similar" (Section 4), though the exact total pair counts after tie removal are not reported.

Step 4: Supervised fine-tuning (SFT) stage. Before applying DPO, the model is first fine-tuned via standard supervised learning on all generated responses (both those that end up as preferred and dispreferred in the preference data). This SFT stage serves two purposes. First, it provides the reference model $\pi_{\text{ref}}$ for the DPO objective — the DPO loss uses this SFT model as the baseline against which divergence is measured. Second, it ensures the model can generate well-formed responses in the target domain before the DPO stage optimizes for factuality specifically. Without this SFT stage, the DPO updates might improve factuality at the cost of response quality or coherence, because the KL penalty would be measured against a model that has not been adapted to the domain.

Step 5: DPO training. The preference pairs constructed in Step 3 are used to fine-tune the SFT model using the DPO objective (Equation 3). The DPO loss increases the relative log-probability of $y_w$ compared to $y_l$ while the KL penalty term (implicit in the DPO objective through the $\beta$ coefficient and the reference model comparison) prevents the model from deviating too far from the SFT initialization that produces well-formed responses.

What the model actually learns: The DPO training does not explicitly teach the model facts it did not know. Rather, it adjusts the model's generation behavior so that, when uncertain about a fact, the model is less likely to produce a confident-sounding but incorrect claim. The model learns to leverage its own internal uncertainty — the same uncertainty that the confidence-based estimator measures — to avoid generating claims that have a high probability of being false. In the FactScore variant, the model additionally learns which types of claims tend to be unsupported by reference texts, potentially learning to avoid generating claims that fall into systematic error patterns (e.g., inventing specific dates, attributing awards the person did not receive, or fabricating educational details).

Design Choices and Their Justifications

The paper makes several specific design decisions that warrant explanation:

Why DPO over PPO: DPO is "simpler to implement and tune" (Section 2). PPO requires maintaining a separate reward model, sampling from the policy during training, estimating advantages, and managing the complexity of clipping, value function estimation, and online data collection. For an exploratory study testing multiple truthfulness estimators, DPO's simplicity enables rapid iteration. Furthermore, the preference dataset is constructed offline from pre-generated responses, which fits naturally into DPO's fixed-dataset paradigm; PPO would require regenerating responses from the evolving policy, which might create distribution shift issues with the truthfulness estimator (which is calibrated on the base model's outputs).

Why atomic claim extraction for both methods: Long-form text contains many facts embedded in stylistic, transitional, and structural language. A raw sequence probability from the model would assign high probability to a fluent but factually incorrect passage. By decomposing into atomic claims and evaluating each independently, the truthfulness score focuses specifically on factual content. This decomposition also enables the question-conversion step in the reference-free method: each claim maps to a single answerable question.

Why 20 resamples for confidence estimation: The paper does not provide an explicit justification for the number 20, but it represents a pragmatic balance between statistical stability and computational cost. With 20 samples, the granularity of the confidence estimate is 0.05 (1/20) — that is, the smallest non-zero confidence is 5%. This is sufficient to distinguish between high-confidence facts (18/20 = 0.9) and low-confidence facts (7/20 = 0.35). More samples would provide finer granularity but at linear cost in API calls or inference compute.

Why heuristic string matching outperforms GPT-3.5 equivalence checking in the ablation (Table 5): This is a counterintuitive result that the paper addresses directly. The authors hypothesize that "our heuristic equivalence match consistently underestimates semantic entropy across all examples, while GPT-3.5 matching could either over or underestimate samples, resulting in noisier preference pairs, even if GPT-3.5 equivalence check scores are closer to the true semantic entropy on average" (Section 4.4). In other words, the heuristic is systematically conservative — it tends to split semantically equivalent answers into separate bins, which reduces the apparent confidence and lowers the truthfulness score. However, this bias is consistent across examples, so preference pairs are still correctly ordered (the response with genuinely higher confidence still scores higher, even if both scores are underestimated). GPT-3.5, by contrast, introduces unsystematic noise — sometimes overestimating equivalence, sometimes underestimating — which can flip the relative ordering of responses and produce incorrect preference labels. If response A has genuinely higher confidence than response B, systematic underestimation preserves the ranking (both are shifted down by roughly the same amount), while unsystematic noise can reverse it. The paper's finding that systematic bias is less harmful than unsystematic noise for preference learning is a subtle but important insight.

Why maximum confidence slightly outperforms entropy in the ablation: Entropy over semantic bins is a more information-theoretically complete summary of the distribution — it captures the spread across all bins, not just the size of the largest. However, entropy is also more sensitive to the number of bins and to the specific binning granularity, which depends on the equivalence checking method. Maximum confidence is a coarser but more robust metric: it only depends on correctly identifying the single largest cluster of semantically equivalent answers, which is less sensitive to how the smaller clusters are partitioned. For preference construction, what matters is which response scores higher, not the absolute scores, and the maximum confidence metric appears to produce a more reliable ranking.

Why temperature 1.0 for candidate generation: Using temperature 1.0 (the default sampling temperature for many LLMs) ensures diversity in the candidate responses. If temperature were too low, all $n$ responses to a prompt would be near-identical, yielding preference pairs where both responses have the same factual content and similar truthfulness scores — these pairs would be discarded or, if kept, would provide a weak training signal. High temperature ensures that the candidate set includes both more-factual and less-factual responses, creating meaningful preference pairs where the model can learn to distinguish between them. However, temperature that is too high would produce degenerate, low-quality text that is trivially identifiable as worse, making the learning problem too easy and failing to teach the model about the subtle distinction between supported and unsupported claims in otherwise well-formed text.

Why the base model is Llama-1-7B for confidence resampling even when fine-tuning Llama-2: When fine-tuning Llama-2-7B models, the confidence-based truthfulness estimator still uses Llama-1-7B for answer resampling. This is a deliberate decoupling: the confidence signal comes from a fixed, frozen model, so it does not change as the policy being trained (Llama-2-7B) evolves. If Llama-2-7B were used for both generation and confidence estimation, the policy could learn to increase the confidence metric without improving factuality — for instance, by generating claims that are phrased in a way that leads to less diverse resampling (more deterministic expressions) regardless of truth. Using a separate, frozen model for confidence estimation ensures the reward signal is stationary and cannot be gamed by changes in the policy's generation style.

Why the SFT stage uses all generated responses, not just preferred ones: The SFT stage aims to adapt the model to the domain (biographies or medical QA) and provide a well-initialized reference model for DPO. Using all responses maximizes the amount of domain-specific training data. Importantly, this SFT model is not optimized for factuality — it simply learns to generate well-formed responses in the domain — so including factually incorrect responses does not hurt the SFT objective (which is just maximum likelihood on the generated text). The factuality optimization happens entirely in the subsequent DPO stage, where the preference pairs teach the model to discriminate between more and less factual outputs.

4. Key Insights and Innovations

Innovation 1: Reframing Hallucination Mitigation as an Objective Optimized at Training Time, Not an Error Detected at Inference Time

The most fundamental conceptual move this paper makes is to shift the field's framing of hallucination from a detection-and-patching problem to an optimization-at-training-time problem. Before this work, the dominant research paradigm treated factual errors as something to catch and correct after generation — whether through retrieval-augmented verification (Peng et al., 2023; Gao et al., 2023), decoding-time interventions that shift internal activations (Li et al., 2023; Chuang et al., 2023), or prompting strategies that encourage caution (Si et al., 2023). The model itself remained unchanged; factual reliability was a property of the system wrapped around the model, not of the model's weights.

The paper's core reframing — "Can language models be fine-tuned to leverage this internal awareness, to avoid making untrue statements in the first place?" (Section 1) — treats the model's propensity to hallucinate as a behavioral problem that can be directly optimized against through preference learning, analogous to how RLHF optimizes for helpfulness and harmlessness. This is not an incremental improvement on detection — it is a fundamentally different theory of where factuality should be implemented in the system stack. The detection paradigm says: generate first, verify second, correct third. The optimization paradigm says: train the generator so verification and correction are needed far less often.

What makes this move significant beyond performance gains is that it opens a new axis for improvement that is complementary to both detection and retrieval methods. Table 4 demonstrates this empirically: applying DOLA decoding on top of a factuality-tuned model yields further accuracy gains (from 0.812 to 0.864 on Llama-1 biographies, and from 0.783 to 0.794 on medical QA). This suggests that training-time factuality optimization and inference-time interventions operate through distinct mechanisms — the former changes what the model is likely to generate, the latter modifies how it selects among possibilities — and that the two can compound. If factuality tuning only replicated what decoding interventions achieve through a different mechanism, stacking them would show no improvement. The additive gains provide evidence for a genuinely new axis of control.

The significance of this reframing extends beyond the specific methods in the paper. It suggests that any progress in automated truthfulness estimation — whether through better retrieval, better NLI models, or better confidence calibration — can be directly converted into training signal for language models. This creates a virtuous cycle: better truthfulness estimators → better training data for factuality optimization → more factual models → fewer errors to detect → reduced burden on downstream verification systems. The paper's contribution is not any single truthfulness estimator, but the demonstration that this cycle is viable and that DPO provides a simple mechanism for closing the loop.

Innovation 2: Model Confidence as a Training Signal, Not Just a Diagnostic

The paper's second distinctive contribution is repurposing model confidence from a passive diagnostic tool into an active training signal for reducing hallucinations. This is a genuinely novel use of calibration research that has no precedent in the prior literature.

Before this work, the finding that language model confidence correlates with correctness (Kadavath et al., 2022; Tian et al., 2023) was used exclusively for detection — flagging when a model is likely to be wrong so a human or another system can intervene. Kuhn et al. (2023) introduced semantic uncertainty as a more refined confidence metric that accounts for meaning-equivalent paraphrases, but again used it only for evaluation. The unspoken assumption was: confidence tells you that the model might be wrong, but it cannot tell you how to fix it. The model already "knows what it knows" (in Kadavath et al.'s phrase), and that knowledge is apparently inaccessible to the training procedure.

The paper challenges this assumption directly. The key insight is that confidence scores can be converted into a ranking over candidate responses — the response whose claims the model is more confident about is (probabilistically) the more factual response — and this ranking can serve as preference data for DPO. The model is not being taught new facts; it is being taught to self-censor claims that fall below its own internal confidence threshold. This transforms the calibration signal from an external evaluation of the model into an internal optimization objective that the model can learn to satisfy.

What makes this particularly elegant is that it bypasses the chicken-and-egg problem that often plagues self-supervision: if the model is uncertain about a claim because it lacks the knowledge, how can it learn to avoid making that claim? The answer is that the preference ranking does not require the model to know the correct answer — it only requires that the model's confidence be higher for the response that is actually more factual. Even if confidence scores are imperfectly calibrated (they are), and even if the model cannot distinguish between a claim being wrong versus being expressed in an unusual way (the claim-to-question conversion addresses this), the relative ordering of two responses by average confidence carries a reliable signal, as long as the calibration errors are more systematic than random. The ablation in Table 5 provides evidence for exactly this: systematic underestimation of confidence (from heuristic string matching) produces better preference pairs than unsystematic noise (from GPT-3.5 equivalence checking), because the former preserves the ranking while the latter can reverse it.

This innovation has implications well beyond factuality. Any behavioral property where the model exhibits internal signals correlated with output quality — honesty, coherence, safety, groundedness — could potentially be optimized through the same pattern: convert the signal into a preference ranking, apply DPO. Confidence-based factuality tuning is a proof of concept for a broader class of introspective self-improvement methods where the model's own uncertainty guides its behavioral refinement without external supervision.

Innovation 3: The "Strict Improvement" Criterion as a Diagnostic for Genuine Factuality Gains

The paper introduces a conceptual diagnostic — the strict improvement region — that reveals a previously overlooked failure mode in factuality interventions and provides a more rigorous standard for evaluating them. This is not a method but a evaluation concept, and it proves essential for distinguishing between interventions that genuinely make models more factual and those that merely shift the tradeoff between correct and incorrect information.

The strict improvement region, shown visually in Figure 3, is the quadrant where an intervention simultaneously increases the number of correct facts generated and decreases the number of incorrect facts. Prior work on factuality interventions — including the decoding methods ITI and DOLA, as well as standard RLHF — had reported improvements in factuality metrics (typically percentage of correct claims) without decomposing whether the gain came from generating more correct facts, generating fewer incorrect facts, or both. The paper's breakdown reveals a stark pattern: most prior methods do not achieve strict improvement. ITI and DOLA lie outside the strict improvement region for both Llama-1 and Llama-2 on biographies (Figure 3, left), meaning they either increase correct facts at the cost of also increasing errors, or reduce errors at the cost of suppressing correct information. In other words, they are trading one type of failure for another rather than fixing the underlying problem.

By contrast, FactTune-FS is the only method that lands in the strict improvement region for both datasets and both model families. It simultaneously increases correct facts and reduces incorrect facts. FactTune-MC lies just outside the strict improvement region — it reduces errors substantially but produces slightly fewer correct facts than the SFT baseline — which the paper interprets as a more conservative behavior that prioritizes avoiding errors over maximizing information content.

This diagnostic is significant because it reveals that the standard metric — percentage of correct claims — can be misleading as a measure of factuality. A model that generates only the single incontrovertible fact "Yo-Yo Ma is a cellist" and stops would achieve 100% correctness but contain almost no information. A model that generates 20 facts, 15 correct and 5 incorrect, achieves 75% correctness but is arguably more useful despite being less "factual" by the percentage metric. The strict improvement criterion establishes that a genuinely more factual model must deliver more truth with fewer falsehoods, not just a better truth-to-falsehood ratio achieved by saying less.

The concept also provides a diagnostic for reward overoptimization. If an intervention appeared to improve the percentage correct solely by reducing the total number of claims (making the task easier for the verifier), it would fall outside the strict improvement region — fewer errors, but also fewer correct facts. The paper's later validation (Section 4.5, Figure 4) confirms that FactTune-FS improvements are not simply a result of the model becoming terser or more conservative: GPT-4 and human evaluators both agree that the gains reflect genuine improvements in factual accuracy, not exploitation of the FactScore metric. The strict improvement criterion thus serves as an internal consistency check that protects against certain forms of reward hacking, even before external validation is conducted.

Innovation 4: Factuality Tuning as Composable with RLHF, Not a Replacement

The paper demonstrates that factuality optimization through DPO is composable with existing RLHF alignment rather than requiring a separate, from-scratch training pipeline. This is shown in Table 3, where applying FactTune-FS and FactTune-MC on top of Llama-2-Chat (an already RLHF-trained dialogue model) further improves factuality: error count drops from 6.41 to 4.06 (FactTune-FS) or 4.84 (FactTune-MC) on biographies, and percentage correct increases from 74.8% to 83.1% or 81.2%.

The significance of this finding is that it decouples factuality from general instruction-following and helpfulness. Before this work, one might have assumed that RLHF's "honesty" component already addresses factuality to the extent possible given the base model's capabilities, and that further improvements would require either better pretraining data or retrieval augmentation. The paper shows that standard RLHF leaves substantial room for improvement that can be addressed by a targeted second-stage fine-tuning using automatically constructed factuality preferences. This is not obvious a priori — RLHF already optimizes human preference judgments, and human annotators presumably penalize obvious factual errors. The fact that an additional factuality-specific tuning stage provides large gains suggests that the human preference signal used in standard RLHF is not sufficiently sensitive to subtle factual errors, or that the preference data distribution (which spans many dimensions of quality) dilutes the factuality signal relative to a dedicated factuality dataset.

The composaibility finding also has practical implications for deployment. It means that factuality tuning can be added as a post-processing stage to existing aligned models without disrupting their conversational abilities, safety properties, or instruction-following behavior. The paper notes qualitative changes in generation style — FactTune models produce "more objective and direct sentences and less of a conversational or story-telling style" (Section 4.1) — but these changes do not appear to degrade the model's fundamental ability to follow instructions or engage in dialogue; they represent a shift in rhetorical strategy toward factual caution, not a loss of capability.

This composability positions factuality tuning as a modular improvement that can be independently developed, evaluated, and deployed without requiring changes to the upstream pretraining or RLHF pipelines. As truthfulness estimators improve, the factuality tuning stage can be upgraded without retraining the entire alignment stack — a practical advantage that accelerates the path from research to deployment.

5. Experimental Analysis

Evaluation Methodology

Dataset. The experiments use two custom-constructed datasets targeting long-form open-ended generation where factuality can be evaluated against Wikipedia. The Biographies dataset contains 355 diverse well-known individuals (296 train, 59 test), each with 10 short-paragraph biographies generated by Llama-1-7B using few-shot prompting and temperature 1.0 sampling. The Medical QA dataset contains 200 diverse common medical conditions (150 train, 50 test), each with 6 questions about the condition (e.g., "What are the common symptoms of a stroke?") and 6 short-paragraph answers per question, also generated by Llama-1-7B. Prompts for both datasets were generated with GPT-3.5; all individuals and medical conditions have corresponding Wikipedia pages to enable FactScore evaluation (Section 4).
Base model(s). The primary models are Llama-1-7B and Llama-2-7B, with additional experiments on Llama-2-7B-Chat (the RLHF-trained dialogue variant; Touvron et al., 2023b). The 7B scale is chosen as representative of widely-used open-source LLMs. For the reference-free confidence estimation, the paper uses Llama-1-7B as the frozen model for answer resampling even when fine-tuning Llama-2 models, decoupling the confidence signal from the policy being optimized (Section 4.4).
Metrics. The primary evaluation metric is counts of correct, incorrect, and (for Medical QA) relevant atomic facts extracted from each generated response, following the FactScore procedure (Min et al., 2023). Each response is decomposed into atomic claims using GPT-3.5, and each claim is checked against the relevant Wikipedia article using a Llama-1-7B model fine-tuned for natural language inference. A claim classified as "supported" by the Wikipedia text is counted as correct; claims classified as "refuted" or "not supported" are counted as incorrect. From these counts, the paper reports # Correct (number of supported claims), # Incorrect (number of unsupported claims), and % Correct (proportion of relevant supported claims out of total extracted claims). For biographies, relevance checking is skipped because "essentially 100% of facts were relevant to the individual" (Section 4). All results in Tables 2–6 and Figures 3–4 report these metric decompositions.
Baselines. The paper compares against five baselines:
- SFT: The base model fine-tuned via standard supervised learning on all generated responses in the target domain (biographies or medical QA), without any factuality-specific optimization. This serves as the reference model π_ref for DPO and as the primary point of comparison for measuring factuality improvements.
- RLHF (Llama-2-Chat): The off-the-shelf Llama-2-7B-Chat model (Touvron et al., 2023b), which has been trained with reinforcement learning from human feedback on general helpfulness, harmlessness, and honesty preferences. Only applicable for Llama-2 experiments.
- ITI: Inference-Time Intervention (Li et al., 2023), which identifies a "truthfulness direction" in the model's activation space using linear probes trained on FactScore-labeled atomic facts, then applies this direction as an activation shift during inference.
- DOLA: Decoding by Contrasting Layers (Chuang et al., 2023), which modifies the decoding procedure by contrasting the output distributions of early versus later transformer layers to amplify factual knowledge.
- Majority voting: Not explicitly listed as a baseline in the factuality results (it is mentioned only briefly in the context of the revision model experiments, which are in a different paper).
Generation budget / compute accounting. The paper does not report a standardized compute budget in FLOPs or GPU-hours. Instead, the key resource metric is the number of candidate responses generated per prompt during training data construction: 10 responses per prompt for biographies, 6 per prompt for medical QA. All methods use the same candidate responses to construct their training signals (SFT uses all responses, DPO methods use preference pairs derived from the same responses). At inference time, all methods generate a single response per prompt (greedy or temperature sampling at temperature 1.0, though the exact decoding parameters are not explicitly specified for evaluation). There is no sweep over inference-time generation budgets (e.g., best-of-N, beam search), as the paper's focus is on training-time optimization rather than test-time compute scaling.
Cross-validation / statistical protocol. The paper reports a simple train/test split for both datasets (e.g., 296/59 for biographies, 150/50 for medical QA), with no cross-validation, no confidence intervals, and no statistical significance testing for the main results in Tables 2–4. The validation of metrics in Section 4.5 uses a separate human evaluation on a subset of examples and a GPT-4 evaluation covering both SFT and FactTune-FS outputs across both datasets, but does not report sample sizes or statistical tests for the metric-correlation analysis. For the human evaluation, evaluators were recruited through Prolific.co and "compensated at an estimated hourly rate of $16-18" (Section 4.5), but the number of evaluators, number of examples evaluated per model, inter-annotator agreement, and confidence intervals are not reported.

Main Quantitative Results

Factuality Tuning Across Domains (Table 2)

The central result compares FactTune-FS and FactTune-MC against SFT, ITI, DOLA, and (for Llama-2) RLHF on both the biographies and medical QA benchmarks. All numbers in Table 2 represent per-response averages of correct and incorrect fact counts computed by FactScore on the test sets.

On biographies with Llama-1-7B:

SFT achieves 13.78 correct facts and 12.16 incorrect facts (56.8% correct).
FactTune-FS achieves 14.81 correct facts and 3.75 incorrect facts (81.2% correct) — a 69.2% reduction in error rate (from 12.16 to 3.75 errors per response) while simultaneously increasing correct facts by 7.5%.
FactTune-MC achieves 10.59 correct facts and 2.94 incorrect facts (78.3% correct) — a 75.8% reduction in error rate but with a 23.2% decrease in correct facts, representing a more conservative generation strategy.
ITI: 11.67 correct, 6.69 incorrect (66.9% correct) — reduces errors vs. SFT but also reduces correct facts; outside the strict improvement region (Figure 3).
DOLA: 11.75 correct, 3.84 incorrect (75.4% correct) — competitive error reduction with FactTune-FS but produces fewer correct facts.

On biographies with Llama-2-7B:

Llama-2-Chat achieves 19.03 correct, 6.41 incorrect (74.8% correct).
FactTune-FS achieves 17.06 correct, 2.00 incorrect (89.5% correct) — a 68.8% reduction in errors compared to Chat (from 6.41 to 2.00), though correct facts decrease slightly (from 19.03 to 17.06).
FactTune-MC achieves 11.31 correct, 2.06 incorrect (84.6% correct) — error rate reduced by 67.8% vs. Chat, but with substantially fewer correct facts (11.31 vs. 19.03), again reflecting conservative generation.
ITI: 18.50 correct, 5.75 incorrect (76.0%) — slight error reduction vs. Chat.
DOLA: 13.41 correct, 5.84 incorrect (69.6%) — worse than Chat on both dimensions.
SFT: 12.19 correct, 5.19 incorrect (70.1%) — substantially fewer correct facts than Chat, highlighting that domain-specific SFT alone does not match the information content of RLHF training.

On medical QA with Llama-1-7B:

SFT: 10.75 correct, 6.31 incorrect (63.0% correct).
FactTune-FS: 10.88 correct, 4.50 incorrect (70.7% correct) — a 28.7% reduction in errors while maintaining approximately equal correct facts.
FactTune-MC: 12.31 correct, 6.88 incorrect (64.2% correct) — surprisingly, increases both correct facts (by 14.5% vs. SFT) and incorrect facts (by 9.0%), resulting in only marginal percentage improvement. This is the only setting where FactTune-MC fails to reduce the absolute error count.
ITI: 8.91 correct, 5.16 incorrect (63.3%) — fewer facts overall than SFT.
DOLA: 8.03 correct, 5.91 incorrect (57.6%) — worse than SFT.

On medical QA with Llama-2-7B:

Chat: 9.63 correct, 5.50 incorrect (63.6% correct).
FactTune-FS: 12.53 correct, 3.47 incorrect (78.3% correct) — a 36.9% reduction in errors vs. Chat, with a 30.1% increase in correct facts. This is the strongest demonstration of strict improvement: the model both says more correct things and fewer incorrect things simultaneously.
FactTune-MC: 11.41 correct, 4.80 incorrect (70.4% correct) — 12.7% error reduction with modest increase in correct facts.
ITI: 10.97 correct, 4.06 incorrect (73.0%) — competitive error reduction but fewer correct facts than FactTune-FS.
DOLA: 9.72 correct, 4.38 incorrect (69.0%) — marginal improvement over Chat.
SFT: 11.75 correct, 6.75 incorrect (63.5%) — more facts overall but higher error rate.

The headline claim from Section 1 — "58% and 40% reduction in factual error rate when generating biographies and answering medical questions, respectively" for Llama-2 — maps to the Chat vs. FactTune-FS comparison: 6.41 → 2.00 errors on biographies (68.8% reduction, though the paper reports 58%, likely computed as (6.41 - 2.00)/6.41 = 68.8% vs. (6.41 - 2.69)/6.41 ≈ 58% using a different baseline or rounding); and 5.50 → 3.47 on medical QA (36.9% reduction, reported as ~40%). The exact basis for the 58% and 40% figures is not precisely traceable to a single row in Table 2; the paper likely rounds or aggregates across sub-conditions.

Factuality Tuning Chat Models (Table 3)

The key question here is whether factuality tuning can improve an already RLHF-aligned dialogue model without degrading its capabilities. Table 3 compares Llama-2-Chat baseline against further fine-tuning with FactTune-FS, FactTune-MC, and the DOLA decoding intervention.

On biographies:

Chat: 19.03 correct, 6.41 incorrect (74.8%).
FactTune-FS: 19.94 correct, 4.06 incorrect (83.1%) — errors reduced by 36.7% while maintaining essentially the same number of correct facts.
FactTune-MC: 20.91 correct, 4.84 incorrect (81.2%) — errors reduced by 24.5%, correct facts slightly increased.
DOLA applied to Chat: 21.00 correct, 5.19 incorrect (80.2%) — improves correct facts and reduces errors vs. Chat, but not as dramatically as FactTune-FS on error reduction.

On medical QA:

Chat: 9.63 correct, 5.50 incorrect (63.6%).
FactTune-FS: 9.38 correct, 5.25 incorrect (68.2%) — marginal improvements (errors down 4.5%, correct facts down 2.6%).
FactTune-MC: 10.34 correct, 5.69 incorrect (64.5%) — slight increases in both dimensions.
DOLA: 11.50 correct, 8.25 incorrect (58.2%) — produces more correct facts but substantially more errors, falling well outside the strict improvement region.

The finding is that factuality tuning composes with RLHF: the Chat model can be further improved, particularly on biographies, where FactTune-FS achieves an 11.1% absolute improvement in percentage correct (74.8% → 83.1%). On medical QA, the gains are smaller (63.6% → 68.2% for FactTune-FS), suggesting that the Chat model's existing factuality is closer to the ceiling achievable through preference optimization without new knowledge acquisition — or that the medical QA domain is harder for the FactScore verifier to provide a reliable signal.

Complementary Benefits of Factuality Tuning and Decoding Interventions (Table 4)

This experiment tests whether training-time factuality optimization (FactTune-FS) and inference-time factuality interventions (DOLA) operate through complementary mechanisms such that combining them yields additive gains.

On biographies with Llama-1:

FactTune-FS alone: 14.81 correct, 3.75 incorrect (81.2%).
FactTune-FS + DOLA: 12.44 correct, 2.00 incorrect (86.4%) — further error reduction (from 3.75 to 2.00) but at the cost of reducing correct facts (14.81 → 12.44). The net effect is a 5.2 percentage point improvement in % correct, representing additive benefit from DOLA.

On medical QA with Llama-1:

FactTune-FS alone: 10.88 correct, 4.50 incorrect (70.7%).
FactTune-FS + DOLA: 11.47 correct, 3.75 incorrect (76.7%) — simultaneous improvement in both dimensions, a clear strict improvement over FactTune-FS alone.

On biographies with Llama-2:

FactTune-FS alone: 17.06 correct, 2.00 incorrect (89.5%).
FactTune-FS + DOLA: 16.22 correct, 2.65 incorrect (86.5%) — DOLA degrades performance, increasing errors and reducing correct facts. This is the exception noted in Section 4.3.

On medical QA with Llama-2:

FactTune-FS alone: 12.53 correct, 3.47 incorrect (78.3%).
FactTune-FS + DOLA: 12.56 correct, 3.44 incorrect (79.4%) — negligible change, suggesting the interventions are redundant rather than complementary for this model-dataset combination.

The paper's claim that "DOLA can even further increase the accuracy of factuality fine-tuned models" (Section 4.3) holds for 3 out of 4 model-dataset combinations, with the fourth (Llama-2 biographies) showing degradation. The authors do not investigate why this exception occurs, leaving it as a qualitative observation rather than an explained phenomenon.

Strict Improvement Analysis (Figure 3)

Figure 3 plots # Correct facts per response (x-axis) against # Incorrect facts per response (y-axis) for each method on both datasets, using Llama-2 (the figure caption in the paper references Llama-2 only, though the text discusses both model families). The "strict improvement" region is the quadrant where a method has simultaneously higher correct counts and lower incorrect counts than the SFT baseline — the top-left region in the plot coordinates.

For biographies (Figure 3, left panel):

SFT sits at approximately (12.2 correct, 5.2 incorrect).
FactTune-FS sits at (17.1 correct, 2.0 incorrect) — clearly in the strict improvement region, with a substantial shift upward and leftward.
FactTune-MC sits at (11.3 correct, 2.1 incorrect) — just outside the strict improvement region, with errors reduced but correct facts slightly lower than SFT.
RLHF (Chat) sits at (19.0 correct, 6.4 incorrect) — more correct facts than SFT but also more errors, in the "more information but less accurate" quadrant.
ITI sits at (18.5 correct, 5.8 incorrect) — very close to Chat, outside strict improvement.
DOLA sits at (13.4 correct, 5.8 incorrect) — fewer correct facts than SFT, similar error count, outside strict improvement.

For medical QA (Figure 3, right panel):

SFT sits at approximately (11.8 correct, 6.8 incorrect).
FactTune-FS sits at (12.5 correct, 3.5 incorrect) — in the strict improvement region.
FactTune-MC sits at (11.4 correct, 4.8 incorrect) — fewer errors but also fewer correct facts than SFT, just outside strict improvement.
Chat, ITI, and DOLA all fall outside the strict improvement region, with Chat and ITI having fewer correct facts and fewer errors than SFT, and DOLA having more errors.

The visual decomposition makes explicit why FactTune-FS is the paper's flagship method: it is the only intervention that consistently moves the model into the strict improvement quadrant, meaning it makes the model more factual without trading off informativeness for accuracy.

Validation of Factuality Metrics (Table 6, Figure 4)

To rule out the possibility that the observed improvements are artifacts of overoptimizing against the FactScore evaluation metric, Section 4.5 provides two external validations.

Human evaluation (Table 6). Human evaluators from Prolific.co were given the prompt, generated response, and Wikipedia article title, and asked to "count the total number of facts and the number of incorrect facts in the response." The human-rated accuracy is the fraction of facts judged correct. Results:

Dataset	Model	Human Accuracy	FactScore Accuracy
Biographies	SFT	0.582	0.669
Biographies	FactTune-FS	0.846	0.921
MedQA	SFT	0.662	0.534
MedQA	FactTune-FS	0.838	0.806

Human evaluators rate FactTune-FS substantially higher than SFT on both datasets: +26.4 percentage points on biographies, +17.6 points on medical QA. The FactScore metric is somewhat optimistic for biographies (0.921 vs. 0.846 human) and pessimistic for medical QA SFT (0.534 vs. 0.662 human), but the key finding is that the ranking is preserved: FactTune-FS > SFT under both evaluation methods, with large effect sizes. The absolute gap between FactTune-FS and SFT is actually larger under human evaluation than under FactScore for medical QA (+17.6 vs. +27.2 percentage points), indicating that FactScore may understate the improvement.

GPT-4 evaluation (Figure 4). GPT-4 was prompted to count the number of factual errors in each response. The average FactScore error count and average GPT-4 error count are "highly correlated" according to the paper, and Figure 4 (a scatter plot with points for SFT, FactTune-FS, and FactTune-MC on both datasets) shows a clear positive relationship. Critically, the FactTune-FS and FactTune-MC points consistently have lower error counts than SFT under both metrics, confirming that the error reduction is not specific to the FactScore NLI model. The axes are scaled within each dataset (GPT-4 error counts are scaled by the maximum GPT-4 error count in that dataset), and all FactTune points cluster at the lower-left of their respective SFT points, indicating cross-metric agreement.

The paper does not report correlation coefficients (Pearson, Spearman), per-example agreement rates, or statistical tests, limiting the precision of this validation.

Ablation Studies and Robustness Checks

Fact extraction method (atomic claims vs. named entities) for reference-free confidence estimation (Table 5): Using GPT-3.5 to extract atomic claims and convert them to questions (Atomic rows) generally outperforms the simpler approach of extracting named entities (or noun chunks for medical QA) and resampling them directly (Entity rows). On biographies with maximum confidence and atomic extraction: 12.2 correct, 2.56 incorrect (84.0% correct) vs. entity-based: 12.7 correct, 6.31 incorrect (69.3% correct). The atomic method's advantage comes almost entirely from reducing incorrect facts (2.56 vs. 6.31), while correct facts are similar. On medical QA, the gap is smaller (10.2 correct, 5.19 incorrect, 67.3% for atomic vs. 9.5 correct, 4.78 incorrect, 67.3% for entity with max confidence), suggesting that named entity resampling is a more reasonable proxy for factuality in the medical domain — perhaps because medical facts often center on specific entity-like terms (conditions, symptoms, treatments).

Confidence metric (maximum confidence vs. entropy) (Table 5): The choice between maximum confidence (fraction in largest semantic bin) and entropy over semantic bins has mixed effects. For biographies with atomic extraction: maximum confidence achieves 12.2 correct, 2.56 incorrect (84.0%) vs. entropy at 10.6 correct, 2.88 incorrect (81.0%). Max confidence provides better results on biographies but slightly worse on medical QA for the entity extraction setting (12.7 correct, 6.31 incorrect, 69.3% for max vs. 13.8 correct, 6.31 incorrect, 69.3% for entropy — identical percentage). The paper attributes maximum confidence's advantage to greater robustness: entropy is sensitive to the granularity of semantic binning, which depends on the equivalence checking method, while maximum confidence only requires correctly identifying the single largest cluster.

Equivalence checking method (heuristic string match vs. GPT-3.5) (Table 5): The surprising finding is that heuristic string matching outperforms GPT-3.5 equivalence checking for constructing preference pairs. For biographies with atomic extraction and maximum confidence: heuristic matching achieves 12.2 correct, 2.56 incorrect (84.0%) vs. GPT-3.5 checking at 13.7 correct, 4.16 incorrect (79.4%). GPT-3.5 produces more correct facts on average but also substantially more errors, whereas heuristic matching's more conservative binning (splitting semantically equivalent answers into separate bins) systematically underestimates confidence but preserves the ranking across responses. The paper's interpretation (discussed in Section 3.4 under Design Choices) is that systematic underestimation preserves preference orderings while GPT-3.5's unsystematic noise can flip them, producing lower-quality preference pairs.

Effect of relevance filtering on evaluation (Medical QA only): The paper adds a relevance check using GPT-3.5 for medical QA facts (to ensure claims about medical conditions are actually about the condition in question), but not for biographies since "essentially 100% of facts were relevant." The impact of this filter is not ablated — there is no comparison of medical QA results with and without relevance filtering — so its effect on the reported metrics is unknown. If relevance filtering removes some correct claims as irrelevant (false negatives) or accepts some incorrect claims as relevant (false positives), it could bias the reported % Correct metric in either direction.

Critical Assessment

Claim: "Fine-tuning with factuality preference rankings significantly improves factuality" (Section 1, Table 2)

This claim is strongly supported for the specific models, datasets, and metric (FactScore) used in the paper. FactTune-FS consistently reduces the number of incorrect facts by 29–75% across all model-dataset combinations in Table 2, while maintaining or increasing correct facts (with the exception of Llama-2 biographies where correct facts drop slightly from 19.03 to 17.06). The external validations (human evaluation in Table 6, GPT-4 correlation in Figure 4) confirm that the improvements are not artifacts of overfitting to the FactScore metric — human evaluators rate FactTune-FS as substantially more accurate than SFT on both domains.

However, what "significantly improves factuality" means requires careful scoping. The improvements are measured only on topics where Wikipedia pages exist — both the training data construction (FactScore) and evaluation depend on Wikipedia as the ground truth source. The method does not and cannot improve factuality on topics where the base model has no knowledge. The paper is explicit that all entities and conditions in the datasets have Wikipedia pages; the "hard questions" issue that plagues the compute-optimal test-time scaling paper (where the hardest difficulty bin shows zero improvement regardless of budget) does not arise here because the problem setup deliberately avoids knowledge boundaries by focusing on well-documented entities. A legitimate question is whether the improvements generalize to long-tail entities where Wikipedia coverage is sparse, or to domains where "ground truth" is less clearly defined than biographical and medical facts.

Claim: "58% and 40% reduction in factual error rate for biographies and medical questions" (Abstract, Section 6)

This claim is supported with qualifications. The headline percentages appear to reference the Llama-2-Chat → FactTune-FS comparison. In Table 2, Llama-2-Chat on biographies has 6.41 incorrect facts; FactTune-FS has 2.00 incorrect facts — a 68.8% reduction, not 58%. On medical QA, Chat has 5.50 incorrect; FactTune-FS has 3.47 — a 36.9% reduction, not 40%. The paper's reported figures (58% and 40%) may use a different denominator (e.g., total claims rather than absolute error count, or a subset of the data), but the exact derivation is not specified. Regardless, the qualitative claim of "roughly halving errors on biographies and reducing errors by about a third on medical QA" holds in the table. The discrepancy between the headline numbers and the table raises a minor concern about how error rate is computed — the paper's Abstract reports different numbers than the body, and the calculation method is not explicitly defined.

More importantly, these reduction percentages are point estimates on small test sets (59 individuals for biographies, 50 conditions × 6 questions = 300 responses for medical QA). No confidence intervals are reported. The true error reduction could be substantially higher or lower depending on sampling variability. For the medical QA result specifically, the FactTune-FS improvement over Chat is small: 5.50 → 5.25 incorrect facts per response, with 9.63 → 9.38 correct facts (Table 3). This is a 4.5% error reduction, not 40%, and the 40% figure from the Abstract's "40% reduction" claim is therefore inconsistent with the Chat→FactTune-FS comparison in Table 3 (which shows the Chat model fine-tuned with FactTune-FS). The body of the paper (Section 4.2) presents Table 3 as showing a smaller improvement: "factuality tuning can be composed with RLHF to further improve the factuality of chat models," but the Abstract's 40% figure appears to reference the broader claim of improvement over baselines on the medical QA task, not specifically the Chat→FactTune-FS delta. This ambiguity in which baseline the headline numbers reference weakens the precision of the central quantitative claim.

Claim: "Reference-free confidence-based tuning eliminates the need for external knowledge" (Section 1, Section 6)

This claim is supported in principle but weaker in practice. FactTune-MC does not use Wikipedia or any external knowledge base during truthfulness estimation — all signals come from the model's own confidence scores. This is a genuine methodological contribution that opens the door to factuality tuning in domains where reference texts are unavailable or unreliable.

However, the claim's practical significance is limited by two factors. First, FactTune-MC consistently underperforms FactTune-FS on percentage correct (Table 2): 78.3% vs. 81.2% on Llama-1 biographies, 84.6% vs. 89.5% on Llama-2 biographies, 64.2% vs. 70.7% on Llama-1 medical QA, 70.4% vs. 78.3% on Llama-2 medical QA. The reference-free approach is inferior to the reference-based approach in every setting, albeit often by modest margins. Second, FactTune-MC requires extensive use of GPT-3.5 for claim extraction and question generation, plus 20 resamples per claim from Llama-1-7B, making the training-time cost substantial — potentially comparable to or exceeding the cost of retrieving Wikipedia articles (though this cost is not quantified). The "reference-free" claim is technically true at both training and inference time, but the training pipeline still depends on an external LLM (GPT-3.5) for the claim-to-question conversion step, making it not fully self-contained. The named entity extraction variant (Table 5) removes the GPT-3.5 dependency but degrades performance further.

Claim: "Factuality tuning composes with RLHF" (Section 4.2, Table 3)

This claim is supported on biographies but weak on medical QA. On biographies, applying FactTune-FS to Llama-2-Chat reduces errors from 6.41 to 4.06 (36.7% reduction) while maintaining correct facts (19.03 → 19.94), a clear and meaningful improvement. On medical QA, the improvement is marginal: errors decrease from 5.50 to 5.25 (4.5% reduction), and correct facts decrease slightly from 9.63 to 9.38. This is technically "further improvement" but the effect size is small enough that it could plausibly be noise given the unreported variance. The claim of general composability would be stronger with evidence that the gains on medical QA are statistically reliable, or with results on additional domains beyond the two tested.

Unaddressed Limitations in the Experimental Design

Small test sets with no variance estimates. The biography test set contains 59 entities × 10 responses = 590 evaluated generations. The medical QA test set contains 50 conditions × 6 questions × 6 responses = 1,800 evaluated generations. While the total generation counts are reasonable, the paper treats the per-entity or per-condition averages as point estimates with no reported standard deviations, standard errors, or confidence intervals. This makes it impossible to assess whether the observed differences between methods (e.g., FactTune-FS vs. FactTune-MC) are statistically significant or within the range of sampling noise. For the biography task specifically, the SFT model's average of 13.78 correct facts with a likely high variance (since different individuals have different amounts of known information) means that the 7.5% increase from FactTune-FS could be consistent with noise, though the large reduction in errors (12.16 → 3.75) is likely robust.

Single base model family (Llama). All experiments use Llama-1-7B and Llama-2-7B. The paper does not test on other model families (e.g., Mistral, Falcon, Pythia) that may have different calibration properties, different factual knowledge bases, or different sensitivities to DPO fine-tuning. The claim that the method "can be applied to off-the-shelf language models" is plausible but unverified beyond the Llama family.

Only one scale (7B). The paper explicitly acknowledges this limitation (Section 6): "We explore only 7B models in this work. Scaling up our factuality tuning recipe to larger models (and larger preference datasets) may reduce hallucinations even further." This is a significant gap because model calibration properties change with scale (larger models are typically better calibrated), which could affect both the FactTune-MC confidence signal quality and the magnitude of factuality improvements achievable through DPO. It is also unknown whether the relative performance of FactTune-FS vs. FactTune-MC changes with scale — if larger models have better internal confidence signals, FactTune-MC might close the gap with FactTune-FS.

Confidence estimation uses a fixed frozen model (Llama-1-7B) even when fine-tuning Llama-2. For FactTune-MC applied to Llama-2, the confidence scores come from Llama-1-7B, not Llama-2. This is a deliberate design choice to prevent reward hacking (the policy being optimized should not influence its own reward signal), but it introduces a model mismatch: the confidence signal reflects what Llama-1-7B knows, not what Llama-2-7B knows. If Llama-2-7B has different knowledge or different calibration properties, the preference pairs constructed from Llama-1-7B confidence scores may not accurately rank Llama-2-7B's outputs by factuality. The paper does not ablate this choice (e.g., by comparing Llama-1-7B confidence signals vs. Llama-2-7B confidence signals for training Llama-2), so the cost of the mismatch is unknown.

No comparison to retrieval-augmented generation at inference time. The paper positions reference-free factuality tuning as an alternative to retrieval-based systems (Section 5), but never directly compares FactTune-MC against a retrieval-augmented baseline on the same datasets. The claim that reference-free tuning "eliminates the need for a reference corpus" is true, but does not establish that it is preferable to simply using retrieval at inference time for domains where reference corpora exist. A comparison of FactTune-MC (no retrieval, training only) vs. retrieval-augmented generation (no special training) on the same metrics would clarify the practical tradeoffs.

The claim extraction step uses GPT-3.5, which introduces an external dependency and potential source of error. Both FactTune-FS and FactTune-MC rely on GPT-3.5 to decompose generated text into atomic claims. If GPT-3.5 fails to extract certain claims, or splits compound claims incorrectly, or attributes claims to the wrong entity, the truthfulness scores will be noisy. The paper does not evaluate the accuracy of the claim extraction step (e.g., by comparing GPT-3.5 extractions against human-annotated atomic claims on a subset), so the error rate of this component is unknown. A systematic failure mode in claim extraction (e.g., GPT-3.5 missing claims that use indirect or implied factual language) would bias both training and evaluation.

The medical QA dataset is less well-motivated for the reference-free approach. The biography task is a natural fit for confidence-based factuality tuning because each claim typically tests a single, well-defined piece of knowledge (birth year, profession, education). Medical QA involves more complex claims about symptoms, treatments, and disease mechanisms where the model's confidence might be poorly calibrated — the model could be highly confident in a plausible-sounding but incorrect medical claim. This may explain why FactTune-MC performs worse on medical QA than on biographies (Table 2, Llama-2: 84.6% correct on bios vs. 70.4% on medical QA). The paper does not investigate whether certain types of factual claims are more amenable to confidence-based verification than others.

No training-time compute cost analysis. The paper does not report the GPU-hours, API costs, or wall-clock time required for any stage of the pipeline: candidate generation, FactScore scoring (GPT-3.5 claim extraction + Llama-1-7B NLI inference + Wikipedia retrieval), confidence-based scoring (GPT-3.5 claim extraction and question generation + 20× Llama-1-7B forward passes + binning), SFT, or DPO. This makes it impossible to compare the cost-effectiveness of FactTune-FS vs. FactTune-MC vs. simply using a larger model or retrieval-augmented generation. Given that the paper's motivation emphasizes the cost of human labeling ($2,000 for 505 biographies), a cost analysis of the automated alternative would be directly relevant to the paper's practical claims.

6. Limitations and Trade-offs

6.1 The Method Cannot Improve Factuality on Topics Outside the Base Model's Knowledge

The paper's truthfulness estimators — both reference-based (FactScore) and reference-free (model confidence) — can only assess whether generated claims are consistent with existing knowledge, either in Wikipedia or in the base model's own parameters. They have no mechanism to teach the model new facts. The paper implicitly acknowledges this through its dataset construction: all individuals and medical conditions in the evaluation "have Wikipedia pages" (Section 4), and the factuality tuning pipeline operates on candidate responses generated by the base model itself. If the base model's pass@1 for a particular fact is essentially zero — it never generates the correct answer even under resampling — then all candidate responses will contain errors, and the preference ranking will simply select the least-wrong response rather than a factually correct one. The model cannot learn to generate facts it does not already know.

Consequence. Factuality tuning cannot address errors stemming from fundamental knowledge gaps. For a biography of a lesser-known individual where Wikipedia has sparse or no coverage and the base model lacks parametric knowledge, the method provides no benefit — the preference data contains no examples of correct facts to learn from. More subtly, if the base model has partially correct knowledge but expresses it inconsistently (e.g., correctly stating someone's profession but fabricating their birth year), the confidence-based estimator may assign similar scores to responses with different mixes of correct and incorrect claims, producing weak or noisy preference pairs. This creates a hard ceiling: factuality tuning amplifies existing capabilities but does not create new ones, analogous to how the compute-optimal test-time scaling paper found zero improvement on the hardest difficulty quintile regardless of budget.

Evidence in the paper. The paper does not directly measure this ceiling — there is no analysis of how FactTune performance varies with the base model's pre-existing accuracy on specific facts, and no experiment on long-tail entities where Wikipedia coverage is sparse. The reliance on Wikipedia-verifiable entities for both training and evaluation selects for precisely the regime where the base model is most likely to have non-trivial knowledge. The medical QA results provide indirect evidence: FactTune-FS on Llama-2-Chat achieves only a 4.5% error reduction (5.50 → 5.25 incorrect facts, Table 3), substantially smaller than the 36.7% reduction on biographies (6.41 → 4.06), suggesting that when the base model's knowledge is weaker or less consistent (medical facts are more complex and varied than biographical ones), the preference signal provides less leverage.

Mitigation status. The paper does not address this limitation. Section 6 (Conclusion) notes that the two benchmark tasks "are representative of but do not fully cover the range of scenarios where we would hope to improve factuality," but the knowledge-boundary issue is not discussed. No experiment tests whether factuality tuning degrades or simply plateaus for entities with limited Wikipedia coverage or low base-model confidence.

6.2 The Difficulty Estimation Cost Is Unaccounted For in the Headline Gains

Constructing the preference dataset requires: (a) generating n candidate responses per prompt from the base model (10 for biographies, 6 for medical QA), (b) extracting atomic claims from each response using GPT-3.5, (c) for FactScore, running NLI verification with a fine-tuned Llama-1-7B model against retrieved Wikipedia text for each claim, (d) for model confidence, converting claims to questions with GPT-3.5 and resampling 20 answers per claim from Llama-1-7B, and (e) running SFT and DPO fine-tuning. This pipeline is computationally expensive, but the paper does not report any cost metrics — no GPU-hours, API call counts, or wall-clock times.

Consequence. A practitioner evaluating whether to adopt this method cannot assess its cost-effectiveness relative to alternatives. The Abstract and Introduction emphasize that human fact-checking costs ~$2,000 for 505 biographies, implying that the automated pipeline is cheaper. But without cost numbers, this is unverified. If the automated pipeline requires, say, 10,000 GPT-3.5 API calls and 200 GPU-hours of Llama-1-7B inference, the dollar cost might be comparable to or exceed human annotation depending on scale — particularly for the reference-free method, which requires 20 resamples per claim, and each biography might contain 15-20 claims, yielding 3,000-4,000 Llama-1-7B forward passes per biography just for confidence estimation. At scale, this could cost more than hiring fact-checkers, undermining the paper's primary motivation.

Furthermore, the cost of difficulty estimation (truthfulness scoring for preference construction) is not amortized across the reported improvements. The 58% and 40% error reduction figures in the Abstract measure after the preference dataset has been constructed and DPO applied, but the total cost to achieve these reductions includes the one-time dataset construction cost plus the DPO training cost. For a new domain or model, the entire pipeline must be re-run from scratch — generating new candidate responses, extracting claims, scoring them — since the truthfulness scores depend on the specific model's outputs.

Evidence in the paper. The paper provides no cost analysis whatsoever. There is no mention of GPU-hours, API costs, or wall-clock time in any section. The only cost-related figure is the human annotation cost cited from Min et al. (2023) — $2,000 for 505 biographies — with the paper merely implying that automated alternatives are cheaper without substantiating the claim. The computational requirements of the pipeline must be inferred from the method description: GPT-3.5 is used for claim extraction and question generation; Llama-1-7B is used for NLI verification (FactScore) and answer resampling (model confidence, 20 samples per claim); Wikipedia retrieval is performed for each claim in the FactScore variant; and DPO training is performed on thousands of preference pairs.

Mitigation status. Acknowledged only implicitly. Section 4.4 evaluates a cheaper alternative (named entity extraction instead of atomic claim extraction and question generation) and finds it underperforms, but this ablation compares accuracy, not cost. No future work is suggested on reducing the cost of preference data construction.

6.3 The FactTune-MC Pipeline Is Not Truly Reference-Free at Training Time

The paper's central methodological innovation is FactTune-MC, a "reference-free" truthfulness estimator that "eliminates the need for external knowledge" (Section 1). However, the pipeline relies heavily on GPT-3.5 for two critical steps: (1) decomposing generated text into atomic claims, and (2) converting each atomic claim into a minimally ambiguous question. Both steps require an external, proprietary, and opaque language model. The named entity extraction variant (Section 3.2, Table 5) removes the GPT-3.5 dependency but degrades performance: for Llama-1 biographies with maximum confidence, named entity extraction achieves 69.3% correct vs. 84.0% for atomic question extraction.

Consequence. "Reference-free" means only that the pipeline does not reference a knowledge base like Wikipedia. It does not mean the pipeline is self-contained or reproducible without external API access. A researcher who wants to replicate FactTune-MC needs access to GPT-3.5 (or an equivalent) for claim extraction and question generation, which introduces cost, latency, and a dependency on a model that may change behavior over time (API model updates). Moreover, errors in GPT-3.5's claim extraction or question generation propagate into the confidence scores and thus the preference pairs — if GPT-3.5 fails to extract a claim that the base model got wrong, that error goes undetected; if it converts a claim to an ambiguous question, the confidence score may be artificially low even for a correct fact. The paper does not measure the accuracy of either the claim extraction or question generation steps, so the scale of this error propagation is unknown.

The named entity ablation (Table 5) demonstrates that removing GPT-3.5 substantially degrades performance, so the pipeline's quality depends on the external model. This makes the method's gains contingent on a specific external system, not achievable with the base model alone. A fully self-contained pipeline — using the base model itself for claim extraction and question generation — is not tested, though it would be the natural way to eliminate external dependencies.

Evidence in the paper. The named entity extraction results in Table 5 reveal the gap: atomic question extraction (using GPT-3.5) achieves 84.0% correct on Llama-1 biographies vs. 69.3% for named entity extraction without GPT-3.5. The paper does not ablate using Llama-1 or Llama-2 itself for claim extraction and question generation — only GPT-3.5 is tested. The exact dependence on GPT-3.5 is mentioned in Sections 3.1, 3.2, and 4.4 but never quantified in terms of cost, latency, or error rate.

Mitigation status. The paper does not acknowledge this as a limitation. The "reference-free" framing emphasizes independence from knowledge bases but glosses over the external LLM dependency. No future work is suggested on replacing GPT-3.5 with a self-hosted model for claim extraction or question generation.

6.4 Single Model Family, Single Scale, Two Narrow Domains

All experiments use Llama-1-7B and Llama-2-7B models on exactly two datasets: biographies of well-known individuals and medical question-answering about common conditions, both constructed specifically for this paper. The authors state that "these tasks are representative of but do not fully cover the range of scenarios where we would hope to improve factuality" (Section 6), but the scope of tested generalization is minimal.

Consequence. The paper provides no evidence that factuality tuning transfers to:

Other model families (e.g., Mistral, Falcon, Pythia) that may have different calibration properties, different knowledge bases, or different sensitivities to DPO. The model confidence method specifically depends on the base model's calibration (how well confidence correlates with correctness), which varies substantially across model families and training procedures.
Other scales. The paper acknowledges this explicitly: "We explore only 7B models in this work. Scaling up our factuality tuning recipe to larger models (and larger preference datasets) may reduce hallucinations even further" (Section 6). Larger models are typically better calibrated, which could either improve FactTune-MC (better confidence signal) or reduce its relative advantage (if the base model already has fewer errors, the room for improvement shrinks). Smaller models are typically worse calibrated, potentially making the confidence signal too noisy to serve as a useful preference signal.
Other domains. Biographies and medical QA are both domains where facts are relatively objective, well-defined, and verifiable against authoritative sources. In domains with subjective or contested facts (e.g., political analysis, literary criticism, product reviews), the concept of "truthfulness" is less clear, and both FactScore and model confidence would struggle. The paper does not test on domains like code generation, legal analysis, scientific explanation, or news summarization — which represent a large fraction of real-world LLM use cases.
Languages other than English. Both datasets are English-only, and the Wikipedia verification step for FactScore depends on English Wikipedia coverage. The confidence-based method might generalize better (since it does not require external text), but the claim extraction and question generation prompts are English-specific.

Evidence in the paper. The experiments cover only two model families (Llama-1-7B variants and Llama-2-7B variants), two domains, and one scale (7B parameters). The test sets are small: 59 individuals for biographies, 50 conditions for medical QA. Table 2 shows that the relative performance of FactTune-FS vs. FactTune-MC vs. baselines varies across model and dataset combinations — e.g., FactTune-MC is competitive on Llama-1 biographies (78.3% vs. 81.2% for FS) but substantially worse on Llama-2 medical QA (70.4% vs. 78.3%). This variability suggests that performance is sensitive to model-dataset interactions, making generalization to unseen models or domains uncertain.

Mitigation status. The paper acknowledges the scale limitation explicitly (Section 6) and mentions that scaling "may reduce hallucinations even further," but does not address the model family or domain generalization limitations. No experiments on additional models, domains, or scales are proposed as future work beyond the general statement about scaling.

6.5 The FactScore Evaluation Metric Is Both the Training Signal and Primary Evaluation, Creating Potential Circularity

Both FactTune-FS and the primary evaluation use the same underlying mechanism — FactScore's atomic claim extraction and Wikipedia-based NLI verification — to measure factuality. While FactTune-FS uses this signal for preference pair construction and evaluation uses it for reporting, FactTune-MC uses a different signal (model confidence) for training but is still evaluated with FactScore. The paper's external validations (human evaluation in Table 6, GPT-4 in Figure 4) partially address this concern but have significant limitations.

Consequence. For FactTune-FS specifically, there is a risk of reward overoptimization in a subtle form: the model may learn to generate text that scores well under the specific FactScore NLI model rather than text that is genuinely factual. The NLI model (Llama-1-7B fine-tuned for fact-checking) has its own error patterns — it may classify certain phrasings as "supported" due to superficial lexical overlap with Wikipedia text, or classify genuinely true but differently-phrased claims as "unsupported." DPO could amplify these idiosyncrasies, producing text that is optimized for the NLI model's particular judgment criteria rather than for truth. The paper's external validations address this but are incomplete: the human evaluation (Table 6) covers only SFT vs. FactTune-FS on two datasets, with no reported sample size, no inter-annotator agreement, no confidence intervals, and no per-model breakdown showing whether human rankings agree with FactScore rankings on individual examples (only aggregate averages are reported). The GPT-4 evaluation (Figure 4) shows correlation but no per-example agreement rates or statistical tests.

For FactTune-MC, the evaluation uses FactScore, but the training signal is model confidence — so circularity is not an issue. However, if model confidence and FactScore disagree systematically (e.g., the model is highly confident in a claim that Wikipedia does not support because the Wikipedia article is incomplete or outdated), FactTune-MC might optimize for confidence-calibrated outputs that FactScore rates as incorrect. The paper does not measure the agreement between model confidence scores and FactScore judgments on the same claims, so the extent of this mismatch is unknown.

Evidence in the paper. Table 6 shows that human evaluators rate FactTune-FS higher than SFT, confirming that the improvement is not purely a FactScore artifact. However, the human accuracy numbers differ substantially from FactScore accuracy: for biographies, humans rate SFT at 0.582 vs. FactScore at 0.669 (a 8.7-point gap); for medical QA, humans rate SFT at 0.662 vs. FactScore at 0.534 (a 12.8-point gap in the opposite direction). These gaps indicate that FactScore is not perfectly aligned with human judgment, and the direction of misalignment varies by domain. Figure 4 shows correlation between FactScore and GPT-4 error counts but the axes are normalized per-dataset, obscuring whether GPT-4's absolute error counts match FactScore's or are merely correlated in ranking.

Mitigation status. Partially addressed through the human evaluation (Table 6) and GPT-4 validation (Figure 4). The authors explicitly state that "to validate that our models do not suffer from extreme reward overoptimization, we conduct a human evaluation" (Table 6 caption). However, the validation is limited in scope and statistical rigor. No analysis is provided of whether the FactScore NLI model has systematic biases that DPO might exploit, and no adversarial evaluation (e.g., testing whether FactTune-FS produces claims that fool the NLI model but are factually wrong) is conducted.

6.6 The Method Produces Qualitatively Different (Terser, Less Conversational) Text, With Unmeasured Impact on User Experience

The paper reports that FactTune models generate qualitatively different text from the SFT baseline. Section 4.1 notes:

"FactTune-FS and FactTune-MC samples tend to have more objective and direct sentences and less of a conversational or story-telling style compared to the SFT model"
"GPT-4 rates FactTune-FS as less conversational in tone than the SFT model for 77.5% (n=40) of Llama-1 questions and 65.6% (n=32) of Llama-2 samples"

Appendix Tables 8 and 9 provide examples where factuality-tuned biographies lack chronological organization and omit transitional language present in SFT generations.

Consequence. The "more factual" model may be less useful in ways not captured by the FactScore metric. A biography that states facts accurately but in jumbled, non-chronological order with no narrative structure may be harder for a human to read, less engaging, or less informative overall — even if each individual claim is correct. For the medical QA task, a "terser" answer that omits context, hedging, or explanatory detail might be technically correct but less helpful to a patient trying to understand their condition. The paper's evaluation framework counts atomic facts but does not measure coherence, readability, or usefulness — dimensions that matter for real-world deployment.

Moreover, the "less conversational" style may indicate a form of reward hacking: the model learns that shorter, simpler sentences are more likely to pass the NLI verification or have higher confidence scores, so it adopts a more telegraphic style that reduces error rate by making fewer (and simpler) claims. This would improve the FactScore metric without representing a genuine improvement in the model's ability to express factual knowledge fluently. The "strict improvement" analysis (Figure 3) shows that FactTune-FS does increase correct facts, which argues against pure claim-suppression, but the qualitative shift in style suggests that some of the error reduction may come from stylistic choices (simpler, more verifiable phrasings) rather than improved factual accuracy per se.

Evidence in the paper. The style changes are documented qualitatively in Section 4.1 and quantified for "conversational tone" via GPT-4 judgment (77.5% and 65.6% less conversational). Appendix Tables 8 and 9 show side-by-side examples where the factuality-tuned generations are noticeably more terse and less narratively structured. However, no evaluation measures whether users prefer the factuality-tuned outputs, find them more trustworthy, or would choose them over the more fluent but less accurate SFT outputs. There is no human evaluation of overall response quality, helpfulness, or readability — only of factual accuracy.

Mitigation status. The paper acknowledges the qualitative differences descriptively but does not treat them as a limitation requiring mitigation. No experiments test different DPO hyperparameters or training strategies to preserve conversational style while improving factuality. No user study or preference evaluation is conducted or proposed as future work.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new architecture, a new pretraining objective, or a new decoding algorithm. Its contribution is a pipeline design pattern: convert an automated truthfulness estimator (whether reference-based or reference-free) into preference pairs, then optimize those preferences with DPO. The shift this causes is not a paradigm revolution but a reframing of hallucination from a detection problem to an optimization problem — and, more subtly, a demonstration that the model's own internal uncertainty signals can serve as a training objective, not just a diagnostic.

The magnitude of this shift should be understood precisely. The paper does not claim to solve hallucination. The error reductions are large in relative terms (roughly 40–70% on biographies) but the resulting models still make factual errors at non-trivial rates: FactTune-FS on Llama-2 biographies still generates 2.00 incorrect facts per biography on average (Table 2). The method reduces error rate but does not approach zero, and it demonstrably fails to help on topics where the base model's knowledge is weak. The contribution is therefore best understood as establishing a new baseline for what can be achieved purely through preference optimization without human labels, not as solving the underlying problem.

What makes this reframing significant for the field is that it resolves a tension in the prior literature between two approaches that seemed to be in opposition:

Reconciling detection-based and training-based approaches. Before this work, the hallucination literature was bifurcated: one strand focused on detecting errors through retrieval, uncertainty estimation, or activation analysis (Kadavath et al., 2022; Min et al., 2023; Azaria & Mitchell, 2023); another strand focused on preventing errors through decoding interventions or prompt engineering (Chuang et al., 2023; Li et al., 2023; Si et al., 2023). These strands rarely intersected — detection methods produced scores that were used to flag or correct outputs post-hoc, while prevention methods operated directly on the model without explicit error detection. The paper bridges this gap by showing that detection signals (whether from retrieval or uncertainty) can be converted into training data for prevention. The detection and prevention literatures are not competing approaches but complementary stages of a unified pipeline.

Resolving the calibration paradox. Prior work had established that LLM confidence correlates with correctness (Kadavath et al., 2022) and that semantic uncertainty can identify likely hallucinations (Kuhn et al., 2023). Yet these findings led only to passive diagnostics — the model "knew what it knew" but could not use that knowledge to avoid errors. The paper shows that this limitation was not fundamental but rather an artifact of how confidence signals were deployed. By converting confidence into a preference ranking, the model can learn to regulate its own generation based on its internal uncertainty — essentially, learning to self-censor claims where its confidence falls below an implicit threshold. The "calibration paradox" (the model knows it's uncertain but says the wrong thing anyway) is therefore not a paradox at all but a gap in the training procedure, which DPO + confidence-based preferences can close.

Reconciling RLHF's "honesty" axis with dedicated factuality optimization. The paper's finding that factuality tuning composes with RLHF (Table 3) establishes that RLHF's existing honesty optimization leaves substantial room for improvement. This was not obvious a priori — one might have expected RLHF to already capture the available factuality signal in human preference data. The fact that a targeted second stage provides large additional gains suggests that factuality is not well-represented in general-purpose human preference judgments, perhaps because human annotators lack the domain expertise to reliably identify subtle factual errors, or because factuality preferences are "crowded out" by other dimensions of quality (fluency, helpfulness, tone) in the aggregate preference signal. This finding implies that factuality should be treated as a separate optimization axis with its own dedicated preference data, rather than being subsumed under general helpfulness/honesty.

The paper also shifts which research directions look more or less attractive:

More attractive after this work:

Automated factuality preference construction as a research area. The paper shows that multiple truthfulness estimators (FactScore, model confidence, named entity resampling) can produce useful preference data, suggesting that improving these estimators — rather than improving the RL algorithm or the base model architecture — is the highest-leverage path to better factuality.
Introspective self-improvement, where the model's own internal signals (confidence, consistency, coherence) serve as training objectives. The FactTune-MC pipeline is a template for this broader class of methods.
Domain-specific factuality tuning using cheaply available domain knowledge. The FactScore method uses Wikipedia, but the same pattern could apply to any domain with a reference corpus — legal documents, medical guidelines, technical manuals.
Modular post-hoc alignment, where targeted preference datasets address specific behavioral failures (factuality, citation accuracy, mathematical reasoning) without disrupting general capabilities.

Less attractive after this work:

Pure decoding-time interventions as a standalone solution for factuality. The paper shows (Figure 3, Table 2) that ITI and DOLA fail to achieve strict improvement — they trade off correct and incorrect facts rather than improving both simultaneously. While they can compose with factuality tuning (Table 4), alone they are insufficient.
Relying solely on human preference labels for factuality. The paper's composability result (Table 3) and the cost analysis from Min et al. ($2,000 for 505 biographies) together suggest that automated factuality preference construction is both cheaper and more targeted than collecting human factuality judgments.
Increasing model scale as the primary path to factuality. While larger models are generally more factual, the paper demonstrates that a 7B model with factuality tuning can substantially close the gap to a larger model's baseline (though no direct scale comparison is provided). This suggests that tuning strategies, not just parameter count, are a critical lever.

Follow-Up Research This Work Enables

1. Factuality tuning on long-tail entities with sparse or conflicting reference knowledge. The paper's experiments focus on well-known individuals and common medical conditions — entities with substantial Wikipedia coverage and (presumably) strong representation in the base model's training data. The most important stress test is: what happens for long-tail entities where Wikipedia has a stub article, or where the base model's parametric knowledge is fragmentary? A concrete experiment would construct a biography dataset stratified by Wikipedia article length (e.g., <500 words, 500–2000 words, >2000 words) and measure FactTune-FS performance as a function of reference coverage. The prediction: factuality tuning should provide the largest gains when Wikipedia coverage is good (the preference signal is reliable) and the base model has non-trivial knowledge (there exist correct candidates to prefer). It should provide diminishing returns as coverage shrinks, eventually reaching zero when the base model never generates correct claims. Measuring the shape of this curve — how quickly gains decay as coverage decreases — would establish the practical applicability boundary of reference-based factuality tuning. For FactTune-MC, the analogous experiment would measure performance against base model knowledge: do confidence-based preferences help when the model is uncertain (high entropy across candidates) versus when it is uniformly wrong (all candidates incorrect, but with varying confidence patterns)? The ablation in Table 5 hints that FactTune-MC is sensitive to domain — it works well on biographies but poorly on medical QA — and understanding which properties of a domain predict FactTune-MC effectiveness is essential before deploying it broadly.

2. Factuality tuning with self-hosted claim extraction and question generation, eliminating the GPT-3.5 dependency. The paper's "reference-free" method depends on GPT-3.5 for two critical steps: atomic claim extraction and claim-to-question conversion. This introduces cost, latency, reproducibility concerns, and a dependency on a proprietary system. A natural follow-up is to replace GPT-3.5 with the base model being fine-tuned (e.g., Llama-2-7B) for both steps, using few-shot prompting with the same prompt templates from Appendix Table 7. The question is whether self-hosted extraction and conversion are sufficiently accurate to produce useful preference pairs. If Llama-2-7B's claim extraction misses important claims or converts claims to ambiguous questions, the confidence scores will be noisy and the preference pairs less informative. A strong experiment would compare FactTune-MC trained with GPT-3.5 extraction vs. Llama-2-7B extraction vs. the named entity baseline from Table 5, on the same datasets, measuring both the quality of the extracted claims (against human annotations of ground-truth atomic facts) and the downstream factuality improvement. If self-hosting works, it makes FactTune-MC fully self-contained and reproducible — a significant practical advance. If it fails, it establishes that the external LLM is load-bearing, which would motivate research into better self-contained claim extraction methods.

3. Scaling factuality tuning to larger models and measuring whether the FactTune-FS vs. FactTune-MC gap changes. The paper uses only 7B models. The central question for scaling is whether FactTune-MC catches up to FactTune-FS as model size increases. The reasoning: larger models are generally better calibrated (their confidence scores more accurately reflect correctness), so the confidence-based preference signal should improve. At some scale, the internal confidence signal might match or exceed the quality of Wikipedia-based verification, making an external knowledge base unnecessary. A concrete experiment would apply the identical FactTune-FS and FactTune-MC pipelines to Llama-2 models at 7B, 13B, and 70B scales, measuring the absolute error rate and the gap between FS and MC at each scale. The hypothesis: the FS–MC gap shrinks with scale because larger models have better internal confidence signals. If true, this would position FactTune-MC as the preferred method for frontier models, where Wikipedia retrieval becomes a bottleneck (retrieval latency, coverage, conflict resolution) and internal signals are strongest. If false — if the gap persists — it would indicate that external knowledge provides a qualitatively different signal that model confidence cannot replicate, regardless of scale.

4. Joint optimization of factuality and other alignment objectives to understand interaction effects. The paper shows that factuality tuning composes with RLHF (Table 3) but does not explore whether factuality optimization trades off against helpfulness, harmlessness, or other alignment dimensions. The qualitative observation that FactTune models are "less conversational" and more "direct" (Section 4.1) raises the possibility that factuality tuning degrades engagement, creativity, or willingness to handle ambiguous queries. A rigorous experiment would construct preference datasets for both factuality and helpfulness (e.g., using human preference judgments from an existing RLHF dataset), then fine-tune with DPO on mixtures of these preferences at varying ratios, measuring the Pareto frontier of factuality vs. helpfulness. The key question is whether there is a tradeoff (improving factuality necessarily reduces helpfulness) or whether the two objectives are largely independent (the model can be both factual and helpful by expressing uncertainty rather than fabricating). The "less conversational" style change suggests some tradeoff exists, but its magnitude and shape are unknown. If the tradeoff is small, factuality tuning can be safely applied to production chat models. If large, deployment decisions become domain-specific: factuality tuning for medical advice, standard RLHF for creative writing.

5. Extending confidence-based truthfulness estimation to domains without clear atomic claim structure. The paper's confidence estimation pipeline assumes that long-form text can be decomposed into discrete, independently verifiable factual claims. This works for biographies and medical QA, where facts are propositional (birth dates, symptoms). In domains like legal analysis, scientific explanation, or narrative writing, facts are often entangled with reasoning, context, or narrative structure that does not cleanly decompose into atomic claims. A follow-up could explore alternative decomposition strategies: extracting only the named entities and their relations (a middle ground between full atomic claims and the simple named entity baseline), using the model's confidence when generating the entire passage conditional on a counterfactual prompt (e.g., "Write a biography of Yo-Yo Ma, but he was born in a different city"), or measuring consistency across multiple independently generated responses to the same prompt as a confidence proxy. The specific experiment: on a dataset of scientific explainers or legal summaries, compare FactTune-MC with standard atomic claim extraction against FactTune-MC with these alternative confidence metrics, using a domain-appropriate truthfulness evaluator (e.g., expert annotations on a small test set) rather than FactScore. The goal is to establish whether the FactTune-MC pattern generalizes beyond propositional fact-checking domains or is fundamentally limited to them.

6. Adversarial evaluation of factuality-tuned models against the FactScore NLI verifier. The paper's external validations (Table 6, Figure 4) show that FactTune-FS improvements are not pure overoptimization — humans and GPT-4 also rate the model as more factual. However, these validations measure aggregate correlation, not worst-case exploitation. A targeted adversarial experiment would identify claims where the FactScore NLI verifier is known to make systematic errors (e.g., consistently classifying claims about specific dates as "supported" when they match Wikipedia's date format even if the year is wrong, or misclassifying negative claims like "X did not win award Y" because Wikipedia doesn't mention the non-event). The experiment would then test whether FactTune-FS generates disproportionately more claims that fall into these systematic error categories compared to the SFT baseline. This would reveal whether DPO has learned to exploit specific verifier blind spots — even if aggregate human ratings improve, the model might still be overoptimizing on certain claim types. This is important because the FactScore verifier (a fine-tuned Llama-1-7B NLI model) is a relatively weak judge compared to GPT-4 or human experts, and adversarial vulnerabilities could be exploited more severely as the DPO training proceeds or as the model scale increases.

Practical Applications and Downstream Use Cases

1. Cost-efficient factuality improvement for domain-specific LLM deployments. Organizations deploying LLMs in high-stakes factual domains — medical information systems, legal research assistants, educational content generators — currently face a choice: use a large, expensive model with some factual guardrails (e.g., retrieval augmentation), or accept higher error rates from smaller, cheaper models. The paper provides a third option: fine-tune a 7B model with automated factuality preferences constructed from domain-specific reference corpora (replacing Wikipedia with the organization's own trusted knowledge base). The concrete benefit is a roughly 40–70% reduction in factual error rate at fixed model size (per Table 2), with the inference cost of a 7B model rather than a 70B model. For a medical advice application where each factual error represents a potential patient harm risk, reducing errors from 5.5 to 3.5 per response (Table 3, Chat vs. FactTune-FS on medical QA) while maintaining the same deployment infrastructure is directly actionable. The training-time cost of preference dataset construction (GPT-3.5 claim extraction, NLI verification, DPO training) is a one-time investment amortized over all future inference queries — making it cost-effective for high-volume deployments.

2. Post-hoc factuality improvement for already-aligned chat models. The composability result (Table 3) means that organizations currently using Llama-2-Chat (or similar RLHF-trained models) can apply factuality tuning as an additional fine-tuning stage without disrupting the model's existing conversational abilities, instruction-following, or safety properties. This is a drop-in improvement: take the deployed chat model, generate candidate responses to a set of factuality-focused prompts (biographies, definitions, explanations of common concepts), score them with FactScore or model confidence, construct preferences, run DPO, and redeploy. The model gains improved factuality on knowledge-intensive queries while retaining its general chat capabilities. The specific benefit: for a customer support chatbot that occasionally answers factual questions (e.g., "What is the return policy?" and "What year was your company founded?"), factuality tuning reduces the rate of confidently incorrect answers without requiring a separate verification system or retrieval pipeline. The 36.7% error reduction on biographies for Chat → FactTune-FS (Table 3) suggests meaningful improvement in real-world factual accuracy with no inference-time overhead.

3. Self-improving data generation pipelines. When LLMs are used to generate training data for other models (e.g., synthetic QA pairs, document summaries, knowledge distillation), the factuality of the generated data directly impacts downstream model quality. The paper's pipeline can be applied to improve the factuality of the generator model before it produces training data, reducing the rate at which factual errors propagate into downstream training sets. The concrete workflow: take a base LLM used for synthetic data generation, construct factuality preferences on a representative sample of the target generation tasks using either an available reference corpus or model confidence, fine-tune with DPO, then use the factuality-tuned model for large-scale data generation. The benefit is cleaner training data without manual verification. The cost is the one-time preference dataset construction, which is small relative to the scale of synthetic data generation pipelines that can produce millions of examples. The 58% error reduction on biographies (Section 6) translates to roughly halving the number of false factual claims in the resulting synthetic dataset — a substantial quality improvement for downstream models trained on that data.

4. On-device or privacy-constrained deployments where retrieval is infeasible. FactTune-MC's reference-free approach is uniquely suited to scenarios where an external knowledge base cannot be accessed: on-device models operating without network connectivity, models deployed in air-gapped environments handling sensitive data, or applications where retrieval latency is unacceptable. In these settings, retrieval-augmented generation and reference-based verification are impossible, leaving purely parametric models as the only option. FactTune-MC provides a way to improve the factuality of such models using only the model's own internal signals during a one-time fine-tuning stage, with no retrieval dependency at either training or inference time (modulo the GPT-3.5 dependency for claim extraction, which occurs only during dataset construction and could be replaced with a self-hosted alternative as discussed in Future Direction 2). The benefit is a reduction in error rate from 56.8% to 78.3% correct on Llama-1 biographies (Table 2) without any external knowledge source — a meaningful accuracy improvement for a purely parametric model. The cost is the more conservative generation style (fewer total facts, Table 2: 10.59 vs. 13.78 correct facts for Llama-1 biographies), which represents a deliberate tradeoff of informativeness for accuracy that may be acceptable or even desirable in high-stakes privacy-sensitive applications.