ArXiv: 2512.20182

🎯 Pitch

FaithLens shows that an 8B-parameter model can beat GPT-5.2 and o3 at hallucination detection—not just labeling claims, but also generating faithful explanations that let users verify why something is wrong. It does this at roughly one percent of the API cost by combining filtered synthetic training data with a rule-based RL reward that asks a weaker model to predict the label from the explanation alone.

1. Executive Summary

This paper introduces FaithLens, a cost-efficient faithfulness hallucination detection model that jointly predicts whether a claim is hallucinated and provides corresponding explanations. Built on an 8B-parameter Llama-3.1-8B-Instruct backbone and evaluated across 12 diverse tasks from LLM-AggreFact and HoVer, FaithLens combines a cold-start supervised fine-tuning stage on synthesized data filtered through three well-defined dimensions—label correctness, explanation quality, and data diversity—followed by a rule-based reinforcement learning stage that optimizes both prediction correctness and explanation quality via a composite reward (using a novice-level model to verify that generated explanations enable correct label prediction). FaithLens achieves state-of-the-art performance, surpassing advanced LLMs such as GPT-5.2 and o3 while operating at a fraction of the inference cost (roughly 0.1versus0.1 versus 8.80–$15.30 for API-based counterparts on 1.2K samples), establishing that a compact specialized model can outperform large general-purpose models when trained on carefully filtered synthetic data with explicit explanation-quality signals—though the method's requirement for a ground-truth-labeled probe set during data diversity filtering means the full training pipeline depends on access to verified labels.

2. Context and Motivation

The Core Problem: Hallucination Detection Needs to Be Both Accurate and Explainable

The fundamental question this paper tackles is deceptively simple: how do we build a practical system that reliably detects faithfulness hallucinations in LLM-generated text AND explains its decisions to users? This matters because LLMs are increasingly deployed in high-stakes settings where outputs must be grounded in provided context—retrieval-augmented generation (RAG), summarization, dialogue systems—and users need more than a binary "faithful / hallucinated" label to trust or act on the system's judgment.

The paper draws a crucial distinction between two types of hallucination (Section 5, Related Work). Factuality hallucinations occur when an LLM's parametric knowledge contradicts real-world facts—for instance, confidently asserting that the capital of France is Madrid. Faithfulness hallucinations occur when the model's output is inconsistent with or unsupported by the given input context, such as a grounding document or retrieved evidence, even if the statement happens to be factually true in the abstract. For example, if a document says "the meeting was held in Paris," but the LLM's summary claims "the meeting was held in Paris on Tuesday," the claim about Tuesday is a faithfulness hallucination—it cannot be verified from the provided document. The paper focuses exclusively on faithfulness hallucination detection.

The practical importance of this distinction is immense. In RAG systems, users expect answers grounded in retrieved documents; a hallucinated claim that sounds plausible but isn't in the context erodes user trust, potentially leading them to reject the entire system. In summarization, faithfulness hallucinations introduce information not present in the source text. The authors cite specific patterns: summarization hallucinations "typically manifest as subtly distorted content from the context" (Section 1), while RAG hallucinations "often ignore the retrieved context and involve conflicting claims." These are qualitatively different error modes, and a detection system must handle both.

Critically, the paper argues that binary predictions alone are insufficient for real-world trust. The introduction (Section 1) identifies "Lack of Explainability" as the first of three key challenges. When a system says "this claim is hallucinated" with no explanation, users cannot localize the error, verify the reasoning, or determine whether to accept or override the judgment. This black-box nature "limits the trustworthiness of detection models." The paper's extension of the task formulation from the standard binary classification (Equation 1) to a joint prediction-and-explanation setting (Equation 2) is not merely a nice-to-have feature—it is presented as a fundamental requirement for deployment.

Why Existing Approaches Fall Short: Three Gaps

The paper identifies three specific limitations in prior work that collectively motivate FaithLens (Section 1):

Gap 1: Lack of Explainability

The dominant paradigm in prior hallucination detection treats the task as pure binary classification. Models are trained to estimate P(y∣doc,claim)P(y \mid \text{doc}, \text{claim}) where y∈{0,1}y \in \{0, 1\}—faithful or hallucinated—and nothing more. This is true of the two main lines of prior work:

API-based LLM approaches (Manakul et al., 2023; Liu et al., 2023c; Lei et al., 2023; Dhuliawala et al., 2024) use carefully designed prompts to query large models like GPT-4o to check for hallucinations. While some of these methods incorporate internal reasoning (e.g., chain-of-thought strategies to improve effectiveness), their primary output is a classification label. The paper acknowledges that these approaches "are inefficient for real-world deployment because they rely on large and advanced models to achieve reliable detection performance" (Section 1). More subtly, even when these models do produce reasoning tokens, these are not structured as user-facing explanations—they are internal computation that may not be coherent, interpretable, or evidence-grounded.

Specialized classifier approaches aim to build cost-efficient models that avoid API dependency. SummaC (Laban et al., 2022) adapts NLI models for document-level faithfulness evaluation but produces only entailment scores. AlignScore (Zha et al., 2023) trains a unified detection model on 4.7M examples from 7 tasks, but outputs only a scalar alignment score. MiniCheck (Tang et al., 2024a) uses synthetic data from Llama-3.1-405B-Inst to train a 7B-parameter classifier—again, producing binary or scored outputs without explanations. FactCG (Lei et al., 2025) improves synthetic data complexity using knowledge graphs but remains a binary classifier. ClearCheck (Seo et al., 2025) comes closest to addressing explainability by training the model to produce chain-of-thought (CoT) reasoning before answering, but as the paper notes, these CoT traces were not designed or optimized as user-facing explanations—they are internal reasoning artifacts. Table 2 shows that ClearCheck's CoT achieves only 77.3 for "Avg" explainability (readability, helpfulness, informativeness averaged) versus FaithLens's 90.4—a substantial gap that illustrates the difference between incidental reasoning traces and explicitly optimized explanations.

The paper explicitly states the consequence: "This makes it difficult for users to localize errors and understand why tested claims are hallucinated, which limits the trustworthiness of detection models" (Section 1).

Gap 2: Inconsistent Generalization Across Tasks

The second gap is that prior methods "are primarily designed for detecting task-specific hallucination and then fail to transfer across different tasks effectively" (Section 1). This is not merely a "performance drops on out-of-domain data" observation—the paper identifies qualitatively different hallucination patterns that emerge across tasks:

  • In summarization (e.g., Agg-CNN, Agg-XSum from Tang et al., 2023), hallucinations typically manifest as "subtly distorted content from the context" (Section 1)—a fact is present in the source document but is altered in the summary.
  • In RAG (e.g., RAGTruth from Wu et al., 2023), hallucinations "often ignore the retrieved context and involve conflicting claims" (Section 1)—the LLM generates information from its parametric knowledge that disagrees with or goes beyond the retrieved evidence.
  • In dialogue summarization (Tofu-MediaS, Tofu-MeetB from Tang et al., 2024b), hallucinations arise from complex multi-turn conversations where speakers' statements must be accurately attributed and condensed.
  • In multi-hop reasoning (HoVer from Jiang et al., 2020), claims require evidence from up to four Wikipedia articles with complex reasoning graphs, making detection a reasoning task rather than a simple matching task.

Even the models "designed for general-purpose scenarios (Tang et al., 2024a; Lei et al., 2025; Seo et al., 2025) still perform unevenly on different tasks" (Section 1). This is visible in Table 1, where models like MiniCheck range from 68.3% F1 (ExpertQA) to 91.0% (RAGTruth)—a 22.7 percentage point spread—while FaithLens achieves both higher average performance (86.4 vs. 80.7) and lower standard deviation (4.6 vs. 7.5), directly addressing this inconsistency.

Gap 3: Lack of High-Quality Training Data with Quality Control

The third gap is arguably the most fundamental from a methodology perspective. "Annotating training data for hallucination detection is costly and often results in low inter-annotator agreement" (Section 1, citing Seo et al., 2025). This has pushed the field toward synthetic data generation, but the paper argues that existing synthetic data approaches lack "well-defined data quality control strategies," which can result in:

  • Incorrect labels propagating through training—the synthetic data generator may produce wrong predictions, especially on ambiguous or borderline cases, and the resulting training data would teach the model to replicate those mistakes.
  • Low-quality explanations or irrelevant reasoning traces that, while superficially coherent, do not actually provide useful signal for learning to detect hallucinations.
  • Skewed difficulty distributions where simpler instances survive filtering preferentially, "ultimately limiting the model's abilities in complex detection scenarios" (Section 1).

The paper's data filtering strategy (Section 3.1.2) is a direct response to this gap—each of the three filtering dimensions targets a specific failure mode of naïve synthetic data generation.

The Tension Across Prior Work: Large Models vs. Efficient Models

The paper's positioning reflects a deeper tension in the hallucination detection literature. On one side, large API-based models like GPT-4o, o1, GPT-4.1, and o3 achieve strong performance but are prohibitively expensive for large-scale deployment (Table 3 shows inference costs ranging from 7.30to7.30 to 15.30 for GPT-4o through GPT-5.2 on 1.2K samples, versus $0.10 for FaithLens). They are also slow and require network access, making them unsuitable for real-time or privacy-sensitive applications.

On the other side, specialized detection models like MiniCheck, FactCG, and ClearCheck are cost-efficient but historically underperform larger models, especially on complex reasoning tasks like HoVer (where MiniCheck achieves 74.9% vs. GPT-5.2's 82.9%). The paper notes that Seo et al. (2025) "found that a small fine-tuned model underperforms larger models by a huge margin, particularly for instances requiring complex reasoning" (Appendix B). This establishes a clear performance gap that FaithLens aims to close—not just matching but exceeding large models while maintaining the cost efficiency of specialized classifiers.

The paper also addresses a subtle methodological issue in prior benchmarks: Seo et al. (2025) identified that the original LLM-AggreFact and HoVer benchmarks contain 9.1% ambiguous examples and 6.6% mislabeled instances. This means prior reported results on these benchmarks may overstate true model performance, and the FaithLens evaluation uses a cleaned, de-noised version of the benchmark, making comparisons more reliable and increasing the difficulty (since models can no longer benefit from label noise that happens to align with their predictions).

How FaithLens Positions Itself

FaithLens positions itself at the intersection of three previously separate research directions:

From the specialized classifier tradition, it inherits the goal of building a compact, cost-efficient model that can be deployed locally without API access. But it rejects the premise that such models must trade explainability for efficiency.

From the chain-of-thought reasoning literature (Wei et al., 2024a; Jacovi et al., 2024), it draws on the idea that explicit reasoning steps improve prediction accuracy. But it goes further by optimizing those reasoning steps to serve as user-facing explanations, not merely internal computation. The architecture generates CoT first, then a structured explanation, then the prediction—a deliberate separation that allows the explanation quality to be independently evaluated and optimized.

From the RL-based LLM optimization literature (Schulman et al., 2017; Shao et al., 2024; Kimi-Team et al., 2025), it adopts GRPO for policy optimization with rule-based rewards. But rather than using a single correctness reward (which is standard for reasoning tasks), it introduces the novel explanation quality reward (Section 3.2.2) that evaluates whether a generated explanation enables a novice-level model (Llama-3.1-8B-Instruct) to predict the correct label. This is a clever operationalization of "explanation quality": an explanation is good if it transfers knowledge—if it enables a weaker model that couldn't originally solve the task to do so after reading the explanation. This is conceptually related to the Feynman technique (if you can't explain it simply, you don't understand it well enough) but operationalized as a machine-learning reward signal.

The unifying insight is that prediction correctness and explanation quality are complementary objectives that can be jointly optimized through a composite reward. The prediction correctness reward (Equation 13) ensures the model remains accurate. The explanation quality reward (Equation 14) ensures the model's reasoning is transferable. Together, they produce a model that is simultaneously more accurate than specialized classifiers and more trustworthy than API-based LLMs—a distinctive combination the paper frames as its central contribution.

Where FaithLens Differs from ClearCheck

Since ClearCheck (Seo et al., 2025) is the closest prior work in concept (it also uses CoT reasoning in a fine-tuned 8B model), understanding the differences is important for situating FaithLens. ClearCheck uses 57K ANLI examples and 25K private synthetic multi-hop data, with CoT traces distilled from Llama-3.1-405B-Inst. It treats the CoT as an internal reasoning scaffold, not as an explicitly optimized explanation. FaithLens differs in several key ways:

  • Data source: FaithLens uses only open-source data (the same ANLI, C2D, D2C sets from FactCG/Lei et al., 2025), not private data, meaning its entire training pipeline is reproducible (Table 4).
  • Data filtering: FaithLens applies a three-dimensional filtering strategy that explicitly checks label correctness, explanation quality, and data diversity—none of which are present in ClearCheck's pipeline.
  • Optimization objective: ClearCheck is trained via multi-task SFT only. FaithLens adds an RL stage with explicit rewards for both prediction and explanation quality, which provides a training signal that cannot be captured by behavior cloning on synthetic CoT traces.
  • Explanation evaluation: The paper evaluates FaithLens's explanations on three quality dimensions (readability, helpfulness, informativeness) and shows they substantially outperform ClearCheck's CoT traces (90.4 vs. 77.3 on average, Table 2).

3. Technical Approach

3.1 Reader Orientation

FaithLens is a fine-tuned language model (an 8B-parameter Llama-3.1-8B-Instruct, further trained via SFT and RL) that takes a grounding document and a claim as input and produces three things in sequence: a chain-of-thought reasoning trace, a human-readable explanation, and a binary prediction of whether the claim is faithful or hallucinated. The system solves the problem of explainable faithfulness hallucination detection — given only a document and a claim, it must not only decide if the claim is supported by the document but also tell the user why, specifically enough that another (weaker) model could make the correct decision after reading the explanation.

3.2 Big-Picture Architecture (Diagram in Words)

The FaithLens system is built in two stages, each producing a progressively stronger model from the same backbone (Llama-3.1-8B-Instruct):

  1. Data Synthesis & Filtering Pipeline: An advanced large reasoning model (DeepSeek-V3.2-Think) is prompted with (document, claim) pairs from existing open-source training data. It generates structured outputs containing a chain-of-thought (CoT), an explanation, and a predicted label. These synthetic outputs then pass through three sequential quality filters — label correctness, explanation quality, and data diversity — that discard low-quality examples. The surviving examples form a curated training set.

  2. Cold-Start Supervised Fine-Tuning (SFT): The base Llama-3.1-8B-Instruct model is fine-tuned on the filtered synthetic data to learn the joint task of generating CoT, explanation, and prediction from (document, claim) pairs. This produces the SFT-initialized model, which can already detect hallucinations and produce explanations, but may be suboptimal — it has only imitated the synthetic data, not optimized for genuine explanation quality or robustness.

  3. Rule-Based Reinforcement Learning (RL) Training: The SFT-initialized model is further optimized using the GRPO (Group Relative Policy Optimization) algorithm with a composite rule-based reward. For each training instance, the model generates a group of candidate (explanation, prediction) pairs. Each candidate receives three rewards: (a) a prediction correctness reward (1 if the binary label matches ground truth, else 0), (b) an explanation quality reward (1 if feeding the generated explanation to a novice-level Llama-3.1-8B-Instruct enables it to predict the correct label, else 0), and (c) a format reward (1 if the output follows the required XML tag structure, else 0). The sum of these three rewards drives GRPO to produce a final model that is simultaneously accurate, explainable, and well-formatted.

Information flows through the trained model at inference time as: (document, claim) → internal CoT generation → explanation generation → binary prediction, with the CoT providing the reasoning scaffold that precedes and supports the user-facing explanation.

3.3 Roadmap for the Deep Dive

  • The training data synthesis procedure (Section 3.1.1): How FaithLens generates the initial synthetic examples with CoT, explanation, and prediction using DeepSeek-V3.2-Think, creating training data for a task that prior benchmarks did not label.

  • The three-dimensional data filtering strategy (Section 3.1.2): The core quality-control mechanism. I will walk through each filter — label correctness (discarding mislabeled examples), explanation quality (measuring whether an explanation helps the same base model become more confident on the correct answer via perplexity reduction), and data diversity (using a K-Medoids clustering and probe-set approach to retain examples that help diverse data types) — explaining exactly how each is computed and why each targets a specific failure mode of naive synthetic data generation.

  • The cold-start SFT objective (end of Section 3.1.2): How the filtered data is used for initial supervised training and what the model learns at this stage.

  • The RL protocol and reward design (Section 3.2): How GRPO works in this context (group-based advantage estimation, KL regularization to the SFT-initialized reference model) and how the composite reward — prediction correctness plus explanation quality plus format — is designed. This is the most technically novel part of FaithLens, so I will detail the operational definition of the explanation quality reward (having a novice-level model re-predict from the explanation), the format reward, and how these signals interact.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a model-building and training methodology paper whose core idea is that a compact detection model can be trained to produce high-quality explanations — not by expensive human annotation — but by a combination of (a) synthetic data with multi-dimensional filtering to ensure label correctness, explanation quality, and data diversity, and (b) reinforcement learning with a composite reward that explicitly optimizes for both prediction accuracy and explanation transferability (as measured by whether a weaker model can correctly predict the label after reading the explanation).

Data Synthesis via Large Reasoning Models

The training pipeline begins with a problem: existing open-source hallucination detection datasets (the ANLI subset, C2D, and D2C sets from Lei et al., 2025, totaling 52,268 examples, as shown in Table 8) provide only (document, claim, ground-truth label) triples. They contain no explanations, no chain-of-thought reasoning, and no structured output that teaches a model why a claim is faithful or hallucinated. To bridge this gap, FaithLens uses an advanced large reasoning model (LRM), specifically DeepSeek-V3.2-Think, to synthesize these missing components.

The procedure is straightforward but deliberately chosen. For each training example (document doc, claim c), the authors query DeepSeek-V3.2-Think with a prompt (shown in Figure 5 of the Appendix) that instructs the model to:

  • First, "think step by step about whether all the information in the claim is fully supported by the document within <thinking> and </thinking> tags"
  • Then, "provide an easy-to-understand explanation for your answer within <reason> and </reason> tags"
  • Finally, "assess the claim's consistency with the document by responding with either 'Yes' or 'No' and wrap your final answer in <answer> and </answer> tags"

The model autoregressively generates three components in sequence: a chain-of-thought reasoning trace CoT̂, a human-readable explanation ê, and a predicted label ŷ. The temperature is set to 1.0. This produces a synthetic sample ŝ = (doc, c, CoT̂, ê, ŷ) for each original training pair.

The choice of DeepSeek-V3.2-Think over other advanced LRMs (such as o3 or GPT-5.2) is motivated by a practical constraint: access to chain-of-thought content. The paper explicitly states that "these models do not allow us to access the CoT content" (Section 4.1, Implementation Details). Since FaithLens needs the CoT as training signal for the SFT stage — the model must learn to produce reasoning before producing the explanation — only models that expose their internal reasoning traces are usable for synthesis. DeepSeek-V3.2-Think provides this access, making it the natural choice.

The resulting 52,268 synthetic samples form the raw material for the filtering stage. Table 8 shows that the initial full data consists of 52,268 examples, of which 35,554 are designated as "Initial SFT Data" (the ANLI subset, C2D, and D2C sets) and 16,714 as "Initial Data For RL" (the CG2C-MHQA and CG2C-Doc sets from Lei et al., 2025). The RL data is not filtered — the authors note that "the RL data we selected consists of verified, high-quality, and more challenging samples by Lei et al. (2025)" (Appendix C).

Data Filtering: Three Dimensions of Quality Control

The synthetic data from DeepSeek-V3.2-Think, while produced by a powerful model, is not assumed to be clean. "Even if we apply well-designed prompts, the synthesized data without quality control could still be noisy or useless" (Section 3.1.2). The paper identifies three failure modes that the filtering strategy must address:

  • Incorrect labels: The LLM may produce a wrong prediction ŷ that contradicts the ground-truth label y_gt. If such examples are included in training, "the related CoT and explanation may appear coherent, but they are internally aligned with an incorrect prediction. Including these samples would cause the model to learn incorrect patterns, which would reduce its detection effectiveness and explanation quality" (Section 3.1.2).
  • Low-quality explanations: Even when the label is correct, the explanation may be uninformative, misleading, or insufficiently grounded in the document. An explanation that merely restates the conclusion without citing specific evidence provides no useful learning signal for producing genuine explanations.
  • Skewed data distribution: Filtering for label correctness and explanation quality tends to retain easier examples that the synthetic LLM handles well. This "may also lead to distribution bias, where the retained data focus on specific tasks and hallucination patterns, ultimately limiting the model's cross-task generalization" (Section 3.1.2).

The paper applies three sequential filters to the SFT data (35,554 initial examples), each addressing one failure mode. Table 8 shows the progression: 14,258 survive label correctness filtering, 4,363 survive explanation quality filtering, 5,004 survive data diversity filtering, and after combining with additional data, 11,929 form the final SFT set.

Label Correctness Filtering

This is the simplest and most direct filter. For each synthetic sample ŝ, the predicted label ŷ from DeepSeek-V3.2-Think is compared against the ground-truth label y_gt provided in the original dataset:

Flabel(s^)=1{y^=ygt}F_{label}(\hat{s}) = \mathbf{1}\{\hat{y} = y_{gt}\}

where F_label(ŝ) is 1 if the sample is retained and 0 if discarded, and 1{·} is the indicator function.

What it computes: a binary keep/discard decision based on exact label match. The sample with its CoT, explanation, and prediction is retained only if the LLM's predicted label matches the known ground truth.

Why this form: exact match is the only criterion that guarantees the explanation direction is consistent with the truth. If ŷ ≠ y_gt, the generated CoT and explanation are reasoning toward the wrong conclusion — they will, by construction, contain logical steps and evidence citations that support an incorrect answer. Training on such examples would teach the model to produce convincing but wrong explanations, which is worse than producing no explanation at all. Using a soft threshold or probability-based filtering would risk retaining examples where the LLM is confidently wrong, undermining the training signal.

This filter removes the largest fraction of data: only 14,258 out of 35,554 examples survive (approximately 40.1%). This high discard rate reflects the difficulty of the hallucination detection task even for advanced LLMs — DeepSeek-V3.2-Think makes incorrect predictions on roughly 60% of the training examples under the standard prompting setup.

Explanation Quality Filtering

After ensuring label correctness, the paper addresses explanation quality. The core insight is that a good explanation should make the correct answer more obvious: it should provide evidence or reasoning that increases a reader's confidence in the correct label. The operationalization measures whether the explanation reduces the model's perplexity on the ground-truth label.

Specifically, the authors use the model M that will subsequently be fine-tuned (Llama-3.1-8B-Instruct) as the evaluator. First, they measure the model's perplexity for the ground-truth label y_gt when conditioned only on the document doc, claim c, and the synthesized CoT CoT̂ (without the explanation):

PPLw/o. exp=PPLM(ygt∣doc,c,CoT^)PPL_{w/o.\ exp} = PPL_M(y_{gt} \mid doc, c, \widehat{CoT})

where PPL_M(·|context) is the perplexity of generating token y_gt given the specified context under model M. Perplexity is the exponentiated negative log-likelihood: lower perplexity means the model assigns higher probability to the correct answer — it is more confident that the correct label should follow from the context.

Then, they include the synthesized explanation ê as additional context and recompute the perplexity:

PPLw. exp=PPLM(ygt∣doc,c,CoT^,e^)PPL_{w.\ exp} = PPL_M(y_{gt} \mid doc, c, \widehat{CoT}, \hat{e})

The sample is retained only if adding the explanation reduces perplexity — that is, if the explanation makes the model more confident in the correct label:

Fexp(s^)=1{PPLw. exp<PPLw/o. exp}F_{exp}(\hat{s}) = \mathbf{1}\{PPL_{w.\ exp} < PPL_{w/o.\ exp}\}

where F_exp(ŝ) is 1 if retained and 1{·} is the indicator function.

What it computes: a binary keep/discard decision based on whether appending the synthetic explanation to the context lowers the base model's perplexity on the ground-truth label. It measures the marginal informativeness of the explanation — does the explanation add information beyond what the CoT already provides that makes the correct answer more predictable?

Why this form: perplexity reduction is a natural proxy for explanation usefulness because it directly measures whether the explanation increases the model's confidence in the correct answer. An explanation that is irrelevant, confusing, or contradictory would not reduce perplexity — it might even increase it by introducing distracting information. Using perplexity rather than requiring the model to generate a new prediction (which would introduce sampling variance and increase computational cost) provides a cheap, deterministic signal. The comparison is specifically against the CoT-only context (not against a no-explanation baseline) because the CoT already contains reasoning — the question is whether the explanation adds value on top of the reasoning already present.

There is a subtle but important design choice here: the evaluation uses the base Llama-3.1-8B-Instruct model, not the already-fine-tuned detection model. This means the filtering evaluates how well the explanation transfers knowledge to a naive model — the same principle used later in the RL explanation quality reward (Section 3.2.2). The consistency between the SFT filtering criterion and the RL reward signal is not accidental; both operationalize "good explanation" as "enables correct prediction by a model that doesn't already know the answer."

This filter is highly selective: only 4,363 out of 14,258 label-correct samples survive (approximately 30.6%). The authors note in Table 8 that 5,004 samples are subsequently added from some other source (the paper's Table 8 shows "Filtered Data from Data Diversity Filtering: 5,004" and "Final Data For SFT: 11,929," suggesting additional unfiltered data — likely the ANLI subset — is added after filtering to reach 11,929). The high discard rate from explanation quality filtering suggests that even when DeepSeek-V3.2-Think predicts correctly, its explanations are often not sufficiently informative to move the base model's probability distribution.

This is a significant empirical finding in itself: producing a correct answer and producing a useful explanation are distinct capabilities, and a model can do the former without the latter. The filtering strategy explicitly separates these two aspects of quality and enforces both.

Data Diversity Filtering

The final filter addresses distributional skew. After label correctness and explanation quality filtering, the surviving data will be biased toward examples where the synthetic LLM both predicted correctly and produced informative explanations — which likely correlates with easier examples. Training exclusively on such examples would produce a model that performs well on easy cases but struggles with complex, ambiguous, or atypical hallucinations.

The diversity filter uses a clustering-based probe set approach. The process has four steps:

Step 1 — Embedding and Clustering. For each (document, claim) pair in the candidate pool, the authors compute a dense vector representation using a sentence embedding model — specifically, Llama-Embed-Nemotron-8B (Babakhin et al., 2025), which is itself based on Llama-3.1-8B and maps text to a fixed-dimensional embedding space. They then apply the K-Medoids algorithm (Park and Jun, 2009) with K=10 clusters and cosine similarity as the distance metric. K-Medoids selects K actual data points as cluster centers (medoids), unlike K-Means which uses centroids (potentially artificial points in embedding space). The medoids are the most centrally located samples in each cluster.

Step 2 — Probe Set Construction. The K medoids form a probe set S_p = {ŝ'_1, ..., ŝ'_K}. These are K=10 representative samples, each from a different region of the data distribution as defined by the embedding space.

Step 3 — Perplexity Evaluation on Probe Set. For each probe sample ŝ'_i = (doc'_i, c'_i, CoT̂'_i, ê'_i, ŷ'_i), the model M (Llama-3.1-8B-Instruct) computes the perplexity of the ground-truth label given the probe's own context:

PPL(s^i′)=PPLM(y^i′∣doci′,ci′,CoT^i′,e^i′)PPL(\hat{s}'_i) = PPL_M(\hat{y}'_i \mid doc'_i, c'_i, \widehat{CoT}'_i, \hat{e}'_i)

This establishes a baseline: how confident the model is on each probe sample using only that sample's own context.

Then, for each candidate sample ŝ under evaluation, the authors recompute the perplexity on each probe sample with the candidate sample prepended as an in-context demonstration:

PPL(s^i′∣s^)=PPLM(y^i′∣s^,doci′,ci′,CoT^i′,e^i′)PPL(\hat{s}'_i \mid \hat{s}) = PPL_M(\hat{y}'_i \mid \hat{s}, doc'_i, c'_i, \widehat{CoT}'_i, \hat{e}'_i)

If PPL(ŝ'_i | ŝ) < PPL(ŝ'_i), it means the candidate sample ŝ provides useful information that helps the model better predict the correct label for probe sample ŝ'_i — the candidate sample's (document, claim, CoT, explanation, label) combination is complementary to the probe sample.

Step 4 — Retention Criterion. The candidate sample ŝ is retained if it improves perplexity on at least half of the probe set:

Fdiv(s^)=1{∣{s^i′∈Sp∣PPL(s^i′∣s^)<PPL(s^i′)}∣≥K2}F_{div}(\hat{s}) = \mathbf{1}\left\{ \left|\{\hat{s}'_i \in S_p \mid PPL(\hat{s}'_i \mid \hat{s}) < PPL(\hat{s}'_i)\}\right| \geq \frac{K}{2} \right\}

where F_div(ŝ) is 1 if retained and |·| denotes set cardinality.

What it computes: a binary keep/discard decision based on whether the candidate sample, used as an in-context demonstration, improves the model's confidence on at least 5 out of 10 diverse probe samples. Samples that help many different types of probe samples — that are broadly useful rather than narrowly specific — are retained. Samples that only help a few probe samples (or none) are likely to be redundant with existing data or too idiosyncratic to generalize.

Why this form: the probe-set approach directly operationalizes "diversity" as complementary informativeness across the data distribution. A sample is considered diverse not because it looks different (that would be a surface-level diversity measure) but because it provides new information that helps with different types of examples. The threshold of K/2 requires the sample to be useful for a majority of the probe set — a sample that only helps one or two clusters is too specialized. The use of K-Medoids (actual data points as centers) ensures the probe set consists of real, representative examples rather than artificial centroids that might not correspond to any actual hallucination pattern.

The choice of K=10 is the only hyperparameter introduced by the filtering process. The authors test K=6, K=14, and K=20 in the parameter study (Appendix I, Tables 11–12) and show that performance is stable across these values, with K=10 giving the best results (86.4 average F1 vs. 85.6–86.1 for the other values). This robustness suggests the method is not highly sensitive to the exact number of clusters, provided K is large enough to capture diversity but small enough that each cluster has sufficient samples.

Relationship to the other filters: the three filters are applied sequentially — label correctness first (cheapest, eliminates the most data), then explanation quality, then data diversity. This ordering is computationally efficient because label correctness filtering removes about 60% of data before the more expensive perplexity computations for explanation quality and data diversity are performed.

Cold-Start Supervised Fine-Tuning

After filtering, the surviving 11,929 samples form the SFT training set D. The base model (Llama-3.1-8B-Instruct) is fine-tuned on these examples to learn the structured output format: generate CoT, then explanation, then prediction, conditioned on (document, claim).

The SFT objective is the standard next-token prediction loss:

LSFT=−Es^∼D[log⁡M(CoT^,e^,ygt∣doc,c)]\mathcal{L}_{SFT} = -\mathbb{E}_{\hat{s} \sim D} \left[ \log M(\widehat{CoT}, \hat{e}, y_{gt} \mid doc, c) \right]

where D is the filtered training set, M is the model being trained, (doc, c) is the input (document and claim), and (CoT̂, ê, y_gt) is the target sequence. The model is trained to maximize the log-probability of the target tokens given the input.

What it computes: the negative log-likelihood of the ground-truth sequence (CoT + explanation + correct label) under the model's predicted distribution, averaged over the training set. The expectation is approximated by a finite sum over mini-batches.

Why this form: this is standard causal language model fine-tuning. The model learns by imitation — it sees examples where a reasoning model (DeepSeek-V3.2-Think) produced CoT and explanations leading to correct labels, and it learns to reproduce similar patterns. The loss is computed over the entire target sequence (CoT, explanation, and label), which means the model learns all three components jointly. The CoT provides a reasoning scaffold that precedes the explanation; training on the full sequence teaches the model to produce reasoning before the user-facing explanation, rather than generating explanation text directly.

Training hyperparameters (Appendix C): the Adam optimizer (Kingma and Ba, 2017) with learning rate 1 × 10⁻⁵, weight decay 0.1, batch size 16, trained for 3 epochs. Distributed training uses DeepSpeed with ZeRO-3 optimization and BF16 mixed precision. These are standard choices for efficient 8B-parameter model fine-tuning; the learning rate is typical for SFT, and ZeRO-3 sharding is necessary to fit the model and optimizer states across GPUs.

The output of this stage is the SFT-initialized model, which can already perform the detection-with-explanation task. However, as the ablation study (Table 5) shows, this model achieves only 82.6 average F1 and 83.8 average explainability — substantially below the full FaithLens (86.4 and 90.4, respectively). The SFT model "can easily memorize the simple training samples and struggles to generalize to complex detection tasks. Also, the model may generate correct explanations but often lacks clarity or informativeness, as it is trained to imitate training data rather than explicitly optimize for explanation quality" (Section 3.2).

This gap motivates the RL stage: SFT alone produces a model that can explain, but the explanations are not explicitly optimized for quality. The imitation learning objective optimizes for similarity to the training distribution, not for downstream usefulness.

Reinforcement Learning Training

The RL stage treats the hallucination detection task as a policy optimization problem. The model M_ours (a policy that generates CoT, explanation, and prediction given a state consisting of document and claim) is optimized to maximize a composite reward that captures both accuracy and explainability.

Reinforcement Learning Protocol: GRPO

The paper uses Group Relative Policy Optimization (GRPO) (Shao et al., 2024), a variant of policy gradient methods that eliminates the need for a separate reward model (a critical advantage, since training a reward model for explanation quality would require labeled data that doesn't exist). GRPO works by sampling multiple outputs from the current policy, computing their relative performance within the group, and updating the policy to favor outputs that outperform the group average.

For each document-claim pair (doc, c), the model generates a group of G=7 candidate outputs. Each candidate consists of an explanation eᵢ and a prediction pᵢ, both conditioned on the same input. (The CoT is generated as part of the model's internal computation but is not separately rewarded; the reward focuses on the final output components.) Each candidate receives a composite reward R_final = R_pred + R_exp + R_format (detailed in the next section).

GRPO computes an advantage Aᵢ for each candidate i based on its relative reward within the group. The exact advantage estimation formula within GRPO normalizes rewards within the group, such that outputs with above-average rewards receive positive advantages and outputs with below-average rewards receive negative advantages. The policy is then updated using a clipped surrogate objective:

LGRPO(Mours)=E(doc,c),{ei,pi}∼Mold[1G∑i=1GLi−βDKL(Mours∣∣Mref)]\mathcal{L}_{GRPO}(M_{ours}) = \mathbb{E}_{(doc,c), \{e_i, p_i\} \sim M_{old}} \left[ \frac{1}{G} \sum_{i=1}^G L_i - \beta D_{KL}(M_{ours} || M_{ref}) \right]

where the individual term is:

Li=min⁡(wiAi,clip(wi,1−ϵ,1+ϵ)Ai)L_i = \min \left( w_i A_i, \text{clip}(w_i, 1 - \epsilon, 1 + \epsilon) A_i \right)

Key components:

  • M_ours is the current policy (the model being optimized)
  • M_old is the policy at the start of the update step (used for sampling candidates)
  • M_ref is the reference policy — the SFT-initialized model, frozen during RL training. This provides a regularization target.
  • w_i = M_ours(e_i, p_i | doc, c) / M_old(e_i, p_i | doc, c) is the importance sampling ratio — the ratio of probabilities of output i under the new and old policies. This corrects for the fact that candidates were sampled from the old policy but evaluated under the new one.
  • A_i is the group-relative advantage of candidate i
  • \epsilon = 0.2 is the clipping parameter — it limits how much the policy can change in a single update by capping the importance ratio at [0.8, 1.2], preventing destructive large updates from high-variance advantage estimates
  • \beta = 0.001 is the KL divergence coefficient — it penalizes the policy for diverging too far from the reference (SFT-initialized) model, preventing catastrophic forgetting of the structured output format and base language capabilities
  • D_KL(M_ours || M_ref) is the Kullback-Leibler divergence between the current and reference policy distributions

What it computes: the expected clipped surrogate objective plus a KL penalty term. The first part (L_i) is the standard PPO-style clipped objective that encourages the policy to increase probability for outputs with positive advantages while limiting the magnitude of each update. The second part (-β D_KL) keeps the policy close to the SFT-initialized reference, which is crucial for maintaining the structured output format (CoT → explanation → prediction) that the SFT stage established.

Why this form: GRPO eliminates the need for a separate value function (critic) model by using group-relative advantages. This reduces memory requirements (no additional model parameters) and simplifies training (no need to train a value function that accurately predicts expected returns). The clipped objective with KL regularization is standard in RL fine-tuning of LLMs and provides stable training dynamics. The choice of G=7 (group size) balances diversity of sampled outputs with computational cost — 7 rollouts per training instance provides enough candidates for meaningful relative comparison without excessive GPU memory.

RL training hyperparameters (Appendix C): learning rate 1 × 10⁻⁶ for the actor (lower than SFT, as is standard for fine-tuning a pre-trained policy), rollout temperature 0.6 (identical to the evaluation temperature, ensuring training and inference distributions match), mini-batch size 16 across 7 GPUs (total batch size 112), trained for 2 epochs, β=0.001, \epsilon=0.2, and a gamma-decay factor α=0.2 (used in the advantage computation to control temporal credit assignment across the sequence). The training is conducted on NVIDIA A800 SXM4 80GB GPUs.

Reward Design: The Composite Reward

The RL training signal comes from a rule-based reward function — no learned reward model is used. The composite reward for each candidate output is the sum of three independent components:

Rfinal=Rpred+Rexp+RformatR_{final} = R_{pred} + R_{exp} + R_{format}

Each component is evaluated deterministically from the model output and ground-truth annotations, with no human involvement.

Prediction Correctness Reward

This is the simplest reward: it checks whether the model's predicted label matches the ground truth.

Rpred={1,if ypred=ygt0,otherwiseR_{pred} = \begin{cases} 1, & \text{if } y_{pred} = y_{gt} \\ 0, & \text{otherwise} \end{cases}

where y_pred is the binary prediction extracted from the model's <answer> tag and y_gt is the ground-truth label from the training data.

What it computes: a binary 0/1 reward for exact prediction accuracy. The model receives full credit (1) if correct, no credit (0) if incorrect.

Why this form: a simple binary reward provides the strongest possible signal for the core task: the model should be accurate above all else. Using a continuous score (e.g., log-probability of the correct token) would be more nuanced but could reward confident wrong answers — a model that assigns 0.9 probability to the wrong label would receive a higher continuous score than one that assigns 0.3 to the correct label, which is counterproductive. The binary reward forces the model to put probability mass on the correct answer to receive any credit.

This reward alone would optimize for accuracy but could lead to degenerate strategies — for instance, the model might learn to always predict "hallucinated" (the majority class in many datasets), which would achieve moderate accuracy but produce useless explanations. The explanation quality reward is designed to prevent this.

Explanation Quality Reward

This is the paper's most novel technical contribution to the reward design. Directly evaluating the quality of free-form explanation text via rule-based verification is fundamentally challenging — there is no simple regex or pattern that distinguishes a good explanation from a bad one. The paper's solution is indirect evaluation through transfer learning: a generated explanation is considered high-quality if it enables a separate, weaker model to correctly predict the label.

Formally, the reward is:

Rexp={1,if ypredMnov(doc,c,e)=ygt0,otherwiseR_{exp} = \begin{cases} 1, & \text{if } y_{pred}^{M_{nov}}(doc, c, e) = y_{gt} \\ 0, & \text{otherwise} \end{cases}

where e is the generated explanation from the current policy, and y_pred^{M_nov}(doc, c, e) is the binary prediction produced by a novice-level model M_nov when conditioned on the document doc, the claim c, and the generated explanation e. The novice-level model is Llama-3.1-8B-Instruct — the same base model used for SFT initialization, but kept frozen and untuned during RL. Its temperature is set to 0.6 during reward computation (matching the rollout temperature).

The prompt used for the novice model during reward computation is shown in Figure 8 of the Appendix. It provides the document, claim, and explanation, and asks the model to produce an <answer> tag with either "Yes" or "No". No additional context or reasoning instructions are given — the novice model must use only the provided explanation to make its prediction.

What it computes: a binary 0/1 reward for explanation transferability. The generated explanation e is presented to a frozen, untuned model that has not been trained on hallucination detection. If, conditioned on e, this novice model correctly predicts the label, the explanation is considered sufficiently coherent and informative to convey the relevant evidence — it transfers the knowledge needed to make the correct decision. If the novice model still gets it wrong, the explanation is insufficient.

Why this form: this operationalization has several desirable properties:

  • It measures explanation quality without a learned reward model. Training a reward model would require human-labeled explanation quality scores, which are expensive, inconsistent (different annotators disagree on what makes a good explanation), and scarce. Using a frozen base model as a zero-shot evaluator eliminates this dependency.
  • It directly tests for knowledge transfer. A good explanation should help someone who doesn't already know the answer to figure it out. The novice model serves as a proxy for a human reader who has read the document and claim but is uncertain about the judgment. If the explanation resolves that uncertainty, it is useful.
  • It prevents reward hacking via vacuous explanations. If the model tried to game the reward by producing explanations like "the answer is no because the claim is not supported" (circular reasoning), the novice model would not gain useful information and would fail to predict correctly — receiving R_exp = 0. Only explanations that contain genuine, evidence-grounded reasoning about the specific document-claim pair can improve the novice model's predictions.
  • It naturally rewards clarity and specificity. Vague explanations ("some parts are not supported") provide less signal than specific ones ("the document states the meeting was in Paris, but the claim incorrectly adds that it was on Tuesday, which is not mentioned"). The novice model will benefit more from the latter.
  • It aligns with the SFT data filtering criterion. The explanation quality filter during SFT also measured whether an explanation helps the base model (via perplexity reduction). The RL reward extends this to a more demanding test: actual prediction correctness rather than mere perplexity reduction, and at inference time (generating a full prediction) rather than scoring. This creates a consistent but escalating standard of explanation quality across training stages.

A subtler design choice concerns the novice-level model selection. The paper tests using different models as M_nov (Appendix I, Study 3, Tables 11–12). Using Qwen-2.5-7B-Inst (a heterologous model, from a different model family) as the novice model reduces performance (85.3 average F1, 88.4 explainability) compared to using Llama-3.1-8B-Inst (86.4, 90.4). The authors hypothesize that "this may be due to different pre-training data, language styles, or sensitivity to instruction formats, which can result in a particular model being unable to correctly predict labels based on the provided explanation." Using a homologous novice model (same model family as the policy model) avoids measurement noise from format incompatibility.

More dramatically, using an expert-level model (DeepSeek-V3.2-Think) as the novice model produces worse results (85.9 F1, 86.5 explainability). The interpretation: "the expert model can ignore the incorrect explanations provided and still predict the correct label. As a result, low-quality explanations are assigned high rewards, which in turn weakens the quality of the explanations generated by the policy model." This is a critical insight — the novice model must be weak enough that it genuinely depends on the explanation, not strong enough that it can succeed despite a bad explanation. The reward signal is only informative when M_nov's predictions are sensitive to explanation quality.

Format Reward

To ensure the model outputs are parseable and follow the expected structure, a format reward checks for the presence of the required XML tags:

Rformat={1,if correct formatting0,if incorrect formattingR_{format} = \begin{cases} 1, & \text{if correct formatting} \\ 0, & \text{if incorrect formatting} \end{cases}

The "correct formatting" criterion is that the entire generated response contains the three required XML tag pairs in the proper structure: <thinking> ... </thinking>, <reason> ... </reason>, and <answer> ... </answer> (as shown in the prompt template, Figure 4). The reward is all-or-nothing — all three tags must be present and properly closed.

What it computes: a binary 0/1 reward for output structure compliance.

Why this form: the format reward serves as a necessary constraint. Without it, the RL optimization might cause the model to drift away from the structured output format learned during SFT, making it impossible to extract the prediction and explanation at inference time. A simple binary reward is sufficient because the format is a syntactic constraint, not a quality metric — either the tags are present or they aren't.

Composite Reward Interaction

The composite reward R_final = R_pred + R_exp + R_format ranges from 0 to 3 for each candidate. The three components are designed to be complementary rather than redundant:

  • R_pred ensures the model remains accurate — it penalizes correct explanations that lead to wrong predictions.
  • R_exp ensures the model produces high-quality explanations — it penalizes correct predictions with uninformative explanations (the novice model would fail despite the correct label in the training data).
  • R_format ensures the output is structured — it penalizes unstructured outputs regardless of their content quality.

A candidate can receive R_final = 3 (perfect: correct prediction, explanation that transfers knowledge, proper formatting), R_final = 2 (correct prediction but poor explanation, or correct prediction and good explanation but malformed), R_final = 1 (only one component correct), or R_final = 0 (failure on all dimensions).

The sum formulation (rather than a product or weighted average) treats each component as independently valuable. A candidate that is correctly formatted and has a correct prediction but poor explanation receives R_final = 2, which still provides a positive signal (compared to 0 or 1) that the policy should move toward correct predictions, just not at the expense of explanation quality. The relative advantage computation within GRPO then determines whether the policy shifts toward the R=2 or R=3 candidates based on which is above the group average — if all candidates have poor explanations (a common outcome early in training), R=2 candidates will have positive advantages and the policy will first learn to be accurate before being pushed toward higher explanation quality.

The ablation study (Table 5) confirms the importance of the explanation quality reward: removing it (- w/o. Explanation Quality Reward) drops explainability from 90.4 to 84.7 while only slightly reducing effectiveness (85.7 vs. 86.4). This demonstrates that R_exp is the primary driver of explanation quality improvements, and that optimizing for explanation quality does not harm (and may slightly help) prediction accuracy — a finding that supports the paper's central claim that trustworthiness and effectiveness are complementary, not competing, objectives.

Summary of Design Choices and Their Justifications

  • DeepSeek-V3.2-Think for data synthesis over other advanced LRMs: provides access to chain-of-thought content, which GPT-series models and o-series models do not expose. Without CoT training data, the model would have no demonstration of the reasoning-before-explanation pattern.

  • Three-dimensional data filtering over simple label-based filtering: label correctness alone retains too many examples with low-quality explanations; explanation quality filtering alone biases toward easy examples; data diversity filtering alone does not guarantee label accuracy. The three filters are complementary and address orthogonal sources of data noise.

  • Perplexity-based explanation quality evaluation rather than correctness-based evaluation: during SFT data filtering, the authors use perplexity reduction rather than requiring the model to generate new predictions because it is cheaper (no sampling variance, no inference cost for multiple runs) and provides a continuous signal within the binary keep/discard framework.

  • K-Medoids with K=10 for diversity filtering: actual data points as cluster centers are more representative than artificial centroids; K=10 balances cluster specificity with statistical reliability, and is shown to be robust to the exact value in parameter sweeps.

  • GRPO over PPO for RL optimization: eliminates the need for a separate value function model, reducing memory requirements and simplifying the training pipeline. The group-relative advantage normalizes rewards without requiring reward normalization heuristics.

  • Novice-level correctness-based explanation quality reward rather than perplexity-based: the RL stage demands a higher standard than SFT filtering — actual prediction correctness rather than perplexity reduction. This forces the policy to generate explanations that are not just informative (reduce uncertainty) but sufficient (enable correct decisions).

  • Homologous novice model (Llama-3.1-8B-Instruct for a Llama-3.1-8B-Inst backbone): avoids format and style incompatibilities that could introduce noise into the reward signal. The novice model must be weak enough to depend on the explanation but compatible enough to process it correctly.

  • Sum-based composite reward over product or weighted average: treats each quality dimension as independently valuable, allowing partial credit and enabling the policy to learn progressively (accuracy first, then explanation quality) rather than requiring simultaneous improvement on all dimensions.

  • No additional data introduced beyond FactCG's training sets: the entire FaithLens pipeline uses the same ANLI subset, C2D, D2C (for SFT) and CG2C-MHQA, CG2C-Doc (for RL) as FactCG (Lei et al., 2025), totaling 28,643 training examples after filtering (Table 4). This ensures fair comparison — FaithLens's improvements come from better data curation and training methodology, not from access to more or better source data.

4. Key Insights and Innovations

Innovation 1: Framing Explanation Quality as Knowledge Transfer, Not Text Similarity

The paper's most conceptually distinctive move is how it defines and optimizes explanation quality. Prior work that produced text alongside predictions—such as ClearCheck's chain-of-thought traces (Seo et al., 2025) or prompted reasoning from API-based LLMs—implicitly treated the generated text as valuable insofar as it led to correct predictions. The quality of that text as an explanation was either unevaluated or evaluated only via surface-level metrics like fluency and relevance. The dominant assumption was that if a model reasons correctly internally, the text it produces will naturally serve as a good explanation.

FaithLens breaks this assumption with a specific operational definition: an explanation is good if it enables a weaker model that could not otherwise solve the task to do so. This is instantiated twice—during SFT data filtering via perplexity reduction on the ground-truth label (Equation 6), and during RL via the explanation quality reward where a frozen Llama-3.1-8B-Instruct must correctly predict the label after reading the explanation (Equation 14). In both cases, the criterion is not "does this text look like a good explanation" but rather "does this text transfer the capability to make the correct judgment."

This is a fundamental reframing. The field's default approach to explanation evaluation—when it evaluates at all—is to use human judgments or LLM-as-a-judge on dimensions like fluency, relevance, and completeness. These are surface-level proxies. FaithLens's knowledge-transfer criterion is a causal test: it measures whether the explanation actually changes a downstream decision-maker's behavior in the intended direction. An explanation that merely restates the conclusion ("the claim is not supported") scores poorly under this criterion because it provides no new information to the novice model. Only explanations that cite specific evidence, point out specific discrepancies, or break down complex claims into verifiable components actually improve the novice model's predictions—and thus receive positive reward.

The significance of this framing extends beyond hallucination detection. It suggests a general principle for training explainable models without human-labeled explanation data: pair the model with a weaker observer and optimize for the observer's performance conditional on the model's output. This could apply to any domain where a strong model needs to explain its decisions to a weaker one—code review, medical diagnosis, legal reasoning. The paper does not claim this generality, but the operationalization is sufficiently abstract to be widely applicable.

The empirical evidence for this innovation is embedded in the ablation study (Table 5). Removing the explanation quality reward (- w/o. Explanation Quality Reward) drops explainability from 90.4 to 84.7 (as measured by GPT-4.1 judging readability, helpfulness, and informativeness), while leaving effectiveness nearly unchanged (85.7 vs. 86.4). This cleanly isolates the effect: the prediction correctness reward alone produces an accurate model, but not an explainable one. The explanation quality reward is what converts accurate internal reasoning into externally useful explanations. More subtly, the study in Appendix I (Study 3) showing that using an expert-level model (DeepSeek-V3.2-Think) as the "novice" degrades explanation quality provides convergent evidence for the knowledge-transfer framing: if the observer is too strong, it can succeed even with bad explanations, and the reward signal loses its discriminating power.

This is not an incremental refinement of explanation generation. It is a new diagnostic concept: explanation quality as measured by transfer learning, operationalized as a reward signal that can be computed automatically. That this works—that optimizing for a weaker model's predictions produces text that humans and LLM judges rate as more readable, helpful, and informative—is the paper's deepest empirical finding.

Innovation 2: Re-founding Data Filtering on Complementary Informativeness, Not Heuristic Cleanliness

The paper's three-dimensional data filtering strategy for synthetic training data appears, at first glance, to be a careful engineering contribution—well-designed filters that improve data quality. But the conceptual innovation runs deeper: the three filters each target a different failure mode of the interaction between the synthetic data generator and the model being trained, and together they constitute a principle for curating synthetic data that goes beyond simple correctness checks.

To see why this is distinctive, consider the dominant approach to synthetic data filtering in prior work. The standard practice is to filter based on label agreement with ground truth—if the synthetic label is correct, keep the example; if not, discard it (this is essentially what FactCG does implicitly by relying on deterministic label generation from knowledge graphs). Some work adds heuristic filters for length, diversity of sources, or confidence scores. These are all absolute quality checks that evaluate each synthetic example in isolation.

FaithLens's approach is relational. The explanation quality filter (Equation 6) evaluates not whether the explanation is good in isolation, but whether it adds value beyond what the CoT already provides—measured as the marginal perplexity reduction when the explanation is appended to the CoT-only context. A perfectly grammatical, fluent explanation that merely paraphrases the CoT would fail this filter. The data diversity filter (Equation 9) evaluates not whether a candidate example looks different from others, but whether it provides complementary information across a probe set spanning the data distribution—measured by whether using it as a demonstration improves perplexity on diverse probe samples.

This reframes data curation from a static quality assessment problem to a dynamic complementarity assessment problem. The question is not "is this example good?" but rather "does this example add information that other examples in the training set don't already provide?" This is a fundamentally different lens, and it addresses a well-known pathology in synthetic data pipelines: that easy examples survive quality filters disproportionately, leading to training sets that are clean but narrow, producing models that are accurate on simple cases but brittle on complex ones.

The empirical evidence for this framing comes from the ablation results (Table 5). Removing data diversity filtering (- w/o. Data Diversity Filtering) increases the standard deviation across tasks from 4.6 to 6.4—a substantial increase in performance inconsistency—while the average F1 drops from 86.4 to 85.0. This pattern is exactly what the complementarity framing predicts: without diversity filtering, the model becomes more specialized to the types of examples that survived the other two filters, losing robustness on less-represented hallucination patterns. The standard deviation metric is particularly informative here because it captures cross-task consistency, which is the paper's explicit goal (Gap 2 in Section 1).

The insight generalizes: when generating synthetic training data with a strong model, the primary risk is not noise but homogeneity—the strong model produces clean examples concentrated in regions of the problem space it finds easy. A filtering strategy that only checks for correctness will amplify this bias. FaithLens's relational, complementarity-based filtering is a principled countermeasure, and one that could be adapted to any domain where synthetic data from an advanced model is used to train a smaller model.

Innovation 3: Jointly Optimizing Predictive Accuracy and Explanation Quality as Explicit, Separate Objectives in Reinforcement Learning

Prior work that trained models to produce reasoning alongside predictions—including ClearCheck (Seo et al., 2025) and various chain-of-thought fine-tuning approaches—used supervised learning on synthetic or human-written reasoning traces. The objective was always a single scalar: maximize the probability of the target sequence. This conflates two distinct goals: producing correct predictions and producing useful explanations. A model trained under this objective can learn to produce text that looks like reasoning without actually improving its predictions, or can produce correct predictions with vacuous reasoning, depending on which pattern dominates the training data.

FaithLens separates these objectives in the RL stage by introducing independent, additive rewards for prediction correctness and explanation quality (Equations 13 and 14). The RL training sees, for each candidate output, two distinct 0/1 signals—one for whether the prediction matched ground truth, one for whether the explanation enabled a novice model to predict correctly. These signals can conflict: a candidate might have the correct prediction but a poor explanation (R = 2 + format), or might have a good explanation but the wrong prediction (also R = 2). The GRPO algorithm learns from the relative advantage of candidates within each group, which means it receives a nuanced training signal about the tradeoff between accuracy and explainability.

This is a small architectural change—two rewards instead of one—but a significant conceptual shift. It acknowledges that prediction quality and explanation quality are orthogonal dimensions that must be explicitly supervised, not assumed to co-occur. The evidence that this separation matters comes from the comparison between the - w/o. Rule-based RL Stage ablation (SFT only) and the full FaithLens (Table 5). The SFT-only model, trained with the standard sequential prediction objective, achieves 82.6 F1 and 83.8 explainability. Adding RL with only the prediction correctness reward (implicitly the - w/o. Explanation Quality Reward condition, since the format reward is present in all RL variants in the main ablation) yields 85.7 F1 and 84.7 explainability—accuracy improves but explainability barely moves. Adding both rewards yields 86.4 F1 and 90.4 explainability—both improve, and explainability improves substantially more. This pattern demonstrates that explanation quality does not automatically follow from prediction accuracy, even when the model is producing explanation-like text. It requires a separate optimization signal.

The innovation here is not the use of multiple rewards per se—multi-objective RL is well-established—but rather the operationalization of the second objective as knowledge transfer to a fixed observer model, and the demonstration that this operationalization produces explanations that humans and LLM judges rate as higher quality along multiple dimensions (Table 2). The paper does not rely on any human-labeled explanation quality data, yet produces explanations that are competitive with or superior to those from much larger models like GPT-4o and o1, as shown in the human evaluation (Figure 3) where FaithLens wins or ties on the majority of examples.

Innovation 4: Establishing That a Compact Specialized Model Can Surpass Large General-Purpose Models on Structured Reasoning Tasks—With a Specific, Replicable Training Recipe

This is the paper's most applied finding, but it carries conceptual weight because of the specificity of the demonstration. It has been known that fine-tuned models can match or approach larger models on narrow tasks (as MiniCheck and FactCG showed for binary hallucination classification). What is new here is that FaithLens surpasses models like GPT-5.2 and o3 on a task that requires structured reasoning and explanation generation, not just classification, and does so using a training recipe that is fully reproducible—open-source base model, open-source training data, and a pipeline that requires no human annotation.

The evidence is in Table 1: FaithLens achieves 86.4 macro-F1 at 8B parameters, versus 86.1 for GPT-5.2 and 82.1 for o3. The cost comparison in Table 3 is staggering: roughly 0.10vs.0.10 vs. 15.30 for GPT-5.2 on 1.2K samples—two orders of magnitude cheaper. Critically, this advantage is not limited to classification accuracy. Table 2 shows that FaithLens's explanations are rated higher than GPT-4o's and competitive with o3's on explainability dimensions, while being generated at a tiny fraction of the cost.

The significance of this result lies in what it implies about the relationship between model scale and structured reasoning. The dominant narrative in the field, reinforced by scaling laws and the performance of models like o1 and o3 on reasoning benchmarks, is that deep reasoning requires large models with massive pretraining. FaithLens demonstrates that for a specific, well-defined reasoning task—examining a document and claim and producing a structured justification for a binary judgment—the reasoning capability can be compressed into a much smaller model through a combination of synthetic data from a reasoning model and multi-objective reinforcement learning. The knowledge transfer happens twice: first from DeepSeek-V3.2-Think to the SFT data, then through the RL process that optimizes for knowledge transfer to the novice model.

This is not a claim that 8B models can reason as well as large models in general. The paper is careful to limit its claims to faithfulness hallucination detection specifically. But the recipe—synthesize structured reasoning traces from a large reasoning model, filter for quality and diversity, then fine-tune a small model with an RL objective that explicitly penalizes explanations that don't transfer knowledge—is general. If it works for hallucination detection, it plausibly works for other structured reasoning tasks where correctness can be verified automatically and where the reasoning can be serialized into text. The paper doesn't make this claim, but the implication is clear.

The FLOPs-matched comparison here is implicit but stark: FaithLens uses ~0.1% of the inference compute of GPT-5.2 (estimated from the cost figures, assuming similar hardware costs) while producing superior results. For deployed hallucination detection systems—where latency, cost, and privacy all favor local models over API calls—this is a transformative result. It suggests that the future of hallucination detection, and perhaps of structured reasoning more broadly, lies not in scaling model size but in distilling reasoning capabilities into compact, specialized models through carefully designed training pipelines.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. All experiments use the cleaned version of LLM-AggreFact (Tang et al., 2024a) and HoVer (Jiang et al., 2020) benchmarks, as curated by Seo et al. (2025), who identified 9.1% ambiguous examples and 6.6% mislabeled instances in the originals. LLM-AggreFact spans 11 faithfulness hallucination detection tasks: Agg-CNN, Agg-XSum (summarization, Tang et al., 2023), ClaimVerify (generative search engine responses, Liu et al., 2023b), ExpertQA (expert-curated QA, Malaviya et al., 2024), FC-GPT (factual consistency of LLM responses, Wang et al., 2023), LfQA (long-form QA, Chen et al., 2023), RAGTruth (RAG hallucination corpus, Wu et al., 2023), Reveal (reasoning chain verification, Jacovi et al., 2024), Tofu-MediaS and Tofu-MeetB (dialogue summarization, Tang et al., 2024b), and Wice (Wikipedia claim entailment, Kamoi et al., 2023). HoVer (Jiang et al., 2020) focuses on complex multi-hop reasoning requiring evidence from up to four Wikipedia articles, with 3/4-hop claims written in multiple sentences. Across the 12 tasks, 1.2K samples form the evaluation set used for cost reporting (Table 3).

  • Base model(s). All primary experiments use Llama-3.1-8B-Instruct (Grattafiori et al., 2024) as the backbone for FaithLens. This choice is motivated by fair comparison with ClearCheck (Seo et al., 2025), which also uses this backbone, and by the model's representativeness as a widely deployed open-source LLM. Generalization experiments (Table 7, Appendix H) additionally test Qwen-2.5-3B-Inst, Qwen-2.5-7B-Inst, Qwen-2.5-32B-Inst (Yang et al., 2024), Llama-3.1-70B-Inst, and Llama-3.1-405B-Inst as backbones.

  • Metrics. The primary effectiveness metric is macro-F1, following Seo et al. (2025) for fair comparison. This is computed per task and then averaged across the 12 datasets (Table 1 reports the overall mean µ and standard deviation σ). For explainability, three GPT-4.1-as-judge dimensions are scored on a 1–5 scale (prompt in Figure 9, Appendix D): readability (clarity, logical sequence, avoidance of ambiguity), helpfulness (does the explanation guide understanding of why the model reached its conclusion), and informativeness (richness of content, specificity of evidence cited, nuanced reasoning). These are reported as percentages (Table 2). For efficiency, inference cost in USD is reported for 1.2K samples from the 12 datasets (Table 3), assuming $0.8/GPU-hour for FaithLens.

  • Baselines. The paper compares against two categories. Advanced API-based LLMs: GPT-4o (gpt-4o-2024-08-06), o1 (o1-2024-12-17), GPT-4.1 (gpt-4.1-2025-04-14), o3-mini (o3-mini-2025-01-31), o3 (o3-2025-04-16), GPT-5.2 (gpt-5.2-2025-12-11), DeepSeek-V3.2-Non-Think (deepseek-chat), DeepSeek-V3.2-Think (deepseek-reasoner), Claude-3.7-Sonnet (claude-3-7-sonnet-20250219), and Llama-3.1-405B-Inst. These are evaluated using the prompt from Seo et al. (2025) (shown in Figure 10, Appendix B), which asks the model to determine if a statement is supported by a document and output [Attributable], [Not Attributable], or [Contradictory]—with the latter two mapped to hallucination. Specialized detection models: AlignScore (Liu et al., 2023a)—a 355M RoBERTa-Large-based model trained on 4.7M examples from 7 tasks, using a 0.5 prediction threshold; MiniCheck (Tang et al., 2024a)—a 7B model trained on 35K private data synthesized from Llama-3.1-405B-Inst; FactCG (Lei et al., 2025)—a 435M DeBERTa-v3-Large model trained on 52K data including the ANLI subset, C2D, D2C, and CG2C sets; ClearCheck (Seo et al., 2025)—an 8B model using 57K ANLI examples and 25K private multi-hop data with CoT distilled from Llama-3.1-405B-Inst and multi-task training. For baselines already evaluated under identical settings by Seo et al. (2025), results are directly adopted. For baselines not covered (e.g., FactCG, GPT-5.2), the authors reproduce results under the same experimental settings. All models are inferred twice to obtain stable results.

  • Generation budget / compute accounting. The paper reports inference cost in US dollars (Table 3) rather than FLOPs or token counts, since API-based baselines have opaque computational costs. The 1.2K evaluation set from 12 datasets is used as the unit for cost comparison, with FaithLens's cost computed at $0.8/GPU-hour on NVIDIA A800 SXM4 80GB GPUs. For the explanation quality reward computation during RL training (Equation 14), the novice-level model generates a single prediction per candidate explanation at temperature 0.6.

  • Cross-validation / statistical protocol. No explicit cross-validation is used for final evaluation—results are reported on the fixed cleaned benchmark from Seo et al. (2025). For the parameter study (Appendix I), different hyperparameter choices are compared directly. For the few-shot prompting study (Appendix I, Study 4), demonstration examples from Seo et al. (2025) are used. The human evaluation (Figure 3, Appendix G) uses 120 samples (10 per dataset) evaluated by three participants with majority voting across readability, helpfulness, and informativeness.

Main Quantitative Results

5.1 Overall Effectiveness: FaithLens Surpasses All Baselines at 8B Parameters

Table 1 presents the headline result: FaithLens achieves 86.4 macro-F1 averaged across 12 tasks, with a standard deviation of 4.6—the highest mean and lowest variance among all evaluated systems. This places it ahead of the strongest API-based LLM (GPT-5.2 at 86.1, σ = 5.9) and substantially ahead of all specialized detection models (MiniCheck at 80.7, FactCG at 78.2, ClearCheck at 80.1). The comparison to Llama-3.1-8B-Inst (the base model before fine-tuning, at 56.3 F1) represents a +30.1 percentage point absolute improvement.

The per-task breakdown reveals several patterns. On the most challenging tasks, FaithLens demonstrates the largest advantages: on HoVer (multi-hop reasoning), FaithLens achieves 82.9 vs. ClearCheck's 80.3 and FactCG's 73.1—a 2.6-point lead over the next-best specialized model. On Agg-CNN (summarization hallucination), FaithLens achieves 84.9 vs. ClearCheck's 72.8—a dramatic 12.1-point gap. On FC-GPT (factual consistency), FaithLens reaches 92.1 vs. GPT-5.2's 89.9. However, on RAGTruth, FaithLens scores 92.2—slightly below GPT-4.1 (93.2) and GPT-5.2 (93.3), and tied with AlignScore (92.2). The standard deviation of 4.6 across tasks is the lowest among all systems (GPT-5.2: 5.9, o3: 6.0, ClearCheck: 6.6, MiniCheck: 7.5), directly supporting the paper's claim about improved cross-task consistency (addressing Gap 2 from Section 1).

Comparing to the base Llama-3.1-8B-Inst, the largest per-task improvements are on FC-GPT (+46.0), Agg-CNN (+41.8), and Tofu-MeetB (+37.4), while RAGTruth shows the smallest gain (+14.0), possibly because the base model already handles this task reasonably well (78.2) relative to others.

5.2 Explainability: FaithLens Generates Higher-Quality Explanations Than API-Based LLMs

Table 2 evaluates explanation quality using GPT-4.1 as a judge on three dimensions (1–5 scale, reported as percentages; prompt in Figure 9, Appendix D). Only explanations corresponding to correctly predicted samples are evaluated. FaithLens achieves 90.4 average across readability (92.4), helpfulness (93.4), and informativeness (85.4). This substantially outperforms:

  • GPT-4o (84.1): FaithLens leads most strongly on helpfulness (93.4 vs. 84.8, +8.6 points) and informativeness (85.4 vs. 73.0, +12.4 points), while readability is comparable (92.4 vs. 94.4).
  • o3 (93.5): o3 scores higher overall, but FaithLens achieves comparable readability (92.4 vs. 97.6) and helpfulness (93.4 vs. 97.6) at two orders of magnitude lower cost.
  • ClearCheck's CoT (77.3): The large gap (+13.1 points) demonstrates that CoT reasoning traces from SFT alone do not serve as high-quality explanations—the RL explanation quality reward is essential. ClearCheck's CoT scores 75.5 when used directly (the "CoT from FaithLens" ablation, where the CoT is presented as explanation without the structured explanation generation), confirming that the CoT and explanation serve distinct roles.
  • Base Llama-3.1-8B-Inst (71.9): The +18.5-point improvement establishes that FaithLens's training pipeline creates explanation capability that the base model lacks entirely.

A robustness check using GPT-5-mini as the judge instead of GPT-4.1 (Table 10, Appendix) shows consistent patterns: FaithLens achieves 90.4 vs. GPT-4o's 82.4 and o3's 91.9, with the ranking of systems largely preserved. However, GPT-4.1's self-evaluation scores are inflated for readability (94.4 by GPT-4.1 judging GPT-4o vs. 90.2 by GPT-5-mini), indicating some degree of self-preference bias in LLM-as-judge evaluations—though this does not affect FaithLens's relative standing.

5.3 Efficiency: Two Orders of Magnitude Cheaper Than API-Based Models

Table 3 reports inference costs on 1.2K samples from the 12 datasets. FaithLens costs **0.10∗∗(at0.10** (at 0.8/GPU-hour), while API-based competitors range from 0.80(DeepSeek−V3.2−Non−Think)to0.80 (DeepSeek-V3.2-Non-Think) to 140.60 (o1). The cost-to-performance ratio is dramatic: FaithLens achieves 86.4 F1 at 0.10,whileo3achieves82.1at0.10, while o3 achieves 82.1 at 8.80 (~88× more expensive for lower performance), and GPT-5.2 achieves 86.1 at 15.30( 153×moreexpensiveforcomparableperformance).ThiscostadvantagestemsfromFaithLensbeingacompact8Bmodelthatrunslocally,avoidingper−tokenAPIpricing.ThepapernotesthatDeepSeek−V3.2−Think,whilerelativelycheap(15.30 (~153× more expensive for comparable performance). This cost advantage stems from FaithLens being a compact 8B model that runs locally, avoiding per-token API pricing. The paper notes that DeepSeek-V3.2-Think, while relatively cheap (1.20), still costs 12× more than FaithLens while achieving lower performance (84.4 F1, Table 1).

Table 4 compares training data efficiency across specialized models. FaithLens uses 28,643 total training examples (11,929 SFT + 16,714 RL) from open-source data (the same sources as FactCG), compared to 4,700K for AlignScore, 35K for MiniCheck (private), and 82K for ClearCheck (private). The data filtering pipeline reduces the initial 52,268 synthetic examples to 11,929 for SFT—a 77.2% reduction—yet produces a stronger model than training on all 52K examples directly (81.2 F1 for - w/o. Data Filtering in Table 5). This demonstrates that data quality, not quantity, is the primary driver of FaithLens's performance.

5.4 Ablation Study: Each Component Provides Complementary Gains

Table 5 reports the ablation results, with detailed per-task breakdowns in Table 9 (Appendix E) and per-dimension explainability scores in Table 10.

  • The base Llama-3.1-8B-Inst achieves 56.3 F1 with σ = 10.9 and 71.9 explainability.

  • Direct SFT on all 52K unfiltered data reaches 79.1 F1, σ = 6.1—a decent improvement from the base model—but cannot provide explanations (the training data consists of binary classification examples without explanation targets).

  • Removing the cold-start SFT stage (- w/o. Cold-start SFT Stage, i.e., RL directly on the base model) yields 83.4 F1 and 88.1 explainability. This is 3.0 F1 points and 2.3 explainability points below FaithLens, showing that SFT initialization provides a meaningful but not essential foundation—RL alone could recover most of the performance, but SFT provides the structured output format and initial reasoning capability that RL then refines.

  • Removing all data filtering (- w/o. Data Filtering, training on unfiltered synthetic data then applying RL) produces 81.2 F1 (σ = 6.7) and 82.3 explainability. The -5.2 F1 drop and -8.1 explainability drop demonstrate that filtering is critical, particularly for explanation quality. The increased σ (6.7 vs. 4.6) reflects the loss of cross-task consistency.

  • Removing only label correctness filtering (- w/o. Label Correctness Filtering) yields 83.5 F1 and 86.0 explainability—a -2.9 F1 and -4.4 explainability drop. This filter primarily affects prediction performance, confirming that incorrect synthetic labels teach the model wrong patterns.

  • Removing only explanation quality filtering (- w/o. Explanation Quality Filtering) yields 85.8 F1 and 83.4 explainability—a -0.6 F1 drop (minimal) but a -7.0 explainability drop (substantial). This is the strongest evidence that explanation quality filtering specifically benefits explainability, consistent with its design (filtering for explanations that reduce perplexity on the correct answer). The near-zero effect on prediction accuracy suggests that label-correct examples with poor explanations still provide useful classification signal, but fail to teach the model how to generate good explanations.

  • Removing only data diversity filtering (- w/o. Data Diversity Filtering) yields 85.0 F1 and 89.3 explainability—a -1.4 F1 drop and -1.1 explainability drop, but notably the standard deviation increases from 4.6 to 6.4. This is the largest increase in σ among all ablations, confirming that diversity filtering's primary role is ensuring consistent cross-task generalization rather than boosting average performance.

  • Removing the entire RL stage (- w/o. Rule-based RL Stage, SFT only on filtered data) yields 82.6 F1 (σ = 6.0) and 83.8 explainability. Comparing to the 52K direct SFT baseline (79.1 F1, no explanations) and to FaithLens (86.4, 90.4) shows that: (a) data filtering alone adds +3.5 F1 to SFT, (b) RL adds +3.8 F1 and +6.6 explainability on top of filtered SFT. The RL gains are approximately evenly split between prediction accuracy and explainability improvements.

  • Removing only the explanation quality reward (- w/o. Explanation Quality Reward, RL with only R_pred + R_format) yields 85.7 F1 and 84.7 explainability—a -0.7 F1 drop but a -5.7 explainability drop. This cleanly isolates the contribution of the explanation quality reward: it accounts for nearly all of the explainability improvement from RL, while prediction accuracy is driven primarily by the correctness reward. This supports the paper's central claim that prediction quality and explanation quality are orthogonal objectives that require explicit, separate optimization signals.

5.5 Claim Decontextualization and Decomposition: Not Needed for FaithLens

Table 6 (with detailed per-task results in Table 9, Appendix F) investigates two preprocessing steps common in hallucination detection pipelines. Claim decontextualization (Choi et al., 2021) resolves coreference and ellipsis in claims using previous claims as context. FaithLens's performance is unchanged (86.4 F1, σ = 4.6) under decontextualization—identical to the original setting. Other specialized models show negligible changes (AlignScore ±0.1, MiniCheck ±0.1, ClearCheck 0.0), while API-based LLMs show small variations (GPT-5.2: 86.1 → 86.0, o3: 82.1 → 82.0). Claim decomposition (Min et al., 2023) breaks each claim into atomic facts, verifies each independently against the document, and labels the claim as supported only if all atomic facts are supported. FaithLens improves marginally to 86.6 F1 (σ = 4.4), a +0.2 gain. Other models similarly show small improvements (GPT-5.2: 86.1 → 85.9, o3: 82.1 → 82.5). The paper argues that "decontextualization and decomposition are not needed for our model as FaithLens can effectively capture the context-dependent relations" (Section 4.3), and that decomposition increases inference time by a factor of 2–4× (depending on the average number of atomic facts per claim) without providing a significant accuracy benefit.

5.6 Human Evaluation: FaithLens Explanations Preferred Over GPT-4o

Figure 3 reports pairwise human evaluation on 120 samples (10 per dataset, Appendix G). Across three dimensions—readability, helpfulness, and informativeness—FaithLens wins or ties with GPT-4o on the majority of comparisons (exact counts are not numerically specified in Figure 3, which shows a bar chart with "Ours Wins," "Tie," and "GPT-4o Wins" segments). Three participants with bachelor's or master's degrees evaluated explanations following the principles in Figure 13, with majority voting determining the final result. This provides an independent validation (beyond LLM-as-judge in Table 2) that FaithLens's explanations are competitive with or superior to GPT-4o's.

5.7 Generalization Across Foundation Models: The Recipe Transfers

Table 7 (detailed in Tables 9–10, Appendix H) tests the FaithLens training pipeline on different model backbones. For the Llama family: FaithLens-8B (86.4 F1, 90.4 explainability) substantially exceeds Llama-3.1-405B-Inst used directly (75.8 F1, 83.7 explainability). For the Qwen family: FaithLens-3B (83.4 F1, 88.3 explainability) outperforms both Qwen-2.5-32B-Inst directly prompted (73.1, 84.2) and the larger FaithLens-7B (84.9, 90.3). The consistent pattern—FaithLens training substantially improves both effectiveness and explainability across model families and scales—suggests the recipe is not specific to the Llama architecture. Notably, FaithLens-3B at 3B parameters achieves 83.4 F1, surpassing o3-mini (79.2), o1 (76.8), and GPT-4o (76.1) while being dramatically smaller and cheaper.

Ablation Studies and Robustness Checks

  • Number of clusters K in data diversity filtering: Tested at K=6, 10, 14, 20 (Tables 11–12, Appendix I). Performance is stable across values: 85.6 (K=6), 86.4 (K=10), 86.0 (K=14), 86.1 (K=20). Explainability similarly stable (89.8–90.4). The method is robust to this hyperparameter, though larger K increases computation time for perplexity calculation.

  • Choice of sentence embedding model for diversity filtering: Comparing Llama-Embed-Nemotron-8B (90.4 explainability, 86.4 F1), Linq-Embed-Mistral-7B (89.7, 86.0), and Gemini-Embedding-001 (90.0, 85.7) shows stable results across different advanced embedding models (Tables 11–12). The recommendation is to use an advanced open-source embedding model for reproducibility.

  • Choice of novice-level model for explanation quality reward: Comparing Llama-3.1-8B-Inst (homologous, 86.4 F1, 90.4 explainability), Qwen-2.5-7B-Inst (heterologous, 85.3, 88.4), and DeepSeek-V3.2-Think (expert-level, 85.9, 86.5)—Tables 11–12. Using a heterologous model reduces explainability, possibly due to format or style incompatibility. Using an expert-level model produces the lowest explainability because "the expert model can ignore the incorrect explanations provided and still predict the correct label," assigning high rewards to low-quality explanations and weakening the training signal.

  • Few-shot prompting for LLM baselines: Adding few-shot demonstrations (Study 4, Tables 11–12) improves GPT-4o from 76.1 → 82.5 F1 and o1 from 76.8 → 85.1 F1, but FaithLens (86.4) still outperforms both. The few-shot examples are those provided in Seo et al. (2025). This confirms that the performance advantage is not an artifact of zero-shot vs. fine-tuned comparison.

  • Using only CoT vs. only explanations vs. both for SFT: Question 1 in variant methods testing (Tables 11–12) compares three SFT configurations. Using both CoT and explanations (the FaithLens design) yields 82.6 F1 and 83.8 explainability. Using only explanations (removing CoT during SFT) drops to 75.8 F1 and 81.9 explainability—a dramatic -6.8 F1 decline, demonstrating that "explanations alone cannot serve the same role as the CoT in improving faithfulness hallucination detection performance." Using only CoT achieves 81.2 F1, close to the CoT+explanation variant but with no explainability metric reported (since the model never learns to produce structured explanations). This justifies the design choice of generating CoT first as a reasoning scaffold, then producing a separate user-facing explanation.

  • Perplexity-based vs. correctness-based explanation quality reward: Question 2 in variant methods testing (Tables 11–12) compares using perplexity reduction (reward = 1 if the novice model's perplexity on the correct label decreases, matching the SFT filtering criterion) versus actual correctness (reward = 1 if the novice model predicts correctly, the FaithLens design). Using perplexity achieves 83.9 F1 and 88.2 explainability vs. 86.4 and 90.4 for correctness. The paper suggests this is because "reducing perplexity is a simpler task compared to using correctness, which limits the model's ability to explore more effective policies during the rule-based RL stage." This result also demonstrates that the escalation from perplexity-based filtering (SFT) to correctness-based reward (RL) is important: the RL stage demands a stricter standard of explanation quality.

Critical Assessment

Does FaithLens genuinely outperform GPT-5.2 and o3 in a practically meaningful way?

The headline claim from Table 1—FaithLens at 86.4 F1 surpasses GPT-5.2 at 86.1 and o3 at 82.1—is supported by the reported numbers. However, the margin over GPT-5.2 is only 0.3 F1 points, which is within the range of statistical noise given a 500–question test set (the paper does not report confidence intervals). The more meaningful practical claim is the cost-efficiency: FaithLens achieves comparable or better performance at roughly 1/150th the inference cost (0.10vs.0.10 vs. 15.30 for GPT-5.2 on 1.2K samples, Table 3). This cost advantage is robust and would hold even if FaithLens were slightly below GPT-5.2 in accuracy. The paper could strengthen the "surpasses" claim by reporting whether the difference is statistically significant, which it does not.

A subtler issue: the evaluation protocol for API-based baselines uses a specific prompt (Figure 10, upper part) that maps three labels to binary outcomes, while FaithLens uses its own structured prompt format (Figure 4). These are not identical tasks—the API-based models are performing a generic entailment task, while FaithLens is trained specifically for hallucination detection. The paper does test an alternative prompt (Figure 10, lower part) that asks API-based models to produce explanations (Table 13, Appendix), and the F1 scores are nearly identical to the standard prompt results (GPT-4o: 76.1 vs. 75.7, GPT-5.2: 86.1 vs. 86.0). This suggests prompt format differences have negligible impact on prediction accuracy, partially mitigating the concern.

Does the data filtering strategy genuinely improve data quality, or does it simply select easier examples?

The ablation results provide strong evidence for the former interpretation. If filtering merely selected easier examples, we would expect: (a) improved average performance but higher variance (since the model would be specialized to easy cases), and (b) explanation quality to degrade or stay flat (since easy examples don't necessarily teach good explanation habits). Instead, Table 5 shows: filtering reduces variance (σ drops from 6.7 to 4.6) and improves explainability (from 82.3 to 90.4). The data diversity filter specifically reduces variance (σ 6.4 → 4.6) without much changing average F1 (85.0 → 86.4), which is consistent with the claim that it prevents distributional skew toward easy examples. These patterns are consistent with genuine quality improvement, not difficulty-based selection.

However, there is a limitation: the filtering process discards 77.2% of the synthetic data (52,268 → 11,929 for SFT). The paper does not characterize what types of examples are filtered out beyond the high-level failure modes. Are certain tasks (e.g., HoVer multi-hop) disproportionately filtered? Does filtering remove genuinely ambiguous cases that might be valuable for robustness? Without this analysis, the claim that filtered data is "higher quality" rests on downstream performance improvements, not on an intrinsic analysis of what distinguishes retained from discarded examples.

Does the explanation quality reward genuinely measure explanation quality, or does it measure something else?

The operationalization—rewarding explanations that enable a novice model to predict correctly—is elegant but carries a potential confound: the novice model (Llama-3.1-8B-Instruct) is from the same model family as the policy model and may share similar biases, shortcut heuristics, or format preferences. An explanation could help the novice model predict correctly not because it conveys genuine evidence about the document-claim relationship, but because it triggers a spurious pattern that the novice model's internal representations latch onto—for instance, always including the phrase "the document explicitly states" might correlate with faithful claims in the training data.

The evidence against this confound is: (a) the human evaluation (Figure 3) and LLM-as-judge evaluation (Table 2) both rate FaithLens's explanations highly, and these evaluations do not share the novice model's potential biases; (b) the ablation showing that expert-level models as the "novice" degrade performance (Tables 11–12) demonstrates that the reward signal is sensitive to the observer's capability level in the theoretically expected direction. However, the paper does not conduct a controlled experiment where explanations are adversarially constructed to exploit the novice model's shortcuts, so the robustness of the reward signal to reward hacking remains an open question.

Does the generalization across foundation models (Table 7) genuinely establish that the recipe is model-agnostic?

The results are positive but limited: only two model families are tested (Llama-3.1-Inst at 8B/70B/405B and Qwen-2.5-Inst at 3B/7B/32B), both decoder-only transformer architectures with similar instruction-tuning paradigms. The recipe might fail on models with substantially different architectures (encoder-decoder, mixture-of-experts, non-transformer backbones) or different pre-training objectives. The claim "the recipe transfers" is supported but only within the tested families. Additionally, the paper does not report the computational cost of training FaithLens variants, making it difficult to assess whether the recipe is practical for resource-constrained settings.

Missing experiments that would strengthen the paper

The paper does not include several analyses that would make the conclusions more robust:

  1. Statistical significance testing for the main comparison (Table 1). With 12 test sets of varying sizes and a 0.3 F1 gap over GPT-5.2, readers cannot determine whether FaithLens's advantage is reliable or within sampling noise.

  2. An analysis of what types of examples are filtered out at each stage of the data filtering pipeline. Understanding whether specific hallucination patterns, difficulty levels, or task types are systematically removed would clarify the filtering strategy's limitations.

  3. A direct comparison of FaithLens's RL recipe against standard RLHF with a trained reward model. The paper argues that rule-based rewards eliminate the need for a reward model, but does not compare against an approach that trains a reward model on human-labeled explanation quality data. This leaves open the question of whether rule-based rewards are genuinely superior or merely cheaper.

  4. Latency benchmarking. The paper reports only cost (Table 3), but FaithLens's sequential generation of CoT → explanation → prediction increases inference latency compared to models that output only a label. For real-time applications, latency may matter as much as cost, and the paper provides no data on this dimension.

  5. An evaluation on out-of-distribution data. All 12 tasks are from the LLM-AggreFact and HoVer benchmarks, which share similar document-claim structures. Testing on a substantially different hallucination detection task (e.g., multimodal, cross-lingual, or with much longer documents) would clarify whether FaithLens's capabilities generalize or are specific to the benchmark distribution.

  6. A FLOPs-matched comparison between FaithLens and API-based models that accounts for pretraining cost. The paper's cost comparison (Table 3) includes only inference cost. A full accounting would amortize the pretraining costs of Llama-3.1-8B-Inst and the training cost of FaithLens (SFT + RL). While API-based models' pretraining costs are unknown, a comparison of total compute would provide a more complete picture of efficiency.

Where the claims hold conditionally

  • "Surpasses advanced LLMs such as GPT-5.2 and o3": Holds for the specific 12-task benchmark suite, but the margin over GPT-5.2 is small (0.3 F1) and not tested for statistical significance. The claim is better supported by the cost-efficiency framing (comparable performance at 1/150th the cost) than by the absolute performance framing.

  • "Can produce high-quality explanations": Strongly supported by automatic evaluation (Table 2) and human evaluation (Figure 3). However, "high-quality" is relative—FaithLens explanations are rated around 90.4 on a 100-point scale (interpreted from the percentages in Table 2), and the paper does not define what score threshold constitutes "high-quality" in absolute terms.

  • "Delivering a distinctive balance of trustworthiness, efficiency, and effectiveness": This is the paper's most robust claim. The three dimensions are independently measured (explainability in Table 2, cost in Table 3, accuracy in Table 1), and FaithLens ranks near the top on all three simultaneously—a combination no other model achieves. GPT-5.2 has comparable accuracy but dramatically higher cost and lower explainability. MiniCheck has lower cost but much lower accuracy and no explainability. The "balance" claim is well-supported by the multidimensional comparison.

  • "Without relying on closed-source and private data": Supported by Table 4, which shows FaithLens uses only open-source data (the same ANLI, C2D, D2C, CG2C sets as FactCG, all publicly available). This claim distinguishes FaithLens from MiniCheck (35K private data) and ClearCheck (25K private data, though the 57K ANLI subset is open). However, the synthetic data generation step uses DeepSeek-V3.2-Think, which is itself a proprietary model—so the training pipeline is not fully open in the strictest sense, even though the training data is.

  • "Cost-efficient": Undeniably supported by Table 3. FaithLens at $0.10 is 1–3 orders of magnitude cheaper than all API-based baselines while matching or exceeding their performance. This claim is the most robust in the paper.

6. Limitations and Trade-offs

The Difficulty Estimation Analogy: FaithLens Requires Ground-Truth Labels for Its Core Training Pipeline

The paper frames FaithLens as a model that can be trained "without human efforts" (Section 3) and "without relying on closed-source and private data" (Table 4). However, both the SFT data filtering pipeline and the RL explanation quality reward fundamentally depend on access to ground-truth binary labels for every training example. The label correctness filter (Equation 3) discards any synthetic sample where DeepSeek-V3.2-Think's predicted label does not match y_gt. The explanation quality filter (Equation 6) measures perplexity reduction on y_gt. The data diversity filter (Equations 7–9) computes perplexity improvements on probe samples' ŷ'_i. The explanation quality reward (Equation 14) checks whether a novice model predicts y_gt after reading the explanation. Every quality signal in the entire training pipeline—both SFT and RL—is anchored to ground-truth binary correctness judgments.

The consequence is that FaithLens's training pipeline is not applicable to domains where binary faithfulness labels are unavailable or expensive to obtain. The paper's framing emphasizes that it avoids human annotation for explanations specifically, and this is true—no human ever writes or rates an explanation. But it does require verified binary labels for all training instances, which for faithfulness hallucination detection are themselves expensive to produce. The existing benchmarks (LLM-AggreFact, HoVer) were constructed through substantial human annotation effort, and Seo et al. (2025) documented that even these curated benchmarks contained 9.1% ambiguous examples and 6.6% mislabeled instances. Extending FaithLens to a new domain with different document types, claim styles, or hallucination patterns would first require constructing a labeled training set with verified binary judgments—exactly the costly annotation process the paper's "without human efforts" framing might lead readers to believe is circumvented.

What evidence exists in the paper: This dependency is acknowledged implicitly but never stated as a limitation. The data synthesis section (3.1.1) notes that the pipeline leverages "open-source training datasets" as the source of (document, claim, label) triples, but does not discuss what happens when such datasets are unavailable. Table 4 shows that FaithLens uses 28,643 labeled examples from FactCG's training sets—all of which carry ground-truth labels from the original benchmarks. The data filtering section (3.1.2) treats y_gt as available without comment. The "Limitations" section at the end of the paper does not mention this requirement.

Mitigation status: Not addressed. The paper does not discuss whether the pipeline could be adapted to work with noisy, automatically generated, or weak labels. The filtering strategies depend on exact label matching and perplexity computation against ground truth, suggesting the approach would degrade substantially with even modest label noise. This is a significant gap because the high cost of annotation is precisely the bottleneck the paper's synthetic-data approach purports to solve—it solves the explanation annotation problem but not the label annotation problem.

FaithLens Outputs Only Binary Decisions, Not Fine-Grained Hallucination Categories

FaithLens is designed and evaluated exclusively on the binary classification task: faithful (label 1) or hallucinated (label 0). This is reflected in the task formulation (Equation 2, Section 2), the SFT training objective (Equation 10), the prediction correctness reward (Equation 13), the binary prediction prompt template (Figure 4), and all evaluation metrics. The model cannot distinguish between different types of unfaithfulness—it cannot tell a user whether a claim contradicts the document, introduces information absent from the document, or misinterprets a passage while using some correct surface elements.

The consequence is that FaithLens provides incomplete information to downstream consumers. In a RAG deployment, a user who receives a "hallucinated" judgment on a claim gains no insight into how to fix it. Should they remove the contradictory statement? Add missing attribution? Rephrase an ambiguous passage? Different error types require different remediation strategies, and a binary verdict masks this distinction. The paper's own motivating example (Section 1) distinguishes summarization hallucinations ("subtly distorted content") from RAG hallucinations ("conflicting claims"), establishing that error modes differ across tasks—yet FaithLens cannot communicate which error mode it detected, only that it detected one. For practical trustworthiness, this is a meaningful gap: a user who sees "hallucinated" without knowing whether the system found a contradiction or merely an unverifiable claim must manually re-examine the entire document-claim pair to understand the nature of the error.

What evidence exists in the paper: The authors explicitly acknowledge this limitation in the Limitations section: "Finally, following standard practice in existing works, FaithLens outputs only binary labels (faithful vs. hallucinated). While more fine-grained hallucination categories may benefit real-world applications, current datasets lack a unified taxonomy for such distinctions. We therefore leave fine-grained hallucination detection as future work." The honesty of this acknowledgment is commendable, but it does not change the practical constraint: users who need to act on hallucination detections (not just flag them) receive less actionable information from FaithLens than they might from a multi-class system.

The case studies in Appendix K (Figures 16–17) illustrate this indirectly. FaithLens's explanations describe what is wrong with the claim in free-form text—e.g., pointing out that the Lanham Act is not mentioned in the document, or that the release year 1940 contradicts the document's 2007 date. The explanations partially compensate for the binary label by providing specific evidence, but the system is not trained to produce a categorical error type (e.g., "unsupported claim," "factual contradiction," "temporal error"). The explanation quality reward (Equation 14) incentivizes the novice model to predict the correct binary label, not to classify the error type, so there is no optimization pressure toward categorical precision in the explanation.

Mitigation status: The paper explicitly defers this to future work, citing the absence of a unified taxonomy in existing datasets. This is a reasonable justification—constructing a multi-class hallucination taxonomy with adequate inter-annotator agreement across 12 diverse tasks would be a substantial research contribution in itself. However, the limitation remains consequential for practitioners who need granular error feedback.

Inference Overhead from Sequential CoT, Explanation, and Prediction Generation Is Not Benchmarked

FaithLens generates its output sequentially in three stages: first a chain-of-thought reasoning trace (<thinking>...</thinking>), then a human-readable explanation (<reason>...</reason>), then a binary prediction (<answer>...</answer>). This is fundamentally serial generation—the model cannot produce the explanation in parallel with the CoT, nor can it skip the CoT and go directly to the prediction (the ablation in Appendix J, Question 1, shows that removing CoT during SFT drops performance from 82.6 to 75.8 F1, confirming that the CoT scaffold is essential for accuracy). The paper also demonstrates that the CoT alone cannot serve as the explanation—"CoT from FaithLens" achieves only 75.5 on the explainability metric (Table 2) versus 90.4 for the full FaithLens output—meaning the explanation generation adds additional tokens beyond the CoT.

The consequence is that FaithLens generates substantially more output tokens per inference call than a binary-classification model of comparable size. The paper acknowledges this partially: "Although this design substantially improves trustworthiness and explainability, it introduces additional inference overhead compared to models of similar size that output only predicted labels" (Limitations section). However, it provides no latency or throughput measurements whatsoever. Table 3 reports only cost ($0.10 for 1.2K samples), which accounts for total GPU time but obscures per-query wall-clock time.

For real-time applications—a user waiting for a hallucination check before displaying a RAG response, or a streaming system that must verify claims before they reach the user—the sequential generation of CoT + explanation + prediction may impose unacceptable latency even if the total cost is low. A binary classifier like MiniCheck outputs a single token (or a short score) in one forward pass; FaithLens must autoregressively generate potentially hundreds of tokens across three structured sections. The paper does not compare the average number of output tokens FaithLens produces per instance against baselines, nor does it measure end-to-end latency on fixed hardware. A deployment that parallelizes multiple FaithLens calls (e.g., verifying multiple claims from one document) could achieve high throughput by batching, but per-claim latency would still be bounded by the sequential generation length.

What evidence exists in the paper: Almost none. The Limitations section mentions the overhead in one sentence. No latency measurements, token-count comparisons, or throughput benchmarks are reported. The variant methods testing (Appendix J, Question 1) quantifies the accuracy cost of removing CoT but not the latency savings. A practitioner evaluating FaithLens for a latency-sensitive application cannot determine from the paper whether the sequential generation makes the model unsuitable.

Mitigation status: Acknowledged as a limitation but not measured or mitigated. The paper frames this as a deliberate choice: the sequential design is essential for trustworthiness and explainability, and the tradeoff is accepted. However, without quantifying the tradeoff (how many milliseconds or tokens does explainability cost?), the acknowledgment is qualitative. Future work on speculative decoding, parallel CoT/explanation generation, or adaptive early-exit strategies could address this, but none are explored.

The Training Pipeline Has Not Been Tested Beyond Textual Faithfulness Hallucination Detection on English Benchmarks

All experiments in the paper use English-language textual benchmarks from two sources: LLM-AggreFact (11 tasks spanning summarization, QA, RAG, dialogue, and fact verification, all in English) and HoVer (multi-hop fact verification on English Wikipedia). The model backbone is English-centric (Llama-3.1-8B-Instruct and Qwen-2.5-Inst, both primarily trained on English data). The paper does not evaluate on multilingual data, multimodal settings (e.g., checking whether an image caption is faithful to the image), or substantially different hallucination patterns (e.g., code hallucination, where "faithfulness" would mean consistency with a specification or test suite rather than a natural-language document).

The consequence is that FaithLens's effectiveness on non-English text, cross-lingual document-claim pairs, long documents exceeding the model's context window, or multimodal faithfulness is completely unknown. The paper's claim to "cost-efficient and effective faithfulness hallucination detection" (Abstract) is empirically validated only within the narrow domain of English textual faithfulness evaluation. The limitation is more acute because FaithLens is a fine-tuned specialized model, not a general-purpose LLM that might transfer zero-shot capabilities across languages or modalities. The entire training pipeline—synthetic data from DeepSeek-V3.2-Think, data filtering using Llama-3.1-8B-Instruct's perplexity, RL with a novice model from the same family—is English-centric and would require substantial re-engineering (new base models, new embedding models for diversity filtering, possibly new synthetic data generation prompts) to adapt to other languages or modalities.

What evidence exists in the paper: The authors explicitly restrict their scope in the Limitations section: "we focus exclusively on textual faithfulness hallucination detection and do not address multi-modal settings. Extending our FaithLens to multi-modal settings would require fundamentally different grounding signals and explanation formats, which are beyond the scope of this study. To ensure the comparability with prior work, we therefore restrict our investigation to the textual domain." This is transparent and well-justified—the prior work the paper compares against (MiniCheck, FactCG, ClearCheck) is similarly text-only—but it means the paper's contribution is, by construction, domain-specific. No cross-lingual evaluation is mentioned or attempted, despite the generalization experiments (Table 7) testing different model families but only on the same English benchmarks.

Mitigation status: Explicitly deferred. The paper treats multimodal and multilingual extension as out of scope and does not propose concrete approaches for adaptation. A practitioner needing hallucination detection for non-English or multimodal content cannot use FaithLens without substantial additional research.

The Data Filtering Strategy's Cost Is Not Amortized in the Efficiency Calculations

The paper's headline efficiency claim—FaithLens costs 0.10for1.2Kinferencesamplesversus0.10 for 1.2K inference samples versus 15.30 for GPT-5.2 (Table 3)—accounts only for inference. It does not amortize the one-time costs of the training pipeline: generating 52,268 synthetic samples from DeepSeek-V3.2-Think, computing perplexity on all candidates for explanation quality and data diversity filtering (which requires multiple forward passes through Llama-3.1-8B-Instruct per candidate), running K-Medoids clustering on 14,258 embedding vectors, and performing SFT and RL training across multiple GPUs for multiple epochs. These are substantial computational costs, and for a deployment where the inference volume is modest, the training cost could dominate the total cost of ownership.

The consequence is that the cost-efficiency comparison in Table 3 is a marginal inference cost comparison, not a total cost of ownership comparison. A team that needs to detect hallucinations on 10,000 queries total would find that FaithLens's training cost (XforSFT+RL+syntheticdatageneration)plusX for SFT + RL + synthetic data generation) plus 0.83 for inference might exceed simply paying GPT-5.2 127.50forAPIcalls.Thepaper′scostadvantageismostcompellingathighinferencevolumes,wherethefixedtrainingcostisamortizedovermanyqueries.Butthepaperprovidesnobreakevenanalysis:atwhatinferencevolumedoesFaithLensbecomecheaperthanAPI−basedalternatives,accountingfortrainingcosts?Withoutthis,the"127.50 for API calls. The paper's cost advantage is most compelling at high inference volumes, where the fixed training cost is amortized over many queries. But the paper provides no breakeven analysis: at what inference volume does FaithLens become cheaper than API-based alternatives, accounting for training costs? Without this, the "0.10 vs. $15.30" framing is potentially misleading for low-to-medium volume deployments.

Furthermore, the data filtering process itself incurs costs that scale with the size of the unfiltered synthetic dataset. The paper generates 52,268 synthetic samples, filters them down to 11,929, and discards 77.2%. The discarded samples represent wasted DeepSeek-V3.2-Think API calls and wasted perplexity computations. If a practitioner wanted to apply FaithLens's pipeline to a new domain, they would need to pay these filtering costs upfront—generating and then discarding a large fraction of synthetic data—before obtaining a cleaned training set. The paper does not characterize the relationship between unfiltered dataset size, filtering discard rate, and final model quality, making it difficult to estimate the total synthetic data generation budget needed for a new domain.

What evidence exists in the paper: Table 3 reports only inference cost. Table 8 reports the number of examples at each filtering stage but not the computational cost of filtering. Appendix C reports that SFT uses 3 epochs on DeepSpeed+ZeRO3 with BF16 and that RL uses 2 epochs across 7 GPUs, but no total GPU-hours are reported. The cost of querying DeepSeek-V3.2-Think for 52,268 synthetic samples is not estimated. The reduced-data ablation (- w/o. Data Filtering in Table 5, 81.2 F1 and 82.3 explainability) shows that filtering is essential for performance, but does not address the cost of filtering itself.

Mitigation status: Partially addressed by the data efficiency framing. Table 4 emphasizes that FaithLens uses fewer training examples (28,643) than AlignScore (4,700K) or ClearCheck (82K), which could be interpreted as lower training cost. But this comparison excludes the synthetic data generation and filtering costs, which are unique to FaithLens's pipeline and not incurred by models trained directly on existing labeled data. The paper does not discuss amortization, breakeven points, or the one-time vs. recurring cost distinction.

FaithLens Has Only Been Validated on a Single, Fixed Test Set of 1,200 Samples Across 12 Tasks with No Statistical Significance Testing

The entire effectiveness evaluation of FaithLens rests on 12 test sets from the cleaned LLM-AggreFact and HoVer benchmarks (Section 4.1), totaling approximately 1,200 samples. The paper reports macro-F1 per dataset and an overall average with standard deviation (Table 1), but provides no confidence intervals, no significance tests, and no bootstrap estimates of the uncertainty around the reported means. When FaithLens achieves 86.4 F1 versus GPT-5.2's 86.1—a difference of 0.3 points—the reader cannot determine whether this difference would replicate on a different sample of test questions or is within the range of sampling variability.

The consequence is that the paper's strongest claim—"FaithLens achieves state-of-the-art performance, even surpassing advanced LLMs such as GPT-5.2 and o3" (Abstract)—is empirically underdetermined for the specific comparison to GPT-5.2, where the margin is razor-thin. A bootstrap confidence interval might well show that the 86.4 vs. 86.1 difference is not statistically significant at conventional levels (p < 0.05), which would mean the correct interpretation is "FaithLens is statistically tied with GPT-5.2 while being ~150× cheaper"—still a strong result, but a different claim than "surpasses." The o3 comparison (86.4 vs. 82.1, a 4.3-point gap) is more likely to be significant, but without formal testing, this is speculative.

The small test-set sizes also affect the reliability of the per-task breakdowns used in the ablation studies (Table 9). With 12 tasks and a total of ~1,200 samples, each task likely contains only ~100 test examples on average. An ablation that shifts F1 by 2–3 points on a single task is based on a small number of classification decisions (a few misclassified examples changing category) and may be noisy. The paper does not report per-task sample sizes, making it impossible for readers to assess the statistical reliability of per-task comparisons.

What evidence exists in the paper: The paper reports only point estimates (mean macro-F1 and standard deviation across tasks). It does not mention statistical testing, confidence intervals, or bootstrap methods anywhere. The cross-validation strategy discussed in Section 3.2 (two-fold cross-validation within difficulty bins for compute-optimal strategy selection) is from the example summary and is not present in the FaithLens paper—this paper uses a fixed train/test split based on the cleaned benchmarks from Seo et al. (2025) without cross-validation. The authors note that they "infer our model twice to obtain stable results" (Appendix C), which addresses inference stochasticity but not sampling uncertainty over the fixed test set.

Mitigation status: Not addressed. The paper reports raw numbers and interprets them as definitive rankings. The Limitations section does not mention test-set size or statistical reliability as a concern. For readers evaluating whether to adopt FaithLens over API-based alternatives, the lack of significance testing introduces uncertainty into the central performance claim. This is a notable gap given the narrow margin over GPT-5.2 and the field's increasing emphasis on rigorous evaluation with confidence intervals.

7. Implications and Future Directions

How This Work Changes the Landscape

FaithLens shifts the conversation around hallucination detection from a binary classification problem with opaque outputs toward a joint prediction-and-explanation task where explanation quality is a first-class optimization objective. Before this work, the dominant paradigm treated hallucination detection as a throughput optimization problem—build the most accurate classifier at the lowest inference cost—with explainability either ignored entirely (MiniCheck, FactCG, AlignScore) or treated as an incidental byproduct of chain-of-thought reasoning (ClearCheck). FaithLens demonstrates that explainability can be explicitly optimized through a composite reward signal without human-labeled explanation data, and that doing so improves rather than sacrifices prediction accuracy.

The conceptual reframing is this: an explanation should transfer knowledge—it should enable a weaker observer to make the correct judgment. This operationalization, instantiated through the novice-model-based explanation quality reward (Equation 14), transforms explanation generation from a text-similarity imitation task (SFT on synthetic CoT/explanation traces) into a causal transfer task (optimizing for whether the explanation changes the novice model's prediction). This is not merely a new metric for evaluating explanations; it is a trainable reward signal that can be computed automatically, requiring no human annotation. The critical empirical finding supporting this reframing is the ablation in Table 5: removing the explanation quality reward drops explainability from 90.4 to 84.7 while leaving effectiveness nearly unchanged (86.4 → 85.7). This cleanly isolates the explanation quality reward as the mechanism that converts accurate internal reasoning into externally useful explanations—accuracy alone does not produce good explanations, even when the model is generating explanation-like text.

The paper also resolves a subtle tension in prior work. Seo et al. (2025) found that "a small fine-tuned model underperforms larger models by a huge margin, particularly for instances requiring complex reasoning," suggesting that compact models hit a capability ceiling on tasks like HoVer. FaithLens breaks through this ceiling: its HoVer F1 of 82.9 surpasses GPT-5.2 (82.9, a tie), o3 (81.1), and ClearCheck (80.3), while its cross-task standard deviation of 4.6 is the lowest among all evaluated systems. The enabling mechanism is not larger model scale or more training data—FaithLens uses fewer training examples (28,643) than ClearCheck (82K) or AlignScore (4.7M)—but rather the quality and complementarity of the training signal. The three-dimensional data filtering (Section 3.1.2) explicitly curates training data to be label-correct, explanation-informative, and distributionally diverse, while the RL stage provides separate optimization pressure for accuracy and explanation quality. This suggests that the "small model ceiling" observed in prior work was at least partly an artifact of suboptimal training pipelines, not an inherent limitation of model scale.

The research direction this work makes more attractive is clear: distilling structured reasoning capabilities from large reasoning models into compact, specialized models through multi-objective RL. FaithLens demonstrates this recipe for hallucination detection; the natural question is whether it generalizes to other reasoning tasks where correctness can be verified automatically (code verification against test suites, mathematical proof checking, logical entailment verification) and where the reasoning can be serialized into CoT + explanation text. The direction FaithLens makes less attractive is scaling model size as the primary route to better hallucination detection: at 8B parameters, FaithLens matches or exceeds models that are orders of magnitude larger (GPT-5.2, o3) at two orders of magnitude lower inference cost. For practitioners, the implication is that investing in training methodology—specifically, synthetic data filtering and multi-objective RL with knowledge-transfer rewards—may yield higher returns than investing in larger models.

However, the paper also identifies a boundary on this shift. The entire training pipeline depends on access to ground-truth binary labels for filtering and reward computation. In domains where such labels are unavailable or expensive, the recipe cannot be applied directly. This means the paper's contribution moves the bottleneck from human annotation of explanations to human annotation of binary labels—a less expensive bottleneck, but not an eliminated one. The shift is therefore best understood as making explanation generation tractable without human effort, while still requiring human-verified labels for the underlying classification task. This is a meaningful but bounded advance.

Follow-Up Research This Work Enables

Extending the knowledge-transfer reward to multi-class or structured hallucination taxonomies. FaithLens's explanation quality reward (Equation 14) is defined over a binary prediction: the novice model either predicts the correct label or not. A natural extension is to define a similar reward for multi-class error-type classification, where the explanation must enable the novice model to identify not just that a claim is hallucinated but how (contradiction, unsupported claim, factual error, temporal mismatch, etc.). The paper acknowledges that "current datasets lack a unified taxonomy for such distinctions" (Limitations). A strong follow-up would first construct a multi-class hallucination dataset with verified error-type labels across multiple tasks, then train a FaithLens variant with a reward defined as R_exp = 1 if the novice model predicts the correct error category. The key question is whether explanations that transfer binary knowledge also transfer categorical knowledge, or whether more detailed explanations are needed and whether the RL signal is strong enough to optimize for that additional precision. This would extend the paper's framework to the more practically useful setting of actionable hallucination feedback.

Testing whether the explanation quality reward genuinely measures evidence grounding or merely exploits novice-model shortcuts. The paper acknowledges this as a potential confound in the Limitations discussion, but does not test it. A rigorous follow-up would construct adversarial explanations that are designed to trigger the novice model's correct predictions without citing genuine evidence from the document—for instance, explanations that always claim "the document explicitly contradicts the claim" when the label is "hallucinated," regardless of the document content. If such vacuous explanations consistently receive high explanation quality rewards, the reward signal is measuring surface-level heuristic matching, not genuine knowledge transfer. Conversely, if the RL training learns to avoid such shortcuts (perhaps because GRPO encourages diversity and vacuous explanations perform poorly on novel probe examples during data diversity filtering), this would strengthen the paper's central claim about the reward design. The experiment would also involve human evaluation of explanations produced under adversarial training conditions to determine whether automatic reward optimization aligns with human judgments of quality in edge cases.

Combining FaithLens's explanation generation with interactive error correction in RAG systems. FaithLens currently produces static explanations: given a document and claim, it outputs a judgment and an explanation, end of interaction. An ambitious extension would embed FaithLens in an interactive RAG pipeline where a user, upon receiving a "hallucinated" judgment with an explanation, can ask follow-up questions ("Which part of the document contradicts this specific phrase?" or "What would a faithful version of this claim look like?") and FaithLens (or a downstream model) generates targeted responses. The paper's explanation quality reward—optimizing for text that enables a weaker model to answer correctly—is directly extensible to this setting by defining a multi-turn reward where the novice model must answer follow-up questions correctly after reading the explanation chain. The challenge would be constructing training data for multi-turn explanatory dialogue without human annotation, perhaps by having DeepSeek-V3.2-Think generate multi-turn explanations for existing (document, claim) pairs and filtering using the same three-dimensional strategy extended to multi-turn coherence.

Evaluating FaithLens on out-of-distribution hallucination patterns: cross-lingual, long-document, and multimodal. The paper restricts evaluation to English text from LLM-AggreFact and HoVer. A stress-test follow-up would evaluate FaithLens on: (a) cross-lingual document-claim pairs (e.g., a German document with an English claim generated by a multilingual LLM), testing whether the embedding-based data diversity filter and novice-model reward transfer across languages; (b) long-document faithfulness (e.g., checking claims against 50+ page PDFs), where the document exceeds the model's context window and the explanation must localize evidence within specific passages—this would test whether FaithLens's structured generation format degrades under document-truncation strategies; (c) multimodal faithfulness (e.g., checking whether an image caption is faithful to the image), which the paper explicitly defers but which would test the limits of the text-only pipeline. The key measurement would be the performance drop-off as the domain shifts from in-distribution English text to these harder settings. A large drop-off would delineate the boundaries of the paper's method; a small drop-off would suggest the filtering and reward strategies capture something fundamental about explanation quality rather than task-specific heuristics.

Scaling the training pipeline to use multiple diverse synthetic data generators. FaithLens uses a single model (DeepSeek-V3.2-Think) for synthetic data generation. The data diversity filter (Section 3.1.2) ensures diversity among the surviving examples but cannot introduce reasoning patterns or explanation styles that the generator never produces. A natural extension is to synthesize training data from multiple large reasoning models with different architectures and training paradigms (e.g., DeepSeek-V3.2-Think, o3 if CoT access becomes available, Claude-3.7-Sonnet with prompting to expose reasoning) and pool the results before filtering. This would produce a training set with genuinely diverse reasoning strategies, which the diversity filter would then curate for complementarity. The hypothesis is that models trained on multi-source synthetic data would achieve even lower cross-task variance (already at 4.6 for FaithLens) and better generalization to entirely novel hallucination patterns that no single generator encountered during its own training. The experiment would require controlled comparisons: FaithLens trained on single-source vs. multi-source data, evaluated on a held-out task not represented in any generator's training distribution.

Measuring and reducing the cost of the data filtering pipeline. The paper reports only final training data counts (Table 8) and inference costs (Table 3), but not the computational cost of the filtering pipeline itself—generating 52,268 synthetic samples, computing perplexity for explanation quality and data diversity filtering, and running K-Medoids clustering. A practically important follow-up would quantify these costs in GPU-hours and API dollars, then investigate cheaper proxies: can a lightweight classifier predict which synthetic samples will survive filtering, avoiding the need to generate full CoT + explanation + perplexity computation for samples likely to be discarded? Can the data diversity filter be approximated by computing embeddings only (skipping perplexity-based complementarity testing) and subsampling clusters? The goal would be to reduce the filtering overhead to make the FaithLens pipeline practical for small teams or new domains without access to large compute budgets. The paper's parameter study showing robustness to the number of clusters K (Tables 11–12) suggests some components may be simplifiable without major performance loss.

Practical Applications and Downstream Use Cases

Cost-sensitive large-scale RAG verification pipelines. FaithLens's most directly actionable application is in RAG systems that must verify the faithfulness of LLM-generated responses against retrieved documents at scale. Consider a customer support system that generates hundreds of thousands of responses per day, each containing multiple factual claims that must be verified before being shown to users. Using GPT-5.2 at 15.30per1.2Ksamples(Table3)wouldcostapproximately15.30 per 1.2K samples (Table 3) would cost approximately 12,750 per million claims. FaithLens at 0.10per1.2Ksamplesbringsthistoapproximately0.10 per 1.2K samples brings this to approximately 83 per million claims—a more than 150× reduction. The 86.4 F1 accuracy (Table 1) means that automated verification would catch the vast majority of hallucinations, with the remaining errors being false negatives that a human reviewer could audit on a sampled basis. The explanations generated by FaithLens (rated 90.4 on average by GPT-4.1 as judge, Table 2) provide the audit trail needed for compliance and quality improvement: when a claim is flagged as hallucinated, the explanation cites specific document evidence, enabling content moderators to quickly verify the system's judgment without re-reading the entire document. This combination of cost, accuracy, and explainability makes FaithLens deployable in production settings where API-based alternatives would be economically prohibitive.

Explainable hallucination detection in high-stakes domains with privacy constraints (legal, medical). In legal document review or medical report verification, claims must be checked against sensitive source documents that cannot be sent to external APIs. FaithLens runs entirely locally as an 8B-parameter model on consumer or datacenter GPUs, satisfying data residency and privacy requirements. More importantly, the binary "hallucinated" label is insufficient in these domains—a lawyer needs to know which clause of a contract contradicts a claim, and a doctor needs to know which lab result the claim misinterprets. FaithLens's explanations (rated 93.4 for helpfulness and 85.4 for informativeness in Table 2) provide this specificity, citing evidence from the grounding document. The paper's case study from LLM-AggreFact (Figure 16, Appendix K) illustrates this pattern: FaithLens's explanation for a hallucination about the Lanham Act explicitly lists the statutes that are mentioned in the document (Truth in Lending Act, Fair Credit Reporting Act, etc.) and notes that the Lanham Act is absent, providing a verifiable evidence trail that a domain expert can confirm without re-examining the full document. For legal and medical applications, this specificity is a hard requirement, not a nice-to-have—and FaithLens is, as of the paper's publication, the only compact model that provides it without sacrificing accuracy.

Data generation for self-improving RAG systems. FaithLens's explanation generation capability enables a self-improvement loop for RAG systems. A deployed RAG model generates a response; FaithLens checks each claim in the response against the retrieved documents, producing binary judgments and explanations; the explanations are then used to construct corrected training examples for fine-tuning the RAG model to produce more faithful responses. For instance, if FaithLens flags a claim as hallucinated and explains "the document states the meeting was in Paris, but the claim adds 'on Tuesday' which is not supported," this explanation can be used to generate a corrected claim ("the meeting was in Paris") and a training pair (document, corrected claim) for the RAG model. The key enabler is that FaithLens's explanations are sufficiently specific to guide correction—the 85.4 informativeness score (Table 2) indicates that explanations contain detailed evidence citations, not just generic flags. This application directly extends the paper's framework: FaithLens, trained to transfer knowledge to a weaker model, would transfer knowledge about hallucination patterns to a RAG model through corrected training data. The efficiency numbers are compelling: at 0.10per1.2Kclaims,generating100Kcorrectionpairswouldcostapproximately0.10 per 1.2K claims, generating 100K correction pairs would cost approximately 8.30, making large-scale self-improvement economically feasible.

When to Prefer This Method

The paper does not articulate an explicit tradeoff matrix against named alternatives, so this section is not included. The paper's positioning is that FaithLens is simultaneously more accurate, more explainable, and cheaper than existing options for textual faithfulness hallucination detection—it is framed as a Pareto improvement rather than a method with clear "prefer A when X, prefer B when Y" tradeoffs. The limitations section acknowledges that FaithLens incurs additional inference latency (not benchmarked), requires ground-truth labels for training (implicit), and is restricted to text-only binary classification—but these are presented as scope limitations of the current work, not as conditions under which a different method would be preferable. A decision rule would therefore require speculation beyond what the paper provides.