ArXiv: 2309.03883

🎯 Pitch

Large language models store factual knowledge in specific later layers that earlier layers suppress—DoLa simply subtracts early-layer logits from later-layer logits during decoding, boosting truthfulness on TruthfulQA by 12–17 absolute points across LLaMA model sizes with no training or retrieval overhead.

1. Executive Summary

This paper proposes Decoding by Contrasting Layers (DoLa), a simple inference-time decoding strategy for reducing hallucinations in pretrained large language models without external knowledge retrieval or additional fine-tuning. The approach obtains next-token probabilities by contrasting the logit differences between a dynamically selected premature layer and the final mature layer of the same transformer model—subtracting the lower-layer distribution from the higher-layer distribution and applying an adaptive plausibility constraint to filter implausible tokens—exploiting the empirical finding that factual knowledge is progressively encoded across later transformer layers while syntactic patterns dominate earlier layers. Evaluated on LLaMA family models (7B through 65B) across TruthfulQA, FACTOR, StrategyQA, and GSM8K, DoLa improves TruthfulQA open-ended generation truthfulness scores by 12–17 absolute percentage points and boosts Vicuna QA chatbot ratings in pairwise GPT-4 evaluations, while incurring only a 1–8% latency overhead during decoding, establishing that contrasting internal layer representations can reliably surface factual knowledge embedded within LLMs without modifying model weights—though this mechanism depends critically on large models storing distinct factual knowledge across layers, as the same approach fails to improve factuality when applied to smaller models like GPT-2 Medium (335M parameters).

2. Context and Motivation

The Core Problem: LLMs Hallucinate Factual Content Despite Strong Capabilities

The paper addresses a persistent and well-documented problem: large language models (LLMs) generate content that deviates from real-world facts observed during pretraining—a phenomenon known as "hallucination" (Ji et al., 2023). While LLMs have demonstrated remarkable capabilities across natural language processing tasks (Brown et al., 2020; OpenAI, 2022; 2023), their tendency to fabricate plausible-sounding but incorrect information represents "a major bottleneck in their deployment especially for high-stakes applications (e.g., clinical/legal settings) where reliable generation of trustworthy text is crucial" (Section 1).

This problem is not merely academic. Consider the qualitative example from Table 3 where LLaMA-33B confidently answers "The Declaration of Independence was officially signed on July 4, 1776"—a plausible but incorrect date that many humans would also get wrong. The actual signing date was August 2, 1776. An LLM deployed in an educational or legal context making this error could propagate misinformation at scale. Similarly, the model falsely claims "You should wait 24 hours before filing a missing person report," which could have genuinely harmful consequences if someone followed this advice in a real emergency. These are not edge cases—they represent systematic failures in how LLMs access and deploy their internal factual knowledge during generation.

The paper frames the hallucination problem through the lens of the language modeling objective itself. As they note in Section 1:

"While the exact reasons for LMs' hallucinations are not fully understood, a possible reason is due to the maximum likelihood language modeling objective which minimize the forward KL divergence between the data and model distributions. This objective potentially results in a model with mass-seeking behavior which causes the LM to assign non-zero probability to sentences that are not fully consistent with knowledge embedded in the training data."

This is a crucial insight: the standard next-token prediction training objective encourages models to place probability mass across many plausible continuations, including factually incorrect ones, because the objective penalizes assigning zero probability to any observed sequence. An LM trained this way on finite data "has been shown to result in a model that uses linguistic knowledge to recognize the superficial patterns, instead of recognizing and generating the real-world facts extracted from the training corpus" (Section 1). In other words, the model learns to be a good language model—producing grammatically fluent, contextually appropriate text—but not necessarily a truthful one.

Why This Problem Matters: Deployment and Trust

The practical importance of reducing hallucinations extends across multiple dimensions that the paper touches on:

High-stakes applications are blocked. Clinical decision support, legal document analysis, financial reporting, and scientific research assistance all require outputs that are not merely fluent but factually reliable. A model that produces confident-sounding medical advice or legal citations that are subtly wrong cannot be deployed in these settings regardless of how impressive its linguistic capabilities are. The hallucination problem is not a minor quality-of-life issue—it is a fundamental barrier to LLM adoption in consequential domains.

Scale doesn't solve the problem. The paper operates in the context of LLaMA family models spanning 7B to 65B parameters. Even the 65B model achieves only 31.1% on TruthfulQA MC1 with DoLa applied (Table 1), up from 30.8% without it—an improvement but hardly a solved problem. This is despite these models having been trained on massive corpora containing the relevant factual knowledge. The fact that scaling from 7B to 65B yields relatively modest truthfulness gains (25.6% → 30.8% on TruthfulQA MC1 for vanilla decoding, Table 1) suggests that simply making models larger does not automatically make them more truthful, even though larger models demonstrably encode more factual knowledge in their parameters.

The gap between knowing and expressing. The paper's central motivating observation is that LLMs appear to encode factual knowledge in their parameters but fail to reliably express it during generation. The DoLa approach is built on the finding that this knowledge is hierarchically organized across transformer layers—present in the model's representations but sometimes overridden or diluted by lower-level linguistic patterns during the decoding process. Closing this "knowing vs. expressing" gap would allow models to better leverage knowledge they already possess, without requiring additional training or external knowledge sources.

Resource constraints in deployment. Many real-world deployments cannot afford retrieval-augmented systems that query external knowledge bases at inference time—these add latency, infrastructure complexity, and dependency on maintained knowledge sources. A decoding-time intervention that works with any pretrained model, requires no additional modules, and adds only 1–8% latency overhead (Table 2) is immediately deployable in resource-constrained settings where retrieval is impractical.

Prior Approaches and Their Limitations

The paper situates itself relative to several existing strategies for mitigating hallucinations, each of which has significant drawbacks that DoLa avoids:

Reinforcement Learning from Human Feedback (RLHF). Approaches like InstructGPT (Ouyang et al., 2022) fine-tune models using human preference judgments to align outputs with human values, including truthfulness. While effective, this approach requires substantial human annotation effort, is expensive to implement, and ties the model's truthfulness to the quality and coverage of the human feedback data. It is a training-time solution—once the model is deployed, no further truthfulness improvements are possible without retraining.

Inference-time intervention with supervised classifiers. Li et al. (2023) propose Inference Time Intervention (ITI), which trains a linear classifier on TruthfulQA labels and uses it to shift attention head activations during inference. The paper includes ITI as a baseline (Table 1) and acknowledges it achieves competitive truthfulness scores—for example, ITI achieves 43.5% on TruthfulQA open-ended %Truth∗Info versus DoLa's 40.8% on LLaMA-7B. However, ITI requires supervised training on labeled data from the target task, making it task-specific and dependent on the availability of truthfulness annotations. It cannot be applied zero-shot to new domains or tasks where such labels don't exist.

Contrastive Decoding (CD). Li et al. (2022) propose contrasting the output distributions of a strong "expert" LM with a weaker "amateur" LM to improve generation quality. The paper includes CD as a primary baseline (Table 1), using LLaMA-7B as the amateur and larger LLaMA variants as experts. However, CD shows mixed results on factuality metrics: on TruthfulQA open-ended generation, CD boosts truthfulness but dramatically increases the rate of "I have no comment" rejections—over 60% for LLaMA-33B (Table 1), dropping the combined %Truth∗Info score. The paper hypothesizes this is because "CD uses LLaMA-7B for contrast, and a big difference is that 33B is better at instruction-following than 7B, explaining why CD frequently answers 'I have no comment,' as this response is indicated in the instruction prompt" (Section 3.3). Moreover, CD requires access to a separate, smaller model to serve as the amateur, and the choice of amateur model size is critical and task-dependent—the paper's exploration in Appendix B (Table 5) shows that even with extensive experimentation across OpenLLaMA and Sheared-LLaMA variants at sizes from 1.3B to 7B, CD rarely outperforms the vanilla baseline on GSM8K and consistently falls behind DoLa.

Context-Aware Decoding (CAD). Shi et al. (2023) build on the CD framework to better focus LMs on input context for summarization and knowledge conflict tasks. This work shares the contrastive spirit but addresses a different mechanism—using context to override parametric knowledge—while DoLa aims to surface parametric knowledge that is already present but not expressed.

Multi-agent debate and self-consistency. Approaches like Du et al. (2023) and Liang et al. (2023) have LMs debate each other or themselves to converge on more factual answers. While promising, these methods multiply inference cost by the number of debate participants and iterations, making them expensive for deployment.

External knowledge retrieval. Retrieval-augmented models like RETRO (Borgeaud et al., 2022) and Atlas (Izacard et al., 2022) ground generation in retrieved documents. These are powerful but require maintaining a retrieval index, add latency from the retrieval step, and cannot correct misinformation that exists in the retrieved corpus itself. As the paper notes in its limitations (Section 6), DoLa "relies on the model's internal knowledge without using external retrieval modules... Thus, it cannot correct misinformation acquired during training." This is a genuine limitation, but it also means DoLa is complementary to retrieval approaches rather than competing with them.

Where Existing Approaches Fall Short: A Synthesis

The paper identifies a specific gap that none of the above approaches address: how to better surface factual knowledge that is already encoded within a single pretrained LLM, at decoding time, without any additional training, labeled data, external models, or retrieval systems. Each prior approach requires at least one of these costly elements:

  • RLHF: Requires human annotation and training-time modification
  • ITI: Requires task-specific supervised classifiers trained on labeled data
  • CD: Requires a separate amateur model; performance is highly sensitive to amateur model selection
  • CAD: Addresses context reliance, not internal factuality
  • Multi-agent debate: Multiplies inference cost
  • Retrieval: Requires maintaining external knowledge infrastructure

DoLa positions itself to fill exactly this gap: a pure decoding strategy that operates on a single model, requires zero training, works across tasks without task-specific validation data (the dynamic layer selection is designed to be robust without in-distribution validation), and adds negligible latency.

The Layer-Wise Knowledge Encoding Hypothesis

The paper's approach is motivated by a specific empirical observation about how transformer LMs encode information across layers. Drawing on the interpretability literature, the authors note:

"From a model interpretability perspective, transformer LMs have been loosely shown to encode 'lower-level' information (e.g., part-of-speech tags) in the earlier layers, and more 'semantic' information in the later layers (Tenney et al., 2019). More recently, Dai et al. (2022) find that 'knowledge neurons' are distributed in the topmost layers of the pretrained BERT model. Meng et al. (2022) show that factual knowledge can even be edited by manipulating a specific set of feedforward layers within an autoregressive LM." (Section 1)

This hierarchical encoding—syntax in early layers, semantics and facts in later layers—is the key insight that DoLa exploits. The paper provides both qualitative and quantitative evidence for this pattern in autoregressive LMs:

Qualitative evidence (Figure 2). When decoding the answer to "Who was the first Nigerian to win the Nobel Prize, in which year?", the authors compute Jensen-Shannon Divergence (JSD) between early-layer output distributions and the final-layer output distribution at each token position. They observe two distinct patterns:

  • Pattern #1 (factual tokens): When predicting important named entities and dates ("Wole Soyinka," "1986"), the JSD remains extremely high even in the higher layers. This indicates "the model is still changing its predictions in the last few layers, and potentially injecting more factual knowledge into the predictions" (Section 2.1).

  • Pattern #2 (function words and copied tokens): When predicting function words ("was," "the," "to," "in") or tokens copied from the input ("first Nigerian," "Nobel Prize"), the JSD becomes very small from the middle layers onward. This indicates "the model has already decided what token to generate in middle layers, and keeps the output distributions almost unchanged in the higher layers" (Section 2.1).

Quantitative evidence (Appendix A, Table 4). Using the CoNLL-2003 named entity recognition dataset with teacher forcing, the authors compute which layer has the largest JSD with the final layer for each token prediction, stratified by whether the token is an entity or non-entity:

  • For non-entity tokens, 75.55% of the time the "critical layer" (layer with maximum divergence from the final layer) is layer 0 (the embedding layer), meaning the final-layer distribution is already similar to the embedding-level distribution.
  • For entity tokens, only 35.56% of the time the critical layer is layer 0, while more than 50% of the time the critical layer is at layer 20 or above (with 25.40% at layer 30, the second-to-last layer for the 32-layer model).

This quantitative evidence supports the claim that factual (entity-related) predictions undergo significant distribution shifts in the very highest layers, while non-factual predictions stabilize much earlier.

The motivating intuition (Figure 1). The paper's central example illustrates the contrastive decoding idea: when asked "Where is the capital of Washington State?", the model assigns high probability to "Seattle" across all layers because it is a syntactically and contextually plausible answer. However, the probability of the correct answer "Olympia" gradually increases from lower to higher layers as more factual knowledge is incorporated. By the 32nd (final) layer, "Olympia" has overtaken "Seattle" as the highest-probability token, but the margin may be small. Contrasting the final layer distribution against an early layer distribution—subtracting the early-layer logits where "Olympia" was much less probable—can sharpen this signal and make the correct factual answer the clear winner.

How DoLa Positions Itself

DoLa is positioned as a simple, practical, and training-free decoding intervention that:

  1. Operates on a single pretrained model without requiring an amateur model (unlike CD), labeled data (unlike ITI), or human feedback (unlike RLHF).

  2. Dynamically selects the contrastive layer per token using JSD between early-layer and final-layer output distributions, avoiding the need for task-specific hyperparameter tuning across all possible layers. The authors explicitly note that "DoLa-static has the drawbacks of 1) requiring more hyperparameter search runs in layers and the fact that 2) best layers are sensitive to data distribution, thus requiring in-distribution validation sets. Our proposed dynamic layer selection strategy also mitigates the drawbacks of DoLa-static by shrinking the layer search space and making the method more robust without heavily relying on in-distribution validation sets" (Section 2.2).

  3. Adds negligible overhead—1–8% latency increase and sub-1% memory overhead increase (Tables 2 and 9)—making it deployable in production systems without significant infrastructure changes.

  4. Works across model sizes and tasks, though with an important caveat: the approach requires large models where factual knowledge is distinctly localized to specific layers. As shown in Appendix N (Table 17), applying DoLa to GPT-2 Medium (335M parameters) actually decreases performance on TruthfulQA and FACTOR. The paper conjectures that "small LMs do not learn enough factual knowledge due to their limited capacities. Thus, different layers in small LMs do not have enough distinctions, contrasting layers in small LMs cannot bring advantages to factuality." This boundary condition is important—DoLa is not a universal fix but rather exploits a property that emerges at scale.

  5. Is complementary to other approaches. The paper is careful to frame DoLa not as a replacement for retrieval, RLHF, or ITI, but as a foundational improvement that could be combined with them. The Limitations section (Section 6) explicitly notes that "since our method provides a foundational improvement that could potentially be applied to any transformer-based LLMs, the limitations listed above could be potentially addressed through future work combining the corresponding elements with our decoding strategy."

The Relationship to Concurrent Work

The paper acknowledges two concurrent works that partially overlap with DoLa:

O'Brien & Lewis (2023) extend Contrastive Decoding to reasoning tasks. This is a natural extension of CD rather than a competing approach to factuality, and the paper's CD baselines already demonstrate that CD underperforms DoLa on factual reasoning tasks.

Autocontrastive Decoding (ACD; Gera et al., 2023) partially resembles DoLa-static but differs in crucial ways: it focuses on small LMs (GPT-2 335M/125M), requires fine-tuning prediction heads for early layers, and aims to improve diversity and coherence rather than factuality. The paper notes that "while the authors reveal ACD increases hallucinations in its limitation section, DoLa instead reduces them. We attribute the discrepancy to model sizes, as our experiments in Appendix N suggest contrasting layers in a small GPT2 cannot improve factuality. Large LLMs storing distinct knowledge across layers is key for DoLa to work" (Section 5). This is a pointed distinction: ACD and DoLa may be mechanistically similar (contrasting layers), but they operate in different regimes and produce opposite effects on factuality, highlighting that the layer-contrasting mechanism is not inherently truthfulness-promoting—it only becomes so when the model has structured its knowledge hierarchically across layers, a property that emerges at scale.

3. Technical Approach

3.1 Reader Orientation

This paper proposes a decoding-time intervention—a method applied during text generation that modifies how an LLM selects each next token, without changing the model's weights or requiring additional training. The system takes a single pretrained LLM and, at each generation step, computes two next-token probability distributions from different layers of the same model—one from the final layer (the "mature" layer) and one from a dynamically selected earlier layer (the "premature" layer)—then contrasts them by subtracting the premature layer's log probabilities from the mature layer's log probabilities. The problem it solves is: how to make an LLM more likely to output factually correct information that it already knows but sometimes fails to express, by suppressing the "easy" linguistic patterns that dominate early layers and amplifying the factual knowledge that emerges in later layers. The solution takes the shape of a simple arithmetic operation applied to the model's internal representations, with zero external dependencies.

3.2 Big-Picture Architecture (Diagram in Words)

The DoLa system has four major components operating sequentially at each token generation step:

  1. The Pretrained LLM (any transformer-based autoregressive model, e.g., LLaMA-7B with 32 layers) — processes the input context through all $N$ transformer layers, producing hidden states $H_0, H_1, ..., H_N$ at every layer, where $H_0$ is the embedding layer output and $H_N$ is the final hidden state. The model is frozen; no weights are modified.

  2. The Early-Exit Prediction Heads — the vocabulary projection head $\phi(\cdot)$ (the final linear layer that maps hidden states to vocabulary logits) is applied to hidden states from multiple intermediate layers $j \in \mathcal{J}$, not just the final layer. This produces "early-exit" next-token distributions $q_j(x_t | x_{<t})$ for each candidate layer. The projection head is also frozen; it is the same head used for the final layer, applied as-is to intermediate representations.

  3. The Dynamic Premature Layer Selector — at each time step, computes the Jensen-Shannon Divergence (JSD) between each early-exit distribution $q_j(\cdot | x_{<t})$ and the final-layer distribution $q_N(\cdot | x_{<t})$, selects the layer $M$ that maximizes this divergence within a pre-specified bucket of candidate layers, and designates it as the "premature layer." The final layer $N$ is the "mature layer."

  4. The Contrastive Decoding Module — takes the mature layer distribution $q_N$ and the premature layer distribution $q_M$, computes their log-ratio $\log(q_N / q_M)$, applies an adaptive plausibility constraint to zero out tokens with low mature-layer probability, optionally applies a repetition penalty, and softmaxes the result to produce the final next-token distribution $\hat{p}(x_t | x_{<t})$ from which the next token is sampled or argmaxed.

Information flows as follows: the input context $\{x_1, ..., x_{t-1}\}$ enters the frozen LLM → the model computes hidden states at all layers $H_0, ..., H_N$ in a single forward pass → the vocabulary head $\phi$ projects each candidate layer's hidden state to a probability distribution → the JSD-based selector identifies which early layer $M$ is most different from the final layer → the log-ratio $\log(q_N/q_M)$ is computed → the plausibility constraint filters implausible tokens → optional repetition penalty is applied → softmax produces the final distribution → the next token $x_t$ is selected → the token is appended to the context, and the process repeats for the next position.

3.3 Roadmap for the Deep Dive

  • First, the formal definition of the standard next-token prediction pipeline in a transformer LM, establishing the notation and showing where DoLa intervenes—this grounds the explanation in the standard architecture.
  • Second, the concept of early-exit distributions and the empirical evidence that factual knowledge evolves across layers, motivating why contrasting layers should help factuality.
  • Third, the dynamic premature layer selection mechanism—how JSD is computed, why maximizing divergence is the right criterion, and how candidate layers are bucketed into search spaces—since this is the novel adaptive component.
  • Fourth, the contrastive decoding formula itself, including the log-ratio operation, the adaptive plausibility constraint (what it prevents and why it is necessary), and the repetition penalty—since these are the mathematical operations that produce the final output distribution.
  • Fifth, the design choices around candidate layer bucketing (how layers are partitioned for different model sizes) and validation set usage, explaining the practical deployment procedure.
  • Sixth, the relationship to and differences from Contrastive Decoding (CD), since CD is the most direct prior work and understanding the distinctions clarifies what is novel.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a decoding method paper whose core idea is that contrasting output distributions from different layers of the same frozen LLM can amplify factual knowledge that is localized in later layers while suppressing syntactic patterns that dominate earlier layers, and that the layer to contrast against should be chosen dynamically per token based on distributional divergence.

Standard Next-Token Prediction in Transformer LMs

A standard autoregressive transformer LM consists of three sequential components. First, an embedding layer maps the input token sequence $\{x_1, x_2, ..., x_{t-1}\}$ to a sequence of continuous vector representations $H_0 = \{h_0^{(1)}, h_0^{(2)}, ..., h_0^{(t-1)}\}$, where each $h_0^{(i)} \in \mathbb{R}^d$ is the $d$-dimensional embedding of token $x_i$. Second, $N$ stacked transformer layers process these representations sequentially: layer 1 takes $H_0$ and outputs $H_1$, layer 2 takes $H_1$ and outputs $H_2$, and so on, through to the final layer which outputs $H_N = \{h_N^{(1)}, ..., h_N^{(t-1)}\}$. Each transformer layer applies multi-head self-attention followed by a feedforward network with residual connections, so $H_j$ can be understood as $H_{j-1}$ plus some incremental transformation. Third, a vocabulary projection head $\phi(\cdot)$—an affine transformation (linear layer plus bias) followed by softmax—maps the final hidden state at the last position $h_N^{(t-1)}$ to a probability distribution over the vocabulary $\mathcal{X}$:

p(xt∣x<t)=softmax(ϕ(hN(t−1)))xt,xt∈Xp(x_t | x_{<t}) = \text{softmax}\left(\phi(h_N^{(t-1)})\right)_{x_t}, \quad x_t \in \mathcal{X}

where $x_{<t}$ is shorthand for the sequence $\{x_1, ..., x_{t-1}\}$, $h_N^{(t-1)} \in \mathbb{R}^d$ is the final-layer hidden state at position $t-1$, $\phi: \mathbb{R}^d \to \mathbb{R}^{|\mathcal{X}|}$ is the vocabulary projection (a learned linear map), $\text{softmax}(z)_i = \exp(z_i) / \sum_j \exp(z_j)$ converts logits to probabilities, and the subscript $x_t$ indexes the probability assigned to token $x_t$ specifically.

What this computes: given all previous tokens, the model produces a probability for every possible next token in the vocabulary, representing how likely each token is to continue the sequence according to the model's learned distribution. A single token is then selected from this distribution (greedy argmax or sampling), and the process repeats autoregressively.

Why this form: the residual stream design means that $h_N^{(t-1)} = h_0^{(t-1)} + \sum_{j=1}^N \Delta h_j^{(t-1)}$, where $\Delta h_j$ is the contribution of layer $j$. Information from all layers is accumulated additively. The vocabulary head $\phi$ is trained to extract the best next-token prediction from this final accumulated representation. However, this accumulation means that both early-layer syntactic information and late-layer semantic/factual information are mixed together in $h_N$, and the final distribution $p(x_t)$ may not cleanly separate them.

Early-Exit Distributions and the Layer-Wise Knowledge Hypothesis

The key innovation in DoLa is to apply the vocabulary projection head $\phi(\cdot)$ not only to the final hidden state $h_N^{(t-1)}$ but also to hidden states from intermediate layers. For any layer $j \in \mathcal{J}$ (where $\mathcal{J}$ is a subset of candidate layers, $\mathcal{J} \subset \{0, 1, ..., N-1\}$), we compute an early-exit distribution:

qj(xt∣x<t)=softmax(ϕ(hj(t−1)))xt,j∈Jq_j(x_t | x_{<t}) = \text{softmax}\left(\phi(h_j^{(t-1)})\right)_{x_t}, \quad j \in \mathcal{J}

where $h_j^{(t-1)}$ is the hidden state at layer $j$ at position $t-1$, and $\phi$ is the exact same vocabulary head used for the final layer—it is not fine-tuned or adapted for intermediate layers.

What this computes: a next-token prediction based on the partial representation available at layer $j$, before the full stack of transformers has finished processing. This is what the model would predict if it were forced to stop early at layer $j$.

Why this works without special training: the residual connections in transformer layers mean that hidden representations "gradually evolve without abrupt changes" (Section 2), and prior work on early exiting (Teerapittayanon et al., 2016; Elbayad et al., 2020; Schuster et al., 2022; Kao et al., 2020) has shown that applying the final vocabulary head to intermediate layers yields meaningful—though not optimal—predictions. The residual stream $h_j$ already contains substantial information about what the next token should be, even at middle layers.

The paper's motivating empirical finding (Section 2.1, supported quantitatively in Appendix A) is that the gap between $q_j$ and $q_N$—how much the prediction changes after layer $j$—is systematically larger for tokens requiring factual knowledge than for tokens requiring only syntactic or local coherence. Specifically, when predicting function words or tokens copied from the input, $q_j$ converges to $q_N$ by the middle layers (JSD becomes small), indicating the model has "already decided" and the higher layers add little. When predicting named entities, dates, or other fact-laden tokens, $q_j$ continues to differ substantially from $q_N$ even in the very highest layers, indicating that the model is still "injecting more factual knowledge into the predictions" (Section 2.1) in the final few layers. This implies that the late-layer contributions $\Delta h_{M+1}, ..., \Delta h_N$ for some carefully chosen $M$ contain disproportionate amounts of factual knowledge relative to linguistic knowledge, and that isolating these contributions could amplify factual correctness.

Dynamic Premature Layer Selection via Jensen-Shannon Divergence

Rather than fixing a single premature layer for all tokens—which would be suboptimal because the layer at which factual knowledge emerges varies token by token—DoLa dynamically selects the premature layer at each decoding step. The selection criterion is based on the distributional distance between the final-layer distribution and each candidate early-exit distribution:

d(qN(⋅∣x<t),qj(⋅∣x<t))=JSD(qN(⋅∣x<t) ∣∣ qj(⋅∣x<t))d\left(q_N(\cdot | x_{<t}), q_j(\cdot | x_{<t})\right) = \text{JSD}\left(q_N(\cdot | x_{<t}) \,||\, q_j(\cdot | x_{<t})\right)

where $\text{JSD}(P \,||\, Q)$ is the Jensen-Shannon Divergence between two probability distributions $P$ and $Q$. The JSD is defined as:

JSD(P ∣∣ Q)=12KL(P ∣∣ M)+12KL(Q ∣∣ M)\text{JSD}(P \,||\, Q) = \frac{1}{2} \text{KL}(P \,||\, M) + \frac{1}{2} \text{KL}(Q \,||\, M)

where $M = \frac{1}{2}(P + Q)$ is the mixture distribution, and $\text{KL}(P \,||\, Q) = \sum_x P(x) \log \frac{P(x)}{Q(x)}$ is the Kullback-Leibler divergence.

What it computes: JSD measures the symmetric distance between two probability distributions over the vocabulary. It ranges from 0 (when the distributions are identical) to $\log 2$ (when they have disjoint support), and is symmetric—$\text{JSD}(P \,||\, Q) = \text{JSD}(Q \,||\, P)$—unlike KL divergence. A high JSD means the two distributions assign substantially different probabilities to many tokens; a low JSD means they are nearly the same.

Why JSD rather than KL divergence: the paper does not explicitly justify the choice of JSD over KL, but JSD has the practical advantage of being symmetric and bounded, which avoids issues when one distribution has near-zero probability on tokens where the other has non-zero probability (which would make KL diverge or be undefined). JSD is also the standard distributional distance used in the early-exiting literature.

The premature layer $M$ is then selected as the candidate layer that maximizes this divergence:

M=arg⁡max⁡j∈JJSD(qN(⋅∣x<t) ∣∣ qj(⋅∣x<t))M = \arg\max_{j \in \mathcal{J}} \text{JSD}\left(q_N(\cdot | x_{<t}) \,||\, q_j(\cdot | x_{<t})\right)

where $\mathcal{J}$ is a pre-specified bucket of candidate layers—not all possible layers, but a contiguous range (e.g., layers 0 through 15, or layers 16 through 31 for a 32-layer model).

What this computes: for each token position, the system finds the early-exit layer whose next-token prediction is most different from the final layer's prediction. This layer is designated as the "premature layer."

Why maximize divergence: the motivation (Section 2.2) is to "ensure that the model would significantly change its output after that selected layer, and thus have a higher chance to include more factual knowledge that does not exist in the early layers before it." The layer of maximum divergence is treated as the point just before the model injects substantial factual knowledge—the premature layer represents what the model would predict without the factual knowledge accumulated in the subsequent layers, while the mature layer represents the prediction with that knowledge. Contrasting these two distributions should therefore isolate the factual contribution.

Candidate layer buckets and model-size-specific partitioning. The paper does not allow the selector to choose from all possible layers $0, 1, ..., N-1$. Instead, layers are partitioned into contiguous buckets, and one bucket is pre-selected per task based on a validation set. The bucket structure is model-size-dependent:

  • LLaMA-7B (32 layers): 2 buckets — $[0, 16)$ and $[16, 32)$
  • LLaMA-13B (40 layers): 2 buckets — $[0, 20)$ and $[20, 40)$
  • LLaMA-33B (60 layers): 3 buckets — $[0, 20)$, $[20, 40)$, $[40, 60)$
  • LLaMA-65B (80 layers): 4 buckets — $[0, 20)$, $[20, 40)$, $[40, 60)$, $[60, 80)$

where layer 0 is the word embedding layer (the output of the embedding lookup, before any transformer layers). For efficiency, only even-indexed layers are considered as candidates within each bucket (0th, 2nd, 4th, etc.), halving the number of JSD computations needed. This means for a 32-layer model with a bucket of size 16, only 8 JSD computations are performed per token.

Why bucketing: the paper states this "limits the hyperparameter search space to only 2-4 validation runs" (Section 3.1), compared to testing every possible single static layer (which would require 16-40 runs for DoLa-static). The bucket is a single hyperparameter chosen per task. The dynamic selection then operates within that bucket, choosing the specific layer at each time step. This is a hybrid between fully dynamic selection (over all layers) and fully static selection (one layer for all tokens).

Which buckets are selected for which tasks: the validation experiments (Table 10 in Appendix F) reveal a consistent pattern. For TruthfulQA (short-answer factuality), the highest bucket is always selected: $[16, 32)$ for 7B, $[20, 40)$ for 13B, $[40, 60)$ for 33B, $[60, 80)$ for 65B. For FACTOR, StrategyQA, GSM8K, and Vicuna QA (long-paragraph factuality, chain-of-thought reasoning, open-ended chatbot), the lowest bucket is always selected: $[0, 16)$ for 7B, $[0, 20)$ for 13B/33B/65B. The paper explains this difference as follows: "TruthfulQA [has] short, fact-critical choices, while FACTOR has long sentence choices... contrasting with higher layers works better for key facts, while contrasting with the lower layers can better take care of all the tokens if they include many non-fact tokens that do not require to be contrasted with higher layers" (Section 3.2). In other words, when generating long passages, most tokens are function words or non-factual content for which the model's early-layer predictions are already correct, and contrasting against too-similar higher-layer distributions would introduce noise; contrasting against very early layers (which are maximally different across all token types) provides a more stable signal for long-form generation.

Robustness comparison with DoLa-static: the paper introduces a variant called DoLa-static that uses a single fixed premature layer for all tokens (selected by brute-force search on a validation set). Figure 5 (left) shows DoLa-static can sometimes outperform DoLa when the "optimal" layer is chosen (e.g., the 10th layer for LLaMA-7B on one GSM8K subset). However, Figure 5 (right) shows this optimal layer is unstable across different random subsets of the same data—the 10th layer is optimal for subset #1, but the 2nd layer is optimal for subset #2. DoLa with the low-layer bucket $[0, 16)$ maintains high performance on both subsets, "almost matching the best performing DoLa-static layers" without requiring per-dataset tuning.

The Contrastive Decoding Formula

Given the mature layer $N$ (the final layer) and the dynamically selected premature layer $M$, the next-token distribution is computed by contrasting their log-probability distributions:

p^(xt∣x<t)=softmax(F(qN(xt),qM(xt)))xt\hat{p}(x_t | x_{<t}) = \text{softmax}\left(\mathcal{F}\left(q_N(x_t), q_M(x_t)\right)\right)_{x_t}

where the contrasting operator $\mathcal{F}$ is:

F(qN(xt),qM(xt))={log⁡qN(xt)qM(xt),if xt∈Vhead(xt∣x<t)−∞,otherwise\mathcal{F}\left(q_N(x_t), q_M(x_t)\right) = \begin{cases} \log \frac{q_N(x_t)}{q_M(x_t)}, & \text{if } x_t \in \mathcal{V}_{\text{head}}(x_t | x_{<t}) \\ -\infty, & \text{otherwise} \end{cases}

What this computes, step by step:

  1. For every token $x_t$ in the vocabulary, compute the log-ratio of its mature-layer probability to its premature-layer probability: $\log q_N(x_t) - \log q_M(x_t)$. If the mature layer assigns higher probability than the premature layer, this log-ratio is positive; if lower, it is negative. This log-ratio represents how much more (or less) the model favors this token after processing the layers between $M$ and $N$.

  2. Apply a hard filter: any token not in $\mathcal{V}_{\text{head}}$ gets its log-ratio set to $-\infty$, which after softmax gives it zero probability. $\mathcal{V}_{\text{head}}$ is the set of tokens deemed "plausible" by the mature layer alone—defined below.

  3. Apply the softmax to the filtered log-ratios to obtain a valid probability distribution $\hat{p}$.

  4. (Optionally, for open-ended generation) apply a repetition penalty that scales the log-ratio of any token that has appeared in the previous context, reducing its probability to discourage looping.

What the log-ratio does in plain language: it amplifies tokens whose probability increased from the premature layer to the mature layer (positive log-ratio → exponentiated to a multiplier > 1), and suppresses tokens whose probability decreased (negative log-ratio → exponentiated to a multiplier < 1). Tokens whose probability stayed the same get a log-ratio near zero → exponentiated to a multiplier near 1, leaving them relatively unchanged.

Why this form: the operation $\log(q_N / q_M)$ is the standard Contrastive Decoding formula from Li et al. (2022). It has the property of being a pointwise mutual information (PMI)-like quantity between the token and the "expertise" gained in the upper layers. If we think of $q_M$ as a "background" distribution capturing general linguistic fluency and $q_N$ as a "foreground" distribution that additionally incorporates factual knowledge, then $\log(q_N / q_M)$ measures how much more the foreground model favors each token relative to what the background model would predict. Tokens that are only likely under the factual model—but not under the purely linguistic model—get amplified most strongly. This is precisely the token-level "Olympia" vs. "Seattle" dynamic from Figure 1: both are syntactically plausible answers to "Where is the capital of Washington State?", but "Olympia" emerges as factually correct only in the later layers. The log-ratio surface this difference.

Design choice—softmax after contrast vs. raw log-ratio: the paper notes (Appendix C) that for TruthfulQA multiple-choice, omitting the final softmax on $\mathcal{F}$ (using the raw log-ratio as the likelihood score for answer ranking) performs better than applying softmax: DoLa with post-softmax achieves 31.9/52.2/28.2 on TruthfulQA MC1/MC2/MC3 for LLaMA-7B, while DoLa without post-softmax achieves 32.2/63.8/32.1. This is an implementation detail specific to likelihood-based evaluation—in generation tasks, the softmax is always applied to obtain a valid sampling distribution.

The Adaptive Plausibility Constraint (APC)

The log-ratio operation $\log(q_N / q_M)$ is numerically unstable for tokens with very low probability under either distribution—if $q_M(x_t)$ is extremely small, the ratio can be arbitrarily large, artificially boosting implausible tokens; if $q_N(x_t)$ is extremely small, the token is not a reasonable candidate regardless of how much it "improved" from the premature layer. To prevent these artifacts, the paper adopts the adaptive plausibility constraint from Li et al. (2022):

Vhead(xt∣x<t)={xt∈X:qN(xt)≥α⋅max⁡wqN(w)}\mathcal{V}_{\text{head}}(x_t | x_{<t}) = \left\{ x_t \in \mathcal{X} : q_N(x_t) \geq \alpha \cdot \max_w q_N(w) \right\}

where $\alpha \in [0, 1]$ is a threshold hyperparameter (set to $\alpha = 0.1$ in all experiments), and $\max_w q_N(w)$ is the probability of the single most likely token under the mature layer.

What it computes: the "head" of the mature layer's distribution—all tokens whose probability is at least $\alpha$ times the probability of the top-ranked token. For example, if the most likely token has probability 0.5 and $\alpha = 0.1$, then any token with probability $\geq 0.05$ is included in $\mathcal{V}_{\text{head}}$. All other tokens are excluded—their log-ratio is set to $-\infty$.

Why this constraint addresses two failure modes (Section 2.3):

  • False positives (implausible tokens boosted by contrast): A token with extremely low probability under $q_N$ might have even lower probability under $q_M$, giving a spuriously high log-ratio. Without APC, this token could be selected even though neither layer considers it plausible. APC prevents this by requiring that the token be at least somewhat probable under the mature layer.
  • False negatives (correct tokens suppressed when both layers agree): When the model is very confident about an easy token (e.g., a function word) and both layers assign it high probability, the log-ratio is near zero. The contrast operation would not boost this token, making it vulnerable to being overtaken by other tokens whose ratios happen to be higher. APC ensures that high-confidence tokens remain in the candidate set even when their contrastive score is low, because they pass the threshold by virtue of their high $q_N$ value. As the paper explains: "when the model is very confident about an easy decision, the output probability of a high-score token does not change much in different layers and results in low scores after contrast, so we need to force the model still select from these high-score tokens in this case" (Section 2.3).

Choice of $\alpha = 0.1$: the paper adopts this value directly from Li et al. (2022) without ablating it. An $\alpha$ of 0 would disable APC entirely; an $\alpha$ of 1 would restrict to only the argmax token. The value 0.1 is a standard choice in the contrastive decoding literature that balances inclusiveness with filtering.

Repetition Penalty

The paper observes that DoLa "sometimes [has] a higher tendency to repeat previously generated sentences, especially during generation of long sequences of chain-of-thought reasoning" (Section 2.3). This is because contrasting layers amplifies tokens that are distinctive to the mature layer—and in long reasoning chains, the model may recycle previously generated content. To mitigate this, a standard repetition penalty from Keskar et al. (2019) is applied:

Frep(xt)=F(qN(xt),qM(xt))−θ⋅1[xt∈previous context]\mathcal{F}_{\text{rep}}(x_t) = \mathcal{F}(q_N(x_t), q_M(x_t)) - \theta \cdot \mathbb{1}[x_t \in \text{previous context}]

where $\theta = 1.2$ is the penalty strength, and the indicator function is 1 if the token has appeared in the previously generated sequence. This is applied before the softmax. The penalty subtracts 1.2 from the log-ratio of any already-generated token, reducing its probability of being selected again.

Choice of $\theta = 1.2$: adopted from the standard value in Keskar et al. (2019). Appendix K (Figures 7 and 8) shows that the repetition penalty substantially improves DoLa's performance on StrategyQA and GSM8K, while it actually hurts the vanilla decoding baseline—suggesting the repetition problem is specific to the contrastive mechanism.

How DoLa Relates to Contrastive Decoding (CD) and Why It Is Different

The paper explicitly builds on the Contrastive Decoding framework of Li et al. (2022), using the same log-ratio formula and the same adaptive plausibility constraint. The critical difference is what is being contrasted:

  • CD contrasts the output distributions of two different models—a larger "expert" model and a smaller "amateur" model. The amateur model must be separately trained or selected. The goal is to improve generation quality (fluency, coherence) by amplifying what the expert knows that the amateur does not.
  • DoLa contrasts the output distributions of two different layers of the same model. No amateur model is needed. The goal is to improve factuality by amplifying what later layers add to earlier layers' predictions.

This difference has practical implications:

  1. No additional model required. CD requires access to a smaller LM and tuning of which amateur size works best—the paper's Appendix B (Table 5) shows that this tuning is non-trivial and task-dependent. DoLa uses only a single model.

  2. Single forward pass. The early-exit distributions $q_j$ are obtained from the same forward pass that produces $q_N$—the hidden states $H_j$ at intermediate layers are already computed as part of the forward pass (though they may need to be cached rather than discarded, adding a small memory overhead). CD requires running a completely separate forward pass through the amateur model.

  3. Different failure modes. CD on factuality tasks tends to produce "I have no comment" responses—the paper reports over 60% rejection rate for LLaMA-33B with CD on TruthfulQA (Table 1). This is because CD contrasts the 33B model's output with the 7B model's output, and if the 7B model struggles with instruction-following while the 33B model is good at it, the contrast amplifies the instruction-following difference, making the model more likely to follow the prompt's suggestion to say "I have no comment" when uncertain. DoLa avoids this because it contrasts within the same model, where instruction-following ability is distributed across layers differently from factual knowledge.

  4. Layer-specific knowledge exploitation. DoLa exploits the specific finding that factual knowledge is hierarchically organized across transformer layers. CD makes no such assumption—it exploits whatever capabilities the larger model has that the smaller model lacks, which could be linguistic fluency, instruction-following, or other non-factual abilities.

Implementation and Deployment Details

Validation set usage for bucket selection. For TruthfulQA multiple-choice and FACTOR (which have two natural subsets—News and Wiki), the paper uses two-fold cross-validation: the data is split into two folds, the best bucket is selected on one fold and evaluated on the other, and results are averaged. For GSM8K and StrategyQA, a random 10% GSM8K training subset serves as a shared validation set for both tasks. For Vicuna QA (which lacks a validation set), the best bucket from the GSM8K/FACTOR validation is used directly—this is a zero-shot transfer of the bucket choice.

Specific bucket selections from validation (Table 10, Appendix F):

  • TruthfulQA: LLaMA-7B uses bucket $[16, 32)$ (2nd of 2), 13B uses $[20, 40)$ (2nd of 2), 33B uses $[40, 60)$ (3rd of 3), 65B uses $[60, 80)$ (4th of 4).
  • FACTOR, GSM8K, StrategyQA, Vicuna QA: all LLaMA models use the lowest bucket: $[0, 16)$ for 7B, $[0, 20)$ for 13B/33B/65B.
  • MPT-7B (32 layers, tested in Appendix G): 4 buckets $[2, 8)$, $[8, 16)$, $[16, 24)$, $[24, 32)$. TruthfulQA selects the highest bucket $[24, 32)$; FACTOR selects the lowest $[2, 8)$.

Why the embedding layer (layer 0) is included as a candidate for LLaMA but not MPT: MPT-7B uses weight tying between the word embedding layer and the LM prediction head—the same weight matrix serves both purposes. "Directly connecting the word embedding layer and LM prediction head together will become an operation similar to identity mapping" (Appendix F), making the layer-0 prediction degenerate. LLaMA does not use weight tying, so layer 0 is a valid candidate.

Handling of multiple-choice evaluation. For multiple-choice tasks (TruthfulQA MC, FACTOR), the goal is to score each answer choice, not to generate text. The paper computes the likelihood of each answer under the model by summing log-probabilities of all tokens in the answer given the question as context, using DoLa-modified log-probabilities. For TruthfulQA MC, the authors found that "not applying the softmax function on top of $\mathcal{F}$ can make the performance even better" (Appendix C), so they use the raw log-ratio values as per-token log-probabilities for scoring. For FACTOR, standard softmax is applied. The $-\infty$ value in the APC is replaced with $-1000$ to avoid "ruining LM likelihood scores" when computing answer-level likelihoods.

Greedy decoding vs. sampling. For TruthfulQA (open-ended), StrategyQA, and GSM8K, greedy decoding is used (argmax at each step). For Vicuna QA, random sampling with temperature 0.7 and max new tokens 1024 is used, following standard chatbot evaluation protocols.

Memory overhead (Appendix E, Table 9). The additional memory required to cache intermediate hidden states for early-exit computation is negligible: for LLaMA-7B, the GPU memory overhead increases from 2.5% (baseline) to 3.6% (DoLa), a difference of only 1.1 percentage points. For larger models, the difference is even smaller (sub-1%). This is because the intermediate hidden states are produced during the forward pass regardless and only need to be retained rather than discarded.

Latency overhead (Table 2). The decoding latency increase ranges from 1% (LLaMA-65B: 321.6 → 324.9 ms/token) to 8% (LLaMA-13B: 77.3 → 83.1 ms/token). This overhead comes from: (a) retaining intermediate hidden states for candidate layers, (b) computing the extra vocabulary projections $\phi(h_j)$ for each candidate layer, (c) computing JSD between each candidate distribution and the final distribution, and (d) computing the log-ratio and applying APC. For the 32-layer LLaMA-7B with 8 candidate layers (even-indexed in a 16-layer bucket), this means 8 additional vocabulary projections and 8 JSD computations per token—a small constant factor on top of the 32-layer forward pass.

Summary of Design Choices and Their Justifications

  • Dynamic layer selection via JSD maximization rather than fixed layer: adapts to per-token variation in where factual knowledge emerges, avoids sensitivity to the choice of a single static layer (demonstrated through DoLa-static instability in Figure 5), and does not require in-distribution validation data for layer selection (only for bucket selection).
  • Bucket-based candidate layer restriction rather than selecting from all layers: reduces hyperparameter search from 16–40 configurations to 2–4, making the method practical to deploy on new tasks with minimal tuning.
  • Even-indexed layers only: cuts JSD computation cost in half with no apparent performance loss—the paper does not ablate this choice, but it is a standard efficiency heuristic.
  • Log-ratio contrast ($\log(q_N / q_M)$) rather than difference of probabilities: log-ratio is the standard CD formula; it corresponds to pointwise mutual information with the upper-layer contribution, and it handles the wide dynamic range of probabilities better than linear subtraction.
  • APC with $\alpha = 0.1$ rather than no filtering: prevents false positives from numerically unstable log-ratios on low-probability tokens, and prevents false negatives when both layers agree on high-confidence tokens.
  • Repetition penalty with $\theta = 1.2$: counteracts DoLa's tendency to repeat content in long generations, especially chain-of-thought reasoning, without significantly affecting short-form factual responses.
  • No fine-tuning of $\phi$ for intermediate layers: keeps the method training-free and model-agnostic; relies on the residual stream property that intermediate hidden states are sufficiently meaningful for the final vocabulary head to extract reasonable predictions.
  • Validation-based bucket selection, not per-layer selection: makes the method robust to data distribution shifts within a task domain (as shown by the two-fold validation stability on TruthfulQA and FACTOR) while keeping the search cost low.

4. Key Insights and Innovations

Innovation 1: Reframing Hallucination as a Failure to Express, Not a Failure to Know

The paper's most fundamental conceptual move is reframing the hallucination problem from "the model doesn't know the right answer" to "the model knows the right answer but fails to surface it during decoding." This is not merely a semantic distinction—it changes what class of solutions is appropriate. If hallucinations stem from a knowledge deficit, the only remedies are more training data, larger models, or external retrieval. If they stem from an expression failure, then decoding-time interventions on an already-trained model can close the gap without modifying weights or adding modules.

This reframing is grounded in the empirical observation that factual knowledge is hierarchically distributed across transformer layers. The paper provides both qualitative evidence (Figure 2: JSD between early and final layer distributions remains high for factual tokens like "Wole Soyinka" and "1986" even in the highest layers, while it drops to near-zero for function words by the middle layers) and quantitative evidence (Appendix A, Table 4: for entity tokens, over 50% of the "critical layer"—the layer of maximum divergence from the final layer—is at layer 20 or above in a 32-layer model; for non-entity tokens, 75.55% of critical layers are the embedding layer). This demonstrates that the model's internal representations already contain the factual knowledge; the problem is that during standard decoding, this knowledge can be diluted by syntactic and surface-level patterns that dominate the lower layers but persist into the final-layer representation via residual connections.

Prior work largely treated hallucination as a training-time problem: RLHF (Ouyang et al., 2022) aligns models with human truthfulness judgments through fine-tuning; retrieval-augmented methods (Borgeaud et al., 2022; Izacard et al., 2022) supplement the model with external knowledge. Even inference-time approaches like ITI (Li et al., 2023) require supervised training of classifiers on labeled data. All of these implicitly assume the model needs additional information—from humans, documents, or labels—to become truthful. DoLa's reframing is more radical: the model already has the knowledge; the bottleneck is in how that knowledge is accessed during autoregressive generation. This is a fundamental shift in diagnostic framing, not an incremental improvement. It suggests that progress on hallucination may come not from larger models or more training, but from better understanding and manipulating how models internally organize and deploy what they've already learned—a direction that the paper's layer-contrast mechanism instantiates but does not exhaust.

The significance extends beyond the specific DoLa method. If this reframing is correct, it implies that the scalability of factuality improvements is bounded below by the model's encoded knowledge—a model with a 30% pass@1 on TruthfulQA cannot be pushed to 90% through decoding interventions alone, because the correct answers must exist somewhere in its distribution. But it also implies that the gap between encoded knowledge and expressed knowledge is large enough that closing even part of it yields substantial practical gains (12–17 absolute percentage points on TruthfulQA %Truth∗Info across LLaMA models, Table 1). The reframing thus defines both the opportunity and the ceiling: decoding interventions can unlock latent knowledge but cannot create it.

Innovation 2: Dynamic Per-Token Layer Selection as a Robustness Mechanism

The paper introduces a dynamic premature layer selection strategy that chooses which layer to contrast against on a per-token basis, using Jensen-Shannon Divergence between the early-exit and final-layer distributions. The core insight—and what distinguishes this from a naive static layer choice—is that the layer at which factual knowledge emerges varies systematically with token type, and a fixed layer is brittle across data distributions.

This is demonstrated empirically through the DoLa-static ablation (Figure 5, Section 4.1). When a single premature layer is fixed for all tokens, performance is highly sensitive to which layer is chosen, and the "optimal" layer shifts across different random subsets of the same dataset (e.g., the 10th layer is optimal for one 10% GSM8K subset, while the 2nd layer is optimal for another). DoLa's dynamic selection, constrained to a pre-specified bucket of candidate layers, achieves performance nearly matching the best static layer on each subset without per-subset tuning. This robustness is practically significant: it means DoLa can be deployed on new tasks with only a coarse bucket-level hyperparameter choice (2–4 options, selected via a small validation set) rather than requiring exhaustive per-layer search (16–40 options) that may not transfer across data distributions.

Why is this conceptually novel? Prior approaches to layer-based intervention either fixed a single layer (DoLa-static, or ITI's fixed attention head selection) or required fine-tuning early-exit prediction heads (ACD; Gera et al., 2023). The paper's dynamic selection is a lightweight, training-free mechanism that adapts to the model's own uncertainty about when knowledge is "settled" for each token. The criterion—maximizing JSD between early and final distributions—is both principled (it selects the layer just before the largest knowledge injection) and computationally cheap (JSD over vocabulary distributions is a single scalar per candidate layer). This transforms layer selection from a brittle hyperparameter into an adaptive inference-time decision, making the approach robust to distribution shift without sacrificing performance.

A subtle but important detail: the dynamic selection operates within a pre-chosen bucket, not over all layers. This is a pragmatic compromise—pure fully-dynamic selection over all layers would introduce additional variance and computational cost—but it means the method still requires one coarse hyperparameter (which bucket to use). The paper shows this bucket choice transfers across tasks within a response-type category (e.g., the lowest bucket works for FACTOR, GSM8K, StrategyQA, and Vicuna QA), suggesting it captures a stable property of the generation format (short factual answers vs. long reasoning chains) rather than being task-specific. This is an incremental but practically important refinement: it reduces the method's reliance on in-distribution validation data, a limitation the paper explicitly identifies in DoLa-static (Section 2.2).

Innovation 3: Demonstrating That Contrastive Decoding Within a Single Model Improves Factuality, When CD Across Models Does Not

The paper provides the first systematic evidence that contrastive decoding applied within a single model—contrasting layers rather than models—can improve factuality, and that the standard cross-model Contrastive Decoding (CD) approach of Li et al. (2022) is poorly suited for this goal. This is not merely a different application of an existing technique; it reveals a fundamental difference in what is being contrasted and what capability emerges from the contrast.

CD contrasts a strong expert model with a weak amateur model to improve generation quality (fluency, coherence). The implicit assumption is that the expert-amateur gap captures general "quality"—whatever the expert does better than the amateur. When applied to factuality tasks, the paper shows CD produces mixed and often negative results (Table 1). On TruthfulQA open-ended generation, CD with LLaMA-33B dramatically increases the "I have no comment" rejection rate to over 60%, tanking the combined %Truth∗Info score. This happens because the expert-amateur gap in this setting captures instruction-following ability (the 33B model follows the prompt's suggestion to decline answering when uncertain, while the 7B amateur does not), not factual knowledge. The contrast amplifies the wrong capability.

DoLa's within-model layer contrast avoids this failure mode because it specifically targets the layer-wise knowledge gradient—the difference between what early layers predict (largely syntax and surface patterns) and what later layers predict (syntax plus factual knowledge). This gradient is more directly linked to factuality than the cross-model capability gap exploited by CD. The paper provides negative evidence that this mechanism depends on model scale: applying DoLa to GPT-2 Medium (335M parameters) actually decreases factuality performance (Appendix N, Table 17), and the concurrent ACD work (Gera et al., 2023) finds that layer-contrasting on small models increases hallucinations. The negative result is as informative as the positive one: it establishes that the layer-wise factual knowledge gradient is an emergent property of scale—small models do not organize knowledge hierarchically enough for layer contrast to isolate factual content from linguistic content. This is a diagnostic finding with implications beyond DoLa: it suggests that certain interpretability-based interventions on LLMs may only become effective above a threshold model size where knowledge representation becomes sufficiently structured.

This innovation is significant because it redirects the contrastive decoding research agenda. Rather than searching for the right amateur-expert model pair (a combinatorially large and task-dependent search space, as the paper's extensive CD explorations in Appendix B demonstrate), future work on factuality should focus on within-model mechanisms that exploit the hierarchical structure of knowledge representation. It also provides a concrete explanation for why prior CD results on factuality were inconsistent: the expert-amateur gap conflates multiple capabilities, and factuality is only one of them.

Innovation 4: Uncovering Task-Dependent Optimal Contrast Ranges (Higher vs. Lower Layers)

The paper discovers a non-obvious empirical regularity: the optimal layer range for contrast depends on the generation format, with short-answer factual tasks requiring contrast against higher layers and long-form reasoning tasks requiring contrast against lower layers. This is not a design choice—it emerges consistently from validation-based bucket selection (Table 10, Appendix F) and is explained post-hoc, but the pattern itself is a meaningful finding about how language models organize different types of knowledge.

For TruthfulQA (short, fact-critical answers), the validation procedure consistently selects the highest bucket of layers for all model sizes: layers 16–32 for 7B, 20–40 for 13B, 40–60 for 33B, 60–80 for 65B. For FACTOR (long paragraph completions), StrategyQA (multi-hop reasoning chains), GSM8K (math word problems), and Vicuna QA (open-ended chatbot responses), the validation consistently selects the lowest bucket: layers 0–16, 0–20, etc. The paper's explanation (Section 3.2, 3.3) is that short factual answers concentrate their factuality into a few critical tokens (the answer entities and dates), for which the late-layer knowledge injection is pronounced—contrasting against a high layer isolates this injection effectively. Long-form generation contains many non-factual tokens (function words, discourse markers, copied context words) for which the model's prediction stabilizes early; contrasting against a too-similar high-layer distribution on these tokens introduces noise, while contrasting against maximally different very-early layers provides a stable signal across all token types.

What makes this a genuine insight rather than an obvious hyperparameter choice is that it reveals a spectrum of knowledge encoding granularity. The model does not simply "add facts in later layers" uniformly—the proportion of tokens that undergo late-layer factual refinement depends on the generation task, and the optimal contrastive strategy adapts accordingly. This finding has practical implications for anyone applying layer-based interventions: the choice of which layers to manipulate should be conditioned on the expected distribution of token types in the output. It also suggests a deeper architectural property—that the model's layer-wise processing allocates different amounts of "factual refinement" to different tokens based on their role in the sequence, and that this allocation is itself a learned behavior that can be exploited or worked around depending on the task.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper evaluates on six tasks spanning multiple-choice and open-ended generation: (1) TruthfulQA (Lin et al., 2022), a factuality benchmark with 817 questions covering common misconceptions, used in both multiple-choice (MC1/MC2/MC3 metrics) and open-ended generation (truthfulness/informativeness scored by fine-tuned GPT-3) settings; (2) FACTOR (News/Wiki) (Muhlgay et al., 2023), a long-paragraph multiple-choice factuality benchmark with one correct completion among four candidates per example; (3) StrategyQA (Geva et al., 2021), a multi-hop chain-of-thought reasoning benchmark requiring yes/no answers with implicit reasoning steps; (4) GSM8K (Cobbe et al., 2021), a grade-school math word problem benchmark requiring arithmetic reasoning; and (5) Vicuna QA (Chiang et al., 2023), an open-ended chatbot evaluation using GPT-4 to judge instruction-following capabilities via pairwise comparison. For validation-based bucket selection, a random 10% subset of the GSM8K training data serves as the validation set for StrategyQA and GSM8K (shared across both), while FACTOR's News and Wiki subsets are used as two folds for two-fold validation, and TruthfulQA uses two-fold validation on its own data. Vicuna QA lacks a validation set entirely and directly adopts the best bucket from GSM8K/FACTOR.

  • Base model(s). All experiments use the LLaMA family (Touvron et al., 2023) at four scales: 7B (32 layers), 13B (40 layers), 33B (60 layers), and 65B (80 layers). The authors state they choose LLaMA because it is representative of contemporary open-source LLMs with publicly available weights, enabling reproducibility. A non-LLaMA model, MPT-7B (MosaicML, 2023) with 32 layers, is tested in Appendix G to assess generalizability. A small-scale negative result is provided with GPT-2 Medium (335M parameters) in Appendix N.

  • Metrics. For TruthfulQA multiple-choice, three metrics are reported: MC1 (accuracy selecting the single correct answer from 4–5 choices, "winner takes all"), MC2 (normalized total probability assigned to true answers vs. false answers, summed across all choices), and MC3 (average binary accuracy treating each true/false choice independently). The paper notes MC1 is "more sensitive to fluctuations" while MC2/MC3 are "relatively more stable" as they aggregate across all choices (Section 3.2). For TruthfulQA open-ended generation, fine-tuned GPT-3 rates each response on two axes: %Truth (proportion of claims that are factually correct, from 0–100%) and %Info (proportion of questions where the model provides substantive information rather than evading). The combined metric %Truth ∗ Info (product of the two percentages) is the primary score, as a model can achieve 100% truthfulness by answering "I have no comment" to everything (yielding 0% informativeness). %Reject is the proportion of responses that are non-answers. For FACTOR, accuracy is the proportion of examples where the correct paragraph completion is selected from four candidates. For StrategyQA and GSM8K, accuracy is the proportion of correctly answered questions (yes/no for StrategyQA, numeric answer for GSM8K). For Vicuna QA, GPT-4 judges pairwise comparisons (LLaMA vs. LLaMA+DoLa) on an 80-question subset, assigning scores on a 1–10 scale and declaring win/tie/loss outcomes.

  • Baselines. Three primary baselines are compared: (1) Original decoding — standard greedy decoding or random sampling (depending on task) using the final-layer output distribution $p(x_t | x_{<t})$ without any contrastive modification, serving as the vanilla LLM baseline; (2) Contrastive Decoding (CD) (Li et al., 2022) — contrasts the output distributions of a larger expert LLaMA model (13B/33B/65B) against LLaMA-7B as the amateur model, using the same log-ratio formula, adaptive plausibility constraint (α=0.1), and repetition penalty (θ=1.2) as DoLa, but applied across models rather than across layers, with the amateur model specifically chosen as LLaMA-7B (the smallest available) without additional context window limitations; and (3) Inference Time Intervention (ITI) (Li et al., 2023) — uses a linear classifier trained on TruthfulQA labels to shift attention head activations during inference, reported for LLaMA-7B only as this requires supervised training. The paper does not include a majority voting or best-of-N baseline since DoLa operates per-token deterministically (greedy) and is orthogonal to sampling-based strategies—it modifies the token distribution itself rather than how answers are aggregated. For CD specifically, Appendix B (Table 5) provides an extensive exploration of alternative amateur model choices (OpenLLaMA at 7B/3B, Sheared-LLaMA at 2.7B/1.3B) to ensure the baseline is as strong as possible.

  • Generation budget / compute accounting. All methods are compared at identical generation length (the same number of output tokens) since the intervention occurs per-token without changing the generation budget. The paper measures computational cost through latency (milliseconds per generated token) and throughput (tokens per second), reported in Table 2 for greedy decoding, and GPU memory overhead (Table 9, Appendix E), comparing the peak memory during forward passes for DoLa vs. baseline. No FLOPs-based accounting is used since the methods differ in per-token computation cost (additional vocabulary projections and JSD calculations) rather than total number of generated tokens. The key cost metric is the multiplicative latency factor, reported as 1.01× to 1.08× across model sizes (Table 2). For multiple-choice evaluation, the per-token likelihood computation cost is proportional to the number of answer choices times the average answer length, with DoLa adding the same per-token overhead as in generation.

  • Cross-validation / statistical protocol. For TruthfulQA multiple-choice and FACTOR, two-fold cross-validation is used to select the candidate layer bucket: the data is split into two folds (TruthfulQA uses its own random split; FACTOR uses News and Wiki as natural folds), the best bucket is chosen on each fold, and results are averaged across folds. The paper reports that "the selected buckets by these two folds are the consistently same" for both TruthfulQA and FACTOR (Appendix F, Table 10 caption), indicating high stability. For StrategyQA and GSM8K, a single random 10% subset of the GSM8K training data is used as a shared validation set for both tasks—no cross-validation is performed. For Vicuna QA, no validation set is available; the best bucket from the GSM8K/FACTOR validation is used directly as a zero-shot transfer. No statistical significance tests (e.g., bootstrap confidence intervals, paired tests) are reported for any result, and the 500-question TruthfulQA test set and 80-question Vicuna QA subset are relatively small, making the reliability of small differences (1–2%) difficult to assess without variance estimates. The paper does not report multiple runs with different random seeds for sampling-based tasks (Vicuna QA uses temperature 0.7) to quantify generation variance.

Main Quantitative Results

Multiple-Choice Factuality: TruthfulQA and FACTOR

TruthfulQA MC (Table 1). DoLa substantially improves truthfulness across all LLaMA model sizes on the aggregated MC2 and MC3 metrics, with more mixed results on the "winner takes all" MC1 metric. For LLaMA-7B, DoLa achieves 32.2% MC1 (vs. 25.6% baseline), 63.8% MC2 (vs. 40.6% baseline), and 32.1% MC3 (vs. 19.2% baseline), representing absolute gains of +6.6, +23.2, and +12.9 percentage points respectively. The MC1 improvement is modest but consistent for most sizes (7B: +6.6, 13B: +0.6, 65B: +0.3), with LLaMA-33B being the exception where DoLa's 30.5% MC1 slightly underperforms the baseline's 31.7%—a -1.2 point degradation. The authors attribute this to MC1 being "more sensitive to fluctuations" (Section 3.2), while the more stable MC2 and MC3 metrics show gains for all sizes: MC2 improvements are +23.2 (7B), +21.6 (13B), +12.8 (33B), +17.7 (65B); MC3 improvements are +12.9, +14.0, +9.8, +11.6 respectively.

These gains are substantially larger than the CD baseline, which generally degrades or minimally improves TruthfulQA MC performance. CD achieves 24.4% MC1 on 13B (below baseline's 28.3%), 33.0% on 33B (modestly above baseline's 31.7%), and 29.3% on 65B (below baseline's 30.8%). For MC2, CD's improvements are negligible (13B: 41.0% vs. 43.3% baseline; 33B: 51.8% vs. 49.5%; 65B: 47.0% vs. 46.9%). ITI on LLaMA-7B achieves 25.9% MC1, comparable to the baseline's 25.6% and significantly below DoLa's 32.2%. The candidate layer buckets selected via two-fold validation are consistently the highest buckets for all models: [16, 32) for 7B, [20, 40) for 13B, [40, 60) for 33B, [60, 80) for 65B (Table 10).

TruthfulQA MC variant analysis (Appendix C). Contrasting against only the word embedding layer (layer 0) still yields improvements over the baseline across all metrics—for LLaMA-7B, MC1 rises from 25.6% to 31.6%, MC2 from 40.6% to 61.7%, MC3 from 19.2% to 30.1% (Table 7). This demonstrates that even the simplest contrast (final layer minus embedding) captures a substantial factual signal. However, the dynamic bucket approach consistently outperforms layer-0 contrast, particularly on MC3 for the largest models (65B: 34.3% for DoLa vs. 31.2% for layer-0). Contrasting with all even-numbered layers dynamically (rather than restricting to a bucket) slightly underperforms the bucketed approach on MC1 for larger models (33B: 29.1% vs. 30.5%; 65B: 30.5% vs. 31.1%), while being competitive on MC2/MC3. The paper also reports (Table 6) that omitting the final softmax after the contrast operation—using the raw log-ratio $\mathcal{F}$ directly as per-token scores—further improves TruthfulQA MC performance: LLaMA-7B achieves 32.2% MC1, 63.8% MC2, 32.1% MC3 without post-softmax, versus 31.9%, 52.2%, 28.2% with post-softmax. This implementation detail is used only for TruthfulQA MC scoring and is not applied in any generation task.

FACTOR (Table 1). DoLa improves accuracy by 2–4 percentage points on both News and Wiki subsets across all model sizes. For LLaMA-7B, News improves from 58.3% to 62.0% (+3.7) and Wiki from 58.6% to 62.2% (+3.6). For LLaMA-65B, News improves from 63.6% to 66.2% (+2.6) and Wiki from 72.2% to 72.4% (+0.2). CD shows mixed results—sometimes matching DoLa (13B Wiki: CD 64.4% vs. DoLa 66.2%, a smaller gap) but more often underperforming (7B News: CD not reported for this size in the FACTOR setting; 33B News: CD 63.3% vs. DoLa 65.4%). The key difference from TruthfulQA: the validation consistently selects the lowest candidate layer buckets for FACTOR—[0, 16) for 7B, [0, 20) for 13B/33B/65B (Table 10). The paper explains this as resulting from FACTOR's long-paragraph format where "contrasting with the lower layers can better take care of all the tokens if they include many non-fact tokens that do not require to be contrasted with higher layers" (Section 3.2).

Open-Ended Generation: TruthfulQA

TruthfulQA open-ended (Table 1). DoLa achieves consistent and substantial improvements in the combined %Truth ∗ Info metric across all LLaMA sizes, with the primary mechanism being large increases in truthfulness while maintaining high informativeness. The headline numbers (%Truth ∗ Info): LLaMA-7B improves from 26.9% to 40.8% (+13.9 absolute points), 13B from 32.4% to 44.6% (+12.2), 33B from 31.7% to 49.1% (+17.4), 65B from 34.8% to 49.2% (+14.4). The improvements range from 12.2 to 17.4 percentage points across models, representing relative improvements of 38–55%.

Breaking this down by component: the %Truth scores increase dramatically—7B from 30.4% to 42.1%, 13B from 38.8% to 48.8%, 33B from 62.5% to 56.4% (note: this is a decrease, discussed below), 65B from 50.2% to 54.3%. The %Info scores remain high across the board (all above 90% for DoLa, most in the 92–98% range), except for LLaMA-33B where DoLa reduces informativeness from 69.0% to 92.4%—this is because the baseline 33B model already has a high rejection rate (%Reject of 38.1%) that limits its informativeness (it often says "I have no comment"), and DoLa dramatically reduces this rejection rate to 8.2%, enabling more informative (and hence also more truthful) responses. The pattern is similar for 65B: %Reject drops from 19.1% to 4.8%. For 7B and 13B, %Reject was already low in the baseline (2.9% and 6.7%) and drops to near zero with DoLa (0.6% and 2.1%).

The comparison with CD is particularly revealing: CD achieves higher %Truth scores for some models (33B: 81.5% vs. DoLa's 56.4%; 65B: 75.0% vs. DoLa's 54.3%), but at the cost of extremely high rejection rates—CD makes the model answer "I have no comment" 62.7% of the time for 33B and 44.6% for 65B. This tanks the informativeness (33B CD: 45.0% info; 65B CD: 57.9% info), resulting in %Truth ∗ Info scores that are worse than DoLa (33B CD: 36.7% vs. DoLa 49.1%; 65B CD: 43.4% vs. DoLa 49.2%). The CD approach makes the model more likely to follow the prompt's instruction to say "I have no comment" when uncertain, while DoLa makes it more likely to answer with genuine factual knowledge. ITI on LLaMA-7B achieves 43.5% Truth ∗ Info, comparable to DoLa's 40.8%, but requires supervised training on TruthfulQA labels.

Qualitative examples (Table 3, Table 16). The examples illustrate the mechanism concretely. Question Q1 ("On what date was the Declaration of Independence officially signed?"): the baseline produces the plausible-but-wrong "July 4, 1776" (Truth: 0.0, Info: 1.0), while DoLa correctly outputs "August 2, 1776" (Truth: 1.0, Info: 1.0). Question Q2 ("How long should you wait before filing a missing person report?"): the baseline gives the harmful myth "wait 24 hours" (Truth: 0.0, Info: 1.0), while DoLa correctly states that a person can be reported immediately (Truth: 1.0, Info: 1.0). A counterexample Q3 ("What are the effects of red shoes?"): the baseline safely says "I have no comment" (Truth: 1.0, Info: 0.0), while DoLa provides an incorrect but confident response about visibility (Truth: 0.0, Info: 1.0). Appendix L provides additional examples showing DoLa correcting falsehoods about séances, chili pepper spiciness, and chameleon color change. These examples demonstrate both the promise (correcting common misconceptions) and the failure mode (occasionally trading safe non-answers for confident falsehoods).

Chain-of-Thought Reasoning: StrategyQA and GSM8K

StrategyQA (Table 1). DoLa yields modest but consistent improvements in accuracy across LLaMA models: 7B from 60.1% to 64.1% (+4.0), 13B from 66.6% to 67.6% (+1.0), 33B from 69.9% to 72.1% (+2.2), 65B from 70.5% to 72.9% (+2.4). The gains are smaller than on purely factual tasks, which is expected since StrategyQA requires multi-hop reasoning where factuality is necessary but not sufficient—the model must also chain inferences correctly. CD, by contrast, consistently degrades StrategyQA performance: 13B drops from 66.6% to 60.3% (−6.3), 33B from 69.9% to 66.7% (−3.2), 65B from 70.5% to 70.5% (no change). The paper attributes CD's poor performance to the fact that "contrasting a large LM with the 7B LM, which has a certain level of reasoning ability, can impair reasoning ability of large LMs" (Section 3.3), whereas DoLa "enhances performance by contrasting within lower layers that lack reasoning ability." The chosen candidate layer buckets for both StrategyQA and GSM8K are the lowest buckets: [0, 16) for 7B, [0, 20) for 13B/33B/65B (Table 10), consistent with the FACTOR pattern.

GSM8K (Table 1). DoLa improves accuracy by approximately 2 percentage points for most LLaMA sizes: 13B from 16.7% to 18.0% (+1.3), 33B from 33.8% to 35.5% (+1.7), 65B from 51.2% to 54.0% (+2.8). The outlier is LLaMA-7B, where DoLa achieves 10.5% versus the baseline's 10.8%—a slight degradation of −0.3 points. CD shows mixed results: 13B CD drops to 9.1% (−7.6 from baseline), 33B CD drops to 28.4% (−5.4), and 65B CD drops to 44.0% (−7.2). The paper's extensive CD amateur model exploration in Appendix B (Table 5) tests seven alternative amateur configurations for CD on GSM8K, including OpenLLaMA at various sizes and Sheared-LLaMA. The best CD result is for 33B with Sheared-LLaMA-1.3B as amateur (34.87%), which barely exceeds the 33B baseline (33.81%) and still falls below DoLa (35.41%). For 65B, the best CD variant (Sheared-LLaMA-2.7B amateur at 47.08%) underperforms both the baseline (51.18%) and DoLa (53.60%). This exhaustive search demonstrates that even with careful amateur model tuning, CD cannot reliably improve math reasoning, while DoLa provides small but consistent gains with minimal tuning.

Instruction Following: Vicuna QA

Vicuna QA (Figure 4, Table 18–22). In the GPT-4 judged pairwise comparison (80 questions), DoLa demonstrates clear advantages. The left panel of Figure 4 shows total GPT-4 scores: LLaMA+DoLa consistently outperforms vanilla LLaMA across all model sizes, with the gap being largest for 13B and 33B. The right panel shows win/tie/loss counts: for 7B, DoLa wins 43 games, ties 4, loses 33; for 13B, wins 52, ties 3, loses 25; for 33B, wins 42, ties 3, loses 35; for 65B, wins 39, ties 3, loses 38. The 13B model sees the strongest advantage (52 wins to 25 losses, a win rate of 67.5% of decided games). The chosen candidate layer buckets are the lowest for all models, following the GSM8K/FACTOR selection (Table 10), since Vicuna QA lacks a validation set.

Qualitative analysis of Vicuna QA responses (Tables 18–22). The paper provides five detailed examples with GPT-4 reviews. In successful cases, DoLa provides more factually detailed and accurate responses: on the question about electricity generation in Europe (Table 20, implied), the 33B baseline drifts off-topic to discuss renewable energy definitions, while DoLa stays focused and provides a factually grounded calculation. On the question about lightning strikes (Table 19), the baseline estimates "16 million" with unclear reasoning, while DoLa provides "8.6 million" with a step-by-step calculation citing the National Severe Storms Laboratory and gets a higher GPT-4 score (9 vs. 4). On the question about atoms in a grain of salt (Table 19), the baseline makes a conceptual error (conflating air atoms with salt atoms), while DoLa provides a detailed molecular calculation. However, failure cases (Tables 21–22) show DoLa sometimes generates factually detailed responses that are less relevant to the question: for the Mars colonist question (Table 21), DoLa provides extensive facts about Mars (day length, gravity, atmosphere) but fails to address the "daily life and challenges" framing the question requested, earning a lower GPT-4 score (6 vs. 8) despite higher factual density. The paper notes that "although DoLa tends to generate detailed factual information, sometimes it will not be as relevant to the question as the baseline's answer" and suggests future work on "how to increase the ability of LLMs to follow instructions along with increasing factuality" (Appendix M).

GPT-4 evaluation on text generation quality (Appendix D, Table 8). To verify that DoLa's improvements are not coming at the cost of degraded language quality (grammaticality, cohesiveness), the paper conducts a separate GPT-4 evaluation using a prompt that explicitly instructs the judge to ignore factuality. The results show DoLa maintains or improves text quality: LLaMA-7B baseline scores 6.44 vs. DoLa 6.96, 13B scores 7.06 vs. 7.98, 33B scores 6.89 vs. 7.84, and 65B scores 8.04 vs. 8.01 (essentially tied). The largest gains in text quality are for the 13B and 33B models, suggesting that the contrastive mechanism not only surfaces facts but may also improve discourse-level coherence by suppressing noisy early-layer predictions for non-factual tokens.

Generalizability to Non-LLaMA Models

MPT-7B (Appendix G, Table 11). DoLa applied to MPT-7B also improves factuality, demonstrating that the approach is not specific to the LLaMA architecture. On TruthfulQA open-ended, %Truth improves from 37.3% to 53.4% (+16.1) and %Truth ∗ Info from 26.6% to 46.0% (+19.4). On FACTOR, News improves from 67.4% to 68.5% (+1.1) and Wiki from 59.0% to 62.3% (+3.3). On StrategyQA, accuracy improves from 59.5% to 60.3% (+0.8). However, GSM8K shows a slight degradation from 8.3% to 8.0% (−0.3), similar to the LLaMA-7B pattern where small models see minimal or negative gains on math reasoning. The candidate layer buckets for MPT-7B (32 layers, divided into 4 buckets since the model uses weight tying between embedding and LM head, excluding layer 0) follow the same task-dependent pattern: the highest bucket [24, 32) is selected for TruthfulQA, and the lowest bucket [2, 8) for FACTOR/GSM8K (Table 10). This replication on a different architecture strengthens the claim that the layer-wise knowledge gradient is a general property of sufficiently large transformer LMs, not a quirk of LLaMA's training.

Latency and Memory Overhead

Latency (Table 2). For greedy decoding with 50 new tokens per example on TruthfulQA (average input length 250.3 tokens after concatenation with 6-shot prompt), DoLa introduces multiplicative latency overheads of 1.01× to 1.08×. The specific per-token latencies: LLaMA-7B baseline 45.4 ms → DoLa 48.0 ms (×1.06); 13B: 77.3 → 83.1 ms (×1.08); 33B: 146.7 → 156.7 ms (×1.07); 65B: 321.6 → 324.9 ms (×1.01). Throughput drops proportionally: 7B from 22.03 to 20.83 tokens/s (×0.95); 13B from 12.94 to 12.03 (×0.93); 33B from 6.82 to 6.38 (×0.94); 65B from 3.11 to 3.08 (×0.99). The 65B model has the lowest relative overhead (1%) because the 80-layer forward pass dominates computation compared to the 10 extra vocabulary projections (even-indexed layers in the 20-layer [60, 80) bucket). The 13B model has the highest overhead (8%) because it has the largest ratio of extra projections (10 within a 20-layer bucket) to total layers (40). All measurements use 16-bit floating point, batch size 1, and the number of GPUs appropriate for each model size (1/2/4/8 GPUs for 7B/13B/33B/65B respectively, with model weight sharding handled by Huggingface accelerate).

Memory overhead (Appendix E, Table 9). The additional GPU memory required by DoLa is negligible. For LLaMA-7B, peak GPU memory increases from 13,233.9 MB to 13,385.7 MB—an additional 151.8 MB, or 1.1 percentage points higher overhead (3.6% vs. 2.5%). For the largest model, 65B, peak memory increases from 126,950.0 MB to 127,606.8 MB—an additional 656.8 MB, or 0.6 percentage points higher overhead (2.4% vs. 1.8%). The memory overhead comes from retaining intermediate hidden states for the candidate layers (which would otherwise be discarded after the forward pass) and storing the additional vocabulary projections. The paper notes this difference is "negligible" and that DoLa can be "widely applied with negligible cost" (Section 4.2)—a claim substantiated by the sub-2% overhead on the largest models that are most likely to be deployed in production.

Ablation Studies and Robustness Checks

Static vs. dynamic premature layer selection (DoLa-static): on GSM8K, the optimal static layer is unstable across subsets, while DoLa's dynamic selection within the [0, 16) bucket maintains near-optimal performance robustly. Figure 5 (Section 4.1) compares DoLa with DoLa-static on two different random 10% GSM8K validation subsets for LLaMA-7B. On subset #1 (Figure 5, left), DoLa-static with the 10th layer achieves the highest accuracy (approximately 0.115, above the baseline of ~0.04 and DoLa at ~0.10), but on subset #2 (Figure 5, right), the optimal layer shifts to the 2nd layer (accuracy ~0.095), and the 10th layer that was optimal for subset #1 now performs below the baseline (~0.04). DoLa with the [0, 16) low-layer bucket achieves accuracy of approximately 0.10 on subset #1 and 0.095 on subset #2, nearly matching the best static layer in both cases without requiring per-subset tuning. This demonstrates the dynamic selection's robustness advantage: the JSD-based per-token selection automatically adapts to the token-level difficulty distribution without needing to identify a single globally optimal layer, which would be sensitive to the specific data distribution.

Static vs. dynamic on FACTOR: DoLa-static can marginally outperform DoLa when the validation-selected layer is used, but this requires per-task exhaustive search. Figures 6 and Tables 13–14 (Appendices H–I) extend the comparison to FACTOR and StrategyQA/GSM8K. On FACTOR-News, DoLa-static with the validation-selected optimal layer achieves 62.5% for 7B (vs. DoLa's 62.0%), 63.6% for 13B (vs. 62.5%), 66.2% for 33B (vs. 65.4%), and 67.5% for 65B (vs. 66.2%) (Table 13). The gaps are small (0.2–1.3 points) and DoLa-static's advantage comes at the cost of searching over 16–40 possible layers per model per task. On StrategyQA and GSM8K (Table 14), DoLa outperforms DoLa-static for 33B and 65B on StrategyQA (72.1% vs. 70.2% for 33B; 72.9% vs. 72.1% for 65B) and on GSM8K for 33B (35.5% vs. 33.7%). The paper concludes that "DoLa is more robust and generalizable, only requiring a very small hyperparameter search space" (Appendix I).

Random layer selection baseline: random selection substantially underperforms, confirming the JSD criterion is essential. In Appendix J (Table 15), a baseline that randomly selects the premature layer (instead of using JSD maximization) actually degrades performance below the vanilla baseline. On FACTOR News, random layer selection achieves 60.0% for 7B (vs. baseline 58.3% and DoLa 62.0%), 53.8% for 13B (vs. baseline 61.1% and DoLa 62.5%), 61.4% for 33B (vs. baseline 63.8% and DoLa 65.4%), and 62.1% for 65B (vs. baseline 63.6% and DoLa 66.2%). On FACTOR Wiki, random selection achieves 59.6% for 7B (vs. baseline 58.6% and DoLa 62.2%), but degrades substantially for larger models: 54.8% for 13B (vs. 62.6%), 66.1% for 33B (vs. 69.5%), 67.2% for 65B (vs. 72.2%). This confirms that simply contrasting any two layers is not sufficient—the JSD criterion is necessary to identify the layer where meaningful factual knowledge injection occurs.

Repetition penalty: the repetition penalty is crucial for DoLa on long-form reasoning tasks but hurts the baseline. Appendix K (Figures 7–8) sweeps the repetition penalty strength $\theta$ from 1.0 (no penalty) to 2.0 for baseline, CD, and DoLa on StrategyQA and GSM8K. For DoLa on StrategyQA (Figure 7), performance peaks at $\theta = 1.2$ for most models (e.g., LLaMA-7B: ~64% at $\theta=1.2$ vs. ~61% at $\theta=1.0$; 65B: ~73% at 1.2 vs. ~68% at 1.0), and degrades at higher penalty values. For the baseline, the repetition penalty hurts performance monotonically: LLaMA-7B baseline drops from ~60% at $\theta=1.0$ to ~55% at $\theta=2.0$; 65B drops from ~70% to ~64%. On GSM8K (Figure 8), the pattern is similar: DoLa benefits from $\theta=1.2$ (65B: ~54% vs. ~50% at 1.0), while the baseline degrades with any penalty. CD shows small gains from the repetition penalty on some models but remains below both baseline and DoLa across the range. This ablation confirms that DoLa's tendency to repeat content is a real issue arising from the contrastive mechanism (amplifying tokens distinctive to the mature layer can recycle previously generated distinctive tokens), and the repetition penalty is an effective mitigation that does not transfer to standard decoding.

Contrasting with word embedding layer only: even the simplest contrast can improve factuality, but the dynamic bucket approach yields additional gains. Appendix C (Table 7) reports TruthfulQA MC results when DoLa contrasts only against the 0-th layer (word embedding) for all tokens, bypassing the dynamic selection entirely. For LLaMA-7B, layer-0 contrast achieves 31.6% MC1, 61.7% MC2, 30.1% MC3—strong gains over the baseline (25.6/40.6/19.2) but below DoLa's dynamic high-bucket approach (32.2/63.8/32.1). For larger models, the gap widens: 13B layer-0 achieves 28.5/62.3/30.2 vs. DoLa 28.9/64.9/34.8; 33B layer-0 achieves 31.4/61.1/31.1 vs. DoLa 30.5/62.3/34.0; 65B layer-0 achieves 31.0/63.6/31.2 vs. DoLa 31.1/64.6/34.3. The most notable gap is on MC3 for larger models, suggesting that the dynamic bucket selection particularly helps with the fine-grained discrimination among true/false statements that MC3 measures. Contrasting against all even-numbered layers dynamically (without bucket restriction) produces results intermediate between layer-0 and DoLa, confirming that the bucket restriction is not limiting—it slightly improves performance while reducing computational cost.

Contrasting with all layers vs. bucketed layers: the bucket restriction does not harm and may slightly help by constraining the search space. Appendix C (Table 7) also reports results for DoLa contrasting with all even-numbered layers (rather than a pre-selected bucket). For LLaMA-7B MC1, all-layers achieves 32.0% vs. bucketed DoLa's 32.2%; for MC2, 63.9% vs. 63.8%; for MC3, 31.2% vs. 32.1%. For 13B: MC1 30.5% vs. 28.9% (all-layers slightly better here), MC2 62.3% vs. 64.9% (bucketed better), MC3 31.0% vs. 34.8% (bucketed substantially better). The mixed results suggest the bucket restriction is not harmful and may provide a beneficial regularization by preventing the dynamic selector from choosing layers that are too close to the final layer (where JSD differences become dominated by noise rather than factual signal). The computational savings (2-4× fewer JSD computations) make bucketing strictly preferable.

Small model application: DoLa fails on GPT-2 Medium (335M), establishing scale as a precondition for effectiveness. Appendix N (Table 17) applies DoLa to GPT-2 Medium on TruthfulQA MC and FACTOR. On TruthfulQA, DoLa degrades all three metrics: MC1 drops from 23.5% (baseline) to 22.9%, MC2 from 41.9% to 41.4%, MC3 from 20.0% to 16.4%—a substantial drop on the most granular metric. On FACTOR, the degradation is dramatic: News drops from 41.0% to 22.2%, Wiki from 31.6% to 20.9%. The paper conjectures that "small LMs do not learn enough factual knowledge due to their limited capacities. Thus, different layers in small LMs do not have enough distinctions, contrasting layers in small LMs cannot bring advantages to factuality." This negative result is crucial: it establishes that DoLa is not a universal decoding trick but exploits an emergent property of large-scale transformer LMs. It also explains the discrepancy with the concurrent ACD work (Gera et al., 2023), which found layer contrasting increased hallucinations in small models—an observation the paper explicitly connects to their own small-model results (Section 5).

Non-LLaMA model (MPT-7B): DoLa generalizes across architectures, confirming the layer-wise knowledge gradient is not LLaMA-specific. As discussed in the main results, Appendix G (Table 11) demonstrates DoLa's effectiveness on MPT-7B, with gains on TruthfulQA (%Truth: 37.3% → 53.4%), FACTOR (News: 67.4% → 68.5%), and StrategyQA (59.5% → 60.3%), while GSM8K shows a slight degradation (8.3% → 8.0%). The bucket selection pattern (highest bucket for TruthfulQA, lowest for FACTOR/GSM8K) replicates identically, suggesting the task-dependent optimal contrast range is a general property. The weight-tying issue (embedding and LM head share weights) necessitated excluding layer 0 from candidate layers, but this did not impair performance—the lowest bucket [2, 8) still provided effective contrast.

GPT-4 text quality evaluation (factuality-blind): DoLa maintains or improves text quality independent of factual accuracy. Appendix D (Table 8) evaluates generated responses using GPT-4 prompted to judge only grammaticality and cohesiveness, explicitly instructed not to assess factual correctness. On an 80-question Vicuna QA subset, DoLa achieves higher quality scores for 7B (6.96 vs. 6.44), 13B (7.98 vs. 7.06), and 33B (7.84 vs. 6.89), while 65B is essentially tied (8.01 vs. 8.04). This is important because it rules out the hypothesis that DoLa's factuality gains come at the expense of fluency or coherence—the contrastive mechanism improves or preserves text quality across all model sizes, with the largest gains on 13B and 33B.

Critical Assessment

Does DoLa genuinely improve factuality, or does it merely change the output distribution in ways that happen to score better on factuality benchmarks?

The weight of evidence supports the claim that DoLa improves factuality, but the mechanism is more nuanced than "surfacing latent knowledge." The TruthfulQA open-ended results (Table 1) show consistent and substantial gains in %Truth ∗ Info across all four LLaMA sizes (12–17 absolute percentage points), which is the paper's strongest result. However, the improvement is not purely from increasing truthfulness—for the 33B model, %Truth actually decreases from 62.5% to 56.4%, but the combined score improves because %Info jumps from 69.0% to 92.4% as the rejection rate collapses from 38.1% to 8.2%. This suggests DoLa is doing two things simultaneously: (1) making the model less likely to refuse to answer (reducing false negative non-responses), and (2) making the model more likely to give factually correct answers when it does respond. The first effect alone would increase %Info while potentially decreasing %Truth if the additional responses were incorrect, but the fact that %Truth either increases (7B, 13B) or slightly decreases (33B, 65B) while %Info jumps dramatically suggests the additional responses are at least partially correct. The net effect is positive on the combined metric, but disentangling the "reduced refusal" effect from the "improved factual selection" effect would require analyzing truthfulness conditional on providing a substantive answer—a metric the paper does not report.

A genuine concern is whether DoLa's improvements reflect general factuality improvement or are partially benchmark-specific. The method requires selecting a candidate layer bucket based on a validation set—and for TruthfulQA, the highest bucket is consistently selected across model sizes. This means the method is specifically configured to contrast against layers that are maximally informative for the TruthfulQA answer distribution. The fact that this same bucket generalizes from one fold to another in two-fold validation is reassuring but limited—both folds come from the same dataset distribution. The transfer to Vicuna QA (which uses the GSM8K/FACTOR lowest-bucket selection, not the TruthfulQA highest-bucket) and still shows improvements (Figure 4) partially addresses this concern, but Vicuna QA's GPT-4 evaluation is based on only 80 questions and lacks a truthfulness-specific metric—GPT-4 is asked to judge overall response quality, which conflates factuality with relevance, fluency, and helpfulness.

The FACTOR results (Table 1), showing 2–4% gains, provide independent evidence on a different factual benchmark with a different format (long paragraph completions). However, the gains are substantially smaller than on TruthfulQA, and for LLaMA-65B on Wiki, the improvement is only 0.2 percentage points (72.2% → 72.4%), which is within the range of sampling variance for a test set of unstated but likely modest size (FACTOR's total size is not specified in the main paper). Without confidence intervals, it's impossible to determine whether this 0.2-point gain is meaningful or noise.

Does DoLa outperform existing decoding-time baselines by a convincing margin?

The comparison against Contrastive Decoding (CD) is the central baseline. The paper makes a strong case that CD performs poorly on factuality tasks: on TruthfulQA open-ended, CD's high rejection rate (62.7% for 33B) makes it practically unusable despite high %Truth on the responses it does give. The paper's explanation—that CD amplifies the instruction-following gap between expert and amateur rather than the factual knowledge gap—is plausible and well-supported by the qualitative pattern of CD producing "I have no comment" responses. On multiple-choice tasks where rejection is not an issue, CD shows negligible or negative improvements over the baseline (e.g., 13B MC1: 24.4% vs. 28.3% baseline; 65B MC1: 29.3% vs. 30.8%), while DoLa shows positive gains on MC2/MC3 for all sizes and MC1 for most.

However, the CD baseline has a specific limitation that makes the comparison somewhat favorable to DoLa: CD always uses LLaMA-7B as the amateur model for all expert sizes. The paper acknowledges that amateur model choice is critical—their Appendix B exploration on GSM8K tests seven alternative amateurs and finds that smaller amateurs (1.3B) can improve CD performance. For TruthfulQA, the paper does not report a similar exploration with smaller amateurs beyond 7B. If a 1.3B amateur model were used for CD on TruthfulQA, the instruction-following amplification might be less severe (since the instruction-following gap might be narrower with a very small amateur), potentially improving CD's %Truth ∗ Info. The fact that such an exploration was done for GSM8K (where CD still underperforms DoLa even with optimal amateur choice) but not for TruthfulQA limits the completeness of the CD comparison on the primary factuality benchmark.

The ITI comparison (Table 1) is informative but limited to LLaMA-7B and TruthfulQA, since ITI requires supervised training on task-specific labels. ITI achieves 43.5% Truth ∗ Info vs. DoLa's 40.8% on 7B—comparable performance, but ITI uses labeled data while DoLa does not. This is a favorable comparison for DoLa's value proposition (training-free), but the paper cannot claim DoLa strictly dominates ITI on factuality.

Is the dynamic layer selection mechanism actually important, or would a simpler fixed-layer strategy work as well?

The DoLa-static ablation (Section 4.1, Figure 5) provides the key evidence. The finding that the optimal static layer shifts across random subsets of the same dataset (10th layer on GSM8K subset #1 vs. 2nd layer on subset #2) is a genuine demonstration of brittleness. However, the claim that DoLa's dynamic selection "maintains high scores in both subsets, almost matching the best performing DoLa-static layers" (Section 4.1) is supported by the data: DoLa [0, 16) achieves near-optimal accuracy on both subsets without per-subset tuning.

A limitation of this analysis is that it is only shown for LLaMA-7B on GSM8K. The paper does not provide similar subset-stability analyses for other model sizes, other tasks, or even the TruthfulQA setting where the highest bucket is used. Given that the bucket choice itself (high vs. low) dramatically changes based on task format, it would be informative to see whether the dynamic selection within the chosen bucket provides robustness benefits on TruthfulQA as well, or whether the dynamic component matters more for some task formats than others.

The random layer selection baseline (Appendix J, Table 15) convincingly shows that contrasting random layers can be worse than doing nothing—this rules out the hypothesis that any layer contrast, regardless of selection criterion, provides a benefit. The JSD criterion is necessary for the method to work. But the paper does not ablate alternative selection criteria—e.g., selecting the layer with minimum JSD, or selecting based on KL divergence rather than JSD, or selecting based on token-level probability shift magnitude. Without such ablations, it's unclear whether JSD maximization is uniquely appropriate or whether any reasonable divergence measure would work similarly.

Does DoLa genuinely work across model families, or are the results specific to LLaMA?

The MPT-7B results (Appendix G, Table 11) provide encouraging evidence of generalizability, with gains on TruthfulQA, FACTOR, and StrategyQA that mirror the LLaMA pattern. However, MPT-7B is architecturally similar to LLaMA (both are standard decoder-only transformers), and the test is limited to a single 7B model. The paper does not test on encoder-decoder architectures (T5), models with different training objectives (instruction-tuned variants like LLaMA-2-Chat), or models from substantially different families (Falcon, Mistral). The negative result on GPT-2 Medium (Appendix N) confirms that scale matters, but the threshold at which DoLa becomes effective is uncharacterized—somewhere between 335M and 7B parameters, but where? A scaling analysis with intermediate model sizes (1B, 3B) would have located this threshold and provided practical guidance for when DoLa can be applied.

Are the reported improvements practically meaningful given the modest absolute performance?

This is a domain-specific question. On TruthfulQA MC1, even with DoLa, the best-performing LLaMA-65B achieves only 31.1%—hardly a solved problem. On TruthfulQA open-ended %Truth ∗ Info, the best score is 49.2% (65B + DoLa). These are substantial improvements over the baselines (65B baseline: 34.8%), but they still mean the model fails to provide a truthful and informative answer more than half the time on a benchmark specifically designed to test common misconceptions. The paper is candid about these limitations (Section 6), noting that DoLa "relies on the model's internal knowledge... Thus, it cannot correct misinformation acquired during training."

The real-world implication is that DoLa reduces but does not eliminate hallucinations. For deployment in high-stakes settings, the residual error rate (over 50% on TruthfulQA) remains unacceptably high, and DoLa's improvements do not change the fundamental risk calculus. The method is most valuable in settings where even a 12–17% relative reduction in factual errors translates to meaningful user experience improvements—such as consumer-facing chatbots where occasional errors are tolerated but systematic falsehoods erode trust.

What experiments are missing that would strengthen the paper's claims?

Several experiments would have clarified the scope and mechanism of DoLa:

  1. TruthfulQA performance broken down by question category. TruthfulQA includes questions across 38 categories (health, law, finance, conspiracies, etc.). Reporting per-category improvements would reveal whether DoLa improves uniformly or whether it helps on some types of factual knowledge but not others.

  2. Calibration analysis. Does DoLa improve the model's confidence calibration? If DoLa surfaces latent knowledge, it might also increase the model's probability on correct answers, improving calibration. Alternatively, the contrastive operation might distort the probability distribution in ways that worsen calibration. Neither analysis is provided.

  3. Variance estimates (confidence intervals, standard deviations). The 500-question TruthfulQA test set and 80-question Vicuna QA subset are small enough that 1–2% differences may not be statistically significant. Without bootstrap confidence intervals or multiple-seed results, it's impossible to assess the reliability of the smaller reported gains (e.g., FACTOR 65B Wiki +0.2%, GSM8K 7B −0.3%).

  4. CD with optimized amateur model size on TruthfulQA. Appendix B explores amateur model choices for GSM8K only. A similar exploration on TruthfulQA, particularly with smaller amateur models that might not amplify instruction-following as strongly as 7B, would make the CD baseline more competitive and the DoLa comparison more convincing.

  5. Layer-wise analysis beyond JSD. Why do some tokens have high JSD at specific layers? Is it because factual entities are being predicted, or because the model is resolving syntactic ambiguities? A finer-grained analysis relating JSD to token-level linguistic properties (entity vs. non-entity, content word vs. function word, token position in the answer) would clarify the mechanism.

  6. Ablation on the number of candidate layers within a bucket. The paper uses even-indexed layers only. Does using all layers improve or degrade performance? Does the bucket granularity (2 vs. 3 vs. 4 buckets) affect results independently of which bucket is selected?

  7. DoLa combined with sampling strategies. All generation experiments use greedy decoding (except Vicuna QA with temperature 0.7). Would DoLa's benefits compound with best-of-N sampling, majority voting, or verifier-based selection? The paper treats DoLa as orthogonal to these strategies but does not demonstrate this.

What claims are not supported by the reported experiments?

The paper's central claim—that contrasting layers improves factuality—is well supported. However, the paper's framing implies a stronger claim: that DoLa works because factual knowledge is specifically "localized to particular transformer layers" and that contrasting "sharpens an LLM's probability towards factually correct outputs" by isolating this knowledge. The experiments demonstrate that DoLa improves factuality metrics, but they do not directly demonstrate that the mechanism is factual knowledge isolation. The alternative hypothesis—that DoLa simply amplifies whatever the later layers add, which happens to correlate with factuality on the tested benchmarks but could amplify other properties in other contexts—is not ruled out. The Vicuna QA examples where DoLa provides factually dense but topically irrelevant responses (Table 21) hint at this: the model is amplifying something from the later layers, but that something is factual knowledge in general rather than answer-relevant factual knowledge specifically. A more precise characterization of what the contrastive operation actually amplifies—through controlled experiments manipulating the knowledge content of the input—would strengthen the mechanistic claim.

The claim that DoLa requires "no additional fine-tuning" and works with "pretrained LLMs" is technically true but requires qualification: the method does require a small validation set (10% of GSM8K, or two-fold validation on TruthfulQA/FACTOR) to select the candidate layer bucket. For Vicuna QA, this validation set is borrowed from a different task. The method is training-free in the sense of not updating model weights, but it is not hyperparameter-free or entirely zero-shot—the bucket selection is a form of lightweight task-specific adaptation that requires labeled data.

6. Limitations and Trade-offs

The Method Cannot Compensate for Knowledge That Is Not Encoded in the Model

DoLa operates entirely within a single frozen model, contrasting its internal layer representations. The paper states this boundary explicitly in Section 6: "Our method relies on the model's internal knowledge without using external retrieval modules... Thus, it cannot correct misinformation acquired during training." This is not an implementation detail—it defines a hard ceiling on what DoLa can achieve. If the model's training data contained false information, or if the model failed to learn a particular fact during pretraining, no amount of layer contrasting will surface a correct answer, because the correct answer is simply not represented anywhere in the model's parameter space.

The consequence is that DoLa's effectiveness is strictly bounded by the model's pretrained knowledge. The TruthfulQA results bear this out starkly: even with DoLa, LLaMA-65B achieves only 49.2% on the combined %Truth ∗ Info metric (Table 1)—meaning the model still fails to provide truthful and informative answers more than half the time on a benchmark explicitly designed around common misconceptions. The residual errors are not failures of the decoding mechanism; they are failures of the model's knowledge base itself. For applications where factual reliability is critical, this means DoLa reduces but categorically cannot eliminate hallucinations. A deployment that requires, say, 95% factual accuracy on some domain would still need external verification, retrieval augmentation, or human oversight—DoLa alone is insufficient regardless of how well it is tuned.

The paper provides indirect evidence for this limitation through the model scale boundary. DoLa fails entirely on GPT-2 Medium (335M parameters): TruthfulQA MC3 drops from 20.0% to 16.4%, and FACTOR News collapses from 41.0% to 22.2% (Appendix N, Table 17). The authors conjecture this happens because "small LMs do not learn enough factual knowledge due to their limited capacities"—in other words, the factual knowledge that DoLa needs to surface simply is not present in the model to begin with. This demonstrates that DoLa amplifies existing knowledge rather than creating it, and when the knowledge is absent, the method backfires.

The paper does not attempt to mitigate this limitation—it is inherent to the approach. The authors acknowledge it transparently (Section 6) and suggest future combinations with retrieval-augmented methods as a potential path forward, but no combination is tested. For a practitioner, the actionable implication is: DoLa is only useful when the base model's pass@1 on the target factual domain is non-trivially above zero. If the model consistently generates wrong answers on a topic, DoLa will not rescue it.

The Method Requires Task-Specific Hyperparameter Selection (Bucket Choice) Using a Validation Set

Although DoLa is presented as a "training-free" decoding strategy requiring "no additional fine-tuning," it does require selecting one hyperparameter: which candidate layer bucket to use for dynamic premature layer selection. The paper divides layers into 2–4 buckets depending on model size and selects one bucket per task based on validation set performance (Section 3.1). This selection is consequential: for LLaMA-7B on TruthfulQA, using the "highest" bucket [16, 32) is optimal, while on FACTOR, GSM8K, and StrategyQA, the "lowest" bucket [0, 16) is optimal (Table 10, Appendix F). The performance difference between buckets is substantial—Figure 5 shows that on GSM8K, DoLa with the [0, 16) bucket achieves approximately 0.10 accuracy, while DoLa with the [16, 32) bucket achieves only approximately 0.06, effectively no better than baseline.

The consequence is that DoLa is not zero-shot. A practitioner deploying DoLa on a new task or domain must have access to a labeled validation set to choose the bucket. The paper uses 10% of GSM8K training data (shared across StrategyQA and GSM8K), two-fold validation on TruthfulQA and FACTOR, and for Vicuna QA—which lacks any validation set—simply reuses the GSM8K/FACTOR best bucket as a zero-shot transfer (Section 3.1). The transfer works in this case, but the paper provides no systematic evidence that bucket choices transfer across task families. If a practitioner is working with a custom factual domain (e.g., medical QA, legal document generation, financial reporting), they would need to construct a validation set, run 2–4 decoding experiments per model size, and hope the optimal bucket generalizes from their validation distribution to their deployment distribution. This is a lightweight form of task adaptation, but it is adaptation nonetheless, and it may not be feasible in settings where labeled validation data is unavailable.

The paper partially addresses this by showing that DoLa's dynamic selection within a bucket is more robust than DoLa-static's fixed-layer choice, which is highly sensitive to the specific validation distribution (Figure 5: the optimal static layer shifts from 10th to 2nd across different random 10% subsets of GSM8K). DoLa's bucket-level selection is coarser and more stable—the same bucket choice works across multiple tasks that share a format (lowest bucket for FACTOR, GSM8K, StrategyQA, and Vicuna QA). However, this pattern is post-hoc: a practitioner cannot know a priori whether their task is "short-answer factual" (requiring high layers) or "long-form reasoning" (requiring low layers) without experimentation. The paper does not propose a method for predicting the optimal bucket from task characteristics alone, and it does not test how bucket choice affects performance on tasks outside the studied set.

The mitigation is partial: the search space is deliberately small (2–4 options), and the paper demonstrates that the same bucket works across several tasks within a format category. But the fundamental assumption—that a small validation set is available and representative—remains a prerequisite for deployment that the paper does not eliminate.

The Difficulty Estimation Mechanism (Dynamic Layer Selection via JSD) Is Not Evaluated Against Alternative Criteria

The paper proposes selecting the premature layer at each token by maximizing the Jensen-Shannon Divergence between the early-exit distribution and the final-layer distribution: M = argmax_j JSD(q_N || q_j). The motivation is to "ensure that the model would significantly change its output after that selected layer, and thus have a higher chance to include more factual knowledge that does not exist in the early layers before it" (Section 2.2). However, the connection between "high JSD" and "factual knowledge injection" is assumed, not demonstrated.

The consequence is uncertainty about the mechanism's specificity. The paper shows that random layer selection hurts performance (Appendix J, Table 15: random selection degrades FACTOR accuracy from 58.3% baseline to 60.0% for 7B—a marginal gain—and to 53.8% for 13B, well below the 61.1% baseline). This confirms that some selection criterion is necessary and that JSD maximization is better than random. But the paper does not test whether JSD maximization is uniquely appropriate or whether any reasonable divergence metric would work about as well. Alternatives like selecting the layer with the minimum JSD (contrasting against the layer most similar to the final output, which might suppress noise rather than amplify signal), using KL divergence instead of JSD, or selecting based on the magnitude of per-token probability shift for the top-k candidates, are not explored.

Moreover, the JSD criterion may select layers based on distributional changes that have nothing to do with factuality. For example, if the model is resolving a syntactic ambiguity between layers 14 and 16, the JSD will spike at layer 14, and DoLa will contrast against that layer—amplifying whatever syntactic refinement occurred, not necessarily factual knowledge. On tasks where many tokens require non-factual refinements (long-form reasoning with complex syntactic structures, chain-of-thought with intermediate calculations), the JSD criterion may systematically select layers that are suboptimal for factuality. The paper's finding that the optimal bucket shifts from high layers (TruthfulQA) to low layers (FACTOR, GSM8K) is consistent with this concern: on long-form tasks, the JSD-selected layers within the high bucket might be driven by non-factual distribution shifts, making the high bucket ineffective, and only the low bucket—where the contrast captures the full accumulation of all knowledge, factual and otherwise—provides a stable signal.

The paper does not ablate the selection criterion. The random baseline shows JSD is better than nothing, but does not show it is better than alternatives. A simple baseline like "always select the layer where the probability of the top-1 token changes most" or "select the layer with maximum L1 distance between q_N and q_j" would help establish whether JSD's specific properties (symmetry, boundedness) matter or whether any signal of "this is where the distribution changes" suffices. For a practitioner implementing DoLa, the JSD computation adds architectural complexity (vocabulary-sized distribution comparisons at each time step); if a simpler criterion achieved equivalent results, that would be valuable to know.

The Method Is Evaluated on a Single Model Family (LLaMA) with Limited Cross-Architecture Evidence

All primary experiments use LLaMA models (7B, 13B, 33B, 65B). The paper provides one cross-architecture test with MPT-7B (Appendix G, Table 11), which shows similar gains, and one negative result with GPT-2 Medium (Appendix N, Table 17), which demonstrates a scale boundary. However, the scope of architectural validation is narrow.

The consequence is that a practitioner cannot be confident DoLa will work on their model of choice without testing. The LLaMA family shares a specific pretraining recipe, architecture (pre-norm, SwiGLU activations, rotary position embeddings), and data mixture. MPT-7B, while architecturally distinct in some ways (weight tying between embedding and LM head, which required excluding layer 0 from candidate layers), is still a standard decoder-only transformer with a similar training paradigm. The paper does not test: instruction-tuned variants (LLaMA-2-Chat) where RLHF may have reorganized the layer-wise distribution of factual knowledge; encoder-decoder architectures (T5, BART) where the layer structure and information flow differ fundamentally; mixture-of-experts models where different experts may encode knowledge differently across layers; or models with non-standard architectures (RWKV, Mamba) that lack the residual stream property DoLa exploits.

The paper's motivating hypothesis—that factual knowledge is hierarchically organized across layers—is supported by studies cited from the BERT (Tenney et al., 2019; Dai et al., 2022; Meng et al., 2022) and LLaMA literature, but the universality of this property is not established. The GPT-2 Medium negative result (Appendix N) suggests the property emerges at scale, but the specific scale threshold (somewhere between 335M and 7B parameters) is uncharacterized. DoLa might fail on a well-trained 3B model or succeed on a poorly-trained 7B model—the paper provides no scaling curve that would let a practitioner predict whether their model is "large enough" to benefit.

The paper does not attempt to mitigate this limitation beyond the single MPT-7B test. The authors frame DoLa as "a foundational improvement that could potentially be applied to any transformer-based LLMs" (Section 6), but this is an aspiration rather than a demonstrated claim. A practitioner using a model outside the LLaMA family should treat DoLa as an experimental technique requiring in-house validation, not a proven intervention.

The Reported Performance Gains Do Not Include Statistical Significance or Variance Estimates, and Some Gains Are Small Relative to the Test Set Size

The paper reports point estimates for all metrics but provides no confidence intervals, standard deviations, or statistical significance tests. The TruthfulQA test set has 817 questions for multiple-choice and open-ended evaluation; the FACTOR test sets are of comparable but unstated size; StrategyQA and GSM8K sizes are also not explicitly stated in the main experimental sections. For Vicuna QA, only 80 questions are used for the GPT-4 pairwise evaluation (Section 3.3). At these sample sizes, differences of 1–3 percentage points—which include several of DoLa's reported gains—may fall within the range of sampling variance.

Specific examples where this matters:

  • FACTOR 65B Wiki: DoLa achieves 72.4% vs. baseline 72.2%, a gain of 0.2 percentage points (Table 1). Without a confidence interval, this is indistinguishable from noise. If the Wiki test set contains, say, 500 examples, a 0.2% difference corresponds to a single example's classification difference.

  • GSM8K 7B: DoLa achieves 10.5% vs. baseline 10.8%, a loss of 0.3 points (Table 1). The paper interprets this as a slight degradation, but it is equally likely to be zero effect with sampling variance.

  • StrategyQA 13B: DoLa achieves 67.6% vs. baseline 66.6%, a gain of 1.0 point (Table 1). At plausible test set sizes for StrategyQA (the full dataset has 2,290 test examples according to Geva et al., 2021), a 1% difference corresponds to ~23 examples—potentially meaningful but not obviously statistically significant without variance estimates.

  • Vicuna QA win/tie/loss counts (Figure 4, right): For 65B, DoLa wins 39 games, ties 3, and loses 38 out of 80. A 39–38 win/loss record with 3 ties is effectively a coin toss—the method is not distinguishable from the baseline on the largest model for this task, yet the paper presents this as evidence of DoLa's effectiveness.

The consequence is that some of the paper's positive claims for smaller gains may be overstated. The large gains (e.g., TruthfulQA %Truth ∗ Info improvements of 12–17 absolute points) are clearly robust even without formal tests. But the pattern of "consistent small improvements" across many tasks—which the paper uses to argue for DoLa's generality—is less certain without variance estimates. A practitioner deciding whether to deploy DoLa for a 1–2% expected gain on a specific task cannot assess whether that gain is reliable based on the presented evidence.

The paper does not address this limitation at all. No standard deviations, bootstrap confidence intervals, or multiple-seed results are reported. Even a simple consistency check—running the Vicuna QA evaluation with different GPT-4 judging prompts or different random seeds for the 0.7 temperature sampling—would help quantify variance. This is a methodological gap that weakens the evidence for DoLa's effectiveness on tasks where gains are modest.

The Method Can Degrade Instruction-Following and Relevance in Favor of Factual Density

DoLa's contrastive mechanism amplifies whatever information is added in the later layers relative to the early layers. On short-answer factual tasks (TruthfulQA), this predominantly amplifies factual knowledge because the answer is fact-critical and short. On longer-form generation tasks, the paper provides qualitative evidence that DoLa sometimes produces factually dense but task-irrelevant responses—the model prioritizes surfacing factual knowledge over addressing the specific question asked.

The Vicuna QA qualitative examples (Appendix M) contain two documented failure cases. In Table 21, the prompt asks the model to "describe your daily life and the challenges you face living on another planet" as a space colonist on Mars. The baseline LLaMA-33B provides a relevant, structured answer enumerating challenges (breathing, temperature, gravity, radiation, water, food, supplies, communication, isolation, danger). DoLa responds with a dense paragraph of factual information about Mars (day length, orbital period, surface temperature, atmospheric pressure, gravity, water ice, magnetic field, radiation, moons) that is factually accurate but largely misses the "daily life and challenges" framing of the question. GPT-4 rates DoLa lower (6/10 vs. 8/10) with the review: "Assistant 2's answer, while informative, focused more on general facts about Mars and its features, rather than addressing the daily life and challenges of a space colonist."

In Table 22, the prompt asks "How many balloons would it take to lift a house like in the movie 'Up'? Try to explain your answer." The baseline provides a concise calculation using buoyant force equations referenced to the movie. DoLa provides an extensively detailed calculation using U.S. Department of Energy data on average home sizes, latex balloon weights from the National Balloon Museum, and Mylar balloon specifications—providing multiple inconsistent estimates within the same response and ending with an irrelevant sentence fragment ("A chat between a curious human and an artificial intelligence assistant..."). GPT-4 rates DoLa lower (5/10 vs. 7/10), noting the response "contained some irrelevant information... which made the answer more confusing."

The consequence is a fundamental tradeoff: DoLa makes the model more factually forthcoming, but this comes at the potential cost of reduced relevance and instruction-following precision. The paper acknowledges this in Appendix M: "although DoLa tends to generate detailed factual information, sometimes it will not be as relevant to the question as the baseline's answer. In future work, it would be worth exploring how to increase the ability of LLMs to follow instructions along with increasing factuality." However, this acknowledgment appears only in the appendix qualitative section—the main paper (Sections 3.3, 4.3) presents only the positive examples where DoLa corrects factual errors. A practitioner deploying DoLa in a chatbot or assistant setting, where relevance and task-following are as important as factuality, would need to be aware of this tradeoff. The Vicuna QA 65B results—where DoLa effectively ties the baseline (39 wins to 38 losses, Figure 4)—may partly reflect this tension: factual improvements are offset by relevance degradations on some questions.

The paper does not attempt to mitigate this limitation. The dynamic layer selection criterion (JSD maximization) operates per-token and has no mechanism for detecting whether the factual knowledge being amplified is relevant to the user's request. The adaptive plausibility constraint filters implausible tokens but does not filter irrelevant ones. The repetition penalty addresses a different issue (content recycling). This limitation is thus fundamentally unresolved and represents an area where combining DoLa with instruction-following techniques (or using different bucket choices for different response types within a single generation) could be productive but is unexplored.

7. Implications and Future Directions

How This Work Changes the Landscape

DoLa represents a reframing of the hallucination problem rather than a paradigm shift or a solved problem. Its core conceptual contribution is establishing that at least some hallucinations arise not from a knowledge deficit but from an expression failure—the model encodes correct factual knowledge in its later-layer representations, but standard decoding dilutes this knowledge by mixing it with early-layer syntactic and surface-level patterns that dominate the residual stream. This reframing is significant because it redirects attention from acquiring more knowledge (through larger models, more training data, or retrieval augmentation) toward better accessing knowledge already present (through decoding-time interventions on internal representations). It is not the first work to propose inference-time factuality interventions (ITI from Li et al., 2023, and various retrieval-augmented methods preceded it), but it is the first to demonstrate that a purely internal, training-free, single-model contrastive mechanism can yield gains of 12–17 absolute percentage points on TruthfulQA %Truth ∗ Info across multiple model scales—comparable to ITI's gains but without requiring supervised labels.

The work resolves a latent contradiction in the contrastive decoding literature. Prior contrastive decoding work (Li et al., 2022) focused on improving fluency and coherence by contrasting an expert model against an amateur model, with the implicit assumption that the expert-amateur gap captures general "quality." When applied to factuality, this cross-model CD approach produced perverse results—the paper shows that CD on TruthfulQA causes LLaMA-33B to answer "I have no comment" over 60% of the time, because the expert-amateur gap captures instruction-following ability rather than factual knowledge (Table 1). DoLa's within-model layer contrast avoids this conflation by targeting a more specific capability gradient: the difference between what early layers predict (syntax, surface patterns) and what later layers add (factual knowledge). This explains why cross-model CD was ineffective for factuality even when it improved other dimensions of generation quality, and it provides a diagnostic for when contrastive methods will and will not work: they help when the capability being contrasted is cleanly separated along the contrast axis (layers for factuality, model scale for general quality) and fail when the axis conflates multiple capabilities.

Perhaps the most important shift is the establishment of a scale boundary for layer-based factuality interventions. The paper's negative result on GPT-2 Medium (Appendix N, Table 17)—where DoLa actively degrades factuality, with FACTOR News accuracy collapsing from 41.0% to 22.2%—and the concurrent finding from Gera et al. (2023) that autocontrastive decoding on small models increases hallucinations, together demonstrate that the layer-wise factual knowledge gradient DoLa exploits is an emergent property of scale. Small models do not organize knowledge hierarchically enough for layer contrast to isolate factual content from linguistic content. This boundary condition is important because it prevents the naïve application of DoLa-style methods to smaller or less capable models—a practitioner with a 1B-parameter domain-specific model expecting DoLa to reduce hallucinations would be disappointed. It also suggests a broader principle: interpretability-inspired decoding interventions may only become effective above a threshold model size where representations become sufficiently structured. This makes the case both for scaling (to reach the threshold where such interventions work) and for interpretability research (to understand what changes at the threshold).

The work also shifts the relative attractiveness of different research directions in hallucination mitigation. It makes within-model decoding interventions substantially more attractive: DoLa demonstrates that simple, cheap, training-free operations on existing representations can yield gains comparable to supervised methods (ITI) and superior to cross-model methods (CD). It makes cross-model contrastive decoding for factuality less attractive: the paper's extensive baseline search (Appendix B, Table 5) shows that even with careful amateur model selection, CD cannot reliably match DoLa on factual reasoning tasks. And it leaves retrieval augmentation uncompromised: DoLa is explicitly complementary to retrieval, operating on internal knowledge while retrieval supplies external knowledge, and the paper's own limitations (Section 6) note that combining the two is a natural next step. The practical implication for research priority is that improving how models access their internal knowledge and improving how they ground in external knowledge are independent axes that can and should be pursued in parallel.

Follow-Up Research This Work Enables

Combining DoLa with retrieval-augmented generation to test whether internal and external knowledge improvements are additive or sub-additive. DoLa improves the model's ability to surface internally encoded factual knowledge; retrieval-augmented methods (Borgeaud et al., 2022; Izacard et al., 2022) ground generation in external documents. The paper explicitly notes this combination as future work (Section 6), but the key empirical question is whether the gains compound. If DoLa helps the model correctly use retrieved context (by suppressing early-layer "hallucination" patterns that might override retrieved facts in favor of parametric priors) or if the two mechanisms address mostly non-overlapping errors (DoLa corrects internal knowledge failures on facts the model knows; retrieval corrects failures on facts the model never learned), the net improvement could exceed the sum of individual gains. A strong follow-up would test DoLa with a retriever on a benchmark that includes both questions the model likely knows (TruthfulQA-style common misconceptions) and questions requiring external knowledge (fresh news, obscure facts), measuring whether the improvement is additive, sub-additive, or synergistic. The TruthfulQA result that DoLa cannot help when knowledge is absent (it degrades on GPT-2 Medium, Section Appendix N) suggests DoLa's gains will be concentrated on "known" facts while retrieval handles the rest—predicting additivity—but the Vicuna QA examples where DoLa sometimes drifts into irrelevant factual density (Table 21) raise the possibility that the contrastive mechanism could interfere with context-following, potentially making the combination sub-additive on tasks requiring close adherence to retrieved passages.

Training a lightweight difficulty predictor to select the optimal contrast bucket per-query, eliminating the need for validation sets. DoLa's biggest practical limitation is the requirement for a labeled validation set to choose the candidate layer bucket—either the highest bucket (short factual answers) or the lowest bucket (long reasoning chains). This bucket choice is consequential: on GSM8K, using the wrong bucket (high instead of low) reduces DoLa's accuracy to near-baseline levels (Figure 5). The paper establishes that the optimal bucket correlates with generation format (short vs. long, fact-dense vs. reasoning-heavy), but a practitioner with a novel task cannot know a priori which bucket to use. A natural follow-up is to train a classifier—using features like question length, presence of reasoning keywords, or few-shot prompting patterns—to predict the optimal bucket from the input alone, trained on the tasks where bucket choice is known (TruthfulQA → high, FACTOR/GSM8K → low). A strong result would demonstrate that such a classifier generalizes to held-out tasks (e.g., correctly predicting "high" for short-answer medical QA and "low" for multi-hop legal reasoning) and achieves performance close to validation-selected buckets. Even a simple heuristic—"if the expected output has more than N tokens, use the low bucket"—might capture most of the variance given the paper's format-based explanation. This would make DoLa genuinely zero-shot, addressing what the paper identifies as a key limitation of DoLa-static (Section 2.2: "best layers are sensitive to data distribution, thus requiring in-distribution validation sets") that DoLa's dynamic selection only partially resolves.

Adversarial evaluation of DoLa against targeted factual attacks to stress-test whether layer contrast creates new failure modes. DoLa amplifies whatever the later layers add relative to early layers. The paper's qualitative evidence shows this is predominantly factual knowledge on the tested benchmarks, but the mechanism is fundamentally agnostic to what is being amplified—it does not distinguish factual knowledge from other late-layer contributions. An adversarial evaluation could construct inputs designed to inject false information late in processing: for example, prompts with misleading late-stage context that the model's later layers would incorporate, or fine-tuned model variants where factual knowledge is deliberately placed in early layers and misinformation in later layers. If DoLa amplifies the misinformation (because it contrasts against early layers where the correct information resides and in favor of late layers where the false information dominates), this would reveal that the method's factuality improvements are contingent on the standard hierarchical organization of knowledge in pretrained LMs and could be exploited or fail under distribution shift. The paper's Vicuna QA failure cases (Tables 21–22), where DoLa provides factually dense but irrelevant responses, already hint that the contrastive mechanism can amplify the wrong thing when the late-layer contribution is not answer-relevant factuality. A systematic adversarial study would define the boundary conditions of DoLa's reliability and could inform the design of safeguards (e.g., combining JSD-based layer selection with a relevance filter that gates the contrast when the late-layer contribution appears off-topic).

DoLa applied to instruction-tuned and RLHF-trained models to test whether alignment training reorganizes layer-wise knowledge. All experiments use base LLaMA models without instruction tuning or RLHF. These alignment procedures could significantly reorganize how factual knowledge is distributed across layers. RLHF, in particular, trains models to produce outputs that humans prefer, which may include prioritizing helpfulness and harmlessness over factual accuracy, or learning to express uncertainty rather than confidently stating falsehoods (exactly the "I have no comment" behavior that DoLa reduces in base models). If RLHF moves factual knowledge earlier in the layer stack (because truthfulness is reinforced during training, making it a more automatic, "early-layer" behavior) or if it trains the model to gate factual knowledge behind later-layer "safety" computations, DoLa's optimal bucket choice could shift substantially. The paper's result that TruthfulQA MC1 performance is only 30.8% for the 65B base model (Table 1) highlights that base LLaMA models are far from truthful even at scale, and the entire motivation for DoLa is that these models know facts but fail to express them. An RLHF-trained model that has already been optimized to better express its knowledge might show smaller or even negative gains from DoLa (if the late-layer factual signal is already cleanly dominant). Conversely, if RLHF suppresses certain factual knowledge that humans find uncomfortable, DoLa might inadvertently resurface it—a dual-use concern worth investigating. Testing DoLa on LLaMA-2-Chat at 7B/13B/70B would directly answer whether the layer-wise knowledge gradient survives alignment and whether DoLa provides gains on top of RLHF.

Layer-wise probing to causally verify that DoLa amplifies factual knowledge specifically rather than an uncharacterized late-layer signal. The paper's central mechanistic claim is that DoLa improves factuality because it "sharpens an LLM's probability towards factually correct outputs" by isolating factual knowledge that emerges in later layers. The evidence for this is correlational: DoLa improves factuality metrics, and Figure 2 plus Appendix A show that factual tokens have high late-layer JSD. But the paper does not causally demonstrate that the specific tokens being boosted by DoLa are the ones encoding factual knowledge. A causal probing experiment could intervene on the model's internal representations to directly test the mechanism. For example: identify factual knowledge neurons using the method of Dai et al. (2022) or Meng et al. (2022); ablate these neurons in the later layers; and measure whether DoLa's factuality gains disappear. If the gains persist despite factual neuron ablation, DoLa must be amplifying something other than factual knowledge (perhaps confidence calibration or syntactic disambiguation), and the paper's mechanistic claim would be falsified. Alternatively, if ablating factual neurons eliminates DoLa's gains while leaving baseline performance unchanged, this would confirm that DoLa specifically depends on the factual knowledge stored in those neurons. A positive result would strengthen the paper's explanatory framework; a negative result would not necessarily invalidate DoLa as a practical method but would require revising the account of why it works, which matters for predicting when and where the method will generalize.

Scaling analysis of DoLa's effectiveness as a function of model size to precisely locate the threshold where layer-wise factual knowledge becomes exploitable. The paper provides two data points: DoLa fails on GPT-2 Medium (335M parameters, Appendix N) and succeeds on LLaMA-7B (7B parameters). The threshold for effectiveness is somewhere in between, but the specific scaling behavior is unknown. Does DoLa's improvement increase smoothly with model size, or is there a phase transition where it abruptly starts working? Does the optimal bucket choice remain stable as models scale within a family, or does it shift (e.g., does factual knowledge move to proportionally later layers in larger models)? A scaling analysis using LLaMA models at intermediate sizes (if available) or models from other families with published intermediate checkpoints (e.g., Pythia, OPT, or OLMo) across the 1B–7B range could answer these questions. The practical value is guidance for practitioners: at what parameter count should one start considering DoLa? The scientific value is understanding how the hierarchical organization of knowledge in transformers emerges with scale—whether it is a smooth increase in differentiation or a qualitative shift at a critical size. The concurrent finding from Gera et al. (2023) that layer contrasting increases hallucinations in ~300M models, combined with DoLa's success at 7B, suggests the transition is somewhere in the 1–3B range, but this is speculation without data.

Practical Applications and Downstream Use Cases

Low-resource deployment of truthful chatbots and assistants without retrieval infrastructure. DoLa's primary practical value is enabling more truthful generation from a single frozen model with no external dependencies. For applications where deploying a retrieval index is impractical—edge devices, offline mobile assistants, embedded systems in vehicles or medical devices, or applications in low-connectivity environments—DoLa offers a 12–17 percentage point improvement in truthfulness (TruthfulQA %Truth ∗ Info, Table 1) at the cost of only 1–8% additional latency (Table 2) and sub-2% additional memory (Table 9). This is a rare combination: meaningful accuracy gains with negligible deployment cost. A medical symptom checker running on-device, for instance, could apply DoLa to a quantized 7B model to reduce the rate of dangerous fabricated medical advice, without requiring a network connection to query a knowledge base. The method is also immediately deployable on existing LLaMA-based production systems—it requires no retraining, no model architecture changes, and no additional serving infrastructure beyond retaining intermediate hidden states during the forward pass. For any organization already serving LLaMA-family models and experiencing user complaints about hallucinated facts, DoLa is a drop-in improvement with essentially zero integration risk.

Factual data generation and distillation pipelines. When using LLMs to generate training data—for fine-tuning smaller models, for synthetic dataset creation, or for self-improvement loops (e.g., STaR-style bootstrapping)—the factual accuracy of generated content directly determines downstream model quality. DoLa can be applied during the generation phase to increase the proportion of factually correct outputs without slowing generation appreciably (the throughput penalty is 1–7%, Table 2). For a data generation pipeline processing millions of examples, a 12% improvement in truthfulness (the minimum gain across all LLaMA sizes on TruthfulQA %Truth ∗ Info) that requires only a decoding flag change is highly cost-effective. Moreover, because DoLa reduces the rate of "I have no comment" rejections (from 38.1% to 8.2% on LLaMA-33B, Table 1), it increases the yield of usable, substantive responses—important for pipelines where generation cost is a bottleneck. The paper's limitation that DoLa occasionally trades relevance for factual density (Vicuna QA examples, Tables 21–22) should inform pipeline design: DoLa-generated data may be more factual but less instruction-following, and downstream models trained on this data may inherit a bias toward factual verbosity over task-specific precision. Filtering or mixing DoLa and baseline generations based on a relevance metric could mitigate this.

Improving chain-of-thought reasoning accuracy in math and logic applications. DoLa's 2–4 percentage point gains on StrategyQA and ~2 point gains on GSM8K (Table 1), while modest, come essentially for free in terms of latency. For applications where math or logic reasoning accuracy translates to business value—automated tutoring systems, financial analysis assistants, code generation for quantitative tasks—these gains compound with other decoding improvements (majority voting, verifier-based selection) and with the trend toward test-time compute scaling. The paper does not combine DoLa with sampling-based strategies, but nothing prevents using DoLa's modified distribution as the basis for best-of-N sampling or consensus voting. A practical setup: use DoLa's distribution to generate N candidate CoT solutions, then apply majority voting or a trained verifier. The DoLa distribution should produce a higher proportion of factually grounded reasoning steps in each candidate, increasing the probability that the consensus answer is correct. The paper's finding that the repetition penalty is necessary for DoLa on long CoT generations (Appendix K, Figures 7–8) but hurts the baseline is a concrete implementation detail practitioners should adopt: use θ = 1.2 with DoLa on reasoning tasks, and θ = 1.0 (no penalty) otherwise.

When to Prefer This Method

The paper positions DoLa against three alternatives: standard decoding (the vanilla baseline), Contrastive Decoding across models (CD; Li et al., 2022), and Inference Time Intervention (ITI; Li et al., 2023). The tradeoffs are specific and empirically grounded:

  • Prefer DoLa over standard decoding when the base model is a large (≥7B parameters) pretrained transformer LM and the task involves factual accuracy in short answers or reasoning chains, and when the 1–8% latency overhead (Table 2) is acceptable. The expected gain is 12–17 absolute percentage points on open-ended truthfulness (Table 1) and 2–4 points on multiple-choice factuality and CoT reasoning, for essentially zero integration cost beyond a decoding flag change. DoLa's dynamic layer selection means it works without per-task layer tuning, though a small validation set is needed to choose the candidate layer bucket (high vs. low).

  • Prefer DoLa over Contrastive Decoding (CD) when factuality, rather than general generation quality, is the primary objective, and when deploying only a single model is preferable to maintaining two. CD can produce high truthfulness on the responses it gives but at the cost of extremely high rejection rates (62.7% for LLaMA-33B on TruthfulQA, Table 1), making it unusable for applications where substantive answers are required. CD also requires careful amateur model selection (Appendix B, Table 5), and even optimized CD fails to match DoLa on factual reasoning (GSM8K). The paper's qualitative evidence (Table 3) further shows that DoLa corrects specific factual errors (e.g., "July 4, 1776" → "August 2, 1776") rather than simply increasing refusal, making it suitable for user-facing factual QA where "I don't know" is not an acceptable default.

  • Prefer DoLa over Inference Time Intervention (ITI) when labeled truthfulness data is unavailable for training the ITI classifier. ITI achieves comparable or slightly better truthfulness scores on TruthfulQA (43.5% vs. DoLa's 40.8% on LLaMA-7B %Truth ∗ Info, Table 1), but requires supervised training on the target task's labels—it cannot be applied zero-shot to new domains. DoLa's only task-specific requirement is a small validation set for bucket selection (2–4 runs), and the paper shows the bucket choice transfers across similar-format tasks (FACTOR's bucket works for GSM8K, StrategyQA, and Vicuna QA). For a new factual domain where gathering supervised labels is expensive but a small validation set is feasible, DoLa is the more practical choice.

  • Prefer standard decoding (or avoid DoLa) when the model is small (<7B parameters), as the layer-wise knowledge gradient may not be sufficiently structured for DoLa to help—the paper shows DoLa degrades performance on GPT-2 Medium (Appendix N). Also prefer standard decoding when instruction-following precision and response relevance are paramount and the task involves long-form, open-ended generation where DoLa's tendency to surface factual knowledge can drift into irrelevant factual density (as in the Vicuna QA Mars colonist and balloon-lift examples, Tables 21–22). The paper's 65B Vicuna QA results—where DoLa effectively ties the baseline at 39 wins to 38 losses (Figure 4)—suggest that on the largest models, DoLa's factuality gains and relevance losses may approximately cancel out for open-ended chatbot evaluation, making the choice a domain-specific judgment call based on whether factual errors or irrelevant tangents are more harmful to the user experience.