Read More — Paper Graph

🎯 Pitch

A 7B parameter language model outperforms a 540B model on a key benchmark just by reweighting output probabilities to emphasize the prompt—no extra training needed. This same trick makes AI assistants 75% more likely to follow quirky instructions, effectively doubling a model's capabilities for free at inference time.

1. Executive Summary

This paper adapts Classifier-Free Guidance (CFG)—an inference-time technique originally developed for text-to-image diffusion models—to autoregressive language modeling and demonstrates that it broadly improves prompt adherence across a wide array of tasks using Pythia, GPT-2, and LLaMA-family models. The core mechanism is a logit-space reweighting that amplifies the influence of the prompt conditioning by interpolating between the conditional and unconditional next-token distributions (example: generating "The dragon flew over Paris, France" with CFG upweights tokens related to dragons and Paris while downweighting unrelated locations like "Queensland"). The paper reports that CFG achieves a 75% human preference rating for GPT4All assistant responses, delivers improvements equivalent to doubling the model's parameter count, and sets a new SOTA on LAMBADA with LLaMA-7B over PaLM-540B—establishing that this training-free, out-of-the-box technique reliably increases prompt-adherence across basic prompting, chain-of-thought, code generation, and chatbot-style prompting settings.

2. Context and Motivation

The Core Problem: Language Models Don’t Stay on Topic

The fundamental problem this paper addresses is deceptively simple: language models lose adherence to their prompts over the course of generation. A user provides a prompt—whether it's a question, an instruction, a few-shot example, or a chatbot system directive—and the model begins generating a plausible continuation. However, as the generation extends, the model's output often drifts from the original intent expressed in the prompt. The paper identifies several manifestations of this problem:

Hallucination [49]: the model generates factually incorrect or unsupported statements that diverge from what the prompt actually requests.
Degradation [38]: the statistical quality of generated text deteriorates as generation continues, producing repetitive or nonsensical content that loses connection to the initial conditioning.
Meandering [76]: the model gradually shifts topics, starting on-prompt but ending somewhere unrelated.

This is not merely a nuisance—it fundamentally undermines the reliability of language models in practical applications. When a model is asked to follow a system-level directive (e.g., "write an enthusiastic response") and instead produces a neutral one (see Table 1), or when it's given a math problem and produces a reasoning chain that veers into unrelated territory, the model has failed at the core task of following instructions. The paper's central hypothesis is stated explicitly:

"focusing more on the prompt at inference-time will result in generations that better align with expected behavior."

Why This Matters: The Shift Toward Inference-Time Control

The significance of this problem has grown as the field's relationship with language models has evolved. The paper articulates several practical concerns that make prompt-adherence critical:

Smaller models, larger ambitions. While the largest models (GPT-4, PaLM-540B) show relatively strong prompt-adherence due to scale alone, smaller and more accessible models struggle. The authors demonstrate this across GPT-2, Pythia, and LLaMA families—models that researchers and practitioners actually use due to hardware constraints. If a technique can make a 7B parameter model behave like a 14B parameter model in terms of prompt-following, that dramatically expands who can deploy reliable language systems.

Complex, multi-part prompting is becoming standard. Modern usage patterns have moved far beyond simple text completion. The paper identifies a taxonomy of increasingly complex prompting paradigms:

Basic zero-shot prompting (Section 3.1): a simple question or instruction, expecting a short answer. Drift is less severe here because the completion is short, but the paper still finds improvements from CFG.
Chain-of-Thought prompting (Section 3.2): the model generates a multi-step reasoning chain before producing an answer. Long chains are especially prone to divergence—the model starts reasoning correctly but loses the thread or fails to produce a parsable answer structure.
Text-to-text generation (Section 3.3): code generation and machine translation require long, structured completions that must remain faithful to the prompt specification throughout.
Chatbot-style multi-stage prompting (Section 3.4): the prompt is split into a system prompt (defining the assistant's behavior, persona, or constraints) and a user prompt (the actual query). The model must satisfy both simultaneously, and the paper observes that systems like Alpaca [77, 59, 3] "often ignore changes to their default system-prompt."

Each of these settings places increasing demands on prompt-adherence, and failures become more costly as the generation length grows. A code generation model that starts writing a correct function but drifts into unrelated logic produces unusable output; a chatbot that ignores its behavioral guidelines may produce harmful or inappropriate responses.

The cost of training-based solutions. The dominant approach to improving prompt-following has been instruction fine-tuning [81, 70] and reinforcement learning from human feedback (RLHF) [56, 4, 6]. These techniques demonstrably improve alignment, but the paper emphasizes their prohibitive cost: they require large curated datasets, significant compute for fine-tuning, and in the case of RLHF, an ongoing human annotation pipeline. The authors state directly that these approaches "are expensive and their compute and data cost may not be accessible to all users." An inference-time technique that works without any additional training—applied to the model as-is—would democratize access to better prompt-adherence.

Prior Approaches and Where They Fall Short

The paper situates itself within two broad traditions of addressing generation control, and identifies specific limitations in each.

Training-Time Solutions: Powerful but Expensive

The most successful approaches to improving prompt-adherence operate during training:

Instruction fine-tuning [81, 70]: models are fine-tuned on datasets of (instruction, response) pairs, teaching them to follow task specifications expressed in natural language. This approach, exemplified by FLAN, T0, and Alpaca, produces models that are demonstrably better at following prompts. However, the paper notes that even instruction-tuned models exhibit the drift problem, particularly with system-prompt modifications in chatbot settings. The authors observe that Alpaca-based systems "often ignore changes to their default system-prompt, and may even expose models to attacks like prompt injection" [36].

RLHF [56, 4, 6]: models are fine-tuned using reinforcement learning where the reward signal comes from human preferences, explicitly training them to produce outputs that humans judge as helpful and aligned. This approach underpins ChatGPT and similar systems, but requires an extensive human annotation infrastructure and careful reward modeling. The paper positions CFG as an orthogonal technique that could complement such approaches without the ongoing cost of human feedback collection.

Controlled generation via pretrained conditional models: CTRL [40] was an early approach that pretrained language models with explicit "control codes" (e.g., "Science", "Romance") that could steer generation toward specific styles or domains. However, this requires purpose-built training from scratch and offers only coarse-grained control over a fixed set of attributes—not the flexible, prompt-level control that modern usage demands.

The common thread: training-time approaches are effective but require model modification. Once a model is deployed, its prompt-following behavior is largely fixed. If a user needs a 7B model to follow instructions better, instruction-tuning or RLHF that model requires resources they may not have access to.

Inference-Time Vocabulary Manipulation: A Growing Research Direction

A parallel line of work has explored controlling generation by directly manipulating the model's output distribution $p(x_n|x_{<n})$ at inference time, without modifying model weights:

Basic heuristics are now standard: temperature scaling [17] sharpens or flattens the token distribution; nucleus sampling [38] restricts generation to the most probable tokens; repetition penalties [31] discourage the model from repeating itself. These are widely used but are content-agnostic—they operate on statistical properties of the distribution without regard to what the prompt actually says. They cannot specifically boost prompt-relevant tokens.

Plug-and-Play Language Model (PPLM) [24] was the earliest work achieving controlled generation through vocabulary distribution modification on vanilla pretrained models. PPLM used Bayes Rule to factorize $p(x_n|x_{<n}, a) \propto p(x_n|x_{<n})p(a|x_n, x_{<n})$ , where $a$ is a desired attribute, and trained a discriminator $p(a|x)$ to steer generation. This approach showed that inference-time steering was possible, but required training a separate classifier for each attribute—reintroducing a training dependency for every new control dimension.

FUDGE [84] extended this idea with "future discriminators" that predict whether a partial completion will eventually satisfy a desired attribute, allowing more fine-grained control. GeDi [43] used class-conditional language models as discriminators to guide generation toward specific styles. NA-DO [50] went further by decomposing the oracle discriminator into simpler components. Each of these works demonstrated that manipulating the vocabulary distribution could effectively control generation, but each required auxiliary models trained for specific control dimensions.

Contrastive Decoding [45] is perhaps the most directly related prior work. This approach computes $f_s(x|y) - f_w(x|y)$ , where $f_s$ is a strong "expert" model and $f_w$ is a weaker one, to generate text that amplifies the expert's strengths while suppressing common patterns. The paper notes this is similar in spirit to CFG's interpolation, but requires access to two different models—a practical constraint that limits deployment scenarios.

The gap these approaches leave: all of these inference-time methods require either auxiliary trained classifiers, or multiple models, or are designed for coarse attribute-level control rather than fine-grained prompt-level control. None provides a simple, training-free, single-model method for saying: "generate text that adheres more strongly to this specific prompt."

The Text-to-Image Inspiration: Classifier Guidance and CFG

The paper draws direct inspiration from the text-to-image generation literature, where the problem of prompt-adherence has been studied extensively in the context of diffusion models.

Classifier Guidance [28] was proposed to enhance the generative quality of diffusion models by using a separate classifier $P_\phi(c|x)$ —trained to predict whether an image matches a text description $c$ —to guide the sampling process. By adding the classifier's gradient $\gamma \nabla_z P_\phi(c|x)$ to the denoising process, the model's output distribution is reweighted to favor images that the classifier associates with the conditioning text. This results in approximate samples from:

$\hat{P}(x|c) \propto P_\theta(x) \cdot P_\phi(c|x)^\gamma$

where $\gamma$ is the guidance strength. At $\gamma = 0$ , this reduces to unconditional generation; at $\gamma = 1$ , conditional generation; and at $\gamma > 1$ , the conditioning is overemphasized, which [28] found improves "inception score at the cost of diversity." This approach has been widely adopted [32, 41, 22] but requires training a classifier that works with intermediate representations of the diffusion process—a non-trivial addition.

Classifier-Free Guidance (CFG) [37] eliminated the need for a separate classifier through a clever observation. Using Bayes Rule, the classifier term can be rewritten as $P_\theta(c|x) \propto \frac{P_\theta(x|c)}{P_\theta(x)}$ , meaning the generative model can serve as its own implicit classifier if it's trained to support both conditional and unconditional generation. The key training requirement is conditioning dropout: during training, the conditioning information is randomly dropped (replaced with a null token) so the model learns both $P_\theta(x|c)$ and $P_\theta(x)$ . At inference time, sampling follows:

$\log \hat{P}_\theta(\epsilon_t|x_{t+1}, c) = \gamma \log P_\theta(\epsilon_t|x_{t+1}, c) - (\gamma - 1) \log P_\theta(\epsilon_t|x_{t+1})$

This is interpreted as a vector arithmetic operation in latent space: take a step of size $\gamma$ away from the unconditional prediction in the direction of the conditional prediction. The paper notes that this formulation enables negative prompting: instead of starting from the unconditional distribution, start from a "negative conditioning" $\bar{c}$ (representing what you don't want) and move toward the positive conditioning $c$ . This yields outputs that emphasize the difference between $c$ and $\bar{c}$ , and has become a widely-used technique in text-to-image generation [29, 1, 23, 65] for improving image quality and adherence to complex prompts.

The critical adaptation challenge: CFG in diffusion models works because the models are specifically trained with conditioning dropout. Language models, however, are trained differently—they see complete sequences and learn to predict continuations, with the "conditioning" simply being the prefix. The key question this paper addresses is: can CFG be applied to language models without any additional training?

How This Paper Positions Itself

The paper's positioning is distinctive along several dimensions:

1. Training-free, architecture-agnostic inference-time control. Unlike text-to-image CFG which requires conditioning dropout during training, the paper observes that autoregressive language models naturally support both conditional $P_\theta(w|c)$ and unconditional $P_\theta(w)$ generation—simply by including or excluding the prefix $c$ from the context window. This is possible because language models are trained on finite context windows and naturally handle variable-length prefixes. The paper states:

"Being able to drop the prefix $c$ is a natural feature. We can thus sample the next $i$ -th token $w_i$ in the logits space."

This means CFG works out-of-the-box on any pretrained language model, without modifying the model, without training auxiliary classifiers, and without conditioning dropout. This is a significant departure from both the text-to-image CFG paradigm and from prior controlled text generation approaches.

2. Operating in logit space. The paper defines the semantic space for CFG operations as the model's logits (the pre-softmax activation values). This choice is justified by reference to work showing that word embeddings and sentence embeddings have strong semantic structures [51, 60, 27, 61], and by the practical advantage that "using the logits avoids network editing and is architecture agnostic." The CFG operation becomes:

$\log \hat{P}_\theta(w_i|w_{j<i}, c) = \log P_\theta(w_i|w_{j<i}) + \gamma\big(\log P_\theta(w_i|w_{j<i}, c) - \log P_\theta(w_i|w_{j<i})\big)$

This is a direct translation of the diffusion model formula into the autoregressive setting, operating at every token generation step rather than at every denoising step.

3. A unified framework across diverse prompting paradigms. Rather than proposing a new prompting technique, the paper positions CFG as a meta-technique that can be layered on top of any existing prompting approach. The experimental design deliberately covers the full spectrum: basic prompting (Section 3.1), chain-of-thought (Section 3.2), long-form generation including code and translation (Section 3.3), and multi-stage chatbot prompting (Section 3.4). The paper also demonstrates that CFG "can stack alongside other inference-time methods like Chain-of-Thought and Self-Consistency, yielding further improvements in difficult tasks." This positions CFG not as a replacement for existing techniques but as a general-purpose amplifier that enhances whatever prompting strategy is already in use.

4. A computational cost analysis that reframes the value proposition. Recognizing that CFG doubles inference FLOPs (two forward passes per token instead of one), the paper directly addresses whether users should simply run a model twice as large instead. Section 4 presents a FLOPs-matched comparison across 9 benchmarks, finding that across 5 out of 9 tasks, there is "a statistically insignificant difference between using CFG and using vanilla prompting with a model of twice the size." This has "enormous implications for training budgets and inference latency due to limited VRAM usage," since the model with CFG achieves comparable performance while requiring half the training compute, half the parameter storage, and potentially less VRAM (if running serially on the same hardware) compared to the larger vanilla model.

5. Explanatory depth beyond performance reporting. Unlike many technique papers that simply report benchmark improvements, this paper dedicates Section 5 to understanding why CFG works: analyzing entropy reduction in the sampling distribution (Section 5.1), comparing CFG's effects to instruction-tuning (Section 5.2), and providing qualitative visualizations of which tokens get upweighted and downweighted at each generation step (Section 5.3). The visualization approach—ranking vocabulary items by $\log P(w_t|w_{<t}) - \log P(w_T|\hat{w})$ at each timestep—is proposed as "an integral part of effective prompt engineering," giving practitioners a tool to understand and debug their prompts' effects on generation. This moves CFG from a black-box technique to an interpretable one.

6. Candid about limitations and risks. The paper acknowledges that CFG "requires tweaking and exploration: $\gamma$ values that might work in one context (i.e. long-form generation) might be poorly suited for another context." It also raises concerns about potential misuse, noting they "have not tested the effects of CFG if used in conjunction with malicious strategies for hacking language models, including but not limited to: prompt injection and prompts aimed at overriding alignment." This frankness about the technique's sensitivity to hyperparameters and potential security implications distinguishes the paper from work that presents techniques as universally beneficial.

The Broader Intellectual Context

The paper can be understood as part of a larger shift in how the field thinks about inference. For years, the dominant narrative was that better performance came from bigger models trained on more data—the scaling laws paradigm. Inference was treated as a fixed-cost step: feed in the prompt, run one forward pass per token, get the output. Techniques like CFG and Contrastive Decoding [45] represent an alternative philosophy: invest more compute at inference time to get better output from a given model, rather than investing that compute in training a bigger model. This inference-time compute can be traded off against training-time compute, with different tradeoffs depending on deployment constraints (VRAM limits, latency requirements, training budgets).

The paper's contribution to this shift is to provide a simple, universally applicable technique that requires no model modification whatsoever. While Contrastive Decoding needs two models and instruction-tuning needs a fine-tuning pipeline, CFG works on any autoregressive language model immediately. This accessibility—combined with the paper's thorough empirical demonstration across models, tasks, and prompting paradigms—positions CFG as a practical tool that practitioners can adopt today, while opening the door for future inference-time compute strategies that push this philosophy further.

3. Technical Approach

3.1 Reader Orientation

This paper presents a method for applying Classifier-Free Guidance (CFG) to autoregressive language models—an inference-time technique that modifies the next-token probability distribution to increase adherence to a provided prompt, without requiring any model retraining, fine-tuning, or auxiliary classifiers. The core problem it solves is that language models, especially smaller ones, tend to drift away from their initial prompt during generation, losing coherence, hallucinating, or failing to follow instructions; the solution "shape" is a simple logit-space interpolation between a conditional distribution (conditioned on the full prompt) and an unconditional distribution (where the prompt is removed from context), parameterized by a single guidance strength hyperparameter $\gamma$ that controls how much the prompt is emphasized.

3.2 Big-Picture Architecture (Diagram in Words)

The system involves five conceptual components, though only the first is a traditional model—the rest are inference-time procedures:

Base Autoregressive Language Model (e.g., LLaMA-7B, Pythia-12B, GPT-2) — the pretrained transformer that takes a sequence of tokens $w_1, \ldots, w_{t-1}$ as input and outputs a probability distribution $P_\theta(w_t | w_1, \ldots, w_{t-1})$ over the next token. This model is used exactly as-is, with no weights modified.
Conditional Forward Pass — at each generation step $t$ , the model processes the full context including the prompt $c$ and all previously generated tokens, producing a conditional logit vector $\ell_{\text{cond}} = \log P_\theta(w_t | w_{<t}, c)$ .
Unconditional Forward Pass — at the same generation step, the model processes only the generated tokens (the prompt prefix $c$ is dropped from the context window, truncated to start from the last token of the prompt), producing an unconditional logit vector $\ell_{\text{uncond}} = \log P_\theta(w_t | w_{<t})$ . This requires a second forward pass through the network.
Logit-Space Interpolation — the two logit vectors are combined using the CFG formula: $\ell_{\text{CFG}} = \ell_{\text{uncond}} + \gamma(\ell_{\text{cond}} - \ell_{\text{uncond}})$ , where $\gamma \geq 1$ is the guidance strength. At $\gamma = 1$ , this reduces to standard conditional generation; at $\gamma > 1$ , it overemphasizes the prompt's influence by extrapolating away from the unconditional distribution.
Sampling Procedure — the interpolated logits $\ell_{\text{CFG}}$ are passed through softmax to produce the final sampling distribution, from which the next token is drawn (potentially with temperature, top-p, or other sampling parameters on top).

Information flow: The prompt $c$ is concatenated with the generation prefix → at each step, two forward passes run (one with prompt, one without) → their logits are interpolated → a token is sampled → the token is appended to the generation → repeat until termination. The only hyperparameter introduced is $\gamma$ , which governs the strength of the guidance.

3.3 Roadmap for the Deep Dive

First, the mathematical derivation of CFG in autoregressive language models (Section 2.2 of the paper), since this defines the core operation and connects it to its text-to-image origins. Understanding why the derivation works without conditioning dropout requires explanation.
Second, the practical implementation details: how the unconditional distribution is obtained, what happens at the boundary between prompt and generation, and what hyperparameters are involved. This is where theory meets engineering.
Third, the negative prompting extension, which generalizes CFG to a three-way interpolation that allows "subtracting" undesired behaviors, since this is the mechanism behind the chatbot experiments in Section 3.4.
Fourth, the connection to the visualization and analysis framework in Section 5, since understanding what CFG computes makes its effects on entropy and prompt-engineering interpretable.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method adaptation paper whose core idea is that the CFG technique from diffusion models can be ported to autoregressive language models by recognizing that dropping the prompt prefix from context naturally yields the unconditional distribution—no conditioning dropout training is required.

The Autoregressive Generation Setting

Before introducing CFG, we must establish what standard autoregressive generation looks like. A language model is a function that maps a sequence of past tokens to a probability distribution over the next token:

$P_\theta(w) = \prod_{i=1}^{T} P_\theta(w_i | w_j < i)$

where $P_\theta$ is the model parameterized by $\theta$ , $w_i$ is the token at position $i$ , $w_j<i$ represents all tokens before position $i$ , and $T$ is the total sequence length.

What it computes: This is the standard autoregressive factorization of the joint probability of a sequence into the product of conditional probabilities—each token's probability given all preceding tokens. The model takes an input sequence and outputs a vector of logits (unnormalized scores) over the vocabulary; applying softmax converts these logits into a proper probability distribution.

Why this form: This factorization is what makes autoregressive generation possible—the model only needs to predict one token at a time, and the prediction conditions on everything that came before. This sequential structure is what allows CFG to be applied at every step independently, since each token's distribution depends on the prefix context.

In a prompted setting, part of the sequence is treated as the "prompt" $c$ (the context, instruction, or beginning text provided by the user), and the model generates a continuation. The prompt occupies positions $w_1, \ldots, w_p$ , and the generated continuation occupies positions $w_{p+1}, \ldots, w_T$ . Standard generation samples from $P_\theta(w_i | w_{j<i}, c)$ —the model conditions on both the prompt and the previously generated tokens at each step.

Deriving CFG for Autoregressive Language Models

The paper's central technical contribution is adapting the CFG formula from diffusion models to autoregressive language models. The derivation proceeds by analogy, then identifies why the adaptation works without the training modifications that diffusion models require.

Step 1: The Classifier Guidance Starting Point

In Classifier Guidance for diffusion models [28], the goal is to sample from a distribution that favors images matching a text description $c$ . Using a separately trained classifier $P_\phi(c|x)$ , the sampling distribution is reweighted as:

$\hat{P}(x|c) \propto P_\theta(x) \cdot P_\phi(c|x)^\gamma$

where $\hat{P}$ is the reweighted distribution, $P_\theta(x)$ is the unconditional generative model, $P_\phi(c|x)$ is the classifier's prediction that image $x$ matches condition $c$ , and $\gamma$ is the guidance strength.

What it computes: This multiplies the unconditional likelihood of an image by the classifier likelihood raised to power $\gamma$ , then renormalizes. At $\gamma = 0$ , this gives the unconditional distribution; at $\gamma = 1$ , the standard conditional distribution; at $\gamma > 1$ , the classifier's opinion is overemphasized, producing images that more strongly exhibit the conditioning attribute.

Why this form: The exponent $\gamma$ provides a continuous knob between unconditional and "over-conditional" generation. The multiplicative combination is the natural way to combine independent sources of evidence about $x$ under a log-linear model—it corresponds to adding log-probabilities.

Step 2: Eliminating the Classifier via Bayes Rule

Ho and Salimans [37] observed that the classifier term can be rewritten using Bayes Rule:

$P_\phi(c|x) \propto \frac{P_\theta(x|c)}{P_\theta(x)}$

Substituting this into the Classifier Guidance formula eliminates the need for an external classifier, provided the generative model can produce both $P_\theta(x|c)$ and $P_\theta(x)$ . This gives the Classifier-Free Guidance formula:

$\hat{P}_\theta(x|c) \propto \frac{P_\theta(x|c)^\gamma}{P_\theta(x)^{\gamma-1}}$

What it computes: The conditional likelihood raised to power $\gamma$ divided by the unconditional likelihood raised to power $\gamma-1$ . When $\gamma = 1$ , the denominator disappears and we get standard conditional generation. When $\gamma > 1$ , the unconditional term in the denominator acts to suppress tokens that are likely even without the prompt.

Why this form: This eliminates the need to train and maintain a separate classifier model. The trade-off is that the generative model must now support both conditional and unconditional generation, which in diffusion models requires purpose-built conditioning dropout during training—the model must learn to generate both with and without the conditioning text.

Step 3: Translating to Logit Space for Diffusion Models

In diffusion models, the sampling procedure operates on predicted noise estimates. Taking the logarithm of the CFG formula and applying it at each denoising step yields the operational form:

$\log \hat{P}_\theta(\epsilon_t | x_{t+1}, c) = \log P_\theta(\epsilon_t | x_{t+1}) + \gamma\big(\log P_\theta(\epsilon_t | x_{t+1}, c) - \log P_\theta(\epsilon_t | x_{t+1})\big)$

where $\epsilon_t$ is the predicted noise at denoising step $t$ , $x_{t+1}$ is the current noisy image, and $c$ is the text conditioning.

What it computes: This performs vector arithmetic in the logit (log-probability) space: start at the unconditional log-probability, then take a step of size $\gamma$ in the direction of the conditional log-probability minus the unconditional log-probability. At $\gamma = 1$ , this gives the conditional log-probability. At $\gamma > 1$ , it extrapolates beyond the conditional, moving further away from the unconditional in the direction of $c$ .

Why this form: Rewritten as $\log P_\theta(\epsilon_t | x_{t+1}) + \gamma(\log P_\theta(\epsilon_t | x_{t+1}, c) - \log P_\theta(\epsilon_t | x_{t+1}))$ , this reveals the operation as "unconditional + $\gamma \times$ (conditional direction)." The geometric interpretation as vector arithmetic in a semantic space is what enables negative prompting and other extensions.

Step 4: The Autoregressive Adaptation

The paper's key insight is that autoregressive language models naturally handle both conditional and unconditional generation without any additional training. The derivation proceeds by unrolling the autoregressive factorization:

$\hat{P}_\theta(w|c) \propto \frac{P_\theta(w|c)^\gamma}{P_\theta(w)^{\gamma-1}} = \frac{\prod_{i=1}^T P_\theta(w_i | w_{j<i}, c)^\gamma}{\prod_{i=1}^T P_\theta(w_i | w_{j<i})^{\gamma-1}}$

The paper states this can be factorized to apply CFG at each token generation step independently:

$\hat{P}_\theta(w|c) \propto \prod_{i=1}^T \hat{P}_\theta(w_i | w_{j<i}, c) \propto \prod_{i=1}^T \frac{P_\theta(w_i | w_{j<i}, c)^\gamma}{P_\theta(w_i | w_{j<i})^{\gamma-1}}$

What it computes: Rather than sampling from the joint distribution of the entire sequence and then applying CFG (which would be intractable), this applies the CFG reweighting token-by-token during autoregressive generation. At each step $i$ , the next-token distribution is reweighted using the conditional probability (given prompt and previous tokens) and the unconditional probability (given only previous tokens).

Why this form: This token-by-token application is computationally feasible because each step already requires a forward pass. The alternative—generating a complete sequence and then reweighting—would require computing probabilities for all possible sequences, which is exponentially large. The factorization assumes that applying CFG locally at each step approximates applying it globally to the full sequence, which the paper implicitly validates through empirical results.

Step 5: The Operational Logit-Space Formula

Taking logarithms to work directly with logits (the pre-softmax values that language models compute internally):

$\log \hat{P}_\theta(w_i | w_{j<i}, c) = \log P_\theta(w_i | w_{j<i}) + \gamma\big(\log P_\theta(w_i | w_{j<i}, c) - \log P_\theta(w_i | w_{j<i})\big)$

This is the formula that is actually implemented. The paper states: "using the logits avoids network editing and is architecture agnostic."

What it computes: For each token position $i$ , compute two logit vectors—one from a forward pass that includes the prompt $c$ in context, and one from a forward pass where $c$ is dropped. Then linearly interpolate: the unconditional logits plus $\gamma$ times the difference between conditional and unconditional logits. The result is a new logit vector that, after softmax, gives the CFG-adjusted probability distribution.

Why this form: Operating in logit space has several practical advantages: (1) logits are the natural output of the transformer's final linear layer, so no additional computation is needed to obtain them; (2) the linear interpolation in logit space corresponds exactly to the desired multiplicative reweighting after softmax; (3) it avoids numerical issues with very small probabilities by working in log-space; (4) it allows standard sampling techniques (temperature, top-p, top-k) to be applied to the CFG-adjusted logits as they would be to any logit vector.

Obtaining the Unconditional Distribution

The critical practical question is: how does one obtain $P_\theta(w_i | w_{j<i})$ —the unconditional distribution over the next token without the prompt? The paper's answer is simple and is the reason CFG works without conditioning dropout:

"language models handle both $P_\theta(w|c)$ and $P_\theta(w)$ naturally due to being trained on finite context windows. Being able to drop the prefix $c$ is a natural feature."

The implementation, as described in Section 3.1, starts the unconditional prompt "at the last token of the initial prompt." This means: to compute the unconditional logits at step $i$ , truncate the context window to remove the original prompt $c$ and feed only the generation prefix (the tokens that have been generated so far) into the model. The model processes a shorter sequence and outputs logits that represent what it would generate if the prompt had never been provided.

Why this works: Autoregressive language models are trained to predict the next token given any prefix, regardless of length. During training, the model sees sequences of varying lengths and learns to condition on whatever context is available. When the prompt is dropped from context, the model simply conditions on the remaining tokens—this is exactly the unconditional distribution $P_\theta(w_i | w_{j<i})$ , since the prompt is no longer part of the conditioning. The model does not need to have been explicitly trained to generate without prompts; it was trained to generate given any prefix, and an empty or shorter prefix is just another possible context.

A subtle boundary detail: The unconditional distribution $P_\theta(w_i | w_{j<i})$ conditions on previously generated tokens but not on the prompt. At the very first generation step ( $i = p+1$ ), there are no generated tokens yet, so the unconditional context is effectively empty—the model would predict based on no prior information. As generation proceeds and tokens accumulate, the unconditional distribution conditions on the generated prefix, which may cause it to gradually diverge from the conditional distribution in content, not just in prompt-relevance. This is the intended behavior: the unconditional distribution represents "what the model would say without the prompt's guidance," and CFG subtracts this baseline to emphasize the prompt-specific contribution.

Computational cost: Each generation step now requires two forward passes—one with the full context (prompt + generation prefix) and one with the truncated context (generation prefix only). This doubles the inference FLOPs per token. The paper explicitly addresses this cost in Section 4, comparing against running a model twice as large without CFG (which would have similar per-token FLOPs). The key tradeoff is that the model with CFG requires half the parameter storage and half the training compute, while achieving comparable performance.

The Guidance Strength Hyperparameter $\gamma$

The guidance strength $\gamma$ is the only new hyperparameter introduced by CFG. Its behavior is characterized by:

$\gamma = 1$ : The formula reduces to standard conditional generation. The unconditional term $(1-1) = 0$ contribution disappears, and $\hat{P}_\theta(w_i | w_{j<i}, c) = P_\theta(w_i | w_{j<i}, c)$ .
$\gamma > 1$ : The prompt's influence is overemphasized. Tokens that are more likely with the prompt than without are boosted; tokens that are equally likely with or without the prompt are unchanged; tokens that are less likely with the prompt than without are suppressed. The magnitude of the effect scales with $(\gamma - 1)$ times the log-probability difference.
$\gamma < 1$ (not explored in the paper): Would move toward the unconditional distribution, deemphasizing the prompt. This direction is not experimentally studied.
$\gamma = 0$ : Would give pure unconditional generation (the prompt is completely ignored).

The paper sweeps different $\gamma$ values depending on the task: $\gamma = 1.5$ is a common default for basic prompting (Section 3.1), low values ( $\gamma \leq 1.5$ ) work best for code generation (Section 3.3.2), and higher values ( $\gamma = 3$ – $5$ ) are used for chatbot negative prompting (Section 3.4). The optimal $\gamma$ is task-dependent and must be tuned; the paper acknowledges this as a limitation: "CFG requires tweaking and exploration: $\gamma$ values that might work in one context (i.e. long-form generation) might be poorly suited for another context."

Why $\gamma$ must be tuned per task: The CFG formula amplifies the difference between conditional and unconditional log-probabilities. The magnitude of this difference depends on how strongly the prompt constrains the generation. For short-answer tasks where the correct answer is one of a few tokens, the prompt strongly determines what should come next, so the difference is large and a modest $\gamma$ suffices. For long-form generation, the prompt has a weaker per-token influence (many tokens are plausible regardless), so a higher $\gamma$ may be needed to maintain adherence without the difference vector being washed out by sampling noise. For code generation, the structure of the programming language imposes strong syntactic constraints regardless of the prompt, so high $\gamma$ might fight against syntactic necessity and produce malformed code.

Negative Prompting: Extending CFG to Three-Way Interpolation

The paper generalizes the CFG formula to support "negative prompting"—specifying what the generation should not look like—by replacing the unconditional baseline with a "negative conditioning" distribution:

$\log \hat{P}_\theta(w_i | w_{j<i}, c, \bar{c}) = \log P_\theta(w_i | w_{j<i}, \bar{c}) + \gamma\big(\log P_\theta(w_i | w_{j<i}, c) - \log P_\theta(w_i | w_{j<i}, \bar{c})\big)$

where $c$ is the desired (positive) conditioning and $\bar{c}$ is the negative conditioning—the prompt specifying what should be avoided.

What it computes: Instead of starting from the unconditional distribution, start from the distribution conditioned on the negative prompt $\bar{c}$ , then take a step of size $\gamma$ away from this negative conditioning in the direction of the positive conditioning $c$ . The result is a distribution that strongly favors what $c$ encourages and strongly disfavors what $\bar{c}$ encourages.

Why this form: This is a direct generalization of the two-term CFG formula. When $\bar{c} = \emptyset$ (empty conditioning), it reduces to the standard CFG formula. The geometric interpretation is compelling: the unconditional point in Equation 7 becomes an arbitrary point in semantic space that we want to move away from. This enables fine-grained control—rather than just saying "generate something related to my prompt," we can say "generate something like my prompt but unlike this other prompt."

The paper applies this in the chatbot setting (Section 3.4) with a concrete use case. The positive prompt $c$ is a modified system prompt (e.g., "The prompt below is a question to answer... decide which and write a sad response"), and the negative prompt $\bar{c}$ is the default system prompt (e.g., "The prompt below is a question to answer... decide which and write an appropriate response"). By moving away from the default system-prompt distribution and toward the modified one, CFG emphasizes the difference between the modified and default instructions. The paper explains:

"This approach not only makes the sampling more prompt-aware in general, but directly emphasizes the difference between our system-prompt and the model's default system-prompt."

This is powerful because the model, having been trained primarily on the default system prompt, strongly defaults to its standard behavior. Simply providing the modified prompt as conditioning may not overcome this default bias; the negative prompting formulation explicitly pushes generation away from the default, making the modification more effective.

Connection to Analysis: Entropy and Token Visualization

The paper's analysis in Section 5 provides insight into what CFG is doing to the logit distributions, which helps explain its empirical effectiveness:

Entropy reduction (Section 5.1). By comparing the entropy of logit distributions across 32,902 datapoints from the P3 dataset using Falcon-7B, the paper finds that "CFG entropy distribution is significantly lower across generation time-steps [than] vanilla prompting, with a mean of 4.7 vs. 5.4." Lower entropy means the model is more "confident" about which tokens to choose—the probability mass is concentrated on fewer tokens. This is consistent with CFG's mechanism: by amplifying the prompt's signal, tokens that are clearly prompt-relevant receive higher probability, while tokens that are ambiguous receive lower probability. This increased confidence reduces the sampling variance that can cause meandering.

Token reordering visualization (Section 5.3). The paper proposes visualizing CFG's effect by ranking vocabulary items according to the difference $\log P(w_t | w_{<t}) - \log P(w_T | \hat{w})$ at each timestep. The top-ranked tokens are those most encouraged by CFG; the bottom-ranked are those most discouraged. The example in Table 3, for the prompt "The dragon flew over Paris, France," shows CFG encouraging tokens about "dragons," "Paris," "cathedral," and "flying," while discouraging tokens about other locations ("Queensland," "Newfoundland"), dates ("1913"), or unrelated topics ("hostages," "voyages"). This visualization confirms that CFG does what it mathematically promises: upweight prompt-relevant vocabulary and downweight prompt-irrelevant vocabulary.

The paper suggests this visualization as "an integral part of effective prompt engineering," since it shows, token by token, what a prompt is emphasizing and what it's suppressing. If a prompt engineer sees that important concepts are being downweighted, they can adjust the prompt formulation.

Practical Implementation Considerations

Beyond the mathematical derivation, several implementation details matter:

Token position for prompt boundary. The unconditional context starts "at the last token of the initial prompt" (Section 3.1). This means: if the prompt is tokens $w_1, \ldots, w_p$ , then at generation step $p+1$ (the first generated token), the unconditional forward pass receives an empty context (or a BOS token). At step $p+2$ , the unconditional pass receives the first generated token; at step $p+3$ , the first two generated tokens; and so forth. The unconditional context grows with generation, always containing the generation prefix but never the prompt itself.

Interaction with sampling parameters. CFG modifies the logits before any sampling heuristics (temperature, top-p, top-k) are applied. The standard pipeline is: compute conditional logits → compute unconditional logits → apply CFG interpolation → optionally apply temperature scaling to the result → optionally apply top-p/top-k filtering → sample from the resulting distribution. This means CFG and sampling heuristics are orthogonal and can be combined freely.

Hardware requirements. CFG requires running two forward passes per generated token (one with full context, one with truncated context). For a model of size $N$ parameters generating $T$ tokens, this uses $2 \times 2 \times N \times T$ FLOPs (roughly double the standard $2NT$ ). If the user has sufficient VRAM to run the model once, they can run CFG by performing two sequential forward passes—doubling latency but not exceeding VRAM. The paper emphasizes this VRAM advantage: "a model using CFG can generally perform just as well as a model twice as large" (Section 4), meaning users can achieve the performance of a 14B model while only needing VRAM for a 7B model, at the cost of doubled inference time.

Applicability to any autoregressive model. The paper tests CFG on GPT-2 (Radford et al., 2019), Pythia (Biderman et al., 2023), LLaMA (Touvron et al., 2023), WizardLM (Xu et al., 2023), Guanaco (Dettmers et al., 2023), CodeGen (Nijkamp et al., 2023), GPT-J (Wang and Komatsuzaki, 2021), Falcon (Almazrouei et al., 2023), RedPajama, Bloom (Scao et al., 2022), mT0 (Muennighoff et al., 2022), and GPT4All (Anand et al., 2023). The technique works across all of them because it only requires access to logits—it never touches model weights, layers, or architecture-specific internals. This architecture agnosticism is a direct consequence of the logit-space operation.

Limitation: doubled FLOPs. The paper is transparent that CFG "almost doubles the computation during inference." For latency-sensitive applications where generating tokens serially is already the bottleneck, doubling the per-token cost may be prohibitive. However, the paper argues this is still valuable because (1) memory constraints are often more binding than compute constraints, and (2) the training cost savings (training a model half the size) can be substantial enough to justify slower inference for many use cases.

4. Key Insights and Innovations

Innovation 1: CFG Works on Autoregressive LMs Without Any Training, Contradicting the Assumption That Conditioning Dropout Is Necessary

The dominant assumption inherited from the text-to-image literature was that Classifier-Free Guidance requires purpose-built training with conditioning dropout. Ho and Salimans [37] established this requirement explicitly: diffusion models must be trained to handle both conditional and unconditional generation by randomly dropping conditioning information during training, so that at inference time the model can serve as its own implicit classifier. The paper identifies that this assumption does not transfer to autoregressive language models—and in fact, language models already satisfy the requirements for CFG without any modification whatsoever.

The conceptual move: The paper recognizes that autoregressive language models trained on finite context windows naturally handle variable-length prefixes. Dropping the prompt from context is not a special operation requiring conditioning dropout; it is simply providing a shorter context, which the model handles as a normal case. The authors state:

"Being able to drop the prefix $c$ is a natural feature."

This observation is simple in retrospect but represents a genuine reframing. Prior work on controlled text generation—PPLM [24], FUDGE [84], GeDi [43], NA-DO [50]—all assumed that steering generation required either training auxiliary classifiers or modifying the base model. Contrastive Decoding [45] required access to two different models (a strong "expert" and a weak "anti-expert"). The paper shows that a single forward pass with the prompt and a single forward pass without the prompt—on the same model—suffices to implement full CFG. Zero training, zero auxiliary models, zero architecture changes.

Why this matters beyond convenience: This finding transforms CFG from a technique that requires buy-in at training time (like conditioning dropout in diffusion models) to a technique that can be applied to any deployed model retroactively. A practitioner with access to a pretrained LLaMA or GPT-2 checkpoint can apply CFG immediately. This is not an incremental improvement in ease-of-use—it is a categorical shift in who can use the technique and when. It means that the entire ecosystem of existing pretrained language models becomes CFG-compatible without any retraining or fine-tuning pipeline.

Evidence: The paper demonstrates CFG across Pythia, GPT-2, LLaMA, Falcon, CodeGen, GPT-J, Bloom, mT0, RedPajama, WizardLM, Guanaco, and GPT4All—models spanning multiple architectures, training paradigms, and scales—with consistent improvements (Table 2, Figures 8–10). No model required modification. The mechanism is architecture-agnostic because it operates purely in logit space, which every autoregressive language model exposes.

A limitation exposed by the finding: The flip side of this insight is that CFG's effectiveness depends entirely on how distinguishable the conditional and unconditional distributions are for a given prompt. If a prompt barely shifts the model's predictions (e.g., the model is already highly entropic on the task, or the prompt is so out-of-distribution that the model ignores it), the difference vector $(\ell_{\text{cond}} - \ell_{\text{uncond}})$ carries little signal, and CFG cannot help. This explains the paper's finding that CFG is less effective on the hardest tasks (ARC-challenge, WinoGrande in Table 2) and that $\gamma$ must be tuned per-task—the guidance can only amplify what the model already weakly knows.

Innovation 2: Negative Prompting via Three-Way CFG Enables Fine-Grained Behavioral Control Without Retraining

The generalization of CFG to support negative prompting (Equation 5 in Section 2.2, applied in Section 3.4) represents a conceptual advance beyond simply adapting two-way CFG from vision to language. In text-to-image models, negative prompting is used to suppress visual artifacts ("bad hands, bad anatomy") or stylistic qualities ("low resolution"). The paper's application to language is fundamentally different: it uses negative prompting to emphasize the difference between two prompts, making it possible to override deeply ingrained model defaults.

The conceptual move: Instead of the standard CFG formulation that moves from unconditional toward conditional, the paper replaces the unconditional baseline with a "negative conditioning" $\bar{c}$ that represents what should be suppressed. The generation moves away from $\bar{c}$ and toward $c$ :

$\log \hat{P}(w_i | w_{j<i}, c, \bar{c}) = \log P(w_i | w_{j<i}, \bar{c}) + \gamma(\log P(w_i | w_{j<i}, c) - \log P(w_i | w_{j<i}, \bar{c}))$

When $\bar{c}$ is set to the model's default system prompt and $c$ is a modified behavioral instruction, this formulation directly targets the model's tendency to revert to its default behavior. The paper explains:

"This approach not only makes the sampling more prompt-aware in general, but directly emphasizes the difference between our system-prompt and the model's default system-prompt."

Why this is distinct from prior work: Prior controlled generation methods (PPLM, FUDGE, GeDi) required training a separate classifier or discriminator for each control dimension. If you wanted to make the model "more enthusiastic" or "more formal," you needed a labeled dataset and a trained model for that specific attribute. The negative prompting formulation, by contrast, allows ad-hoc behavioral specifications through natural language differences between $c$ and $\bar{c}$ —no attribute-specific training required. The same CFG mechanism that boosts prompt-adherence in basic QA can, by changing the choice of $\bar{c}$ , implement style transfer, persona modification, or safety constraint enforcement.

Evidence: The human evaluation in Section 3.4 (Figure 5) demonstrates this concretely. Participants were shown generations from GPT4All with vanilla sampling vs. CFG with negative prompting ( $\bar{c}$ = default system prompt, $c$ = modified system prompt like "write a sad response" or "write a seductive response"). At $\gamma = 3$ , 75% of evaluators preferred the CFG output for system-prompt following, while user-prompt relevance remained undegraded (52% preference, not significantly different from random). This shows that negative prompting can dramatically improve adherence to behavioral instructions without sacrificing responsiveness to the user's actual query—a separation of concerns that training-based approaches often fail to achieve.

The deeper implication: This finding suggests that system-level behavioral control can be implemented as a runtime parameter rather than a training property. A single deployed model could switch personas, styles, or safety constraints by changing $\bar{c}$ and $c$ at inference time, without loading different model weights or running additional fine-tuning. This has practical implications for multi-tenant deployments where different users or applications need different behavioral configurations from the same base model.

Innovation 3: CFG Provides Performance Equivalent to ~2× Model Scale, Establishing Inference-Time Compute as a Substitute for Training-Time Compute

While the paper's headline result—CFG improves benchmark scores—is expected given the technique's mechanism, the magnitude and systematicity of the improvement relative to model scaling is a distinctive empirical finding with implications for how the field thinks about the pretraining-inference tradeoff.

The conceptual move: Section 4 directly asks the question a skeptical practitioner would pose: since CFG doubles inference FLOPs, is it not better to simply run a model twice as large without CFG? The paper answers this with a FLOPs-matched comparison across 9 benchmarks using models from the GPT-2, Pythia, and LLaMA families. The finding is striking:

"across 5 out of 9 tasks, there is a statistically insignificant difference between using CFG and using vanilla prompting with a model of twice the size at $p = .01$ , according to ANCOVA regression analysis. Of the significantly different tasks, 2 favor CFG and 2 favor vanilla."

This means that, on average, a model with CFG performs comparably to a model with twice the parameter count at the same inference FLOP budget. The equivalence is not universal—LAMBADA and SciQ strongly favor CFG, while WinoGrande strongly favors the larger vanilla model—but the central tendency is that CFG recovers roughly the performance gain of doubling model size.

Why this matters beyond performance: This is not merely a convenient result; it reshapes the economic calculus of model deployment. Training a model with $N$ parameters costs roughly $6ND$ FLOPs (where $D$ is training tokens), and storing it requires roughly $2N$ bytes of memory (at 16-bit precision). Doubling parameters doubles both costs. If CFG can achieve the same performance gain at inference time with zero additional training cost and zero additional memory, it means:

Training budget savings: The resources that would have been spent training a 14B model can instead train a 7B model, with the difference in training cost being pure savings.
VRAM savings: A 7B model with CFG fits in half the GPU memory of a 14B model without CFG, enabling deployment on smaller hardware. The paper explicitly calls out "limited VRAM usage" as a key constraint that CFG alleviates.
Democratization: Researchers and practitioners with limited compute can access the effective performance of larger models without the capital expenditure of training or hosting them.

A critical nuance the paper does not fully explore: The doubled inference FLOPs translate to doubled latency if running serially (two sequential forward passes per token). The paper emphasizes FLOPs and VRAM but does not discuss wall-clock time implications. For interactive applications where latency matters more than throughput, the tradeoff may be less favorable. However, for batch processing or offline generation where throughput dominates, the FLOPs-matched comparison is the right metric.

The connection to scaling laws: This finding can be interpreted as an early empirical contribution to what would become the study of inference-time scaling laws—the analogue of pretraining scaling laws (Hoffmann et al., 2022) applied to test-time compute. Just as Chinchilla scaling laws established that pretraining FLOPs should be allocated between model size and data quantity in a specific ratio for optimal performance, CFG demonstrates that inference FLOPs can be productively "invested" in guidance to improve output quality from a fixed model. The tradeoff curve—how many effective parameters does each unit of inference compute buy?—is not fully characterized in this paper, but the §2× equivalence finding is a concrete data point on that curve.

Evidence: Figure 11 in Appendix C.2 shows the regression analysis underlying this claim, with ANCOVA p-values in Table 4. The LAMBADA result is the most dramatic: LLaMA-7B with CFG ( $\gamma = 1.5$ ) achieves 81.3% accuracy, outperforming PaLM-540B's 77.9%—a model roughly 77× larger. This specific comparison overstates CFG's benefit (PaLM uses greedy decoding and is not a LLaMA-family model, so other architectural and training differences confound), but the within-model-family FLOPs-matched analysis is the rigorous test of the scaling claim.

Innovation 4: CFG's Effects Are Quantitatively and Qualitatively Distinct from Instruction-Tuning, Opening a Complementary Axis for Model Improvement

A natural hypothesis when encountering CFG is that it might simply be approximating instruction-tuning at inference time—both techniques aim to improve prompt-following, and both operate on the model's output distribution. The paper's analysis in Section 5 directly tests this hypothesis and largely rejects it, establishing that CFG and instruction-tuning are complementary rather than redundant.

The conceptual move: The paper conducts a large-scale comparison (32,902 datapoints from the P3 dataset using Falcon-7B-base vs. Falcon-7B-Instruct) examining token-level logit distributions. The findings reveal a nuanced picture:

Entropy reduction is similar in magnitude: CFG reduces mean logit entropy from 5.4 to 4.7 (Figure 6a), comparable to instruction-tuning's effect. Both techniques make the model more confident in its token choices.
But the specific tokens being upweighted are largely different: Figure 6b shows that CFG shares roughly 50% of the top-p=0.9 vocabulary tokens with the vanilla prompted model, while the overlap between CFG and instruction-tuning is lower—they are not converging on the same token rankings despite both reducing entropy.
Similarity between CFG and instruction-tuning is context-dependent: Table 8 reveals that on specific dataset types (e.g., Winogrande, SuperGLUE tasks), CFG and instruction-tuning show the highest agreement, while on others (e.g., RottenTomatoes sentiment, COPA), they diverge sharply. Longer, more specific prompts correlate with higher CFG-instruct agreement (Spearman $r_s = 0.05$ , while modest, is statistically significant).

Why this matters: If CFG were simply approximating instruction-tuning, it would be a cheaper but redundant alternative—useful only when instruction-tuning is unavailable. The finding that the two methods differ in how they reshape the vocabulary distribution means they are complementary axes of improvement. A model that is instruction-tuned (to broadly follow task specifications) can still benefit from CFG (to specifically adhere to this particular prompt), and vice versa. The paper does not experimentally combine the two (e.g., applying CFG to an instruction-tuned model) but the analysis strongly implies that such combinations would yield further gains.

The diagnostic contribution: The visualization technique proposed in Section 5.3—ranking vocabulary items by $\log P(w_t | w_{<t}) - \log P(w_t | \hat{w})$ at each timestep—provides a general-purpose tool for understanding what any prompt is actually doing to a model's predictions. The example in Table 3 (for "The dragon flew over Paris, France") shows tokens about "dragons," "Paris," and "cathedral" being upweighted while "Queensland," "1913," and "hostages" are downweighted. This is not just a visualization of CFG; it is a visualization of the prompt's effect on the model's semantic space, which persists even at $\gamma = 1$ (standard generation). The paper notes this can serve as "an integral part of effective prompt engineering"—if a prompt engineer observes that important concepts are not being upweighted, they can revise the prompt formulation before deploying it.

A negative result with positive implications: The paper's finding that CFG and instruction-tuning are not equivalent is, in a sense, a negative result—the simple hypothesis is rejected. But it opens the door to more sophisticated combinations: applying CFG on top of instruction-tuned models to get the benefits of both, or using CFG's token-level visualization to debug why an instruction-tuned model is failing on a particular prompt. The paper's analysis section transforms CFG from a black-box performance booster into an interpretable tool for understanding and improving prompt effectiveness.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates across multiple datasets rather than a single benchmark. For basic prompting evaluations (Section 3.1), a suite of zero-shot benchmarks is drawn from EleutherAI's Language Model Evaluation Harness [33], including: LAMBADA [58] (sentence completion requiring broad discourse context), TriviaQA [39] (closed-book QA), SciQ [5] (scientific QA), HellaSwag [85] (commonsense reasoning), PIQA [12] (physical commonsense reasoning), BoolQ [18] (yes/no QA), ARC-easy and ARC-challenge [19] (science reasoning), and WinoGrande [69] (pronoun resolution). For chain-of-thought experiments (Section 3.2), GSM8K [21] and AQuA [48] are used—both arithmetic reasoning benchmarks with multi-step solution requirements. For code generation (Section 3.3), the HumanEval benchmark [16] is used, consisting of 164 Python programming tasks where the prompt is a function signature and docstring, and correctness is evaluated against unit tests. For machine translation (Appendix D.1), WMT14 fr-en [13] is evaluated using BLEU score. For analysis experiments (Section 5), a dataset of 32,902 datapoints is sampled from the P3 dataset [70], with replication on the Open-Assistant Dataset [42]. For chatbot experiments (Section 3.4), a custom evaluation set of 25 system prompts and 46 user prompts is constructed, producing 1,740 random combinations sampled for human evaluation.
Base model(s). The paper evaluates across an unusually broad range of model families and scales. For basic prompting: GPT-2 family (small, medium, large, XL; Radford et al., 2019 [62]), Pythia family (160M, 410M, 1B, 1.4B, 2.8B, 6.9B, 12B; Biderman et al., 2023 [11]), and LLaMA family (7B, 13B, 30B, 65B; Touvron et al., 2023 [78]). For chain-of-thought: WizardLM-30B [83] and Guanaco-65B [25]. For code generation: GPT-J [79] (for exploratory experiments), and CodeGen-mono family (350M, 2B, 6B; Nijkamp et al., 2023 [54]). For machine translation: Bloom-3B [72], RedPajama-Incite-Base-3B, and mT0 [52]. For chatbot: GPT4All-J v1.3-jazzy [3]. For analysis: Falcon-7B-base and Falcon-7B-Instruct [2], with replication on RedPajama-3B. The breadth of model families is deliberate—the paper aims to demonstrate that CFG is architecture-agnostic, and the authors state they believe PaLM 2-S* (not used here) is "representative of the capabilities of many contemporary LLMs." The inclusion of models from 160M to 65B parameters enables the FLOPs-matched scaling analysis in Section 4.
Metrics. For basic prompting benchmarks, the primary metric is accuracy (% correct), with task-specific grading: substring matching rather than exact match is used for TriviaQA based on manual analysis showing exact matching disqualified valid answers like quoted responses. For chain-of-thought, the metrics are accuracy on the final answer and percentage of chains producing a valid (parsable) answer structure. For HumanEval, pass@k (k = 1, 10, 100) is used following the definition from Chen et al. [16]: k samples are generated per problem, a problem is considered solved if any sample passes unit tests, and total fraction solved is reported. For machine translation, BLEU score is used. For chatbot evaluation, human preference ratings are collected for two axes: system-prompt following and user-prompt relevance, with participants shown blinded pairs (CFG vs. vanilla) and asked which better follows each aspect. For analysis experiments, vocabulary overlap in top-p=0.9, perplexity correlations (Spearman $r_s$ ), and token-level logit entropy are measured.
Baselines. The primary baseline across all experiments is vanilla sampling: standard autoregressive generation from the same model with $\gamma = 1$ (equivalent to disabling CFG, since the formula reduces to standard conditional generation). For chain-of-thought experiments, the baseline is the standard few-shot CoT prompting approach of Wang et al. [80]. No separately trained verifier models or majority voting baselines are used for the primary tasks (unlike the companion paper on test-time compute scaling, which uses process reward models). For the HumanEval code generation experiments, the baseline is standard sampling from CodeGen models without CFG at the same temperature. The FLOPs-matched comparison in Section 4 compares CFG-enabled models against vanilla models of approximately twice the parameter count, using the same benchmark suite, with ANCOVA regression analysis on log-transformed accuracy vs. FLOPs variables.
Generation budget / compute accounting. The paper measures inference cost in FLOPs per token (floating point operations), computed based on model parameter count and sequence length. A standard autoregressive generation step requires approximately $2N$ FLOPs for a model with $N$ parameters (one forward pass). CFG requires two forward passes—one conditional, one unconditional—so it uses approximately $4N$ FLOPs per generated token, roughly doubling the per-token inference cost. This is the basis for the FLOPs-matched comparison: a model with $N$ parameters using CFG is compared against a model with $2N$ parameters using vanilla sampling, since both consume approximately $4N$ FLOPs per token. The paper explicitly notes that "CFG almost doubles the computation during inference." For the HumanEval pass@k experiments, $k$ samples are generated per problem, and the total generation budget scales accordingly. No latency or wall-clock time measurements are reported; the analysis focuses exclusively on FLOPs as the compute metric.
Cross-validation / statistical protocol. For the FLOPs-matched comparison in Section 4, ANCOVA (analysis of covariance) regression is used to test whether the accuracy-vs-FLOPs regression lines for CFG-enabled models differ significantly from those of vanilla models at different scales, with significance threshold set at $p = 0.01$ following Rutherford [67]. For the chatbot human evaluation (Section 3.4), 611 votes from 71 unique voters are collected on 1,740 random prompt combinations; statistical significance of preference ratios is assessed against a binomial baseline of 50% (no preference). For the code generation exploratory experiment with GPT-J (Appendix D.2), a p-value of 0.01 is reported for the improvement in programming language consistency. No cross-validation over dataset splits is performed, as the paper evaluates on standard benchmark test sets directly rather than selecting hyperparameters from validation folds (in contrast to the test-time compute paper, which requires cross-validation for its compute-optimal policy selection).

Main Quantitative Results

Basic Prompting: Zero-Shot Benchmarks

The headline finding for basic prompting (Section 3.1, Table 2 in the paper—actually presented as a split Figure 2a and 2b in the manuscript) is that CFG with $\gamma = 1.5$ improves accuracy over vanilla sampling ( $\gamma = 1$ ) across most benchmarks and model families, with improvements that are "nontrivial and consistent" except for ARC-challenge and WinoGrande.

For the GPT-2 family (Figure 8), the pattern is clear: CFG at $\gamma = 1.5$ outperforms vanilla sampling at $\gamma = 1$ on nearly every task, with the size of the improvement varying by benchmark. The LAMBADA benchmark shows the most dramatic relative improvement: GPT2-small improves from 32.6% to 44.6% (a 12 percentage point absolute gain), GPT2-medium from 43.0% to 55.8% (+12.8 points), GPT2-large from 47.7% to 60.5% (+12.8 points), and GPT2-xl from 51.2% to 62.5% (+11.3 points). SciQ shows similarly large gains: GPT2-small from 64.4% to 70.8% (+6.4 points), with the effect persisting through all model sizes. BoolQ and HellaSwag show moderate improvements (1.5–8.3 percentage points depending on model size). PIQA and ARC-e show smaller but consistent gains (1–5 percentage points). WinoGrande actually degrades slightly with CFG across all GPT2 sizes (e.g., GPT2-xl drops from 58.3% to 55.6%), and ARC-challenge shows mixed results (GPT2-medium drops from 25.0% to 23.9%, while GPT2-xl improves from 28.5% to 30.0%).

For the Pythia family (Figure 9), the pattern replicates with similar magnitudes. The LAMBADA improvements are again the largest: Pythia-160M from 32.8% to 47.4% (+14.6 points), Pythia-1.4B from 61.6% to 72.7% (+11.1 points), Pythia-12B from 70.4% to 80.6% (+10.2 points). SciQ shows consistent 4–8 point improvements across all sizes. TriviaQA shows essentially no improvement or slight degradation (e.g., Pythia-12B drops from 33.4% to 32.1%), which is notable—CFG does not help and may slightly hurt on this closed-book QA task. ARC-challenge and WinoGrande remain weak spots, with small or negative effects across Pythia sizes.

For the LLaMA family (Figure 10), the results are consistent with the trends from GPT-2 and Pythia, but at higher absolute performance levels. The most notable result is LAMBADA with LLaMA-7B: vanilla achieves 73.6%, and CFG with $\gamma = 1.5$ achieves 81.3%, which the paper highlights as "outperforming the current SOTA (zero-shot) of PaLM-540B (77.9%)." This comparison is somewhat apples-to-oranges (different model architectures, different training data), but the within-family comparison is rigorous. LLaMA-13B on LAMBADA improves from 76.2% to 82.2% (+6.0 points), LLaMA-30B from 77.5% to 83.9% (+6.4 points), and LLaMA-65B from 79.1% to 84.0% (+4.9 points). The diminishing returns with scale suggest CFG provides proportionally more benefit to smaller models—consistent with the intuition that smaller models need more help with prompt-adherence. For SciQ, LLaMA-13B shows a dramatic jump from 91.1% to 95.1%, approaching ceiling performance. The weakest results are again WinoGrande (LLaMA-7B drops from 67.1% to 65.5%) and ARC-challenge (mixed, with some improvements at larger scales: LLaMA-13B from 47.8% to 54.2%).

Sweeping over $\gamma$ values. The figures in Appendix C.1 (Figures 8, 9, 10) show accuracy as a function of $\gamma$ (swept from 1.0 to 1.5 or beyond) for each model size and benchmark. The curves generally show monotonic improvement from $\gamma=1$ to $\gamma\approx 1.3-1.5$ , with some tasks showing a plateau or slight decline at higher $\gamma$ . The paper does not systematically sweep $\gamma$ values beyond 1.5 for basic prompting, so the optimal $\gamma$ may be higher for some tasks—this is an acknowledged limitation.

Deliberative Prompting: Chain-of-Thought

For chain-of-thought prompting (Section 3.2, evaluated on GSM8K and AQuA using WizardLM-30B and Guanaco-65B), the headline finding is that low CFG strengths ( $\gamma \leq 1.5$ ) both increase the percentage of chains yielding valid (parsable) answers and boost accuracy, while high $\gamma$ values continue to increase validity but degrade answer quality, creating a clear tradeoff.

The paper reports results through Figure 3 (for GSM8K with WizardLM-30B) and Figure 15 (for AQuA). The key patterns:

Invalid answer rate decreases with $\gamma$ . Without CFG, a non-trivial fraction of chain-of-thought generations fail to produce a parsable answer—the model diverges from the expected output structure before reaching a conclusion. CFG reduces this invalid percentage consistently. However, the paper does not report exact invalid-rate numbers in the main text; these are visible in the figures but not tabulated.
Accuracy peaks at moderate $\gamma$ . For GSM8K with WizardLM-30B, accuracy increases from the vanilla baseline at $\gamma = 1$ to a peak at $\gamma \approx 1.1-1.3$ , then declines at higher $\gamma$ values. The paper states: "for large values, although the model returns more valid results, the quality of the chains is also impacted, and overall the model performances degrade." The same pattern is observed on AQuA (Figure 15).
Qualitative examples demonstrate the mechanism. Table 14 shows a GSM8K example with Guanaco-65B where vanilla sampling produces a reasoning chain that "diverges and results in an incorrect answer which doesn't preserve the answer format." CFG at $\gamma = 1.5$ produces a correct answer with proper formatting. Table 15 shows a WizardLM-30B example where both vanilla and CFG sampling share the same chain prefix, but the vanilla chain diverges to a wrong answer while CFG maintains the correct reasoning.

The paper explicitly acknowledges it has "only scratched the surface of exploring CFG's interactions with CoT" and suggests variations where different parts of the prompt ( $w_p$ , $w_{\text{cot}}$ , etc.) receive different guidance weights—experiments that were not run but are flagged for future work.

Text-to-Text Prompts: Code Generation

For code generation (Section 3.3, evaluated on HumanEval [16] with CodeGen-mono models at temperatures 0.2, 0.6, and 0.8), the headline finding is that low CFG ( $\gamma \leq 1.5$ ) consistently improves pass@1 rates across model sizes and temperatures, while high CFG ( $\gamma > 1.5$ ) degrades performance, and improvements diminish at pass@10 and pass@100, consistent with CFG trading off diversity for prompt-adherence.

Main results (Table 2 in the paper, for temperature = 0.2). For CodeGen-350M-mono, pass@1 improves from 11.0% (vanilla) to 11.8% at $\gamma = 1.1$ and 11.4% at $\gamma = 1.25$ , then declines to 10.9% at $\gamma = 1.5$ and 8.6% at $\gamma = 2.0$ . For CodeGen-2B-mono, pass@1 improves from 19.5% to 20.4% at $\gamma = 1.1$ and 20.9% at $\gamma = 1.5$ , then drops to 16.5% at $\gamma = 2.0$ . For CodeGen-6B-mono, the pattern mirrors CodeGen-2B—the results appear duplicated in the paper's tables (Table 7 shows identical values to Table 6), which may indicate a typesetting error. The full results across temperatures are in Tables 5, 6, and 7 in Appendix C.3.

Temperature interaction (Figures 12–14). At higher temperatures, the relative benefit of CFG changes. The paper's figures show pass@k curves for different $\gamma$ values and temperatures. At temperature 0.6, the pass@1 gains from CFG are more pronounced for some model sizes (e.g., CodeGen-350M-mono improves from 8.9% to 10.0% at $\gamma = 1.1$ ; CodeGen-2B-mono from 15.9% to 18.3% at $\gamma = 1.5$ ). At temperature 0.8, the gains persist but are noisier. This temperature-dependence suggests CFG interacts with sampling stochasticity—at higher temperatures, the unconditional distribution is more entropic, so subtracting it sharpens the conditional signal more dramatically.

Pass@k analysis. The paper observes that "without CFG, many tasks exhibit small nonzero passing rates while having 0% rate with CFG. This explains the decreasing improvement of CFG in pass@k for large k, as larger k significantly boosts the passing rate of difficult tasks where the rates are low but nonzero." In other words, CFG helps on tasks where the model has some capability (pass@1 > 0), but on tasks where the model absolutely cannot solve the problem ( $\approx$ 0% pass@1), CFG eliminates the rare lucky samples that large- $k$ sampling would otherwise catch. This is a genuine tradeoff: CFG improves expected quality at the cost of eliminating low-probability creative solutions that occasionally succeed.

Task-by-task breakdown (Figure 4, and Appendix Figures 15a–15c). For CodeGen-350M-mono at temperature 0.2 and $\gamma = 1.25$ , the paper breaks down individual HumanEval tasks into those where CFG outperforms, ties, or underperforms vanilla sampling. The scatter plots (Figures 15a–15c) show that CFG outperforms on more tasks than it underperforms, but there are clearly tasks where CFG is harmful. The paper does not provide a detailed categorization of which task types benefit vs. suffer—this is left as qualitative observation.

Exploratory GPT-J experiments (Appendix D.2). In a separate exploratory experiment, GPT-J is prompted to generate code in specific programming languages (C, Java, Python). With vanilla sampling, the model generates code in the correct language 73% of the time; with CFG at $\gamma = 1.25$ , this increases to 86% ( $p = 0.01$ ). At higher $\gamma$ (1.5, 1.75), the accuracy drops back to 81% and 77% respectively (Table 12). A second experiment prompts CodeGen-350M-mono to generate an image-rendering function and evaluates syntax correctness, return type, return shape, and output quality. CFG at $\gamma = 2$ improves correct syntax from 419/1600 to 574/1600 (137% of baseline), correct return type from 289 to 546 (189%), correct shape from 284 to 538 (189%), and reduces L2 distance to reference from 0.111 to 0.090 (123% improvement relative to error reduction). These results support the HumanEval findings but use smaller-scale, less standardized evaluations.

Negative Prompting: Improving Assistants

For chatbot-style prompting with GPT4All (Section 3.4, using 25 system prompts × 46 user prompts = 1,740 combinations), the headline finding is that CFG with negative prompting (setting the default system prompt as $\bar{c}$ and the modified system prompt as $c$ ) achieves a 75% human preference rate for system-prompt following at $\gamma = 3$ , without degrading user-prompt relevance, as shown in Figure 5.

The human evaluation (611 votes from 71 unique voters) asked participants to compare two blinded generations—one with vanilla sampling using the modified system prompt, one with CFG using the modified prompt as $c$ and the default prompt as $\bar{c}$ —on two questions: (A) which output better follows the system prompt, and (B) which output better follows the user prompt. The results in Figure 5 show:

System-prompt following preference for CFG peaks at $\gamma = 3$ with 75% preference. At $\gamma = 1$ (which is vanilla sampling with the modified prompt alone, no negative prompting), preference is at 50% (chance). The preference rises to roughly 60% at $\gamma = 2$ , 75% at $\gamma = 3$ , and then declines to roughly 65% at $\gamma = 4$ , 55% at $\gamma = 5$ , and below 50% at $\gamma = 6$ .
User-prompt relevance remains stable. At $\gamma = 3$ , the paper reports 52% preference for CFG on user-prompt relevance—statistically indistinguishable from chance. This means CFG with negative prompting improves adherence to behavioral instructions without sacrificing responsiveness to the user's query. At $\gamma \geq 4$ , user-prompt preference begins to degrade.

A qualitative example is shown in Table 1: a system prompt asking for "an enthusiastic response" to the query "What was the Cambridge Analytica scandal?" Without CFG, the model "ignores the system-level directive" and produces a standard factual response. With CFG (and presumably negative prompting emphasizing the difference from the default system prompt), the model produces an enthusiastic response beginning "Oh my goodness! What a scandal!" while still containing the factual content. Additional qualitative examples in Appendix Table 16 show similar patterns for more extreme system prompts like "write a seductive response."

The paper notes that this approach directly addresses a known failure mode: "systems like Alpaca [77, 59, 3] often ignore changes to their default system-prompt, and may even expose models to attacks like prompt injection." By explicitly subtracting the default system prompt's influence, CFG makes behavioral modifications more effective.

FLOPs-Matched Comparison: Model Size vs. CFG

The FLOPs-matched analysis (Section 4, Figures in Appendix C.2, Table 4) asks: given the same inference FLOP budget, should a user deploy a model of size $N$ with CFG or a model of size $2N$ without CFG? The headline finding is that across 5 out of 9 benchmarks, there is no statistically significant difference between the two options ( $p > 0.01$ by ANCOVA), and among the 4 tasks with significant differences, 2 favor CFG and 2 favor the larger vanilla model.

The methodology: for each benchmark, accuracy is plotted against FLOPs per token at inference, using data points from all three model families (GPT-2, Pythia, LLaMA) at all sizes. Blue points represent vanilla models (no CFG); red points represent models with the best-performing CFG strength. Logistic regression curves are fit to each group (blue and red), and ANCOVA tests whether the regression lines differ significantly.

The results by benchmark (Table 4 for p-values, Figure 11 for plots):

LAMBADA: CFG wins significantly ( $p < 0.001$ ). The red regression line (CFG) sits clearly above the blue line across the full FLOPs range—CFG provides a genuine compute-efficiency advantage for sentence completion.
WinoGrande: Vanilla wins significantly ( $p = 0.003$ ). The blue line (vanilla) sits above the red line—CFG hurts on this pronoun resolution task, and the larger model without CFG is the better investment of FLOPs.
SciQ: CFG wins significantly ( $p = 0.008$ ). CFG provides better accuracy per FLOP on scientific QA.
TriviaQA: Vanilla wins significantly ( $p = 0.008$ ). The larger vanilla model outperforms CFG on closed-book QA.
HellaSwag ( $p = 0.012$ ), PIQA ( $p = 0.030$ ), ARC-c ( $p = 0.216$ ), BoolQ ( $p = 0.345$ ), ARC-e ( $p = 0.355$ ): No significant difference. For these five benchmarks, a model of size $N$ with CFG is statistically indistinguishable from a model of size $2N$ without CFG.

The paper's interpretation emphasizes the practical implications: "this indicates that, overall, a model using CFG can generally perform just as well as a model twice as large. This has enormous implications for training budgets and inference latency due to limited VRAM usage." The key caveat is that the equivalence is not universal—it depends on the task—and the paper does not provide a predictive theory for which tasks will benefit.

Analysis: Entropy, Instruction-Tuning Comparison, and Token Visualization

For the analysis experiments (Section 5, using 32,902 datapoints from P3 with Falcon-7B-base and Falcon-7B-Instruct), the headline findings are: CFG reduces sampling entropy to levels comparable to instruction-tuning, but achieves this through different vocabulary reorderings; CFG and instruction-tuning are most similar on longer, more specific prompts; and the token-level visualization confirms that CFG upweights prompt-relevant vocabulary and downweights irrelevant vocabulary.

Entropy reduction (Section 5.1, Figure 6a). The mean logit entropy across generation timesteps is 5.4 for vanilla prompted $P(y|x)$ , compared to 4.7 for CFG at $\gamma = 1.5$ . The instruction-tuned model shows similar entropy reduction. This lower entropy means the probability mass is concentrated on fewer tokens, making the model less likely to sample rare tokens that could cause meandering. Figure 6b shows that the number of tokens in the top-p=90% of the vocabulary decreases correspondingly.

CFG vs. instruction-tuning vocabulary overlap (Section 5.2, Figure 6b). CFG shares roughly 50% of its top-p=0.9 vocabulary with the vanilla prompted model, but the overlap with the instruction-tuned model is lower. This means CFG and instruction-tuning are "largely not overlapping" in which specific tokens they promote, despite both reducing entropy. The paper states: "CFG is not having a similar effect on the vocabulary logits as instruction-tuning."

When CFG and instruction-tuning align (Tables 8, 9, 10). The paper identifies conditions where the two approaches are most and least similar. Longer prompts correlate with higher CFG-instruct agreement (Spearman $r_s = 0.05$ ). Harder phrases (higher perplexity for instruction-tuned models) also correlate with higher agreement (Table 7b). The P3 datasets where CFG is most similar to instruction-tuning include SuperGLUE, SciQ, and CosE tasks—generally, tasks with well-specified constraints. Where they are least similar: open-ended, vague prompts like "How do you feel about your current weight and eating habits?" or "What do you usually do when you meet a guy for the first time?"

Token visualization (Section 5.3, Table 3). For the prompt "The dragon flew over Paris, France," the paper visualizes the vocabulary ranked by $\log P(w_t|w_{<t}) - \log P(w_t|\hat{w})$ at each generation step. The top-ranked tokens at early steps relate to flying ("swoop," "circled," "darted"), dragons ("Dragon," "dragons," "Winged"), and Paris landmarks ("Cathedral," "Basil," "Mosque"). The bottom-ranked tokens include unrelated locations ("Queensland," "Newfoundland"), dates ("1913," "1944"), and irrelevant topics ("hostages," "voyages"). As generation proceeds, the visualizations show how CFG continues to reinforce the prompt's semantic fields at each step, maintaining topic coherence.

The paper describes this visualization as "an integral part of effective prompt engineering," since it shows practitioners what their prompts are actually emphasizing and suppressing in the model's vocabulary distribution.

Ablation Studies and Robustness Checks

The paper's ablation studies are distributed across multiple sections and appendices rather than being centralized. Here are the non-trivial ablations:

Effect of varying $\gamma$ on basic prompting accuracy (Appendix C.1, Figures 8–10). For each model family and benchmark, accuracy is shown as a function of $\gamma$ from 1.0 up to approximately 1.5–2.0. The curves generally increase from $\gamma = 1$ to a peak, then plateau or slightly decline. The optimal $\gamma$ varies by task: LAMBADA peaks at approximately $\gamma = 1.3$ – $1.5$ across model sizes; HellaSwag shows continuing improvement up to $\gamma = 1.5$ for most models; WinoGrande degrades monotonically with increasing $\gamma$ for nearly all models, suggesting CFG is fundamentally harmful for pronoun resolution tasks. The paper does not systematically sweep beyond $\gamma = 1.5$ for basic prompting, leaving the high- $\gamma$ regime largely unexplored in this setting.

Effect of varying $\gamma$ on code generation pass@k (Appendix C.3, Tables 5–7, Figures 12–14). Pass@1, pass@10, and pass@100 are reported for $\gamma = 1.0, 1.1, 1.25, 1.5, 1.75, 2.0$ at temperatures 0.2, 0.6, 0.8. The optimal $\gamma$ for pass@1 is consistently between $1.1$ and $1.5$ , with performance degrading beyond $1.5$ . For pass@10 and pass@100, the optimal $\gamma$ is lower or $\gamma = 1$ is already optimal—CFG provides little to no benefit for high- $k$ metrics, and at $\gamma = 2.0$ pass@10 and pass@100 are substantially below baseline across all models and temperatures. For example, CodeGen-350M-mono at temperature 0.2: pass@100 drops from 22.0% (vanilla) to 17.6% at $\gamma = 2.0$ . This ablation confirms the diversity-precision tradeoff: CFG sharpens the distribution around prompt-relevant tokens, reducing the chance of sampling the rare correct solutions that high- $k$ evaluation captures.

Effect of varying $\gamma$ on chain-of-thought validity and accuracy (Figures 3 and 15). For GSM8K and AQuA with WizardLM-30B and Guanaco-65B, the invalid answer rate (chains that fail to produce a parsable final answer) decreases with increasing $\gamma$ , but accuracy peaks at low $\gamma$ ( $\approx 1.1$ – $1.3$ ) and declines thereafter. This ablation reveals that CFG's benefits for chain-of-thought come primarily from enforcing output structure (validity) rather than improving reasoning quality—at high $\gamma$ , the chains are structurally valid but substantively worse.

Effect of varying $\gamma$ on chatbot human preferences (Figure 5). For GPT4All with negative prompting, system-prompt following preference peaks at $\gamma = 3$ (75%) and user-prompt relevance remains undegraded at this value but degrades at $\gamma \geq 4$ . This ablation identifies a clear sweet spot where negative prompting improves behavioral adherence without sacrificing query responsiveness, and a degradation regime where overemphasis on the system prompt harms user-prompt relevance.

Comparison of CFG vs. instruction-tuning (Section 5.2, Figures 6–7, Tables 7–10). This can be viewed as an ablation on the hypothesis that CFG approximates instruction-tuning. The finding that CFG and instruction-tuning have largely non-overlapping top-p vocabulary (Figure 6b) and that their similarity varies with prompt characteristics (Table 7) falsifies the simple approximation hypothesis. This is a negative result—the two techniques are not equivalent—with positive implications for their complementarity.

Temperature interaction with CFG for code generation (Tables 5–7, Figures 12–14). By evaluating CFG at three different temperatures (0.2, 0.6, 0.8) for each model size on HumanEval, the paper implicitly ablates the interaction between CFG and sampling stochasticity. The finding that CFG provides larger relative improvements at higher temperatures for some configurations (e.g., CodeGen-2B-mono pass@1 at temperature 0.8: from 12.3% to 14.9% at $\gamma = 1.5$ , a 2.6 point gain, vs. temperature 0.2: from 19.5% to 20.9%, a 1.4 point gain) suggests that the unconditional distribution becomes more diffuse at higher temperatures, making the subtraction operation more impactful.

Prompt language consistency for GPT-J (Appendix D.2, Table 12). The confusion matrix for programming language generation shows that CFG at $\gamma = 1.25$ improves correct-language accuracy from 73% to 86%, but higher $\gamma$ (1.5, 1.75) reduces it to 81% and 77%. This non-monotonic effect mirrors the HumanEval findings and confirms that CFG strength must be tuned per-task.

Machine translation evaluations (Appendix D.1, Table 11). For 0-shot translation on WMT14 fr-en, Bloom-3B with CFG at $\gamma = 1.10$ improves BLEU from 14.16 to 15.81, but $\gamma = 1.25$ drops back to 14.16. RedPajama-Incite-3B improves from 15.04 to 17.24 at $\gamma = 1.10$ and 17.78 at $\gamma = 1.25$ . For 1-shot translation, CFG provides no improvement or slight degradation—the 1-shot example already provides strong conditioning, making the unconditional subtraction less beneficial. For mT0 (a prompt-tuned model), CFG provides no improvement, suggesting that "prompt-tuned models might already be at the pinnacle of possible 0-shot performance." This is a useful boundary condition: CFG helps most when the base model's conditioning is weak or ambiguous.

Replication across model architectures and training paradigms. While not a formal ablation, the paper's evaluation across GPT-2 (standard pretraining), Pythia (controlled training with public data), LLaMA (efficient pretraining), CodeGen (code-specialized), Falcon (recent architecture), Bloom (multilingual), mT0 (prompt-tuned), and GPT4All (fine-tuned assistant) serves as a robustness check on the claim that CFG is architecture-agnostic. The technique works across all families, though with varying optimal $\gamma$ values.

Critical Assessment

The experiments in this paper collectively support the claim that CFG improves prompt-adherence across a wide range of tasks and models, but the strength of support varies substantially by claim, and several important questions remain unaddressed.

Claim 1: "CFG improves the performance of Pythia, GPT-2 and LLaMA-family models across an array of tasks: Q&A, reasoning, code generation, and machine translation, achieving SOTA on LAMBADA with LLaMA-7B over PaLM-540B."

What is supported: The benchmark results in Table 2/Figure 2 clearly show that $\gamma = 1.5$ improves accuracy over $\gamma = 1$ for most tasks across all three model families. The improvements are consistent in direction and nontrivial in magnitude. The LAMBADA SOTA claim is technically true—LLaMA-7B with CFG scores 81.3% vs. PaLM-540B's 77.9%—but comparing a LLaMA model with CFG to a PaLM model without it confounds model architecture, training data, and the CFG technique itself. The within-family comparisons (e.g., LLaMA-7B with CFG vs. LLaMA-7B without) are the rigorous evidence, and these show genuine improvement.

What is not fully supported: "Across an array of tasks" is overstated for machine translation—the paper reports a single BLEU score improvement on one language pair (WMT14 fr-en) with mixed results (Bloom-3B benefits at $\gamma = 1.10$ but not at $\gamma = 1.25$ ; 1-shot and mT0 show no benefit). The machine translation evaluation is thin compared to the other tasks. The claim of reasoning improvement is also mixed: chain-of-thought experiments show improvements in answer validity and moderate accuracy gains at low $\gamma$ , but the paper acknowledges it has "only scratched the surface" and does not report exact accuracy numbers for the CoT experiments in the main text. The reader must rely on figures for these results.

Missing evidence: The paper does not evaluate CFG on summarization, dialogue, or open-ended creative generation beyond the chatbot experiments—tasks where prompt-adherence is arguably most critical. No evaluation is performed on safety-relevant tasks like toxicity reduction or factuality, despite the paper's own acknowledgment that CFG could be misused for prompt injection. For a technique presented as broadly applicable, the task coverage has notable gaps.

Claim 2: "CFG brings improvements equivalent to a model with twice the parameter-count."

What is supported: The FLOPs-matched analysis in Section 4 is the paper's most rigorous experiment. The ANCOVA results in Table 4 show that for 5 out of 9 benchmarks, there is no statistically significant difference between a model with CFG and a model with twice the parameters without CFG. This is a well-executed analysis with appropriate statistical methodology and a clear finding.

What is not fully supported: The claim of "equivalence" ignores the 4 benchmarks where there is a significant difference—2 where CFG wins (LAMBADA, SciQ) and 2 where the larger model wins (WinoGrande, TriviaQA). The paper's abstract and introduction phrase this as "CFG brings improvements equivalent to a model with twice the parameter-count" without qualification, but the results show a mixed picture. A more accurate summary would be: "CFG recovers performance comparable to ~2× model scale on most benchmarks, but not all." Additionally, the FLOPs comparison counts only inference FLOPs, not training FLOPs. While the paper correctly notes that a 7B model is cheaper to train than a 14B model, the inference-FLOP-matched comparison is the relevant metric for deployment; the training cost savings are a separate benefit.

Missing evidence: The paper does not compare CFG against instruction-tuning with matched inference budgets. A natural question: does CFG on a base 7B model outperform an instruction-tuned 7B model at the same inference FLOPs? This comparison would help practitioners choose between applying CFG to a base model vs. investing in instruction-tuning. The analysis in Section 5.2 shows CFG and instruction-tuning are different, but does not compare their performance on downstream tasks.

Claim 3: "CFG can stack alongside other inference-time methods like Chain-of-Thought and Self-Consistency, yielding further improvements in difficult tasks."

What is supported: The chain-of-thought experiments (Section 3.2) show that CFG improves CoT answer validity and modestly improves accuracy. The paper mentions Self-Consistency in the abstract but provides no experiments on Self-Consistency—this is a claim in the abstract not supported by any experimental evidence in the paper.

What is not supported: The "stacking" with Self-Consistency is entirely unevaluated. Self-Consistency [80] involves sampling multiple reasoning chains and taking a majority vote over final answers. Combining this with CFG would require specifying how CFG interacts with multiple samples (does each sample use the same $\gamma$ ? Are multiple $\gamma$ values used?)—none of this is explored. The abstract's claim overreaches the paper's experimental content.

Claim 4: "CFG can be used to increase the faithfulness and coherence of assistants in challenging form-driven and content-driven prompts: in a human evaluation we show a 75% preference for GPT4All using CFG over baseline."

What is supported: The human evaluation in Section 3.4 provides compelling evidence that CFG with negative prompting improves system-prompt following without degrading user-prompt relevance. The 75% preference at $\gamma = 3$ is statistically meaningful (611 votes from 71 unique voters, though the paper does not report confidence intervals or significance tests for this specific number). The qualitative examples in Tables 1 and 16 are convincing.

What is not fully supported: The chatbot evaluation tests a specific model (GPT4All-J v1.3-jazzy) with specific types of system prompts (the 25 listed in Appendix G.1). These prompts are largely stylistic ("write a rap response," "write a sad response," "write an enthusiastic response") rather than safety-critical or instruction-following in the RLHF sense. The paper claims CFG improves "faithfulness and coherence" but evaluates only prompt-adherence, not factual faithfulness. "Coherence" is not directly measured—no automatic coherence metrics (e.g., entity grid [7]) are reported, and human evaluators were asked about prompt-following, not global coherence.

Missing evidence: The paper does not evaluate CFG on instruction-following benchmarks like AlpacaEval, MT-Bench, or the various IFEval datasets. These would provide standardized, replicable evaluations of instruction-following quality. The human evaluation, while well-designed for its specific question, does not substitute for standardized benchmarks that the community can compare against. Additionally, only one chatbot model is evaluated; there is no evidence that the negative prompting approach transfers to other assistant-style models.

General weaknesses across claims:

The paper does not report confidence intervals or standard errors for any benchmark results. Table 2/Figure 2 reports point estimates (e.g., "22.7 / 23.0" for ARC-c on GPT2-small) without any indication of variance. Given that many of these benchmarks have test sets of varying sizes, and the improvements are often 1–3 percentage points, uncertainty quantification is essential for interpreting whether small gains are reliable. The ANCOVA analysis in Section 4 is the only exception, reporting p-values for regression line comparisons.
The $\gamma$ tuning problem is acknowledged but not systematized. The paper uses different $\gamma$ values for different experiments ( $\gamma = 1.5$ for basic prompting, $\gamma = 1.1$ – $1.5$ for code, $\gamma = 1.1$ – $1.3$ for CoT, $\gamma = 3$ for chatbot), but provides no methodology for selecting $\gamma$ a priori. The statement that "CFG requires tweaking and exploration" is honest but limits practical deployability—a practitioner facing a new task has no guidance on choosing $\gamma$ beyond trial and error.
The test sets for basic prompting benchmarks are not explicitly sized. LAMBADA, HellaSwag, PIQA, WinoGrande, ARC, BoolQ, SciQ, and TriviaQA have different numbers of test examples, and the paper does not report these sizes. Some benchmarks (e.g., WinoGrande) are known to have test sets small enough that a few percentage points of difference may not be statistically significant.
The HumanEval benchmark has only 164 tasks. The pass@k results in Tables 2 and 5–7 are based on relatively few programming problems, which limits the precision of the estimates, especially for pass@100 where the computation involves an unbiased estimator with variance that grows with $k$ . The paper does not address this statistical limitation.
The open-ended generation experiments are largely qualitative. The chatbot evaluation is the only human study; the code generation evaluation is based on automated tests rather than human judgment of code quality; the chain-of-thought experiments show only a few qualitative examples. For a technique that primarily affects generation quality (rather than classification accuracy), the lack of human evaluation across most tasks is a significant gap.
No comparison with contrastive decoding. Contrastive Decoding [45] is the most directly comparable inference-time technique (it also modifies logits using a difference between two distributions). The paper acknowledges this prior work but provides no experimental comparison, making it impossible to assess whether CFG's single-model approach is competitive with or superior to two-model contrastive decoding.
The multi-turn revision setting is not explored. The paper tests CFG on single-turn generation (one prompt, one continuation). Many practical applications involve multi-turn dialogue where prompt-adherence must be maintained across turns. Whether CFG's effects compound or diminish across multiple turns is unknown.

Summary of experimental support: The paper provides strong evidence that CFG improves basic prompting performance on standard NLP benchmarks, with particularly robust effects on LAMBADA and SciQ. The FLOPs-matched comparison is a rigorous analysis that supports the ~2× scaling equivalence claim with appropriate caveats. The chatbot human evaluation provides evidence for negative prompting's effectiveness. However, the paper's broader claims about reasoning, code generation, and machine translation are supported by thinner experiments (small benchmarks, limited human evaluation, qualitative rather than quantitative analysis of CoT improvements). The claim about stacking with Self-Consistency is entirely unevaluated. The absence of confidence intervals, the $\gamma$ tuning problem, and the lack of comparison with contrastive decoding or instruction-tuning on standardized benchmarks are the most significant experimental gaps. A more conservative reading of the evidence would conclude that CFG is a useful technique for improving short-form prompt-following and sentence completion, with promising but not yet fully validated extensions to longer-form generation, code, and assistant-style prompting.

6. Limitations and Trade-offs

The γ Tuning Problem: No Methodology for Selecting Guidance Strength a Priori

The assumption or constraint. The paper introduces a single new hyperparameter, the guidance strength γ, and demonstrates that its optimal value varies substantially across tasks, prompting paradigms, and even individual examples. CFG is presented as a broadly applicable technique, but its deployment requires selecting γ—and the paper provides no principled method for doing so other than sweeping values on a development set. The authors acknowledge this openly in the Conclusion:

"CFG requires tweaking and exploration: γ values that might work in one context (i.e. long-form generation) might be poorly suited for another context."

This is not a minor caveat. The experimental results show that the difference between a well-chosen γ and a poorly-chosen one can be the difference between substantial improvement and substantial degradation. On HumanEval, CodeGen-350M-mono pass@1 at temperature 0.2 goes from 11.8% at γ = 1.1 (improvement over 11.0% baseline) to 8.6% at γ = 2.0 (22% relative degradation, Table 2). On chain-of-thought reasoning, accuracy improves at low γ but degrades at high γ (Figure 3). On chatbot generation, user-prompt relevance remains undegraded at γ = 3 but degrades at γ ≥ 4 (Figure 5). The optimal γ varies not just by task but by model size and temperature—CodeGen-2B-mono at temperature 0.8 peaks at γ = 1.5 (14.9% vs. 12.3% baseline) while at temperature 0.2 peaks at γ = 1.5 (20.9% vs. 19.5%), but the magnitude of improvement and the penalty for overshooting differ across these settings (Tables 5–6).

The consequence. A practitioner facing a new task, model, or prompting paradigm has no guidance on how to select γ. The paper sweeps γ = 1.5 as a default for basic prompting, γ = 1.1–1.5 for code generation, γ = 1.1–1.3 for chain-of-thought, and γ = 3 for chatbot negative prompting, but these values were determined post-hoc through experimentation on those specific benchmarks. There is no predictive model, no heuristic, and no calibration procedure described. In practice, this means deploying CFG on a new task requires either (a) running a full γ sweep on a held-out development set—which may not be available, especially in zero-shot settings—or (b) guessing and hoping the guess is within the narrow range where CFG helps rather than harms. This is a significant barrier to out-of-the-box deployment, undercutting the paper's framing of CFG as a simple, universally applicable technique.

The problem is compounded by the fact that γ interacts with other sampling hyperparameters (temperature, top-p, top-k). The paper evaluates CFG at different temperatures for code generation but does not systematically characterize this interaction or provide rules for joint selection. A practitioner using nucleus sampling with p = 0.9 and temperature 0.8—both common defaults—has no way of knowing whether the γ that worked at temperature 0.2 in the paper's experiments will transfer.

What evidence exists in the paper. The evidence for this limitation is spread throughout the experimental sections but never aggregated into a systematic analysis. Figures 8–10 in Appendix C.1 show accuracy as a function of γ for basic prompting, revealing that the optimal γ is task-dependent but rarely exceeding 1.5 for these benchmarks. Tables 5–7 and Figures 12–14 show the γ dependence for HumanEval, with clear non-monotonic behavior. Figures 3 and 15 show γ curves for chain-of-thought, with the accuracy peak at low γ. Figure 5 shows the γ curve for chatbot preference, with a clear peak at γ = 3 and degradation on both sides. The paper never collects these into a unified analysis of the γ selection problem, nor does it propose any solution.

Mitigation status. The paper does not attempt to mitigate this limitation. It acknowledges the problem in the conclusion but frames it as a property of the technique rather than a gap to be addressed. No heuristics, calibration procedures, or adaptive γ selection methods are proposed. The visualization technique in Section 5.3—ranking vocabulary by the difference between conditional and unconditional log-probabilities—is suggested as a prompt engineering tool, but it visualizes the effect of the prompt itself (which exists at γ = 1) rather than the effect of varying γ; it does not help a practitioner select γ. The paper's future work section does not explicitly flag γ selection as a problem to solve. For a limitation that the authors themselves identify as the primary practical barrier to using CFG, this is a notable gap in the paper's contribution to deployability.

The 2× FLOPs Cost: Doubled Inference Compute With No Latency Analysis

The assumption or constraint. CFG requires two forward passes per generated token—one with the full prompt context and one with the prompt truncated. This roughly doubles the per-token inference cost from $2N$ FLOPs to $4N$ FLOPs for a model with $N$ parameters. The paper is transparent about this cost:

"CFG almost doubles the computation during inference."

However, the paper frames the cost primarily through a FLOPs-matched comparison against larger models (Section 4), arguing that a model of size $N$ with CFG achieves comparable performance to a model of size $2N$ without CFG at the same inference FLOP budget. This framing implicitly assumes that doubled FLOPs is acceptable because the alternative (deploying the larger model) would cost the same at inference time and substantially more at training time. But this analysis is incomplete in several critical ways.

The consequence. The FLOPs-matched comparison addresses throughput (tokens generated per unit compute) but entirely ignores latency (wall-clock time per token). These are not equivalent in practice. A model of size $2N$ requires two sequential forward passes through a network with twice the parameters, while CFG requires two sequential forward passes through a network with $N$ parameters. The latency of each forward pass scales roughly linearly with parameter count (for a fixed sequence length and hardware configuration). So CFG on an $N$ -parameter model has latency roughly equal to a vanilla $2N$ -parameter model at the same sequence length—but this means CFG provides no latency advantage over simply using the larger model.

Where CFG does provide an advantage is in memory: an $N$ -parameter model with CFG requires approximately $2N$ bytes of VRAM (at 16-bit precision), while a $2N$ -parameter model requires approximately $4N$ bytes. For VRAM-constrained deployments, this is genuinely valuable—the paper emphasizes "limited VRAM usage" as a key constraint. But the paper does not discuss the latency implications or provide any latency measurements, even though for interactive applications (chatbots, code assistants, real-time translation), latency is often the binding constraint, not VRAM or throughput.

A deeper issue: the paper's FLOPs analysis counts only the per-token generation FLOPs, not the prompt processing cost. In long-prompt settings (chain-of-thought with extensive few-shot examples, chatbot with long system instructions), a substantial fraction of total inference FLOPs is spent on processing the prompt in the prefill phase. CFG's unconditional forward pass truncates the prompt, so it processes a shorter sequence during prefill—meaning the FLOPs penalty is less than 2× for long prompts. But this is not analyzed or accounted for in the paper's cost model. The effective FLOPs multiplier depends on the ratio of prompt length to generation length, which varies across use cases and is never characterized.

What evidence exists in the paper. The FLOPs-matched comparison in Section 4 and Appendix C.2 provides regression lines of accuracy vs. FLOPs per token for vanilla models and CFG models. The ANCOVA analysis in Table 4 shows that for 5 of 9 benchmarks, the two regression lines are not significantly different—meaning that the accuracy-per-FLOP curves overlap. This is the evidence for the claim that CFG is FLOPs-competitive with a 2× larger model. However, no latency measurements are reported anywhere in the paper. No wall-clock times, no throughput numbers, no discussion of whether the two sequential forward passes can be batched or parallelized. The hardware configuration (GPU type, batch size, sequence lengths) is not specified for any experiment.

Mitigation status. The paper does not attempt to mitigate the latency concern or provide latency measurements. The cost analysis in Section 4 focuses exclusively on FLOPs and training cost savings, while the inference cost discussion remains at the level of FLOP accounting. The paper does not discuss potential optimizations—for example, reusing the key-value cache from the conditional forward pass for the unconditional pass (which shares the generation prefix), or running the two forward passes in parallel on separate devices. The conclusion mentions "inference latency due to limited VRAM usage" as a topic for future work but does not address it in the current paper. This leaves a critical gap for practitioners evaluating CFG for latency-sensitive deployments.

Single-Domain Evaluation: No Evidence Beyond Short-Form English Benchmarks and One Code Benchmark

The assumption or constraint. The paper evaluates CFG on a specific set of tasks: standard NLP benchmarks (LAMBADA, HellaSwag, PIQA, ARC, BoolQ, SciQ, TriviaQA, WinoGrande), two arithmetic reasoning datasets (GSM8K, AQuA), one code generation benchmark (HumanEval), one machine translation language pair (WMT14 fr-en), and one custom chatbot evaluation (with GPT4All). All of these tasks require generating relatively short, well-defined outputs—sentence completions, multiple-choice answers, arithmetic solutions, Python functions, translations of individual sentences, or single-turn assistant responses.

The paper does not evaluate CFG on any of the following, despite positioning CFG as a general-purpose technique for language generation: open-ended text generation (story completion, essay writing), multi-turn dialogue, text summarization, question answering with long-form answers, retrieval-augmented generation, or any task requiring integration of external knowledge. The paper also does not evaluate CFG on safety-relevant metrics (toxicity, factuality, hallucination rate, bias) despite acknowledging in the conclusion that CFG could be misused:

"We have not tested the effects of CFG if used in conjunction with malicious strategies for hacking language models, including but not limited to: prompt injection and prompts aimed at overriding alignment."

The consequence. The paper's claim that CFG is effective "across an array of tasks" and that it "can be used broadly as an inference-time technique in pure language modeling" (abstract) is supported only for the specific task types evaluated. The core mechanism of CFG—amplifying the prompt's influence on the next-token distribution—has different implications for different generation scenarios:

For short-form generation with clear answers (LAMBADA, multiple-choice QA), CFG helps by sharpening the distribution toward the prompt-relevant tokens. This is the regime where the paper's evidence is strongest.
For long-form structured generation (code), CFG helps up to a point (γ ≤ 1.5) but higher guidance harms pass@k, suggesting a tradeoff between prompt-adherence and solution diversity that is not fully characterized.
For open-ended generation (creative writing, dialogue, storytelling), CFG's entropy reduction (mean entropy 4.7 vs. 5.4 vanilla, Section 5.1) could produce repetitive or overly conservative text—a failure mode that is not evaluated.
For tasks requiring factual accuracy, CFG amplifies whatever the model "believes" given the prompt, but if the model's prior is factually wrong, CFG could amplify confident hallucinations rather than correct them. This is not evaluated.

The absence of safety evaluations is particularly notable given the paper's own concern about misuse. If CFG increases adherence to prompts, and prompts can be adversarial (prompt injection, jailbreaking), then CFG could increase the success rate of attacks. The paper acknowledges this possibility but does not test it. Similarly, if CFG makes the model more likely to follow system-level instructions, it could amplify harmful instructions in the system prompt or make the model more susceptible to prompt-leaking attacks where the system prompt contains sensitive information that the model should not reveal.

What evidence exists in the paper. The evidence for this limitation is largely negative—the absence of certain evaluations. The paper explicitly states:

"We have not tested the effects of CFG if used in conjunction with malicious strategies for hacking language models"

and:

"It's possible that there are unforeseen effects induced by an increased adherence to parts of the prompt. We tried to explore this at length, both quantitatively and qualitatively, and we designed tasks that might reveal such behavior. However, we cannot conclude this method is risk-free."

This is an honest acknowledgment, but the paper's tasks were not designed to test safety or robustness—they are standard academic benchmarks. The "tasks that might reveal such behavior" are not described or reported. The chatbot evaluation tests stylistic instruction-following (e.g., "write a sad response") not safety-relevant behaviors.

Mitigation status. The paper does not attempt to mitigate this limitation through additional experiments. The authors call for standardized safety benchmarks in their conclusion:

"We advocate for standardized benchmarks aimed more squarely at language-model risk (including, possibly, pairs of models along with known prompt injections). Such standardized benchmarks could help us unit-test an advancement like CFG before releasing it into the wild."

This is a reasonable suggestion but it is a call for future community infrastructure, not a mitigation provided by the paper. The paper does not attempt even small-scale safety probing (e.g., testing whether CFG makes GPT4All more susceptible to known jailbreak prompts, or whether it increases toxicity as measured by standard toxicity classifiers). This leaves the safety implications of CFG entirely uncharacterized.

The Negative Prompting Baseline Is Not a Standard Baseline—It Relies on the Default System Prompt as an Implicit Negative

The assumption or constraint. The negative prompting extension (Section 3.4, Equation 5) replaces the unconditional distribution in standard CFG with a "negative conditioning" distribution $P_\theta(w_i | w_{j<i}, \bar{c})$ , where $\bar{c}$ is a prompt specifying what should be suppressed. In the chatbot experiments, $\bar{c}$ is set to the model's default system prompt (e.g., "The prompt below is a question to answer, a task to complete, or a conversation to respond to; decide which and write an appropriate response.") and $c$ is set to the modified behavioral prompt (e.g., "...write a sad response").

This choice of $\bar{c}$ is natural—the goal is to override the model's default behavior—but it introduces a significant assumption: that the model's default system prompt is known and accessible. For many deployed chatbots (ChatGPT, Claude, Gemini), the system prompt is proprietary and not disclosed. For fine-tuned models where the system prompt was embedded in training data rather than explicitly provided at inference time, there may be no clear "default system prompt" to use as $\bar{c}$ .

More fundamentally, the paper does not ablate the choice of $\bar{c}$ . Is the default system prompt the optimal negative prompt? Would a more generic negative prompt (e.g., "write a neutral, generic response") work better across different system prompt modifications? Would an empty negative prompt (reducing to standard CFG) work nearly as well? The paper provides no comparison of different $\bar{c}$ choices, so the sensitivity of the approach to this design decision is unknown.

The consequence. The negative prompting results in Section 3.4 cannot be interpreted as evidence that CFG with negative prompting is generally effective for chatbots—they demonstrate effectiveness for one specific model (GPT4All-J v1.3-jazzy) with one specific choice of $\bar{c}$ (that model's default system prompt) applied to 25 specific system prompt modifications. A practitioner deploying CFG on a different chatbot model would need to identify an appropriate $\bar{c}$ , and there is no methodology for doing so.

The paper's framing of negative prompting also conflates two distinct mechanisms: (1) standard CFG, which moves away from the completely unconditional distribution, and (2) negative prompting, which moves away from a specific undesired distribution. The ablation that would separate these is missing: the paper does not compare standard CFG (with $\bar{c} = \emptyset$ , which reduces to Equation 7) against negative prompting (with $\bar{c} = \text{default system prompt}$ ) on the chatbot task. It is therefore impossible to determine whether the benefit comes from negative prompting specifically or from standard CFG applied in the chatbot setting. The paper reports only the comparison between vanilla sampling (γ = 1, no CFG at all) and negative-prompting CFG—not the comparison between standard CFG and negative-prompting CFG.

What evidence exists in the paper. Figure 5 shows human preference for CFG with negative prompting compared to vanilla sampling. The paper does not show a curve for standard CFG (with empty $\bar{c}$ ) on this task. Table 16 in Appendix E shows qualitative examples of negative prompting, but no examples of standard CFG on the same prompts for comparison. Section 2.2 derives the negative prompting formula as a generalization of standard CFG, but Section 3.4 does not report standard CFG results on the chatbot task. The default system prompts used for GPT4All are listed in Appendix G.1, but no sensitivity analysis around different $\bar{c}$ choices is performed.

Mitigation status. The paper does not attempt to mitigate this limitation. There is no ablation of $\bar{c}$ choices, no comparison of negative prompting vs. standard CFG on the chatbot task, and no discussion of how practitioners should select $\bar{c}$ for their own models. The authors do not acknowledge this as a limitation. The finding that negative prompting works for GPT4All with its default system prompt as $\bar{c}$ is a proof of concept, but the generalizability of this finding to other models, other system prompt modifications, and other choices of $\bar{c}$ is entirely unestablished.

CFG Does Not Help—and Sometimes Hurts—on Tasks Requiring Diverse Solutions, and the Diversity Tradeoff Is Not Characterized

The assumption or constraint. The paper presents CFG as a technique for improving prompt-adherence, and the experimental design reflects this—most benchmarks evaluate whether the single best output is correct (accuracy, pass@1). However, the paper acknowledges in multiple places that CFG reduces diversity. From the text-to-image literature that inspired CFG:

"When γ > 1 then $\hat{P}$ overemphasizes the conditioning, which as noticed by [28] results in a better inception score at the cost of diversity."

From the code generation results:

"the consistent improvement on pass@1 rates and the reduced effect on pass@100 rates support our hypothesis that CFG strengthens the adherence to the prompt at the small cost of reduced variability and creativity."

And more specifically, the paper notes that CFG can eliminate low-probability correct solutions:

"without CFG, many tasks exhibit small nonzero passing rates while having 0% rate with CFG. This explains the decreasing improvement of CFG in pass@k for large k, as larger k significantly boosts the passing rate of difficult tasks where the rates are low but nonzero."

This is a fundamental tradeoff: CFG sharpens the distribution toward prompt-relevant tokens, which improves the expected quality of a single sample but reduces the probability of sampling rare but correct solutions that deviate from the model's default prompt-conditioned behavior. The paper acknowledges this tradeoff but does not characterize it quantitatively beyond pass@k metrics on HumanEval.

The consequence. For applications where diversity matters—creative writing, brainstorming, dialogue systems that should produce varied responses, code generation where multiple valid solutions exist—the diversity reduction could be a significant drawback. The paper provides no direct measure of diversity (e.g., distinct-n, self-BLEU, embedding variance, human judgments of diversity) for any task. The only diversity proxy is the pass@k analysis on HumanEval, where CFG's benefit over vanilla sampling decreases as k increases (from consistent pass@1 improvements to mixed pass@10 results to negative pass@100 results at high γ, Tables 5–7).

Even within the benchmarks where CFG improves accuracy, it's unclear whether the improvement comes at the cost of answer diversity. On LAMBADA, the correct answer is a single token (the final word of the passage)—diversity is not relevant. On multiple-choice QA, the model's output is a choice label (A/B/C/D)—diversity is not relevant. But on tasks like TriviaQA, where the correct answer can be expressed in multiple valid forms (e.g., "Mark Twain" vs. "Samuel Clemens"), CFG's entropy reduction could suppress valid alternative phrasings, potentially hurting substring-match accuracy even when the model "knows" the answer. The paper reports that CFG does not help on TriviaQA (and may hurt, e.g., LLaMA-7B drops from 56.0% to 52.7%, Table 2b), which is consistent with this hypothesis but not directly analyzed.

For creative or open-ended tasks, the diversity tradeoff could mean that CFG makes generations more "on-prompt" but also more predictable, repetitive, or boring—a failure mode that the paper's evaluation framework cannot detect because it uses correctness-based metrics.

What evidence exists in the paper. The HumanEval pass@k results (Tables 2, 5–7; Figures 12–14) provide the primary evidence. For CodeGen-350M-mono at temperature 0.2: pass@1 improves from 11.0% to 11.8% at γ = 1.1 (+7.3% relative), but pass@100 drops from 22.0% to 20.1% at the same γ (−8.6% relative). At γ = 2.0, pass@1 drops to 8.6% (−21.8%) and pass@100 drops to 17.6% (−20.0%). This pattern—pass@1 benefiting at low γ while pass@100 degrades—is consistent across model sizes and temperatures. The paper interprets this as evidence for the diversity tradeoff but provides no direct diversity measurement.

The entropy analysis in Section 5.1 (Figure 6a) provides indirect evidence: CFG reduces mean logit entropy from 5.4 to 4.7. Lower entropy means fewer tokens receive substantial probability mass, which mechanically reduces the diversity of sampled outputs. However, the paper does not connect this entropy measurement to any downstream diversity metric.

The ANCOVA analysis in Section 4 (Table 4) shows that CFG provides significant benefits on LAMBADA and SciQ, significant harm on WinoGrande and TriviaQA, and no significant difference on the remaining 5 benchmarks. WinoGrande, in particular, shows a consistent negative effect of CFG across all model families (e.g., LLaMA-7B drops from 67.1% to 65.5%, Figure 10). WinoGrande requires resolving pronoun references where multiple interpretations may be grammatically valid but differ in real-world plausibility—a task where the diversity to consider alternative interpretations might be important. The paper does not analyze why CFG specifically harms WinoGrande, but the diversity tradeoff is a plausible hypothesis.

Mitigation status. The paper does not attempt to mitigate the diversity tradeoff. No diversity-preserving variant of CFG is proposed (e.g., annealing γ over the course of generation, applying CFG only to specific token positions, or using adaptive γ based on estimated uncertainty). The paper does not analyze which types of tasks are most affected by the diversity tradeoff or provide guidance on when diversity concerns should dissuade practitioners from using CFG. The conclusion does not flag diversity reduction as a limitation. For a technique that explicitly trades diversity for prompt-adherence (following the text-to-image CFG precedent), the lack of any diversity characterization is a significant gap in the experimental narrative.

The Difficulty Estimation Mechanism Is Not Studied—CFG's Effectiveness on Hard vs. Easy Problems Is Not Characterized

The assumption or constraint. The companion paper on test-time compute scaling (the prior example in this analysis) identifies problem difficulty as the critical variable determining whether inference-time techniques help or hurt, finding that beam search degrades performance on easy problems due to verifier over-optimization and provides no benefit on the hardest problems where the base model lacks capability. This paper applies CFG uniformly to all examples within a benchmark, reporting only aggregate accuracy without breaking down results by problem difficulty or by the base model's initial success probability on individual questions.

The paper provides no difficulty-binned analysis, no pass@1-conditioned analysis, and no investigation of whether CFG's effects are uniform across easy, medium, and hard examples within each benchmark. This is a significant gap because the mechanism of CFG—amplifying the prompt's signal in the logit distribution—has fundamentally different implications depending on whether the base model already generates the correct answer with high probability (easy examples), occasionally generates it (medium examples), or almost never generates it (hard examples).

The consequence. Without difficulty-conditioned analysis, practitioners cannot determine when CFG is worth the doubled inference cost. Several plausible—but untested—hypotheses about difficulty-dependent behavior follow from CFG's mechanism:

Easy examples: The base model already produces the correct answer with high probability. CFG sharpens the distribution, which should further increase the probability of the correct answer and reduce variance. This would produce modest accuracy gains but at the cost of doubled FLOPs, which may not be cost-effective if the base model is already near ceiling.
Medium examples: The base model produces the correct answer with non-trivial but sub-dominant probability (say, 10–50% pass@1). CFG amplifies the prompt signal, which should boost the correct answer's probability if it is prompt-consistent. This is where CFG should provide the largest benefit—pushing the model from "sometimes correct" to "usually correct."
Hard examples: The base model's pass@1 is near zero—it almost never produces the correct answer regardless of sampling. CFG cannot help here because amplifying the prompt signal does not create new capabilities; it only reshapes the existing probability distribution. If the correct answer receives near-zero probability, CFG's amplification cannot make it likely. Worse, if the model has a systematic misconception (producing a plausible but wrong answer with high confidence), CFG could amplify that wrong answer, making things worse.

This difficulty-dependent hypothesis is directly analogous to the companion paper's finding that test-time compute helps on medium-difficulty problems but not on the hardest ones, and that aggressive optimization can hurt on easy problems. The present paper provides no analysis that would confirm or refute this hypothesis for CFG.

The closest result is the observation that CFG's benefits diminish with model scale for LAMBADA: LLaMA-7B improves from 73.6% to 81.3% (+7.7 points), while LLaMA-65B improves from 79.1% to 84.0% (+4.9 points). This suggests CFG helps less as model capability increases—consistent with the idea that CFG benefits the regime where the prompt signal is weak relative to the model's unconditional bias. But this is model-size analysis, not within-model difficulty analysis, and it does not reveal whether CFG helps on the hardest examples that even large models get wrong.

What evidence exists in the paper. The paper provides no difficulty-binned analysis, no distribution of per-example accuracy gains/losses, and no investigation of which types of examples benefit most from CFG. The scatter plots in Appendix C.3 (Figures 15a–15c) show per-task pass@1 with and without CFG on HumanEval, but these are not analyzed by task difficulty or baseline pass rate. The paper notes that "without CFG, many tasks exhibit small nonzero passing rates while having 0% rate with CFG" but does not systematically characterize which tasks lose their low-probability correct solutions.

Mitigation status. The paper does not attempt to characterize difficulty-dependent effects. This limitation is not acknowledged by the authors and no future work direction is suggested for difficulty-conditioned CFG application (e.g., estimating whether a given prompt is in the regime where CFG helps and adaptively applying γ). Given the centrality of difficulty-conditioned analysis in the companion inference-time scaling paper, this is a notable gap—it means the two papers study complementary aspects of inference-time compute allocation (what strategy to use vs. how much guidance to apply) without either providing the full picture of when CFG should and should not be used on individual prompts.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper makes a deceptively simple intervention—a logit-space interpolation formula applied at inference time—that reshapes how the field should think about the relationship between prompting, model scale, and inference compute. The shift is not a paradigm overthrow, but rather a reclassification: prompt-adherence moves from being a property that must be trained into models (via instruction-tuning, RLHF, or scale) to one that can be dialed up at inference time on any pretrained model, without touching a single weight. This has downstream consequences for how the field allocates effort between training and inference.

The central reframing: prompt-adherence as an inference-time knob, not a training property. Before this paper, if a practitioner needed a 7B model to follow instructions more reliably, the available paths were: fine-tune it on instruction data, apply RLHF, or scale up to a larger model. All three require substantial compute, curated data, and model access. CFG introduces a fourth path: keep the model exactly as-is and spend extra FLOPs at inference time to sharpen its focus on the prompt. This is not a complete substitute for instruction-tuning—the paper's analysis in Section 5.2 shows that CFG and instruction-tuning alter the vocabulary distribution in "largely not overlapping" ways—but it provides an orthogonal axis of improvement that previously did not exist. The implication is that model capability and prompt-adherence can be partially decoupled: a model can possess knowledge without reliably deploying it in response to prompts, and CFG helps bridge that gap without modifying the knowledge itself.

This reframing matters because it shifts the burden of prompt-adherence from the training pipeline to the inference procedure. Training a model to follow instructions well requires anticipating what kinds of instructions users will give—a fundamentally open-ended problem. Inference-time guidance, by contrast, adapts to the specific prompt at hand. A model that was never trained to "write a sad response" can be nudged toward sadness at inference time by subtracting its default behavior and amplifying the modification (Section 3.4). This does not make the model capable of sadness—it merely emphasizes whatever sad-leaning tendencies already exist in its distribution—but it is more flexible than baking every behavioral variation into training data.

Reconciling contradictory intuitions about model scale and prompt-following. The paper resolves a tension that was visible but unarticulated in prior work. On one hand, scaling laws suggested that larger models are better at everything, including prompt-following. On the other hand, practitioners observed that even large models frequently ignore system prompts (e.g., Alpaca-based systems "often ignore changes to their default system-prompt," as the paper notes). The FLOPs-matched analysis in Section 4 provides a reconciliation: for the same inference FLOP budget, a model of size $N$ with CFG performs comparably to a model of size $2N$ without CFG on most benchmarks (5 of 9 tasks show no significant difference, Table 4). This means that prompt-adherence is not purely a function of model scale—it is a function of how inference compute is allocated. A smaller model that invests half its per-token FLOPs in guidance can match a larger model that spends all its FLOPs on standard generation. The practical consequence is that scaling model size is not the only—or even the most cost-effective—way to improve prompt-following.

Shifting the inference-time compute landscape. The paper enters a growing conversation about inference-time compute as a resource to be optimized, alongside the companion paper on test-time compute scaling for math reasoning. CFG occupies a specific point in this landscape: it is a training-free, single-model, per-token intervention that requires no auxiliary verifiers, no search algorithms, and no multi-sample aggregation. This distinguishes it from beam search against process reward models, majority voting, best-of-N sampling, or contrastive decoding with two models. The paper demonstrates that CFG can "stack alongside other inference-time methods like Chain-of-Thought and Self-Consistency" (abstract), suggesting that the inference-time compute budget can be allocated across multiple complementary mechanisms—guidance for per-token prompt-adherence, chain-of-thought for multi-step reasoning structure, and self-consistency for answer selection. The composability of these techniques is not fully explored in this paper, but the demonstration that CFG can be layered on top of chain-of-thought (Section 3.2) without modification establishes the principle.

Making controlled generation accessible without training auxiliary models. Prior controlled generation techniques—PPLM, FUDGE, GeDi, NA-DO—all required training separate classifiers or discriminators for each control dimension. Even Contrastive Decoding [45] requires access to two models (a strong "expert" and a weak "anti-expert"). CFG eliminates this dependency entirely: the same model serves as its own implicit classifier by simply dropping the prompt from context. This dramatically lowers the barrier to entry for controlled generation, making it accessible to practitioners with no training infrastructure and no labeled data. The paper's demonstration across 12+ model families (GPT-2, Pythia, LLaMA, Falcon, CodeGen, GPT-J, Bloom, mT0, RedPajama, WizardLM, Guanaco, GPT4All) establishes that this accessibility is not theoretical—it works on models spanning multiple architectures, training paradigms, and scales, and in some cases works better on smaller models (LLaMA-7B gains +7.7 points on LAMBADA vs. +4.9 for LLaMA-65B, Table 2b), meaning the technique is most valuable where resources are most constrained.

Opening a new diagnostic for prompt engineering. The vocabulary visualization technique in Section 5.3—ranking tokens by their log-probability difference between conditional and unconditional distributions—provides a window into what a prompt is actually doing to a model's semantic space. Before this work, prompt engineering was largely trial-and-error: write a prompt, generate outputs, see if they match expectations. The CFG difference vector makes the prompt's influence interpretable at the token level, showing which concepts are being upweighted and which are being suppressed. The paper presents this as "an integral part of effective prompt engineering," and while the visualization does not require running CFG at generation time (the difference vector exists even at $\gamma = 1$ ), it emerges naturally from the CFG framework. This transforms CFG from a black-box performance booster into a diagnostic tool for understanding model-prompt interaction.

What becomes less attractive. The paper's findings suggest that training separate classifiers for controlled generation (the PPLM/FUDGE paradigm) is largely unnecessary when the base model can serve as its own implicit classifier. The negative prompting extension further reduces the need for attribute-specific classifiers: rather than training a "sadness detector" to steer generation, one can simply contrast the desired prompt against a negative prompt. This does not eliminate the need for fine-grained control—CFG can only amplify what the model already knows—but it covers a substantial range of use cases without any auxiliary model training. The paper also suggests that scaling model size purely for improved prompt-following may be less cost-effective than previously assumed, given that CFG can recover roughly the prompt-adherence benefit of ~2× scale at the same inference FLOP budget for a majority of tasks. However, the paper does not argue against scaling in general—larger models have other benefits (more factual knowledge, better reasoning) that CFG cannot replicate, and the ~2× equivalence does not hold for all tasks.

Follow-Up Research This Work Enables

Adaptive $\gamma$ selection based on prompt difficulty or model uncertainty. The paper's most pressing open problem is the $\gamma$ tuning problem—the optimal guidance strength varies across tasks, models, and even individual prompts, but no selection methodology exists. A strong follow-up would develop a predictive model for optimal $\gamma$ based on features computable before generation: the entropy of the conditional distribution on the first few generated tokens, the KL divergence between conditional and unconditional logit distributions, or the length and specificity of the prompt (building on the finding in Section 5.2 that longer prompts correlate with higher CFG-instruct agreement). A concrete experiment: for each of the 500 LAMBADA test examples, sweep $\gamma$ and record the optimal value; train a lightweight regressor on prompt-level features (number of tokens, average token frequency, conditional entropy at position 1) to predict optimal $\gamma$ ; evaluate whether predicted- $\gamma$ CFG significantly outperforms a fixed $\gamma = 1.5$ baseline. A negative result—finding that optimal $\gamma$ is unpredictable from pre-generation features—would also be informative, suggesting that dynamic $\gamma$ adjustment mid-generation (increasing $\gamma$ when the model appears to be meandering) is necessary.

CFG for safety-critical instruction-following: a systematic red-teaming study. The paper explicitly flags that CFG's effects on prompt injection, jailbreaking, and alignment override are untested. A natural and important follow-up would evaluate CFG against standardized adversarial prompt benchmarks. Concretely: take a set of known jailbreak prompts (e.g., from the Anthropic harmlessness dataset or the AdvBench benchmark) and test whether CFG at various $\gamma$ values increases or decreases the success rate of attacks on instruction-tuned models like LLaMA-2-Chat or Vicuna. The hypothesis is bidirectional: CFG could increase attack success by making the model adhere more strongly to the entire prompt including the adversarial payload, or it could decrease success by amplifying the system prompt's safety instructions over the user's adversarial content. The negative prompting formulation is particularly interesting here: setting $\bar{c}$ to a jailbreak prompt and $c$ to an empty or safety-oriented prompt could create a "de-jailbreaking" mechanism. A careful study would measure both attack success rate and false positive rate (refusing benign requests) as a function of $\gamma$ , producing a safety-utility Pareto frontier for CFG-guided models. The paper's chatbot evaluation framework (Section 3.4) could be adapted for this purpose by replacing stylistic system prompts with safety-relevant ones.

Combining CFG with instruction-tuning: does inference-time guidance add value on top of already-aligned models? The paper's analysis in Section 5.2 shows that CFG and instruction-tuning alter vocabulary distributions in "largely not overlapping" ways, suggesting complementarity. But the paper never tests CFG on instruction-tuned models—all experiments use base models. A direct follow-up would evaluate CFG applied to instruction-tuned variants of the same models: LLaMA-2-Chat, Falcon-Instruct, Mistral-Instruct, or the Pythia fine-tuned variants. The key question: does CFG provide additional prompt-adherence gains on top of instruction-tuning, or do the two techniques saturate the same capability ceiling? The experiment would replicate the basic prompting benchmarks from Section 3.1 using instruction-tuned models with and without CFG, sweeping $\gamma$ to find optimal values (which may differ from the optimal $\gamma$ for base models—instruction-tuned models may require lower $\gamma$ because their conditional distributions are already sharper). A positive result (instruction-tuned models further improved by CFG) would establish that guidance and training-based alignment are genuinely complementary axes. A negative result (instruction-tuning saturates the benefit that CFG provides) would bound CFG's utility to base models and pre-alignment scenarios.

Difficulty-conditioned CFG: when does amplifying the prompt help vs. hurt? The companion paper on test-time compute scaling establishes that inference-time interventions have starkly different effects depending on problem difficulty—beam search helps on medium problems but over-optimizes on easy ones and provides no benefit on hard ones. This paper provides no difficulty-binned analysis of CFG's effects. A well-motivated follow-up would re-analyze CFG's benchmark results broken down by per-example baseline accuracy (pass@1 without CFG for each question in TriviaQA, for instance) to determine whether CFG's benefits are concentrated in a specific difficulty regime. The hypothesis, grounded in CFG's mechanism: CFG should help most on medium-difficulty examples where the model has some probability of producing the correct answer but is pulled off-course by its unconditional biases; it should help least on very easy examples (already near ceiling) and very hard examples (correct answer has near-zero probability regardless of guidance). The HumanEval pass@k results (Tables 5–7) provide suggestive evidence—CFG helps pass@1 but hurts pass@100, implying it suppresses low-probability correct solutions—but a systematic analysis across all benchmarks with per-example difficulty bins would directly test the hypothesis. If confirmed, this would enable adaptive CFG application: estimate difficulty from the first few token predictions, then apply guidance only when the example falls in the medium-difficulty regime.

CFG for multi-turn dialogue and long-form generation coherence. The paper's evaluation is almost entirely single-turn: one prompt, one generation, short output. For dialogue systems and long-form generation (story writing, article drafting, multi-step assistant interactions), prompt-adherence must be maintained across turns and over thousands of tokens. A natural extension would evaluate CFG on multi-turn dialogue coherence and long-form text coherence. Concretely: use a multi-turn dialogue benchmark (e.g., MultiWOZ, DailyDialog, or a custom assistant interaction dataset) and measure whether CFG reduces the rate of contextual contradictions, persona violations, or topic drift across turns. For long-form generation, use a narrative coherence metric (e.g., entity grid overlap [7], or human evaluation of topic maintenance) on completions of 500+ tokens from prompts like "Write a story about a dragon in Paris" with and without CFG. The hypothesis is that CFG's per-token guidance should prevent the slow drift that characterizes long-form generation without guidance, but the per-token cost (2× FLOPs) accumulates over long sequences, making the compute tradeoff steeper. This experiment would need to measure coherence per unit FLOP and per unit wall-clock time to provide actionable guidance for practitioners. A negative finding—that CFG's benefit diminishes over long generations because the unconditional distribution increasingly conditions on the generated prefix (which is itself prompt-influenced), making the difference vector shrink—would be an important boundary condition for the technique.

CFG vs. Contrastive Decoding: a direct experimental comparison. The paper acknowledges Contrastive Decoding [45] as the most closely related inference-time technique but provides no comparison. A rigorous follow-up would benchmark CFG against contrastive decoding on the same tasks, models, and FLOP budgets. Contrastive decoding requires two models (an "expert" and a "weak" model) and computes $f_s(x|y) - f_w(x|y)$ ; CFG requires one model and computes $\ell_{\text{uncond}} + \gamma(\ell_{\text{cond}} - \ell_{\text{uncond}})$ . A matched experiment would compare: (a) a 7B expert + 1.4B weak model with contrastive decoding vs. (b) a 7B model with CFG, both at matched per-token FLOPs (since contrastive decoding also requires two forward passes, one through each model). The evaluation would span the basic prompting benchmarks from Section 3.1 plus a long-form generation task where CFG's diversity tradeoff might matter more than contrastive decoding's. The key questions: does CFG's single-model approach achieve comparable or better prompt-adherence than contrastive decoding's two-model approach? Does the choice of "weak" model in contrastive decoding provide finer-grained control than CFG's unconditional baseline? What happens when CFG is applied on top of contrastive decoding (three forward passes: expert, weak, and expert-without-prompt)? This would map the inference-time compute allocation landscape more completely.

Practical Applications and Downstream Use Cases

On-device deployment with memory-constrained hardware. The most directly actionable implication of the FLOPs-matched analysis (Section 4) is that a model of size $N$ with CFG achieves comparable performance to a model of size $2N$ on a majority of benchmarks, while requiring half the VRAM. For deployment on consumer GPUs (e.g., an RTX 3060 with 12GB VRAM), this is the difference between being able to run a 7B model (with CFG) vs. being forced to use a heavily quantized or offloaded 13B model (without CFG). The training cost savings are equally significant: training a 7B model costs roughly half the FLOPs of a 13B model (assuming Chinchilla-optimal training where model size and data scale together). For organizations building on-device assistants, local code completion tools, or privacy-preserving language tools that must run entirely on user hardware, CFG provides a path to matching larger-model prompt-following without exceeding memory budgets. The latency cost (roughly 2× slower generation due to two forward passes per token) is the tradeoff, but for batch processing or non-interactive use cases where throughput matters more than latency, this is often acceptable. The specific numbers from the paper: LLaMA-7B with CFG at $\gamma = 1.5$ achieves 81.3% on LAMBADA, outperforming the 77.9% of PaLM-540B (a ~77× larger model), though this comparison should be contextualized by architectural and training differences between LLaMA and PaLM. More rigorously, within the LLaMA family, LLaMA-7B with CFG achieves accuracy comparable to LLaMA-13B without CFG on most benchmarks (Table 2, Figure 10).

Cost-efficient batch inference for benchmarking and data generation. Many organizations run large-scale batch inference: evaluating models on benchmark suites, generating synthetic training data, scoring millions of candidate completions for filtering or distillation. In these settings, throughput (tokens per dollar of compute) is the primary metric, and doubling per-token FLOPs is acceptable if it yields quality improvements that would otherwise require a larger model. The paper's results suggest that for tasks like sentence completion (LAMBADA), scientific QA (SciQ), and code generation (HumanEval pass@1), CFG provides accuracy gains that are competitive with doubling model size. A concrete workflow: a team generating training data for an instruction-following model could (a) use a 13B model with greedy decoding, or (b) use a 7B model with CFG at $\gamma = 1.5$ , achieving similar quality at lower total cost (since the 7B model is cheaper to host and the doubled inference FLOPs are compensated by the per-token cost reduction from the smaller model). The paper's machine translation results (Appendix D.1, Bloom-3B BLEU improves from 14.16 to 15.81 with $\gamma = 1.10$ ) and code generation results (CodeGen-350M pass@1 from 11.0% to 11.8% at $\gamma = 1.1$ ) provide specific data points for estimating the quality-vs-cost tradeoff curve for these specific tasks.

Prompt engineering diagnostics and automated prompt optimization. The vocabulary visualization technique in Section 5.3—computing the difference $\log P(w_t|w_{<t}) - \log P(w_t|\hat{w})$ at each generation step and ranking tokens by this difference—can be used as a prompt debugging tool independently of whether CFG is used for generation. A prompt engineer testing a new instruction or few-shot template can visualize which tokens the prompt is upweighting and downweighting, surfacing cases where important concepts are being suppressed or irrelevant concepts amplified. For example, if a prompt asking about "renewable energy adoption" is upweighting tokens about "fossil fuels" and "coal" while downweighting "solar" and "wind," the prompt engineer knows the prompt formulation is activating the wrong semantic associations. This visualization does not require CFG at generation time—the difference vector exists at $\gamma = 1$ —so it adds zero inference cost beyond the two forward passes needed to compute it once. For automated prompt optimization systems (e.g., DSPy, automatic prompt engineering), this difference vector could serve as a training signal: modify the prompt to maximize the probability of desired tokens in the difference vector and minimize undesired ones, providing a gradient-free optimization objective grounded in the model's own semantic space.

Overriding default behaviors in multi-tenant chatbot deployments. The negative prompting results in Section 3.4—75% human preference for CFG-guided system prompt adherence at $\gamma = 3$ without degrading user-prompt relevance—have direct application to multi-tenant chatbot platforms where a single base model serves multiple applications with different behavioral requirements. A customer service bot needs to be helpful and empathetic; a technical documentation bot needs to be precise and factual; a creative writing assistant needs to be imaginative and playful. Rather than serving separate fine-tuned models for each persona (each requiring separate model loading, VRAM, and maintenance), a single base model can switch behaviors by changing the system prompt ( $c$ ) and using the default system prompt as the negative ( $\bar{c}$ ) in the CFG formula. The paper's human evaluation shows that this approach works for stylistic modifications (enthusiastic, sad, seductive, rap-style responses) on GPT4All, and the same mechanism should apply to persona modifications, formality adjustments, and domain-specific behavioral constraints. The caveat is that this requires knowing the model's default system prompt to use as $\bar{c}$ , which may not be accessible for proprietary models. For self-hosted or open-weight models, the default system prompt is controlled by the deployer, making this a practical option today. The paper's results also suggest a $\gamma$ sweep would be needed per persona to find the sweet spot (Figure 5 shows the preference peak at $\gamma = 3$ for GPT4All; this value is likely model-specific).